RLP: Reinforcement as a Pretraining Objective
Authors: Ali Hatamizadeh
Syeda Nahida Akter
Shrimai Prabhumoye
Jan Kautz
Mostofa Patwary
Mohammad Shoeybi
Bryan Catanzaro
Yejin Choi
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Pretraining
Reasoning
Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: A verifier‑free reinforcement pretraining framework for language modeling.

Abstract: The dominant paradigm for training large reasoning models starts with pre-training using next-token prediction loss on vast amounts of data. Reinforcement learning, while powerful in scaling reasoning, is introduced only as the very last phase of post-training, preceded by supervised fine-tuning. While dominant, is this an optimal way of training? In this paper, we present RLP, an information-driven reinforcement pretraining objective, that brings the core spirit of reinforcement learning---exploration---to the last phase of pretraining. The key idea is to treat chain-of-thought as an exploratory action, with rewards computed based on the information gain it provides for predicting future tokens. This training objective essentially encourages the model to think for itself before predicting what comes next, thus teaching an independent thinking behavior earlier in the pretraining. More concretely, the reward signal measures the increase in log-likelihood of the next token when conditioning on both context and a sampled reasoning chain, compared to conditioning on context alone. This approach yields a verifier-free dense reward signal, allowing for efficient training for the full document stream during pretraining. Specifically, RLP reframes reinforcement learning for reasoning as a pretraining objective on ordinary text, bridging the gap between next-token prediction and the emergence of useful chain-of-thought reasoning. Pretraining with RLP on Qwen3-1.7B-Base lifts the overall average across an eight‑benchmark math‑and‑science suite by 19%. With identical post‑training, the gains compound, with the largest improvements on reasoning‑heavy tasks such as AIME25 and MMLU‑Pro. Applying RLP to the hybrid NVIDIA-Nemotron-Nano-12B-v2-Base increases the overall average from 42.81% to 61.32% and raises the average on scientific reasoning by 23%, demonstrating scalability across architectures and model sizes.

Published as a conference paper at ICLR 2026
RLP: REINFORCEMENT AS A PRETRAINING OBJECTIVE
Ali Hatamizadeh 11, Syeda Nahida A Shrimai Prabhumoye Jan Kautz', Mostofa Patwary', Mohammad Shoeybi', Bryan Catanzaro', Yejin Choi NVIDIA Carnegie Mellon University?, Boston University Stanford University abat www.amiiiiimeeeiiiaaaacam com, sprabhumoyeenvidia. com
ABSTRACT
The dominant paradigm for training large reasoning models starts with pre-training using next-token prediction loss on vast amounts of data. Reinforcement learning, while powerful in scaling reasoning, is introduced only as the very last phase of post-training, preceded by supervised fine-tuning. While dominant, is this an optimal way of training? In this paper, we present RLP, information-driven reinforcement pretraining objective, that brings the core spirit of reinforcement the last phase of pretraining. The key idea to treat chain-of-though as an exploratory action, with rewards computed based on the information gain it provides for predicting future tokens. This training objective essentially encourages the model to think for itself before predicting what comes next, thus teaching an independent thinking behavior earlier in the pretraining. More concretely, the reward signal measures the increase in log-likelihoon of the next token when conditioning on both context and a sampled reasoning chain, com- pared to conditioning on context alone. This approach yields verifier-fre -free dense reward signal, allowing for efficient training for the full document stream during pretraining. Specifically, RLP reframes reinforcement learning for reasoning a pretraining objective on ordinary text, bridging the gap between next-token pre- diction and the emergence of useful chain-of-thougl! reasoning. Pretraining with RLP on QWEN3-1. 1.3--1.7B-11 BB-BA lifts the overall average across an eight- cight-benchmark math-and-science suite by 19%. With identical post-training the gains compound, with the largest improvements on reasoning rravyning-heavy tasks such as AIME25 and MMLU-Pro. Applying RLP to the hybrid NEMOTRON-NANO. 1AANO-12B-V2 22 increases the overall average from 42.81 % to 61.32% and raises the average on scientific reasoning by 23%, demonstrating scalability across architectures and model sizes.
deaming-explorationn
INTRODUCTION
Large Language Models (LLMs) pretrained with next-token prediction loss have demonstrated broad utility, but this objective does not explicitly encourage long-range reasoning or integration with world knowledge. Consequently, state-of-the-art models (Guo et al.) 2025 Yang et al. 2025) rely on post-training objectives such supervised fine-tuning (SFT) and reinforcement learning with human or verified feedback (RLHF, RLAIF, RLVR) (Ouyang et al. 2022 Lambert et al., 2024 to
induce complex reasoning abilities. In contrast, human comprehension is not linear token-by by-n-b-yetteen process, but rather a parallel integration of input with prior knowledge (Baumgaertner et al. 2002
Hagoort et al., (2004) Metzner al. 2015). Current pretraining lacks such mechanisms, limiting the model's ability to reason and ground language in world knowledge during learning.
To fill this gap, we propose Reinforcement Learning Pre-training (RLP) which treats Chain-of-Thought (CoT) generation as an explicit action taken before predicting each next token. As shown in Fig [ the model first samples an internal thought, then predicts the observed token from the same context augmented with that thought. The training signal is the increase in log log-liieldhood of the observed token when the thought present compared a no- mo-think baseline. This yields a verifier free and dense reward that assigns position-wis wise credit wherever thinking improves prediction. Because
Work done during internship at NVIDIA Equal contribution

---

Published as a conference paper at ICLR 2026
Impact RLP
Paradigm Comparison
Aspect
Mext Token Prediction
rewinds based on information pre tain modis that rossin befon
Rayae Pyee
sparto (oann
luttier inproves accuracies after training
Pipeirns
RLP bridgee pretrs ring and resezring with scsaable damse mecert
NTP: Input Moxt sokon
RL. Inpul CoT soan Wwrttee-e Derta irtormation Ga Reward
Overview of
Nest Token Prediction
RE.P
No- No-Thine bassline
PEMAl- xx)
Dense nembiiary
plata, mako foot using nunlight
Photognth procese
plars, slgae, ane some tackets use to nann own fodd usiig
Psologythesa the proces
<thinb The sentence describaa how planes, algan. knowledig saye this process rallia on energy from sun, the nest soken thely sunlight
Thought Policy
paf [ect
Fo
"othinko
Information Gain Reward
GOT track next token
Figure 1: Visualization the RLP framework. A chain-of-thought is sampled before next-token prediction. Rewards are computed by contrasting the predictor conditioned on the CoT with No-think EMA baseline, yielding a verifier-free, dense signal. We list the advantages of RLP over the traditional pretraining objective (top right) and show the impact after end-to-end training (top left).
the signal is defined for ordinary text with teacher forcing, RLP reframes reinforcement learning for reasoning as reinforcement pretraining the same streams used for maximum likelihood.
Unlike post-training with verifiable rewards, which requires task-specific checkers or curated solutions, RLP is verifier-free: the signal is computed directly from log-evidence under the model and baseline, allowing uniform application to domain agnostic web-scale text. Compared to reinforcement pretraining via prefix- prefix-matching maaching rewards (RPT) (Dong et al.) (2025), which uses sparse binary reward and often relies on proxy- moxel filtering of "easy" tokens, RLP provides a continuous improvement signal every position and trains on the full documents. This eliminates the need preselect high-entropp tokens or couple training to separate small model. Prior RPT demonstrations also depend on distilled checkpoints with strong prior reasoning ability, which clouds whether the method helps base models. RIP is designed to shape thinking in base models by rewarding only those thoughts that measurably help next-token prediction.
This work makes the following key contributions: We introduce RLP, a verifier-free information- gain objective that augments next-token prediction by rewarding thoughts in proportion to their predictive utility. We develop a practical and stable training algorithm that interleaves reinforce- ment updates with standard likelihood training via group-relative advantages, clipped surrogate for thought tokens, and a slowly updated Exponential Moving Average (EMA) baseline. We provide theoretical guarantees linking expected reward to reductions in cross-entropy and to a computable lower bound, ensuring both interpretability and tractability. We conduct comprehensive experi- ments showing that RLP outperforms strong baselines, remains robust after strong post-training, generalizes across diverse corpora, and scales effectively to larger model sizes and hybrid architec- tures- tures-establishing it as a broadly applicable reinforcement pretraining objective.
Our empirical validation is comprehensive, assessing the efficacy of RLP along four key axes. First, we evaluate its performance relative to traditional next-token prediction baselines. On the QWEN3- 1.7B-BASE model, RLP outperforms continuous pretraining by +17% and RPT by nearly +4%. We show the advantage persists even when the baseline uses 35x more data to match FLOPs, confirming the gains arise from methodology rather than compute. Second, we demonstrate the robustness of these improvements, showing they are not transient. As shown in Fig T when subjected to an identical, strong post-training regimen, the foundational advantages of RLP compound, allowing our final model to surpass its conventionally trained counterparts by significant 7-8% margin. Third, unlike methods requiring narrow, curated datasets, RLP successfully extracts a powerful reasoning signal from diverse, general-purpose web curpora-establishing its versatility across data domains (Table 4). Finally, we confirm its scalability and sechitecture-agnostie power. When applied M'EMOTRON-NANON 2B-v2), RLP achieves staggering
to 12B hybrid
2

---

Published as a conference paper at ICLR 2026
35% relative improvement over heavily trained baseline while using just 0.125% of the data- testament to its remarkable data efficiency and broad applicability across LLM families and sizes.
2
METHODOLOGY
We introduce RLP, pretraining peerraining-time procedure that explicitly induces reasoning. As illustrated in Fig. [ RLP inserts short Chain-of- -finnol-T (CoT) before next-token prediction and measures how much that thought improves the model's prg-probability of the observed token relative baseline. This improvement, which is a log-likelihood ratio, a verifier- free, dense reward available at every position in ordinary text corpora. By valuing thoughts in proportion to their predictive benefit, RLP turns reinforcement pretraining into learning to think on the same data used for standard next-token training.
no-think
Parameterization and roles.
We separate the components for clarity:
Thought policy / predictor Te(c, | xee)) and po Tice, share exactly the same network and parameters 0. The network first samples a CoT and then, conditioned on the concatenated prefix (I<e, Ct), scores the next token 24.
No-think baseline Po (zy x<e)) (parameters Φ) an EMA teacher of the current network used to score the same token without any CoT channel.
Thus, there is single model that both generates the thought and predicts the next token given that thought; the EMA teacher provides the no-think counterfactual.
Classical next-token objective. Given text sequence a (Jo. .T) and position 6, the standard next-token objective for predictor 9n is
(NTP(T)
E [lo on (2t m<)e]]
For distributions p and g on the next token, we define Cross-entropy (CE) as
CE(p, 4) def
E log g(x)].
(2)
Using p² x22e) for the data distribution over T4, maximizing equation is equivalent to minimizing E CE(pp,. (.(. x<t)]] We include equation / only for context as our training does not include standard NTP loss ferm. Instead, RLP optimizes information-gal objective defined
below and updates parameters only through the tokens of the sampled thoughts.
2. REASONING AS AN ACTION
RLP augments next-token prediction with a sampled thought. At each position i, the policy draws latent CoT random variable
24 ( me)))
and we write a for realization. The network then predicts Ia with the reasoned scorer Po(- &eee, C.). As a no-think counterfactual we use P- x.e)), the EMA teacher queried on the same context without providing the COT.
EMA teacher instantiation and schedule. We instantiate the EMA teacher match the current model on the first batch (ø +- 0), and thereafter update it after each optimizer step via
< T (1-7)0,
0.999.
This choice makes P& moving counterfactual that is (i) current enough to provide informative comparisons and (ii) intentionally lagged to mitigate reward hacking. If the baseline were frozen, the counterfactual would drift too far from the evolving model; if it tracked the model without lag, the log-likelihood ratio would collapse toward zero and invite degenerate strategies. The post-update averaging yields one-step-lagged, smoothed teacher that stabilizes training.
3

---

Published as a conference paper at ICLR 2026
2.2 INFORMATION-GAII REWARD
With teacher forcing on the next token, define the reasoned and baseline log-evidence
(ct) po Fet,
(3)
SEMA log po(20 x<t).
(4)
The information-gen gain reward is the logliikelhhoo ratio
r(c)) Sprea(ee) SEMA:
which compares the reasoned scorer with no-think baseline on the observed next token. Rewards computed under teacher forcing for each I. When updating the policy, treal r(c)) as a constant with respect to 0 (no backpropagation through pe or P&): see 22.4
2.3 EXPECTED IMPROVEMENT IDENTITY
Proposition (CE reduction). For any fixed (Fet,
E [r(c)) CF(p", Pol (ee)) - CF(p", :pl c))- 2e-pp"
where p"( (. | Iet) the data distribution over Jy. Maximizing the expected reward therefore maximizes the predictive usefulness of the thought for the next token.
Proposition (Lower bound via marginalization over thoughts). Let To(#t (ee)) be the distribution over CoTs and define the collapsed predictor
pa(x x<<))=
E =imxo()
[po(x x<c,
Then for any realized 24,
< Po xec), and (() E(r(c)) s E log (0(x x
E Spread (t)]
po x(t)
The CoT-conditioned objective is thus a computable lower bound on the improvement one would obtain after marginalizing thoughts. Refer to 8.1 of the appendix for the proofs of the propositions.
2.4 RLP OBJECTIVE AND OPTIMIZATION
RLP optimizes the thought policy to produce thoughts that increase predictive evidence. Our training does not include the standard next-token loss in equation [ information-gain objective
Instead, optimize only the
J(Ø)
BEiN G C (:1re)) r((c)),
E
E
(6)
or, equivalently, we minimize the negative information gainn loss Lia(0) (G). Gradients are applied only the thought tokens; r(c)) ) treated constant (no backpropagation through po or p&)
Group-relative baseline (inclusive mean with correction). To reduce variance, for each context we sample G thoughts (c(() G Timl and use a corrected inclusive mean baseline. Let
G
F = Σ
G
j=l
We define the ad advantages
(149)
G
(6)
with no gradient propagated through
G
This multiplicative factor removes the (1 ) unbiased estimator with low variance.
shrinkage inherent to the inclusive mean, yielding an
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 RLP for next-token prediction with information gain
1: Inputs: dataset D, group size 2 2, clipping (ce, ch), EMA decay (0, 1), learning rate 17.
2: Model: single network with parameters 0 used both as (i) thought policy my and (ii) reasoned predictor po; EMA baseline Pa-
3: Initialization: mark & as uninitialized.
4: while training do
5: Set the behavior snapshot Oold f 0.
>> used for the current sampling pass
6: Sample minibatch {(x <ts 2zg
(b) (b)
)
D.
7: For each b, sample G thoughts ct ~ Tom. (- xtt) with |c (b.i)
2 1.
8:
if is uninitialized then
9:
D lazy init of EMA teacher Compute baseline log-evidence (teacher forcing, grad) SEMA per equation 3
10:
Compute reasoned log-evidence syeal and rewards (6..)) as per equation and equation
11:
12:
Group baseline fz(b) and A(b.s) (inclusive with correction; sg is stop-grad) per
equation
7
Per-token importance ratios and clipped surrogate for (b,,) with prefix prefix!! (b.1),
13:
Pu (b.i) = exp log Te(El0))) prefix - log Tomn (1(0.) prcfix!!.1))
L (b.5)
elip To!
En min
Pi (b.1)
sg (A((ii)))), clip(pu (b.6), Ee, 1 En) sg sgg(gg (A((.))))
14:
Policy update on thought tokens:
[(0) BG
Σ Eb=1 Σ d
θ - V)C(8)
15:
EMA update baseline: o - T o + - T)
16:
Output: trained policy/predictor (shared 6) and EMA baseline ø.
Per-token importance ratios and clipped surrogate. We update the log-probability of the thought
tokens with clipped surrogate. Let E!") be the u-th token in (8) and prefix (Iret, ( With behavior parameters Oola used to sample the thoughts, define the per-token importance ratio
_ = exp( log (e( prefix(") - log To.. (E(1) prefix("))
We write clip(p; ce, 1 + (n) for elementwise clipping and denote stop-gradient by sg(+). The surrogate loss is
Celip( (0) -E
min pbo sg(A()), clip(p!"): Ee, 1 En) sg(A()))
(8)
2.5 REWARD PROPERTIES AND GUARANTEES
Does thinking actually help? The reward r(c:) is positive exactly when the model that used the sampled thought assigns higher probability to the observed next token than the EMA baseline that did not think. In expectation over the data distribution, this equals the reduction in cross-entropy between reasoned scorer and the no-think baseline (Prop.
Positionwise credit at every step. independently at each position t as
Since the task is next-token prediction, the reward is computed
r(c))
Lo ()) log Po x<t).
Credit is attached exactly where the thought changes predictive probability, yielding scalar per token and removing the need for a learned value function or any external verifier.
Putting it all together. Algorithm composes the above pieces into single training loop. Specifi- cally, multiple thoughts are sampled per position and information-gaa rewards are computed against a moving EMA counterfactual. Group-relative advantages formed and the shared network is updated only on the thought tokens via the clipped surrogate in equation 8 In this case, the improvements originate from learning generate thoughts that systematically raise predictive evidence.
5

---

Published as a conference paper at ICLR 2026
Benchmark
Mbsse
McPT
MRLP
Mbase +Post
McrT +Post
MRLP +Post
AIME25
2.25
3.96
5.02
5.32
5.89
7.05
MATH500
48.45
57.52
58.48
61.92
62.70
64.30
GSM8K
51.16
72.85
74.48
78.22
78.70
80.50
AMC23
25.94
31.25
31.25
35.00
34.38
36.50
Minerva
15.30
19.03
21.19
25.30
26.10
27.80
MMLU
50.08
41.95
56.14
58.36
59.00
61.50
MMLU @ I[4]
44.85
40.00
52.18
56.00
58.53
61.00
MMLU-PPo
28.17
27.81
34.62
37.85
39.92
42.40
MMI.U-Pro@119)
23.95
24.61
30.80
36.53
38.49
41.30
GPQA
25.25
26.26
28.28
30.93
29.27
33.33
GPQA @ [[4]
27.52
24.75
27.02
31.52
30.01
34.97
Math Avg
21.35
30.77
31.74
34.29
34.63
36.03
Science Avg
34.50
32.01
39.68
42.38
42.73
45.74
Science Avg@ 1[4]
32.11
29.79
36.67
41.35
42.34
45.76
Overall
30.32
30.85
36.03
39.34
39.90
42.51
Table I: Quantitative benchmarks for -wen3-1.7B-Bar showing the impact of RLP. Shaded columns indicate RLP variants; "Post" indicates SFT RLVR post-training.
3 EXPERIMENTAL SETUP
We experiment with SWWEN--1.7B-BASE (Yag et al. (2055) and then scale our experiments to larger NEMOTRON-NANO- - (Nano 2025 model
We apply RLP diverse set of datasets across two settings: (i) SFT-style reasoning corpora, including a math-centric set (OmniMath (Gao et al. 2024) and mixed math + general- reasoning sets (OpenThhouhts (Guha et al., 2025). Nemotron Crossthink (Akter al. 2025 )); and NemmoroonCrossthinn
RLP.
(ii) general general-purpose purpose pretraining corpora, covering academic papers (ACAD), math textbooks (Math- Text), and open-ended web pages QA pairs from Common Crawl Weeb-Crawl)(Nam)) 2025). We train with RIP for 1B input tokens using general pretraining corpora (Drr) to evaluate its effect in an end-to-ent RLP can be applied for every token in a document but in our experiments we randomly select one token per document. Hence, the number of tokens for which the reward is applied is far less than IB.
pretraining pipeline. We denote this model as MRLP. Note that theoretically
Continuous Pretraining. To ensure compute equivalent comparison with MRLP. do continuous pretraining on the base model denoted by Mbase with the same tokens used in RLP. We denote this model McPT. This serves an additional baseline for our experiments.
Dost-Training All models undergo SFT stage on OpenThougghs data (Guha al. 2025). To further enhance, we apply Reinforcement Learning with Verifier Rewards (RLVR) using MATH dataset (Hendrycks et al., 2021b). This two-stage post-training pipeline provides an evaluation framework to verify that gains from RLP persist under strong alignment, while also revealing how much additional improvement can be achieved through subsequent post-training. For consistency, models are trained with identical SFT and RLVR receipes, ensuring that any observed differences in downstream accuracies can be attributed to the pretraining condition (Maase vs McPT vs MRLP).
3.1 EVALUATION METRICS
We conduct a thorough benchmark assessment using series of tasks using NeMo-Skillf
Details about hyper-parameters for each below phases and prompt used for RI.P can found
Appendix To
http///wwwIIIA//Mc---.ooo... Sk1118

---

Published as a conference paper at ICLR 2026
Benchmark
Mbase
MRLP
Mbase +Post
MRLP +Post
MATH500
79.95
78.68
83.47
87.05
GSM8K
72.31
85.98
94.22
94.90
AMC23
70.63
57.19
62.19
75.00
Minerva
22.61
39.48
40.76
42.78
MMLU
54.12
78.76
73.55
78.17
MMLU@ [[4]
48.01
79.48
75.23
77.90
MMLU-Pro
24.16
53.13
61.78
67.38
MMLU-Pro@@ [[4]
27.13
55.76
73.21
66.96
GPQA
25.25
39.90
41.41
48.00
GPQA@ [[4]
22.47
48.86
52.15
49.62
Math Avg
61.38
65.33
70.16
74.93
Science Avg
34.51
57.26
58.91
64.52
Science Avg@ I[4)
32.54
61.37
66.86
64.83
Overall
42.81
61.32
65.31
68.09
Table 2: Quantitative benchmarks for NEMOTRON-NANO- 12B-v2, showing the impact of RLP.
Shaded columns indicate RLP variants; "Post" indicates SFT RLVR post-training.
Math Reasoning (MATH AVG). We consider four diverse math benchmarks GSM8K (Cobbe et al.
202I). MATH-500 (Hendrycks et al. 2021c), Minerva Math (Lewkowyez al. 2022). AMC23. We
report Pass@I @ average of 8 runs for these.
Science Reasoning (SCIENCE AVG). For conceptual science and specialized knowledge, evaluate on MMLU (Hendrycks et al. 2021a) , MMLU-Pro (Wang et al. (024),, and the graduate-level STEM benchmark GPQA-Diamond (Rein al., 2024). For science benchmarks, we report the average greedy and @ I scores from 4 runs (SCIENCE AVG @ [4)).
4 RESULTS
Table the performance of QWEN3-1.7B-BASS under different pretraining and post-training objectives. First, RLP consistently outperforms both the Mbase and MMPTT across nearly all benchmarks, with especially strong gains We see that MRLP is relatively on average 19% and 17% better than Mhase and McPT respectively. This highlights the effectiveness of dense, verifier-free reinforcement signals for instilling reasoning capabilities during pretraining. Second, the benefits of RLP persist even after strong post-training (SFT RLVR). While all models improve after post-training, .MRLP achieves the highest scores with the overall average substantially higher than both Mbsse by 8% and McPT by 7% relatively. This indicates that RLP establishes robust reasoning foundations that are not washed out by downstream alignment but instead compound with post- pest-training. We observe particularly large gains in science domains, with MRI.P +Post achieving +-3 points over McPT +Post. This trend suggests that RLP not limited to mathematical reasoning but also generalizes effectively to other domains. The ability to strengthen performance in science benchmarks highlights that RLP fosters a broader class of multi- step oppanationnddrvenn reasoning skills, moving beyond domain-speciff improvements and pointing toward a more versatile foundation for reasoning in LLMs. Overall, the results demonstrate that RLP not only induces reasoning ability during pretraining but also synergizes with post-training, leading to models with stronger and more durable reasoning abilities than those trained with next-token prediction continuous pretraining.
saasoning-heavy tasks such AIME25 and MMLU-Pro.
Scaling Model Size and Architecture We further scale RLP to NEMOTRON -ATTRON--NNO-1130 12B-2
(Nano] 2025) (Mbsse). ), hybrid Trans---ransoommer language model of 12B parameter size. this comparison we take intermediate checkpoint of NEMOTRON-NANO-. 12B-12 trained till 19.8 trillion tokens and apply RLP for 250 million tokens only. Mbase on the other hand is trained for 20 trillion tokens. In addition, the setup used for QWEN3- 1.7 7B-7BB-A BASE in Table [ The results as shown in Table 2 confirms that, regardless of model size and families, RLP not only yields very large improvement the base
employ identical post-training pipeline (SFT - RLVR), mirroring
7

---

Published as a conference paper at ICLR 2026
stage (Overall 42.81% to 61.32%, 43% relative gain) but that these gains persist and continue to compound after strong post training. After SFT + RLVR, the RLP trained model improves from 61.32% to 68.09%, maintaining clear margin over the compute matched baseline (65.31%) The largest relative gains are in scientific reasoning as the Science Avg rises from 34.51% % to 57.26% at the base stage and further to 64.52% after post training. compared to 58.91% % for the continuously pretrained baseline. This pattern closely mirrors our findings on Qwen3, and demonstrates that RLP scales effectively both larger parameter counts and different architecture family, while remaining compatible with strong downstream aligmment. Furthermore, we validate that RLP scales effectively to larger backbones by applying to QWEN3- 1----BSB, where it improves the overall average from 60.66% to 65.00% after training on IB tokens, with particularly strong gains in scientific reasoning. Full results for this scaling experiment are provided the Appendix [2
RPT Comparison Following the experimental setup in RPT (Dong et al. 2025), we trained Mbase on both RPT and RLP methods for one epoch under tokens and flop matched com- pute budgets before evaluating on our benchmark suite. In the token matched setting, we trained both models on Omni-MATH (Gao et al. 2024) using the same number of input tokens. As we apply RLP to single token per doc- ument while RPT applied on multiple tokens per document, the number of tar- get tokens for which reward is applied is substantially larger for RPT.
Model
Math Avg
Science Avg
Avg
Token-Matched
MRPT
47.50
35.88
41.69
MRLE
49.62
37.07
43.35
Conversely, for the flop-matched training. Nemotron-
Flop-Matched
both models are trained CrossThink (as detailed in Appendix [2] for one epoch on the same data, and we ensure that the number of target tokens for which reward is applied is same in both training runs. As summarized in Table Ta- ble 3 under Token-Matched setting, RLP achieves uniformly higher aggregates: Math Avg improves by an absolute +2. (+4.5% relative), Science Avg by +1.19%(++.33)) relative), and Overall Avg boosts by +1.66% (+4.0% relative). In addition, under the flop matched setting, the improvement even more prominent. RLP achieves a 20. 20.12% relative improvement on average over RPT. Method- ologically, RPT applies reinforcement only tokens pre-selected by an auxiliary assistant via entropy filtering and optimizes sparse, binary next-token correctness signal that ignores the CoT content, limiting where the signal can be applied. contrast, RLP evaluates each sampled CoT by the information gain it provides for the observed next token and updates at all positions without an auxiliary filter which yields consistently better averages under the matched setting above. Crucially, this dense, per-token information reward supplies richer credit assignment than RPT's sparse binary signal and, in our matched experiments, empirically yields better performance.
MRPT
36.66
34.38
35.68
MRLP
45.95
38.76
42.86
Table 3: Token- and flop-matched comparisons of RLP and RPT using S.WEN3-1.7B-B... model.
ABLATIONS
Does RLP provide generalizable improvements across diverse corpora? A key advantage of RLP is its scalability to large, diverse corpora, unlike RLVR, which relies on small, curated reasoning datasets and raises concerns about generalizability. Prior work (Chen al. 2025 Setlur al. 2025) highlights the need for complex reasoning corpora to sustain RL improvements, but such datasets are costly to curate and impractical at pretraining scale. For these ablations, we apply RLP to QWEN3-1. 11--1..B-BASE for 200 steps- ceipssitiiizzing 770M input tokens- the rest of the setup fixed.
As illustrated in Table 4 RLP delivers consistent gains across all corpus families, eliminating concerns that RL. based pretraining only benefits curated reasoning data. Relative to Mbase average improves by 7-9% with strongest gains on Nemotron-Croosttinnk (SFT-style) and Web-Crawl (general- purpose corpora). Unlike prior work (Akter et al. 2025), where RL gains were limited math and weakened under mixed data, RLP achieves simultaneous improvements across all benchmarks, demonstrating genuine cross-domate transfer. Even purely non-reasoning general corpora such as web-crawl, RLP extracts a reasoning signal that scales with data diversity (Appendix [12). Table 4 illustrates that unlike prior work (Liu al. 2025b Zhou et al. 2025), RLP can be applied any
8

---

Published as a conference paper at ICLR 2026
Model
Dataset
Type
Math Avg
Science Avg
Science Avg@ [[4]
Avg
Mhsse
35.96
34.50
32.11
34.19
Nemotron- Crossthink [170M]
Equal Ioput Token
37.1
35.76
32.15
35.01
McPT
Nemotron- Nemotron-Crossthine Crossthink [6B]
Equal FLOPs
43.90
37.74
32.47
38.04
Dpr[IB]
PT Data Mix
45.34
32.14
29.33
35.60
OmniMath [170M]
46.48
40.27
37.54
41,43
OpenThoughts [170M]
SFT
47.64
40.84
35.88
41,45
Nemotron-Crossthink (170M)
49.76
42.54
37.78
43.36
MRLP
ACAD [170M]
47.68
40.59
36.87
41.71
Math-Texi [170M]
General
48.07
40,46
36.32
41.62
Web-Crawl [170M]
48.87
40.75
36.77
42.13
Dpr[IB]
PT Data Mix
46.35
39.68
36.67
40.90
Table 4: RLP across diverse corpora. RLP trained on six SFT-style and ceneral-purpose datasets yields consistent gains, indicating transferable reasoning from mixed/open-ended data.
data format like academic papers, textbooks, web-craw/ as well as SFT style data. Overall, RLP is scalable, aggnosicc pre-training augmentation that enhances both reasoning and accuracy.
Does the improvement sustain under compute equivalent baselines? A critical question is whether RLP's gains stem from its unique RL-based pretraining or simply higher compute. Standard next-token pretraining quantifies compute by input tokens, but RLP adds rollout costs not captured by this metric. For fair comparison, we evaluate against McPT baselines under: (a) equal Input Tokens Seen and (b) equal total Compute FLOPs. RLP fixed to Tap = 170M tokens; the token- matehe McPT [170M] continues pretraining on 170M tokens (Input Token), while the FLOP-matcher budget corresponds 6B tokens for CPT (MerT [sB])(see Appendix I.].
MRLP outperforms McPT trained on the same 170M tokens and maintains a clear advantage even against a compute-mate matched McPT exposed to 6B tokens (35x more data). Despite this disparity, RLP achieves a 5.3% gain on average (compare McPT Cemotron-Crossthin vs MRIP Gemotron-Crossthing
In Table 4]
[6B] [170M]), with consistent improvements across math and science benchmarks. These results show that RIP'ss gains stem not from more efficient use of compute, not larger budgets, validating the effectiveness of our approach.
Is RLP comparable to CPT with high-quality reasoning data? High-quality reasoning corpora have shown to substantially boost base model reasoning ability when used in continuous pretraining (CPT) or mid-training (Wang al. 2025 Gandhi al. (025). This raises the important question of whether CPT can match or even surpass RLP under such favorable conditions. To investigate this, we conduct CPT on both reasoning-centric, centrie, Nemotron Crossthink and general pretraining (Drt) datasets, each using 170M tokens. Our results in Table 4ishow that even with high quality reasoning data, RIP consistently outperforms CPT by a significant margin. Specifically, MRLP outperforms and 5% pre-training data mix
Cemotron-Crossthink
Mcrr, showing an average gain of 8% on Nemotron-Crossthin (DPT) on IB tokens. These results highlight two key insights. First, while CPT benefits from reasoning-dense corpora, it remains sensitive to domain skew-evident in the weak science accuracy on Dpr--whrree whereas RLP generalizes more evenly across disciplines. Second, the consistent margin by which RLP outperforms CPT, even in the presence of high quality reasoning data, underscores that the gains of RLP are not merely due to data quality but stem from the algorithmic design itself. This reinforces the conclusion that RLP provides generalizable mechanism for leveraging reasoning data during pretraining, complementing rather than being overshadowed by high-quality corpus selection.
Ablations on rollout count, completion length, and KL weight. Fig. 2 visualizes the trends across three settings: (a) rollouts, (b) completion length, and (c) KL. Please look into $11 for more detailed numbers and per-task breakdowns. More rollouts help up to = 16 (Overall 42.17%); = 4 and already reach 41.33 11138% and 41.95%, while G = 32 decreases slightly 41.75% (Fig. 2 Increasing completion length gives the largest gains. Specifically, overall rises from 11.50% at 64 to 42.17% at 2048, with Math/Science moving from 1.12% 21.88% to 48. 23..06%/36.22
(Fig. 2b). Extending to 4096 yields 42.21% at roughly twice the thought budget, so we default to 2048. Furthermore, a
KI. anchor does not help. Specifically, B 10-4 and 10-3 give 41 35% and 41 11.44%, compared to
9

---

Published as a conference paper at ICLR 2026
50
50
bek
30
Accurii
30
Moth Aug
Hoth AvB
Hath Avg
So orse Any
23
Sclance Ang
Ang
Ouatell Arg
Science
Basadire
10
Rocelira Scdenie
Rasodire Mart
Ence lon Ounrtl
Baselire Ovent
Blssdine Ovanc
258
1024 2013 0096
compiction longoh (L)
(a) Number of rollouts
(b) completion length
(c) KL coefficient
Figure 2: Ablations Qwen3-1.7B. Curves report Math/Sciencc//veraall averages. Dashed lines mark the base model.
42.17% at 3 = O, and also increases memory and step time (Fig.
2 We therefore use G = 16,
completion length 2048, and /3 0 in later experiments.
RELATED WORK
Next-Token Prediction.
Next-token prediction is the standard pretraining objective for LLMs:
predict the next word from prior context (Shannon, 1951; Bengio et al. 2003). Scaling with Transformers (Vaswani et al. 2017) enabled landmark and state-of-the-art systems (Radford et al.
2018: Brown et al.] 2020 Smith et al. 2022 Bi et al.] 2024 Nano 2025 Yang et al.) 2025). Anticipating tokens across corpora induces syntactic, semantic,
pragmatic structure that transfers broadly. Alternatives include masked language modeling (Devlin et al. 2019) and span corruption (Raffel al. 2020), but next-token prediction remains dominant for its alignment with left-to-right generation and strong downstream accuracy across tasks. In this work. we add verifier- free dense reward during pretraining that leverages reasoning before prediction.
Verifier-Free Rewards in Post- Dost-Training. Recent work explores verifier-fres rewards. Yuan et al. (2024) uses iterative DPO where, after SFT, the model judges its own candidates to create preference pairs. Liu et al. (2025b) trains with incentive RL on SFT corpora. Zhao et al. (2025) proposes RL from an internal feedback while using the model's ' confidence as reward. RLP. P. in contrast, is a GRPO-style pretraining objective. It operates any text data including web-crawl. academic papers and SFT datasets and optimizes continuation quality beyond next-token prediction. Because these methods target post-training policies, direct comparisons not well-posed.
7 CONCLUSION
We introduce RLP, a reinforcement pretraining objective that rewards chain-of-thought by its infor- mation gain for next-token prediction. Unlike traditional approaches that defer RL to post-training, RLP instills reasoning during pretraining. yielding gains that persist and compound after alignment. Experiments across datasets, domains, and architectures show that RLP consistently outperforms compute- compute-matched matched baselines and scales efficiently to large hybrid models, establishing reinforcement pretraining as principled and general alternative to likelihood-only training.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett. Mojan Javaheripi. Piero Kauffmann, James R. Lee. Yin Tat Lee, Yuanzhi Li. Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL
waarg/absst / 2412 08905
Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturima, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-crossthinkk: Scaling self-learnig beyond math reasoning, 2025. URL https://
arxiv. wwxivoorg/abs///ooo /2504 13941
10

---

Published as a conference paper at ICLR 2026
Annette Baumgaertner, Coreelis Weiller, and Christian Büchel. Event-related fmri reveals cortical sites involved in contextual sentence integration. Neuroimage, 16(3):736-74) 745, 2002.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb): 137--111555555 2003.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02955 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason nemetronn Advancing math and code reasoning through reinforcement learning, 2025. httpt/ttss://///tteeeaa org/ abs 2505. 6400
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL prttps://arxiv,.
abss/210.. 14162
AL, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li. J. L. Cai. Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang. Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu. Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu. Shengfeng Ye. Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, W. I.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li. Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025a.
URL pottps://azxxe
org/ abs /2001.25011000 12948
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo. Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan
11

---

Published as a conference paper at ICLR 2026
Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang. Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu. Zilin Li. Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025b. URL https:tt////c httpp:://axxi...tttt
www.v. org/ abs 2 2412 19437
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers). pp. 4171-4186, 2019.
Qingxiu Dong. Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, and Furu Wei. Reinforcement pre-training, arXiv preprint arXiv:2506.0800 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL
hww.g//bb///// 01307
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang. Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni- math: A universal olympiad level mathematic benchmark for large language models, 2024. URL httts:://rxxi.. /tttts:://axxl..rrr//b////////// wwtp:////graaa///////////////// 220.0//9811.90015
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang. Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https: //aa:iv//xxvvvi org abs aaaaaaas22555 2506. 04178
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Peter Hagoort, Lea Hald, Marcel Bastiaansen, and Karl Magnus Petersson. Integration of word meaning and world knowledge in language comprehension. science. 304(5669):438 441, 2004.
Seungwook Han, Jyothish Pari, Samuel J Gershman, and Pulkit Agrawal. Position: General intelli- gence requires reward-based pretraining. In Forty-second International Conference on Machine Learning Position Paper Track.
12

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://
arxiv. abs /bw.abb....0000000
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021c.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasonner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025, URL
25tp:://.bb/2253.3222 24290
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https //github. wgw
Albert Q. Jiang, Alexandre Sablayrolles. Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel. Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL URL httph:tt///attt wtg/ wwwp///gg.....aooaeae /2w.. 04088
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15:: 151224, 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models, 2025a. URL //apivi/////vv...//a http://ww....ogas////a/aaaaaaaas 2205..255... 24864
Wei Liu, Siya Qi, Xinyu Wang, Chen Qian, Yali Du, and Yulan He. Nover: Incentive training for language models via verifier-free reinforcement learning, arXiv preprint arXiv:2505.16022, 2025b.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL htt.:httt///prreeeeeeeee http::///wprreeeeeeeeoe.. net. htr//forunee forum? d=
Bkg6RiCqY7
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol- preview with a 1.5b model by rl. DeepScaleds wwweeeeSrupsssigg--ee 2025. Notion Blog.
not cttttiiionssne///eooooo..o
1- Model boweel-b---aalinnnggggg------aa- RL- 1ww.169989221468000
Paul Metzner, Titus von der Malsburg, Shravan Vasishth, and Frank Rösler. Brain responses to world knowledge violations: A comparison of stimulus-and fixation-triggered event-related potentials and neural oscillations. Journal of Cognitive Neuroscience, 1015):0 2015.
NVIDIA Nemotron Nano. Efficient hybrid namba-transformeet reasoning model. arXiv preprint arXiv:2508. /4444, 2025.
Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brun- dyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das. Ayush Dattagupta, Olivier Delalleau, Leon Derezynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining
13

---

Published as a conference paper at ICLR 2026
Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabalesh- warkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang. Zhilin Wang. Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report, 2024. URL
org/ http//bbb/222...0 117044
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024. URL http::tttpppteeeeeeeee net/Eorum?id Tidddddd-i77789333
Amrith Setlur, Matthew Y. R. Yang, Charlic Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning explore enables extrapolation of test-time compute for Ilms, 2025. URL hhttstts:: http://axxvvvvv org abs / 2506. 09026.
Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1): 50-64, 1951.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im Im: Training multi-billion parameter language models using model parallelism. arXiv preprint 10000019,,,0803,, 2019.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turinn turing nig 530b, a large-scale generative language model. arXiv preprint ar arXhv:2201.11990, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024. URL wwtp::ttps/// http//aaxivv.v wwtpw/wwwo//./////...... 01574
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Revisiting mid- mid-training in the 2025. Notion Blog.
era of rl scaling. httppss///tnnuzllccoooooo/
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint ar Xix:25055009388 2025.
Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.100 3, 2024.
14

---

Published as a conference paper at ICLR 2026
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint WXXv:2505.19599) 2025.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21499: 2025.
15

---

Published as a conference paper at ICLR 2026
8 APPENDIX
8.1 PROOFS
In this section, we provide the proofs supporting the methodology $2 We first prove the tokenwise cross-entropy (CE) reduction identity (Prop. D). then the lower bound via marginalization over
thoughts (Prop. 2 Finally, we state and prove Prop. 3 which formalizes the positionwise-credi
claim described in | under teacher forcing, averaging the expected tokenwise information-
rewards across positions recovers the expected per-token sequence-level CE improvement.
For convenience, we recall the key definitions from the main text: the reasoned and baseline log-evidence (p)e( log Neet; C4) and SEMA log Po x<t)) (equation 3 the
information-ge -gan reward r(e)) (pre(())) - SEMa (equation 5); and the cross-entropy CF(p.)) q) def Egwpl log ((x)] (equation 2
8.2
PROOF OF PROPOSITION (EXPECTED IMPROVEMENT IDENTITY)
Fix the context <<t and realized thought ct and let p (z) == p' x<e) denote the data distribution over xę at this position. By the reward definition equation together with equation
Proof of Proposition]] 7
r(c)) = log Po(ar Zeer 9t) - log Po( x=x).
Taking expectation with respect 24 P and using linearity of expectation,
E
[r(c)]=
E se-ppL
log (x (<2),C))
E zempp
log xx(e
so each expectation of By the definition of cross-entropy equation 2 CE(p, q) Ex-pl- log g (x)), a log- log-liieli equals the negative cross- coos-entropp::
E #e>pp
log
(e(r))) ci] CE(P', pol x)))
log [EE)
CD(p', Dol. x<t)).
me-pp
Substituting into the previous display yields
E
[r(ct) CF(p" .pol x<ee))
CF(p". xe),
xp-p:
which is the desired identity.
8.3
PROOF OF PROPOSITION 2 (LOWER BOUND VIA MARGINALIZATION OVER THOUGHTS)
Proof of Proposition
2 Fix (I)), and recall Spred(a) log (xt &ee,, Cx) and po(x (ree) =
(a Pe Tet, 2)).
(i) Jensen bound.
Conditioning on (Ieee 2.) and taking expectation over e, ~ Te(. xer))
Beeewo/SSped (c)) Ee, log pa (een) ct) s log lee po(a. ( ca) log ( (2t x<());
where the inequality is Jensen's inequality applied to the concave function log(-). This proves (i) pointwise for the realized 2t.
(ii) Bound on J(O). By definition of the reward in equation 5 and teacher forcing (see equation 3).
Pa(x x<() Po (w+ xxx))
(0) 18 Ee [ppeacc))) - SEMA
< E log (x
x - Po x log
where the inequality uses part (i) and the outer expectation is over (I<e, 2x) ~ D. This proves (ii).
Tightness. Equality in (i) (and hence in (ii) holds precisely when po (x Øeee 0e) almost surely constant in e, under ( (- 2e) (e.g., when the predictor ignores the thought when the thought policy is degenerate).
16

---

Published as a conference paper at ICLR 2026
8.4 SEEEWWISE-T-SEEUEENNE CONNECTION UNDER TEACHER FORCING (POSITIONWISE CREDIT)
This subsection formalizes the claim 22. that summing positionwise CE improvements recovers the sequence-level (per-token) improvement. The following proposition is new to the appendix and not required elsewhere; clarifies how tokenwise rewards aggregate the sequence level under teacher forcing.
connection under teacher forcing). Let x (Z1,.... IT) be drawn from the data distribution p" (x) and fix a policy ne (cel x<t)), the reasoned scorer pe (--(1a I<t, ct). and the no-think baseline Pa(- xet). Define the sequence-level (per-token) cross-entropy for the baseline and the (stochastic) reasoned scorer by
Proposition 3 Bakeewwssett--eqqeee
CEeea(p' (po) BEND
log (o Rea)
T
T
(F (Po((T (P((T)
E. EREN
(e t=1
[log s2c ci]
T
Then the average over positions of the expected tokenwise information-ee gaiiioonngaininn rewards equals the per-token sequence-level CE improvement of the reasoned scorer against the baseline:
T
E
Eeg Cel-mall (1ecola)) = CE .Po) - CEsoo(p" . pol
T
1=1
Proof. (i) Conditional independence under teacher forcing. At position t, teacher forcing samples the target token from the data channel while the thought is sampled from the policy given the same prefix:
p2-pp(( xee));
m xet).
Hence
p( ve x<ee) (e x<ce) p" (os ret), i.e. a I x Tec.
This implies Ex,-p- (1226)))
E (f
(ii) Positionwise CE reduction. By Proposition I for any fixed (x<e, a),
Excopt CE(p", Po (<))) CE(p", pp<( (<n.
Taking expectation over a ~ n. x<ee) and using linearity of expectation gives
Ec, [r. CE(P"
x<)))) [p(('ppp'>>>p Ee, CL(P"", pp xee,, c)).
(iii) Sum over positions. Average the identity in (ii) over 1,..., T and over D:
T
En
E.,
E.,se,C[rro [r(a))]
E.
T
t=1
T CElp", po( m<))
T
E., CF(p', p Tees ct))
Ex
En
T
T 2m1
By the definition of cross-entropy in equation and the chain rule for likelihoods,
Enrop' (+Ixe)) log (x 220)] CE(p", . x2()),
and similarly for the reasoned scorer inside the -xeeecctaaion. Therefore the two on the right are exactly CEsec(p" ,Do) and CEsen "o"".poorrp (Tol)))) as defined above, yielding the claimed equality.
17

---

Published as a conference paper at ICLR 2026
9 WHY RELATIVE ADVANTAGES Do NoT REWARD BAD THOUGHTS
9.1 PROOF OF MONOTONIC IMPROVEMENT
may seem paradoxical that, when all thoughts perform poorly (r(e)) 0), the group-relative formulation still labels one "better" and reinforces it. Does this mean the model is being trained favor bad reasoning? We demonstrate that, mathematically, this mechanism sound: the update remains an unbiased gradient step J(), ensuring monotonic improvement even in such cases.
1.Objective.
For context Tet and target token Zz:
J(O) Rewn [r(e),,
r(c) log po Fett c) log ( xee)).
Maximizing J reduces cross-entropy versus the no-think baseline. Ignoring stop-gradients, the policy gradient is
VeJ = Eowng (r(c) Ve log Te(c).
(10)
2. Group-relative advantages are unbiased. We draw G2 2 thoughts (.),...,, To and form
7 == 1 Σ
C r(c6))
(11)
j=1
10 G-1 ((()))) -r).
(12)
Let μ = E[r(c)]. Then
([A() c00) = -r(c()) 14
(13)
and
E G
(1) V log l(
VoJ(O).
(14)
i=1
Hence, the estimator is unbiased. Even if all rewards are negative, the update follows the correct gradient direction.
Why positive advantage for the "least-bad" rollout correct. As the model learss, it gradually increases the probability of generating thoughts that help prediction and decreases the probability of those that do not. This process, known the replicator dynamic, captures how relative advantages drive steady improvement over time:
*(c) = x(c) [r(c) -4],
(15)
whose improvement rate is
d
dr [(((T)) =Vrr(r))a 0.
(16)
Even if all r(c) < 0, shifting probability mass from more-negative to less-negative thoughts increases J. Thus, positive advantage for the least-bad thought reflects correct relative improvement, not misaligned reward.
Monotonic expected improvement. size
With unbiased gradient estimator 9 Rs V and small step
E + ag)] ~ + a J(0),
(17)
ensuring monotonic improvement in expectation.
18

---

Published as a conference paper at ICLR 2026
5. The gradient does not blindly increase harmful thoughts. A remaining concern is that thought with negative reward r(c) < might still receive a positive advantage A(c) > it is simply less harmful than its peers, apparently encouraging bad reasoning. However, the gradient update does not blindly amplify such thoughts; reallocates probability mass among them in a way that improves the expected objective.
First, because the advantages are defined as
A = (r(c)
with f= c Σ r(( j=1
the total E; A(c(t)) = O. Hence, even if every reward negative, the update is sero-sum: probability increases only for thoughts that are less negative than average, while it decreases for those that are worse. This shift raises the expected reward (0) because the expected improvement rate is
d ((((T)) Var, [arx[r(c)] 2 0.
dr
Thus, the method performs relative reallocation and guarantees monotonic ascent in expectation.
Second, a positive advantage A(c) > 0 does not deterministically increase the corresponding r(c) on the next update; it increases it in expectation. The policy gradient on thought tokens,
VoLiG x A(c) V log (c),
acts on the relative usefulness of each thought, not absolute reward value. Over repeated steps, the model raises the log-evidence log (0x Izce, c) for those thoughts that contribute more to prediction, thereby increasing their expected r(c) relative to the slowly moving EMA baseline Ps.
Third, the EMA baseline prevents artificial reward inflation. Because Po lags behind 0 through slow exponential moving average, any transient or spurious improvement r(c) dissipates as the baseline catches up. Sustained positive advantages arise only when the model genuinely improves predictive likelihood relative to the no-th no-think counterfactual.
Finally, while positive advantage can momentarily reinforce thought whose raw reward remains negative, this update is not pathological. simply redirects probability toward the least harmful reasoning pattern available, reducing overall loss. Over time, these relatively better thoughts typically evolve into genuinely helpful ones as their predictive evidence increases, ensuring that the training process remains stable and aligned with maximizing (()).
9.2 NUMERICAL ILLUSTRATION OF RELATIVE ADVANTAGE UPDATES
To make the abstract dynamics more concrete, we present a simple numerical example showing how the group-relative advantage mechanism improves the expected objective J[;;; r) even when all rewards are initially negative. Note that in this illustrative example we denote the expected reward as J(x;; r) to emphasize its dependence on the discrete policy over thoughts ı and fixed rewards r;- Conceptually, this corresponds to the same information- objective J(0) introduced in the main text, expressed here in a simplified form.
We consider four sampled thoughts cı, c2 c3, C4 with policy [TTI, m2, T3, m4). initialized uni- formly. For each thought, the information- reward is
Fitt, Gi) log Pa Fee)); and the group size is G with mean reward
r; log (
Σ ri. The gloup-relative advantage is
A (r: F) = 5 (r:-)), -),,
and we assume each thought has length |cl| = 4 so that token weight is A,/4 The policy is updated by an exponentiated-gradien (replicator) step
Nnw (i) x (ita(i) exp(nA.), with r 0.5,
19

---

Published as a conference paper at ICLR 2026
and the expected objective is J(T:)) = Ex T.P.
Although a positive advantage can momentarily reinforce a thought whose raw reward ry is still negative, this update is not pathological. Because advantages are computed relative the group mean, a positive A, simply indicates that c; is less harmful than its peers. Increasing its probability reallocates mass away from worse alternatives, thereby improving the expected objective J. Over subsequent updates, the model typically adapts to make these less-harmful thoughts genuinely helpful, raising/ ri expectation.
Iteration 1: all thoughts are harmful (r; < 0), but one is least bad.
(0) [0.25, 0.25, 0.25, 0.25], (1) [- 0.80, - 0.60, - 0.50, 0.300
Mean and advantages:
,()) = -0.55,
A 0.3333,
0.0600,0 +0.667,400066
Per-token weights: A | 0.0833, -0.0167, +0.0167,+0008333, Note that ed has 11 -0.30 < O yet receives a positive advantage A1 +0.3333, so every token in e₁ gets positive gradient. Policy update with 17 0.5 gives
(11) x π (1)) (exp((0.4 = 00.210,, 0.2566, 0.2932),
yielding ;((())())))))) -0.5000 and J( (1; (1) -0.5284. This is a small but consistent
improvement.
Iteration 2: dense updates improve r on Ca, Ca.
(2)) [00.00 -0660, -0.35,00.10) -0.10] Update:
p(2) 00.4625,
A [-0.4500, 0.1833, +5.1500, +0.4833).
(²2) x (exp((00.)) = 0.2113,, 0.2668, 0.3601].
Expected objective: ((1(1);((1))) = -..411,, (2) = -0.3856.
Iteration 3: the least-bad thought becomes genuinely helpful.
(3) [-0.80, -.600, -0.20, +0.05).
(3) -0.3875,
( -..55000 0.2883,, +0.5833).
Policy update:
0.272,00000007,244444440,, and the expected objective improves again: (3) = 0.2991,, ([x(()); = 0.2244.
π (3) x π (2) (c = (0.1127,0.168)) 0.2772, 0.4420].
As seen in the above, in Iteration 1, all rewards negative, yet C2 (the least bad)has positive advantage, showing how the dense loss pushes probability toward less harmful thoughts and increases
J. Since rewards tied to log-evidence, these positive gradients directly improve the corresponding r(c) values, leading to less-negative and eventually positive rewards in later iterations.
10 EXPERIMENTAL SETUP
RLP: We employ RLP on both base and intermediate checkpoints using diverse datasets. To facilitate this, we use Hugging Face (2025) as the RL training backbone and deploy training using 32 H100 80GB SXM5 GPUs for 170M to 10B tokens. We train the base models with key settings including a constant learning rate of 1e 1e-6, batch size of 512 and a maximum context length of 2048 tokens. Each generation step contains 512 unique prompts sampled from the dataset, and performing 16 rollouts with temperature 0.7. We set KL coefficient to O across all runs.
Continuous Pre-training: We continuously pretrain the Mbasc model using both general pre- training and specialized post-training corpus to draw comparison between pretraining and RLP training objective. For this experimentation, we use Megatron-LM (Shoeybi et al., 2019) as the pretraining backbone and continuously train on 32 H100 80GB SXM5 GPUs for 170M to 10B tokens depending on the data size and comparison requirement. During training, we use the AdamW optimizer (Lohhhhlovv & Hutter 2019) with 3n 0.9. 3, = 0.95 and weight decay of 0.1. We use 2- waw tensor and pipeline parallelaalllisiee parallelism to train the model. We set the maximum value of learning rate to le minimum le -e and use batch size of 6M tokens with 8192 context length.
20

---

Published as a conference paper at ICLR 2026
For supervised fine-tuning (SFT), the OpenThoughts dataset (Guha et al. 2023). We filtered examples that did not include a final answer. With this filtering scheme, the total number of samples for SFT post-training is 45, 6024. For RLVR, used the The Mathematics Aptitude Test of Heuristics (MATH) dataset (Hendrycks et al. 2021b) with 7, 500 examples. This dataset includes problems from various subjects such algebra, geometry, number theory and precalculus. We trained models in all RLVR experiments for epoch with a global batch size of 1024 and used cosine annealing and an initial learning rate of le-6.
Post-Training
Prompt Given a context Tic we ask the model to reason about the target token using the following prompt, p.
System Prompt,
"You are continuation iittiiiuation-nnd-rre -nd--andd---as reasoning assistant. You receive the prefix of a context, problem, solution, or derivation. First, briefly think between < //tink>> and < //thitiii /think> about what should come next. Then, after < /think>> continue the text in the SAME style as the prefix (notation, LaTeX, tone), focusing on the next few steps rather than jumping to final boxed answer. Do not restate the question or add meta commentary; simply continue the content."
11 EXTENDED ABLATION DETAILS
Table S.I reports per-task accuracies for each setting, and Fig. provides the corresponding curves for (a) rollout count, (b) completion length, and (c) KL coefficient. Unless stated, each sweep holds the other two dimensions at the best configuration (16 rollouts, completion length 2048, ,3 0).
Rollout count. Increasing G improves accuracy up to G = 16, where Overall reaches 42.17% (from 34.03%, + points). The largest taskwise lifts = 16 relative the base are GSM8K (+222.96), MATH--50 (+ 13.85), MIVA (+7.20), MMLU (+6.35), and MMLU-PRO (+6.20), while GPQA unchanged (27.51 vs 27.52). Moving from G
to = 32 slightly lowers Overall to 41.75 (-0.42), driven mainly by GPQA (-2.13), with other tasks nearly flat (e.g.. MMLU-PRO +-0.79, MMLU 00.24). This suggests diminishing returns once the group-relative estimator is already well-sampled.
Completion length. Capacity on the thought channel dominates performance. Very short completions underperform sharply: at length 64, Overall is 11.50 and Math averages 1.12. Increasing to 512 raises Overall to 24.65 and Math to 22.63. The main jump occurs between 512 and 1024 (Overall +14.24 38.89; GSM8K +88.55; MATH-500 +36.85). Extending 2048 adds a smaller but consistent gain (Overall 42.17, +3.28 over 1024; Math/Science 48.06/ 06/36.222 //366/.). Pushing to 4096 gives only marginal change (Overall 42.21, +0.04; small taskwise shifts such as MMLU-PRO +0.64 and GSM8K 0.62), so 2048 is the preferred trade-off.
KL coefticient. Adding a token-level KL toward a fixed reference does help overall. At 3 10-4 and 10-3 Overall is 41.35 and 41.44 (-0.82 and -0.73 vs 3 = 0). There are isolated improvements (MMLU-PRO +1.43 at 10-4, 44; AMC23 +1.88 10 10-3), but these are offset by broader declines (e.g., GSM8K - 1.26 and 2.82; GPQA- -2.01 and -0--1. -..51). The KL term also increases memory use and step time. We therefore keep 3 = the main recipe.
In summary, the appendix table provides the taskwise breakdown behind these trends, and the figure shows the smooth saturation with rollouts, the strong length-driven regime change between 512 and 1024 tokens, and the lack of net benefit from KL.
12 ADDITIONAL ABLATIONS
Does the improvement sustain if we make Pretraining compute equivalent to RLP? For both comparisons, the configuration for RLP remains fixed, based on a budget of Timp = 170M input tokens. First, we establish baseline by continuing the pretraining of the base model on an identical 170M tokens (Base + CPT, Input Token). Second, to create a equiveeuiv bascline, we first approximate the total computational cost of RLP. The effective token budget, T flop- be estimated
21

---

Published as a conference paper at ICLR 2026
Table S.I: Ablations on rollout count, completion length, and KL weight with QWEN3-1, 1. 1.7B-BASE. All numbers denote accuracy (%)
Tasks (%)
Macro avg (%%
Model Variant
MATH500
GSMSK AMC23
Mincrva
MMLU
MMLU-Pto
GPQA
Math
Science
Overall
Baseline
Qwen3-1.7B-Bal
48.45
54.16
26.94
15.30
44.85
23.95
27.52
35.
96
32.11
34.03
Ablation: roliouts
nam rolloutse4
59.45
74.79
33.44
21.78
50.83
28.81
26.52
47.37
35.39
41.38
num rollouts=88
61.70
76.93
30.62
22.06
30.88
30.53
26.77
47 83
36.07
41.95
oum_rollouts=16 num_rollouts=32
62.30
77.12
30.31
22.50
51.20
30.15
27.51
36.29
42.17
60.45
77.26
22.29
50.96
30.94
47.74
Ablation; completion longth
completion.lenggleeb
1.00
completion. Jength= 128
1.73
3.17
0.94
0.05
29.04
13.94
12:37
1.47
18.45
9.96
2.95
13.86
2.81
0.16
37.19
17.09
15.15
5.02
23.14
14.08
21.35
46.58
16.25
42.27
19.82
17.93
22.63
26.67
24.65
complction www.ghhooooootittoooo 1024
58.20
75.13
28.80
20.47
48.36
38.89
completion length=20488
62.30
77.12
30.31
22.50
51,20
30.15
27.51
48.06
36.29
42.17
Sumplelion.eength--00055
62.00
76.50
30.60
22.80
51.30
30.79
27. 27.27
47.96 98
36.45
42.21
Ablation: KZ weight
10
61.35
75.86
28.00
21.50
51.00
31.58
25.50
46.68
36.03 03
41.35
10
60.90
74.30
19
20.73
50.73
30.80
26.00
47.03
35.84
41.44
B =
62.30
77.12
30.31
22.50
51.20
30.15
27.5
48 06
36.29
42.17
Model
Dataset
Math Avg [[8]
Science
Science Avg [[4]
Average
Musse
35.96
34.50
32.11
34.19
Only
48.23
41.64
36.77
42.21
MRLP
Only Science
49.17
39.65
38. 36.26
42.36
Combined
49.76
42.54
37.78
43.36
Table S.2: Ablation on math, science, and combined domains. RLP shows particularly strong generalization in presence of multi-domain data.
by summing the tokens used for gradient updates (Tinp) and the tokens processed during the rollout phase:
Trtpp (n x loey bs ittrs) Timp
where n the number of rollouts per instance, leeg is the sequence length, bs the batch size and iters is the number of steps RLP has gone through. This calculation re- sults in an effective budget of approximately 6B tokens for our model. We therefore train second, more powerful CPT baseline 6B tokens (Base + CPT, Flop Usage), holding all other hyperparam- constant.
Benchmarks M
MRLP
MATH500
78.81
81.15
GSMBK
90.36
94.04
RLP resonates well in presence of multidomain data. Recent works have shown tremendous improvement in rea- soning tasks, particularly in mathematics, through RLVR (Liu et al. 2025a Luo et al. 2025 Hu et al.) 2025). How- ever, these methods are often tied to the complexity of queries, limiting their scalability. To draw a parallel, we evaluate RLP on Nemotron- Crossshhin using different blends of math and science data. As shown in Table S.2 training only on math yields substantial math improve- ments, but comes at the cost of weaker generalization to science. Conversely, training only on science improves science accuracy, but underperforms in math compared to math-only training. Strikingly, combining both domains provides the best overall average, indicating that RLP is able to leverage complementary signals from multiple do- mains without diluting the benefits within each. This suggests that RLP not only scales beyond
AMC23
55.94
57.81
Minerva
37.96
40.26
MMLU
76.56
80.59
MMLU@ I[4]
74.03
79.31
MMLU-Pro
$9.20
65.53
MMILU-Proo1114 1[4]
54.00
61.75
GPQA
44.44
48.15
GPQA@ [[4]
40.40
44.70
Math Avg
65.77
68.32
Science Avg
60.07
64.76
Science Avg 1[4]
56.14
61.92
Overall
60.66
65.00
Table S.3: RLP training with QWEN3- 14B-BASE model.
22

---

Published as a conference paper at ICLR 2026
single-domain specialization but also thrives in multidomain settings where diverse reasoning styles reinforce one another.
Effectiveness of RLP with Scaling LLM. Validating RLP on larger, state-of-the-ar model sizes is indeed essential to confirm that our gains hold as parameter counts increase. To address this, we conducted a new set of experiments applying RLP to the M = -WWNN-114--BBR general pretraining corpus (DPt) for IB tokens. As shown in Table S.3) RLP delivers substantial improvements even on this stronger, significantly larger baseline. Applying RLP improves the overall average from 60.66% to 65.00%, with particularly notable gains in scientific reasoning where the average score improves from 60.07% to 64.76%. These results confirm that the dense, verifier-free signal provided by RLP remains effective at scale, success- fully extracting reasoning capabilities that are not fully utilized by standard pretraining alone.
model and
training
How early RLP can be applied? Previously, we have confirmed that RLP can be integrated to intermediate checkpoints from last stage of pretraining. However, it unclear whether the gains sustain we pick a very early checkpoint for RLP training. Inspired by the find- ing of Han et al.] and to study how early RLP can be introduced, additionally evaluate a much earlier check- point. Concretely, we take a NEMOTRON-NAL -ANONO- 2BBv2 model trained on only 4T tokens (about 20% of the full 20T pretraining budget) and apply RLP for IB tokens on same pretraining corpus DpT. As shown Table S.4 even at this early stage, RI.P is highly effective: with only IB tokens, Math Avg more than doubles (from 21.93 to 10.14), Science Avg @ [[4] improves by points (from 5.69 to 11.9)), and Overall Average increases by 12 points (from 12.05 24.08). While
Benchmarks
Mbase
MALP
MATH500
30.15
62.38
GSM8K
29.56
81.42
AMC23
22.81
37.81
Minerva
5.19
18.93
MMLU
11.59
13.10
MMLU@ 1[4]
8.73
20.68
MMLU-Pro
4.93
6.11
MMLU-ProG 1[4]
2.66
7.50
GPQA GPQA@ 1[4]
9.10
11.20
5.68
7.70
Math Avg
21.93
50.14
Science Avg
8.54
10.14
Science Avg@ [[4)
5.69
11.96
Overall
12.05
24.08
strongest final results come from applying RLP later in pretraining, these find- ings indicate that RLP can already yield large gains when the model has seen only a small fraction of the standard pretraining budget.
Table S.4: Comparison NEMOTRON- NANO-120 1170-12B-11 12B-v2 4T Base and Base+RLP across benchmarks.
FLOP matched comparison between RLP and RPT. We would like to clarify that even though RLP can be theoretically applied to tokens at every position the document, practice we only apply for one token per document. This token is selected randomly and not through any criteria as in the case of RPT. For the experiments in Table 3] we have matched the number of input tokens for both RLP and RPT settings (we train both methods for epoch of the we want to highlight that the number of target tokens for which reward is calculated is much larger for RPT compared to RLP (since we don't 't apply the RLP reward every token in the document). Hence, the setting in Table is in favor of RPT. Additionally, we don't 't include the compute needed to pre-select tokens using an external LLM for RPT.
documents). But
To directly address the head to head flop matched comparison, we run a controlled experiment using Comotron-CrossThine data. We deploy both RLP and RPT for only one epoch on the same data, i... the number of target tokens for which reward is calculated is similar in both cases. RI. achieves 16.23% relative improvement in Overall Avg and consistently outperforms RPT on both math and science aggregates. These results confirm that the gains in Table are not artifact of mismatched settings. Even under stricter, target-matche matched conditions, RLP provides stronger and more general improvements.
As shown in Table 3. RLP achieves a 16.23% relative improvement in Overall Avg and consistently outperforms RPT on both math and science aggregates. These results confirm that the gains in Table 3 are not an artifact of mismatched settings. Even under stricter, provides stronger and more general improvements.
conditions, RLP
23

---

Published as a conference paper at ICLR 2026
Continuous pretraining with longer context length. Owen3-1.7B-Basst is indeed eventually extended to 32K context window, but as described in the Qwen3 technical report, this happens only in third long-com long-context stage after the model has already been pretrained for 30T+ tokens at a much shorter context (4,096 tokens) and then further trained on snowledge-intensive data. Our conceptually closer these first two stages; continue pretraining pretraining mixture (Dpt), which consists almost entirely of relatively short documents without long- range dependencies. In this regime, substantially increasing the context length does not obviously provide additional learning signal, but does change the optimization landscape and the effective batch and gradient statistics.
CPT experiments
To evaluate the effect of longer context length, we conduct an additional controlled experiment where we keep all CPT hyperparameters fixed and only increased the context length from 8K to 32K. The resulting model, denoted (Mcrr(32)). is compared to our original (Mcrr(8K) in Table S.5 The result suggests that for our pretraining corpus (Dpt). which rarely contains long documents that would actually utilize a 32K window, the 8K context configuration is at least strong as, and in practice strictly better than, 32K context configuration under matched compute and hyper- parameters. Therefore, while we agree that context length an important design choice, in our specific setup, using 8K rather than 32K does weaken the CPT baseline; if anything, the longer context hurts optimization without yielding downstream benefits. Importantly, all compar- isons between RLP and CPT are made against the stronger 8K CPT configuration.
Benchmarks
McPr(8K)
McPr(32K)
AIME25
3.96
3.33
MATH500
57.52
51.80
GSM8K
72.85
60.44
AMC23
31.25
25,00
19.03
17.46
MMLU
41.95
42.19
MMLU@ [[4]
40.00
40.55
MML.U-Pr
27.81
27.08
MMLU-Pro@1[[]
24.61
22.87
GPQA GPQA@ I[4]
26.26
25.76
24.75
24.21
Math Avg
36.92
31.61
Science Avg
32.01
31.68
Science Avg I[4]
29.79
29.21
Effect of EMA over RLP training. In RLP, the EMA baseline (px) acts as a dynamic no-think counterfactual, providing a reference fog-likelihood for each next token. The decay rate 7 controls comparison difficulty: if 7 is too low, the baseline updates too quickly and the reward collapses toward zero; if too high, it becomes stale and yields artificially easy gains.
Overall
32,90
30.83
Table S.5: Comparison of CPT models with 8K vs 32K context length.
To justify our choice of we ran sensitivity study on Qwen3-1.77 1.n03-117BBBaee with 0.99, 0.995, 0.999, 0.9995. As shown in Table S.6 performance forms a bell-shaped curve with clear peak at 0.999.
Model
Math Avg
Science Avg
Overall Avg
Musse
N/A
35,96
32.11
34.03
0.99
45,20
36.31
38.82
0.995
45.18
37.36
39.21
MRLP
0.999
45.98
37.38
39.54
0.9995
45.64
36.84
39.20
Table S.6: Effect of temperature on performance. Best result highlighted.
Across this range, training remained stable and we did not observe divergent or unstable behavior in any of our runs. Concerns that the model could "game" the objective by degrading the baseline do not manifest because the haseline updated only via the EMA of the student parameters: for the baseline degrade, the student must degrade first, which is immediately penalized through the primary reward term log pa. Thus the EMA baseline provides stable, meaningful measure of information gain.
Wall-clock time of RLP training versus SFT. We conduct a direct comparison using 32 H100 GPUs with global batch size of 512 and a 32K context length. As shown in the table below, RLP incurs an expected overhead due to the generation phase (G = 16 rollouts). While SFT, which has a similar computational profile to standard Continuous Pretraining (CPT), achieves throughput of
24

---

Published as a conference paper at ICLR 2026
92.34 samples/s (approx. 5.5s per step), RLP operates at 41.07 samples/s (approx. 12.5s per step).
This results in a per-step slowdown factor of roughly 2.25x.
Method
Batch Size
Rollouts (G)
Time/Step (s)
Throughput (samples/s)
Relative Speed
SFT
512
N/A
5.54
92.34
1.00x
RLP
512
16
12.47
41.07
0.44x
Table S.7: Comparison of SFT and RLP training efficiency.
However, this per-step cost must be viewed in the context of convergence efficiency and total compute. While RLP is 2.25 x slower per iteration than SFT/CPT, is drastically more data-efficient. As detailed in Table 4 RLP achieves overall average accuracy of 43.36% on the dataset using only 170 tokens. In contrast, the FLOP-matched CPT baseline required processing 6B tokens (roughly 35x more data) account for the compute difference, yet only reached an accuracy of 35.60% Thus, while RLP processes tokens slower due to rollouts, the dense reward signal extracts significantly more reasoning capability per FLOP, yielding performance margin (+7.76%) that standard training cannot replicate even with substantially higher data volume.
-rrossshnn
Final perplexity after post-training. We confirm that the model's perplexity on ordinary tokens does not degrade; in fact, it significantly improves. Unlike standard RLHF, where optimizing for an external reward often causes the model distribution drift away from natural language, our reward signal is the log- ligeliiel of the next token itself. Therefore, by definition, RLP is optimizing for prediction accuracy.
Model
Nemotron- Crasoonnn-rosoiiikkk PPL.
Nemotron-Crus<Think NLL.,
Wikitext-103 P
Wikitext- NLL
Mrsi (Qwen-1.7B) MRLP (Ours)
2.91
1.06
S,83
1.77
2.36
0.86
4.48
1.50
Table S.8: Perplexity and NLL comparison on Nemotron CrossThink and Wikitext- 103.
Mathematically, maximizing the RLP reward is equivalent to minimizing the cross-entropy of the reasoned predictor against the data distribution (Proposition 1). As shown in the table below, our empirical results confirm this theoretical guarantee: MRLP achieves consistently lower Perplexity (PPI.) and Negative Likellhoo (NI.L) compared to the base model. Crucially, this improvement holds for both the reasoning- intensive Nemotron CrossThink dataset and the gemeral-domain Wikitext- 103 benchmark, demonstrating that the "thoughts" generated by the model successfully compress information to better predict ordinary text.
-intiinngittesivvee
Computational Cost and FLOP Analysis RLP. potential concern comparing RLP against CPT is the perceived computational burden of autoregressively generating long reasoning traces. This would indeed be prohibitive if the rollout policy were applied at every token position in a sequence. In practice, however, RLP is applied to only one randomly sampled token per sequence, which dramatically reduces the computational burden. Instead of scaling with Looc x .CCor, the rollout cost scales with Loor per sequence. This design choice makes RLP computationally feasible and allows us to interleave it with standard training efficiently. In addition, autoregressive generation involves bottleneck compared parallel processing. We agree that this affects wall-clock time due to memory bandwidth constraints, but it does not incorrectly skew the FLOP calculation used for the baselines.
In Appendix 12,
we calculated the FLOP-equivalent budget by summing the tokens used for gradient updates and the tokens generated during rollouts. We compared RLP (170M input tokens) against a CPT baseline trained on 6B tokens. This 35x increase in data for the baseline is rigorous upper bound for two reasons:
The FLOPs of forward pass for generating one token is approximately 2NV (where N is parameter count). The cost of training on one token (forward + backward) is approximately 6N. By equating one generated token one trained token in FLOP calculation, we are effectively penalizing RLP (counting generation as 3x more expensive than it theoretically is in terms of FLOPs).
Operation Count:
25

---

Published as a conference paper at ICLR 2026
Total Compute: Even with the overhead of 16 rollouts of length 2048 per document, the total floating-pois operations performed by RLP on 170M documents are comparable to (or less than) performing standard forward/backward passes on the 6B tokens used in the Mcrr[6B] baseline.
While autoregressive generation is indeed slower in terms of wall-clock time, the purpose of the baseline is to compare compute efficiency. RLP applied to single token per document is highly efficient, and McPr[6B] baseline represents compute- which RLP still outperforms significantly (Overall Avg 42. 13% vs 38.04%).
On Self- Reffeeetertnti and the Semanties of the RLP Reward. A natural concern for any method that leverages model-internal signals is whether the learning dynamics risk becoming self- rewardiin increases in internal confidence rather than genuine improvements in correct- ness reasoning ability. In RLP, however, the reward structure is explicitly grounded in the data rather than in unconstrained model self-agreement.
Reward is anchored to ground-truth tokens. For each sampled position, the reward
r(c)) log I<t, c)) log Ps x<te)
is defined with respect the ground-truth next token 2y from the corpus. Proposition 1 shows that, =<t)) this reward equals the reduction in cross-entropy achieved by conditioning on the thought ce. A thought therefore receives positive reward only it moves probability mass toward the correct continuation in the true data distribution. Increased confidence on an incorrect continuation strictly decreases the reward. This prevents the model from benefiting by simply inflating logit magnitudes or reinforcing patterns unrelated to accuracy.
in expectation over ~ p-(( (-1
EMA baseline prevents degenerate self-consistency loops. RLP compares each thought- conditioned prediction an exponential moving average (EMA) baseline Pa evaluated on the same context and same ground-truth token. If the current parameters shift toward patterns that improve internal consistency but harm prediction of the observed token, the relative likelihood under pe falls and the corresponding thought receives negative advantage. Group-relative normalization and advantage clipping further ensure that thoughts cannot win reward by global logit scaling alone; only thoughts that contribute meaningful information about the next token outperform the EMA teacher expectation.
External evaluations validate correctness rather than internal consistency. The most impor- tant empirical question is whether internal information gain translates into better reasoning on verifiable tasks. Across GSM8K, MATH500, MMI.U-Pro, GPQA, and other benchmarks with objectively checkable answers, RIP-trained models consistently outperform both the base model and compute-matched continuous- preeraannnn baselines- even when the latter consume substan- tially more training tokens at equal FLOPs. Notably, these gains persist after a strong post-training pipeline involving SFT and RLVR with external verifiers. If RLP were primarily amplifying internal confidence without improving correctness, these advantages would be expected to collapse or become fragile under verifier supervision. Instead, RLP P-initiaalize models remain ahead, partic- ularly on reasoning-hen heavy domains, indicating that the learned thoughts encode genuinely useful information and not merely self-reinforcing patterns.
Overall, the formulation of the RLP reward ensures that the model is optimized for meaningful reductions in predictive error on the underlying data distribution, while empirical evidence confirms that these internal information gains translate into improved external reasoning performance.
Generalizability across Architectures and Data Distributions To rigorously assess the univer- sality of our approach, we evaluated RI. on two models chosen specifically for their significant divergence in both architectural design and data provenance: Qwen3-1.7B- BBBe and Nemotron-Nar Nano- 12B-V2. These distinct settings demonstrate that RLP is not limited to single model family or training recipe.
Architectural Heterogeneity: The models represent fundamentally different backbone architec- tures. Qwen3-L.7B-B.. utilizes standard, pure Transformer architecture. In contrast, Nemotron- Nano-12B- V2 is a Hybrid Mamba2- -rannfforrrfrrrr which integrates State Space Models (SSM) with attention layers and employs distinct tokenizer and training recipe. The fact that RLP translates
26

---

Published as a conference paper at ICLR 2026
effeettiely to this hybrid architecture- driving an increase in overall average accuracy from 42.81% to 61.32% ppovidesvv strong evidence that the method architecture- architecture-agnostic.
-riiviin
Data Lineage Independence: The models also rely distinct pretraining distributions. Nemotron- Wano-12B-W V2 was trained from scratch on a massive mixture of raw public data (including Web Crawl, arXiv, and (penWeeMath) and synthetic data generated by a diverse ensemble of teacher models. As detailed in recent technical reports, this ensemble extends well beyond the Qwen family include DeepSeek-RI (DeepSeek-At AI al. 2025a). Mixtral-&x22B-vv
8x22B-v0.J (Jiang et al. 2024). DeepSeek- V3 (DeepSeck-Al et al. 2025b). Phi-4 (Abdin et al., 2024). and Semotron-4-3444 (Nvidia et 2024).
By validating RI.P on a Hybrid Mamba model trained on a multi-source ensemble distribution, distinct from the pure Transformer Qwen baseline, we confirm the method's scalability and robustness across disparate experimental environments.
Analysis of thought traces. We examine the model's internal thought traces understand how RLP shapes reasoning. A manual analysis of 50 randomly sampled traces reveals consistently high-quality reasoning. characterized by grammatical correctness and strong contextual relevance, confirming that the information gaaiiniooonnnnnnnnn objective effectively suppresses unproductive generation. We observe three qualitative properties: (1) focused, context-aware reasoning, where thoughts orient the model to the specific logical step required by the context; (2) continuation-ssy formatting, acting as internal scratch work that mirrors the prefix's tone and notation without meta-commentary and (3) utility-divee anticipation, where thoughts predict structural moves (e.g., upcoming rules or algebraic steps) to maximize the likelihood the ground-truth continuation. An illustrative example follows.
Prefix
"West Virginia students invited to enter Ornament Competition CHARLESTON, W. Va.
(AP) West Virginia students in kindergarten through 12th grade may enter the First Lady Student Ornament Competition this fall. First lady Cathy Justice asking all students participate in the 18th annual event. It is open to students in public and private schools well as those who are home-schooled. Students are asked to create a sunteracker"-themee ornament for tree to be displayed at the Culture Center in Charleston during the holidays, according to a news release from Gov. Jim Justice's 's office. There will be four divisions according to grade, and a winning class will be chosen from each division. The winning ornaments will be donated in January to the West Virginia State Museum. The four winning classes will"
Thought Trace
"The article outlining contest logistics. The next sentence will likely add a specific detail such as what the winning classes receive, how the ornaments will be displayed, or other submission guidelines. Maintain the neutral news tone and extend the informational structure already established."
13 DATA BLEND EXTENDED RESULTS
To further examine whether RLP learns transferable reasoning beyond narrowly curated datasets, we evaluate it across broad spectrum of corpora spanning both structured reasoning data and open-ended pretraining distributions. All experiments start from QWEN3- 1.7B-BASSS and apply RI. for 200 steps, consuming 770M input tokens, while keeping all other training settings fixed.
We consider two primary corpus families. The first consists of SFT-style reasoning datasets, including OmniMath (ao et al. 2024), OpenThouughs (Guha et al. 2025), and Temotron-CrossThiin (Akter) al.) 2025). These datasets contain structured question-answer pairs with explicit reasoning content and represent the typical setting where reinforcement based methods are expected to perform well. The second family consists of general-purpose pretraining corpora, including academic papers (ACAD), math textbooks (Math- MMtth-Tex),, and open-ended web crawl data. These datasets are not curated specifically for reasoning and more closely resemble large-scale pretraining mixtures.
27

---

Published as a conference paper at ICLR 2026
Benchmark
OmniMath
ThenThoughts
Nemotron-Crosthhnk
ACAD
Math-Text
Web-Crawl
MATH500
57.95
59.55
62.65
59.75
60.03
61.58
GSM8K
74.82
74.80
79.97
76.10
75.95
76.48
AMC23
32.50
32.81
30.91
31.88
34.38
35.00
Minerva
20.63
20.96
25.46
22.98
21.92
22.43
MMLU
55.72
55.84
56.72
56.35
56.02
55.93
MMLU@ [[4]
50.85
50.84
52.11
51.51
50.72
50.11
MMLU-Pro
35.81
34.55
38.58
35.11
36.06
37.02
MMLU-
31.47
30.57
35.10
31.06
31.86
32.43
Pro@ [[4]
GPQA
29.29
25.76
32.32
30.30
29.29
29.29
GPQA@ 1[4]
30.30
27.27
26.14
28.03
26.39
27.78
Math Avg
46.48
47.03
49.76
47.68
48.07
48.87
Science Avg
40.27
38.72
12.54
40.59
40.46
40.75
Science
37.54
36.23
37.78
36.87
36.32
36.77
Avg@ [[4]
Overall
41.43
40.66
43.36
41.71
41.62
42.13
Table S.9: Quantitative benchmarks for QWEN3-1.J 1.7--117B---0 1.7.-BBSSSS BB-B,S,, showing the impact of RLP on different data blends. Shaded columns indicate general pretraining corpus.
Table 4 shows that RLP consistently improves over the base model across all corpus types. Impor- tantly, the gains are not confined to math-centric SFT data. While the strongest improvements occur on Nemotron-Can Crrtonn--rosssnnn within the SFT family, substantial gains are also observed when training on purely general corpora such as academic papers and web crawl data. This demonstrates that RLP does not depend on carefully constructed reasoning traces. Instead, extracts a transferable reasoning signal even from heterogeneous, weakly structured text.
notable contrast emerges when compared with prior RL-based approaches that report improvements concentrated in high-quality math data and limited transfer to broader domains. In experiments, models trained with mixed-domain or open-ended corpora simultaneously improve math, science, and professional benchmarks. There is evidence of Nomainn-spccfff overfitting degradation on math when incorporating diverse data. Rather, diversity appears to strengthen general reasoning performance.
Table $.9 provides a task-level breakdown. Across MATH500, GSM8K, MMLU, MMLU-Pro, GPQA, and related metries, improvements observed regardless of whether the underlying training corpus is structured SFT data or general pretraining text. Even web-scale crawl data yields competitive math and science averages, suggesting that RLP leverages latent reasoning patterns embedded in broad distributions.
Overall, these results support three conclusions. First, RLP scales across corpus families without requiring specialized reasoning datasets. Second, it exhibits genuine cross-domain transfer rather than narrow task adaptation. Third, data diversity amplifies the learned reasoning signal instead of diluting it. Together, this positions RLP as both reasoning robustness and general benchmark accuracy.
pretraining augmentation that enhances
28


Reviews:
[{"review_id": "DcRcACf3L5", "reviewer": "Reviewer_UUEW", "summary": "The paper proposes how to use RL during pretraining - where the model can be trained to output generations that increase the likelihood of next token prediction on the original corpus. The reward is the log-likelihood improvement of the dataset token when conditioned on both the context and the sampled reasoning trace, compared to conditioning on the context alone. This produces a dense, verifier-free reward signal that can be computed on ordinary pretraining text, enabling reinforcement-style updates during pretraining without external verifiers or curated datasets. Therefore, this allows for training the model to “think before predicting,” encouraging internal reasoning behavior that improves predictive utility and persists through post-training.", "questions": "Based on the weaknesses, can the authors talk more about when is the right time to introduce RL in the pretraining phase - too early and the reward is less meaningful. Furthermore, can the authors talk about when to use this - for example, if the context is a knowledge / fact then this method won't be too useful. \n\nIts also worth citing this work: https://arxiv.org/abs/2502.19402, which recommends this direction.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a clean and scalable way to insert RL in the pre-training phase. In addition the RL signal is dense, thus scalable and data efficiency, outperforming strong baselines on reasoning benchmarks while using only a fraction of the data.", "weaknesses": "The paper defines reward purely in terms of internal log-likelihood improvement, making the entire learning process self-referential. The model is effectively rewarding itself for being more confident, not necessarily for being more correct. This creates a risk that RLP amplifies patterns that increase internal consistency without corresponding improvements in truthfulness or reasoning quality.\n\nWhile the authors frame the method as encouraging reasoning, there’s little evidence that the model is actually learning to reason. The improvements could just as well come from implicit regularization or longer-context modeling rather than genuine emergence of reasoning. There’s no analysis of the thought traces themselves to support the interpretation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6z27LwnPoJ", "reviewer": "Reviewer_TMzP", "summary": "This paper proposes Reinforcement Learning Pre-training (RLP), an objective that moves RL-based reasoning into the pretraining phase. The model learns to generate an internal Chain-of-Thought before next-token prediction, using a verifier-free \"information gain\" reward compared to a \"no-think\" EMA baseline. Experiments on 1.7B and 12B models show RLP improves reasoning benchmarks over standard pretraining, with gains persisting after post-training.", "questions": "Please refer to weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Conceptual Novelty: The key novelty is integrating RL-based reasoning into pretraining, not just post-training. The proposed \"information gain\" reward is verifier-free and self-contained, allowing it to be applied to general text corpora without needing curated datasets.\n2. Strong Empirical Gains: The method shows significant empirical gains over several key baselines, including a continuous pretraining baseline that is matched for total FLOPs.\n3. Compounding Improvements: The pretraining gains from RLP are shown to compound, rather than be \"washed out,\" by standard post-training (SFT + RLVR), suggesting it builds a stronger reasoning foundation.", "weaknesses": "1. Unclear Computational Cost & Baseline Fairness: The computational cost analysis and baseline comparison are a key weakness. RLP appears significantly more expensive than standard pretraining (e.g., 16 rollouts at every token). The main \"compute-matched\" baseline (CPT on 6B tokens vs. RLP on 170M) is confounded by seeing 35x more data. A clearer comparison would be a CPT baseline trained on the same data for the same FLOPs. \n2. Marginal Gains Over RPT: The comparison with RPT is not that convincing. As shown in Table 3, the performance difference between RLP and RPT is marginal. Given that RPT uses a simpler sparse, binary reward, this small gap raises questions about the necessity of RLP's more complex, dense reward. Furthermore, the paper does not provide sufficient detail to assess if the \"matched data and compute\" setting for this comparison was strictly fair.\n3. Limited Scale of Experiments: The experiments are limited to small/medium models (1.7B, 12B). Demonstrating large relative gains on small models (which have near-random performance on some hard tasks) is less convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4aTjt27syD", "reviewer": "Reviewer_T8xM", "summary": "The paper proposes a method called RLP for enhancing model reasoning without a verifiable reward. The method maximizes the improvement in next‑token prediction when conditioning on a reasoning trace, relative to ordinary prediction. RLP reports superior performance on math and science tasks.", "questions": "Intuitively, the simplest way to obtain a high improvement reward is to push up the teacher model’s cross‑entropy. I assume the EMA on teacher weights helps prevent such collapse. Still I have a few questions:\n\n- Did you observe such collapse in any experiments? If so, which modifications helped prevent it?\n\n- Does the model’s final perplexity (on ordinary tokens) degrade after this post‑training, especially when RLP is applied on standard pre‑training data?\n\nIn methods like GRPO, generation speed becomes the bottleneck. Could you report the wall‑clock time of RLP training versus SFT? I realize your implementation may not be fully optimized, but wall‑time comparisons would still be informative", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is intuitive and easy to implement.\n\n- The ablation study is diverse and comprehensive.", "weaknesses": "- Despite promising results, the objective of maximizing improvement could, in principle, degrade the model’s base predictions (to inflate the reward).\n\n- Nemotron‑nano‑12B‑v2 and Qwen3‑1.7B‑Base are closely related models. As far as I know, Nemotron was refined with data generated by the Qwen3 family. Experiments on more independent models (e.g., Llama) would strengthen the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OmJj9m0xyc", "reviewer": "Reviewer_kYzE", "summary": "This work introduces a new method for applying RL right after the pre-training phase. In particular, this involves generating reasoning traces with a base model and computing a reward signal based on how well these reasoning traces increase the likelihood of ground-truth next-tokens labels in a given dataset. The authors call this method RLP and test it on top of a Qwen 1.7B base transformer model and a Nemotron hybrid model, evaluating directly on generation-based tasks.", "questions": "I have detailed what I found to be the biggest areas of improvement and my questions about this work in the Weaknesses section of my review. I would encourage the authors to address these points and help clarify any potential misconceptions for the discussion phase.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- Overall, the exposition of the method is clear, and the simple theoretical considerations based on Jensen's inequality in section 2.2 to show the loss is a sensible optimization objective are concise and not overemphasized.\n- The authors apply their method to a hybrid SSM like Nemotron. Moreover, they also experiment with different data sources in Table 4. This degree of diversity is nice to see.\n- While code is not released, the authors do provide training and hyper-parameter details in Appendices 9 and 10, which provide helpful transparency to the method and baselines considered.", "weaknesses": "**Main concerns**\nMy main concerns are about the soundness of the paper's empirical validation. The main quantitative results in tables 1, 3, and 4 feature a Qwen3 1.7B base model. This base model is extended and compared with RLP on downstream reasoning tasks. There are several aspects about how the evaluation is carried out and the baselines that prompted my concerns:\n1) Evaluating base models without any alignment on generation tasks and without even \"few-shot\" examples is doomed to underperform, as these models have never been primed for question-answering. Thus, I am unsure how relevant it would be to compare RLP with continued pretraining and base models trained on fixed non-masked datasets (which make up the bulk of the reported results), as pretraining corpora are purposefully not meant to imbue the models with alignment capabilities. The RLP traces generated with low temperature (appendix 9) and filtered based on likelihood should inherently make the model's logits more skewed (something that even occurs in RL with random rewards [1]).\n2) Looking at the hyperparameters for the continued pre-training baseline (CPT), these appear quite odd and suboptimal (e.g., using a much smaller context than what Qwen3 was pretrained with in the first place). Beyond pretraining, the \"post-training\" procedure in Table 1 uses the OpenThought dataset that was conceived for finetuning already post-trained \"instruct\" models, and is much smaller and less diverse than common datasets used for the first SFT phase.\n3) One of the key downsides of RL is its disproportionate cost compared to other training stages. Yet, with one exception, all \"continued pre-training baselines\" seem to only equate the size of RLP's \"seed\" dataset, which I believe implies that the training budget and tokens used for CPT would be disproportionately more (likely well over an order of magnitude). There is a single baseline in Table 4 that is described to be \"equating FLOPs\", yet, looking at Section 10 in the Appendix, this FLOP calculation does not take into account the autoregressive bottleneck of generating reasoning traces with RL. If I am not missing something, I believe this likely implies the training time for CPT would still be much larger.\n\nBased on 1), 2), and 3), I find it hard to extrapolate that the proposed method would provide any benefit to an actual LLM training pipeline and would be worth the high computational demands of autoregressive generation at an early training stage. In order to provide convincing evidence, I would suggest comparing their final model directly to the \"instruct\" version of Qwen 3 1.7B, but I suspect that given 2), even their final post-trained version would significantly underperform with respect to this properly aligned baseline.\n\n**Other**\n1. I did not find the role of the EMA model to compute the baseline to be properly explained. While the authors briefly mention the reward hacking hypothesis, this does not seem to be empirically validated. Empirically, did the authors find divergent behaviors without EMA? Keeping EMA in memory also requires storing a separate copy of the network's weight, with immediate additional cost concerns that are not currently discussed.\n2. When comparing with RPT, the authors mention 1 epoch training \"with matched data and compute\". Yet, it's unclear to me how both can be true at the same time, as RPT performs token-level filtering before training. To equate compute, RLP would have to train on a much smaller number of unfiltered datapoints from Omni-MATH.\n3. Without strong empirical validation or analysis, the overall contribution of the paper seems quite limited to me. Generating reasoning traces during pretraining was already done by [2] (which the authors do cite), while rewarding reasoning traces with the log probabilities of ground-truth completion tokens was already done by [3] (currently, not cited).\n\n\n[1]  Shao, Rulin, et al. \"Spurious rewards: Rethinking training signals in rlvr.\" arXiv preprint arXiv:2506.10947 (2025).\n\n[2] Hatamizadeh, Ali, et al. \"RLP: Reinforcement as a Pretraining Objective.\" arXiv preprint arXiv:2510.01265 (2025).\n\n[3] Cetin, Edoardo, Tianyu Zhao, and Yujin Tang. \"Reinforcement Learning Teachers of Test Time Scaling.\" arXiv preprint arXiv:2506.08388 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nXKKG6daa1", "reviewer": "Reviewer_cQyL", "summary": "This paper introduces RLP, a pretraining paradigm that integrates RL into training reasoning LLMs. RLP reframes CoT generation as an action taken before next-token-prediction, and this action is rewarded by a verifier-free information gain which measures how much the CoT improves the model's likelihood of predicting the correct token. The no-think model adopts a DQN fashion update to keep its distribution close to the behavior policy. I think the authors' proposed method is a clever way to overcome the sparse reward and verifier-must conditions in applying RL to training a reasoning model.", "questions": "The no-think EMA baseline appears to serve a similar role to the target network in DQN, where the update frequency is a critical hyper-parameter that balances target stability against the staleness of the learning signal. In RLP, how sensitive are the training stability and final performance to the EMA decay rate? Did the authors experiment with different decay schedules for $\\tau$, and can they characterize this trade-off?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper presents a foundational shift by moving RL to the pretraining stage, and addresses the core limitation of standard next-token prediction.\n* The information-gain reward is a key innovation, providing a dense, self-supervised signal on any text without needing external verifiers.", "weaknesses": "* While showing scalability, the most detailed experiments and ablations are on a relatively small 1.7B parameter model. The paper would be strengthened by a deeper quantitative analysis on a larger, more state-of-the-art model size (e.g., 7B+). Also, I would like to see the comparison with an instruct-tuned model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)