Native Reasoning Models: Training Language Models to Reason on Unverifiable Data
Authors: Yuanfu Wang
Zhixuan Liu
Li xiangtian
Chaochao Lu
Chao Yang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: Native Reasoning Training (NRT) cultivates reasoning in base LLMs using only Q&A pairs. It rewards self-generated logic for correct answers, achieving SOTA results for verifier-free methods without needing costly, human-written reasoning examples.

Abstract: The dominant paradigm for training large reasoning models—combining Supervised Fine-Tuning (SFT) with Reinforcement Learning with Verifiable Rewards (RLVR)—is fundamentally constrained by its reliance on high-quality, human-annotated reasoning data and external verifiers.
This dependency incurs significant data-collection costs, risks embedding human cognitive biases, and confines the reinforcement learning stage to objectively assessable domains like mathematics and coding, leaving a vast landscape of unverifiable tasks unaddressed.
To overcome these limitations, we introduce Native Reasoning Training (NRT), a novel framework that cultivates complex reasoning by having the model generate its own reasoning traces using only standard question-answer pairs, thereby obviating the need for expert-written demonstrations.
NRT reframes the training problem by treating the reasoning process as a latent variable. 
It employs a unified training objective that models reasoning as an optimization problem, intrinsically rewarding paths that increase the model's likelihood of producing the ground-truth answer.
This unified perspective allows us to analyze intrinsic failure modes of prior methods, such as policy collapse, and systematically design more robust reward aggregation functions, creating a self-correcting feedback loop where the model learns to \textit{think} in ways that resolve its own uncertainty.
Empirical evaluation on Llama and Mistral model families demonstrates that NRT achieves state-of-the-art performance among verifier-free methods, significantly outperforming standard SFT baselines and prior verifier-free RL methods.
Our approach yields particularly strong performance gains in complex reasoning domains and exhibits high robustness to policy collapse, offering a general, scalable path toward building more powerful and broadly applicable reasoning systems.

Published as a conference paper at ICLR 2026
NATIVE REASONING MODELS: TRAINING LANGUAGE MODELS TO REASON ON UNVERIFIABLE DATA
Yuanfu Wang! Zhixuan Liu
Xiangtian Li
Chaochao Lu!
Chao Yang
'Shanghai Artificial Intelligence Laboratory SSanggaai Jiao Tong University sharkwyf& gmail. com, 1zx931244940sjtuu edu. (lixiangtian, luchaochao, [angchao]@pjlas org.cn
ABSTRACT
The prevailing paradigm for training large reasoning models-combinnng Super- vised Fine-Tuning (SFT) with Reinforcement Learning with Verifiable Rewards (RLVR)-is is fundamentally constrained by its reliance high-quality, human- annotated reasoning data and external verifiers. This dependency incurs signifi- cant data-collection costs, risks embedding human cognitive biases, and confines the reinforcement learning stage to objectively assessable domains like mathe- maties and coding, leaving a wide range of unverifiable tasks beyond its scope. To overcome these limitations, we introduce NRT (Native Reasoning Training), novel framework that cultivates complex reasoning by having the model gen- erate its own reasoning traces using only standard question-ane answer pairs, thereby obviating the need for expert-written demonstrations. NRT reframes the train- ing problem by treating the reasoning process as a latent variable. It employs a unified training objective that models reasoning as an optimization problem, in- trinsically rewarding paths that increase the model's likelihood of producing the ground-truth answer. This unified perspective allows us to analyze intrinsic fail- ure modes of prior methods, such as policy collapse, and systematically design more robust reward aggregation functions, creating a self reiforeinnor feedback loop where the model learns to think in ways that resolve its own uncertainty. Em- pirical evaluation on Ilama and Mistral model families demonstrates that NRT achieves state-of-the-art performance among verifier-free methods, significantly outperforming standard SFT baselines and prior verifier-free RL methods. Our approach yields particularly strong performance gains in complex reasoning do- mains and exhibits high robustness to policy collapse, offering a general, scalable path toward building more powerful and broadly applicable reasoning systems.
INTRODUCTION
Large Language Models (LLMs) have evolved into powerful Large Reasoning Models (LRMs), demonstrating remarkable abilities complex tasks like mathematics, coding, and multi-step problem-solving. The prevailing approach is two-stage pipeline: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning with Verifiable Rewards (RLVR) (DeepSeck-Al et al., 2025). In the first stage, SFT trains the model to imitate human-writter reasoning processes, often chain- of-thought demonstrations (Wei et al. 2022). a cold-start phase that provides a foundational ability structure its thoughts. The second stage, RLVR, refines and generalizes this ability, using an ex- ternal verifier-such as a unit test for code or a final answer check for a math problem-to reward the model for any reasoning trace yielding correct output, freeing from strictly mimicking single numan-providee path (Shao et al. 2024).
Despite its success, the SFT+RLVR paradigm has two fundamental limitations. First, it depends critically on the initial SFT phase, which requires expensive, high-quality, hunan-annotated reason- ing data. This data can embed human priors and biases, constraining the model's 's performance and its search for novel, more effective reasoning strategies. Second, reliance on external verifier con- fines the RLVR stage to objectively assessable domains. This leaves a vast landscape of tasks-suee
"Corresoonding author.

---

Published as a conference paper at ICLR 2026
as open-ended question answering, creative writing, and summarization -where reasoning is crucial but correctness is subjective and unverifiable. Attempts to bridge this gap often use proxy rewards. Some methods use judge model to score outputs (Vacareanu et al., 2024 Guan et al. 2024), but this introduces rubric design complexity and is constrained by the judge's s own capabilities. Other self-rewarding approaches Chen et al. (2024) train a model to generate its own feedback but can be limited by the model's ' initial capacity and often suffer mode collapse, where the policy converges to simple, low-entropy outputs. These limitations motivate a fundamental question: how can we culti- vate and refine reasoning abilities in language models using only question-answer ansser pairs, especially for domains where external verifiers are unavailable?
In this work, we introduce Native Reasoning Training (NRT). a novel framework to cultivate rea- soning abilities without supervision on the reasoning trace itself. The term Native signifies that the reasoning process is not imitated from human demonstrations but emerges as the model discovers how to connect question to its answer. treats the reasoning process as latent variable to be discovered, not imitated. NRT uses RL to explore potential reasoning traces, intrinsically rewarding those that increase the model's likelihood of generating the ground-truth answer. This creates a self- reinforcing feedback loop where the model learns to generate reasoning traces that resolve its own uncertainty. Crucially, this process requires only standard anestion-answer pairs, obviating the need for expert-written reasoning demonstrations and external verifiers. NRT unlocks reasoning training for a vast range of problems where verifiers are unavailable.
Our contributions are threefold: (1) A Unified Framework for Verifier-Free Reasoning. We intro- duce NRT, unified framework that models reasoning as a latent variable and optimizes the marginal log-likelihood of the answer. Our framework contextualizes prior methods, enabling a theoretical analysis of their reward structures and failure modes like policy collapse. (2) A Powerful, Princi- pled Training Method for Unverifiable Data. We propose principled training method with novel reward schemes (e.g., NRT-WS) derived from our framework. This method enhances reasoning by focusing on the model's uncertainty, requires only question-answer pairs, and eliminates the need for expert demonstrations or external verifiers, thus lowering data costs and broadening applicability. (3) State-of-the-Ar Performance through Refined Reasoning. We demonstrate empirically that NRT achieves state-of-the- art results, significantly outperforming SFT baselines and prior verifier- free methods on diverse reasoning benchmarks. Our analysis confirms that NRT is highly robust to training instabilities like policy collapse that affect other RL methods.
2 RELATED WORK
Our work on developing reasoning in LLMs builds on three paradigms:
Reinforcement Learning with Verifiable Rewards (RLVR). The predominant approach for train- ing state-of-the-ard reasoning models is RLVR, which requires an objective, external verifier to con- firm the correctness of model's final answer. While highly effective, this paradigm is largely confined to domains like mathematics and code generation where ground -ruth can be programmat-
ically checked (Shao et al. 2024 DeepSeek-AT al., 2025 OpenAl al. 2024). The foundational method is Group Relative Policy Optimization (GRPO) (Shao et al. 2024), built on general policy optimization algorithms like PPO (Schulman et al. 2017b). Its success spawned algorithmic refine- ments that improve optimization stability (Liu et al. 2025 MiniMax et al. 2025 Zheng et al. 20255 or adapt to problem difficulty YYu al. 2025a). Despite these advances, the core reliance on external verifiers constrains these powerful RL techniques to objectively assessable problem domains.
Proxy Rewards and Self-Rewarding, To extend reasoning training to more general domains, re- searchers developed proxy reward methods. One strategy uses powerful LLM judge to score outputs against predefined rubric or human preferences (Lee et al. 2023) Guan et al. 2024 Yuan et al. 2024). However, this approach introduces rubric design complexity and is bottlenecked by the judge's own capabilities and biases (Gao et al., 2023 Lightman et al.) 2023). Another direc- tion leverages self-reward, where the model generates its own feedback. These techniques often reward high-confidence or consistent answers, identified via mechanisms like majority voting or en- tropy minimization (Zuo et al. 2025 Cheng al. 2025). These objectives, while innovative, can penalize exploration, reduce response diversity, and reinforce the model's initial biases (Cui et al.
2025).
2

---

Published as a conference paper at ICLR 2026
Reinforcement Learning on Unverifiable Data. The most related work trains reasoning directly on non non-verifiable data, dispensing with external verifiers and proxy judges. These methods pioneer using the ground-truth answer itself the core reward signal. One approach treats the reasoning trace as a latent variable and optimizes for internal steps that maximize the likelihood of the correct final answer, often using the evidence lower bound related objectives (Tang et al., [2025] Zhou eet al., (025). Another emerging class uses the model's predictive confidence in the final answer as an intrinsic reward to guide policy updates (Yu et al. 2025b Chen et al. 2024). Recently, Di- rect Reasoning Optimization (DRO) introduced reasoning reflection reward that identifies salient tokens in reference answer guide training on open-ended tasks (Xu al., 2025). Our work, Native Reasoning Training (NRT), falls within this scope. It advances this paradigm with unified framework that formalizes these intrinsic reward mechanisms and can cultivate complex reasoning from base model using only question-answer pairs, without requiring reasoning demonstrations.
3 METHOD
We first review Supervised Fine-Tuming (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR), to contextualize our approach. Then we detail the training objective, optimization process, reward shaping, and mechanisms for enforcing structured reasoning trace of our proposed method.
3.1 PRELIMINARIES: STANDARD PARADIGMS FOR REASONING TRAINING
Supervised Tuning (SFT). SFT adapts pre-trained model To to mimic expert demonstra- tions from a dataset D. where each instance is a triplet (z, y²) of an input question, an expert reasoning trace, and final answer. The objective is maximize the conditional log-likelihood of the expert sequence o* = (z", expressed as:
JSFT (0) [(,,"" [log Te(2" (o(2",""(=)) ,=1=))]
As language models are auto-regressive, this log-likelihood the token sequence o* (I,,..,,An) is decomposed as:
log (0)00²
Σ log (t(2,,tt)))
This phase instills strong prior for reasoning structure, preparing it for subsequent optimization.
Reinforcement Learning with Verifiable Rewards (RLVR). Unlike imitation-based SFT, Rein- forcement Learning (RL) optimizes the policy no maximize an expected reward R(x,0) for generated output = (z,y). y). The general objective is:
max (o)) o)].
In RLVR, the reward is determined by an external verifier assessing the correctness of the final answer y against the ground truth y". The reward is typically binary:
Rveatter (y: y²) 1 (y=r")+
(4)
where 1 is the indicator function. The RLVR objective is thus:
JRLVR (0; v. 5²* (- [Ey=xe(-12,)) Reventer (y: y yr)]]
(5)
This objective, optimized via policy gradient algorithms like PPO (Schulman al. 2017a) or GRPO Shao et al., 2024), reinforces any reasoning trace z that yields a correct answer. The efficacy of RLVR contingent on reliable verifier, confining its application domains like mathematics and code generation.
3.2 NATIVE REASONING TRAINING
Native Reasoning Training (NRT) cultivates reasoning without explicit supervision on the reasoning trace, requiring only the input question and the reference answer y* The core idea is to treat the
3

---

Published as a conference paper at ICLR 2026
SFT Model TSFT
Verifier
Taksing Fornat Response Length
Rubrics
Groundtruthy
Math Problem
equation 0) (o), to simpliy and this equation. main step is </thinks
Solve 105 18 0 o.
the solution Aboxed(3).
Generated Reasoning Process
e Generated Answer
PPO/GRPO...
Question
Reinforcement Learning with Verifiable Reward (RLVR)
Estimate Difficulties
Pleve Ne
Base Model T
<thinla-/tthinko
Creative Kids Coloring Pages Coffee and...
Ø Null Reasoning Procesa
e
Reference Answer y"
00
Generate
Tube
Generate etsy title coloring pages witkin 140 ch...
<thenks To create a compaling title that these constraints... ... tnn the theme keywords "Creative Kide crthinko
Creative Kids Celoring Pages Coffee
Generated Reasoning Process
Reinforce
Reference Answer y
Question
VaJ(y)
Vi Valograasaale
V mula (0 is, Miss
Reward Calculation
Jnbby Gracleec
Tiken Protuction Geadun
Native Reasoning Training with General SFT Data
Figure 1: Comparison of Reinforcement Learning with Verifiable Rewards (RLVR) and our Native Reasoning Training (NRT). (Top) RLVR uses an external verifier to reward reasoning that yields an answer y matching the ground-truth y". This approach is constrained by its need for verifier. (Bottom) NRT operates on general SFT data, using only a question x and a reference answer y" trains the model generate latent reasoning trace by intrinsically rewarding traces that increase its own predictive confidence in the reference answer. This self-reinforcing process removes the need for external verifiers or expert-writte written reasoning.
reasoning trace z as latent variable and intrinsically reward traces that increase the model's own predictive confidence in the reference answer.
We formalize this principle the token level. Let the reference answer be y" (vi 17), A per-trace reward, R(= 0), defined by applying differentiable aggregation function f RR" R to the model's token-level conditional probabilities c;(2, 0) To (v," (x, 2, v.
R(2, 0) f(, 0), (2, 0). ..... or (2.
(6)
The NRT objective is the expectation this reward, encouraging the generation of helpful traces:
J(0) = (e--xo(( [R(,0)]
Maximizing this objective teaches the model to generate reasoning that is helpful for its token-by- token predictive process. We optimize this objective using off-policy reinforcement learning, As derived in Appendix the gradient can be estimated from batch of traces {2x 1 sampled from an older policy Told:
K
VeJ(O) R
Tn(
RiZk, 0)
V log (T(2)(2))
Trace Reward
Trace Policy Gradient
(8)
(1)(())(
Vo log no (v, L Uer
Token Reward Signal
Token Prodiction Gradient
4

---

Published as a conference paper at ICLR 2026
where (8) is the importance sampling ratio, (r,)(() is the conditional probability of token y and (0) 0f the reward function's sensitivity ci dec
This gradient provides two complementary learning signals. The first term is policy gradient update reinforcing holistically useful traces (z+) via the overall Trace Reward. The second a weighted, per-token policy gradient for answer prediction, which updates the model to better predict answer tokens (y,)) given the trace (zz), weighting each token's 's gradient by the Token Reward Signal. This dual mechanism jointly refines the trace generation and answer prediction policies, enabling self-correction of the latent reasoning process without external trace supervision.
3.2.1 INTRINSIC REWARD SHAPING VIA AGGREGATION FUNCTIONS
The choice of differentiable aggregation function f is central to NRT, shaping the intrinsic re- ward landscape by defining two key gradient components Equation [ the overall Trace Reward R(z, 0) = f(c), which scales the trace policy gradient; and the Token Reward Signal o;;;; Sc, c,. which weights each answer token's 's policy gradient.
8f
Different forms of f prioritize different aspects of the prediction task, such as holistic accuracy or improving the least-confident predictions. Table [ summarizes several aggregation functions and their resulting gradient components, derived in Appendix
Intrinsic reward shaping schemes derived from different aggregation functions f. C = (ut ( p., 2, w22) is the conditional probability of the i-th token of the reference answer.
Table I:
Scheme
Aggregation Function f(c)
Trace Reward R(2, 0)
01 Token Reward Signal D0,
Sequence Log-Prob (logP)
f(c) Σ j-1 log cj
log moly"" x, No(y"
Sequence Probability (P)
(c) IT S/
<o(""<>,
f(e) (II,-) 1/T
Geometric Mean (GM)
(To (z,,
$ (mols 2))
Arithmetic Mean (A.)
[(c) E//11
E
Weighted Sum (WS)
f(e) >T 2I
The Sequence Probabiibyyl and Arithmetic Mean (AM) schemes correspond to prior work, but unified framework reveals their potential pitfalls. AM, for instance, can be dominated by high- probability (easy) tokens, rewarding trivial or null traces (2) that fail to help with difficult tokens, which cause the policy collapse observed in our experiments (Sec|4.3).
This motivates using more robust functions like Geometric Mean (GM). The GM scheme is more robust than AM because a single near-zero probability token collapses the entire reward, forcing the model to perform well on all tokens.
The Weighted Sum (WS) scheme offers certainty. A powerful heuristic is to focus on difficult tokens, measured by their low probability Cr,basc Thsee (yāl Ix, 0, y24) from baseline model bescc e.g. Told, with a null trace (z 0). Setting weights inversely proportional to these probabilities encourages reasoning that resolves the model's own uncertainty. Table details two such implementations.
granular control by targeting points of highest
Table 2: Practical implementations of the Weighted Sum scheme that prioritize difficult tokens.
Weights wy are based on the model's baseline uncertainty, measured by Ci Tusee (vC (zc, 0, y2y))
Weighting Scheme
Weight Formula (w:)
Trace Reward R(z,0) Token Reward Signal o.e,
Inverse-Proobee Probability (1/p)
20 X
Σ/
Log- Probabiiiity (ologp)
W% X log Ca.e
T
1= Cy log Cy.hase
C2 log Case
These targeted weighting strategies use the model's baseline uncertainty to shape rewards, creating self-reinforcing loop that encourages reasoning address its predictive weaknesses.
5

---

Published as a conference paper at ICLR 2026
3.2.2 REWARD STABILIZATION
To stabilize policy gradient updates, the reward in Equation | is replaced with robust advantage estimate, A& (0), inspired by GRPO (Shao et al. 2024). For each prompt we generate a group of K candidate traces {2x lk=1 and compute their relative advantages.
The advantage calculation is a two-step process, First, clipped reward, R2 (0), is computed to focus the learning signal on traces that outperform a baseline. The baseline, Rease f(Cbase), the reward from an empty trace (= = (). The clipped reward for trace 2g is:
R2 (0) max (0, R(zk, 0) Rosse).
Second, these clipped rewards { Bkjkk= normalized to have zero mean and unit variance within their group. This yields the final advantage for trace Zg:
(0) = R2 mean( RR))
(10)
Rtd
where (R') and std( R') are the mean and standard deviation of the clipped rewards for the group. This group-wise normalization converts absolute rewards into relative ranking, significantly reducing gradient variance and focusing optimization on identifying the most effective reasoning traces among candidates.
3.2.3 STRUCTURAL FORMAT SUPERVISION
We employ structural format supervision to teach the model to clearly distinguish the reasoning trace from the final answer y. This supervision guides the model adopt specific output format: concatenation of (1) a start-of- rffaasoinng token sturt, (2) the reasoning trace 2, (3) an end-of end-of-thoughe thought token lend, and (4) the final answer y.
Instead of rigid constraint, we apply supplementary format- superviiu loss, Tifomat- This is targeted cross-entropy objective applied only the special tokens:
Lormat (0) - log @ (z) log T [r, Estarts 2).
(1I)
This format loss encourages the model to generate well-formed structures without harshly penalizing exploration. To ensure all samples are structurally complete for the policy update, we manually append fend if it is missing at the maximum generation length, thereby maximizing data utility.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Models and Dataset. We use the pretrained Llama-3.2-3B, Mistral-7B-v0.3 and Llama-3. 1-8B mod- els (Dubois et al. (222) for all experiments. All methods, including the SFT baseline and NRT vari- ants, are trained on a shared 200K-sample random subset of the falu-3-sft-mixe
dataset (Lambert et al. 2024). The dataset contains only question-answer pairs (ar, v²), with expert- written reasoning traces )=*). This setup ensures a fair comparison, as all methods start from the same un-tuned checkpoint and use identical data.
Methods. We evaluate our proposed Native Reasoning Training (NRT) methods against a compre- hensive set of baselines. The compared methods are: (1) SFT, standard Supervised Fine-Tuning model, our primary baseline. We also include our re-implementations of prior work, viewed as spe- eific instances of our NRT framework (see Table 5 in Appendix D for detailed comparison): (2) JLB (Tang et al.) (025). corresponding to our Sequence Probabbbyyyy scheme, maximizing the log- probabiiiity of the entire answer sequence; (3) Verifree (Zhou al., 2025), which aligns with the Sequence Probability scheme and maximizes the joint probability of the answer sequence; and (4) RLPR YYu et al. 2025b), an instance of the Arithmetic Mean scheme that maximizes the average probability of individual answer tokens. Finally, we test three NRT variants based on proposed reward shaping schemes: (5) NRT-GM, which uses the Geometric Mean to balance reward across all answer tokens; (6) NRT- WS (1/p). Weighted Sum scheme that prioritizes difficult tokens using their inverse baseline probability as weights; and (7) NRT-WS (- log a Weighted Sum scheme that uses the negative log baseline probability to focus the most challenging tokens.

---

Published as a conference paper at ICLR 2026
Table 3: Comprehensive evaluation across suite of nine general and reasoning benchmarks. The comparison is conducted on Llama-3.2-3B, Mistral-7B-v0.3, and Llama-3.1-88 1--B- models, all fine- tuned on the same 200k subset of tulu-uulue0 ------3-ff--ie---- xtuur..... NRT, particularly the weighted-sun variant, NRT-WS (- log demonstrates significant performance gains over all baselines. Bold and underline indicate the top two results per model.
"Our baseline implementations.
General Reasoning
Question Answering
Math
Code
Instruct
Method
Overall
BBH
MMLU
DROP PopQA
TQA
GSM8K MATH
HEval
IFEval
Llama- 1.ama-3.2--3- BB-Base
SFT
30.2
44.3
24.6
23.3
46.2
31.7
8.5
55.2
46.3
36.4
JLB
28.2
43.4
18.3
43.8
22.3
9.3
57.8
42.0
34.1
Verifree"
24.1
14.7
57.5
38.7
RLPR
28.4
43.4
23.0
18.3
44.8
33.0
57.8
43.3
35.7
NRT-GM
29.4
23.9
18.3
47.5
46.0
9.1
61.8
39.3
37.3
NRT-WS (1/p)
35.0
47.0
26.7
16.7
42.4
12.5
60.8
41.7
38.0
NRTWWS logp)
26.1
20.7
44.2
58.8
44.3
39,9
Mistral- 7B-v0.3
SFT
36.7
53.6
28.6
23.7
47.4
30.0
12.5
66.3
61.0
40.0
JLB
40.1
54.0
32.0
22.7
43.8
8.3
12.1
63.0
62.0
37.6
Verifrce
43.3
55.0
44.4
18.0
47.5
56.7
19.2
60.2
61.3
45.1
RLPR
40.4
55.1
19.7
23.7
43.4
65.3
19.0
66.6
60.3
43.7
NRT-GM
44.3
58.3
46.6
20.7
48.3
57.0
20.7
68.4
59.0
47.0
NRT-WS [1/2)
43.4
56.8
40.0
19.3
54.3
62.0
19.3
63.2
62.3
46.7
NRT-WS log
42.6
56.1
30.6
20.0
51.6
63.7
22.5
63.0
66.7
46.3
Llama-3.1 3.1-- 1-----BBse
SFT
38.0
36.7
30.3
45.5
29.0
17.8
74.7
58.3
46.0
JLB
40.2
59.0
36.1
31.0
46,9
54.0
19.0
74.3
58.0
49.0
Verifree*
35.7
58.3
33.5
27.7
46.6
54.3
19.4
76.3
59.3
48.1
RLPR
41.2
58.7
32.5
29.3
45.6
65.0
27.8
77.8
61.3
50.8
NKT-GM
54.3
66.1
48.7
26.7
47.3
70.3
32.2
76.3
55.3
$4.9
NKT-WS [1/p)
50.3
65.1
34.5
32.7
46.8
66.3
29.1
75.9
60.7
53.3
NRT-WS logp)
51.0
66.7
52.2
31.0
46.1
76.0
30.7
77.8
59.0
56.2
Evaluation. We evaluate all models on nine diverse benchmarks, groppmmarrs..uuupedddd into five categories: Gen- eral Reasoning (BBH (Suzgun al. 2022), MMLU (Hendrycks al. 2020)), Question Answer- ing (DROP (Dua et al. 2019). PopQA (Mallen et al. 2022). TruthfulQA (Lin et al. 2021) ). Math (GSM8K (Cobbe al. (021). MATH (Hendrycks et al.] (02T)), Code Generation (HumanEval (Chen et al. (021)), and Instruction Following (IFEval (Zhou et al., (2023). We report the pri- mary metric for each benchmark (Table 3 and the average score across all nine gauge overall performance. See Appendix G for further details.
Implementation Details. All RL methods are trained using GRPO Shao et al. 2024) with a learn- ing rate of le-5 and batch size of 256. We generate reasoning traces up to 2048 tokens. A small sureat-supervision loss (weight 0.3) encourage the specified output format. See Appendix
4.2
MAIN RESULTS
NRT establishes a new state-of-the-art for verifier-free reasoning, with particularly strong gains in complex, reasoning-intensin domains. As shown in Table its NRT methods achieve the highest average scores on Llama-3.2-42 3.2-3B, Mistral-7B-v0.33 and Llama-3. 3.1BBB models. On the 8B model, our best variant, NRT-WSS (- log scores 56.2 on average, +10.2 point gain over the SFT model (46.0) and a +5.4 point improvement over the strongest prior method, RLPR (50.8). This trend holds for the 3B model. where NRT-WSS (- log achieves 39.9 average, outperforming the SFT baseline by 3.5 points. Notably, on the smaller 3B model, other verifier-free RL methods (JLB, Verifree, RLPR) fail to surpass the SFT baseline, highlighting both the task's 's difficulty and NRT's challenging reasoning benchmarks. On GSM8K, NRT-WS (- log p) boosts the 8B model's score from 29.0 (SFT) to 76.0, far surpassing RLPR's 65.0. Similarly, NRT-GM improves the BBH score to 54.3, substantial + 13.1 points over RLPR,
robustness. The benefits
most pronounced

---

Published as a conference paper at ICLR 2026
demonstrating that NRT effectively elicits latent reasoning skills for complex, multi-step problems. For qualitative case studies illustrating these improvements, see Appendix !|
NRT's success is driven by its principled reward shaping, which guides the model to resolve its own uncertainty and outperforms simpler aggregation methods. While simpler variants like NRT-GM show significant improvement, the weighted-sum (WS) schemes consistently deliver the best results. These schemes validate the core hypothesis: guiding the model to generate reason- ing that resolves its own uncertainty highly effective. By prioritizing tokens the model finds difficult (i.e., those with low baseline probability), NRT-WS (- log becomes the top-performing method overall for both model scales, securing the highest scores on GSM8K, MMLU, DROP, and Evl on the 8B model. Its superior performance over methods that use simpler aggregation functions- succh as RIPR (Arithmetic Mean) and JL.B (Sequence (og-Probability) pproves that the structure of the intrinsic reward is key factor in the success of verifier-free learning.
4.3 ANALYSIS OF TRAINING DYNAMICS OF THE REASONING PROCESS
NRT effectively avoids the common RL pitfalls of mode collapse and policy degradation, main- taining diverse, lengthy, and high-quality reasoning throughout training. A significant chal- lenge RL-based fine-tuning is mode collapse, where the policy converges to simple, low-reward outputs, ceasing to explore more complex strategies. We track the evolution of the reasoning process z during training across three metries: entropy, token length, and semantic quality (evaluated via an LLM judge; see Appendix H for details). As shown in Figure 2 the RLPR baseline exhibits a rapid degradation on both 3B and 8B models. It converges to a state producing short. low-entropy reason- ing traces (Figures 2a 2db that receive consistently low quality scores (Figures 2e This confirms that the baseline settles into a poor local optimum, generating repetitive or trivial content that fails to support logical deduction. In contrast, all NRT variants sustain high-entropy and long reasoning processes while maintaining or improving reasoning quality throughout training. This alignment between structural metries (length, entropy) and semantic quality verifies that the NRT models are not merely "bbbbliin" to increase length, but are actively engaging in substantive reasoning steps.
NRT's superior training stability stems from a reward mechanism that incentivizes the gener- ation of genuinely helpful reasoning over superficial shortcuts. As analyzed in our framework (Sec 3.2. the RLPR baseline's collapse can be attributed to its Arithmetic Mean reward, which may inadvertently reward empty or trivial traces if they maximize the probability of a few easy-to- predict answer tokens. NRT-GM, through its geometric mean, penalizes any single token with a low probability, requiring the model to generate holistically helpful reasoning. More powerfully, NRT- WS schemes force the model to confront its weaknesses by up- up-weighting difficult tokens. To earn a high reward, the model must conduct complex, multi-step reasoning to resolve its most significant predictive uncertainties. This mechanism prevents the quality degradation observed in the baseline and translates directly to the superior downstream performance in Table 3
4.4 ANALYSIS OF GROUNDTRUTH TOKEN PROBABILITIES
NRT's targeted reward shaping effectively teaches the model to resolve its predictive weaknesses, significantly boosting confidence on the most difficult tokens. To understand how NRT improves the model's predictive capabilities, we analyze the confidence change for each ground-truth answer token. We categorize tokens by difficulty, measured by the SFT model's pre- diction entropy (Hsrr). As Figure Ba shows, this difficulty distribution highly skewed: while most tokens are easy for the baseline to predict (low entropy, median 0.125), a long tail of highly uncertain, hard tokens exists. These hard tokens are precisely where reasoning most critical.
Weighted-sum NRT schemes uniquely focus learning on these hard-to-predict tokens, where other methods fail or degrade performance. Figure plots the relative improvement in token probability (P/ PSET) against the token's 's baseline entropy, where a ratio 1 indicates improved con- fidence over the SFT baseline. The results reveal a clear divergence: RLPR and NRT-GM methods show negligible gains, with confidence hovering around or even dipping below the baseline for high- entropy tokens. In contrast, both weighted- sum (WS) variants exhibit a strong positive correlation. For the highest-entropy tokens, NRT-WS (1/p) increases the assigned probability by up to 15%, providing direct evidence that our targeted reward shaping works as designed, forcing the model to generate reasoning that addresses its most critical points of confusion. By focusing the learn-
8

---

Published as a conference paper at ICLR 2026
ALPR
NRT-GM
NRT--GM
NRT-WS (Lip)
NRT-WS Cl/p)
NRT-WS log
NRT-WS
Training
Training
(a) Reasoning Process Entropy (Jaam-332--BB
(b) Reasoning Process Entropy (Llama-3.1-80)
2000-
basiin))
baselee
37500
3000
750
500
1.00
2000
300
400
500
600
700
800
1.00
200
300
400
500
600
700
800
Taining
(c) Reasoning Process Length (Llama-3.2-3B)
Reasoning Process Length (Llama-3.1885
3.0
3.0
0.6
0.6
0.4
0.4
0.2
0.2
3.0
0.0
300
400
800
1.00
200
500
600
Traiiing Step
Training Step
(c) Reasoning Process Quality (Llama-3.2-3B)
(f) Reasoning Process Quality (Llama-3. 1-8B)
Figure 2: Evolution of the reasoning process during RL fine-tuning across three dimensions: di- versity (entropy; a, b), length (c, d), and semantic quality (e. f). Quality is measured by an LLM- as-a-judge using 0-1 score (see Appendix HH. While the RLPR baseline suffers rapid collapse across all metrics, producing short, repetitive, and low-quality reasoning, NRT variants sustain high- entropy. lengthy, and semantic reasoning on both Llama-3.2-3B and Llama-3.1-8P models. This demonstrates our approach prevents mode collapse while maintaining reasoning integrity.
ing signal where it is most needed, NRT-WS creates powerful self-rein renfrreeffore mechanism that translates directly to the enhanced reasoning capabilities in our main results.
5 CONCLUSION
We introduce Native Reasoning Training (NRT), a framework that overcomes the dependencies of the SFT+RLVR paradigm. By treating reasoning latent variable. NRT intrinsically rewards the model for generating reasoning traces that increase its predictive confidence in the reference answer. This self-reinforcing mechanism cultivates complex reasoning using only question-answer

---

Published as a conference paper at ICLR 2026
Normaltzed Fequency
0.6
Mean: 0,631
Median: 0.125
Bolahive Token Probablity
1.125
3.15-
MAT4WS 12p0
RLPS
NAT-MM
Sari Change)
0.4
2.330
MXT-WS (-loga)
3.075
0.3
1.050
3,025
1.000
0.0
0.975
Entropy Hor
Entropy Hor
(a) Distribution of token entropy
(b) Relative token probability
Figure 3: Analysis of how NRT improves prediction of ground-truth tokens, particularly for those the SFT baseline finds most uncertain (high entropy, HsFT). (a) The distribution of token entropy, showing that while most tokens are easy to predict. long tail of high-entropy tokens exists. (b) Change in relative token probability (P/ PsFr). Weighted-sum NRT schemes (WS) provide the largest confidence gains for high-entropy tokens, effectively targeting the model's weaknesses.
pairs, climinating the need for expert-written demonstrations and external verifiers. Our experiments show NRT achieves state-of- the-art results for verifier-free learning, significantly outperforming SFT baselines and prior methods in complex reasoning domains. We show that our unified frame- work, especially via robust reward shaping like NRT-WS to target model uncertainty, effectively resists policy collapse. Ultimately, NRT extends reinforcement learning general and unverifiable domains, offering a scalable path toward more powerful and broadly applicable reasoning systems.
Limitations and Future Directions. NRT opens several exciting avenues for future exploration. The design of the reward aggregation function is a key new lever for influencing model behavior. While our work demonstrates the power of principled, handcrafted functions, we see significant po- tential in exploring adaptive or even fully learned reward schemes, which could further automate and enhance the reasoning acquisition process. In terms of computational resources, NRT's sampling- based approach represents a trade-off that prioritizes performance and applicability in verifier free domains. We view this as a worthwhile investment and anticipate that future work on more sample- efficient optimization algorithms could improve its efficiency. Finally, our current study focuses on fine-tuning, but applying NRT's principles during pre-training is compelling next step. This could instill foundational reasoning abilities directly into the model from its inception, paving the way for a new generation more inherently capable reasoning systems.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility, we provide comprehensive details of our methodology. The theoret-
ical framework for Native Reasoning Training (NRT), including our objective function and reward
schemes, is presented in Section 3 with complete mathematical derivations in Appendices B and C
Our experimental setup is described in Section supported by details on the dataset (Appendix D
complete list of training hyperparameters (Appendix [F. and the full training procedure in Algo-
rithm [ (Appendix E). All evaluations were conducted using the standardized OLMES framework, with benchmark configurations detailed in Appendix G
ACKNOWLEDGMENTS
This work was supported by Shanghai Artificial Intelligence Laboratory.
REFERENCES
Haolin Chen, Yihao Feng. Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, and Huan Wang. Language models are hidden reason- ers: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint MXX1:2441..04280 2024.
10

---

Published as a conference paper at ICLR 2026
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXhv:2107.03374, 2021.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang. and Furu Wei. Reasoning with exploration: An entropy perspective on reinforcement learning for Ilms, 2025. URL httth:ttt://// ptactpps//aaxiv.ooroer abs 2506 14758
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 14168, 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22611 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. Deepscck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12942 2025.
Dheeru Dua, Yizhong Wang. Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprchension benchmark requiring discrete reasoning paragraphs. arXiv preprint 193300903..0161, 2019.
Yann Dubois et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Yuling Gu, Oyvind Tafjord, Bailey Kuchl, Dany Haddad. Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.084466 2024.
Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, and Amelia Glaese. Deliberative alignment: Reasoning enables safer language models. arXiv preprint 1rXiv:2412.163393 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.:::0003038744 2021.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15122 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00261 2023.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0795: 2021.

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non- paraameetttic memories. arXiv preprint arXiv:2212.10511 2022.
MiniMax, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, et al. Minimax-ml: Scaling test-time compute efficiently with light- ning attention. arXiv preprint arXiv:2506. 13585, 2025.
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed EI-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, and al. Openai ol system card, 2024. URL https:
orq hbs//bs//e 2412.16722
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017a
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017b. |ttps://///ax/x abs///bb///// 1707. .63477
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint 00XV:2400.003300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihí framework. arXiv preprint arXiv: 2409. 19256, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärf, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.090 2022.
Yunhao Tang, Sid Wang, Lovish Madaan, and Rémi Munos. Beyond verifiable rewards: Scaling re- inforcement learning for language models to unverifiable data. arXiv preprint arXiv:2503. 19618, 2025.
Robert Vacareanu, Anurag Pratik, Evangelia Spiliopoulou, Zheng Qi, Giovanni Paolini, Neha Anna John, Jie Ma, Yassine Benajiba, and Miguel Ballesteros. General purpose verification for chain of thought prompting. arXiv preprint arXiv:2405. 00204, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of tf-toughh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24832 2022.
Yifei Xu, Tusher Chakraborty, Srinagesh Sharma, Leonardo Nunes, Emre Kıcıman, Songwu Lu, and Ranveer Chandra. Direct reasoning optimization: Lims can reward and refine their own reasoning for open-ended tasks. arXiv preprint arXiv:2506. 13351, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangje Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya- Vi-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system scale, 2025a. URL. https:
7/arxiv. verxiivooe www.vvrrg/bb.... 22../25511111... 14476)
Tianyu Yu. Bo Ji, Shouli Wang. Shu Yao. Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. Rlpr: Extrapolating rlvr to general domains without verifiers. arXiv preprint arXiv:2506.1 182544 2025b.
12

---

Published as a conference paper at ICLR 2026
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbuatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.1022,3 3, 2024.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization. arXiv preprint 2807:22577.1877, 2025.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folowiig evaluation for large language models, arXiv preprint arXiv:2311007911, 2023.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493, 2025.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuckai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint 200:25044.0844 2025.
13

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS (LLMS)
We declare the use of Large Language Models (LLMs) in this research work. The LLMs serve supportive role in the following aspects of this project:
Writing and Language Polishing: LLMs assist in improving the clarity, readability, and grammati- cal correctness of the manuscript. This includes refining sentence structure, improving word choice, and ensuring consistent terminology throughout the paper.
Code Development Assistance: LLMs provide assistance in writing and debugging experimen- tal code, including data preprocessing seripts, training pipelines, and evaluation frameworks. The models help with syntax checking, code optimization suggestions, and implementation guidance for standard machine learning practices.
Literature Review Support: LLMs assist in reading and summarizing research literature identify relevant prior work and contextualize our contributions within the existing body of knowledge. This includes assistance with understanding complex technical concepts and identifying key papers the field.
The core research ideas, experimental design, theoretical framework, and scientific contributions presented in this work are original contributions by the authors. The LLMs do not contribute to the fundamental research conception, hypothesis formulation, or interpretation of results. All experi- mental work, data analysis, and conclusions are conducted and drawn by the human authors.
B FULL GRADIENT DERIVATION FOR THE NRT OBJECTIVE
This section provides a complete derivation for the policy gradient our generalized token-level objective. The objective is designed to train model to generate latent reasoning traces that improve its prediction of a ground-truth answer y".
B. FORMULATION OF THE TOKEN-LEVEL OBJECTIVE
Let y" (yi y7) be the ground-trufe truth answer for given input æ. Our model first generates a latent reasoning trace 2 from a policy To To(2 (z|z). Conditioned on this trace, the model predicts the answer. For a given trace 2, let (2 0) be the conditional probability of the i-th ground- -ruth token:
c1 0) no(y? [r, 2, 2.2)))
(12)
We define per-trace reward function, R(z, 0), by applying a differentiable aggregation function f R + R to the vector of these conditional probabilities:
R(z, 0) (c1 e2(2, (2(
(13)
This function f evaluates the quality of entire sequence y". Examples for include the arithmetic mean, f(c) = Σ, c₁ or the geometric mean, f(c) (II,
based how well it facilitates the prediction of the
The overall training objective J(f) is the on-policy expectation of this per-trace reward:
J(O)= = Eswnd( [[2,, 0)] = Σ ((2,),))(2,,).
(14)
B.2 GRADIENT DERIVATION WITH IMPORTANCE SAMPLING
To improve sample efficiency, employ importance sampling enable off- off-policy updates using traces sampled from an older, fixed policy vold- The objective equation 14 is rewritten as:
J(0) = Esonma(tll))
[= R(= 0)
(15)
Let r(2,0) ra(=)m) independent of 8, move the gradient operator inside:
mai denote the importance ratio. Since the expectation is over distribution
V (oJ(O)
E ( (V)(((2,
(16)
14

---

Published as a conference paper at ICLR 2026
Applying the product rule for differentiation yields two terms:
(o(r(z,0) 0)) (Vor(z, [[= (V..
(17)
Term Policy Gradient Term. The first term of equation 17 the standard policy gradient
component, By applying the log-derivative trick, Vg gV log to the importance ratio, we have:
Ver(z, 8) ()V, log r(2, 0) r(2, 0 V log (z(2)))
(18)
This gives the first term of the gradient as:
(Vor(2, R(z, 0) = r(z, 0) R(2, 8) Ve log (2)2)))
(19)
This term adjusts the trace generation policy my m to favor traces with high reward R(2,
Term 2: Differentiable Reward Term. The second term of equation 17 accounts for the de- pendency of the reward function R(2,0) the model parameters through the token proba- bilities (z 0). Using the multivariate chain rule and the log-derivative trick, we differentiate R(z,0) [(T
T
as Voci(z,0)
Vo 0)
Σ
(20)
dc:
i=1
0f
(ogg
Zcg
T
0f
ci(z, 1
The full second term is therefore:
T
Of dc: cz V2 log To ver).
r(z, (V,R((() = r((,
Σ 1=1
(21)
This term constitutes weighted supervised learning signal, where each ground-truth -tuth token's ' log- likelihood gradient scaled by its contribution to the overall trace reward.
B.3 FULL GRADIENT ESTIMATOR
Combining the expressions in equation 19 and equation 21 yields the full gradient of the objective:
(0) = E (x ((2,0) R(z, 0) log (0(1))
(22)
T 0f dcy
ci(z, 0) log 2,21)
In practice, estimate this expectation using Monte Carlo approximation with batch of K traces {2x |k sampled from (ou((22)) ()). Let (0) r(2,, 0), (0) c2(2k, 9), and let (1, 8f be the partial derivative evaluated (0), CT (T)) The gradient estimator is:
K
1
VoJNNe (0)
(t)
R(zk. B)
V log To (2x|2)
K
Reward
Policy Gradient
(23)
(.)C()))
Ve log To(y (z, v2 2k)
Token Reward Signal
Token Prediction Gradient
This estimator provides a principled method for training a model to generate helpful reasoning traces, it rewards both the trace generation and token prediction policies based their tribution to the aggregated objective. All quantities are readily computable for each trace 2 in batch.
15

---

Published as a conference paper at ICLR 2026
C GRADIENT DERIVATIONS FOR COMMON AGGREGATION FUNCTIONS
In this section, we instantiate the general gradient estimator from equation 22 for several concrete choices of the aggregation function f. For each case, first define the function, then compute its partial derivative 0f which is required for the Differentiable Reward Term. This allows us derive specialized form of the full gradient. The key term we derive for each case is the "Token Reward Signal", a;c;
de;
0f (2, 0).
deg
C.I SEQUENCE-
LOG-PROBABIIE
Let be the sum of the log- gg-probabiliites, equivalent to the log-probability of the sequence y".
R(2, f(c)
log log No (y x., 2)-
(24)
The partial derivative with respect to o; is simple:
0f
(25)
dc:
c2
The token reward signal becomes a constant:
af dc,
cz(2,0) 0) c;(2,0) 0) (2(2,) = 1.
(26)
The full gradient simplifies to:
( = E Ez- 0(2,0) R(2, loggo
T
(27)
Σ
Vo log To vo 2.2))
This gradient has clean interpretation: it is sum of the standard policy gradient term, where the reward is the log-probability of the answer, and standard maximum likelihood (supervised) term for the answer tokens.
C.2 SEQUENCE-LEVEL PROBABILITY
Let be the product of the conditional probabilities, which corresponds to the sequence-level prob- ability To (yyy"2z 2).
T
R(2, 0) f(c) II c To(y" 2).
(28)
j=1
The partial derivative of f with respect to cj is the product of all other probabilities:
- R(2,0)
as
II 9
(29)
dc,
(2(2
Jfi
The token reward signal for the i-th token is therefore:
0s cx(2, 0) cx(2.)) 0)
R(z,0 A)
ca (2,0))R((2 R(2,0).
(30)
Oc; Substituting this into the general gradient formula equation
22 we can factor R(2, 0):
(@) Eswma((>))) ((2. (22,0))(2200 0)
V log (2(2))
(31)
V log To(ui 2..0))
where the trace reward R(2,6 8) acts as a scaling factor for both the trace policy gradient and the standard supervised gradients for the answer tokens.
16

---

Published as a conference paper at ICLR 2026
C.3 GEOMETRIC MEAN
Let be the geometrie mean of the conditional probabilities.
T
R(2, 0) f(c)= =
II
Cj
(32)
Using the chain rule, the partial derivative with respect to c; is:
+-1
[
Cs) 1/T
0f dcN T
T
R(2
=
II jil
T
T,((22
(33)
The token reward signal is therefore scaled by 1/T:
0f dc;
R(z, 0)
R(2,0)
(2, 0) T. (F,((, (2 (2,0)
(34)
T
Similar to the sequence-leve probability case, we can factor out the reward R(2, 0):
VoJ E gnmuls ((2,0))(2,0))
Vo log (0(2)))
(35)
T
Vo log oo
T
The gradient has similar form to the sequence-level probability case, but the supervised signal averaged.
C.4 ARITHMETIC MEAN
Let be the arithmetic mean of the conditional probabilities.
=
1
Σ
Cj-
(36)
j=1
The partial derivative with respect to e; constant:
Øs dc; T
(37)
The token reward signal for the i-th token is:
0f
c2,20) cz(2,0)
(38)
De,
Plugging this into the general formula gives the full gradient:
((2,0) [(22, () log (2|2)
VoJ(O) = E. Ex-ma((Z))
(39)
T T
a(2 Vol E
In this case, the supervised gradient for each token y is weighted by its own conditional probability (2, 0), encouraging the model to focus tokens it is already confident about.
17

---

Published as a conference paper at ICLR 2026
C.5
WEIGHTED SUM
Let be weighted sum of the conditional probabilities, where {wi) are non-negative weights.
7
R(z, - f()) Σ wyeg.
(40)
The partial derivative with respect to e; is simply the corresponding weight:
0
wi.
(41)
dc,
The token reward signal for the i-th token is:
0s
(42)
dc,
The full gradient is generalization the arithmetic mean case:
V ( = E ma0=(()) f(2.0)
R(2, R(2,0)Vo logmo((())))))
(43)
Σ wicl(z, 1o
This formulation allows for explicit control over the importance of each token in the supervised learning objective, for example, by assigning higher weights w; to more critical tokens in the answer. The arithmetic mean is special case where w; 1/T for all i.
D DATASET DETAILS
The primary training dataset used across all our experiments 200K-sample subset randomly selected from zulu-3-sft-mixtuee (Lambert et al., 2024). This is diverse, large-scale col- lection high-quality instruction-following data. A key characteristic of this dataset is that ex- clusively contains question-answer pairs (x,y") y²) and does not include any human-annotated "gold" reasoning traces (=*). The detailed composition of our 200K-sample subset is provided in Table 4
Comparison with Baseline Dataset. A crucial distinction between NRT approach and the baseline methods lies in the training data and the nature of the responses used during optimization. 5 a summary of these differences. While all methods operate verifier-free setting (i.e., without requiring external tools to check reasoning steps), their underlying data assumptions and response characteristics vary significantly:
JLB (Tang et al. 2025) is demonstrated on "unverifiable" data, specifically -amoof, which contains long-form proofs without distinct, short-form answer for automated checking. This differs from our setup, where the ground truth is short, verifiable answer y*
Verifree (Zhou et al. 2025) utilizes WebData, a custom 61K sample dataset derived from WebInstruct. This dataset is intentionally curated to have very short answers (fewer than tokens), enforcing strong constraint on the response length.
RLPR Yu et al. 2025b) is trained on 77K non-mathematical prompt set. While it generates some reasoning, average response length remains short (around 13 tokens).
NRT (Ours), in contrast, is trained on general-purpose instruction-following dataset (tulu-3-sae (eftmmixture) that lacks pre-existing reasoning steps. Unlike methods designed for short-form outputs, our training data's reference answers (y²) are often extensive, averaging over 400 tokens. The model must therefore learn to generate its own reasoning traces "natively" to bridge the gap from prompt to these complex answers. This highlights NRT's s ability to elicit reasoning from standard question-answer pairs, a key departure from baselines that either rely on specialized long- form data (JLB) constrained to short responses (Verifree, RLPR).
18

---

Published as a conference paper at ICLR 2026
Table 4: Composition of the 200k-s training dataset, culu-3-sft-mixture
randomly
selected from the
Source
Count
Percentage
(12=adapp-dee/jeesoonabbb math regen 149960
33,036
16.13%
23,778
11.61%
112-adapt-dee//ulu.22 100_
22,088
10.79%
20,010
9.77%
22==adatt-deeuuluuu3..
19,378
9.46%
ii--ddapt-ee//mmminaaath
hat.://lle..-ttteeeeeommmaaaaaa- saathhddcotamminted
18,999
6.84%
11,115
5.43%
al2-adaap--dev/ulluy333 appe__mat_2_ggmneee
11,081
5.11%
ai2-adapt-dew/uulu. v3.9 wildjailbreak decontaminated 50k
11,069
5.40%
ai2 ai2-adapt-dev/uulu v3.9 synthetic finalresp wildguardmixxtriin...
10,935
5.34%
uiZ-adapp-dev/perronnnuuu.c. _od.d__2.ed9e99999
7859
3.84%
iiz-udapptdeev/personaubbb wttp:uiiiiaaaaaaaaaaaaaaa wtt.:/aaamaaaullsseer33333222392
6483
3.17%
ai2-adaap--dcvyuluu..3 v3.9
a22-adapp-dde//cccooo..ommm http://uuuu_u_l.oppppoooooooauba
4529
2.21% %
2441
1.19%
ai2-adapt-dewwuulu v3.9 sciriff 10k
2177
1.06%
ai2-adapt-dev/no.nmm robots converted
2114
1.03%
ai2-audapt-devvoasstl converted
1606
0.78%
ai2-adapt-dev/ullu.v v3.9 wt..:u............e.eee.. ..able-gpt.kk
1062
0.52%
ai2 aa-aadpttt-e//uu www.eaaddddd.ddeeeeeoooooo. repeated 10
0.02%
TOTAL
204800
100.00%
Table 5: Comparison of verifier-free RL methods. This table details each method's aggregation function f(c), reward ormalization, and the characteristics of the responses used for training. Note the significant difference in average response length between our NRT methods, which generate long reasoning traces spontancously, and the baselines, which often use datasets with short or specialized responses.
Method
Aggregation Function
Reward Normalization
Average Response Length
JLB
f(c) Σj=1 log Cj f(c) IT;=1 9j f(c) + Σラ= Cf
RLOO
Unknown.
Verifree
RLOO
S tokens.
RLPR
Uses clipped reward Tfinal = clip(r - r', 1), where r' is the score for generating y* without reasoning 2.
12.52 tokens.
f(c) = (IT=
1/T
NRT-GM
Normalized advantage of the clipped reward difference, max(0,r-2) r²). where r' is the score for generating y with empty reasoning
415.97 tokens.
NRT-WS
f(c) ΣJ=1 Wz3g
Normalized advantage of the clipped reward difference, max(0,r r- r"). where r' is the score for generating y with empty reasoning 0.
415.97 tokens.
19

---

Published as a conference paper at ICLR 2026
Algorithm Native Reasoning Training (NRT)
Input:
Pre-trained language model Tg-
Training dataset D {(x, y")}
Differentiable aggregation function ʃ (see Table T).
Hyperparameters: batch size B, traces per prompt K, learning rate 77, format loss weight Mormat-
Output:
Trained reasoning model s-p-
Initialize:
I: Initialize sampling model: Told t 78.
2: for each training step do
3: Sample minibatch B {(T,, Mz }} from D.
4: Sample a group of K traces { [= ~ Tota (z|ar, tsmn).
5: Append fend any incomplete trace 2k- Calculate baseline reward: Rouse s- --f(fTor(y? (x, 0, y v2
6:
7:
Calculate trace reward: Rs <- (((ooly xr, 2k, y2 1i=
8:
Compute clipped reward: R2 4- max (0, Rg R Rease (Eq. 9).
9:
Compute advantages: {Aa]
(I')
(Eq. 100.
Rtd(R))
10:
Compute NRT loss gradient
11:
Calculate token reward signals: {St
of
cakk 1i=1-
Be:
12:
LK f- Ag V log (T(==())) (x)) Et= Se,t Vo log Tel (w, 22 =k).
13:
> Compute format loss gradient
14:
Lkk, formatt Vo log N (tsun xx) + log To (tetaual x zx)).
15:
V Tatotal < ΣE V NRT + Aroemat V I. [...format)
16:
Update model parameters: 4- 0 - 1) V Laotal
17:
Update sampling model: Told ng.
18:
end for
19:
return To.
E THE NRT TRAINING ALGORITHM
The Native Reasoning Training process implemented as an off-policy reinforcement learning loop. detailed in Algorithm] I The training cycle begins by initializing the policy me from suitable SFT checkpoint. Then, for each prompt (x,y") in a training batch, the following steps are executed.
First, a group of K candidate reasoning traces { is sampled from frozen, older policy Told- This decouples data generation from the gradient update, enhancing stability. Next, an intrinsic reward is calculated for each trace by evaluating the current policy's confidence in predicting the reference answer y*- To reduce variance, these rewards are stabilized using the GRPO-inspired advantage mechanism from Section 3.2.2] rewards clipped against a no-reasoning baseline and then normalized group-wise to produce final advantage estimate Ak-
Finally, the advantage is used compute composite gradient signal defined in Equation g This gradient combines policy gradient update for the entire trace 21. with a weighted supervised update for the answer tokens y". A supplementary formaatsuperviiion loss helps enforce the desired output structure. After the model parameters 0 are updated, the sampling policy Vold is synchronized with the new policy, completing the loop. This iterative process allows the model to progressively refine its own reasoning abilities without external trace supervision.
F TRAINING HYPERPARAMETER, AND IMPLEMENTATION DETAILS
All our Reinforcement Learning experiments, including the NRT variants and re-implemented RL baselines, are trained using the GRPO algorithm (Shao et al. 2024) based on the VERL (Sheng et al.] 2024 training framework. To ensure a fair comparison. the re re-imlemeeted RL baselines incorpo- rate the same reward stabilization (Section (.2.2) and structural format supervision (Section (32.3) our NRT methods. This was necessary adaptation, as the original RL. methods were designed for
20

---

Published as a conference paper at ICLR 2026
instruction-tune -uned models and not directly applicable to the base models used in experiments, which require explicit guidance to generate structured reasoning. We leverage Dr.RRPO (Liu et al. 2025 for length normalization, which is designed to enhance stability and performance. The models are trained for a single epoch over the 200K-sample dataset.
Table 6: Hyperparameters for NRT and baseline RL training.
Parameter
Value
General Training Configuration
Learning Rate
le-5
Learning Rate Scheduler
constant
Weight Decay
0.01
Training Epochs
1
Training Batch Size
256
Max Prompt Length
4096
Max Response Length
4096
Thinking Format SFT Loss Coef.
0.3
GRPO Algorithm Parameters
Advantage Estimator
GRPO
Gamma
Length Normalization
DrGRPO
PPO Mini-batch Size
64
PPO Clip Ratio (Low)
0.2
PPO Clip Ratio (High)
0.28
Generation
Rollout Backend
vLLM
Generations per Prompt (N)
8
Temperature
Top_P
Max Generation Tokens (Reasoning)
2048
Regularization
KL Loss Coefficient
0.0
Entropy Coefficient
0.0
During the RL rollout phase, we generate 8 responses for each prompt the training batch, with each generated reasoning trace capped at maximum of 2048 tokens. This process is acceler- ated using the vLLM inference engine (Kwon et al.) 2023). To encourage the model to explicitly generate its reasoning process within the desired structure, we incorporate with a coefficient of 0.3. This loss incentivizes the model to use the < t/thikkstaart/ and think end special tokens. We did not employ KL divergence or entropy regularization in our experiments. The key hyperparameters for our training setup are summarized in Table
auxiliary SFT loss
G EVALUATION BENCHMARKS DETAILS
To ensure our results are rigorous, reproducible, and comparable to prior and future work, we adopt the Open Language Model Evaluation Standard (OLMES) toolkit for all benchmark evaluations [ OLMES is standardized, open-source framework designed to mitigate common sources of variance LLM evaluation, such as subtle differences in prompt formatting, the choice of few-shot exam- ples, and score normalization techniques (Gu et al. 2024). By using unified evaluation harness, we can confidently attribute performance differences to our training methods rather than to incon- sistencies in the evaluation setup. Our evaluation suite comprises nine diverse benchmarks, selected to provide a holistic assessment of model capabilities across general knowledge, reasoning, truthful- ness, mathematics, reading comprehension, code generation, and instruction following. The specific configuration for each benchmark, including the prompting strategy. number of in-context examples
lenai
21

---

Published as a conference paper at ICLR 2026
Table 7: Details of the evaluation benchmarks and their configurations within the OLMES frame- work. CoT denotes Chain-of- -hain-of-Thhuugt prompting.
Benchmark
Domain
Prompting Strategy
Shot Count
Metric
MMLU
General Knowledge
Zero-shot CoTT CoT
0
Accuracy
BBH
General Reasoning
Accuracy
DROP
Reading Comprehension
Zero-shot Chat
F1 Score
PopQA
Fact-based Q&A
Standard Prompting
15
Accuracy
TruthfulQA
Truthfulness
Standard Prompting
Accuracy
GSM8K
Mathematical Reasoning Mathematical Reasoning
Zero-shot CoT
0
Accuracy
MATH
Zero-shot Reasoning
Accuracy
HumanEval
Code Generation Instruction Following
Standard Prompting
Pass
IFEval
Standard Prompting
Pass @ 10
(shots), and primary evaluation metric, is detailed in Table 7 provided within the OLMES framework.
All configurations are standard settings
H SEMANTIC QUALITY EVALUATION DETAILS
To assess the intrinsic quality of the reasoning processes generated during training, we implement automated evaluation pipeline using the grok-4.1-fast 1-fast model as a judge. We prioritize identifying logical coherence and faithfulness in the reasoning trace rather than strictly checking the final answer format. To ensure reproducibility and minimize variance in the scoring, 0.0. For every model checkpoint analyzed, we randomly sample validation set and query the judge model. The specific prompt template used for this evaluation is detailed in Table 8 inputs are inserted into the placeholders { {reasoning), and truutn prior to inference.
set the temperature to 100 instances from the
I QUALITATIVE ANALYSIS OF NATIVE REASONING MODELS
This section provides a more in-depth qualitative analysis the reasoning processes learned by NRT models, supplementing the quantitative results in the main paper. We first perform aggregate analysis of the vocabulary that emerges in the generated reasoning traces. We then present specific case studies that illustrate how this learned reasoning corrects the failures of baseline models individual problems, demonstrating the practical impact of our approach.
1.1 EMERGENT VOCABULARY OF REASONING
To better understand what the model learns to "think." " we analyze the vocabulary of the reason- ing traces (2) generated by our best-performing model, NRT-WS (olgg)). and compare it the vocabulary of the ground-truth answers (y*).
NRT autonomously develops distinct "language of reasoning" rich with procedural and meta- cognitive terms. As shown in the word cloud in Figure 4p. the generated reasoning processes are dominated by words associated with problem-solving "given", and "using". This indicates the model is not merely repeating the problem statement but actively constructing a plan of attack. More revealingly, the frequency analysis in Figure 4a highlights the words whose usage increases most dramatically in the reasoning trace compared to the ground-truth data. Terms like "premise," "developed," "reasoning," " and "mentioned" show the highest increase, suggesting the model has learned externalize its meta-cognitive process: laying out premises, explaining its logic, and framing its internal queries.
procedures, such as "let". "step", "solve",
The model successfully learns to disentangle the reasoning process from the final answer, confin- ing answer-specific formatting to the output. While the reasoning and ground-truth word clouds (Figures 4b) and 4dh share problem-solving terms, there is clear separation of function. The
22

---

Published as a conference paper at ICLR 2026
committed
regarding
s
provide
Lf
hour
py t
pased
resgense
honmany
mode
Code
right
number
equation ion
return aata
transidion
different mean
Tind
year
Left
Huggnr
casagesulte
worrd
tchar
acts
Tettot
key user
Text
thout
tre
first
mentlored
reques
viven
will
explaration
Sentence
acking
o
ee
pe
fi
List
helgfll
Epecific
data
a
help
cost
st
count
Lil
m
Typee
Li
developed
onglih
x 10
10f
10f
used cal
ees usepup departt
day s um people
calcula attee
S
step
baak
aum
aamount
so
answer
coecle
need
name
new
the
poond
unction
Breste
C
Frac
string p
ona
(a) Words with highest increase.
(b) Word cloud of generated reasoning process (:).
10-
conditio em in "G
emusing
naxinum
U year year.
Saater hour pl
mod
kan
lution
correr
number
rect
new miad Bennine
WeW
equat
output
thlle se Equad In
Deg S tep
simpiited
Function
nome
das wer
infty
pe-
du-
the
Lime
system
siphs
co
Le
cost
new
result lengch
lue
Poqwer
aapprox
rx
term
part
baced
cqulv
iinl
S
data
str
ingiing
de
a newwere
ght
Find
wi
TOS
Lime
O
area
Tedel
uue
given
Fetuur uurn
event
f
fra
one
fina
sum
text
Vi
day
pra
4
6x10
Smoothed
10
dt
int caw.calcaattt
e
he
dis denteneedin
list
(c) Words with highest descrease.
(d) Word cloud of ground-truth answer (y").
Figure 4: Qualitative analysis of the vocabulary learned by NRT-WS (- log (b) The word cloud of the generated reasoning process is rich with procedural terms like 'let', 'step'. and 'solve'. (a) NRT specifically learns to use meta-cognitive words like 'premise' and 'explanation" in its reason- ing. (d) The ground-truth word cloud is more focused on the problem's nouns and final answer. (c) Conversely, NRT learns to suppress answer-specific formatting like 'boxed' from its reasoning traces. Together, these show NRT organically develops a distinct "language of reasoning".
frequency analysis in Figure 4c shows that tokens overwhelmingly associated with final answers, such as the formatting token "boxed" (used in math problems to demarcate the final numerical an- swer), effectively suppressed in the reasoning trace. This demonstrates that NRT, without any direct supervision on vocabulary, successfully teaches the model to differentiate between the "think- ing" part of the task and the "answering" part, correctly adhering the intended latent reasoning structure. This learned separation is a key factor in its ability to generate clean, step-by-step logic that leads to improved final answers.
1.2 CASE ANALYSIS OF GROUNDTRUTH TOKEN PROBABILITIES
Our hypothesis is that a successful training method should not only lead to the correct answer but also increase the model's confidence in that answer by generating a sound reasoning trace. The NRT objective, (- [log (o(y1| 2)]. is designed expressly for this purpose: it rewards the generation of reasoning processes (2) that make the ground-truth answer (y²) more probable.
To provide a conerete, quantitative example of this mechanism, we analyze a single prediction from task. Table[] the performance of the baseline SFT model
a fact-based question-answerte against our NRT-trained models and other baselines. While most models produce the correct year, the models trained to reason first generate an internal monologue that weighs evidence and historical facts.
The result is a striking increase in predictive confidence for the best model. The standard SFT model is uncertain, assigning only a 30.8% probability to the correct answer, "1852". In contrast, after generating its native reasoning trace by citing external knowledge and evidence, our NRT-WS (1/p) model's 's confidence in the same answer jumps to 50.4%, a relative increase of over 63%. This case study provides direct evidence that NRT can successfully train the model to generate reasoning
23

---

Published as a conference paper at ICLR 2026
processes that strengthen its conviction in the correct solution, validating the principle of our approach.
1.3 CASE STUDY OF GENERATED RESPONSES
To provide qualitative illustration of how different training methods affect the model's emergent reasoning, we present case study in Table I0 The example features an open-ended, ambiguous query that does not have a single verifiable "correct" answer, making an excellent bed for practical reasoning ability.
The SFT model responds directly without a reasoning process, acknowledging the question's ambi- guity and offering to help if more information is provided. While polite, it is not proactive.
The RLPR model, trained with basic intrinsic reward, develops a metacognitive reasoning process to "break down the key factors to consider." However, its reasoning remains high-level, and the resulting answer offers only generic institutional examples (e.g.. Harvard Business School) rather than specific, actionable course recommendations.
In contrast, our NRT models leverage the latent reasoning process to navigate the query's 's vague- ness more effectively. The NRT-GM model reasons about general evaluation criteria like curriculum and instructor quality, leading to a helpful final answer that outlines the structure of a hypothetical course. The NRT-WS models exhibit even more exploratory internal monologue, actively brain- storming potential interpretations and considering relevant entities (e.g., Udemy, Coursera, ICF). For NRT-WS (1/p), this richer reasoning allows it to synthesize far more useful final answer with specific, actionable suggestions. Interestingly, while NRT-WS (-log p) follows a similar exploratory reasoning path, its final answer confidently hallucinates a non-existent program called "the Mod- ule Coaching Course." This overall comparison demonstrates that NRT fosters more robust and practical form of reasoning, where the model learns think through ambiguity to generate helpful and comprehensive responses.
24

---

Published as a conference paper at ICLR 2026
Table 8: The complete prompt template used for the ILLM-as-a-judge semantic evaluation. The prompt fed the model single continuous context.
Evaluation Prompt
You verifying the "reasoning process* quality, not the final answer formatting.
Score quality of thought process for the given reasoning trace. Focus on:
Logical coherence and absence of contradictions.
2. Faithfulness to the provided problem statement.
3. Correct mathematical algorithmic steps (even if arithmetic slips occur later).
Absence hallucinated facts unjustified leaps.
Return JSON with fields:
"score": float in [O, 11. "justification": "one paragraph that cites concrete issues or strengths", "flags": ["optional short tags such as hhalucination', "missing-step" ....")
Problem:
{problem)
Model reasoning trace:
[reasoning]
#*# Ground-trun answer:
[ground truth)
Give higher scores when the reasoning is consistent and well gell-ggoonded, and lower scores when it self contradttaad,,,,, irrelevant, or clearly flawed.
25

---

Published as a conference paper at ICLR 2026
Table 9: Case study on impact of native reasoning on predictive confidence. For the same query, we compare the final answer probability from a standard SFT model and models trained with different reasoning-bes based objectives. Models that generate an internal reasoning process show significantly boosted confidence in the correct answer.
Model
Generated Reasoning Process (x)
P(y (2,
Query (z:)
Questions: ['when did luois braille die?"]
Solution:
Reference Answer (v")
1852
SFT
N/A
0.3083
RLPR
<I Althik.starrt > Sure, let's determine the solution for the input "Questions: ['when did luois braille die? 'I" and output "March 6, 1852" based on the reason "The output answer is the answer for the given questions." " thi nk end |>
0.0375
NRT-GM
< <|think.start To answer the task "New input: Questions: ['when did luois braille die?"] Solution:"
0.1354
appears fact-finding or historical question that can answered with a date. The solution provided is needed to match the format and substance of the task, which asks for a fact known to be true. The task is example of fact-based question-answer task in the form of natural language question and its expected answer. According to historical account, Louis Braille, a French educator and inventor of reading and writing system for those who are blind, died in 1852 at the age of 43. Given that the task is to answer the question "when did Louis Braille die?", the correct
is <I thi th_nnd
NRT-WS (1/p)
< <Ithink start >> This question is about person (Luois Braille) that was born in 1809 and died in 1852 (cf Wikipedia "Louis Braille", snippet: "Louis Braille January 1809 Jamuary 1852)...)). You don't I need to understand Natural Language Processing to do this, you just need to search for the answer online... Evidence: Louis Braille, the inventor of braille, died on January 6, 1852, at the age of 43... It clear that Louis Braille, the creator the braille system of writing, died January 6, 1852... < thtiikk end
0.5042
NRT-WSS (-
< Atthinksstarr >> answer "when louis braille die?" we need to consult knowledge about Louis Braille, who invented the braille script. Hc died in 1852... For Louis Braille, he was man who made lifesaving system in this world for those with vision problems. Louis Braille died on January 6, 1852. So, the answer 1852... Louis Braille was born on January 1809 Coupvray, fle-de-France. HE died on 6 January 1852 at the age of forty-two... Louis Braille, the inventor of the Braille system for the visually impaired, died on January 6, 1852... The solution as per your need: January 1852...< [think_end!>>
0.3150
26

---

Published as a conference paper at ICLR 2026
Table 10: A case study comparing the generated reasoning process (2) for different methods sample query from the training dataset.
Query (x): What is good 7 module coaching course?
Method
Generated Reasoning Process (2) and Generated Answer (v)
SFT
It's great that you're looking to learn more and improve your skills through a 7- module coaching course! However, without specific criteria or your area of interest, it's challenging to recommend a particular course. There are many excellent coach- ing courses availahle on various topics, including leadership, personal development, career coaching, and more. If you could provide more details about your specific area of interest the type of coaching you's re looking for. I'd be happy to suggest some options.
RLPR
Reasoning:
To find good 7 module coaching course, let's break down the key factors to consider when selecting coaching course with 7 modules.
Answer: Determining the "hest" 7 module coaching course can depend on variety of factors, such as the specific goals of the individual or organization... here are few examples of 7 module coaching courses that highly regarded in various fields.. well-regarded course this category could offered by the Harvard Business School Executive Education or the Leadership Institute... It's always good idea to research the course content, the credentials the instructors, and the reputation of the provider before making a decision.
NRT-GM
Reasoning: To answer the question "What is a good module coaching course?"... ... I can't provide a specific recommendation... without additional context about what lype of coaching you're interested in... good coaching course should cover a com- prehensive curriculum, be laught by experienced and qualified professionals, and offer practical exercises... You might want to consider the following criteria when evaluating coaching course: and Ratings...
Course Content... Instructor Qualifications.. Reviews
Answer: A good 7 module coaching course could cover a variety of topics depending on the specific area of coaching you't re interested in. Here's 's a general outline of what a module coaching course might look like... Overall, good 7 modulc coaching course would provide comprehensive introduction to the field of coaching, covering the core skills, theories, and ethical considerations necessary for effective coaching practice.
NRT-WS (1/p)
Reasoning: To answer... it's important to understand that the "best" coaching course will depend on your specitic needs, interests, and goals. There many online plat- forms offering coaching courses, such Udemy, Coursera... Some popular 7-module coaching courses include "Coaching Skills Foundations" by the International Couch Federation (ICF).. "The Life Coach School: The Ultimate Life Coach Training Pro- gram" by Brooke Castillo. These courses cover various aspects coaching... It's also important to consider the accreditation...
There are many coaching courses available, and the " "goodness" of course can vary... However, good 7 module coaching course should cover a range of topics including coaching techniques, ethical considerations, client communication, goal setting, and more... Some popular coaching courses include those offered by the International Coach Federation (ICF), the Institute for Professional Excellence in Coaching (iPEC), and the Co- Ac-Actvve Training Institute. Remember to choose course that aligns with your career goals and interests.
Answer:
NRT-WS logg))
Reasoning: To provide a recommendation for good 7 module coaching course, need more context... For a general coaching course, here are some popular options: 1. The Life Coach School... 2. The Tony Robbins Coaching Certification Program... 3. The Co-Active Coaching Certification Program... A good module coaching course can vary depending the specific area... platforms like Coursera, Udemy, and Linkedln Learning ofler wide range of coaching courses...
There are many excellent coaching courses available, but the "7 Module Coaching Course" is comprehensive and highly recommended program. It de- signed to provide individuals with the skills and kuowledge needed to become ef- fective coaches... The course is structured into seven modules, each focusing on a different aspect of coaching, including communication, goal setting, motivation, and more...
Answer:
27


Reviews:
[{"review_id": "4iKRTX5MV2", "reviewer": "Reviewer_q7do", "summary": "The paper introduce NRT - a new LLM framework for verifier-free reasoning .The new method is based on new reward functions which defined on model generated think traces. Next they optimize the model using off-policy RL.  The update is based on 2 parts: (1) Trace reward (2) Token reward for right answer prediction. They propose 3 new schemas for reward: geometric mean, arithmetic mean and weighted sum that that prioritize difficult tokens. \nThe authors empirically demonstrated their models using 2 small base models (Llama - 3B / 8B)", "questions": "How would you modify NRT reward aggregation function  to encourage short answers and penalize too long answers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is original. It proposed a more  general schema which unify 3 previous verifier free methods  (JLB, Verifree, and RLPR) . The paper is well written. The quality of ablation study is good.", "weaknesses": "No obvious weak points", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3gBX8Qpvsl", "reviewer": "Reviewer_QaV8", "summary": "This paper proposes a new framework called \"Native Reasoning Training (NRT)\" aimed at addressing the limitations of current training paradigms for large-scale reasoning models (SFT+RLVR). By treating the reasoning process as a latent variable and using the model’s own confidence in predicting standard question-answer pairs as an intrinsic reward, NRT eliminates the need for expert-annotated reasoning trajectories or external verifiers. However, the authors do not explain why this approach works. Since the quality of the reference answers cannot be guaranteed, it is unclear whether this method is justified. The experiments were conducted only on Llama-3.1-8B and Llama-3.2-3B models, which are too small to adequately validate the authors’ claims. Furthermore, in the experimental section, all comparative methods were evaluated under the authors’ own baseline implementations, raising doubts about the reliability of the results. The authors could have instead used existing baseline settings for a more convincing comparison.", "questions": "1. Why are the results generated with the \"think\" mode disabled used as the reference answers?\n\n2. Why wasn't the method's effectiveness validated on larger-scale models?\n\n3. Why weren't the existing baseline experimental setups and results adopted?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "- This paper is commendable for its exceptional clarity in presentation, which significantly facilitates the review process. The precise and well-structured descriptions allow readers to quickly grasp the core contributions of the work, as well as to readily identify its strengths and potential limitations. \n- The cover letter provides a remarkably clear and concise summary of the manuscript's key findings. This thoughtful presentation is highly efficient, as it enables reviewers to understand the essence of the paper at a glance, thereby saving considerable time and effort during the initial evaluation.", "weaknesses": "- The author did not explain why the method is effective. The author's \"native\" responses are derived from the \"think-off\" mode. Since the quality of the reference answers cannot be guaranteed, I cannot confirm whether this approach is reasonable.\n- The experiments in the paper were conducted solely on the Llama-3.1-8B and Llama-3.2-3B models. The models are too small to adequately validate the author's claims, and the persuasiveness of experimental results from smaller-scale models is relatively weak. In contrast, many existing methods validate their claims on larger models [1].\n- In the experimental section, all comparative methods were evaluated under the \"Our baseline implementations\" setting. The reliability of the experimental results is questionable, as the author could have chosen existing baseline settings for the experiments instead.\n\n[1] Yunhao Tang, Sid Wang, Lovish Madaan, and Remi Munos. Beyond verifiable rewards: Scaling reinforcement learning for language models to unverifiable data. arXiv preprint arXiv:2503.19618, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WDNSYlQCWW", "reviewer": "Reviewer_YbHE", "summary": "The paper proposes Native Reasoning Training (NRT), a verifier-free reinforcement learning framework for reasoning models. Instead of relying on Reinforcement Learning with Verifiable Rewards (RLVR)—which needs an external programmatic checker for correctness—NRT treats the reasoning trace as a latent variable and optimizes rewards that reflect how much a generated trace increases the model’s likelihood of producing the ground-truth answer. The authors present a unified view of intrinsic rewards and discuss several aggregation schemes (e.g., arithmetic mean, geometric mean, weighted-sum that prioritizes difficult tokens) to avoid failure modes such as policy collapse. They couple this with a GRPO-style group advantage estimator. Experiments use a standardized evaluation harness across nine benchmarks (e.g., MMLU, BBH, GSM8K, MATH, HumanEval, DROP, PopQA, TruthfulQA, IFEval). Reported results claim state-of-the-art among verifier-free methods and robustness to training instabilities.", "questions": "Correlation with reasoning correctness: Can you show that higher intrinsic rewards (under your aggregation) correlate with externally judged reasoning quality (e.g., logic checkers, stronger LLM judges) rather than only internal likelihood gains?\n\nWhen verifiers exist: On math/code tasks where RLVR is available, how does NRT compare under similar compute? Could NRT complement RLVR (e.g., pre-train with NRT, then fine-tune with RLVR)?\n\nChoice of aggregation: Why is the weighted-sum superior to geometric mean in practice? Please provide ablations isolating the effect of each aggregation on collapse mitigation and accuracy.\n\nGeneralization to open-ended outputs: Since rewards depend on ground-truth answers, how does NRT extend to truly open-ended tasks without reference answers?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Clear stance vs RLVR: positions NRT for unverifiable tasks, removing dependency on external checkers. \n\nUnified reward view with explicit analysis of aggregation pitfalls  and a weighted-sum that targets low-probability “hard” answer tokens. \n\nStability measures: GRPO-style relative advantages over an empty-trace baseline; practical rollout details are transparent. \n\nStandardized evaluation harness across a broad benchmark suite, improving comparability.", "weaknesses": "Self-referential reward: Even with improved aggregation, the intrinsic signal is driven by the model’s own likelihood over the ground-truth answer. This can inflate confidence without proving better reasoning. The paper lacks external evidence that higher intrinsic reward correlates with logically valid intermediate steps.\n\nScope vs RLVR: While NRT addresses unverifiable domains, the paper does not show how it competes where verifiers do exist (math/code). The claims are limited to “verifier-free SOTA,” which weakens broader impact. \n\nAggregation novelty is incremental: GM and difficulty-weighted sums are reasonable but not conceptually new. The unification is helpful but largely consolidates known heuristics. \n\nEvidence gaps: No analysis with external judges or rule checkers to validate that selected traces embody correct reasoning, rather than patterns that merely raise answer likelihood.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QGSSfVXNDl", "reviewer": "Reviewer_iE85", "summary": "This paper introduces a unified verifier-free framework (NRT) that treats the reasoning trace as a latent variable and maximizes an intrinsic reward computed only from question-answer (QA) pairs without expert CoT or external verifiers. The authors say this re-frames prior methods, exposes failure modes (e.g., policy collapse), and yields new reward aggregations, etc and present several experiments focusing on the llama model family.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper presents a unified formulation that subsumes several verifier-free ideas into one token-level objective and gradient, making design levers more explicit by casting them as choices of $f$, which is a helpful perspective.\n\n* NRT relies only on QA pairs which can potentially help lessen/reduce dependence on costly human chain-of-thought (CoT) annotations or judge models.\n\n* It appears that NRT might avoids the mode collapse observed in RLPR, maintaining diverse reasoning traces while still converging on higher answer accuracy.", "weaknesses": "* The experimentation is quite limited in restricting attention to just two models within the same model family; it's unclear how well these results can generalize to other more model families as a result. Results on bigger models, alternative corpora, and domains with very long outputs or strong factuality constraints would strengthen the claims. \n\n* The results in Table 3 do not seem to show consistent or state-of-the-art results by NRT as claimed in the paper; the results seem mixed as claims of significant gains are made only on the average score. The individual scores tell quite a different story. Moreover, it's unclear what variant of NRT to use consistently over others and why as the results, again, seem to be mixed.\n  * Just as one example, TruthfulQA hardly moves in the results; if NRT induced genuinely more truthful reasoning, I would expect clearer TruthfulQA gains, no?\n\n* The paper also notes prior baselines use different datasets and response lengths (Appendix D, p18), while NRT trains on long-answer QA (longer avg response). From this, it seems like the data regimes also differ across prior work; Appendix D (Table 5, p18) emphasizes that earlier baselines trained on short-answer data (Verifree) or specific long-form proofs (JLB), while NRT trains on a general instruction-following mix with long reference answers. \n  * Longer reference answers can provide denser supervision (e.g., more tokens with explicit targets) and thus richer learning signals even before any reward optimization. Instruction-tuned datasets like tulu-3 encode more linguistic variety than the narrower math-reasoning data used by RLPR. So when the paper attributes higher benchmark scores to NRT, if I understand correctly, how do we separate how much comes from the NRT objective vs. the richer data regime? In effect, two variables have changed at once: the learning rule and the data distribution. From my view, that weakens the causal chain needed to claim “NRT is a better training method” rather than “NRT + better data does better.”\n\n* One thing that caught my attention is that the paper re-implements RL baselines and, “to ensure a fair comparison,” injects the same reward stabilization and format-supervision used by NRT into those baselines, because the original methods were designed for instruction-tuned models (Appendix F, p20). Doesn't this change the baselines’ training conditions from what their original authors intended? I have not seen the paper quantify how this affects each baseline’s behavior (e.g., ablations of RLPR with/without format loss, length normalization, etc.). Without such controls, I fear that these modified baselines may risk confound how fair of a comparison this is.\n  * More specifically, my concern is that this may actually change what those baselines were originally optimizing for. The original RLPR paper, for instance (if I remember correctly), did not use format-supervision or clipping relative to a \"no-reasoning\" baseline; those are distinct design choices that can likely alter gradient magnitudes and trace-length dynamics. As such, when the paper shows NRT outperforming RLPR (Table 3, p7), it’s unclear whether NRT is truly better than the RLPR method or just better than their internally modified version of RLPR. For a fair method-level comparison, the baselines should be reproduced under their original settings and under the shared settings etc. with results reported for both. Otherwise, it's difficult in my view to isolate the causal factor of improvement.\n  * Similarly, it seems that NRT can benefit from some extra things like reward clipping, group-wise normalization, and a mild format loss to maintain thinking/reasoning delimiters (Appendix F); however, these may be problems that prior baselines did not address in the same way (i.e., the converse of the earlier point: if those aids disproportionately help NRT but aren’t tuned equally for baselines, the comparison again favors NRT not because its underlying intrinsic-reward formulation is superior, but because the training recipe is more stable).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)