Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
Authors: Xin Xu
Clive Bai
Kai Yang
Tianhao Chen
Yang Wang
Saiyong Yang
Can Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reasoning
Reinforcement Learning with Verifiable Rewards
Long Chain-of-Thought
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Thinking-Free Policy Initialization, a stage prior to RL that can accelerate RL convergence to a higher performance ceiling and naturally yield reasoning-efficient models

Abstract: Reinforcement Learning with Verifiable Reward (RLVR) effectively solves complex tasks but demands extremely long context lengths during training, leading to substantial computational costs. While multi-stage training can partially mitigate this, starting with overly short contexts often causes irreversible performance degradation, ultimately failing to reduce overall training compute significantly. In this paper, we introduce **T**hinking-**F**ree **P**olicy **I**nitialization (**TFPI**), a simple yet effective adaptation to RLVR that bridges long Chain-of-Thought (CoT) distillation and standard RLVR. TFPI employs a simple *ThinkFree* operation, explicitly discarding the thinking content via a direct *&lt;/think&gt;* append, to reduce token usage during inference. Training with *ThinkFree*-adapted inputs improves performance and lowers token consumption, even in the original slow-thinking mode. Extensive experiments across various benchmarks have shown that {\method} accelerates RL convergence, achieves a higher performance ceiling, and yields more token-efficient reasoning models without specialized rewards or complex training designs. With TFPI only, we train a 4B model to reach 89.0% accuracy on AIME24 and 65.5% on LiveCodeBench using less than 4K H20 hours.

Published as a conference paper at ICLR 2026
THINKING-FREE POLICY INITIALIZATION MAKES DISTILLED REASONING MODELS MORE EFFECTIVE AND EFFICIENT REASONERS
Xin Xu Clive Bai", Kai Yang". Tianhao Chen', Yang Wang". Saiyong Yang* Can Yang!.3.4.
'Department of Mathematics, HKUST 'HY, Tencent
Big Data Bio- thoelntelligence Lab, HKUST "State Key Laboratory of Nervous System Disorders, HKUST Department of Mathematics, HKU
Github Repo: [GitHub Page]
Models: [Huggingface Models]
ABSTRACT
Reinforcement Learning with Verifiable Reward (RLVR) effectively solves com- plex tasks but demands extremely long context lengths during training, leading to substantial computational costs. While multi-stage training can partially miti- gate this, starting with overly short contexts often causes irreversible performance degradation, ultimately failing to reduce overall training compute significantly. In this paper, we introduce Thinking-Frre Free Policy Initialization (TFPI), a simple yet effective adaptation RLVR that bridges long Chain-of-Thouggh (CoT) distilla- tion and standard RLVR. TFPI employs a simple ThinkingFFeee operation, explic- itly discarding the thinking content via a direct </thi//thhii nkk> append, to reduce to- inputs improves
ken usage during inference. Training with PhinkinggFreeaddappee performance and lowers token consumption, even in the original slow-thinking mode. Extensive experiments across various benchmarks have shown that TFPI accelerates RL convergence, achieves higher performance ceiling, and yields more token-efficient reasoning models without specialized rewards or complex training designs. With TFPI only, we train 4B model to reach 89.0% accuracy AIME24 and 65.5% on LiveCodeBench using less than 4K H20 hours.
INTRODUCTION
Reasoning is fundamental aspect of human cognition, and equipping artificial intelligence (AI)
with strong reasoning capabilities is critical for its deployment and applications (Morris et al., 2023;
Huang et al., 2024; Xu al., 2025c). Progress in pretraining (Shao al., 2024; Yang et al., 2024;
Chen et al., 2025b), supervised fine-tuning (SFT) (Yu et al., 2023; Xu et al., 2024; Tong et al., 2024;
Ye et al., 2025; Muennighoff et al.. 2025), rigorous evaluation (Rein al., 2024; Phan al., 2025; Xu et al., 2025b), reinforcement learning (RL) (Jaech et al., 2024; Guo et al., 2025) has significantly enhanced the reasoning abilities of large language models (LLMs)). Among these, RL with verifiable rewards (RLVR) stands out as an effective approach that enables large language models (LLMs) to generate long Chains-of-Thoughh (CoT) (Wei et al., 2022) spontaneously, and empowers them with unprecedented performance on challenging reasoning tasks. Thus, these RLVR-trained LLMs are termed as long-CoT LLMs, slow-thinking LLMs, or large reasoning models (LRMs).
Compared with initializing from base LLM, starting from SFT-distilled LRM typically yields better results and accelerates convergence in RLVR (Guo et al., 2025; An et al., 2025). However, SFT-distilled LRMs often produce excessively long responses during the rollout stage of RLVR, which necessitates a large training context length for RLVR. Using such large training contexts also incurs substantial computational costs. A common mitigation strategy multistage RLVR, which begins with a relatively "short" context and gradually increases the training length (Luo et al., 2025b; An et al., 2025). Nonetheless, Zeng et al. (2025a) argue that multistage training might cause
Corresponding Authors.

---

Published as a conference paper at ICLR 2026
Stage
32K RL
Stage
DeegSicallR
Avgi32 (86)
Accurtcy (8)
4K.-4K BK-16K
JNK-SZK
20
Mage
DS D5-L.g
25
T
10
Direct JL.
50D
1000
1560
2K
4X
BK
16K
10
12 34
8120
Hours
Tokens (K)
Figure I: Our proposed TFPI accelerates the convergence of RLVR to a higher performance ceiling (left) and yields more token-efficient reasoning models (right). Left: avg032 versus training com- pute, measured in H20 hours. "Direct RL" refers to directly training Qwen 1rre3-444 4B with a 32K context window using DAPO, while "TFPI RL" denotes running 32K-context DAPO after initialization with our 3-stage TFPI. The x-axis for TFPI uses a linear scale during the TFPI phase, followed by a logarithmic scale, with the transition indicated by black vertical line. Right: Average accuracy on 4 reasoning datasets (AIME24/25, Beyond AIME, and GPQA) versus average output tokens. Points the upper-left region indicate better performance. Baseline names and their corresponding numbers are listed in Table 3. Red dots denote different stages of our TFPI.
irreversible performance degradation. Moreover, even multistage training demands significant com- putational resources. For instance, training 4B model while progressively increasing the maximum context length from 40K to 48K to 52K tokens requires approximately 8K H800 GPU hours (An et al., 2025). These limitations underscore the need for more efficient RLVR training methods.
In this work, we introduce Thinking-Free Policy Initialization (TFPI) simple yet effective adap- tation to RLVR that bridges long Chain-of-Thought (CoT) distillation and standard RLVR. We first observe that a ThinkingFree substantially reduce token usage during inference (Section 3.1). Training inputs improves performance and lowers token consumption, even when
explicitty discarding the thinking content via a direct
</think> append with BrankknngFree-ddappee evaluated in the original slow-thinking mode (Section 3.2). We then formally define TFPI in Sec- tion 3.3. As illustrated in Figure 1, TFPI accelerates RL convergence, achieves a higher performance ceiling, and produces more token-efficient reasoning models without requiring specialized rewards or complex training designs (Section 4).
as follows: We find that ThinkingFree can substantially reduce inference costs for distilled LRMs, and that training with capability. @ We propose TFPI, fast, low-cost initialization phase for long-CoT RL that accelerates RI convergence and exhibits transfer across domains, even when trained solely on mathematics. @ We show that long-Coo CoT RL following TFPI achieves a higher performance ceiling while producing more token-efficient LRMs without the need for specialized rewards complex training designs, offering an effective and efficient route to training high-perffrming LRMs. We provide both behavioral and parameter-level analyses and reveal that TFPI not only preserves the slow-thinking reasoning pattern but also enables substantially faster rollouts in subsequent long-Col CoT RL stages.
Our contributions
inputs enhances dlw-thinkking
2 PRELIMINARY
Notation. In this paper, define LLM parameterized by 0 policy Tg. Let denote query and DD the set of queries. Given a response to query I, its likelihood under the policy me is expressed ( z) I1 Ty (yt x, (et)) where ly/ denotes the number of tokens in y. A query-response pair (z, 3) is scored by a rule-based outcome reward r(x, y) E {0, 1}, indicating whether the response y aligns with the ground truth of x..
RLVR Algorithms. RLVR algorithms are central to the recent success of LRMs, with prominent examples including Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Group Relative Policy Optimization (GRPO) (Shao et al., 2024). PPO constrains updates to remain close to the previous policy Tew via clipping. GRPO bypasses the need for the value model by computing relative adyantages among responses to the same query. A widely adopted GRPO variant, DAPO (Yu
2

---

Published as a conference paper at ICLR 2026
al., 2025), adds clip-higher dynamic sampling, and token-level policy seale--gradiett losses to stabilize training. The objective given by (D) EznD [JDAPO((, (0,z)),
lud
M min (r..(0)) Aies clip(rs,. Stow: Snigh) Ait))
20, =
18
s.t. 0 < [M h Es..quivaLent (wi +}}1 G.
(I)
Here, {u:) ~ (+2=) and G is the number of generated responses to each query (i.e., the group size) and the importance ratio (0) and advantage Ai, of token yi,, are:
(0) To (y ve, Tew (Ve,aa, )t,ct))
A = r(., mean ((r(z, x)) std (fr(2, (.))
respectively, where all the tokens in y share the same advantage A,
concise overview of PPO and GRPO is provided in Appendix A.1. Across all experiments, we adopt DAPO for fair comparison. In principle, any RLVR method can be applied our TFPI stage.
3
METHODOLOGY
3.1 THINKING-FREE MODE ENABLES MORE EFFICIENT REASONING
To generate response y ~ Ta(- x) for a query x using an SFT-distilled LRM Te. the query is typically formatted with chat template. Template illustrates the template adopted by the Qwen model family (Yang et al., 2025a). We define ThinkingFree as an operator that transforms an input query into modified query x' TreekingFree(()), in which the thinking content is explicitly omitted. Under this transformation, response generation follows y no(- x²). This mechanism provides explicit control over whether reasoning content is present absent in the generated output (see Template 2). Additional examples using other chat templates are provided in Appendix A.2. Hereinafter, use æ to denote a query formatted with the thinking template (e.g., Template 1), and x' or (hiikingFrre(()) to denote the corresponding thinking-free free version (e.g., Template 2).
<lim_sturt > system'ssmmmmm systemmyyPllessee yPleasse reason step by step, and put your final answer within [ }. <lim_end) >n im start "user' ( (x)}
Template (Thinking Mode)
< im end) > An jim start] > assistant)y
Template 2 ThinnkingFFee Mode) < <lim_start >yyttrressysmmtt paaestemmmmPllaaaa nPlease reason step by step, and put
your final answer within |boxed] J.aredd[{<<llnr <lim end >> \n jim start [>user' un (x)}
<)im <immend) \n< (i start assistant n nththink>>>>> 'n' ////<inh
During inference, converting into its thinking-frree version x' can substantially reduce to- ken consumption. To assess how this transformation affects the reasoning capability of SFT- distilled LRMs, we evaluate DeepSeek-Distilles www.-iiiitlllleeeeereeeeeeee Sorrdd-ween-1 11nn--.- 5B (abbreviated DS-1.5B) and Qwen3-eee -BB on the AIME25. Detailed experimental setup is delayed to Appendix B.I. As shown in Figure 2 (Left), applying the ThinkingFrre reduces the number of output tokens by more than 70% for both models. It is worth noting that Qwen 4B is fast-slow fusion model, whereas DS-1.55 53 is an SFT-distilled long-CoT model; nevertheless, both exhibit the same trend.
3.2 THINKING-FREE TRAINING IS BENEFICIAL TO SLOW-THINKIN
To train an SFT-distilled LRM, the training response length should not be too short (Setlur et al., 2025), as this is detrimental to testing performance (An al., 2025). evidenced by the dotted line in Figure 2 (Right), training Qwen- 1-een-3440 4B with a 4K response length using DAPO indeed leads to a substantial drop in performance on AIME25, Given that using the Thinking Free variant for inference significantly reduce token consumption, we pose an audacious question: the TrinkingFree operation to all input queries during the rollout stage of RLVR? Moreover, will this approach be beneficial to preserving the original slow-thinking capability of the trained LLM?
apply
Surprisingly, RL trained with ThinkingFree rollout can slightly improve accuracy and reduce token consumption when evaluated in thinking mode, with very short training context

---

Published as a conference paper at ICLR 2026
20.0
thinking
Chinking-free
thinking
174K
16.5K
70
15.0
10.0
173%
171%
avg 32
15
10
taken
5.0
20
60
80
100
DS-1.5B
Qwen334B
training step
Figure 2: Results of the meta-experiment on the ThinkingFree operation. Left: Average output
tokens in thinking mode and ThinkingFree mode on AIME25. Right: Evolution of avg032 and average output tokens on AIME24 with thinking-mode evaluation over training steps under 4K train- ing response length (both training in thinking mode and ThinkingFreee mode).
lengths. Figure 2 (Right) illustrates the training dynamics of Owen-3 3-BB trained with 4K response length under the ThinkingFree transformation of queries (detailed settings are in Ap- pendix B.2). Even with 4K output length, ThinkingFree RL increases the accuracy on AIME25 in thinking mode by approximately 2%, while reducing output tokens by around 20%. In contrast, standard RLVR with a 4K length reduces avgê 32 by more than 40%. These results suggest that ap- plying ThinkingFree Free during rollout can yield steady improvements with minimal training compute.
3.3 THINKING FREE POLICY INITIALIZATION
Hu et al. (2025) introduce a pre-pretraining stage using formal language accelerate convergence of pretraining, while Wang et al. (2025b) propose mid-training stage between pretraining and RL- zero to facilitate RLVR. Inspired by these works, we ask: can a dedicated stage for SFT-distilled LRMs improve the efficiency and effectiveness of standard RLVR scaling? ThinkingFred RL, which enhances slow-thinking within short training context windows, requires substantially less compute than standard RLVR. Initializing RLVR with a l'hinkingFree RL, policy could therefore reduce roll- tokens while achieving stronger downstream performance. We term this step as Thinking Free Policy Initialization (TFPI), a stage preceding standard RLVR for SFT-distilled LRMs that aims to lower rollout costs, raise the ceiling of reasoning ability, and accelerate convergence.
Consider the RLVR objective (O) = Ex [RRIVR (0, æ)], where JRiVR (0, x)) denotes the per-
example objective of any RLVR algorithm (e.g., DAPO in eq. (1) or GRPO in eq. (5)). For the TFPI stage, we use a modified objective: (0) ENoD [JRLVR (6, x²)], where x'' = //inkinnFFeee(t) In the rollout stage of TFPI, G responses are generated conditioned on x': { To (. x'). The importance ratios and advantages (eq. (2)) are then adapted as:
TE. (yo. x' v1,ce)
A,,== r(x' v.) mean (tr(x', (fr(2'00))2))) std((r(e', M3 );)))
(3)
x', (t)
where Template 1 and Template 2 showcase the correspondence between and x' and r(x', v) r(z, 3) since the ThinkingFree operator does not alter the ground-truth answer of the original prob- lem. In our experiments, we instantiate RLVR with DAPO, i.e., JRLVR (0) JDAPO (0) (see eq. (1)). Note that Heuristics for RLVR (e.g., multi-stage RL, data curricula) can be directly applied to TFPI.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
In this section, we provide a brief overview of the key experimental setup, including training proce- dures, baselines, and evaluation details. Additional information can be found in Appendix C.
Training Details. We build on the VeRL codebase (Sheng et al., 2024) with RLVR, following the DAPO recipe (Yu et al., 2025), which enables dynamic sampling and clipping higher. All methods use the same hyperparameters (batch size 256, learning rate 10-6, no warm-up) and rollout set- tings (temperature 1, topp 1, topk 11, 8 rollouts/problem). Training is conducted DS-1 1
4

---

Published as a conference paper at ICLR 2026
Table 1: Results of TFPI vs. direct RL across different benchmarks, "Avgēk" denotes the average accuracy (%) over k random generations (i.e., pass@1). All models are evaluated in thinking mode. The total training compute for the 3 stages of TFPI equals that of "Direct RL" for fair comparison. Darker colors in the cell background denote better results within each model group.
Mathematics
Multi-Task
Code LiveCode Avgê8
Instruction
Overall
Models
AIME Avg332
AIME 25
Beyond AIME
GPQA Avg88
IFEval Avg84
Overall
Avga32
Avgê8
Avg.
SeepSeek-Dist.ie 1aee-111---eee-111-- 1.e
Initial Model
29.6
23.0
16.3
22.0
Direct RI
33,9
27.1
11.9
19.2
18.2
41.8
25.3
TTPI stage
32.1
26.9
13.9
29.3
18.4
39.5
26.7
TFPI stage?
36.8
28.4
14.5
27.8
20.5
42.3
28.4
-TFPI stage
40.1
30.8
13.8
29.6
19.9
40.8
29.2
Qwan3-4B
Initial Model
73.6
68.3
43.4
56.8
54.9
64.9
60.3
Direct RT
75.7
67.0
43.6
56.3
52.5
66.0
60.2
TFPI stage
75.2
67.8
42.4
57.9
55.3
66.0
60,8
TFPI stage 2
76.0
68.2
44.7
57.8
54.8
64.8
61.0
TEPI stage 3
79.9
70.6
46.7
58.5
57.0
70.2
63.8
DeepSeek-Distt Ditt--iiisiil1-ee----1
Initial Model
56.3
40.0
25.0
36.9
39.5
55.3
42.2
Direct RL
$7.9
40.4
26.6
38.0
38.3
56.7
43.0
TFPI stage
59.4
40.4
29.2
49.0
39.6
56.2
45.6
TFPI stage 61.8
43.9
31.5
48.0
42.0
57.1
47.4
TTPI stage
62.0
44.6
31.1
46.8
47.8
3wen3-4B, and DS-7B using Polaris-53K (An et al., 2025). Direct RLVR uses a maximum out- put length of 16K for DS-1. 5B/DS-7B and 32K for Owen3-4B, while TFPI adopts multi-stage training: 2K+ 4K- 8K for DS 1. 5B/DS 75 and 4K- 16K for Qwen3-4B
Baselines. We compare TFPI with direct RIVR training from SFT-distilled LRM ("Direct RL"" under matched total training compute (Table 1). To assess TFPI as a pre-RLVR stage, we run "TFPI RL. with similar compute and compare against "Direct RL"" (Table 2), also including competitive LRMs of the same size, such as Polaris (An et al., 2025), DeepScaleR (Luo et al., 2025b), Skywork- ORI (He et al., 2025), and so on. For efficiency analysis, we compare with several RL-based efficient reasoning baselines (Table 3) under the same evaluation for fair comparison, including TLMRE (Arora & Zanette, 2025), AdaptThink (Zhang et al., 2025b), AutoThink (Tu al., 2025), Laser (Liu et al., 2025a), LIMax (Aggarwal & Welleck, 2025), and ThinkLess (Fang et al., 2025).
Evaluation Details. Our evaluation benchmarks cover: Math Reasoning: AIME24/25 and Be- yondAIME (ByteDance-SSed) 2025). @ Multi-Task Reasoning: GPQA-Diamond (Rein et al., 2024). Code Generation: LiveCodeBench (Jain et al., 2024). + Instruction Following: IFE- val (Zhou et al., 2023). Following Guo et al. (2025), generate multiple outputs (ranging from to 32 depending on the size of the test set) per problem and report pass@ accuracy. Note that -ut-of-domain evaluation. For IFEval, report the
0 is in-domain evaluation, and ? strict prompt accuracy. All evaluation scripts are adapted from the DeepscaleR codebase (Luo et al., 2025b). Following the officially recommended decoding parameters' we the temperature 0.6 and top-p to 0.95 for the thinking mode, and 0.7, 0.8 for the thinking-fres mode, respectively. The maximum output length 32k for all configurations, except for the Qwen3-44 4B thinking mode, where 48k is used to reduce the clipping ratio. Further details are provided in Appendix C.3, and the effects of changing decoding parameters are presented in Appendix F.4.
4.2 TFPI ENHANCES THE NHOW-THNKINGG OF DISTILLED REASONING MODELS
To evaluate the impact of TFPI the slow-thinking capabilities, we present the results of TFPI versus "Direct RL" under the same training compute in Table 1. From the table, we have:
TFPI substantially enhances the slow-thinking capabilities of distilled LRMs even when trained with a small response length. For example, on DS-1. 5B, TEPI Stage 1 raises the overall
See Qwen3-4B Hugging Face page and DS-1.5B Hugging Face page.

---

Published as a conference paper at ICLR 2026
Table 2: Results (%) of RL after TFPI ("TFPI+RL") vs. "Direct RL" across different benchmarks. "Avg@k" denotes the average accuracy (%) over k random generations (i.e., pass@1). For LRMs marked with "*" results are taken from the corresponding reports (see Appendix C.4); results of 4B models are from our own runs with 48K response length. All models are evaluated in thinking mode. The total training compute for "TFPI+RL" matched to that of "Direct RL" for fair comparison.
Darker colors in the cell background denote better results.
Mathematics
Multi-Task
Code
Instruction
Overall
Models
AIME 24
AIME 25
Beyond AIME
GPQA Avg08
LiveCode
IFEval
Overall
AvgP32
Avg032
Avges
Avg8s
AvgP4
Avg.
DeepSeck R1*
79.8
65.0
42.4
71.5
64.3
86.1
68.2
Seed-1 S-Thinkingg
86.7
74.0
48.0
77.3
64.9
87.4
73.0
Claude4 Opus Thinking"
75.5
79.6
48.9
89.7
S1l--333B-Thinking
85.7
81.5
71.1
35.7
83.4
Qwen3-4B Direct RL
78.8
71.5
46.4
56.2
54.3
65.1
62.0
Qwen3-4B TFP1 stage
79.9
70.6
46.7
58.5
57.0
70.2
63.8
Qwen3-4B TFPI + RL
80.8
76.0
49.5
61.1
55.7
71.3
65.7
Swve2--B-25077-Tineee
84.7
79.2
51.5
66.4
62.4
68.0
68.7
-TPPI only
89.0
81.2
51.3
70.1
65.5
66.7
70.6
average accuracy from 22.0% to 26.7% (+4.7%) despite being restricted to 2K training response length. Results of DS-1. 5B continue improve through stages
on AIME25, accuracy increases from 23.0% (initial model) to 26.9% after Stage 1, 28.4% after Stage 2, and 30.8% after Stage 3, representing a total gain of +7.8%. Similar improvements are observed for Qwen3-4B and DS-7B. These findings indicate that TFPI enables effective training under low-cost settings (short context windows), and that combining this with multi-stage RL yields substantial accuracy gains.
@ Compared with "Direct RL", TFPI delivers faster and larger accuracy improvements under the same training cost. TFPI outperforms "Direct RL" in nearly all configurations. For example, Owen3--pp 3--4-4B 4B with TFPI attains 63.8% overall accuracy versus 60.2% for "Direct RL" (+3.6%), while DS-7B improves by +4.8% (47.8% vs. 43.0%). Given that the equal training compute, these results imply that TFPI achieves convergence more efficiently than conventional long-Co'T RL training.
Improvements of TFPI indicate some degree of transfer beyond mathematics. Although TFPI trained solely Polaris-53K (math-specific data), it demonstrates great out-of-domain improve- ments as well. For example, on DeepSeek -ireek---iise -Disti -Qwen-1. 1.5B 5B, GPQA accuracy increases 1.
from 16.3% 29.6%, LiveCodeBench from 17.7% 19.9%. and IFEval from 36.6% 40.8% after Stage 3. Notably, improvements mathematical benchmarks are often consistent across successive training stages, whereas other domains sometimes exhibit fluctuations (c.g.. GPQA for DS-7B and LiveCodeBench for (S-1.5B) This suggests that incorporating more diverse training data spanning multiple domains could be helpful for TFPI.
4.3 TFPI AS FOUNDATION FOR RLVR ACHIEVE HIGHER PERFORMANCE
Table shows the representative results between "TFPI + RL" with "Direct RI." under the same training cost (full results in Table 9 in Appendix F.2). The results lead to the following observations:
TFPI can raise the upper bound of RL-trained performance. RL training on a TFPI-trained model can still yield notable improvements, particularly in mathematics. For example, adding an RL stage after TFPI for Qwen3-4E boosts AIME25 accuracy from 70.6% to 76.0% (+5.4%)) Similarly, Beyond AIME scores increase by more than 2%. Across different model scales, applying RL after TFPI consistently achieves higher accuracies than "Direct RL" under the same compute budget. For instance, Qwen3-47 the overall accuracy rises from 62.0% (Direct RL) to 65.7% (TFPI+RL)) These results suggest that incorporating TFPI as an intermediate stage between long CoT distillation and standard RLVR can be beneficial for elevating final performance.
TFPI + RL is an effective and efficient strategy for training high-performing LRMs. From Figure 1 (Left), the total compute cost of all three TFPI stages amounts to less than 20% of stan- dard RL training with 32k token sequences. From Table 2, for DS-7B, TFPI+RL achieves ap- proximately the same overall performance as response length 16K, whereas Polaxis-7B-Preee 7B-Preview follows a 16k + 24k + 32k pro- gression during RL. Similarly,
eww despite using maximum
employs 40K
48K + 52k length

---

Published as a conference paper at ICLR 2026
Table Comparison of the Thinking-Free inference mode of TFPI with efficient reasoning baselines across various reasoning tasks. "Avgêk" denotes the average accuracy (in %) over k generations (i.e., passe 1), and "Toks" indicates the average output length thousands of tokens (K). Models with are trained from DeepScal SeeepscaleRR11 5B, while the remaining are from DS- 1661155, 5B. Darker cell background colors indicate better results.
AIME 24
AIME 25
Beyond AIME
GPQA
Overall
Models
2vgē32
Toka
2vg
Toka
avgê8
Taks
avge8
Toks
Avg.
Toks
1YTLRE--S--E
0.1)
27.6
12.9
24.8
12.5
9.2
11.9
14.8
7.5
19.1
11.2
2) Adap<Thinkk11.55 1.5 (4 0.05)
28.1
8.0
22.6
7.9
10.0
4.8
14.8
5.1
18.9
6.5
3) AutoThink-DS-1 SB-Sge
30.3
10.2
25.2
9.1
9.4
17.4
7.0
20.6
4) Laser-DE-1 1096-1144999155
30.3
8.3
24.9
7.4
9.7
7.4
21.1
4.7
21.5
6.9
5) Auto Shink-SSSSS
38.9
8.7
28.9
7.7
11.6
27.3
5.4
26.7
6 LB-Ma'
27.2
3.2
26.3
2.9
9.1
3.1
32.4
2.3
23.8
2.9
7) Thinkless-1-5--1
28.4
11.3
24.1
11.1
8.1
11.7
20.3
12.7
20.2
11.7
DS-1.5B (Thinking)
29.6
16.7
23.0
16.5
8.7
14.4
16.3
9.8
19.4
14.3
DS-1.5B (Thinking-Free)
12.4
5.7
10.9
4.4
4.4
0.9
8.0
3.6
TFPT stage
21.9
1.6
15.3
1.4
8.7
1.3
32.9
0.8
19.7
1.3
TFPI stago
31.5
3.4
24.2
3.1
10.1
2.9
35.3
1.6
25.3
2.7
TFPI stage
37,5
5.3
28.4
5.0
12.4
4,9
35.6
2.6
28.5
4.4
schedule and consumes approximately 8K H800 GPU hours, while our TFPI+RL requires only about 1.5K H800 GPU hours (see Appendix C.I) and achieves superior performance under 48K testing length. Even without a subsequent RL stage, TFPI alone allows DS-1 15-1 outperform DeepScaleR, which uses a maximum training length of 24K. Using only TFPI (4K-8K-16K), Owen3-4B-2500 achieves 89% accuracy AIME24. Remarkably, this 4B model outperforms Swen3-235BBThhnkee
in math reasoning and code generation. These findings suggest that TFPI serve both strong standalone training approach and as an efficient foundation for subsequent RI., producing competitive LRMs with significantly reduced compute requirements.
4.4 TFPI IMPROVES THE TOKEN EFFICIENCY OF DISTILLED REASONING MODELS
We compare the thinking-free inference with other RL-based efficient reasoning baselines in 1.5B size in Table 3 (see also Table 10 in Appendix F.2). We draw the following conclusions:
Both accuracy and token efficiency steadily improve after stage 3 of TFPI. For DS-1. 5B, thinking-free accuracy on AIME24 increases from 29.6% (initial model) to 37.5% (TFPI stage 3), while the average output length remains substantially shorter than that of the original thinking mode (5.3K vs. 16.7K tokens). A similar trend is observed for Owen3-4B, where accuracy improves from 26.9% to 75.1% across stages, with output lengths still far below those of the original thinking model. These results demonstrate that TFPI naturally produces more token-efficient LRMs, offering an alternative pathway to train models that deliver both high accuracy and reduced output length.
Compared with other RL-hased token control methods, TFPI achieves the best perfor- mance-efficiency trade-off without specialized reward or training designs. In DS-1. 53, both TFPI stage and stage 3 outperform almost all baselines in terms of overall accuracy while main- taining competitive or lower token usage. We visualize the overall accuracy-token usage trade-off' in Figure (Right), where TFPI consistently lies on the Pareto frontier across different stages. This observation motivates a rethinking of existing
reasoning RL designs" that rely heav- ily on specialized length reward shaping: training with TFPI offers an alternative paradigm which a strong slow-thinking LRM can be obtained, and more efficient variant can be realized simply by switching to the thinking- free mode without any additional length-control mechanisms.
ANALYSIS
5.1
WHY TEPI LEADS TO BETTER THINKING-MODE INFERENCE?
In this section, we delve deeper into why TFPI, which leverages void thinking content in Template 2, can generalize to enhance reasoning in Template 1. We analyze DS-1. 5B from two perspectives:
I Behavioural level: The learned verification behaviours after </think> during TFPI gen- eralize to the slow- thinking verification occurring within the <think> and </think>. The blue lines in Figure show the ratio of verification steps (i.e., the number of verification steps divided by
7

---

Published as a conference paper at ICLR 2026
veriAcation (%)
12
verifiration bokonn
2K 4K
4K .BK
20-
2K 2K-4k
4K-UK
16000
10-
4000
Verifiaation ratio tokonss
18
14000
46
3000
verification
14
toloens
10000
8000
1000
0000
500
1000
1500
500
1000
1500
training steps
training steps
Figure 3: Behaviour-Levee Analysis of DS-1. 5B over the TFPI Training Course. The ratio of verification steps and the average output tokens over training steps on the training set in thinking- free mode (Left) and on AIME25 in thinking mode (Right) in 3 stages of TFPI.
Layer Index
Direst REXX
T2PI 2X
TF214K
TFR1 BR
0.00
0.01
cosite similority
0.00
0.06
0.005
PC3
922
0.04
0.005
-0.003
0.02
00005
-0.03
-0.01..01 ww..g 8.00 8.15
-0.01
PC2
0.02
0.03
PCI
250
500
training
750
1000
1250
1500
1750
Figure 4: Paraveter-Level Analysis. Left: PCA projection of model parameters from DS-1.5B
to final checkpoints. TFPI (blue) starts at A. passes through intermediate points (B1, B2, B3), and ultimately converges the Direct RL final checkpoint (C). Right: Cosine similarity between parameter updates of TFPI-trained checkpoints and (C-A) across layers during training.
the total number of steps; see Appendix D for details) for the training set (thinking ffree mode) and AIME 25 dataset (thinking mode). We observe that the verification ratio exhibits similar trend:
a rapid drop in Stage 1, followed by steady growth in Stage 2, and a sharp increase in Stage 3. Notably, the sharp decline in Stage resembles information compression process. In Stages and 3, the model begins to explore more extensively (Figure 3 Right), which may explain why TFPI achieves superior performance. As verification believed to be vital for slow-thinking rea- soning (Setlur et al., 2025), the observed generalization of verification behavior suggests a transfer from thinking-free training to inference in thinking mode.
Parameter level: TFPI explores the parameter space more extensively at a faster pace, with its parameter update directions progressively aligning with those of "Direct RL." As shown Figure (Left), the PCA visualization of the initial model, TFPI-trained checkpoints, and Direct RL check- points exhibits distinct trajectories in parameter space. The TFPI begins from the initial model (A), moves through intermediate points (B1), (B2), and (B3), and ultimately converges to region near the Direct RL final checkpoint (C). This indicates that TFPI traverses a larger and more diverse region of parameter space before reaching a point close the RL-trained model. Such broad explo- ration may help explain why TFPI can lead to better LRMs. Furthermore, Figure (Right) shows that the cosine similarity of parameter updates between the TFPI-trained checkpoints and the Direct RL final checkpoint steadily increases across nearly all layers throughout training. This suggests that during TFPI training, the parameter updates share similarities with those in standard long-CoT training. To make our analysis more robust, additional results are given in Appendix F.I.
5.2 ABOUT REASONING PATTERN AND ROLLOUT SPEED
TFPI Preserves the Reasoning Pattern of Thinking Mode. In thinking mode (Template 1), response y comprises a long thinking section and a concise answer nm In ThinkingFree mode (Template 2), the thinking section is omitted, and nn contains shorter reasoning path. While standard RL tends to lengthen the thinking part, TFPI increases the length of y due to the absence of explicit thinking. As shown in Figure 5 (Left), for DS-1 5B trained with TFPI and evaluated in
8

---

Published as a conference paper at ICLR 2026
580
with TPPI
9000
without TEPI
560
tokens answer tokens
2X 44K
4K BBK
8000
340
answer
520
7000
6000
1500
150
299
448
597
746
896
training steps
training steps
Figure 5: Left: For TFPI with DS-1. on AIME25 in thinking mode, showing the average num- ber of answer tokens (excluding the thinking part) and the ratio of answer length total response length over training steps. Right: For long CoT RL with DS-1.58 showing the average number of tokens during rollout on the training set over training steps, with and without the TFPI stage.
Table 4: Additional results of TFPL. All models are evaluated in thinking mode.
Models
AIME
AIME 25
Beyond AIME
GPQA
LiveCode
IFEval
Overall
Stage
Schedule Sensitivity,
Qwen3-4B
4K - 8K + 16K
79.9
70.6
46.7
58.5
57.0
70.2
63.8
8K + 16K
78.0
70.2
46.7
59.5
55.8
64.5
62.4
16K only
77.1
71.5
45,8
56.4
54.4
66.0
61.9
Multi i -Stage Direct RL, Qwen3-44
TFPT Stage 3 Multi-Stage Direct RL
79.9
70.6
46.7
58.5
57.0
70.2
63.8
64.9
48.6
32.5
55.2
51.0
65.2
52.9
Qwen3-14B
Initial Model
81.7
73.2
50.2
65.0
63.5
67.3
66.8
Direct RL
81.7
74.5
50.8
64.8
61.8
68.2
66.9
TTPI stage 3
83,2
76.0
50.5
65.6
63.8
67.8
67.8
Owen3-148 Tokens (K)
Initial Model
16.8
20.8
22.8
9.4
16.0
17.1
Direct RL
17.9
22.1
24.5
10.4
17.4
18.5
TFPI stage 3
13.8
17.3
18.3
7.3
13.8
14.1
thinking mode, my remains stable at 500-580 tokens, whereas 1/yy] rises as the total length lyl decreases. This indicates that TFPI preserves the core reasoning pattern of slow-thinking rather than drifting toward an excessively extended "slow-slow thinking" behavior.
TFPI Speeds Up Rollout for Long-CoT RL Training. Another advantage of TFPI is its ability to speed up the rollout stage in standard long-Coff CoT RL training. As shown in Figure (Right), when directly performing RL from DS-1. 5B, the average output tokens during rollout on the training set start at over 9K, decrease to around 7.5K within the first 300 steps, and then fluctuate around 7.5K in the later stages. In contrast, when RL is performed after the TFPI phase, the average output tokens start at only 6K and the maximum length is below 7K tokens.
5.3 SCALING UP MODEL SIZE
To assess the effect of TFPI on larger reasoning models, we conducted experiments with Qwen3-14P under the same settings as Qwen.34B. As shown in Table 4, TFPI still outperforms Direct RL under the same training compute. Notably, Direct RL yields results only comparable to those of the initial model (66.9% vs. 66.8% on average). We hypothesize that Polaris-53K is, in some sense, too easy for Qwen3- 14B (with over 78% rollout accuracy), leaving only approximately 30% effective data after dynamic sampling (excluding 0/8 and 8/8 prompts). Nonetheless, TFPI re- mains effective in this setting. Additionally. TFPI reduces the average number of output tokens on the test set by 23.8% even in thinking mode (14.1K 18.5K). These results highlight the effective- ness of TFPI for larger model sizes, and we believe that with more challenging, higher-quality data, TFPI can further realize its scalability potential.

---

Published as a conference paper at ICLR 2026
5.4 ABLATION STUDY
Stage Schedule Sensitivity. To assess sensitivity to the stage schedule, we train Qwen3 -B under alternative schedules (8K~>168 ->16k and -onyy with approximately matched compute budgets. As shown in Table 4, all schedules outperform the Direct RL baseline and achieve comparable perfor- mance on math benchmarks, indicating that TFPI is robust to the choice of schedule. Nevertheless, 4K - 8K - 16K yields the best overall results (63.8% vs. 62.4% and 61.9%) We hypothesize that this stems from an implicit data curriculum induced by dynamic sampling: beginning at 4K improves training efficiency and emphasizes easier problems early on.
As discussed in Section 3.2 and Figure 2 (right), applying Direct RL with a 4K response length causes a substantial performance drop AIME25 for Qwen3-4B. In contrast, TFPI trains stably even at 4K. This indicates that, even in a multi-stage setting, the initial stage length should not be too short. To rule out the potential artifact introduced by multi-stage training, we run Direct RL with the identical multi-stage schedule on Qwen B. As shown in Table 4, Direct RL still incurs a substantial performance decrease under the 4K - 8K 16K schedule. Additional details are provided in Appendix F.7.
Multi-Stage Direct RL.
RELATED WORK
LRMs & Efficiency RLVR. RLVR has enabled the development of numerous high-performing large reasoning models (LRMs) (Yang et al., 2025a; Zeng et al., 2025a; An et al., 2025), inspiring research on model behaviors (Liu et al., 2025b; Wang et al., 2025a), novel algorithms (Liu et al., 2025b; Zheng et al., 2025), multimodal extensions (Meng et al., 2025; Xiao et al., 2025; Wang et al., 2025c), tool integration (Jin et al., 2025; Feng et al., 2025; Li et al., 2025a; Xue et al., 2025; Team et al., 2025a), and other related directions (Xu et al., 2025a; Zhang et al., 2025a; Chen et al., 2025c; Zhang et al., 2025c). RLVR can be applied directly to base L.MMs ("RL zero") (Zeng al., 2025b; Guo et al., 2025) or initialized from SFT-distilled long-CoT models, the latter typically yielding stronger results (Luo et al., 2025b; An et al., 2025). major challenge for RLVR is the cost of training with long contexts, as longer outputs are often necessary for harder tasks (Shrivastava et al., 2025: Zeng et al., 2025a), consuming high computational resources. Multi-stage RLVR mitigates this by starting with shorter contexts and gradually extending them (Luo et al., 2025b; An et al., 2025: He al., 2025), while algorithmic approaches modify GRPO reduce length bias (Yu al., 2025; Liu et al., 2025b; Wang et al., 2025a). Orthogonal to these strategies, we introduce TFPI as lightweight stage before RLVR, improving efficiency and strengthening slow-thinking mode at inference with minimal training cost, thus facilitating more effective subsequent RLVR.
Efficient Reasoning. To address the issue of overthinking (Chen et al., 2024; Sui et al., 2025), con- siderable efforts have been made, including prompt-based methods (Muennighoff et al., 2025; Yang et al., 2025c; Fu et al., 2025a; Chen et al., 2025a; Fu et al., 2025b), SFT-based approaches (Kang et al., 2025; Ma et al., 2025; Munkhbat al., 2025: Luo al., 2025a), and RL-related designs. RL- related approaches can be further categorized into length-based reward shaping (Team et al., 2025b; Aggarwal & Welleck, 2025; Arora & Zanette, 2025; Liu al., 2025a), integration of fast and slow thinking (Fang et al., 2025; Zhang et al., 2025b; Lou et al., 2025; Tu et al., 2025; Jiang et al., 2025; Zhang et al., 2025d), and thinking budget control (Li al., 2025b; Hammoud et al., 2025; Wen et al., 2025). These methods primarily trade accuracy for efficiency and rely on specialized reward functions or training strategies to encourage more efficient reasoning. In contrast, our proposed TEPI naturally yields even more efficient LRMs without specialized rewards or training designs.
7 CONCLUSION
After recognizing the benefits of ThinkingFreee for both inference and the training of distilled reason- ing models, we introduce TFPI, a cost-efficient intermediate stage between long-CoT distillation and standard RL training. As a strong initialization point, TIPI accelerates RL convergence, enhances attainable performance, and promotes more token-efficient reasoning without complex reward shap- ing or elaborate training pipelines. We further explain the factors behind TFPI's success from both the behavioral and parameter levels. Overall, TFPI provides a complementary path for building "tokee-efffcieet" LRMs, offering an effective and efficient alternative current RL paradigms.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was partially supported by an Area of Excellence project (A0E/E-6001/24-N) a Theme-based Research Project (T32-615/24-R), and the Innovation and Technology Commission (ITCPD/179) from the Research Grants Council of the Hong Kong Special Administrative Region, China. We would like thank all reviewers for their helpful suggestions in improving this paper.
REFERENCES
Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint 1xXiv25503.0469 2025.
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: post-training recipe for scal- ing reinforcement learning on advanced reasoning models, 2025. URL https: //hk/n///u....... github. wttpllg//bllooooo 2025/Polars
Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint arXiv:2502.04463, 2025.
ByteDance-Seed. Beyondaime:
Advancing math reasoning evaluation beyond high school olympiads. https: / http:///gg...a.eeaaaaaaaeeeeeeee /tt///aasattt////eaaaaaeee Seed/ BeyondAIME, 2025. Hugging Face repository.
Runjin Chen, Zhenyu Zhang, Junyuan Hong, Souvik Kundu, and Zhangyang Wang. Seal: Steerable reasoning calibration of large language models for free. arXiv preprint arXiv:2504.07986 2025a.
Tianhao Chen, Xin Xu, Zijing Liu, Pengxiang Li, Xinyuan Song, Ajay Kumar Jaiswal, Fan Zhang, Jishan Hu, Yang Wang, Hao Chen, et al. Gpas: Accelerating convergence of Ilm pretraining via cradient-preservve activation scaling. arxiv preprint StKivv::500.220049 2025b.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang. et al. Do not think that much for 2+ 3=? on the overthinking of ol-like lims. arXiv preprint arXiv:2412.21187 2024.
Zigeng Chen, Xinyin Ma, Gongfan Fang, Ruonan Yu, and Xinchao Wang. Verithinker: Learning to verify makes reasoning model efficient. arXiv preprint 2tXiv22505.17941 2025c.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: Llm learns when to think. arXiv preprint arXiv:2505.13379 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengguan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.1 1536, 2025.
Yichao Fu, Junda Chen, Yonghao Zhuang, Zheyu Fu, Ion Stoica, and Hao Zhang. Reasoning without self-doubt: More efficient chain-of-thouge through certainty probing. In ICLR 2025 Workshop on Foundation Models in the Wild, 2025a.
Yichao Fu, Xuewei Wang. Yuandong Tian, and Jiawei Zhao. Deep think with confidence. arXiv preprint arXiv:2508. 15260, 2025b.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501 11e...12444 12948, 2025.
Hasan Abed AI Kader Hammoud, Kumail Alhamoud, Abed Hammoud, Elie Bou-Zeid, Marzyeh Ghassemi, and Bernard Ghanem. Train long. think short: Curriculum learning for efficient rea- soning. arXiv preprint S..i::2508.094444 2025.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.2233 223112, 2025.
11

---

Published as a conference paper at ICLR 2026
Michael Y Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. Between circuits and chomsky: Pre-pretraining on formal languages imparts linguistic biases. arXiv:2502.19249, 2025.
arXiv preprint
Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li. Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyu- manshan Ye, Ethan Chemn Yixin Ye, et al. Olympicarena: Benchmarking multi nulti-discipline cog- nitive reasoning for superintelligent ai. Advances in Neural Information Processing Systems, 37: 19209-19253, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1672) 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 2024.
Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, and Furu Wei. Think only when you need with large hybrid-reasoning models. arviv preprint arXiv:2505. 14631,, 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning, arXiv preprint arXiv:2503.09516, 2025.
Yu Kang. Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of-thought without compromising effectiveness. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 24312-24320 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383 2025a.
Zheng Li, Qingxiu Dong, Jingyuan Ma, Di Zhang, and Zhifang Sui. Selfbudgeter: Adaptive token allocation for efficient Ilm reasoning. arXiv preprint arXiv:2505. 1274, 2025b.
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. Learn to reason efficiently with adaptive length- reward shaping. arXiv preprint arXiv:2505.15611 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1 zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qing- ping Yang, and Shuangzhi Wu. Adacot: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning. arXiv preprint arXiv:2505.1 11896, 1896, 2025.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. Ol-pruner: Lenmth-harmonizing fine-tuning for ol-like reasoning pruning. arXiv preprint Aix: 12570, 2025a
Michael Luo. Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol -prrreve with 1.5b model by scaling rl, 2025b. Notion Blog.
Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length- compressible chain-of-thought tuning. arXiv preprint 2rXiv:2502.0960,, 2025.
12

---

Published as a conference paper at ICLR 2026
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang. Junjun He, et al. Mm-eureka: Exploring the frontiers of multi- modal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07366, 2025.
Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Alek- sandra Faust, Clement Farabet, and Shane Legg. Levels of agi for operationalizing progress the path to agi. arXiv preprint ar3iv:231 00022311102266 02442, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint MrXiv:2501.19399 2025.
Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, and Se- Young Yun. Self- training elicits concise reasoning in large language models. arXiv preprint arXiv:250.200122, 2025.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang. Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. Humanity's last exam. ArXiv preprint, has/2501.142444 2025. URL htttps://axxiv.ooo ats//b////// /2011 14249.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint Xiv: 1et.:7707003.00 2017.
ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang. Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb rea- soning models with reinforcement learning. arXiv preprint E.Xiv:2504113144 2025.
Amrith Setlur, Matthew YR Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Sim- chowitz, and Aviral Kumar. e3: Learning explore enables extrapolation of test-time compute for llms. arXiv preprint arXiv:2506.09026 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Rumxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 1409.19256, 2024.
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning. arXiv preprint wtXiv:2508.00726, 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.2053. 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint htt.::2551...99999 2025b.
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-awee rejection tuning for mathematical problem-solving. ArXiv preprint, abs/2407. 13690, 2024. URL https:// /taa:///rrivv http:///awwvvv.oa//....s/s wtg/ 2207. 13690.
13

---

Published as a conference paper at ICLR 2026
Songjun Tu, Jiahao Lin, Qichao Zhang. Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. Learning when to think: Shaping adaptive reasoning in rl-style models via multi-stage rl. arXiv preprint arXiv:2505.10832. 2025.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01939, 2025a.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.205 205 2025b.
Zhenhailong Wang, Xuehang Guo, Sofia Stoica, Haiyang Xu, Hongru Wang, Hyeonjeong Ha, Xiusi Chen, Yangyi Chen, Ming Yan, Fei Huang, et al. Perception-aware policy optimization for mul- timodal reasoning. arXiv preprint arXiv:2507.060 064488 2025c.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma. Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agar- wal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:// wwwpppparrrsiipppp.e.ooe..p ww/papppppppcccccccccccccccce.tt saapperrrfllsseeeeeeeeeeeeeeeeee waales/paper/22222 22a/2/////hh///// html.
Hao Wen, Xinrui Wu, Yi Sun, Feifei Zhang, Liye Chen, Jie Wang. Yunxin Liu, Ya-Qin Zhang, and Yuanchun Li. Budgetthinker: Empowering budget-aware I1m reasoning with control tokens. arXiv preprint arXiv:2508. 2025.
Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Advancing multimodal reasoning capabilities of multimodal large language models via visual perception reward. arXiv preprint arXiv:2506.07211 2025.
Xin Xu, Tong Xiao, Zitong Chao, Zhenya Huang, Can Yang, and Yang Wang. Can ilms solve longer math word problems better? ArXiv preprint, abs/2405. 14804, 2024. URL https: // //arx/
/405. 14804.
Xin Xu, Tianhao Chen, Fan Zhang, Wanlong Liu, Pengxiang Li, Ajay Kumar Jaiswal, Yuchen Yan, Jishan Hu, Yang Wang, Hao Chen, et al. Double-checker: Enhancing reasoning slow-thinking llms via self-critical fine-tuning. arXiv preprint arXiv:2506.2125 2025a.
Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. arXiv preprint arXiv:2502.00334 2025b.
Xin Xu, Jiaxin Zhang, Tianhao Chen, Zitong Chao, Jishan Hu, and Can Yang. Ugmathbench: A diverse and dynamic benchmark for undergraduate- level mathematical reasoning with large language models. arXiv preprint arXiv:2501. 137661666.11.66, 2025c.
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Zejun Ma, and Bo An. Simpletir: End-to- end reinforcement learning for multi-turn tool-integrated reasoning. https: /sip://irimimi..... notion. www.:essttererr..eeeeeeee 2025. Notion Blog.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self- ooff-imprvvement arXiv preprint arXiv:2409.12122 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Chenxu Yang, Qingyi Si, Mz Dai, Dingyu Yao, Mingyu Zheng, Minghui Chen, Zheng Lin, and Weiping Wang. Test-time prompt intervention. arXiv preprint arXiv:2508.022. 02511, 2025b.
14

---

Published as a conference paper at ICLR 2026
Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit in reasoning models. arXiv preprint SrXi:22504115955, 2025c.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387 2025.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. ArXiv preprint, abs/2309. 12284, 2023. URL https: //arxiv. www//o//aaooo. /ass//bb/// 2309. 12284.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 2025.
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arxiv preprint arx:v:2508. 06471, 2025a.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 1020503..882922 18892, 2025b.
Fuxiang Zhang, Jiacheng Xu, Chaojie Wang, Ce Cui, Yang Liu, and Bo An. Incentivizing Ilms self-verify their answers. arXiv preprint arXiv:2506.01 369, 2025a.
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthink: Reasoning models can learn when to think, arXiv preprint arXiv:2505. 13417, 2025b.
Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng. Critique-grpor Advancing Ilm reasoning with natural language and numerical feedback. arXiv preprint arXiv:2506.033 036506 2025c.
Xiaoyun Zhang, Jingqing Ruan, Xing Ma. Yawen Zhu Haodong Zhao, Hao Li, Jiansong Chen, Ke Zeng, and Xunliang Cai. When continue thinking: Adaptive thinking mode switching for efficient reasoning. arXiv preprint arXiv:2505. 15400, 2025d.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507. 180711 2025.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instructioonfoolowinng evaluation for large language models. arXiv preprint 1,7711, 2023. doi: 10.4 URL https:/ httpsss//axivvoo// abs/
15

---

Published as a conference paper at ICLR 2026
A BACKGROUND AND PRELIMINARY
A.1 RLVR ALGORITIMM
Proximal Policy Optimization (PPO). PPO (Schulman et al., 2017) constrains the policy update within a proximal region of the old policy Thy through the clipping mechanism. Specifically, PPO employs the following objective (we omit the KL regularization term hereinafter for brevity):
(vro(O) BEnD, yoote (llr)
min (re(0).c,, clip (r:(0), - -,,
(4)
with the importance ratio of the token y/ is defined as r2(0) =ewe (r.Ve))
the advantage As of y
estimated by value model, and is the clipping range of importance ratios.
Group Relative Policy Optimization (GRPO), GRPO (Shao al., 2024) bypasses the need for the value model by computing the relative advantage of each response within a group of responses to the same query. Specifically, GRPO optimizes JGRPO = E2 [GRRPo (0,))] x)], where:
JGRPO (0, x)=
min (re,(0)An,,, clip
(5)
G
Here [v: ~ To (-)x), G' is the number of generated responses to each query (i.e., the group size), and (0), Ai,t is computed in eq. (2).
Numerous variants have been proposed to improve GRPO. For example, DAPO (Yu et al., 2025) introduces token-level normalization and dynamic sampling: Dr GRPÒ (Liu et al., 2025b) removes length bias to prevent incorrect responses from growing longer over time; and Wang et al. (2025a) train selectively on forking tokens (see also Section 6). Our TFPI orthogonal these RLVR algo- rithms. That is to say, any RLVR algorithm can be applied to our proposed TFPI stage. To isolate the effect RLVR algorithms, we employ DAPO as our RLVR algorithm in all experiments for fair comparison.
A.2 ThinkingFree OPERATION
In Section 3.1, we have shown chat templates for both the original query x and its thinking-free version x' = SriininnFFre((2 for Qwen models. Here, we showcase one additional example under the DeepSeek (Guo et al., 2025) template as below.
Template (Thinking Mode (DeepSeck)) Please reason step by step, and put your final answer within [bboxe{):<<(usr// (qqeettttrrtsustoo <Assistatt |>\n
Template (Thinking-Free Mode (DeepSeek)) Please reason step by step, and put your final answer within \ [ (queesttrr/uttstoo {question (Asssst(snnn /i ///h<nk
B META EXPERIMENTS
B. TOKEN CONSUMPTION OF ThinkingFree
We provide the detailed experimental setup for the meta-experiment in Section 3.1 in this appendix.
For DS-1.52. 5B, we use the decoding parameters suggested by Guo et al. (2025) in thinking mode: temperature = 0.6, top-p = 0.95, and top-k -1 For Qwen3 4B, we adopt the recommended settings from Yang al. (2025a) in thinking mode: temperature = 0.6, top-p 0.95, and top-k 20. For ThinkingFree, we set temperature = 0.7, top-p = 0.8, and use both top-k = -1 and top- k= = 20. The maximum output length is fixed at 32K tokens for both modes. For evaluation, sample 32 generations per query on AIME 2025 and report the average number of output tokens. TTee parameters are given in Table and the results shown in Figure 2 (Left).
16

---

Published as a conference paper at ICLR 2026
Table 5: Decoding Parameters of Meta-Experiment in Section 3.1
Thinking Mode
Thinking-Free Mode
DS-1.5B
0.6, top-p = 0.95, and top-k = -1 T 0.7, top-p = 0.8, and top-k
Qwen3-4R
T = 0.6, top-p 0.95, and top-k 20 T = 0.7, top-p 0.8, and top-k = 20
B.2 DETAILED SETUP OF ThinkingFree TRAINING
We provide the detailed experimental setup for the meta-experiment discussed in Section 3.2 in this appendix.
For training, we use the DAPO (Yu et al., 2025) algorithm, with configurations identical to those Appendix C.I, except that the maximum output length is set to 4K. For evaluation, we set the maximum ouiput length to 48K and perform testing in thinking mode as described in Template 1. All other evaluation parameters follow Appendix C.3. Note that the initial avg 32 value in Figure (Right) is higher than the value reported in (Yang al., 2025a) (68.2 vs. 65.6), because adopt the RoPE scaling method described in Polaris (An et al., 2025).
C EXPERIMENTAL DETAILS
In this appendix, we provide the details of our main experiments Section 4.
C.I TRAINING DETAILS
We build the VeRL codebase (Sheng al., 2024), with the RLVR loss following the DAPO recipe (Yu et al., 2025). Specifically, the RLVR loss is defined in eq. (1). For our TFPI, it becomes
EgnD [JDAPO x')) x' TrinkingFree(x)).
For fair comparison, we use identical hyperparameters across methods. For clip-higher, we set 8low = 0.2 and Shigh 0.28. Training performed with a batch size and mini-batch size of 256, learning rate of 10-6, and no warm-up scheduling. Both KL. divergence loss and entropy loss are excluded.
For rollout, we use temperature 1, topp 1, and topk -1. We generate 8 rollouts per problem. Experiments are conducted on DS-1. 5B, Qwen3-4B, and DS-7B, with Polaris-53K (An et al., 2025) as the training dataset. In principle, we could apply dataset filtering at each training stage to accelerate training An et al. (2025). However, for fairness, deliberately use the full training set for all experiments.
For Direct RLVR, the maximum output length is set to 16K for DS 5B and DS- DS-7B, and 32K for Qwen3-44 4B. For TFPI, we adopt a multi-stage training strategy (An et al., 2025; Luo et al., 2025b);
DDS11.5B 5B and DS 7B: 2K
4K 8K.
1.Qee3344 4B: 4K
8K
16K.
Our experiments are conducted with 32 H20 GPUs. A summary of the number of training steps and training time is provided in Table 6. To provide a comprehensive view of training compute, we also report the number of processed tokens and the average MFU in Table 7. Notably, our method achieves higher MFU and therefore can generate more tokens than "Direct RL" within the same wall-clock time. This arises because shorter responses during RL training reduce rollout-stage "bubble", yielding practical efficiency advantage.
C.2 BASELINES
To evaluate the efficacy of TFPI, compare TFPI with direct RLVR training from SFT-distilled LRM ("Direct RL" for short) under the same total training compute, i.c., the combined compute of the three TFPI stages equals that of direct RLVR (Table 1). To examine the effect of inserting TFPI
17

---

Published as a conference paper at ICLR 2026
Table 6: Training Steps and Time of Main Experiments. "Kh" denotes one thousand H20 Hours.
DS-1 5B
Qwen3-4B
DS-7B
DS-1.5B
Qwen3-4B
DS-7B
TFPI Stage 1
2K. IK steps
4K, 100 steps
2K. 1K steps
0.84 kh
0.35 kh
1.73 kh
TFPI Stage 2
4K. 440 steps
8K, 56 steps
4K. 232 steps
0.62 kh
0.52 kh
0.82 kh
TFPI Stage
8K, 440 steps
16K, 64 steps
8K. 144 steps
1.21 kh
0.91 kh
0.94 kh
TFPI Total
2.66 kh
1.79 kh
3.50 kh
Direct RLVR
16K, 456 steps
32K, 20 steps
16K, 280 steps
2.67 kh
1.9 kh
3.52 kh
TFPI: RLVR
16K, 472 steps
32K, 192 steps
536 steps
2.49 kh
13.5 kh
6,55 kh
TFPI + RLVR Total
5.16 kh
15.4 kh
10 kh
Direct RLVR Total
16K. 896 steps
32K, 216 steps
16K, 820 steps
5.17 kh
15.7 kh
10.1 kh
Table 7: Total processed tokens and the average MFU (in %) of Main Experiments.
DS-1. 5B
Qwen 3-BB
DS- 7B
DS-1 1BB
Qwen3-44
DS- 7B
TFPI Stage
2.37 10"
5.00 x 10
2.75 x 10
69.27
66.28
73.73
TFPI Stage 2
2.26 x 10
5.02 x 10a
1.20 x 109
69.54
66.67
73.79
TFPI Stage 3
3.92 x 109
4.94 x 10a
1.24 10
68.66
66.76
73.76
TFPI Total
8.55 x 109
1.50 x 109
5.19 x 109
Direct RLVR
7.48 x 109
6.48 x 10
4.77 x 109
68.42
61.57
72.83
TFPI: RLVR
6.26 x 109
5.80 x 10"
7.98 x 109
66.20
60.08
72.58
TFPI RLVR Total
1.48 1040
7.30 x 109
1.27 1040
Direct RLVR Total
1.44 10
7.36 x 10
1.38 x 1010
68.11
62.24
72.89
stage before RLVR, apply TFPI to SFT-distilled model ("TFPI RL"), continue with standard RLVR, and compare the results with "Direct RL" using approximately the same training compute (Table 2). We also include several high-performing LRMs of the same model size from previous works for reference, including Polaris (An et al., 2025), DeepScaleR (Luo et al., 2025b), Skywork- ORI (He al.. 2025), and AReal-RL. For both Table and Table all models are evaluated in thinking mode (see also Appendix C.3). To assess the impact of TFPI reasoning efficiency, we compare our TFPI-trained model with various RL-baaed efficient reasoning baselines, including TLMRE (Arora & Zanette, 2025), AdaptThink (Zhang et al., 2025b). AutoThink (Tu et al., 2025), Laser (Liu et al., 2025a), L1Max (Aggarwal Welleck, 2025), and ThinkLess (Fang et al., 2025) (Table 3). The training and testing settings of these baselines, reported in their original papers, are summarized in Table For fair comparison, we standardize the testing parameters to topp 0.95, topk -1, and T = 0.6 with 32K maximum length, following the recommendations of DeepSeek-RR R1 (Guo et al., 2025).
Table 8: Training and evaluation details of efficient reasoning baselines from original papers. We unify the evaluation setting for fair comparison in Table 3. Details are provided in Appendix C.2 and C.3.
Model
Training Duta
Training Length
Test Length
Evaluation Details
DeepSeek-Distill -iteetiill-wee-111.1
TLMRE-DS-J.8 (.1)
3.2K examples from NuminaMath
32K
10 generations for AIME24
AdapyThink. 1.58 (5 0.05)
DeepScaleR 40K DeepScaleR-40k DeepSculeR
16K
16K
T 0.6. 16 generations for AIME24
TutoThink-DS- D8- St-gee3
8K 16K 24K
0.6,
Laser-DF-1.4094 10-.1-141.....55
32K
16 generations for AIME24
DeepscaleR-1. PBrrrviie
SuonThink-Sgge3 Max
DeegScaleR-HOK
8K - 16K - 24K
T = 0.6, generations
DeepScaleR R-20- 40K
4K
8K
Thinkless-L.90 --5-RLL
DeepScalRR-OOk
C.3 EVALUATION DETAILS
To comprehensively evaluate model capabilities, we employ a diverse set benchmarks covering mathematical reasoning, multi-task reasoning, code generation, and instructionfollowing
18

---

Published as a conference paper at ICLR 2026
1. Mathematical reasoning: We evaluate AIME24, AIME25, and Be- yondAIME (ByteDance- 2025). For AIME24 and AIME25, we report pass@1 accuracy using 32 samples per problem (avge 32); for BeyondAIME, report avg@8.
2. Multi-task reasoning: We evaluate on GPQA-Diamond (Rein et al., 2024) and report passel with 8 samples per problem.
3. Code generation:
We assess coding ability on LiveCodeBench (Jain et al., 2024) (2024- 08-2025-01 subset, aligned with DeepSeek-RI (Guo et al., 2025)), reporting passêl with 8 samples per problem.
4. Instruction-following We evaluate on IFEval (Zhou al., 2023) and report pass@l of the strict prompt accuracy with 4 samples per problem.
All evaluation codes are adapted from the DeepscaleR (Luo et al., 2025b) codebase, where vLLM (Kwon et al., 2023) is leveraged to accelrrate inference. For IFEval, we use the same codes provided by the official paper (Zhou et al., 2023).
We provide our decoding parameters as follows:
Table 1:
We set the temperature 0.6 and topp 0.95. For LRMs trained from DS-1.5B and DS-73, we use topk = -1 with a maximum sequence length of 32K tokens. For LRMs trained from Qwen3- -4B, 48K tokens, applying RoPE scaling as proposed by An et al. (2025).
topk 20 with maximum sequence length of
Table 2:
We the temperature 0.6 and topp 0.95. For LRMs initialized from DS-1. 5B and DS-7B, we use topk = -1 with a maximum sequence length of 32K tokens. For LRMs initialized from Qwen: -BB we use topk 20 with a maximum quence length of 48K tokens, again applying RoPE scaling as proposed by An et al. (2025). For models marked with **" we report the results from their original publications (see Ap- pendix C.4).
Table 3: For efficient reasoning baselines listed in Table 8 and the thinking mode of initial models, we set topp 0.95, topk 1, and T 0.6, with maximum sequence length 32K tokens (48K for 3wen3- 4B). For the thinking-free mode of initial models and our TFPI, we set topp 0.8, topk = 20, and T 0.7 with maximum sequence length of 32K tokens, following Yang et al. (2025a).
C.4 SOURCE OF SOME RESULTS IN TABLE
Results for LRMs marked with s*n are taken directly from the Reed-1.5-Thiie report (Seed al., 2025) and the corresponding Hugging Face page of Qwen3-2507, Note that the LiveCodeBench test set subsets of these results, and the metric of IFEval may differ from those in our experiments; their results are included for reference only.
D ANALYSIS DETAILS
As verification is an important indicator of slow-thinking capabilities (Setlur et al., 2025), we duct experiments in Section 5.1 to examine how verification can generalize to slow-thinking, even when trained with TFPI (thinking-free mode). Following Yang et al. (2025b), for the experiments in Figure 3, we segmented the reasoning trajectories using "nn'n'' as delimiters and classified each step according whether it contained verification- phrases such "wait". "let me verify", "let me check", "cheeking"" "verifying", or "double-check".
E THE USE OF LARGE LANGUAGE MODELS
We only used LLMs (specifically GPT-5) for language refinement during the writing process of this work; all ideas, experimental designs, and analyses were conducted by the authors. The text polished by GPT-5 was subsequently reviewed and verified by us ensure the quality of the content.
19

---

Published as a conference paper at ICLR 2026
Layer Index
0.07
0.06
cosine simllarity
0.05
0.04
0.03
0.02
0.01
0.00
250
500
750
1000
1250
1500
1750
training steps
Figure 6: Cosine similarity between parameter updates of TFPI- Frriinraieed checkpoints and an early "Direct RL" update of DS-1. across layers during training.
F MORE RESULTS
F.1 ADDITIONAL RESULTS OF PARAMETER-LEVEL ANALYSIS
Figure 4 (right) reports the cosine similarity of parameter-update vectors between TFPI-trained checkpoints and the final Direct RL checkpoint. For completeness, Figure presents the cosine similarity between TFPI-trained checkpoints and early Direct RL checkpoint (step 456) from the DS-1. 53 run used in Table 1. We observe stronger alignment with the final Direct RI. checkpoint than with the staly-stage checkpoint.
F.2 COMPLETE RESULTS OF TABLE 2 AND TABLE 3
Due to the page limit, present only representative results in Sections 4.3 and 4.4, with the com- plete results provided in Tables 9 and 10, respectively.
F.3 THINKING VS. THINKING-FREE INFERENCE
To compare thinking and thinking-free modes, Table 11 reports accuracy and average output token counts across training stages of TFPI for Qwen3-4R. The accuracy gap between the two modes narrows from Stage to Stage 3. In Stage 3, the thinking mode achieves an average accuracy of 63.9%, whereas the thinking-free mode achieves 60.0%. Meanwhile, the thinking-free mode produces substantially fewer tokens on average (9.9k vs. 15.3k). One may choose the thinking-free mode to trade a modest decrease in accuracy for improved efficiency, switch between modes to balance accuracy and cost.
F.4 ABOUT DECODING PARAMETERS
The decoding parameters used in our main-text experiments are detailed in Appendices and C.3. Following the official Hugging Face recommendations for each model, we adopt different settings for the thinking and thinking- free modes. To assess robustness,
additionally report results under unified configuration: temperature = 0.6, top-p = 0.95, top-k 20, and maximum output length of 32k tokens. As shown Tables 13, 14, 15, and 16, all of our conclusions remain unchanged under this unified setting.
Polaris (An et al., 2025) reports that longer-context inference with RoPE scaling improves perfor- mance on harder questions and evaluates at 90K tokens. We find that 90K is prohibitively slow, and 48K suffices based on the clip ratio analysis (Table 12). As shown in Table 15, constraining the maximum output length 32K slightly degrades the performance of the thinking mode for models RL-tuned from Owen3-44 3-BB. We therefore recommend 48K context length for Qwen3 - 4B-based RI--tuned models.
20

---

Published as a conference paper at ICLR 2026
Table 9: Results (%) of RL after TFPI ("TFPI+RL") vs. "Direct RL" across different benchmarks. "Avg@k" denotes the average accuracy (%) over random generations (i.e., pass@1). For LRMs marked with "*" results are taken from the corresponding reports (see Appendix C.4); all other results are from our own runs. All models are evaluated in thinking mode. The total training compute "TFPI+RL" is matched to that of "Direct RL" for fair comparison.
Mathematics
Multi- Task
Code
Instruction
Overall
Models
24
Beyond AIME
GPQA Avgea
LiveCode Avges
IFEval Avgê4
Overall
Avg832
Avg032
Avge8
Avg.
DeepSeck RI*
79.8
65.0
42.4
71.5
64.3
86.1
68.2
Seed-1.5 Tedi-I.5-Tinnking
86.7
74.0
48.0
77.3
64.99
87.4
73.0
Claude4 Opus Thinking
75.5
79.6
48.9
89.7
Qwen3-235B-Thinel
85.7
81.5
71.1
55.7
83.4
Qwen3-SB DS-R 08528-Qwe3--81
75.3
67.0
43,2
61.7
51.7
66.0
61.7
82.8
74.6
50.6
61.4
$9.0
73.9
67.0
1. Size
DeepScaleR-1e
37.8
31.6
19.1
21.9
40.5
27.3
DS-1.58 Direct
37.2
28.5
24.6
19.5
39.5
27.0
DS-1.5B TFPI stage
40.1
30.8
13.8
29.6
19.9
40.8
29.2
DS-1.50 TFPI RL.
42.3
32,9
15.1
27.8
20.9
41.0
30.0
4B Size
Sirris-4B-Previiee
73.2
70.7
39.9
54.9
39.7
63.9
$7.0
Qwen3-4B Direct
78.8
71.5
56.2
54.3
65.1
62.0
Qwen3-4B TPPI stage
79.9
70.6
46.7
58.5
57.0
70.2
63.8
Qwen3-4B TFPI + RL
80.8
76.0
49.5
61.1
55.7
71.3
65.7
Owen3-1B-25077 18-25007Thinningg
84.7
79.2
31.5
66.4
62.4
68.0
68.7
TTPP only
81.2
70.1
65.5
66.7
70.8
Size
1Reel-hhha--oe hoha-RL--788
61.5
46.1
35.4
57.5
44.2
Skywork-OR Beooor--RR-7B------ BRR--B-Preview
61.2
46.4
31.2
35.2
43.3
55.4
45.5
Polaris-7B 7B- Preview
70.6
48.8
37.0
34.1
43.9
55.6
48.3
DS-7B Direct RI.
62.3
47.9
30.1
36.9
42.8
57.1
46.2
DS-7B TEPI stage
62.0
44.6
46.8
42.1
60.2
47.8
DS-7B TFPI RI.
65.3
47.1
33.2
45.9
43.3
57.3
48.7
Table 10: Comparison of the Thinking-Free inference mode of TFPI with efficient reasoning base- lines across various reasoning tasks. "Avgłk" denotes the average accuracy (in %) over k genera- tions (i.e., pass?l), and "Toks" indicates the average output length in thousands of tokens (K).
AIME 24
AIME 25
Beyond AIME
GPQA
Overall
Models
avg832
Toks
2vg832
Toke
avges
Taks
avg8a
Toks
Avg.
Taks
DeepSeek-Diee Priatill-Wwen-1 -Qwen-1.
TLMRE-DS-LSB 0.1)
27.6
12.9
24.8
12.5
11.9
14.8
19.1
11.2
Adap(Think-1.55 1.55 (8
0.05)
28.1
8.0
22.6
7.9
10.0
4.8
14.8
5.1
18.9
6.5
AutoThink-1D DS DS-1.5B--... 1.5g-Stage3
30.3
10.2
25.2
9.1
9.4
8.7
17.4
7.0
20.6
8.7
Lser--EELA099-11
30.3
24.9
7.4
6.9
Initial Model (Thinking)
29.6
16.7
23.0
16.5
8,7
14.4
16.3
9.8
19.4
14.3
Initial Model (Thinking-Free))
12.4
5.7
10.9
4.4
4.4
3.4
4.2
0.9
8.0
3.6
TFPI stage
15.3
1.3
TFPI stage
31.5
3.4
24.2
3.1
10.1
2.9
35.3
1.6
25.3
2.7
IFPI stage
37.5
28.4
5.0
12.4
35.6
28.5
4.4
DeepscaleR-1 SB-Preview
AutoThink-Stuge
38.9
28.9
7.7
11.6
27.3
26.7
7.4
L1-1,5B-Max Thinkless- 1.50s---1-BBB- 1.5B-RL.
27.2
3.2
26.3
2.9
9.1
3.1
32.4
2.3
23.8
2.9
11.3
24.1
11.1
8.1
11.7
20.3
12.7
20.2
11.7
Qwen3-4B
Initial Model (Thinking)
73.6
18.0
68.3
22.3
43.4
23.5
56.8
10.7
60.5
18.6
Initial Model (Thinking-Free)
26.9
7.2
20.2
5.5
11.0
4.2
45.0
2.3
25.8
4,8
IFPI stage
31.1
3.4
23.2
47.8
36.5
3.0
TFPI stage?
63.7
52.8
8.0
33.0
7.0
50.9
50.1
6.1
TFPI stage
75.1
10.5
66.9
12.7
42.1
12.7
55.9
3.7
60.0
9.9
Qwen3-4B -B-nIn-tutttt-250
60.4
46.1
7.4
33.0
64.4
51.0
6.9
21

---

Published as a conference paper at ICLR 2026
Table 11: Comparison of the Thinking-Free inference mode with the Thinking mode across various reasoning tasks after TFPI. "Avgžk" denotes the average accuracy (in %) over k generations (i.e., pass@l), and "Toks" indicates the average output length thousands tokens (K).
AIME
Beyond
GPQA
Overall
Models
zvg832
Toks
avg
Toks
avgg8
Toks
avges
Toks
Avg.
Toks
Qwen3-48 3-4B
Initial Model (Thinking)
73.6
18.0
68.3
22.3
43.4
23.5
56.8
10.7
60.5
18.6
Initial Model (Thinking Free)
26.9
4.2
4.8
TFPI stage (Thinking)
75.2
13.8
67.8
17.1
42.4
16.8
57.9
7.5
60.8
13.8
TFPI stage (Thinking--rree
43.9
3.7
31.1
3,4
23.2
3.4
47.8
1.5
36.5
3.0
IIFI stage (Thinking)
76.0
13.4
68.2
44.7
16.7
57.8
7.7
61.7
13.7
TFPI stage?,
63.7
7.2
52.8
8.0
33.0
7.0
50.9
2.3
50.1
6.1
TFPI stage (Thinking)
79.9
15.1
70.6
18.7
46.7
18.8
58.5
8.6
63.9
15.3
stage Qwen3 4B 1--t---Bi-tnt---1 Instruct-2507 2507
75.1
12.7
3.7
9.9
60.4
7.9
46.1
7.4
33.0
7.3
64.4
4,8
51.0
6.9
Response Length
16K
32K
48K
64K
80K
90K
Avgê32
42.6
64.1
68.3
68.5
68.5
68.5
Clip Ratio
55.7
25.7
5.2
1.2
0.6
0.0
Table 12: AIME25 results for Owen -BB: Avg0 32 and clip ratio across maximum output lengths.
F.5 ABOUT TRAINING TEMPERATURE
In our main experiments, we set the training temperature to 1.0 (Appendix C.I). Motivated by Polaris( An al., 2025), which reports "optimal" temperature of 1.4 for Qwen3-4R, conduct an ablation on training temperature using Qwen3-4BB As shown in Figure7, performance at 1.4 comparable to 1.0 for both TFPI Stage I and Direct RL (measured by AVG332 on AIME25), suggesting that our results are robust to this choice of training temperature.
F.6 ABOUT THE STAGE BOUNDARY AND STAGE LENGTH
Following prior multi-stage training practice (An et al., 2025; Luo et al., 2025b), we increase the training context length when the clip ratio shows a renewed inflection and validation performance begins to plateau. Figure 8 illustrates this behavior for TFPI Stage on Qwen3-4B; step 100 is natural transition point.
Training context length trades off training throughput and effective learning signal: longer contexts slow training, whereas shorter contexts constrain learning on harder questions. We therefore begin with a shorter length and expand as training stabilizes. To set stage lengths, we prioritize effective signal: with 8-sample rollouts, 0/8 8/8 correctness contributes no learning signal (Yu al., 2025), so for candidate length L, we compute the share of questions with 1/8-7/8 correctness as a proxy for effective training mass. We choose I. preserve throughput while maintaining substantial ef- fective mass at each stage. For Qwen3-4PF L=4K yields 28K/53K questions the 1/8-7/8 band, whereas L=8K yields 29K/53K; thus we start 4K for efficiency and move 8K once validation gains plateau. Note that the effective sets across stages are not disjoint; for instance, the same ques-
Table 13: Re-evaluate the meta-expeni exteriexpei in Figure 2 (Left) under unified evaluation setting (temperature 0.6, top-p= 0.95, top-k = 20). "# Tokens" denotes the average number of tokens on AIME25 and "A" is the percentage of reduction in token consumption.
Model
# Tokens (Thinking)
# Tooeens (Thinking- Free)
A
DS-1.55 5B (original setting) Unified Setting
16.5K
4.4K
-73%
16.4K
3.6K
-78%
Qwen3-de 4BB (original setting)
17.4K
5.1K
-71%
Unified Setting
17.4K
4.7K
-73%
22

---

Published as a conference paper at ICLR 2026
Table 14: Results of TFPI vs. direct RL across different benchmarks. "Avgłk" denotes the average accuracy (%) over k random generations (i.e., passel). ). All models are evaluated in thinking mode. The total training compute for the stages of TFPI equals that of "Direct RL" for fair comparison. This table corresponds to Table under unified decoding setting.
Mathematics
Multi-Task
Code
Instruction
Overall
Models
AIME Avg832
AIME Beyond AIME Avg032
GPQA Avgê8 8
LiveCode AvgeB
IFEval Avg04
Overall
Avg08
Avg.
DeepSeek-Dist Qwen-1.. 5B
Initial Model
29.1
25.5
16.3
40.1
22.9
Direct
35.5
27.9
20.2
38.1
25.5
TFPI stage
31.8
26.7
12.5
30
39.5
26.6
-TFPI stage
35.1
27.6
13.4
29,7
44.0
28.1
TFPI stage
36.1
30.1
14.9
28.5
41.8
28.7
Qwen3-4B
Initial Model
71.6
64.1
41.5
56.6
54.9
64.9
58.9
Direct RI.
73.4
62.2
41.1
56.2
52.1
66.0
58.5
TFPI stage
74.7
67.4
42.1
57.9
55.3
66.0
60.6
TFPI stage
75.6
67.5
44.6
57.8
54.6
64.8
60.8
TFPI stage
78.7
69.5
58.5
56.8
70.2
63.3
DeepSeek-Disee Sei--Distilll Qwen-7B
Initial Model
53.3
40.5
25.0
36.8
37.6
55.6
41.5
Direct RL
56.8
41.6
26.7
37.4
39.6
58.2
43.4
TFPI stage
58.8
39,6
28.8
49.4
40.1
56.6
45.5
-TFPI stage
61.7
42.9
31.5
47.5
42.0
57.5
47.2
TTFPI stage
62.6
43.2
30.7
47.1
42.4
59.3
47.5
Table 15: Results (%) of RL after TFPI ("TFPI+RL") vs. "Direct RL" across different benchmarks. "Avgōk" denotes the average accuracy (%) over random generations (i.e., pass@1). For LRMs marked with "** results are taken from the corresponding reports (see Appendix C.4); All models are evaluated in thinking mode. The total training compute for "TFPI+RL" is matched to that of "Direct RL" for fair comparison. This table corresponds Table 2 under unified decoding setting.
Mathematics
Multi Task
Code
Instruction
Overall
Models
ATME
AIME
Beyond ATME
GPQA Avgês
LiveCode Avqe8
IFEval Avq84
Overall
Avg832
Avge32
Avge8
DeepSeek RI" Behhik.5-Thinking"
79.8
65.0
42.4
71.5
64.3
86.1
68.2
86.7
74.0
48.0
77.3
64.9
87.4
73.0
Claude4 Opus Thinking"
75.5
79.6
48.9
89.7
Qwen3-235H Thinking
81.5
71.1
83.4
Qwen3-4B Direct
75.8
Qwen3-4B TFPL stage
78.7
69.5
46.1
58.5
56.8
70.2
63.3
Qwen3-4B TTPP RL
77.9
70.8
47.5
60.8
54.5
71.3
63.8
tion may score 3/8 4K and 6/8 at 8K. In all experiments, train the full Polaris-533 53K dataset to ensure fairness. However, for better efficiency, one can curate stage-specific subsets based on the effective-signal criterion by filtering questions that are 0/8 or 8/8 correct under the current context length.
F.7 MULTI-STAGE DIRECT RL (4K + 8K 16K)
Figure (right) shows that applying Direct RL with 4K response length causes a substantial per- formance drop on AIME25 for Qwen3- 4B. In contrast, TFPI trains stably even at 4K. suggesting that-even in multi-stage setting-the initial stage length should not be too short. We posit that enabling multi-stage training with short response lengths is a key benefit of TFPI. To rule out the possibility that this effect an artifact of staging, run Direct RL with the identical multi-stage schedule on Qwen3-4B. The complete results are reported Table 17, with training details pro- vided in Table 18.
23

---

Published as a conference paper at ICLR 2026
Table 16: Comparison of the Thinking-Free inference mode TFPI with efficient reasoning base- lines across various reasoning tasks. "Avgêk" denotes the average accuracy (in %) over k genera- tions (i.e., pass@l), and "Toks" indicates the average output length in thousands of tokens (K). Models with are trained from DeepScaleR- 1.......B 5B, while the remaining are from DS 1 5B. This table corresponds Table under a unified decoding setting.
AIME
Beyond AIME
GPQA
Overall
Models
avgg
avgg 32
avgês
Toks
avg08
Toks
Avg.
Toks
TLLMR---S--.55 (o 0.1)
27.6
12.9
24.8
12.5
9.2
11.9
14.8
7.5
19.1
11.2
2) Adap[Think 1.5B 0.05)
28.1
8.0
22.6
7.9
10.0
4.8
14.8
5.1
18.9
6.5
AutoThink-DS-1 155-1_3B-
30.3
10.2
25.2
9.4
8.7
17.4
7.0
20.6
8.7
4) Laser DE-DDE-L-00 L4096 1,5B
30.3
8.3
24.9
7.4
9.7
7.4
21.1
4.7
21.5
6.9
Autol SinlSheh-SS
38.9
28.9
11.6
7.8
5,4
7.4
L1-1.5B-Mas
27.2
3.2
26.3
2.9
9.1
3.1
32.4
2.3
23.8
2.9
7) Thinkless-1 1-5-RRLL
28.4
11.3
24.1
11.1
8.1
11.7
20.3
12.7
20.2
11.7
DS-1.5B (Thinking)
25.5
9.0
9.8
DS-1.5B (Thinking-Free)
13.3
5.1
11.0
3.6
4.0
2.6
6.2
0.9
8.6
3.0
TTPPI stage
23.9
1.5
16.2
1.4
9.1
1.2
36.4
0.8
21.4
1.2
TFPI stage
28.6
25.0
10.6
2.9
1.6
2.7
TFPI stage
37.7
5.2
28.4
5.1
14,5
4.9
34.1
2.6
28.7
4.4
T=1.4
T-1.0
T-1.0
avgess
avg@32
68
67
44 training steps
80
20
56
training steps
Figure 7: AVG832 (%) on AIME25 for Qwen3 4B across training steps under different training temperatures (1.4 1.0). Left: Stage of TFPI; Right: Direct RI.
Table 17: TFPI v.s. Multi-Stage Direct RL. All models are evaluated in thinking mode.
Mathematics
Multi-Task
Code
Instruction
Overall
Models
AIME 24
AIME 25
Beyond AIME
GPQA AvgRs
LiveCode Avg08
IFEval Avg84
Overall
Avg832
Avg032
Avgl8
Avg.
Initial Model
73.6
68.3
43.4
56.8
54.9
64.9
60.3
TFPI
75.2
67.8
42.4
$7.9
55.3
66.0
60.8
Stute Direct RI. Stage
49.2
36.0
22.6
51.4
44.0
64.5
44.6
TFPI stuge
76.0
68.2
44.7
54.8
61.0
Multi-Stuge Direct RI.
58.4
42.3
28.2
52.9
48.4
63.2
48.9
TFPI stage 3
79.9
70.6
46.7
58.5
57.0
70.2
63.8
Stuge Direct RL. Stage
64.9
48.6
32.5
55.2
51.0
65.2
52.9
ellp ratio Lokens
.2500
88
-2400
86-
cip matoo
Ptf
-2.300
tokeos
Accurscy
2200
10.0
Wh
2.000
82
7.5
22000
training steps
training stops
Figure 8: Qwen3-4B 4B TEPI Stage Left: clip ratio and average output tokens over training steps; Right: MATH500 validation accuracy. Step 100 marks turning point.
24

---

Published as a conference paper at ICLR 2026
Table 18: Training Steps, Time, and MFU of TFPI v.s. Direct RL with Qwen3-4B:
Training Steps
Training Time (kh)
MFU (%)
TFPI Stage Multi-Stage Direct RI. Stage
4K, 100 steps
0.35
66.28
4K, 100 steps
0.45
66.82
TFPI Stage
8K, 56 steps
0.52
66.67
Multi-Stage Direct RL Stage 2
8K, 56 steps
0.51
66.31
TFPI Stage
16K, 64 steps
0,91
66.76
Multi-Staa Stage Direct RL Stage 3
16K, 64 steps
0.97
65.99
TFPI Total
1.79
Multi-Stage Direct RL Total
1.93
G REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide detailed description of our method in Section 3.3 and present all experimental setups in Section 4.1 and Appendix C, including training configurations and eval- uation protocols. All evaluation datasets, the training set, the training framework, testing codes, as well as the efficient reasoning baselines used in our experiments, are publicly available and prop- erly referenced (see Appendix C). We will also release our codes and training checkpoints upon acceptance for reproducibility.
25


Reviews:
[{"review_id": "9ePGP36srR", "reviewer": "Reviewer_dBLU", "summary": "The paper proposes Thinking-Free Policy Initialization (TFPI), a short, low-cost RLVR stage that trains an SFT-distilled long-CoT model using `thinking-free' prompts (explicitly closing a <think> section with </think>). The hypothesis is that (i) thinking-free rollouts dramatically cut sequence length and compute; (ii) training the policy on this setting nonetheless improves slow-thinking performance when evaluated with normal CoT; and (iii) TFPI provides a stronger starting point for subsequent long-context RLVR, raising final performance and further improving token efficiency. Experiments across across math datasets, GPQA, code (LiveCodeBench) and instruction following (IFEval) support these claims.", "questions": "1. Stage schedule sensitivity: Only (2K->4K->8K) and (4K->8K->16K) are tested. Do different schedules (e.g., more stages, skipping a stage, starting at 8K) matter? Is TFPI effective if started at 8K or 16K (eg, for hard datasets even thinkfree could lead to more than 8K tokens with Qwen3 models)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a simple and practical method that shows promising results both in terms of training efficiency and performance.\n2. Under similar compute budgets, TFPI improves slow-thinking performance while yielding shorter outputs in thinking-free inference.\n3. The paper shows several useful ablations and analyses, and compares to several baselines.", "weaknesses": "**I do not find any major weaknesses in the paper.** Nonetheless some aspects of the paper could be improved:\n\n\n1. Table 5 in Appendix shows that the full RL method is trained only for 20 steps, in order to match the compute with the baselines. However, it isn't clear whether the same would hold if the compute for all methods increased? Therefore it remains unclear, whether TFPI only works in low train compute settings, or even if the goal is to squeeze last bit of performance from RL training?\n2. The paper uses Polaris-53K dataset, however given they had an extensive study on optimal temperature for RL showcasing the optimal temperature to be 1.4 for Qwen3-4B, why did authors choose to use 1.0? This might also mean, that performance of direct RL could be further improved by just changing this hyperparameter (though it might benefit TFPI as well).\n3. PCA Analysis: Distances in low-dimensional projections can be misleading in high-dimensional weight spaces. The cosine similarity plot (alignment with the direct RL direction) is suggestive, but adding baselines (e.g., alignment to a separate training run, or to the early direct-RL update) would make the analysis more robust. Because if both TFPI and direct RL share the same reward and data, some alignment is expected.\n\n\nSome Writing Suggestions:\n\n1. The matched compute is using wall-clock, but that alone doesn't give full picture, since full utilization of gpu may not be happeining. So providing other variables such as total processed tokens, rl steps (already provided in appendix), etc would further add value.\n2. `Generalizes across domains even when trained solely on mathematics.` Given modest but inconsistent gains (without provided statistical significance) outside math, I would call it `some transfer` rather than `generalizes across domains.`.\n3. Please bold the numbers in all relevant tables (such as Table 1, 2)\n4. I think parts of the paper could be simplified, equations for PPO and GRPO might be unecessary, or could be moved to appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qaRAJmUdC1", "reviewer": "Reviewer_gUP8", "summary": "This paper proposes a fast, low-cost initialization phase for long-CoT RL called Thinking-Free Policy Initialization (TFPI). By eliminating the thinking process during generation, it significantly reduces the training cost of Long CoT RL. The implementation is also relatively straightforward: by simply replacing the Rollout in the existing DAPO method with inference that omits the thinking process, it simultaneously lowers RL computational costs and improves performance.", "questions": "Why are the hyper-parameter settings inconsistent across comparative experiments? For instance, the hyper-parameters of thinking Mode and thinking-free mode in Table 4 are different.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1）By replacing the rollout in the existing DAPO method with thinking-free inference, this work achieves simultaneous reduction of RL computational cost and improvement in performance, which is of great significance to RL training.\n2）The experimental results are comprehensive, demonstrating that TFPI not only lowers computational cost but also enhances slow-thinking reasoning ability, with validations conducted across domains and settings. In addition, the experimental section discusses, from both behavioral and parameter perspectives, why TFPI is able to improve slow-thinking reasoning performance.", "weaknesses": "1）The description of the proposed method in the main text appears rather brief, making it difficult to grasp all the method details clearly.\n2）In the experimental section, no details are provided for the choice of hyper-parameters such as temperature, topp, topk, or maximum output length.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y783Jrs5D0", "reviewer": "Reviewer_vwux", "summary": "This paper introduces Thinking-Free Policy Initialization (TFPI), a training method for large reasoning models designed to improve performance and efficiency while reducing computational costs. The core idea here is to use a \"Thinking-Free\" operation during a multistage Reinforcement Learning process. This is achieved by modifying the input template to explicitly omit the model's chain-of-thought tokens (`</think>` append). The training follows a curriculum (CL) where the context length is progressively increased. The authors conduct experiments on various model sizes and benchmarks, demonstrating that TFPI can achieve strong reasoning performance with significantly less training compute compared to direct long context (includes CoT tokens) RL training.", "questions": "1. The paper's central claim hinges on the efficiency of the 'Thinking-Free' template. Could the authors conduct the critical ablation experiment: train a model using the exact same multi-stage curriculum ($2K \\rightarrow 4K \\rightarrow 8K$) but with the standard 'thinking' template enabled throughout? Please provide a direct comparison of both the final performance and the computational resources (e.g., GPU hours and memory usage) consumed against the TFPI-trained models. This is essential to determine if the gains come from the template modification or simply the curriculum.\n\n2. How do the authors explain the performance fluctuations and, in some cases, degradations observed in out-of-domain tasks (e.g., GPQA for DS-7B) in the later stages of TFPI? \n3. Could the authors elaborate on the rationale for comparing the multi-stage TFPI against a single-stage \"Direct RL\" baseline, rather than a multi-stage \"thinking\" baseline, which would serve as a more direct control for the curriculum aspect of the training?\n4. Your evaluation uses inconsistent decoding parameters (such as lower `top-p` for thinking-free) and max sequence lengths (32K vs. 48K) across different models and modes. This introduces confounding variables. How can you demonstrate that the observed efficiency gains are a direct result of the TFPI training, and not an artifact of these different evaluation settings?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles the significant and practical challenge of the high computational costs associated with training large reasoning models (LRMs) using long-context Reinforcement Learning with Verifiable Reward (RLVR).\n- The authors provide an extensive evaluation across multiple model sizes (1.5B, 4B, 7B) and a diverse set of reasoning benchmarks, including mathematics (AIME), multi-task reasoning (GPQA), code generation (LiveCodeBench), and instruction following (IFEval). The inclusion of both in-domain and out-of-domain tasks is noteworthy.\n- The results presented are strong and demonstrate high efficiency. For example, the paper reports that a 4B model trained with TFPI achieves 89.0% accuracy on AIME24 using less than 4,000 H20 GPU hours, which is good.\n- The paper includes both behavioral (verification analysis) and parameter-level (PCA) analyses to provide insight into why TFPI is effective. This adds value to the empirical results and helps to understand the underlying mechanics of the proposed method.", "weaknesses": "The primary weaknesses of this paper relate to the experimental design and the isolation of the core contribution, which makes it difficult to definitively attribute the observed gains to the proposed method.\n\n- The most significant weakness is the absence of a crucial baseline. The paper compares TFPI (multi-stage RL with thinking-free templates) against \"Direct RL\" (single-stage RL with a long context). However, it fails to compare against a model trained with the **same multi-stage schedule ($2K \\rightarrow 4K \\rightarrow 8K$) but using the standard \"thinking\" template**. Without this ablation, it is impossible to disentangle the performance gains and determine whether they stem from the proposed \"Thinking-Free\" template modification or simply from the known benefits of a multi-stage, curriculum-based training schedule. The improvements could be an artifact of the training curriculum rather than the novelty of TFPI. Clearly, ablations studies are needed.\n\n- The paper acknowledges that multi-stage RLVR (starting with short contexts and gradually increasing them) is a \"common mitigation strategy\". The core technical contribution thus appears to be the addition of a simple template modification to this existing paradigm. While combining existing ideas can be effective, the lack of the aforementioned ablation makes it difficult to assess the true novelty and impact of the \"Thinking-Free\" operation itself.\n\n- The paper claims that TFPI exhibits generalizability across domains, even when trained exclusively on mathematics. However, the results show inconsistencies that weaken this claim. For example:\n\n    - In Table 1, the performance of the DS-7B model on GPQA drops from 49.0% after Stage 1 to 46.8% after Stage 3.\n    - For the DS-1.5B model, performance on LiveCodeBench fluctuates significantly across stages (18.4% $\\rightarrow$ 20.5% $\\rightarrow$ 19.9%).\n        These fluctuations suggest that the out-of-domain improvements are not consistently monotonic and may be sensitive to the training stage.\n\n- The primary comparison is between TFPI's multi-stage strategy and a single-stage \"Direct RL\" approach. Even with matched total compute, these are fundamentally different training strategies, making a direct comparison challenging to interpret. Furthermore, evaluation settings are inconsistent, with different max sequence lengths (32K vs 48K) and different decoding parameters for \"thinking\" vs. \"thinking-free\" modes, which complicates a fair assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WiVVcey9fM", "reviewer": "Reviewer_Z381", "summary": "This paper introduces Thinking-Free Policy Initialization (TFPI), a lightweight pre-training stage for reasoning LLMs before RL with verifiable rewards. TFPI removes explicit <think> content, training models to output only final solutions under short contexts. Experiments on DS-1.5B, DS-7B, and Qwen3-4B show that TFPI improves reasoning accuracy while reducing compute. When followed by normal RL, TFPI further enhances performance and speeds up convergence.", "questions": "- what happens if we remove the think segment entirely at inference after training—does accuracy remain similar?\n- How are the stage lengths (2K→4K→8K) determined in practice? Are they tuned for stability or fixed heuristically?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Efficiency: TFPI significantly reduces rollout tokens and compute cost while maintaining or improving reasoning accuracy.\n- Generality: The method integrates seamlessly with existing RLVR frameworks (e.g., GRPO, DAPO) and scales consistently across 1.5B–7B models.\n- Insightfulness: Analysis reveals that TFPI fosters implicit verification-style reasoning and provides better initialization for subsequent long-context RL.", "weaknesses": "- The paper adopts a multi-stage TFPI schedule (e.g., 2K→4K→8K→16K) but does not justify how these stage boundaries are chosen. It is unclear whether they are based on empirical tuning, stability constraints, or model-size scaling rules.\n- The paper only evaluates TFPI on models up to 7 B parameters. It remains unclear whether the approach scales effectively to larger reasoning models (e.g., 32 B or 14B)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)