Incentivizing LLM Reasoning via Reinforcement Learning with Functional Monte Carlo Tree Search
Authors: Kongcheng Zhang
QI YAO
Baisheng Lai
Jiaxing Huang
Wenkai Fang
Dacheng Tao
Mingli Song
Shunyu Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Reinforcement Learning
Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: In this work, we propose ***R**einforced **F**unctional **T**oken **T**uning* (RFTT), a novel reinforced fine-tuning framework that empowers Large Language Models (LLMs) with learn-to-reason capabilities. Unlike prior prompt-driven reasoning efforts, RFTT embeds a rich set of learnable functional tokens (*e.g.*, \<analyze\>, \<verify\>, \<refine\>) directly into the model vocabulary, enabling chain-of-thought construction with diverse human-like reasoning behaviors. Specifically, RFTT comprises two phases: (1) supervised fine-tuning performs prompt-driven tree search to obtain self-generated training data annotated with functional tokens, which warms up the model to learn these tokens for initial reasoning capability; and (2) online reinforcement learning further allows the model to explore diverse reasoning pathways through functional token sampling without relying on prompts, thereby facilitating effective self-improvement for functional reasoning. Extensive experiments demonstrate the superiority of the proposed RFTT on mathematical benchmarks and highlight its strong generalization capability to other general domains. Moreover, the performance of RFTT exhibits consistent gains with increased test-time computation through additional search rollouts. Our code and dataset are available at https://github.com/sastpg/RFTT.

Published as a conference paper at ICLR 2026
INCENTIVIZING LLM REASONING VIA REINFORCE- MENT LEARNING WITH FUNCTIONAL MONTE CARLO TREE SEARCH
Kongeheng Zhang', Qi Yao", Baisheng Lai Jiaxing Huang", Wenkai Fang", Dacheng Tao Mingli Song', Shunyu Liu
!Zhejiang University, 2 AAlibaba Cloud Computing, 3Generative Al Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798 zhangkc0z edu. cn,
inc. com, shunyu. 1iu. cs gma com
ABSTRACT
In this work, we propose Reinforced Functional Token Tuning (RFTT), novel rein- forced fine-tuning framework that empowers Large Language Models (LLMs) with learn-to-reason capabilities. Unlike prior prompt-driven reasoning efforts, RFTT embeds rich set of learnable functional tokens (e.g., <analyze>, <verifys fy>, <refine>) directly into the model vocabulary, enabling chain-of-thought con- struction with diverse human-like reasoning behaviors. Specifically, RFTT com- prises two phases: (1) supervised fine-tuni fine-tuning performs prompt-driven tree search to obtain reff-generate training data annotated with functional tokens, which warms up the model to learn these tokens for initial reasoning capability; and (2) online reinforcement learning further allows the model to explore diverse rea- soning pathways through functional token sampling without relying on prompts, thereby facilitating effective self-imprrvemmnt for functional reasoning. Extensive experiments demonstrate the superiority of the proposed RFTT on mathematical benchmarks and highlight its strong generalization capability to other general domains. Moreover, the performance of RFTT exhibits consistent gains with in- creased test-time computation through additional search rollouts. Our code and dataset are available at https: http//ggtttt///t/ com/ pastpg/RE" RTTT..
INTRODUCTION
Recent advances in Large Language Models (LILMs), particularly exemplified by OpenAT-ol (Jaech et al., 2024) and DeepSeek-RI (Guo et al., 2025), have demonstrated sophisticated reasoning ca- pabilities with remarkable success across various professional domains (Zhu et al., 2024; 2025; Tu et al., 2025), such as mathematical analysis (Hendrycks et al., b021b; Cobbe et al., 2021; He et al., 2024; Zhang et al., 2025b) and algorithmic programming (Chen et al., 2021b; Austin et al., 2021; Zhuo et al., 2024). Despite the encouraging results achieved, learning to reason remains crucial yet challenging task for LLMs as it necessitates high-quality reasoning data for training (Xu et al., 2025), especially for smaller L.LMs with 7B or 8B parameters (Guan al., 2025). Thus, previous attempts often rely on stronger models or human annotators to generate high-quality Chain-of-Thoughe (CoT) data (Min al., 2024b; Lightman et al., 2024: Huang et al., 2024; Wang al., 2024a); however, such approaches inevitably incur substantial costs (Wang et al., 2024b; Guan et al., 2025; Zhang et al., 2025c) and are prone limited scalability (Li al., 2023; Ahn et al., 2024).
To address these limitations, existing studies explore self-improvement mechanisms where LLMs can iteratively refine their reasoning capabilities through self generated rationales without training data (Hao et al., 2023; Zhang et al., 2024a; Wu et al., 2024; Zhang et al., 2024b; Yao et al., 2024; Fang et al., 2025). For instance, early techniques such as CoT prompting enable models to generate multi-step reasoning paths for unlabeled questions. followed by rejection sampling to select high-confidence solutions as training data (Yuan et al., 2023; Liu et al., 2024c; Brown et al., 2024). More recently, DeepScek-RI RI (Guo al., 2025) and its follow-up works (Zeng et al., 2025; Xie et al.,
Corresponding author.

---

Published as a conference paper at ICLR 2026
2025; Yu et al., 2025; Hu et al., 2025; Zhang et al., 2025a) have demonstrated that Reinforcement Learning (RL) offers a powerful paradigm for autonomously incentivizing LLMs with advanced reasoning capabilities, leading the spontancous emergence of complex "aha moment" behaviors.
However, some studies (Shah et al., 2025; Yue et al., 2025) have raised critical perspectives, suggesting that RL may not introduce fundamentally new reasoning capabilities, but rather increase the likelihood of sampling correct outputs that the initial model already knows. Moreover, RL enhances sampling efficiency at the cost of reducing reasoning diversity, potentially limiting the exploration capability of LLMs. This reveals fundamental dilemma: effective learning to reason necessitates both a strong initial reasoning capability and sustained exploratory diversity. Without sufficient initial competence. the model may struggle to generate informative feedback during self pelf- training. Furthermore, a lack of exploratory diversity can cause the model to converge prematurely on suboptimal reasoning patterns, hindering the discovery of more optimal reasoning paths.
In this work, we propose Reinforced Functional Token Tuning (RFTT), a novel reinforced fine-tuning framework that introduces learnable functional tokens (e-g., <analyze>, <verify>, <refine>) into the model vocabulary to facilitate self-training learn-to-reason. Technically, RFTT operates in two phases: (1) During the Supervised Fine-Tuning (SFT) warmup phase, RFTT employs functional prompt-guided Monte Carlo Tree Search (MCTS) to construct reasoning trees, where both correct and incorrect solution paths merged to synthesize human-like reasoning paths. These self-generated paths explicitly connect reasoning nodes through functional tokens, thereby forming structured training data for learning to reason with these tokens, which equips the model with inítial reasoning capability. (2) In the online RL phase, the model transitions from prompt- prompt-guided to token- guided reasoning by sampling functional tokens to autonomously expand various reasoning nodes. Through diverse exploration of reasoning trees, the model reinforces high-value reasoning paths, ultimately achieving autonomous self eelf-improvement of functional reasoning capabilities.
Our core contributions are summarized as follows:
We propose a new learn-to-reasee paradigm that pioneers the integration of learnable functional tokens into the model vocabulary, enabling the model to establish internalized loken-guides reasoning patterns rather than relying on external prompt-guided constraints.
We devise RFTT, a novel reinforced fine-tuning framework for learning to reason. The SFT phase bootstraps initial reasoning through self-generaten training data annotated with functional tokens, while the RL phase enables diverse exploration of reasoning paths through token guided tree traversal, achieving self-improvemment for functional reasoning.
Extensive experiments conducted on various mathematical benchmarks demonstrate that RFTT yields sigmificantly superior results to state-of-the-ar counterparts. Notably, the performance of RFTT continues to improve as the number of search rollouts increases during inference.
2 RELATED WORKS
Learn-to-Reason The "learn-to-reason to-reaso"" paradigm aims to endow LLMs with complex reasoning ca- pabilities through dedicated training. Due the scarcity of stepwise annotated reasoning trajectories, previous methods have primarily focused on two strategies: synthesizing long-CoT (Wei et al., 2022; Kojima et al., 2022) trajectories for supervised fine-tuning (Trung et al., 2024; Zhang et al., 2024a; Guan et al., 2025; Min et al., 2024a), and generating reasoning preference pairs for preference opti- mization (Rafailov et al., 2023; Lai et al., 2024; Liu et al., 2025). More recently, DeepSeek-RI (Guo et al., 2025) demonstrated that reinforcement learning with verifiable rewards (Lambert et al., 2024) (RLVR) can elicit advanced reasoning abilities without curated trajectories, similar to traditional reinforcement learning (Jiang et al., 2021; 2022; Liu et al., 2023; 2024b). However, subsequent studies (Shah et al., 2025: Yue et al., 2025) indicate that RLVR may degrade the exploration capability and reduce the reasoning diversity of LLMs. In RFTT, mitigate this issue by employing functional tree search to enhance the exploration of the solution space.
Tree-Search Reasoning, While most LLMs have typically adopted auto-regressive reasoning manner, there has been a trend to engage in more complicated reasoning architectures like trees (Wang et al.. 2023b;a; Koh et al., 2024; Zhang et al., 2024c: Ding et al., 2025). Recently, various methods for exploring tree structures have been devised to identify optimal reasoning trajectories. e.g., tree of thought (Yao et al., 2023), graph of thoughts (Besta et al., 2024), and Monte Carlo tree search (Hao
2

---

Published as a conference paper at ICLR 2026
Fanctional Tree Search
Supersised Fine- Tuning
Question (x). What is the smallest odd number with four different prine factors?
Fumctional Tokens in RL
Dynamk Sampling
Quecken
Insial Polley
Werma
Jsbeustion
Reasoning Path (y):
SFF Tless
Reisoring Patlo
sclerify-Well, we are asked find snallest ood rumber that has four distinct prine factors, </dlnnfye
Pollcy LLM
Hios srower 94
Reinforcement Learning
veriy
refine
roubquest Firsirst should iduntify the snallest odd prime numbers s/subquestiion
Paleyy
Action Probebility
snallest odd number with four different prins foctors, we stort The four smallest odd prims rumbers 3,5, 7c/sset_steep
-next_ To
Surppi da
everifyo
MCTS Sampiing
Rowwrd e
recheck...
Figure I: A conceptual illustration of reasoning path generation based on functional tree search and our training framework. RFTT comprises two phases: supervised fine-tuning warmups the model with initial reasoning capability by functional token-annotated data, while online reinforcement learning allows the model to directly sample functional tokens from its vocabulary to autonomously expand reasoning trees for diverse exploration.
et al., 2023; Chen et al., 2024; Zhang et al., 2024a). Furthermore, several studies (Shen et al., 2025; Hou et al., 2025) also incorporate tree search for RL training. However, these direct tree-search methods struggle with limited exploration due to the homogencous reasoning paths and the large search space. Although rStar (Qi et al., 2024) utilizes functional prompts in tree search to diversify and constrain the search process, they only act as external constraints in the inference phase while we introduce learnable functional tokens during model training facilitate effective and efficient exploration, thus improving the model's intrinsic reasoning ability.
3 METHODOLOGY
In this section, we first provide the problem formulation. Then we further detail the proposed two phase leam-to-reason framework of RFTT. The whole procedure is outlined in Figure 1.
3.1 PROBLEM FORMULATION
We consider a complex reasoning task as a multi-step reasoning generation problem, which de- composes the problem into sequence of simpler intermediate steps through structured reasoning operations. Assuming problem requires T intermediate reasoning steps, the reasoning path can be denoted as r = (No, 81, 82, ]: where 8o x and 8 is the 1-th intermediate reasoning steps. Furthermore, we model this process as a tree search, where the root node represents the problem æ, the edges denote the action space A (which can be special tokens or prompts), and children nodes are next steps generated by a policy model Ta with parameter 0 under corresponding actions. Each distinct path from the root to the terminal node denotes a sequence of decision- decision-making steps that constitutes potential solution trajectory т. For any given problem æ, the subset of its solution space T = {Tr ....... TN) } can be extracted from the tree structure. Our aim is to sample diverse reasoning paths and ultimately derive optimal solutions.
Considering the problem setup above, we can transfer the task of better reasoning to better selecting actions based on previous steps. In this work, we focus on how to inspire intrinsic reasoning abilities with functional tree search. The conventional natural language generation task often regards vocabulary-Ievel token sampling as actions. If we adopt vocabulary-Ievel actions to conduct tree search on LLMs, the action space can be extremely large, leading to a sharp increase in computational complexity and reduced search efficiency. Thus, we design series of human-like reasoning behaviors as the action space of the L.M policy. During the preliminary warmup phase for SFT, we employ functional prompts as the action space to induce the model for tree node generation with different semantics, while transitioning to functional tokens for exploratory tree search during RL optimization.
3.2 FUNCTIONAL MONTE CARLO TREE SEARCH FOR DATA GENERATION
The trajectories generated by direct tree search methods are often homogeneous due to the inherent preference for syntactic patterns in L.MSs. To enrich LLMs with sophisticated reasoning capabili-
3

---

Published as a conference paper at ICLR 2026
ties, first analyze the reasoning behaviors of leading LRMs. Inspired by recent work (Gandhi et al., 2025; Luo et al., 2025), we categorize several functional tokens as actions to emulate how hu- mans engage in complex reasoning: <clarify> (a1). <analysis> (a2), <subquest inn> (aa), <next_stee>> (04), <di rect_eensseee (as), <verify (as), <ref trefine> (07) and <output>> (as). We then introduce MCTS guided by functional prompts as actions before the SFT phase to obtain diverse reasoning nodes. Through iterative MCTS rollouts, we can extract multiple reasoning trajectories
{TI, 72...., TN3 for each question from tree structure. More details of MCTS procedures and functional token descriptions with corresponding functional prompts are presented in Appendix D and Appendix E. respectively. The following steps describe how we utilize these trajectories to construct structured human-like reasoning paths for SFT.
To construct a com- plex reasoning structure that incorporates self-verification for SFT, we generate an ex- tra node via cross verification between two trajectories and thereby seamlessly con- catenate them. Given a question and its ground truth y, we can obtain a set of can- didate reasoning trajectories T stated before. We first choose a correct trajec- tory with the highest average process re- wards given by a process reward model Te arg max R(T:), where Te is the set of correct trajectories and R(T,) repre- sents the average rewards of all intermedi- ate steps trajectory T₁. We then identify subset To T consisting of incorrect trajectories that intersect with Te- From Two extract the reasoning path with the lowest average reward: Tu = arg min R(T,). If Tw is non-empty, we define the overlapping nodes between To and Tw as r+ = Te n Tw. and the distinct nodes of them Tet
Cross Verification.
Cross Verification
Functional Prompt
Compared similar correct steps before, where does the current step go wrong?
L.
@N a4
Intersection
Q4
a5
Reasoning Paths with Functional Tokens
Brunch Merging
Verifieation
<ap-clarify ".."
Saz ang-analysis.. slag"
rursuuuusstion ..."/
sapree step .. ...s/ay"
@N
Sagrdiiect
...s
Backtrace
Saiiverity ... "las
aapreefine .la"""""
Intersection
cashoupput "ing""
An illustration of cross verification and branch merging for SFT data construction.
Figure 2:
T and T Tu r+ Through this approach, we can yield a node S, ~ kg (: Tr T {p}) that verify why an error occurs in Tn using a prompt p "Compared to similar correct steps before, where does the current step go wrong?".
We adopt branch merging to form structured reasoning paths that feature self- verification and self-correction: Ty -+ U T U {s,} UT,+. Finally, concatenate intermediate reasoning steps in trajectories by functional tokens, generating a trainable reasoning path DSFT [Qaery (<a>>8/<0o>>) for supervised fine-tuning, where denotes the string concatenation operation, is the intermediate reasoning step, and a; is the corresponding functional token.
Branch Merging,
3.3 REINFORCED FUNCTIONAL TOKEN TUNING
Our primary goal is empower LLMs with self-training learn-to-reason capabilities. To achieve this, we introduce two-phase training strategy: the SFT phase learns functional tokens for initial reasoning capability by self-generated training data annotated with these tokens, while the RL phase further enables autonomous diverse exploration of reasoning paths through functional token-guided tree search, thereby reinforcing high-value reasoning paths.
3.3. SUPERVISED FINE-TUNING
In this stage, an initial base model To is warmuped up on the dataset DSFT that consists of the tuples of "(question, reasoning path)" to obtain the SFT model TSFT. Specifically, we treat functional tokens as new special tokens that can be embedded directly into the model vocabulary. During the SFT phase, the model learns to reason with these functional tokens without relying on prompts.
3.3.2 REINFORCEMENT LEARNING WITH FUNCTIONAL TREE SEARCH
After being trained on the data with functional tokens, the policy model Te can explicitly and dynamically select functional tokens and further generate the specific reasoning steps associated with these tokens. This enables the transition from prompt-guided to token- guided reasoning just by sampling functional tokens. Through this design, the policy model can reinforce the use of key
4

---

Published as a conference paper at ICLR 2026
functional token combinations via form of online tree search, significantly improving its learning efficiency. Specifically, the policy model learns by iteratively searching, evaluating the process in each rollout, and updating its parameters, illustrated in Figure 1.
Like AlphaZero (Silver et al., 2018), the functional token is sampled based on the log- oolliiklihood scores if there exist unexplored actions, otherwise select them according the UCT score:
arg (=x)) (o(a)soa), U(x)) + 0, UCT(s,, a) arg max)e (C)(s;;a), otherwise,
Q(se, a) N(s,,a)
(s,))
N(s,,a)
where (s+) denotes unexplored functional tokens of intermediate reasoning step 8t, and 50.r repre- sents all intermediate steps from timestep 0 to t. In UCT score, (s²) denotes the number of times step 8t has been visited previous iterations, N (s; a) denotes the number of times action a has been selected after step 8t, (x., a) is the cumulative reward value of selecting action after step 8t obtained from previous simulations, and e a constant that balances exploitation and exploration.
Furthermore, the reward function based on the rule-verifies answer extractor ANS(-) (-) is defined as:
(50: Oes St+1) (0:r; ar, 8t+1) KL((),
ANS( (8t-1) (NS(s 1) = y/ 0.1, ANS(ANSSSSS)) st-1) * mnnll. 7 y, KL(t)= log ANS(#:-1) = = null,
To (0:s))
RM(Su:;, at, 8(+1)
TTSF (8o:t)
where the SFT model TSFT serves as the reference model, B is KL. coefficient, and a is the process reward. We can use the Process Reward Model (PRM) to obtain the value of a. Note that will be assigned a value
if only outcome rewards are used.
During online RL, the model parameters are optimized using the Reinforce++ algorithm (Hu, 2025). The core policy objective combines clipped updates with normalized advantage values through the following loss funetion:
(o) = EE min [T(0e((0:2) Ae, clip
Ie [ae (ae(s0:))
e, + A.)
TRau (ax) (0.a)
Th (at s0:e)
where Tee is the previous policy model, e is the clipping coefficient, and the normalized advantage value A, is calculated based on the reward Rt.
Overall, the transition from prompt-guided to token-guided reasoning enables the effective and efficient exploration of the reasoning paths during the RL phase. The policy model learns to reinforce high-value reasoning paths through functional token-guided tree search and achieves self-improvement for functional reasoning.
3.4 EFFECTIVENESS ANALYSIS
We briefly provide insights into why RFTT can outperform current reinforced fine-tuning methods.
High entropy exploration. Diverging from independent rollouts like GRPO, our method imple- ments tree-structured branching strategy at each node guided by the probability of functional tokens. A low-entropy node over functional tokens (probability concentrates on one action) sup- presses branching because alternatives are rarely selected. Therefore, this mechanism adaptively allocates exploration resources to paths in the reasoming space that have increasing entropy, thus promoting a broader exploration of diverse reasoning patterns.
Distinct advantage attribution. Our functional tree search sampling produces a large number of trajectories with shared tokens but divergent branches. Through implicit advantage attribution, the RL algorithm enables the model to effectively internalize the distinctions among different reasoning behaviors. Since shared tokens receive the same aggregated advantage signals from the entire group, while branching tokens obtain distinct advantages specific to their exploration paths, they contribute differently to the final optimization objective.
Consequently, the optimization process improves exploration efficiency while facilitating precise learning at critical reasoning steps in different states.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
To demonstrate the effectiveness of the proposed RFTT method, we conduct experiments on multiple mathematical and general reasoning benchmarks. Our evaluation seeks to answer the following questions: (1) Does RFTT outperform existing reinforcement fine-tuning baselines? (Section 4.3 and Appendix B) (2) How do the configurations of functional tokens and different training/rollout strategies affect the performance? (Section 4.4) (3) Can RFTT expand the exploration boundaries to a higher capacity ceiling than the initial model? (Section 4.5)
4.1 EXPERIMENTAL SETUPS
Datasets. For training, we only adopt the training set of the MATH (Hendrycks et al., 2021b) dataset as our training data. For evaluation, we employ five established mathematical reasoning benchmarks: MATH-500 (Lightman et al., 2024), GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), Olympiad Bench (He et al., 2024), and AMC Moreover, we evaluate on general reasoning datasets like MMLU-Pro (Wang al., 2024c), GPQA (Rein et al., 2024), CommonsenseQA (Talmor et al., 2019), FOLIO (Han et al., 2024), TableBench (Wu et al., 2025), and CRUXEval (Gu et al., 2024) to validate the generalization of RFTT. To ensure evaluation efficiency and prevent data contamination, we use the MATH-500 for MATH evaluations while retaining full test sets for other datasets. More information about our training and evaluation dataset listed in Appendix C.
Base Models. RFTT is generally applicable to a wide range of LLMs. In our experiments, we select --A.MMA--.--88-nssreee (Dubey et al., 2024), Qwvnn-25577-Innsrruett (Yang al., 2024b), and Qwen-3-4B-BBl -ase (Yang et al., 2025) as backbones. By focusing on LLMs with relatively small parameters under 1OB, we expect that RFTT enables the smaller model learn reason via self-play, ultimately achieving results comparable to or exceeding larger LLMs in complex reasoning tasks.
Baselines. We compared RFTT against four categories of baselines in our experiments: (1) Superior LLMs, including leading closed-source models and open-source models; (2) Model fine-tuning methods, including SFT. ReFT (Trung et al., 2024), MCTS-DPO (Xie al., 2024), TreeRL (Hou et al., 2025) and GRPO (Shao et al., 2024) with Deepseek-RI prompt template; (3) In-context learning methods, including Few-shot CoT and CoT+SC@ 4. (4) Tree search methods, including ResT-MCTS* (Zhang al., 2024a), rStar (Qi et al., 2024) and LLaMA-Berry -Brry (Zhang al., 2024c).
Evaluation Metric. We employ Pass@ accuracy across all benchmark datasets as our evaluation metric, with correctness determined through comparison between generated outputs and ground truth. To extract answers reliably, we require the model to wrap its final answer in boxed }
4.2 IMPLEMENTATION DETAILS
All experiments 8x 5880-880GB GPUs While performing MCTS to collect diverse reasoning paths via various functional prompts, we use Qwen-225-7B-Innstruc as the generator and math- sespherd-mistral----ooe (Wang et al., 2024b) as the process reward model. Note that for the Qwen experiment, the model was trained on its own generated data. For convenience, we used this data directly other experiments without regenerating it. We present the performance results after both SFT and RL, and the findings are consistent. We set the maximum search depth to 15 and perform 16 rollouts per question. For sampling each step, we employ the vL.LM engine (Kwon et al., 2023) to accelerate with the temperature set to 0.9, top p set to 0.8, and max tokens set to 1024. We spend approximately one day searching through 1.2k questions using 64 concurrent processes, and then gather 1k SFT data as described in Section 3.2.
We use LLaMA-Factory (Zheng et al., 2024) and OpenRLHF (Hu et al., 2024) for SFT and RL, respectively. During SFT. the training batch size is 128, the learning rate is 7 and the cutoff length 8192. We train the initial model for 10 epochs on about Ik CoT data with functional tokens and select the checkpoint with the best performance as the initial policy model for RL. During RL, implement functional token-guided MCTS to sample different reasoning paths for question. After that, we can choose to score the intermediate reasoning nodes in the tree. At each training step, the model searches for 16 reasoning paths per question with batch of 16 distinct questions. The learning rate for the policy model is 5e-7, 7, the temperature is 0.95, and the KL coefficient is 0.01.

---

Published as a conference paper at ICLR 2026
Table 1: Accuracy of our proposed RFTT and baselines across different mathematical reasoning benchmarks. The best results in each box are highlighted in bold. The proposed RFTT significantly boosts the performance of smaller LLMs across all datasets.
In-Domain
Close-Domain
Model
Setting
AVERAGE
MATH-300
GSMSK
SVAMP
Olympiad Bench
AMC
Superior
GPT-4o
Closed-source
76.6
96.1
93.8
43.3
47.5
71.46
GPT-4o mini ol-preview
Closed-source
70.2
93.2
89.2
35.8
55.0
68.66
Closed-source
85.8
93.0
2wen-2.5-14B-Imnse
Open-source 14B
80.0
93.2
39.4
52.5
71.34
Small LLMs
Zem-shot CoT
58.6
80.6
78.9
31.4
50.0
59.90
10.00
Few-shot CoT
64.8
87.8
80.6
35.3
$7.5
65.20
15.30
CoT+SC@44
67.0
ReFT
79.2
94.4
91.7
45.8
70.0
76.22
116.32
GRPO
75.6
92.7
88.4
42.9
67.5
73.42
113.52
Qwen-3-4B..n Bren-3-4B-Rasee
MCTS-DPO
73.8
70.12
110.22
TreeRL.
82.2
95.3
92.8
46.5
70.0
77.36
117.46
SFT Warmup (Ours)
68.2
89.5
85.6
38.9
62.5
68 94
19.04
RFTT (Ours)
83.4
48.1
79.46
19..56
Zero-shot CoT
72,0
10.00
Fow-shor CoT
75.6
91.6
90.2
36.5
52.5
69.28
12.94
CoT+SC@4
76.4
92.1
90.6
38.2
50.0
69.46
13.12
RcFT
75.8
38.1
71,46
15.12
GRPO
76.2
93.7
90.6
36.3
$5.0
70.36
14.02
Qwen-2.5-7B-Instruct
MCTS-DPO
74.6
37.2
69.82
13.48
TreeRI.
78.4
94.8
92.3
39.6
62.5
73.52
17.18
SFT Warmup (Ours)
73.2
91.9
87.4
36.8
60.0
69,86
13.52
RFTT (Ours)
79.8
95.2
93.0
40.3
70.0
75.66
19.32
Zero-shot CoT
50.6
84.5
78.2
18.6
17.5
49.88
10.00
Fow-shot CoT
53.0
82.0
20.0
20.0
$2.20
t2.32
CoT+SC@4
52.2
86.0
85.6
19.7
25.0
53,70
12.82
ReFT
55.0
24.9
42.5
59,50
19.62
LLaMA-3.1-B--mass
GRPO
54.0
83.4
24.0
22.5
54.42
14.54
MCTS-DPO
53.6
87.4
82.7
23.1
25.0
54,36
14.48
TreeRI,
57.8
62.14
112.26
SET Warmup (Ours)
53.2
86.4
82.9
24.6
$7.40
17.52
RFTT (Ours)
60.2
91.9
87.5
29.8
55.0
64.88
115.0
Table 2: Performance of RFTT on out-of-domain benchmarks. Despite being trained only math datasets, RFTT exhibits strong transferability.
Model
Setting
MMLLU-Pro
GPQA
Commonsense(A
FOLIO
TableBench
CRUXEval
Few-shot
Qwen-2.5-7B-insttt
RFTT (Ours)
57.2
35.6
82.8
73.9
44.4
$7.5
Few-shor CoT
48.3
32.8
76.9
64.5
32.8
38.6
RFTT (Ours)
50.8
33.4
78.1
65.6
34,2
39.5
4.3
MAIN RESULTS
Results Different Reasoning Benchmarks. We conduct comprehensive evaluation of the effectiveness of RFTT across different mathematical benchmarks. Table presents a comparative
analysis between our framework and state-of-the-art baselines. We highlight three key findings: (1) RFTT can significantly improve the complex reasoning proficiency of small-parameter LLMs. For example, LLaMA-3.1 --s--uuttttt initially achieves 50.6% accuracy on the MATH-500 benchmark using the CoT prompting technique. However, after training by RFTT, its Pass accuracy improves 60.2%, surpassing self-consistency sampling (CoT+SC@4) Similarly, 2. RFIT achieves performance on par with Qwen-2.5-1493 1---trrtttttt (79.6% vs. 80% on MATH-500), indicating that smaller models have developed robust reasoning capabilities without any reasoning prompts. (2) RFTT outperforms ReFT various mathematical reasoning tasks. We observe that RFTT achieves consistent performance improvements over ReFT, notably surpassing it by an average margin of 5% across different benchmark evaluations. This result demonstrates the robustness of RFTT and the enormous potential for further self-training. (3) RFTT has also shown strong generalization ability other challenging reasoning problems, including AMC, Olympiad Bench,
with
7

---

Published as a conference paper at ICLR 2026
LLaMA3.1-988 1-nntrrre-
Qwan2.5-77 Snn-ntteee
AlMC (26)
60
MATHAre (%6)
80-
MATH (86)
30-
70-
AMC
70-
40-
60
50
60-
RPTT (0I-MOTS w/c85
30
50
50
RPPT DEL 2014
50
a1 Pamire
20
40
12
16
40
12
20
* RRoloout Number
Ralloout Number
# Rollout Number
RRllout Number
Figure Performance gains under scaling up the inference-tine time computation.
GPQA, and MMLU-Pro. As mentioned in Section 4.1, our training dataset only consists of the training data from MATH, and there may be a risk of over-optinization on well-known testing benchmarks such as MATH-500 and GSM8K. Our results in Table 2 reveal that RFTT remains effective even when applied to unseen general problem sets. More results can be found in Appendix B.
Scaling Inference- Time Computation. We integrate Iearnable functional tokens into the model vocabulary in RFTT, enabling the model to establish internalized token-guided tree search. We use MCTS to augment the policy model trained by RFTT without any guidance of reward mod- els. Specifically, we use the likelihood (confidence) of majority voting as the reward value in the MCTS simulation stage. Figure 3 shows the performance of scaling inference-time computation by comparing the accuracy of policy models with different training strategies across different rollout numbers. The accuracy one rollout corresponds the Pass@1 accuracy of the policy model. We highlight two key observations: (1) Scaling inference-ting computation enhances mathematical reasoning across all benchmarks, albeit with distinct trends observed. On MATH-500, the policy models demonstrate slow improvement after 16 rollouts, whereas on AMC, the accuracies continue to improve steadily. (2) With and 20 rollouts on MATH-500 and AMC benchmarks, respectively, the performance of the trained policy model exceeds ol oprrevie demonstrating its effectiveness.
Tree Search Comparison. Our pro- posed RFTT can autonomously se- lect the next high-value functional to-
Table 3: Comparison of different tree search methods.
Dataset
Method
Accuracy
Time per Question
ken based on partially generated reasoning trajectory, thus enabling a more efficient exploration of the rea- soning space without frequently inter- rupting the generation process. Here, we compare the accuracy and search efficiency of RFTT with other tree- based reasoning methods on LLaMA-
rStrr
92.1%
276
GSM8K
LLaMA-Bery
94.9%
339 s
Ours
95.2%
81 s
rStar
61.0%
526 s
MATH-500
LLaMA-Berry
69.4%
674 s
Ours
72.0%
131
Instruct. Specifically, we measure the average time needed for a single question and the proportion of correct answers under 16 rollouts across datasets. As shown in Table 3, our framework achieves higher accuracy while substantially reducing time complexity compared to other methods. It outperforms rStar, which relies extensive node expansion to select potentially better action, and L..MMABBrryy.. which generates a complete solution instead of a single step for each node.
4.4 ABLATION STUDIES
We first investigate our design of functional tokens. To further demonstrate the advantage of our method, we ablate core components of RFTT: (1) RFTT w/o SFT Warmup directly trains the original model using rule- based RL; (2) RFTT w/o MCTS employs random sampling and rule-barc based RL after SFT Warmup; (3) RFTT w/o PRM employs MCTS sampling and rule-based RI after SFT Warmup.
The necessity and generalization of functional tokens. We conduct an ablation study on MATH-500 using Qwen-2.5-77 In-trrcttt trained after RFTT by masking the possibility of a certain functional token. Results Table 4 highlight the complementary effectiveness all functional tokens. Notably, tokens like <verify> and <refine> have significant contribution, and their absence leads to performance degradation. Moreover, we generate responses on MATH-500 and prompt GPT-4o to classify each reasoning step by type. Figure 4 shows that the semantics of functional tokens align strongly with their intended behaviors. This supports our design of introducing eight numan-inspiree functional tokens, which serve compact yet expressive action space for reasoning exploration.
8

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study on different components RFTT.
Model
Setting
MATH-500
GSMBK
SVAMP
Olympiad Bench
AMC
AVERAGE
RFTT w/o SFT Warmup
74,8
91.2
90.6
37,9
52.5
69,40
Owen-2.5-7B-Inssie
RFTT w/o MCTS
75.2
93.8
91.2
39.0
62.5
72.34
RFIT w/o PRM
77.2
94.1
92.4
38.4
67.5
73,92
RFTI (Ours)
79.8
95.2
93.0
40.3
70.0
75.66
RFTT w/o SFT Warmup
54,8
84.9
83.7
24.0
17.5
53.00
LLaMA-3,1-8B-Innn
RFTT w/o MCTS
56.2
91.5
84.0
29.1
42.5
60.66
RFTT w/o PRM
57.4
92.2
86.3
28.6
50.0
62.90
RFTT (Ours)
60.2
91.9
87.5
29.8
55.0
64.88
We additionally perform an analysis using Deepseek-RI on 1. 1,000 randomly selected questions from MMLU-Pro (a dataset that spans STEM, social sciences, law, and health). The sampled reasoning trajectories are broken down into discrete steps using predefined rules (e.g., newline delimiters). We then employ GPT-4o assess whether each step could be mapped one of functional tokens. As presented in Table 5, approximately 98.4% 4% the steps aligned with the intended semantic coverage of our token set, which demonstrates the generation of functional tokens across diverse tasks.
Table 4: Ablation study Table 5: Coverage rate of on masking different functional tokens.
functional tokens on gen- eral domains.
94.4
4.3
0.3
1.0
0.0
0.0
0.0
0.0
Intended behavior
a2
5.2
2.6
0.0
800
-
0.0
1.3
93.8
4.7
0.1
0.1
0.0
0.0
Functional Tokens
Functional Coverage Tokens
Accuracy
Rate
14.5
1.9
69.
11.5
1.5
0.1
Porcorlage
2
0.0
022
0.2
12.2
80.9
1.0
4.3
w/o a
79.0%
token a
8.2%
w/o ag
77.6%
token ag 13.9%
88
0.0
0.3
2.3
0.3
92.1
5.3
0.0
w/o ag 79.4%
token 0s 4.3%
47
2.0
0.7
20
w/o
73.4%
token
28. 7%
%
0.0
0.0
0.8
0.2
2.6
0.0
0.0
96.4
w/o
75.0%
token Os 14.5%
a1
a3 +5
a6
28
w/o as 72.8%
token ds 13.8%
Semantics
w/o ay 72.6%
token a7 12.3%
Figure 4: The relationship between semantics
All (Ours) 79.8%
token as 2.7%
and behaviors of functional tokens.
The Effectiveness of SFT Warmup. The SFT phase embeds the functional tokens into the model vocabulary, enabling self-improvement with token-guided exploration during the RL phase. To verify effectiveness of our SFT phase, we conduct experiments to directly train the original model without SFT warmup using pure RL. As compared in the first two lines of Table even with the same random sampling method, the model warmuped with SFT (RFTT w/o MCTS) surpasses the pure RL model (RFTT w/o SFT warmup) by large margin (2.9% for Qwen-2.5-785 Instruuttt and 7.7% for LLaMA-3.1 1-8.--3.--8---ttttee much more effectively after SFT initialization.
indicating that the aaken-guided reasoning can explore the reasoning space
The Effectiveness of MCTS. Our functional token-driven MCTS enables a more effective exploration of the solution space in both inference and training. We conducted ablation study on MCTS sampling during the RL phase to investigate the influence of data diversity, as shown in Table The outcomes reveal that random sampling RL yields lower performance compared sampling with MCTS, thereby underscoring the significance of data diversity in learning a better policy. The results show a 1.5% performance gap between random sampling (mean 72.34) and MCTS-enhanced sampling (mean 73.92) after rule-based RI. across 5 benchmarks. Thus, we conclude that functional token-driven MCTS can explore diverse reasoning paths, providing superior foundation for RL.
The Effectiveness of PRM. The results in Table 6 demonstrate that the use of PRM in RFTT consistently outperforms that of rule-based RL (RFTT w/o PRM) by about 2% on both LLaMA and Qwen on average. This might benefit from fine-grained rewards for intermediate reasoning steps, which mitigate eиror accumulation in long reasoning paths. Therefore, the integration of PRM is valuable for reasoning where rulc-based rewards fail to capture intermediate reasoning quality.
4.5 DISCUSSTON
Training Dynamics. We demonstrate the training curve and Pass @ metric of Qwen-2.5-7B 2.55-7BB-BInttrrcc trained by RFTT on challenging AIME-24 benchmark in Figure 5. As depicted, the entropy exhibits

---

Published as a conference paper at ICLR 2026
Response Length
Entropy
AIME24 Pass@
6000
RFTT MCTS
0.7
RFTT wc METS
RFTT MCTS
Length (Token)
5000
RFTT MCTS
0.6
RFTT we MCTS
60
Initial
Accuracy
4000
Value
0.5
40
0.4
0.3
2000
20
www.MM MAN
0.2
1000
0 50
100 150 200 250
50
100
150
200
250
0
50
100
150 200 250
Training Steps
Training Steps
Number of samples
Figure 5: The training dynamics of RFTT on Qwen-255-78-Instruucc during RL..
trend of first increasing then decreasing, suggesting shift from diverse exploration at the early stage to the exploitation of high-quality reasoning paths in the later stage. The response length increases sharply within 50 training steps and then gradually increases when using PRM in RFTT. The searching time of the policy model undergoes continuous improvement during the RL training process, developing the capacity for exploration and optimization with appropriate incentives. We observe that as the search depth increases, the policy reinforces predefined human- like reasoning behavior, including the self-correction mechanism and the exploration of diverse problem-solving solutions, thereby evolving to tackle challenging tasks. Besides, the model trained after RFTT achieves a higher pass @ l score than the initial model when is large, which proves the effectiveness of our approach in promoting the model's exploration capability, achieving higher capability ceiling finally. Please refer to Appendix G for more discussions.
Computational Costs. Detailed infor- mation regarding the computational resources for our method is provided in Table 7. The integration of Monte Carlo Tree Search (MCTS) into the Reinforcement Learning (RL) stage results in about four-fold increase in training time, despite a reduction in the number of generated tokens (due to shared prefixes) per training step. This overhead is attributed to the MCTS implemen- tation, wherein the exploration of multiple search branches disrupts the batched and parallel nature of the standard sampling process. We anticipate that future optimizations to the MCTS sampling algorithm could mitigate this computational burden.
Table 7: Comparison of computational cost.
Method
Phase
Time (min)
Token Usage (k)
Data Construction
SEF
min question
rollout
RETT w/o MCTS
RL
min step
$11.4 step
RFTT (Ours)
RL
12.1 min / step 3.8. 216.8 / stepp 4
CONCLUSIONS
This work proposes RFTT, a reinforced fine-tuning framework that equips LLMs with advanced reasoning capabilities. By embedding learnable functional tokens into the model vocabulary, RFTT enables LLMs to internalize human-like reasoning behaviors, eliminating reliance on external prompts. Our approach advances the learning paradigm by bridging functional token-guided reasoning capa- bility with diverse exploration during model training, offering a promising direction for developing assurce--ffficient generalizable reasoning capabilities in smaller LLMs. Extensive experiments vali- date the effectivennss of RFTT, achieving significant performance gains and demonstrating scalability with inference-time search. Currently, our work is primarily training on mathematical reasoning tasks; its effectiveness on broader reasoning domains needs further exploration.
ACKNOWLEDGEMENT
This work is supported in part by the Hangzhou Joint Funds of the Zhejiang Provincial Natural Science Foundation of China under Grant No. LHZSD24F020001, in part by the Zhejiang Province High-Level Talents Special Support Program "Leading Talent of Technological Innovation of Ten- Thousands Talents Program" under Grant No. 2022R52046. in part by the Fundamental Research Funds for the Central Universities under Grant No. 2021FZZX001 -23, and in part by the advanced computing resources provided by the Supercomputing Center Hangzhou City University. Dr Tao's research is partially supported by NTU RSR and Start Up Grants.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper presents work whose goal is to advance the field of machine learning, enhancing the reasoning capabilities of LLMs. There are many potential societal consequences of our work, none of which we feel will raise potential violations of the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we present the detailed experimental setup, including hardware, time, and hyperparameters the MCTS procedures and two-phase training procedures Section 4.2. The algorithm description in Section 3.2, Section 3.3, and Appendix D is provided in detail to ensure clarity and repeatability. The relevant code and dataset used training are made public in this url.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Gpt-4 technical report. arXiv preprint arXiv:2303.08771 2023.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In Conference of the European Chapter of the Association for Computational Linguistics, 2024.
AI Anthropic. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 3(6), 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint arXi::2108.0773 2021.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In AAAI Conference on Artificial Intelligence, 2024.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:240..21787, 2024.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process supervision without process. In Conference on Neural Information Processing Systems, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, 1lya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 2rXiv:2107.03374 2021b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10. 14168, 2021.

---

Published as a conference paper at ICLR 2026
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.226117 2025.
Yifu Ding, Wentao Jiang, Shunyu Liu, Yongcheng, Jing, Jinyang Guo, Yingjie Wang, Jing Zhang, Zengmao Wang. Ziwei Liu, Bo Du, et Dynamic parallel tree search for efficient 11m reasoning. arXiv preprint arXiv:2502.16235 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21785 2024.
Wenkai Fang, Shumyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Mingli Song, and Dacheng Tao. Serl: Self-play reinforcement learning for large language models with limited data. In Advances in Neural Information Processing Systems, 2025.
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree- search can guide large language model decoding and training. arXiv preprint arXiv:2309.17199 2023.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.00 20000003333333 30707, 2025.
Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In International Conference on Machine Learning, 2024.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking, arXiv preprint arXiv:2501.045 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szab6, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev. FOLIO; natural language reasoning with first-order logic. Proceedings of Empirical Methods in Natural Laguugge Processing, pp. 22017-22031, 2024.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang. and Zhiting Hu. Reasoning with language model is planning with world model. In Conference on Empirical Methods in Natural Language Processing, 2023.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Annual Meeting of the Association for Computational Linguistics, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Conference Neural Information Processing Systems, 2021b.
12

---

Published as a conference paper at ICLR 2026
Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. Treerl: LLM reinforcement learning with on on-policy tree search. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics, pp. 12355-12369) 2025.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint urXiv:2501.03262. 2025.
Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An easy-to-use, scalable and perfoorerree rihf framework. arXiv preprint arXiv:2405.1114: 2024.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zere An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXív:2503 24290, 2025.
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. 01 replication journey-part 2: Surpassing ol -preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16453 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16770, 2024.
Haobo Jiang, Jin Xie, and Jian Yang. Action candidate based clipped double q-learning for discrete and continuous action tasks. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 7979-7986, 2021.
Haobo Jiang, Guangyu Li, Jin Xie, and Jian Yang. Action candidate driven clipped double q-learning for discrete and continuous action tasks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv preprint arXiv:2407.0/44 074466, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Conference on Neural Information Processing Systems, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In ACM Symposium on Operating Systems Principles, 2023. 2023..
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng. and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. arXiv preprint 2rXiv:2406.1862, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T)"" ulu 3: Pushing frontiers in open language model post-training. arXiv preprini 24X1:2411.151222 2024.
Minzhi Li, Taiwei Shi, Caleb Ziems, Min- Kan, Nancy F. Chen, Zhengyuan Liu, and Diyi Yang. Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. In Conference on Empirical Methods in Natural Language Processing, 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. International Conference Learning Representations, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Shunyu Liu, Yihe Zhou, Jie Song, Tongya Zheng, Kaixuan Chen, Tongtian Zhu, Zunlei Feng, and Mingli Song. Contrastive identity- aware learning for multi-agent value decomposition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 1 11595-11603 2023.
13

---

Published as a conference paper at ICLR 2026
Shunyu Liu, Jie Song, Yihe Zhou, Na Yu, Kaixuan Chen, Zunlei Feng, and Mingli Song. Interaction pattern disentangling for multi-agent reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (6:8157-81122 2024b.
Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song. and Dacheng Tao. A survey of direct preference optimization. arXiv preprint arXiv:2503. 1701, 2025.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In International Conference on Learning Representations, 2024c.
Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang. GengRu Chen, Wenbo Su, and Bo Zheng. Deconstructing long chain-of-thouse A structured reasoning optimization framework for long cot distillation. arXiv preprint arXiv:2503. 16385, 2025.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09411 2024a.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang. Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report slow-thinking reasoning systems. arXiv preprint arXiv:2412.09411 2024b.
Sungjin Park, Xiao Liu, Yeyun Gong, and Edward Choi. Ensembling large language models with process seward-guided tree search for better complex reasoning. arXiv preprint arXiv:2412.1579, 2024.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In North American Chapter of the Association for Computational Linguistics, pp. 2080-2094, 2021.
Avinash Patil. Advancing reasoning in large language models: Promising methods and approaches. arXiv preprint arXiv:2410.01560, 2025.
Zhenting Qi. Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang. and Mao Yang. Mutual reasoning makes smaller Iims stronger problem-solvers. arXiv preprint arXiv:2408.061199 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Conference Neural Information Processing Systems, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proc q&a benchmark. In First Conference on Language Modeling, 2024.
Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre- training, arXiv preprint arXiv:2504.04000 04022, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang- Wei Hong, Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with chain-of- tction-thought enhances Ilm reasoning via autoregressive search. arXiv preprint ar arXiv:2502.02500 2025.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
14

---

Published as a conference paper at ICLR 2026
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-11 44, 2018.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. In North American Chapter of the Association for Computational Linguistics, 2019.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with Ilms. arXiv preprint 2al::2501./2999 2025.
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self-improvement of Ilms via imagination, searching, and criticizing. In Conference on Neural Information Processing Systems, 2024.
Luong Quoc Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. In Annual Meeting of the Association for Computational Linguistics, 2024.
Rong-Cheng Cheng Tu, Zhao Jin, Jingyi Liao, Xiao Luo, Yingjie Wang, Li Shen, and Dacheng Tao. Mllm- guided vlm fine-tuning with joint inference for zero-shot composed image retrieval. arXiv preprint arXiv:2505.19700 2025.
Ke Wang. Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi. Renrui Zhang, Lingi Song. Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. In International Conference on Learning Representations, 2024a.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Wei Lee, and Ec- Se-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Annual Meeting of the Association for Computational Linguistics, 2023a
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. sheh-shepherd: Verify and reinforce lims step-by-step without human annotations. In Annual Meeting of the Association for Computational Linguistics, 2024b.
Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessan- dro Sordoni. Guiding language model reasoning with planning tokens. arXiv preprint arXiv:2310.05707 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023c.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Conference Neural Information Processing Systems, 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Conference on Neural Information Processing Systems, 2022.
Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xeron Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Tongliang Li, Zhoujun Li, and Guanglin Niu. Tablebench: A comprehensive and complex benchmark for table question answering. In Association for the Advancement of Artificial Intelligence, pp. 25497-25506, 2025.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.0067 2024.
15

---

Published as a conference paper at ICLR 2026
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong. Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, 2025.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan. Timothy P Lillicrap, Kenji Kawaguchi. and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning, arXiv preprint arXiv:24005. 00455,, 2024.
Fengli Xu, Qianyue Hao, Zefang Zong. Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprini 1xXi:2501.00668 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1rx15:2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 2024b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025.
Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang. Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng. Li Shen, and Dacheng Tao. Mulberry: Empowering milm with ol-like reasoning and reflection via collective monte carlo tree search. arXiv preprint 24X.v22412.18319, 2024.
Jian Yao, Ran Cheng, Xingyu Wu, Jibin Wu, and Kay Chen Tan. Diversity-aware policy optimization for large language model reasoning. arXiv preprint arXiv:2505.2343: 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Conference on Neural Information Processing Systems, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint ar arXiv:2503. 14476, 2025.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308. 01825, 2023.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint ar arXi::2504. (tt.:22504.3337.,, 2025.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: LLM self-training via process reward guided tree search. In Conference Neural Information Processing Systems, 2024a.
Di Zhang, Jiatong Li, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b. arXiv preprint wwXiv:2406.07334 2024b.
Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yugiang Li, Wanli Ouyang, and Dongzhan Zhou. Llama-berry: Pairwise optimiza- tion for ol-like olympiad-level mathematical reasoning, 2024c.
16

---

Published as a conference paper at ICLR 2026
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937, 2025a.
Junjie Zhang, Guozheng Ma, Shunyu Liu, Haoyu Wang, Jiaxing Huang, Ting-En Lin, Fei Huang, Yongbin Li, and Dacheng Tao. A simple" motivation" can enhance reinforcement finetuning of large reasoning models. arXiv preprint arXiv:2506. 2025b.
Xi Zhang, Xiaolin Wu, Jiamang Wang, and Weisi Lin. Learning grouped lattice vector quantizers for low-bit IIm compression. arXiv preprint arXiv:2510.20984 2025c.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint 2rXiv:2403.13377 2024.
Hanwei Zhu, Haoning Wu, Yixuan Li, Zicheng Zhang, Baoliang Chen, Lingyu Zhu, Yuming Fang. Guangtao Zhai, Weisi Lin, and Shiqi Wang. Adaptive image quality assessment via teaching large multimodal model to compare. In Advances in Neural Information Processing Systems, pp. 3261 32611-32629, 2024.
Hanwei Zhu, Yu Tian, Keyan Ding, Baoliang Chen, Bolin Chen, Shiqi Wang, and Weisi Lin. Agenticiqa: An agentic framework for adaptive and interpretable image quality assessment. arXiv preprint ar arXiv:2509.20 26006, 2025.
Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 2024.
17

---

Published as a conference paper at ICLR 2026
Appendix
TABLE OF CONTENTS
More Discussion of Related Works
19
Additional Results
20
C Experimental Details
21
MCTS Procedure
22
Functional Tokens with Prompts
23
F Reasoning Cases
25
Further Discussion and Limitations
28
G.1 How do Functional Tokens Benefit?
28
G.2 Is SFT Warmup with Functional Tokens Needed?
28
G.3 Discussion of Response Lengths.
29
G.4 Limitations
29
18

---

Published as a conference paper at ICLR 2026
A MORE DISCUSSION OF RELATED WORKS
Drawing inspiration from AlphaGo (Silver al., 2016), advanced self-play methods further integrate Monte Carlo Tree Search (MCTS) to explore high-quality reasoning paths by constructing reasoning trees (Hao et al., 2023: Zhang et al., 2024a;b; Guan et al., 2025). However, these direct tree-search approaches face two fundamental challenges: (1) LLMs tend to generate homogeneous reasoning paths due to their inherent preference for syntactic patterns established during training (Wei et al., 2022; Wang et al., 2023c; Patil, 2025); (2) the high- -igh-dimensional search space further limits the exploration capabilities of LLMs, as the combinatorial vocabulary space leads to an exponential growth in candidate reasoning paths (Zhang et al., 2024a).
recent work rStar (Qi et al., 2024) introduces functional prompts (e.g., "propose a one-step thought", "rephrase the question") that steer tree search by simulating human-like reasoning behaviors, thereby diversifying node exploration while constraining the search space. However, rStar operates purely as inference-time augmentation without internalizing the reasoning capabilities through model training. The functional prompts act as external constraints rather than learned patterns. As a result, each inference of rStar must rely iterative tree-search traversal across different functional prompts to seek correct solutions. Compared to direct reasoning methods (which directly generate CoT responses without relying on search), this exhaustive searching approach results in substantial number of redundant LLM inference calls.
More recently, several studies (Hou et al., 2025; Shen al., 2025) also incorporate searching mechanisms for RL training. TreeRL (Hou et al., 2025) first samples trajectories and then identifies tokens that have high entropies. After that, the model continues to generate the rest responses from the positions of these tokens. As we discussed before, LLMs tend to generate homogeneous reasoning paths due their inherent preference for syntactic patterns established during pre-training. For advanced reasoning patterns such as self-reflection and self-correction that are not learned in pre-training, it still difficult to explore them simply by sampling from high-entropy tokens. What's more, TreeRL indeed adopts vocabulary-level actions conduct tree search on LLMs, which has larger search space and reduced search efficiency. Satori (Shen et al., 2025) introduces three reasoning actions and adopts the Restart and Explore (RAF) algorithm. Specifically, RAE augments training data by randomly backtracking trajectories and inserting reflection tokens in their intermediate step. Its exploration is static and manually designed rather than adaptive training dynamics. Moreover, random insertion of reflection tokens at intermediate steps may be suboptimal.
Therefore, introduce functional tokens to form structured training data after MCTS, enabling the model to establish internalized token-guided reasoning patterns. After SFT, the model can perform diverse reasoning tree search by sampling the functional tokens the model vocabulary without relying on any prompts during inference. This enables the model to automatically discover and reinforce optimal functional token combinations through reward signals, rather than relying on manual backtracking to restart exploration. Moreover, compared to other MCTS -basdd methods in Table 8, we additionally introduce cross verification and branch merging, enabling the connection of both correct and incorrect solution paths for more efficient reasoning paths.
Table 8: Performance of RFIT wider reasoning domains.
Method
different reasoning pattern
self-verification
self-correction
AlphaMath (Chen et al., 2024)
X
X
AlphaLLM (Tian et al., 2024)
X
XX
KNKA
X
X
X
ReST-MCTS* (Zhang et al., 2024a)
A
X
X
TS-LLM (Feng al., 2023)
X
LF-MCTS (Park et al., 2024)
X
rStar (Qi et al., 2024)
MCTSr (Zhang al., 2024b)
X
X
Ours
/
19

---

Published as a conference paper at ICLR 2026
Table 9: Accuracy of our proposed RFTT and baselines across different mathematical reasoning benchmarks. The best results in each box are highlighted in bold. The proposed RFTT significantly boosts the performance of smaller LLMs across all datasets.
In-Domain
Close-Domain
Model
Setting
AVERAGE
MATH-500
GSMSK
SVAMP
Olympiad Bench
AMC
Zero-shot Col
46.8
75.8
78.1
17.5
46.78
Few-shot CoT
47.2
77.7
77.3
17.4
17.5
47.42
CoT+SC@4
49.8
79.5
80.5
19.6
22.5
50.38
LLaMA-3.2-3B Instruct
ReFT
52.4
78.5
81.6
22.8
35.0
54,06
SFT Warmup (Ours)
50.6
77.4
79.1
20.6
35.0
52.54
RFTT (Ours)
56.2
82.9
84,2
24.5
47.5
59.06
Zero-shot CoT
68.4
89.6
80.7
37.9
62.5
67.82
Few-shot CoT
75.8
82.1
40.5
67.5
71.70
CoT+SC@4
76.2
93.7
82.9
41.2
70.0
72.80
Qwen-3-14B-Base
ReFT
95.7
92.5
49.8
80.0
80.76
SFT Warmup (Ours)
78.6
86.8
43.1
72.5
RFTT (Ours)
89.0
97.2
95.3
53.0
85.0
83,90
Table 10: Comparison with entropy-based RL methods for promoting exploration. The best results in each box highlighted in bold.
In-Domain
Close-Domain
Model
Setting
AVERAGE
MATH-500
GSM8K
SVMMP
Olympiad Bench
AMC
Clip-Cov DAPO
76.4
92.5
89.1
43.3
70.0
74.26
79.4
93.5
90.1
44.7
70.0
75,54
Qwen-3-4B Base
RFTT
83.4
96.1
94.7
48.1
75.0
79.
RFTT Clip-Cov
83.8
95.7
95.4
75.0
79.72
RFTT DAPO
85.0
96.4
95.2
49.8
77.5
80.78
Clip Cov
54.6
89.3
84.4
25.0
27.5
56.16
DAPO
56.4
90.1
84.5
25.3
27.5
56.76
LLaMA-3.1-8 1-..3-8----sstti Instrucct
RFTT
60.2
91.9
87.5
29.8
$5.0
64.88
RFTT Clip-Cov
60.8
92.6
88.2
30.4
$5.0
65, 40
RFTT DAPO
62.4
92.7
88.2
B ADDITIONAL RESULTS
Different Model Sizes. Section 4.3, we provide the results on models with 4B, 7B, and 8B param- eters. We further provide the performance of RFTT on models on L.LaMA-33--BB-nsmmmu (Dubey
et al., 2024) and Dowen-3-14-BBatt (Yang et al., 2025) in Table to validate the effectiveness of RFTT across different model backbones and parameter sizes.
Iterative Improvement on Policy Models. We conduct experiments LLaMA-3.1-8B --nstruuttttt for one iter- ative update the MATH dataset. Specifically, we collect the explored reasoning paths with the correct an- swers in the first round RL phase, and further use them for the second round supervised fine-tuning on the policy TSF once more. We introduce R as a comparable baseline, which uses iterative MCTS to collect training data for self-training. In Table 11, we list the results of iterative updates on two datasets, demonstrating continuous enhancement of the initial policy's performance. Notably, our framework achieves superior results compared to R (7.9% vs. 1.7% improvement MATH-500), despite the latter employing an extra reward model to guide its searching.
Table 11: Performance gains under self-im inffimpprveeent.
Dataset
Method
1st Iter
2nd Iter
Ratio f
ReST-MCTS*
85.0
86.2
1.4%
GSM8K
Ours
88.1
2.0%
ReST-MCTS*
47.6
48.4
1.7%
MATH-500
Ours
53.2
57.4
7.9 %
Comparison with entropy- based RL methods. We further compare with DAPO (Yao al., 2025) and Clip-Cov (Cui et al., 2025) that explicitly examined entropy mechanisms to enhance RL for exploration, presented in Table 10. Note that DAPO adds diversity term ( to the original RL objective to facilitate exploration. while Clip-Cov restricts gradient updates of tokens with excessively high covariance. RFTT is fundamentally
fundamnntlll orthogonal these entropy-based RI. tricks:
20

---

Published as a conference paper at ICLR 2026
Table 12: Data information curated from non-mathematical reasoning domains.
MMLU-Pro
MMLU-Pro (biology)
MMLU-Pro (physics)
MMLU-Pro (chemistry)
MMLU-Pro
Data Source
HumanEval
(health)
(law)
Samples
267
144
328
135
126
160
Table 13: Performance of Qwen 2.5-7B-II-Bsuuuu being trained on general domain data.
MMLU (STEM)
MMLU (Professional Medicine)
MMLUU (Professional Law)
Settings
GPQA
CommonsenseQA
MBPP
Few-shot CoT
81.1
77.6
50.4
36.4
81.6
79.2
RFTT (Ours) 86.9
83.1
55.8
38.8
85.2
83.8
they act at the RL optimization objective, while our method promotes exploration by improving the diversity of sampling. We operate at different levels but do not conflict with each other. The results show that adding these tricks to loss functions can further improve the performance of our method.
Broader Generalization. To further explore generalization, we have additionally conducted ex- periments using training data curated from non non-mathematical general domains including MMLU- Pro (Wang et al., 2024c) and HumanEval (Chen et al., 2021a), as shown Table 12. We evaluate the performance on three different reasoning domains: (1) Commonsense reasoning benchmarks includ- ing Medicine (MMLU (Hendrycks et al., 2021a)), Law (MMLU), and CommonsenseQA (Talmor et al., 2019); (2) Science reasoning benchmarks including STEM (MMLU) and GPQA (Rein et al., 2024); (3) Code generation benchmarks including MBPP (Austin et al., 2021). The results in Table 13 show broader improvements when RFTT is exposed to more diverse reasoning data, indicating strong potential for cross-doman generalization.
Token Frequency. We analyze the combinations of used functional to- kens on MATH-500, GPQA, and MMLU. Table 14 shows that in math and science domain (MATH-500 and GPQA), the model tends to engage in analysis, reflection, and correction. While in commonsense reasoning (MMLU), it prefers to clarify the problem and then think step by step without reflection.
Table 14: Token usage in different reasoning domains.
Dataset
Top-3 Combinations
MATH-500
a2tastastac, aatastautas datastattactar a2tastastag as autastas, da 01+02+04+aatt aitasta4, a tastaatas. ds a1tasta4+a0
GPQA
MMLU
Reward hacking analysis. In practice, we com- bine outcome-based reward supplemented by PRM score signals, which aims to balance be- tween both accuracy and quality of the reason- ing process. We further analyzed the average process rewards (ranging from -0.5 to 0.5) of trajectories generated by models on the MATH- 500 dataset. The results in Table 15 show that the average PRM scores of incorrect trajectories are much lower than those of correct trajectories, which may alleviate the problem of reward hacking.
Table 15: Potential reward hacking analysis when rewarding reasoning chains.
Correct Samples
Incorrect Samples
Avg. Score
Qwen-3-4B-BP Qwen-2.5-7B-Inntmee -LaMA--311-8B-Inste
0.38
-0.04
0.34
0.02
0.31
-0.05
C EXPERIMENTAL DETAILS
Dataset Information. We list detailed information of our training and evaluation dataset in Table 17. Specifically, we select 1000 and 3994 questions from the training set of MATH for SFT and RI,, respectively. Recent studies have revealed that focusing more difficult problems can better enhance reasoning capability (Min et al., 2024b; Team et al., 2025), thus we increase the proportion of difficult questions (Level 4 and Level 5) in our training. The data distribution of SFT and RL phases are shown in Figure 6.
Evaluation Method. In evaluating the performance of superior LLMs, take the relevant metrics from their official technical reports (Achiam et al., 2023; Anthropic. 2024; Liu et al., 2024a; Dubey al., 2024; Yang et al., 2024a). For non-reported metrics, report their accuracies on corresponding
21

---

Published as a conference paper at ICLR 2026
60
30
SPT
STT RL
Tokens
RLL.
25
20
Pocentag
Langt
1500
50
15
20-
10
1000
10
300
Lavel Lavil
Lavd
Lev14 Lewid
Nanbo:
Prolysha Prealsske
Loodl
Levd2
Levl1
Levi4 Love
Alatta
Toeay
Difficulty
Subject
Average Tokens
Figure 6: Detailed statistical information of the training dataset in two-phase training.
Table 16: The CoT template for evaluation. The variable question will be replaced with the specific problem when testing.
You are a mathematics expert that solves problems step by step. Note that you should output your final answer with the format of "The answer is: boxed <ANSWER>> where <ANSWER> should be numeric result math expression. Question: [question]
Let's think step by step.
datasets using the official API with zero-shot (original question without any prompts). For the results of CoT, we demonstrate the CoT prompt template used to evaluate in Table 16.
D MCTS PROCEDURE
The MCTS algorithm is capable of balancing exploration and exploitation within vast environmental spaces, enabling it to quickly identify high-quality steps trajectories. Due to its focus on exploring unseen states and sampling high-quality trajectories, which is key aspect of online sampling in reinforcement learning, MCTS has been widely applied in reinforcement learning tasks, such as AlphaGo and AlphaZero (Silver et al., 2018). The standard MCTS algorithm consists of four stages: selection, expansion, simulation, and backpropagation.
(I) Selection. Starting from root node (question) so x, a series of internal nodes { 82, ... 81) are chosen until reaching a leaf node 51. the selection phase, we use the Upper Confidence Bound applied to trees (UCT) select each node for balancing exploration and exploitation:
arg max mo(a(so:)) U(s:) + 0,
(5)
arg ((x UCT(s), otherwise,
Q(s).a)
Nss:)
UCT(s;, a) N(s t
(6)
N(s,,a)
where A(s)) represents the of possible actions (functional tokens) that be selected after the current step st, (st) denotes unexplored functional tokens after the current step St, and Te(a)soa) the probability sample functional token based on the current state s0:1- In UCT score, N (s:) denotes the number of times step 8t has been visited in previous iterations, N (st, a) denotes the number of times action a has been selected after step 8t, ( a) the cumulative reward value of selecting action after step and c is constant that balances exploitation and exploration.
In LLM sampling, each action in MCTS corresponds to a reasoning step. The sequence of generated steps forms the current state. Selecting an action in MCTS means generating the next step, and the next state obtained by appending this step to the already generated steps.
(2) Expansion. If the selected leaf node si is not terminal node, it is expanded by sampling an action from A = {al, 02, a3, a4, a5, ).). We then use the action a; E AA (functional prompt in SFT or functional token in RL) to guide the LLM to generate corresponding reasoning step. Notably, the expansion of certain actions has dependency relationships for efficient exploration. For example, <clarify> (a1) can only be expanded after the root question, and <subquestion> (ax) can only expand the <next_ <next_step> (a4) and <direct_answer>s answer> (aş). All prerequisite dependencies presented in Table 18.
22

---

Published as a conference paper at ICLR 2026
Table 17: The detailed information of five datasets.
Table 18: The dependencies between actions.
Dataset
Train
Test
Domain
Previous Action
Next Action
MATH
7500
5000
In- In-Domain
clarify (01)
03; As
GSM8K
7473
1319
Out-of of-Domain
analysis (a2)
03, 04. ds
SVAMP
1000
Out-of-Domait
subquestion (as)
G4,
Olympiad Bench
675
Out-of-Domain
next_step (04)
azy 04, (o Ag; ds
AMC
40
Out-of-Domaal
direct_answer answer (as)
06 ag
GPQA
448
verify (as)
u4; (5s (66; 47,,
MMLU-Pro
12032
Out-of-Domain
refine (a7) 03 04. doy Ag. Us
(3) Simulation. The simulation phase involves randomly selecting an action from expanded node, generating the corresponding step, and continuing this process until terminal node is reached. We define the terminal node as achieving either the final answer or the maximum permitted depth.
(4) Backpropagation. Based on the results of the simulation, backpropagation updates the Q and N values for all nodes along the path from the expanded node to the root node as follows:
( a) Q(+:, a)) +r;
(s + a),,, N(s))
Σ EncAtoo) (s),aa)
(8)
where is the terminal reward obtained from the simulation reaching the terminal node.
E FUNCTIONAL TOKENS WITH PROMPTS
We present the functional tokens and prompts that were designed for experiments.
<clarify> (a1): Organize the conditions of the given complex question and express them in clearer form to avoid misunderstanding.
<analysis> (a2): Analyze the general problem-solving idea and the relevant knowledge.
<subquestion> (ag): Break down complex reasoning problems into manageable sub aub-problems.
<next_ mext_step> (a,): Propose the next intermediate step based on the existing reasoning steps.
sdirectaanswed (aş): Complete the remaining steps step-by-ste to reach the final answer.
<verify> (as): Reflect if there are any mistakes in the preceding reasoning steps, considering LLMs might make computational logical errors in reasoning step.
<refine> (a7): Correct certain previous steps if the errors are identified, ensuring that each step is accurate to achieve a final outcome.
<output> (ag): Output the final answer that is the expected format.
Prompt for Functional Token <clari fy>>
You are an Al assistant to help me clarify questions by restating the original question and task a clear and comprehensive manner. In your clarified version, ensure that all information from the original question fully expressed. Following some useful examples.
Original Question: Increasing radius a cylinder by units increased the volume by y cubic units. Increasing the height of the cylinder by 6 units also increases the volume by cubic units. If the original height is 2, then what is the original radius?
We are given problem involving a cylinder where increasing the radius by units and increasing the height by 6 units cach results in an increase in volume by cubic units. Our goal is to find the original radius of the cylinder, given that the original height is 2.
Clarified Question:
Original Question: wooden model of square pyramid has base edge of 12 and altitude of cm. A is made parallel to the base the pyramid that separates into two pieces: a smaller pyramid and a frustuu. Each base edge of the smaller pyramid is 6 em and its altitude is em. How many cubic centimeters are in the volume of the frustum?
23

---

Published as a conference paper at ICLR 2026
Clarified Question: Let's understand what is being asked. Given wooden model of square pyramid has a base edge of and altitude 8 Also given cut parallel the base creates a smaller pyramid and frustum, Note the smaller pyramid has a base edge 6 em and an altitude 4 em. We need to find the volume of the frustum.
Question: uesr_ cesr_qqeetion]
Prompt for Functional Token < <ana lysis>
Given a math problem and an existing incomplete solution, your task is to provide brief insight for the following problem. Note that it's required provide any specific solution process calculation, but give general guidance of sovling this problem. Usually one or two sentences are enough.
Question: uvesr_question
Prompt for Functional Token <subquesticnt
Given math problem and an existing incomplete solution, please propose sub-problem that can be solved independently based on my current progress. The output format limited to: "Let's now". Where ... indicates the omitted sub-problem that you should in. Note that not propose sub-prote uub-problems that have already been solved in previous steps.
Here is the input, please follow the restricted output format.
Question: Lesr_ duerr_qeettion
Prompt for Functional Token <next_stepp
Given a math problem and an existing incomplete solution, your task to propose one next step in smooth and proper way.
If existing steps are provided, you need to briefly analyse the prohlem from scratch and then output the first step. Otherwise, you need to output correct next step of the existing solution, following the ideas of the existing steps.
Your output should be single reasoning step that may include reasoning, detailed calculations process, choosing answers, etc.
Question: uesr [essr_quettio]
Prompt for Functional Token <direct_answex
Given a math problem and an existing incomplete solution, your task is to complete the remaining solution in a smooth and proper way. You need to give step-hy-step solutions to the problem following the ideas of the existing steps, and do not repeat any existing steps.
Question: uesr_ pesss_quetiion
Prompt Functional Token <verify>
Given problem and its previous solution steps, there is correct subsequent solution and your subsequent solution. Please meticulously review your subsequent solution step by step and identify any mistakes occur.
# Correct solution
(correct solution]
Your subsequent solution (solution]
24

---

Published as a conference paper at ICLR 2026
F REASONING CASES
Our constructed data consists of diverse human- reasoning behaviors, including double check, self-reflection, and self-correction. The following are some data examples of diverse reasoning paths in mathematical reasoning.
Case I: Self- Verification
# Problem: Complex numbers a, 6, form equilateral triangle with side length 18 in the complex plane. If la b + c| = 36, find lab ac + bcl.
# Output: (clarify) The complex numbers a, b, and e form an equilateral triangle with side length 18 in the complex plane. Additionally, we know that (a + b 36. Our task is to find the value of lab + ac + bel. (/clarify)
(analysis) Use the properties equilateral triangles in complex plane, where the centroid (average) of the vertices is involved, and relate it to the given magnitude. Utilize the fact that for an equilateral triangle formed by complex numbers b, and the expression a² + + c² = ab ac + bc. (/analysis) (subquestion) First, Let find the value of + b + c. (/subquestion)
(next_stte step) Since the centroid of the equilateral triangle formed by a. b. and is given by a+b+e, and we know la + 6 6++ c| 36, we can find la + b+ directly as 36. However, the next logical step is to use this information to find
6
Given b + c| 36, can denote recognize that a + + è a complex number with magnitude 36, without loss of generality, we can assume a - b b 36c "# for some angle 9. For simplicity, we can consider the case where 0 = 0, so + b + 36. This assumption simplifies the calculation without affecting the final result due to the properties of magnitudes.
b b+ 2, where
36. The specifie process for this step
So, we have:
a + c 36.
/tep)
(subquestion) Next, I need find the value of lab + bel. (/subquestion) (next_step) Morcover, I will the identity for the of products of roots polynomial.
lab + ac + bcl (a + b c)² (a²
+ cc)
(/next_stepp
(subquestion) Then, we must calculate a² + 6² e². (/subquestion) (nxx__seppi Using the property of equilateral triangles in the complex plane, know that a² + b2 c² ab be. However,
also express a² + b² + d² terms of a + + and ab bc:
a² + 2 + = (a + + c)² 2(ab + ac + bc).
Given a + b+ == 36, we substitute to get:
2(ab ac bc).
Now, let's solve for ab ac + be. (/next step) (subquestion) To continue, need solve for ab + bc. (/subquestion) (next_step) We already have:
+ 62 + c² = 36 2ab + ac + bc).
And since a² b2 + c ab + + bc, we substitute get:
ab + ac + be = 362 2(ab + ac + hc).
Rearrange to solve for + ac + be:
3(ab ac + bc)
362
36
1296
ab be
432.
3
3
Now, let's find lab + ac + be). (/next step)
(verify) Hold on, let me confirm that. The given steps correctly use the propertics of equilateral triangles in the complex plane, including that the centroid (average) of the vertiees is involved and the identity ac bc. The calculation for ab + ac + be is correct:
a² + b² c²
3(ab ac + bc) 36
25

---

Published as a conference paper at ICLR 2026
ab + ac + be 1296 432.
The magnitude of a real number is the number itself, so lab + ac + bcc 432. There are no mistakes in the logic or calculations. (/verify)
(output) So the answer is 432 (/output)
Case 2: Self-Refection and Self-Correction
# Problem: Suppose that f(x) =
2m++8 For what value of b does f (x) 1 2n 7
2x
Output: (analysis) To find the value of which the inverse function (=) 1-2s matches with [(x) 2 start by understanding that if y f(x), then f/=====+/] (v). This means substituting
2
f(x) into the given form of f fxx(x) should yield x. (/analysis)
(next_step) First, substitute f(() 2245
into the equation for f (x)). giving us:
2x
2x+8
This is based on the principle that applying function and its inverse in succession returns the original input, ie., (f((())) (/next step) (next_step) Simplify the right-hand side:
(2x b) -
2x
+ 2-2
2
2
2m+6
The simplification involves combining terms and recognizing that dividing by a fraction equivalent to multiplying by its reciprocal. (/next_ step)
(next step) For the expression to equal x, we set the simplified form equal to at:
+
To solve for b, we equate the numerators (since denominators are already the same and non-zero), leading to:
2x 2r + 2.
(/next_step) (direct_answer) From the equation 2z = 2x + - 2, subtract 2r from both sides to isolate
b
which simplifies to:
b = 2.
Thus, the value of that satisfics the condition is = 2, confirming the expected answer. /ddrecttanswert (verify) But have a feeling that it's little complex. Maybe I should consider a simpler approach. (/verify) (analysis) To find the value of b, first express (xx) in terms of f(x) and then solve for b. (/analysis) (subquestion) First, I should solve for (xx) in terms
(/subquestion)
step) Start with y = fx) 2 Swap x and y to get 2545. (/next_step)
Then solve y to find f (x)
(subquestion) Next, 1 need to solve for y in the equation x 25+5 (/subquestion)
(next step) Multiply both sides by 2y b get rid denominator. then isolate y. (/next step)
(next_step) Multiply both sides by 2y + to obtain (2y + b) = 1. Then distribute x on the left side to get 2zy bær = 1. (/next_step)
(direct_answ)) answer) Next, will isolate y. Starting from the equation 2ry bu
2xy + bx =
Subtract 2ny from both sides:
2xy
26

---

Published as a conference paper at ICLR 2026
Divide both sides by b:
1 2
y/
Since know that f (x)) 1-2s
can equate the expressions for y::
2z
2xy
1 2x
b
2x
the above equality hold 2, numerators and denominators must match. Therefore, we can compare the denominators directly:
b 2r. 2x = 2 4
(/direct answer)
(verify) Oops, seem to get two different results. Actually, I'm getting confused. think should review carefully about my previous solutions.
The mistake occurs when dividing by and incorrectly expressing y 1-2sy
The correct form should be l-he
As result, the value of is incorrectly determined. (/verify) (refinc) Correct the expression for after isolating it:
1-bx 2
Equate the given f {x) 1=2x:
bc
2r
2r
2xr
Since the denominators are the same, equate the numerators:
1 ba 1 2x
Subtract from both sides:
bb 22
Diiidde both sides by - (assuming 0):
b 0
Thus, the value of b is 2. (/refine)
(output) The answer is: (/output)
We provide the following response 2een-2255---Inttruue comparisons before and after training to demonstrate the effectiveness of our method.
Problem from MATH
Semicircles of diameter 2" are lined shown. What is the area. in square inches, of the shaded region in 1-foot length of this pattern? Express your in terms of I.
1'
Initial Response of Qwen-2 1irenn225--70 2.5-7BB-nnste
To determine the area of the shaded region in I-foot length of the given pattern, we need to analyze the pattern and calculate the area of the semicireles involved. First, note that foot equal to inches. The pattern consists of semicircles of diameter 2 inches, which means semicircle has radius of inch.
The area of one semicircle is given by: Area of one semicircle = } = (1)²² = 1... Next, we need to determine how many semicircles fit into a 12-inch length. Since each semicirole has diameter of inches, the number of semicireles 12 inches is 12 6.
27

---

Published as a conference paper at ICLR 2026
The pattern alternates between semicircles above and below the horizontal line, forming repeating sequence of semicircles. The shaded region consists of the top semicircles and the bottom 3 semicircles. The total area of the shaded region is the sum of the areas these semiciroles: Total area of shaded region = 6 x - = 3T.
Thus, the area of the shaded region in I-foot length of the pattern is 3m
Response Trained by RFTT
(clarify) We are given pattern where semicircles 2iameter inches are lined up in specific arrangement. The task is to find the area, in square inches, of the shaded region within a I-foot (which is 12 inches) length of this pattern. The answer should be expressed in terms of и. (/clarify) (next step) First, need to calculate the radius of each semicircle. The diameter of each semicircle is inches, so the radius is half of the diameter, which 1 inch. (/next_step) (subquestion) Next, Let me calculate the area one semicirele. //subquestion)
(next_step) The area of a full circle with radius = 1 inch is πr² = =T(1²² = ग square inches. Therefore, the area of one semicircle is half of that, which is square inches. (/nexi_step) step)
(subquestion)/ Moreover, could attempt to determine how many semicircles fit into 12-inch length. (/subquestion)
(next_step) The pattern consists of alternating upper and lower semicircles, each with a diameter of inches (radius 1 inch). Upper semicircles curve upward, and lower semicircles curve downward, forming repeating wavy pattern. The pattemn repeats every 2 inches. In each 2-inch segment, there is upper semicircle and lower semicircle. (/next_step) (next_stee) Then, could attempt to calculate the total area of semicircles in 12 inches. each inch segment, the shaded area is the sum of the area of one upper semicirele and one lower semicirele: G + 2 T). Since the pattern repeats every 2 inches, 12-inch length contains 12 Thus, the total of the shaded region is ( (6 (verify) Wait, the question arises: have we correctly calculated what comprises the "shaded" region? Upon reassessment, we acknowledge each row aligns exactly with the previous, consisting of precisely half periods in overlapped symmetry, with no shaded region lawyer above certain baseline symmetry points. (/verify)
2 = 6) such segments.
6T ) (/next step)
(output) So the answer is
6
(/output)
G FURTHER DISCUSSION AND LIMITATIONS
G.1 How DO FUNCTIONAL. TOKENS BENEFIT?
There are two main challenges in applying MCTS during training (Guo et al., 2025): (1) Unlike traditional games with well-defined action spaces, the combinatorial nature of vocabulary leads to exponential growth in candidate reasoning paths; (2) Training fine- fine-grained value model to effectively guide the search process remains highly challenging. The introduction of functional tokens with rich semanties can diversify node exploration while constraining the search space. Notably, we can use the intrinsic probability of functional tokens, which adjust dynamically during training to guide its search process, thereby estimating the need for training an extra value model. Furthermore, the functional tokens naturally divide the reasoning process into series of fine- fine-grained steps, which can seamlessly integrate with PRM.
G.2 IS SFT WARMUP WITH FUNCTIONAL TOKENS NEEDED?
Recently, Deepseek RI-Zero (Guo et al., 2025) has demonstrated the emergence of self-reflection in the LLM's response during pure RL phase. We also encounter the occurrence of this kind of feature when conducting pure RL on 2.en-2.5-7BB-nne But such a pattern occurs at extremely low rate (about 1.5%) on Qwen-2.5-7B. 2.---B----nttttr and there is even no such case observed on LLaMA-3.J 1.-MA--.---B--.. I-8B-Instruct. The power of RL is to reinforce high-quality reasoning patterns like self-reflection and exploration of different approaches, thus maximizing expected rewards by leveraging increased test-time computation. However, in a model with smaller parameters, the limitation arises from their inherent preference for syntactic patterns established in pretraining, which further restricts the reinforcement of sophisticated reasoning patterns. Therefore, SFT warmup with functional tokens play an important role in faster convergence and stable training.
(RFTT w/o SFT Warmup), as shown in Table 19.
28

---

Published as a conference paper at ICLR 2026
Table 19: Emergence of self-reflection and self-correction during pure RL on Qwen-2.5-7B-Instte
Question: ARST, shown, sin j. What is sin [asy]]..[/asy]
Response: Given that A RST is right triangle with the right angle S, we know that sin R
apposile
Here,
Iypoiensse
78S
< Then, have: we can find the length ST using the Pythagorean thcorem:57 = vRS² RT = v 2 22
RT' 221 and RS 5. Therefore. we have:sin R = 5 3. This implies that v21 2. which is incorrect based the given information. Instead, should use the correct value of sin R = and find the correct lengths. Let's =
denote the length RT and the length of RS
a Now,
In ARST, sin I' byporenuse
apposite
RS
However, we need to re-ovaluate the correct trigonometric relationship.
Ra
Since sin R 3. we know that cos R v sin?
(3)
1/21 right triangle,
25
25
sin T = cos R. Therefore: sin T = cos R
V2T 6
v2I
Thus, the value of sin is:
Average Length
KL
RETT weo SPT Warmap
RETI w/o MUTS
0,025
2000
0,030
0,020
1900
0,025
Total Leiggh
800
0,020
1800
0.015
700
1700
0,010
600
0,005
1600
0.005
500
1500
0.000
100
200
500
400
100
200
3000
400
Traiting
Training stop
Figure 7: The training curve of RFTT w/o SFT Warmup and w/o MCTS during RL.
G.3 DISCUSSION OF RESPONSE LENGTHS.
Recent research suggests that the reasoning capabilities of the model may benefit from longer response lengths (Guo et al., 2025). In Figure RFTT w/o MCTS on LLaMA-2 C.MMA--..-3B-Instttee
present the training curve of RFIT w/o SFT Warmup and
as an example. We conduct RFTT w/o SFT Warmup with the prompt template in Table 16. Surprisingly, we find that after initial drop in the response lengths of the policy model, the lengths rebound but persistently remain below the response lengths observed in the original pretrained model. Moreover, we observe that as the response lengths drop. the KL divergence between the current policy and the initial model increases during training period 4; while the lengths increase, the KI. divergence drops during training period (2. This indicates potential correlation between response lengths and KI, constraints. It seems that the subsequent increase in response lengths cannot be attributed to the exploration of sophisticated behaviors, but rather primarily stems from the impact of the KL penalty.
In contrast, the response lengths of RFTT w/o PRM in Figure 5 do not exhibit a downward trend. This is likely due to the benefits of our functional boken-guided tree search, which provides diverse explo- ration space. Furthermore, the use of PRM in MCTS introduces fine- mne-grained reward assignments in intermediate functional tokens. Such mechanism significantly mitigates the exploration limitations imposed by the KI. divergence penalty. potentially elucidating the observed length expansion in RFTT with PRM during the training process. This highlights the effectiveness of RFTT, where the functional tokens serve as bridge that seamlessly combines the employment of MCTS and PRM for learning to reason.
G.4 LIMITATIONS
While our method improves reasoning via functional tokens and reinforcement learning, it may also introduce new risks. For example, the use of reasoning tokens could be exploited through token injection attacks to manipulate model behavior. Additionally, self eeffiimprovement via reinforcement learning raises concerns about unintended capability amplification or instability. More generally, enhanced reasoning ability does not eliminate common LLM issues such bias, overconfidence, or the generation of incorrect but convincing outputs. We acknowledge these risks and believe they warrant further study functional reasoning methods evolve.
29


Reviews:
[{"review_id": "qjnaO51tMI", "reviewer": "Reviewer_b1q7", "summary": "This paper proposes to train LLMs to reason by introducing new special tokens (i.e., functional tokens in this paper) and combining it with tree search. The proposed method warm-up the model with SFT data that teach the model how to follow the functional tokens. The second stage trains the model to generate diverse reasoning paths with MCTS. The results show that the proposed method (RFTT) improves the performance over RL fine-tuning on math reasoning tasks.", "questions": "- Line 64: What does human-like reasoning tree mean?\ne\n- Line 68: What does \"prompt-guided\" mean?\n- Line 69: Why the exploration is diverse needs to be explained.\n- Line 172: How do you tell the intersection?\n- Line 166-Line 180: The notations are very hard to parse.\n- Line 210: Unclear what ANS predicts.\n- Line 348: It's not rigourous to use \"siginifcantly improve\" when there are no confidence interval on the results.\n- Line 352: What is self-consistency sampling?\n- Line 363: I'm confused. I thought you need to add functional tokens to the dictionary also in RL/SFT training, but it reads like you only add them to the dictionary while doing tree search?\n- Line 365: Without reward model or any other reward signal, how do you perform MCTS?\n- Line 204: UCT score's definition is missing. Though it was a classic notion in MCTS, it's better to clarify the definition in the context of this paper.\n- Line 262: What is \"MATH dataset\"? Needs reference.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of combining tree search and RL during training is a promising idea since it may improve data efficiency significantly.", "weaknesses": "- Writing is not clear. See my questions.\n- Training model with Equation 4 with the data collected by MCTS sampling (Equation 1) is biased. The PPO's objective needs to sample states and actions from the policy pi_old, but MCTS sampling draws actions based on UCT (Equation 1), which won't yield an accurate expectation on the clipped objective L_RL\n- The ablation study (Table 3) shows the setting of ablating SFT-warmup, MCTS, and PRM, but it's unclear that if MCTS and PRM are required. One possibility is that RFT + SFT-warmup would yield similar performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pR1BgOot85", "reviewer": "Reviewer_Hzqr", "summary": "The paper proposes **Reinforced Functional Token Tuning (RFTT)**, a two-phase framework to endow LLMs with learn-to-reason abilities by internalizing *functional tokens*. Phase I performs SFT on training data generated by a functional-prompt-guided MCTS that stitches together correct and incorrect branches via cross verification and branch merging, yielding structured, token-annotated reasoning paths. Phase II replaces prompt guidance with token-guided exploration and optimizes with Reinforce++ under a KL penalty to a reference SFT model, optionally using a PRM for step-level rewards. On math benchmarks (MATH-500, GSM8K, SVAMP, OlympiadBench, AMC), RFTT improves small models (≤10B) markedly over CoT, self-consistency, ReFT, and GRPO, and shows monotonic gains with more inference-time rollouts. Ablations indicate benefits from SFT warmup, MCTS, and PRM, while token masking highlights the importance of `<verify>` and `<refine>`. The authors also report training dynamics, cost trade-offs, and limitations. Overall, the contribution is a principled way to compress a vast vocabulary-level action space into a compact, semantically meaningful policy over functional tokens, enabling more effective exploration and reinforcement of reasoning behaviors.", "questions": "1. Could you elaborate the motivation of using PRM? How does it compare with ORM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work is original in elevating “functional prompts” (e.g., from rStar-style search) into learnable tokens that become part of the model’s action space and are optimized end-to-end. This shift from external, prompt-level constraints to internalized, token-level policies is a clean conceptual advance that reduces search branching at the word level while preserving diversity (§3.1–3.3, pp.3–5). \n\nThe qualityof the methodology is supported by a careful algorithmic design—cross-verification and branch-merging produce structured SFT targets (§3.2, p.4), token-guided MCTS with UCT balances exploration/exploitation (Eqns. (1),(5)–(8), pp.4,23), and PRM-based process rewards address long-chain credit assignment (§4.4, Table 3 on p.8). Empirically, the paper demonstrates clear, consistent gains across multiple backbones (Qwen-3-4B, Qwen-2.5-7B, LLaMA-3.1-8B) and datasets, with strong improvements versus ReFT and GRPO and favorable scaling with additional rollouts (Table 1, Figure 2, pp.6–7). \n\nThe exposition is clear, with a unifying diagram (Figure 1, p.3), explicit token definitions and prompts (Appendix C, pp.18–19), and transparent training/eval details (§4.2, p.6). RFTT offers a general recipe to couple MCTS and RL via a compact, behavior-level action space, pushing small-model reasoning toward larger-model performance, and the ablations convincingly attribute where the gains come from (Table 3, p.8; Table 4 and Figure 3, p.8).", "weaknesses": "1. The OOD benchmarks are still somewhat in-distribution, as they are essentially math reasoning problems. It would be interesting to test models on other reasoning benchmarks, like FOLIO (logical reasoning), CruxEval (code reasoning), TableBench (table reasoning), etc.\n\n2. Baseline methods are not comprehensive. Performance comparisons with more recent tuning based methods (e.g. rStar-math, Satori, MCTS-DPO) and non-tuning based methods (e.g. rStar, RAP) would bring more insights into the strength of this work.\n\n3. Scalability test of the method is missing. Experiements on smaller/larger models would also be interesting.\n\n4. Showing these heuristic designs of reasoning actions also work on other reasoning domains, like code generation, for synthesizing training data, would also strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YDQuy1j2a6", "reviewer": "Reviewer_Wmaw", "summary": "The paper proposes Reinforced Functional Token Tuning (RFTT), a two-phase framework to enhance LLM reasoning via functional token–guided Monte Carlo Tree Search. During the SFT phase, the model learns reasoning behaviors, such as <analyze>, <verify>, <refine>) from self-generated trajectories. In the RL phase, it explores diverse reasoning paths using these tokens as actions, optimized by process and outcome rewards. Experiments on math and general reasoning benchmarks show improved performance over GRPO and ReFT, suggesting functional tokens enable structured, self-improving reasoning in smaller LLMs.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and well organized, with a logical flow from motivation to methodology and experiments. Figures and tables effectively illustrate the framework and results.\n\n2. The method is sound, with detailed descriptions of the two-phase (SFT + RL) pipeline, reward design, and MCTS procedure. All experiment settings are reasonable and detailed across different domain and model scale size.", "weaknesses": "1. The overall framework is somewhat similar to Satori (Feb 2025) and TreeRL (Jun 2025), as well as other recent RL-based reasoning frameworks that employ token-level actions or tree-structured exploration. It would be helpful to clarify how your method differs from these tree-RL-like approaches and to provide side-by-side results with these baselines [1, 2].\n\n2. The current comparisons include only ReFT and GRPO, omitting strong recent baselines such as TreeRL and DAPO. Adding at least one of these baselines—or discussing the expected difference under a comparable compute budget—would make the evaluation more convincing [2, 3].\n\n3. The paper attributes its improvement to high-entropy branching, but it remains unclear whether this approach provides a clear advantage over existing entropy-based RL methods, which often achieve similar goals without the added cost of increased training time. Since several recent studies have explicitly examined entropy mechanisms to enhance RL for reasoning LMs [4], a comparison with those methods would make the claim more solid and help clarify whether the added training cost is justified.\n\n\n[1] Satori: Functional Token Guided Reinforcement Learning for Structured Reasoning, Feb 2025.\n[2] TreeRL: Tree-Structured Reinforcement Learning for Multi-Step Reasoning, Jun 2025.\n[3] DAPO: Diversity-Aware Policy Optimization for LLM Reasoning, May 2025.\n[4] The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models, May 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBsTvS9qmt", "reviewer": "Reviewer_xic9", "summary": "The paper proposes Reinforced Functional Token Tuning (RFTT), a novel reinforcement-based fine-tuning framework that equips large language models with learn-to-reason capabilities. Unlike prior prompt-driven reasoning methods, RFTT introduces learnable functional tokens (e.g., <analyze>, <verify>, <refine>) directly into the model’s vocabulary to enable structured and human-like reasoning behaviors. The framework consists of two stages: supervised fine-tuning with functional tree search to teach token usage, and online reinforcement learning to encourage autonomous exploration of reasoning pathways. Experimental results on mathematical and general reasoning benchmarks demonstrate that RFTT significantly outperforms baselines and continues to improve with increased test-time computation.", "questions": "1. Please provide a analysis of the potential **reward hacking** problem that may arise when rewarding reasoning chains, where the LLM could exploit the reward model by generating answers that align with its biases or merely appear correct to maximize the reward.\n2. How sensitive is RFTT to the choice and number of functional tokens? Would the performance drop if some tokens (e.g., <verify>, <refine>) were removed or replaced?\n3. During functional tree search, what criteria are used to select the optimal reasoning branch?\n4. How to get the dependencies between actions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of learnable functional tokens provides an interpretable and flexible mechanism for modeling reasoning behaviors within the LLM’s vocabulary.\n2. The two-phase framework combining supervised functional tree search with reinforcement learning.\n3. Experiments demonstrate consistent and significant improvements on mathematical and general reasoning benchmarks", "weaknesses": "1. The training framework may computationally expensive due to the integration of Monte Carlo Tree Search, which limits scalability to larger models.\n2. The semantics of functional tokens are predefined by design, which may constrain flexibility and introduce human bias into the reasoning process.\n3. Rewarding reasoning chains may cause reward hacking, where the model learns to produce text that merely appears logically sound rather than genuinely correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)