RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning
Authors: Qianyue Hao
Sibo Li
Jian Yuan
Yong Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
reasoning
reinforcement learning
Primary area: reinforcement learning
TL;DR: This paper proposed to utilize RL at inference-time to navigate the reasoning logics of LLMs, which significantly enhances the reasoning capabilities of multiple LLMs across various tasks with less than 3K parameters in the RL navigator model.

Abstract: Despite rapid advancements in large language models (LLMs), the token-level autoregressive nature constrains their complex reasoning capabilities.
To enhance LLM reasoning, inference-time techniques, including Chain/Tree/Graph-of-Thought(s), successfully improve the performance, as they are fairly cost-effective by guiding reasoning through external logical structures without modifying LLMs' parameters.
However, these manually predefined, task-agnostic frameworks are applied uniformly across diverse tasks, lacking adaptability.
To improve this, we propose **RL-of-Thoughts (RLoT)**, where we train a lightweight navigator model with reinforcement learning (RL) to generate task-adaptive logical structures at inference time, enhancing LLM reasoning.
Specifically, we design five basic logic blocks from the perspective of human cognition.
During the reasoning process, the trained RL navigator dynamically selects the suitable logic blocks and combines them into task-specific logical structures according to problem characteristics.
Experiments across multiple reasoning benchmarks (AIME, MATH, GPQA, etc.) with multiple LLMs (GPT, Llama, Qwen, and DeepSeek) illustrate that RLoT outperforms established inference-time techniques in most cases and improves up to 13.4% in challenging situations.
Remarkably, with less than 3K parameters, our RL navigator is able to make sub-10B LLMs comparable to 100B-scale counterparts.
Moreover, the RL navigator demonstrates strong transferability: a model trained on one specific LLM-task pair can effectively generalize to unseen LLMs and tasks.
Our code is open-source at https://github.com/tsinghua-fib-lab/RL-LLM-Reasoning.

Published as a conference paper at ICLR 2026
RL OF THOUGHTS: NAVIGATING LLM REASONING WITH INFERENCE-TING REINFORCEMENT LEARNING
Qianyue Hao; Sibo Li, Jian Yuan, Yong Lif
Department of Electronic Engineering, BNRist, Tsinghua University
Beijing, China
ABSTRACT
Despite rapid advancements in large language models (LLMs), the token-level autoregressive nature constrains their complex reasoning capabilities. To en- hance LLM reasoning. inference-time techniques, including Chain/Tree/Graph-orf Thought(s), successfully improve the performance, as they are fairly cost-effective by guiding reasoning through external logical structures without modifying LLMs' parameters. However, these manually predefined, task-agnostic frameworks are applied uniformly across diverse tasks, lacking adaptability. To improve this, we propose RL-of-Thought (RLoTT), where we train lightweight navigator model with reinforcement learning (RL) to generate task-adaptive logical struc- tures inference time, enhancing LLM reasoning. Specifically, design five basic logic blocks from the perspective of human cognition. During the rea- soning process, the trained RL navigator dynamically selects the suitable logic blocks and combines them into task-specific logical structures according to prob- lem characteristics. Experiments across multiple reasoning benchmarks (AIME, MATH, GPQA, etc.) with multiple LLMs (GPT, Llama, Qwen, and DeepSeek) illustrate that RLoT outperforms established inference-time techniques in most cases and improves up to 13.4% in challenging situations. Remarkably, with less than 3K parameters, our RL navigator is able make sub- L0B LLMs com- parable to 100B-scale counterparts. Moreover, the RL navigator demonstrates strong transferability: model trained on one specific L.M-task pair can ef- fectively generalize to unseen L.LMs and tasks. Our code is open- at fib- lab/ RL- LLM Reasoning
INTRODUCTION
Recent years have witnessed unprecedented advancements in large language models (LLMs), achiev- ing remarkable success across diverse natural language tasks (Chang et al. 2024), , including transla- tion (Xu al. 2024), semantic analysis (Lan et al. 2024b and information retrieval (Hao et al. 2024). Despite these advancements, the inherent leven-level autoregressive nature of LLMs poses a significant limitation for complex reasoning tasks (Zhao al. 2023), such solving mathematical problems (Ahn et al., 2024) or answering intricate questions (Zhuagg et al. (023). These tasks require sophisticated logical structures and long-term dependencies that go beyond the scope of simple sequential token prediction, leaving considerable gap between current LLM capabilities and the demands of advanced reasoning applications.
Plentiful research has been devoted to enhancing LLM reasoning. On one hand, fine-tuning ap- proaches attain substantial improvements on pretrained LLMs (Zhong et al., 2024 DeepSeek-Al et al. 2025; Team et al. 2025). However, these methods demand massive computational resources and large-scale datasets, being costly to implement. On the other hand, inference-time techniques, exemplified by Chain-of-Thousee
(Wei et al., 2022), Tree-of-Thoughhe of of-Thoughts (Besta et al. 2024). offer a lightweight alternative by enhancing reasoning through predefined external logical structures. While cost-effecttve, their logical structures rely manual design and are task-agnostic, lacking the adaptability to diverse reasoning tasks.
(Yao et al. (203)), and Graph-
'Qianyue Hao and Sibo Li contribute equally to this work.
Yong Li is the corresponding author, email: 1iyog070tss inghua edu.cn

---

Published as a conference paper at ICLR 2026
Addressing such limitations in inference-time techniques presents significant challenges. First, reasoning tasks span various domains, including mathematics, STEM, commonsense, etc., where tasks in each domain exhibit diverse characteristics, making it infeasible manually design logical structures specified for each task. Second, complex reasoning tasks often require multiple steps, where the problem-solving status evolves after each step, requiring dynamic adjustments to the logical structure for subsequent reasoning. Therefore, predefined logical structures fail to adapt to the changes, limiting their effectiveness in stepwise reasoning tasks. These challenges highlight the need for more adaptive inference-ting time techniques to handle reasoning tasks with diversity and dynamics.
Facing these challenges, we introduce ThoughTs (RLoT), framework that leverages rein- forcement learning (RL) at inference time to enhance the reasoning capabilities of LLMs. Specifically, we model long-sequence reasoning as a Markov Decision Process (MDP) and design five human cognition-inspired basic logical blocks potential actions for decision-making. Within the MDP framework, we train an RL agent, namely the navigator model, to dynamically select and combine these blocks along the reasoning process, constructing task-specific logical structures and thereby enhancing the LLM's s ability to handle complex reasoning tasks. We conduct experiments across a wide range of reasoning benchmarks, including AIME (Olympic mathematics), MATH (elementary mathematics), GPQA (STEM), StrategyQA (commonsense), etc. The results demonstrate that our RLoT design outperforms various established inference-time techniques in most cases while being compatible with multiple well-known LLMss, such as GPT, Llama, Qwen, and DeepSeek, Remark- ably, our RL navigator, which contains less than 3K parameters. is able to enhance the performance of sub- 10B LLMs, making them comparable to much larger LLMs with 10x parameters. Moreover, the RL navigator exhibits strong transferability: model trained with one specific LLM on one task domain can effectively generalize to unseen LLMs and tasks without fine-tuning.
In summary, the main contributions of this work include:
We propose RL-of-Thought (RLoT), inference-ting time technique that leverages RL to adaptively construct task-specific logical structures, enhancing LLM reasoning.
We conduct extensive experiments to verify the effectiveness our method to improve L.M reasoning across various tasks. Compatible with multiple widely known LLMss, RLoT outperforms established inference-time techniques by up to 13.4%.
We demonstrate the transfcrability and efficiency of our method, where the trained navigator model can transfer across various LLMs and reasoning tasks. With 3K parameters, it enhances multiple sub-10B L.MSs to be comparable to 10x larger counterparts.
2 PRELIMINARIES
2.1 LARGE LANGUAGE MODELS (LLMS)
Large language models (LLMs) are class advanced neural networks characterized by parameter scales up to billions, primarily trained through next-token prediction objectives. Given a sequence {W1, W2, ..., w; the models output wy to maximize the observation likelihood in the corpus as:
II P(w. W2, 11)
Recent advancements in LLMs, exemplified by architectures like the GPT series (Brown et al. 2020
Kalyan 2023) Achiam al.] 2003), the Llama family (Touvron et al.) 2023: Dubey et al. 2024).
etc, have demonstrated remarkable proficiency across diverse natural language understanding and generation tasks, including semantic parsing, lingull translation (Zhao al. 2023 Chang et al. 2024). Meanwhile, extensive researches integrate inference-time techniques like Chain-of-Thoup
(CoT) (Wei et al. 2022) and Tree-of-Thoughts (ToT) (Yao et al. 2023) to enhance the multi-step reasoning capability of LLMs. On the other hand, fine-tuning strategies leverage Outcome Reward Models (ORM) and Process Reward Models (PRM) to optimize the reasoning process through reward- guided learning (Lightman et al.) 2023 Wang al. 2024c Luo et al.) 2024). These approaches address both structural limitations of auto-regressive decoding and the challenge of maintaining
logical coherence in complex tasks (Xu et al. 2025)
2

---

Published as a conference paper at ICLR 2026
2.2 MARKOV DECISION PROCESS (MDP)
Markov Decision Process (MDP) provides the core framework for sequential decision-making problems. An MDP is mathematically defined by the tuple (S, p, p, , P. Rt, where S is the state space, and E (S)) represents the probability distribution over initial states, with A being the set of all probability distributions over S. The action space is denoted by A. Given specific action taken in particular state, the state transition probability function P : S x A + (SS) and the reward function define the likelihood of transitioning between states and the reward associated with
R S:S x A each action. At each time step i, the agent chooses an action ae E A in state E S, receives reward rt, and transitions to the next state 8t+11 The agent's objective in an MDP is to maximize total accumulated reward over time, which is the sum of the discounted rewards obtained at each step. The cumulative reward at time step t expressed as G = Ek=o 7 retks where is the discount factor that weighs the significance of future rewards.
3
METHODS
OVERVIEW
Legend
PRM response
Self-evaluation state
Navigator
Prompt
response
step tseac supocis
Compstnoa:
Trained module
Frozen module
Compleiity remsinel:
Train & Inference
Train only
RL agent
Action space
*
Selected action:
Risa) 80.53
Decompose
Basic logic blocks
Reason one step
primising phn
Decompose
Feteh problem
Sieph:
Steph--1: decomppse
Dehate
Step1: Debate
StepN: Refine
StepN+1
score step
Refine
PRM reward
Terminate
Reasoning process
S Decompooe probln vihtekk
e Malke
e Resson cuacily 1
plats
Cttck the cuertsi
i Mettmm
Subtask I...
2ubtask....
3ubeask3
Plan I...
Plan2...
Plan3...
c this step colculate...
We aomeat tiese mistakes.
The firel
We suminaiie lle subsada below...
We Find the the best plan glan!....
Reasen one slep
Decompose
Debate
Refine
Terminate
Basic logic blocks (action space)
Figure Framework of RL-of- Thoughts (RLoT). We train an RL agent as the navigator, which dynamically selects and combines basic logic blocks along the reasoning process, constructing task-specific logical structures for each task and thereby enhancing the LLMs' ability to handle complex reasoning tasks.
In this paper, we propose enhance LLM reasoning inference time with RL, namely RL-of- Thoughts (RLoT)). The overall framework of RLoT is illustrated in Figure [ Specifically, we first model long-sequence reasoning as Markov Decision Process (MDP), where we design the action space, state space, and state transition mechanism, making the sequence of decisions within an episode correspond to the generation of logical structure for reasoning (Section Within this MDP framework. we train an RL agent, referred to the navigator model, which conducts sequential decision-making to construct task-specific logical structures, thereby enhancing L.LMs' capability to address complex reasoning tasks (Section 3.3)
3.2 LLM REASONING AS MDP
To leverage the sequential decision decision-making capability RL for adaptively designing logical structures inference time based on problem characteristics, first model long-sequence reasoning

---

Published as a conference paper at ICLR 2026
specially designed MDP. Within this MDP, the sequence of decisions in an episode corresponds to the generation of a logical structure for reasoning. The designs of the state, action, reward, and state transition are follows.
State. The state space is designed to capture how the current solving status of the task is after steps of reasoning, thereby supporting dynamic adjustments to the logical structure for subsequent reasoning. We employ a self-evaluation mechanism to extract concise and informative states during the reasoning process. Specifically, we prompt the LLM itself to evaluate the current reasoning steps from three major and seven detailed aspects, which are listed in Table T
Table 1: Aspects for state self-evaluation.
Major
A: Correctness
Complexity
C: Completeness
I: Correctness of modeling
Bl: Complexity to the final answer
Cl: Closeness to the final solution
Detailed
A2: Clarity for further reasoning
B2: Alternative methods in further reasoning
C2: Completeness within the step
A3: Correctness calculation
For each detailed aspect, we require the LLM to assign a score from to 3, which is then aggregated to form the state of the MDP. This approach summarizes complex reasoning steps into low-dimensional states, offering comprehensive overview of the changing problem-solving status during the reasoning process, facilitating the RL agent in adjusting the strategy for subsequent reasoning accordingly. Please refer to detailed self-evaluation prompts in Appendix K.T
Action. When addressing difficult and complex problems, humans often employ specific cognitive strategies. For example, we break down complex tasks into smaller components and review previous steps when encountering anomalies in the solution. As evidenced by previous research, understand- ing and applying these cognitive strategies significantly enhance the reasoning capabilities of LLMs (Wu et al. 2024 Xue et al.; 2024).
With inspiration from human cognition, design five "basic logic blocks" that can be flexibly combined, constituting the action space in our MDP. By selecting and cascading the blocks, the agent thereby constructs flexible logical structures, paving reasoning pathways from the initial problem to final solutions. In detail, the basic logic blocks include:
Reason one step: Perform reasoning for a single step of the current task, which may not directly lead to the final answer but contributes the overall process.
Decompose: Break the current task into simpler subtasks and execute them sequentially. Then, we prompt the LLM to briefly summarize the results of these subtasks as the final result of this action.
Debate: Generate multiple plans or approaches for the task hand and compare them to identify the most promising one. Then, we prompt the LLM to reason one step further based on the selected plan.
Refine: Review and revise the current reasoning step to improve clarity and correctness.
Terminate: Based on all the previous steps, provide the final answer to the original problem and show it in specified format. This action marks the conclusion of the reasoning process.
We illustrate the structures and detailed prompts for each blocks in Figure I and Appendix K.2
Reward. To evaluate the quality of the intermediate results after the agent selects an action during the long-sequence reasoning process, we employ the Process Reward Model (PRM) to score the intermediate results and set the PRM score of the intermediate result after each action as the single- step reward for this action.
State Transition. In our MDP design, the state transition is straightforward. During the reasoning process, executing specific action based the current moblem-solvvigg state is to prompt the LLM to continue reasoning using the logical structure corresponding to that action. After reasoning, the new problem-solving state is obtained through the aforementioned self-evaluation approach.
Also, we impose a few simple restrictions on the state transition, ensuring the correctness and rationality of the constructed logical structures. First, once the answer is already presented in the
4

---

Published as a conference paper at ICLR 2026
response after executing some action, further actions are permitted except for "Terminate.. Second, the "Refine" action is automatically converted to "Reason one step", when it appears as the first action, refinement to the original problem is needed. Finally, to avoid the reasoning process being too long, we limit the maximum number of actions, and after reaching the limitation, the "Terminate" action will be automatically executed.
3.3 TRAINING OF THE NAVIGATOR MODEL
Within the MDP framework outlined above, given a specific L.M type and a kind of reasoning task, we train the navigator model. Under this formulation, the training process constitutes a standard RL problem within discrete action space. Consequently, our framework is dggorihh-aggnsee for the employment of arbitrary off-the-shelf RL algorithms for training. To enhance the learning for challenging reasoning tasks, extract hard questions from the training set of the target task, ie., questions that the LLM cannot answer when directly prompted. Then, we use these problems for training the navigator model, from which we randomly select one in each episode and repeat multiple times.
allowing
We illustrate the training and inference pipeline of RLoT in Figure 1 During training, hire PRM to provide reward signals. The parameters of both the PRM and the LLM are kept fixed from pre-trained models, and only parameters of the navigator model, i.e., the RL agent, are updated. This significantly reduces the computational cost, making the training process highly efficient. After training, the PRM model is no longer required, and the trained navigator model is used directly. Given an intermediate reasoning state, the navigator selects an action, which is then used to prompt the LLM to continue reasoning using the logical structure associated with the selected action. By repeating this, the navigator model is able to guide the LLM in solving challenging reasoning tasks with task-specific logical structures.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Reasoning Tasks. We conduct comprehensive evaluation of our RLoT method across wide range of reasoning tasks, encompassing benchmarks in mathematics, STEM, and commonsense question answering. For the mathematics domain, we adopt Olympic-level datasets, the AIME24 and AMC23 as well as elementary math datasets GSM8K (Cobbe et al. 2021) and MATH (Hendrycks) al. 2021c). These are widely recognized as representative benchmarks. In the STEM tasks, we test on the MMLU-STEM (Hendrycks et al.] 2021 a) and GPQA (Rein et al., (023) datasets, which span various STEM domains and a range of difficulty levels. To evaluate the commonsense reasoning ability, we employ the Strategy (Geva et al., 202T) benchmark, which presents challenging multi- hop questions across diverse contexts. These benchmarks cover various domains, difficulties, and task types, forming systematic evaluation of the reasoning ability.
LLMs. Our RLoT framework is designed to be independent of specific LLMs, allowing it to be compatible with any off- shelf LLM. To evaluate this, test approach using four repre- sentative LLMs: Qwen2.5- 7B- IIIItructt Qwen2.5-14B Instruct (Yang et al., 2024a). Llama3. Instruct (Dubey al. 2024). GPT-4o-mini (Hurst et al., 2024), and DeepSeek-R 1-Distil-lltl 7B (DeepSeek-AI et al. (025). In this paper, we mainly focus on sub-10B L.Mss, which are often constrained in handling complex reasoning tasks due their relatively smaller size. We expect that our RLoT design can substantially enhance smaller LLMs by adaptively generating task-specific logical structures at inference time, thereby making their reasoning capabilities comparable to. or even exceeding, those of much larger LLMss. Without the need to modify the LLM's parameters, this approach will be fairly computationally efficient. Note that we abandon the results of L Instruct on the Olympic-level datasets since the capability of this base LLM is too limited to solve these challenging problems.
Baselines. We compare RLoT against various baselines designed to enhance L.M reasoning at inference time. First, we evaluate single-round duestion-answering techniques, including direct
5

---

Published as a conference paper at ICLR 2026
question answering (Direct QA), zero-shot Chain-of-Thougeee (Zero-shot CoT), and few-shot Chain- of Thought (Few-shot Co (Wei et al. 2022). For Zero-shot CoT, we employ prompts with "Let's
think step by step", and for Few-shot Col, we include specific few-shot examples for each benchmark in the prompts as outlined in prior work (Yang et al. 2024b Hu et al. 2023 Rein et al. 2023 Wei
al.) 2022). Additionally, Thought (CoT-SC) (Wang et al. (023) and
consider multi-round techniques, including celf-consittent Chain-of-
(ToT) (ao et al. (023). Following the original settings, perform majority voting across four reasoning samples in CoT-SC, and we implement a logical tree with two layers and five nodes per layer in ToT. These multi-round approaches facilitate comparison and voting across diverse reasoning paths, thereby enhancing the reasoning capability of LLMs in complex tasks.
4.2 TRAINING DETAILS
In implementation, the navigator model is simple three-layer multilayer perceptron (MLP) with the Dueling Network architecture (Wang et al. 2016). The model merely contains 2,566
parameters in total, where the lightweight design ensures efficient training and inference. We employ the Buable-Dueling-DQ algorithm (Mnih et al.) 2015 Van Hasselt et al. (0116) Wang et al. 2016) for optimization. By integrating Double Q-learning mitigate value overestimation and the Dueling architecture to separate state and advantage representations, this algorithm significantly improves stability during the training process. We train the navigator model for 3,000 episodes with Qwen2.5- 14B 1-5-144-Innttii Instruct on the MATH benchmark, where the learning curves presented in Figure 2 and Appendix A indicate strong convergence. Furthermore, in Appendix A.. we investigate the impact of employing alternative RL algorithms and extending training episodes, where the results consistently demonstrate similar convergence.
5-14B-instrucQ@MATH
Qwen2.5-
0.5
0.50
0.4
Reward
0.45
0.3
0.2
Episode
Figure 2: Learning curves during RL training of navigator models.
To obtain reward signals for actions, utilize the Math-Shepherd as the PRM (Wang al. (024c). This model is trained by previous researchers using automatically constructed process-wise wise su- pervision data, redueing heavy reliance manual annotation and thereby achieving remarkable performance. For reproducibility, please refer to more detailed settings and the illustration of the whole pipeline in Appendix
4.3 OVERALL PERFORMANCE
We use the obtained navigator model enhance the reasoning of multiple LLMs across different reasoning tasks. Following existing works like CoT-SC, we perform multiple repeated inferences for each task and filter out the trails that do not meet self-consistency, enhancing the robustness of reasoning, where the results are presented in Table 2 For the bascline performance,
prioritize
using the results reported the official technical reports each LLM (Yang et al. 2024a] Dubey et al. 2024: Hurst et al. 2024), if available. Otherwise, evaluate the performance through own experiments following the settings established previous works (Fu et al., 2023 Yang et al. 2024b
Yao et al. 2023).
The results show that our method performs well, consistently outperforming the established inference- time baselines across almost all reasoning tasks when combined with various LLMss. Notably,

---

Published as a conference paper at ICLR 2026
Table 2: Overall evaluation of RLoT's capability enhance multiple LLMss reasoning across different tasks. The bold numbers indicate the best performance in each group of experiments, and the underlined numbers indicate the best baseline method.
LLM
Method
Olympic math
Elementury muth
STEM
Commonsensse
Average
AIME24
AMC23
MATH GSM8K
GPQA
MMLU-STEM
StrtegyQA
Direct QA
13.33
62.53
Zero-shot CoT
16.67
62:50
78.56
94.23
38. 38.39
85.63
75.98
64.57
Owen2.5-14B-lnsteee
Few-shot CoT
6.67
55.00
80.00
94.80
45.50
85.06
78.60
63.66
CoT-SC
6.67
47.50
$0.04
94.08
45.54
86.71
80.06
62.94
ToT
10.00
55,00
79.50
93.78
45,08
86.55
78.17
64,01
(ours)
$0.38
51.34
Direct QA
10.00
42.50
74,64
91.58
31.25
80,94
68.85
57.11
Few-sbot CoT
13.33
45.00
75.50
91.60
36.40
81.16
74.38
59.62
Qwen2.5-7B --Issstruct
CoT-SC
6.67
ToT
13.33
50.00
73.80
$1.33
36.60
82.62
70.45
39.74
RLoT (ouns)
23.33
60.00
76.70
92.87
44.64
85.06
79.04
63,95
Direct QA
49.12
84.83
32,
71.58
70.74
61.68
Zero-shot CoT
49.74
85.37
32,80
72.85
73.07
62.77
Llama3.1 BB-Instruct
Fow-sbot CoT CoT-SC
48.52
84.50
33.03
70.66
72.05
61.75
51,74
87.04
33,48
73.29
78.8
64.89
RLoT (ouns)
56.56
90.07
46.88
80.56
84.42
71.70
Direct QA
13.33
76.58
93.33
61.17
Zero shot CoT
6.67
67:50
76.76
93.93
40.20
85.76
78.17
64.14
GPT-fo-mini
Few-shot CoT
6.67
57.50
93,448
06
62.13
CaT-SC
6.67
45.00
76.84
93.66
46.42
86.55
82.53
62.52
TrT
6.67
50,00
74.30
93.33
14,42
K5,92
76.42
61,58
RLoT (ours)
20.00
70.00
77.36
93.86
54.02
88.23
82.68
69.45
Direct QA
46.67
60.00
92.27
95.74
54.47
83.61
79.45 19.48
73.18
Zero CoT
$3.33
62.50
91.48
95.20
53.13
85,28
89
74.26
DeepSeek-Rt R1-
Few-shot CvT
56.67
67.50
92.54
94.38
56.47
87.47
79.33
76.34
Distil Qwen-
ToT
50.00
55.00
95.18
94.54
55.13
86.55
80.91
73.90
RLoT (ours)
63.33
96.56
82.92
approach brings about substantial improvements in the GPQA benchmark, which is a challenging task where LLMs generally perform poorly. Specifically, when implemented with Llama3. Instrrctt we achieve a 13.4% tasks. Meanwhile, performance boost. Among the baselines, CoT--CC performs the best across most find that despite being more complex in design, ToT performs poorly on many reasoning tasks, which is also reported in previous studies (Wu et al. 2024 Zhang et al. 2222aa] Qi et al. 2024). Furthermore, we report the computational overhead and implementation latency of RLoT in Appendix C and also compare RLoT with more recently proposed test-time scaling baselines in Appendix By directly generate specific logical structures for each question without requiring search-and-tril our method reached the best performance while maintain a low cost.
4.4 PARAMETER SIZE EFFICIENCY
In this section, we demonstrate the parameter size efficiency of our RLoT method in enhancing the reasoning capability of sub-10B LLMs, making them comparable to LLMs with several times more parameters. Specifically,
select three 10B LLMs, including Qwen2.5-14BBInne Instruct, and GPT-4o-mini and their respective large-scale counterparts. It is worth noting that the parameter size of GPT-4o series models was estimated in previous studies (Abacha et al.) 2024).
Llama3. 1-8B-
In Appendix D we present the performance of these models across various reasoning tasks using Few-shot CoT, which the standard technique commonly used in official technical reports Yang
et al. 2024a] Dubey et al.) 2024 Hurst et al.) 2024) and previous studies (Liu et al., 2024 ret 2024 Tran et al. 2024; Kumar et al.) 2024 Yu et al.) 2024) in evaluation of LLMs. We also show the performance of the 10B LLMs after enhancement with RLoT. The results indicate that our RL-based navigator, which contains fewer than 3,000 parameters, significantly enhances the performance of sub-10B LLMs. making them comparable to much larger counterparts with around 10x more parameters. Specifically, RLoT empowers the sub-100 10B LLMs to comparable to, compensating most of the performance gap, surpassing their larger counterparts, demonstrating its efficiency.
7

---

Published as a conference paper at ICLR 2026
4.5 TRANSFERABILITY
To better illustrate the transferability of our navigator model, we conduct further experiments regarding transferring across different LLMs and reasoning tasks, respectively.
Transfer across Different LLMs. To verify the transferability of RLoT across different LLMs, respectively train navigator models with three different LLMs, namely Owen2.5-14B 1wen2.5-14B-Instructt, Llama3. I-ssruuut and GPT-4o-mini, on the MATH benchmark. Without any fine-tuning, we cross-test the obtained navigator models to enhance other LLMs the MATH benchmark and present the results in Table 3
Instruct,
The results indicate that the trained navigator model exhibits strong transferability across different LLMs. When implementing the navigator model to enhance the reasoning capabilities of a specific LLM, we find that, regardless of whether the navigator model trained on the same LLM different one, its performance remains consistent. In all cases, the enhanced LLM outperforms the best well-known inference time baseline.
Table 3: Evaluation of RLoT's transferability across different LLMs. We train navigator models with three different LLMs the MATH benchmark and cross-test the obtained navigator models with other LLMs. We also list CoT-SC, the best baseline method, for comparison.
Test
Method
Train
Swee2.5-144B-Imss @MATH
Llamai3 I-St-Instruct
GPT-4o-mini @MATH
@MATH
Qwen2.5- -B--nnttucttMMaaa
80.38
56.56
77.36
RLoT (ours)
Llama3. 1--8---Issttuu IB-srucct MATH
81.48
53.60
78.14
GPT-4o-mini@l MATH
80.84
56.94
78.08
CoT-SC
80.04
51.74
76.84
Transfer across Different Reasoning Tasks. To verify the transferability of RLoT across different reasoning tasks, we respectively train navigator models with Qwen2.5-14B-Ing Innt-tttttt on three different benchmarks, namely MATH, GPQA, and StrategyQA. Without any fine-tuning, We cross-test the obtained navigator models to enhance the reasoning capabilities of Qwen2.5-14B- 1nB-truutt on the other tasks and present the results in Table
The results indicate that the trained navigator model owns strong transferability across different reasoning tasks. When utilizing the navigator model to enhance the reasoning capabilities of LLMs for specific task, we observe that, regardless of whether the navigator model is trained on the same task or a different one, performance remains largely consistent. In most cases, surpasses the best-performing inference-time baseline.
Furthermore, we find that the navigator models trained on mathematical (MATH) and STEM (GPQA) problems exhibit better transferability to each other. However, the transferability between the navigator trained on commonsense problems (StrategyQA) and the former two is relatively limited, which is intuitive given the inherent relations and differences between domains of mathematics, STEM, and commonsense reasoning.
Table 4: Evaluation of RLoT's transfcrability across different tasks. We train navigator models with Qwen2.5-14B wwen2.5-14B-Instruee three different tasks and cross-test the obtained navigator models other tasks. We also list CoT-SC, the best baseline method, for comparison.
Test
Method
Train
Qwen2.5. J-I4--nstett Qwen2.5- 1-12225---B--ttrroe @GPQA
Owen2.5-14B-Inon
MMATHH
@StrategyQA
Qwen2.5-14B 3--14B-Instructe MATH
80.38
51.34
RLoT (ours)
Qwen2.5. 1-B.5--11B-Insuuue @GPQA
80.76
53.57
80.64
Qwen2.5-144 14B Instruct Stratteyy
79.94
50.22
81.37
CoT-SC
80.04
45.54
80.06
8

---

Published as a conference paper at ICLR 2026
Table 5: Typical patterns in the task-specific logical structures generated by RLoT ("Reason" is short for "Reason one step").
Task
MATH
GPQA
StrategyQA
Instance number
5000
448
687
Reason-Refine
Reason-Refine
Reason-Debate
Major 2-step pattern
Reason-Decompose
Reason-Debate
Reason-Refine
Cesompose-Refine-Raase Reason-Reffie-Dee Debate
Reason-Refine-Debatte
Season-Deccmpose Debate
Major 3-step pattern
Reason--eecomppos--Reiii
Reason-Refine-Dcchat
4.6
TYPICAL REASONING PATTERNS
The experimental results above have demonstrated RLoT's capability to enhance LLM reasoning with task-specific logical structures. In Table 5 we summarize typical patterns observed in the logical structures generated by RLoT when solving tasks from the MATH, GPQA, and StrategyQA benchmarks using -wen2.5-14BBInnsnree
From the 2-step patterns, we observe that the Reason-Refine mode emerges frequently. Particularly in MATH and GPQA, which require massive mathematical calculations, this pattern compensates for the LLMs' relatively poor calculation abilities, leading to more reliable results, In the 3-step patterns, operations like Decompose and Debate are frequently employed, which help break down challenging problems or facilitate discussions to explore potential solutions. Additionally, the Refine operation is often used before and after the Decompose and Debate steps, ensuring correct integration with preceding and succeeding reasoning processes. These typical reasoning patterns exhibit strong interpretability, further validating the capability of RLoT generate task-specific logical structures that enhance LLM reasoning, For better understanding, we provide an example in Appendix B.] illustrate how these logical structures empower the correct solution problem.
4.7 ABLATION STUDIES, ANALYSES, AND DISCUSSIONS
To better justify our designs, we verify the effectiveness of our logic block designs with ablation studies. To examine the impact of each block, we remove each of them and train navigator, where results in cobler0 that all blocks are effective.
Table 6: Ablation study results for each logic block.
LLM
Ablation
GPQA MATH
Strategy
Average
w/o Decompose
75.42
31.92
77.00
w/o Debate
74.02
36.61
77.58
62.74
-went..--nnttuut-7B
w/o Refine
75.76
41.29
72.93
63.33
Full RLoT
76.70
44.64
79.04
66.79
w/o Decompose
76.08
41.74
79.33
65.72
w/o Debate
74.68
45.31
78.75
66.25
GPT-4o-mini
w/o Refine
76.44
51.56
74.38
67.46
Full RLoT
77.36
54.02
82.68
71.35
Also, we conducted a series of analyses, and discussions. We quantify the contribution of each logic block in Appendix[[ We discuss the reliability of the self-evaluation state design in Appendix [F] We analyze the necessity employing RL to train the navigator model in Appendix G We illustrate the role of the process reward model (PRM) in training the navigator in Appendix

---

Published as a conference paper at ICLR 2026
5 RELATED WORKS
5.1 INFERENCE TIME REASONING OF LLMS
To improve LLMs' reasoning capability, plentiful research has investigated inference- time techniques without the need for model updates. On the one hand. predefined external logical structures are widely applied as basic solutions. Most notably, Chain-of-Thouggh (CoTT) (Wei et al. 2022) incorporates intermediate reasoning steps within the prompt, enhancing the model's abilities in complex tasks. As a subsequent advancement, CoT with Self-Consistency (CoT-SC) (Wang et al. 2023 further refines CoT by generating multiple reasoning chains and filtering out those that do not meet self-consistency, thus increasing the reliability. Moreover, the concept of Tree-of- Tree-of-Thoughht Thoughts (ToT) (Yao et al., 2023) and Graph-of- Thoughts (GoT) (Besta et al., 2024) has been introduced, where the reasoning process is represented as a graph, enabling exploration and backtracking from more promising outcomes. Despite their success, these methods rely on task-agnostic logical structures that applied uniformly across diverse tasks, lacking flexibility.
On the other hand, recent researchers have proposed more adaptive inference-time approaches. From hiring the eecooposs--nnalye--ree Search discover more effective logical structures (Wu et al. 2024 Wang al. 2024a), these approaches dynamically compose appropriate logical structures for specific tasks to enhance the reasoning performance. However, the cost, limiting the efficiency of reasoning. In contrast, the proposed RLoT method employs RI to train a navigator agent. With the trained navigator, our design can directly select and combine basic logic blocks and generate task task-specific logical structures, enabling more adaptive reasoning while eliminating the searching cost.
procedure (Xue et al. (024) to utilizing Monte Carlo Tree
design incurs massive extra computational
5.2
RL AND LLMS
RL has become vital in the development of LLMS (uu et al. 2025 Hao et al. 2025). One important direction is aligning L.LMs with human preferences, where the key method is Reinforcement Learning from Human Feedback (RLHF) (Christiano et al.) [0177] Ouyang et al. 2022). In RLHF, LLMs are fine-tuned as actors in RL based on feedback derived from human preferences. Recently, extensive helpful and harmless LLMs have been created using RLHF (Bai et al., 2022: Casper et al. 2023). ).
Beyond this, RL is also applied enhance the reasoning capability of LLMs. In this context, reward signals are derived from Outcome Reward Models (ORM) (Kazemnejad et al.
2024) or Process Reward Models (PRM) (Lightman et al., 2023] Wang et al., 2024c) [uo et al.) 2024), providing
feedbacks for the LLMs' reasoning process during fine-tuning. By applying RL to LLMs with these
rewards, LLMs can iteratively improve their performance in multi-step reasoning tasks (Havrilla
ct al. 2024a b Shao et al. 2024 DeepScek-AL Al et al. 2025).
In summary, existing RL techniques have significantly enhanced LLMs' capabilities by updating model parameters, yet such fine-tuning demands substantial computational resources when applied to each pre-trained LLM. In this paper, the proposed RLoT method applies RL at inference time, leveraging the power of RL to train a lightweight navigator rather than the entire parameters of LLMs, our method achieves low cost and wide compatibility with various pre-trained LLMs.
6 CONCLUSIONS
In this paper, we propose RL-of-Thoughhs (RLoT), an inference-ting technique that utilizes RL to train a navigator model, which adaptively constructs task-specific logical structures, and thereby enhances the reasoning capabilities of LLMs. Through extensive experiments across various bench- marks, we demonstrate the effectiveness of our method in improving the reasoning capability of various widely known LLMs. Additionally, we show the strong transferability and efficiency of our approach, where the trained navigator model can effectively transfer across different LLMs and unseen reasoning tasks. With fewer than 3K parameters, our navigator model enables multiple sub-10B LLMs to attain performance comparable to larger LLMs with up to 10 times the parameter size. Our work highlights the potential of RL at inference time in enhancing the reasoning capabilities of L.LMs, paving the way for more adaptive and efficient LLM reasoning in the future.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This study uses fully open-source or publicly available models and datasets, adhering to their respective licenses. All resources are properly cited in Section [ and Section The selected datasets and models are well-established, representative, and free from bias or discrimination.
REPRODUCIBILITY STATEMENT
For Reproducibility, we describe the general experimental settings in Section we list the imple- mentation details in Appendix D and our source code are anonymously open source at https t //github. www.co//ttsitaaaaeeeoooooceeesee nnghua fib- aaa/b/// RL- LLM Reasoning
ACKNOWLEDGGENNTS
This work was supported in part by the National Key Research and Development Program of China under 2024YFC3307500, Tsinghua-Toyy Toyota Joint Research Institute ditei-disciplinaryy Program, and Beijing National Research Center for Information Science and Technology.
REFERENCES
Deciphering and enhancing commonsense reasoning in Ilms from the perspective of intrinsic factual knowledge retrieval. 2024. http:///pperreviewnee///ooo wtt.:///o/ooummmmdmMbaaa (oddMMbtAnno8y:
Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. Medee: A benchmark for medical error detection and correction in clinical notes. arXiv preprint arXiv:2412.112660 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprini arXiv:2402.00157. 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmano, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving claborate problems with large language models. In AAAI, pp. 17682-17690. AAAI Press, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. NeurIPS, 2020.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.152117 2023.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1-45, 2024

---

Published as a conference paper at ICLR 2026
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems. 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang. Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang. R. Chen, R. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie. Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun. Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan. Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL httptttss////aiiieaataa wwwp//v.orrreeeooo abs///b/////// 12948)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance. arXiv preprint 1xXi:23305.17300 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL), 2021.
Haixia Han, Jiaqing Liang, Jie Shi, Qianyu He, and Yanghua Xiao. Small language model can self-correct. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp- 18162-18170, 2024.
Qianyue Hao, Jingyang Fan, Fengli Xu, Jian Yuan, and Yong Li. HIm-cite: Hybrid language model workflow for taxt- scientific citation prediction. arXiv preprint arXiv:2410.09112. 2024.
Qianyue Hao, Lin Chen, Xiaogian Qi, Yuan Yuan, Zefang Zong, Hongyi Chen, Keyu Zhao, Shengyuan Wang, Yunke Zhang, Jian Yuan, and Yong Li. Reinforcement learning in the era of large language models: Challenges and opportunities. researchgate, 2025.
12

---

Published as a conference paper at ICLR 2026
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi. Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642 2024a.
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. Glore: When, where, and how improve Ilm reasoning global and local refinements. arXiv preprint arXiv:2402. 10963, 2024b.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.038// 2021c.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy Chen, Shafiq Joty, and Furu Wei. Preference optimization for reasoning with pscudo feedback. arXiv preprint arXiv:2411.16345, 2024.
Katikapalli Subramanyam Kalyan. A survey of gpt-3 family large language models including chatgpt and gpt-4. Natural Language Processing Journal, pp. 100048, 2023.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint 1rXiv:2310.03714, 2023.
Shanu Kumar, Saish Mendke, Karody Lubna Abdul Rahman, Santosh Kurasa, Parag Agrawal, and Sandipan Dandapat. Enhancing zero-shot chain of thought prompting via uncertainty- strategy selection. arXiv preprint arXtv:2412.00353, 2024.
Xiaochong Lan, Yiming Cheng, Li Sheng, Chen Gao, and Yong Li. Depression detection on social media with large language models. arXiv preprint arXiv:2403. 2024a.
Xiaochong Lan, Chen Gao, Depeng Jin, and Yong Li. Stance detection with collaborative role-infused llmebased agents. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pp. 891-903, 2024b.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 2023.
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acemath: Advancing frontier math reasoning with post-train post-training and reward modeling, arXiv preprint 1rXiv:2412.15084. 2024.
Liangehen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng. Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06677 06592, 2024.
13

---

Published as a conference paper at ICLR 2026
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedbaace Advances in Neural Information Processing Systems, 46594, 2023.
Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using Ilms to zero-shot check their own step-by-step reasoning. In ICLR. OpenReview.net, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):52--533 2015.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Zhenting Qi. Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller Iims stronger problem-solvers. arXiv preprint arXiv:2408.06199 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint 1.2222, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang. Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du Chonghua Liao, Chuming Tang, Congcong Wang, Dehao Zhang. Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang. Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma. Shupeng Wei, Sihan Cao. Siying Huang. Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He. Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang. Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with llms, 2025. URL htttt: /ttp///aavvv///////ai//aa./...aa org/ / abs /501 12599
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
Hieu Tran, Zonghai Yao, Junda Wang, Yifan Zhang, Zhichao Yang, and Hong Yu. Rare: Retrieval- augmented reasoning enhancement for large language models. arXiv preprint arXiv:2412.02830, 2024.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforeement learning with double q- learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Dian Yu, Haitao Mi, Jinsong Su, and Dong Yu. Litesearch: Efficacious tree search for 1lm. arXiv preprint arXiv:2407.003320, 2024a.
Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q Im- proving multi-step reasoning for llms with deliberative planning. arXiv preprint arXiv:2406. 14283, 2024b.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Iims btep-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9426 9426-9439, 2024c.
14

---

Published as a conference paper at ICLR 2026
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR. OpenReview.net, 2023.
Yingxu Wang and Vincent Chiew. On the cognitive process of human problem solving. Cognitive systems reseurch, 11(1):81-92, 2010.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models, Advances in 2022.
neural information processing systems,
Jinyang Wu, Mingkuan Feng, Shuai Zhang, Feihu Che, Zengqi Wen, and Jianhua Tao. Beyond examples: High-level automated reasoning paradigm in in-context learning via mets. arXiv preprint arXiv:2411.18478, 2024.
Fengli Xu, Qianyue Hao, Zefang Zong. Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprini wrXiv:2501.09668 2025.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of I1m performance in machine translation. arXiv preprint arXiv:2201.088: 004111 2024.
Shangzi Xue, Zhenya Huang, Jiayu Liu, Xin Lin, Yuting Ning, Binbin Jin, Xin Li, and Qi Liu. Decompose, analyze and rethink: Solving intricate problems with human-like reasoning cycle. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemment arXiv preprint arXiv:2409. 1ati:22499.1122.... 12122, 2024b.
Cheng Yang. Chufan Shi, Siheng Li, Bo Shui, Yujiu Yang, and Wai Lam. Llm2: Let large language models harness system 2 reasoning, arXiv preprint arXiv:2412.20372. 2024c.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang. Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: Thought-augmennted reasoning with large language models. Advances in Neural Information Processing Systems, 37:113519 2024d.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. NeurlPS, 2023.
Xiaodong Yu, Ben Zhou, Hao Cheng, and Dan Roth. Reasonagain: Using extractable symbolic programs to evaluate mathematical reasoning. arXiv preprint arXiv:2410. 190566 2024.
Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Lllam--brry: Pairwise optimization for ol-like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884, 2024a.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.1077 2024b.
Zhenru Zhang, Chujie Zheng. Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXivv2501.07301, 2025.
15

---

Published as a conference paper at ICLR 2026
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303.182233 2023.
Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai ol: Opportunities and challenges of agi. arXiv preprint arXiv:2409.
2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. ICLR. OpenReview.net, 2023.
Yuchen Zhuang, Yue Yu, Kuan Wang. Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools. Advances in Neural Information Processing Systems, 36: 50117-50143, 2023.
16

---

Published as a conference paper at ICLR 2026
A TRAINING DETAILS
A. LEARNING CURVES
In Figure 3 show the learning curves during RL training of all navigator models used exper- iments in the main text. We use sliding window averaging to smooth the reward, and from the learning curves, we observe good convergence the RL training.
Qwen2. 314B-Inttruct@MATT
Qwen2.5-1466 2.-22-1---.sstuuuttttttteeee
0.50
Reward
0.4
0.3
0.35
0.30
500
1900
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Episode
Episade
GPT-4o-mini@MATH
Meta Letta-Llama-33 3.1 3.1-8B.nm in..trruut@@
0.5
0.4
0.4
Rewarr
0.3
0.3
0.1
1000
1500
2000
2300
3000
1000
1500
2000
2500
Episode
Episode
Figure 3: Learning curves during RL training of navigator models.
A.2 ANALYSIS
In Figure we extend the training duration to 5,000 episodes. We observe that the model essentially reaches convergence around 3,000 3,000 episodes, with only marginal performance gains achieved thereafter. Additionally, we explore training the navigator model using the Double-DQN algorithm. While the training stability decreases slightly due the absence of the Dueling network architecture, the model ultimately converges to a performance level comparable to that of the Double- Soooll--Deiing-QO..
owen2.5 insTr Double Double-Dueling DON
qwen2.5. 14b-instrurtgh MATH, Double DON
0.50
0.50
Reward
0.45
0.42
0.40
0.35
4000
Episode
Episode
Figure 4: 5k episodes learning curves during RL. training with different algorithms.
17

---

Published as a conference paper at ICLR 2026
B CASE STUDY
B. CORRECT CASES
Problem: How many square units are in the region satisfying the inequalities Sy Ige |x/5 and Sy 'le x/--33
Few-shot CoT
RLoT
(CuT Esexpieo)
Problcs (Prokics)
Frbbian: (Proiie))
Plone nouse reantly UNE mere step toad coment drp hen,
Pleaie genentte (snnwer)).
probien. Whep the
box raay skgn
Carainll lefs stan onaly giver inequlliiss
To rind the ans of region snidiingg y the inopailtiie 'ge 145 and Vo-all + 15, we ueed undyee boundacee defased thas inoquatiios fed insmrcttion poirre ohe line, solie exputions Sy
The not dap is deterite the painn oftammatioo. For v ta i H4 a.l. l.. For i + Now or know the peiro inrrmnaccion and (ann mee STG IoserS ligat
m-a St...
las
Coneect elachaaic caloulation.
check refise father
DONOT croduct
internat quadriantrrll with vartices 315n1, trenntree lite (3) (Brlggeed sigggg $60.50 505 Statfglnee (91 aff (Brvighnn Thi- quadiiataral ormmetic about the yours and can drrded asso two conameent right trengles
calculation
Your anages so far is comeet..
non sny world indeed be palcalioe the oft oft region brunded doca lasa Ilis ession forn dianesed chombaaa issps contered ong's
Basa imanal bava base Siiee an ana bispal B11 The ancs
Woree ceu undersanninn.
Caneet gounctic snderstaniig..
Thes,
Plose thmee different altrraaivv plans de skyp.
prablam
Plail Use th forrela & aroa dhocllos.
Intogats betweee curvec.
regine miiegfes.
nee permurg Pontr Giver dagrrrttttttttteeo momeir
D- 1--15 (.5)
1 8--00 (1..
pote caksdate dac
mathod
Thas deaae of regien saiifrin the inequaliio Vi' ae (0 My 2a giss Weesure umant
01
Figure 5: A case study comparing few-shot CoT and RLoT representative problem in the MATH dataset. Results are both generated by 1ren2.5-14B-Insse
As shown in Figure analyze representative case to illustrate the detailed working procedure of RLo'T. The problem sampled from the test set of MATH (Hendrycks et al. 2021c) benchmark, which involves calculating the area between two functional curves, requiring combination of algebraic and geometric knowledge. While the Few-shot CoT method attempts to generate the answer in one step. RLoT adopts a task-specific reasoning pathway, which includes Crason-RReiie-Debule typical pattern for solving complex math problems shown in Section 4.6)
Both Few-shot COT and RLoT successfully calculate the intersection points of the curves. However, the response of Few-shot CoT fails to accurately identify the shape of the shadowed area, leading to incorrect bases for the triangles and, ultimately, an incorrect result. In contrast, RLoT utilizes the Refine action to clarify the intermediate results and correctly identifies the shadowed area as rhombus. Subsequently, the Debate action proposes multiple plans and prioritizes directly using the area formula for a rhombus over dividing it into triangles. By avoiding unnecessary geometric division, RLoT achieves the correct answer.
B.2 FAILURE CASES
To provide comprehensive understanding of the boundaries of RLoT, we also analyze typical failure cases observed during our experiments. Common failures can be categorized into two primary modes: (1) Navigator input errors caused by self-evaluation noise, and (2) Navigator output errors caused by suboptimal action selection.
As shown in Figure 6 failure happens due to self-evaluation noise. The robustness of the navigator depends partly on the accuracy of the state representation derived from the LLM self-evaluation. In this case, the LLM generates an incorrect equation (wrongly deriving 34 = 24p 36) but fails to identify this error during the self-evaluation phase, hallucinating a "True" status for correctness of modeling. This noise in the state input misleads the navigator: instead of selecting the necessary "Refine" action to correct the mistake, the Navigator perceives the current step as valid and selects "Debate" to explore alternative plans.
18

---

Published as a conference paper at ICLR 2026
Problem: polynomial Sx"3 3x*2 4x factor Sx19 px"6 qx r.S Enter ordered triple S(p.q.r).S?
State self-evaluation error
Problem: (Problen).
ONE
DONOT stops
pot
peysionial per ge KPoo Salving
equation (el) tomns of Ketk By 24p Bp
Wrong equiion.
(Problem and ressoning steps).
Please evaluste the corzent step from the following aspects.
Commancee modkling Trun, Bud state evalurtion.
Altentalivv metieds Buther osasoning True
Here is problem and several reasoting: (Prblle and previous stops). Camantly have sever alemrativ for solving the problem the
dholld be Refine, hus bad saate coalantion liads Debate,
Figure 6: A failure case study that is caused by wrong state self-evaluation.
As shown in Figure 7 failure happens due to action selection error, where the navigator selects a suboptimal action despite receiving accurate state information. this case, the LLM derives a correct intermediate result (2os(21//7) + -1/2) () and correctly evaluates it as "Truc". However, the navigator triggers a "Refine" action instead of proceeding with reasoning or termination. This unnecessary intervention forces the LLM modify correct step, leading to "over-correctton" where valid reasoning is replaced with an erroneous identity, ultimately causing the solution to fail.
Problem: Compute
"[ccos"3 Vftac (2 (pi) (7) \cos"3 Vfruc (4 'pi) [7) \coos" Mfrac (pi) [7).]
Action selection error
Protiem: (Problem)
Pizaa exarlly ONE mare Iscal on the cumant her, and
DUNOT
Therefore, (2yp)
Correct equation.
reasoning seps).
evaluate cureat step from following aspect..
Cocrectness modelnng: Trae...
ovaliation.
Altemative methods further reasoting: Palse
Hore is problem and soveril novaming: (Preblem and provicne supps) Please check and refine the currert though there, and DONOT conduct further ressoning calculaioo.
Nathing wrong in the cureent but Refine is sclected.
Thersfon, have:
Tooooooda (os T0l Thus, Jabe))
the
Figure 7: A failure study that is caused by wrong action selection.
While these failure modes exist, our extensive quantitative results demonstrate that RLoT achieves performance gains over baselines, indicating that these failure cases represent minority of the inference trajectories. In our design, the RL training process effectively optimizes the navigator to minimize action selection error,, and the structured self-evaluation prompt is designed mitigate state noise errors as much as possible.
19

---

Published as a conference paper at ICLR 2026
C COMPUTATIONAL OVERHEAD AND LATENCY
As shown in Table 7 RLoT improves the reasoning ability of LLMs with a low cost comparing with complex inference-time techniques. The results show that at inference-time, RLoT's token consumption only accounts for 78 % of ToT, and the consumption is comparable to CoT-SC. This because the trained navigator can directly select appropriate logic blocks without searching like ToT, improving the performance and reducing cost. Besides, our navigator with <3k parameters has almost negligible inference cost itself.
Though RL training incurs extra costs, our lightweight navigator makes training efficient, and the trained navigator can transfer across tasks, diluting the cost. Specifically, our navigator is trained for 3000 episodes, and each episode consumes an average of 5004 input and 1448 output tokens. Shared across all test questions, it only adds less than 5% input and output tokens per question.
Table 7: Token consumption per question across tasks. The bold numbers indicate our method.
Madd
Mothed
AJME34
AMC23
MATH
GSMRK
CPQA
XMLLUSSEM
Stratego
Avcrae
Trget
Ouiput
Inpat
Oslpst
Ingeri
Output
Ingest
Output
Direci QA
653
1453
535
1131
-5-
352
55
183
1.5
-60
210
30
259
235
577
Zenoakst
1955
1341
1921
466
336
1559
25.3
CoT.SC
7835
5350
7685
4109
2014
1330
4339
543
6385
1945
4442
1059
1556
897
4884
2169
TuT
5504
10844
3835
9020
2713
6227
3510
4312
5063
7234
4707
5365
5760
6401
500]
0950
Tirect OA
66
1714
611
1417
44
366
58
182
159
516
-34
13
275
233
705
Zem-shit OiT
667
1985
463
1180
47
385
78
181
532
41.3
33.
401
665
Qv Inszuct
16890
7131
15341
1933
1843
5500
1612
1351
77.6
ThT
1380
9150
128
7631
6575
7534
5767
5301
5771
5109
1053
1560
1937
3013
5163
6134
RIaT [aum]
20111
41008
13876
2096
1437
587
476
3112
559
414
1738
669
6235
1466
Dereci UA
-44
431
199
135
811
13
2011
71
407
Zem-itte Crll
73
455
64
215
175
56.1
91
365
20
415
85
431
Llara3 Intruct
Fice-dae Eni
510
412
1101
173
1765
158
417
222
311
Inl"
35714
4.327
5212
4750
35.5
5794
5351
3150
41230
4858
4790
4250
Klal Tsun)
4451
1981
2942
682K
9518
3587
440
1486
3849
1894
4920
1795
Direct
1350
571
404
223
165
357
175
13
201
Zeroshet CuT
614
1455
585
1192
50
422
08
390
160
525
50
326
16
382
236
(40
GPT--o-niiii
Few-lax Cul
Cat-3C
1853
1965
1823
1202
2030
513
406
1107
205
1575
555
1119
215
3601
409
254
1200
602
4020
5735
5649
5175
5541
1467
RLaT leurs)
26529
4779
19515
3973
3453
1575
3869
1948
4372
1890
1982
788
1612
675
9637
2044
Direst CiT
616
Dontl-poee--
Desphickk-tt
Poee-ber CoT
Carse
3100
1360
13135
1778
3096
31899
5180
5130
7288
1438
865
19071
1901
1867
13606
5353
545
1305
17583
1750
831
30555
5196
1881
2063
2012
7830
1550
4191
1197
050
5061
1381
1315
4873
501
1905
RIaT
13556
21878
15684
31915
4971
6050
8347
12384
1750
5415
10546
10881
Aerrage
Firec 0A
0A4
4074
656
2131
144
1305
151
430
253
1535
658
103
462
508
Foundee due
2600
3068
3455
755
1050
1129
563
565
474
Colsal
NG44
13440
9719
M231
3135
5835
0982
1800
$314
3001
2111
2111
2234
1893
6412
3321
5450
14992
4654
4543
1933
5429
5421
3250
6192
5225
3307
4295
REaT plas training coel
21649
6779
16863
18664
3782
2867
.934
3317
5461
5522
3749
2509
3392
2637
6485
5174
Besides token consumption, we show the average solving time (min) per problem along with the standard deviation across repeats in Table S
Table 8: Average solving time (minutes) per problem across various benchmarks. Standard deviations
from 3 repeats are shown in parentheses.
Method
AIME24
AMC23
MATH
GSMSK
GPQA
MMLU-STEM
StrategyQA
Average
Direct QA
0.44(0.02)
0.25(0.01)
0.13(0.00)
0.05(0.00)
0.17(0.01)
0.08(0.00)
0.05(0.00)
0,17(0.01)
Zero shot CoT
0.44(0.02)
0.25(0.02)
0.13(0.1)
0.06(0.00)
0.17(0.02)
0.08(0.00)
0.06(0.00)
0.17(0.01)
Fow-shot CoT
0.52(0.03)
0.42(0.02)
0.170.01)
0.18(0.01)
0.33(0.02)
0.17(0.02)
0.10(0.01)
0.27(0.02)
CoT-SC
2.09(0.13)
1.)
0.64(0.03)
0.71(0.07)
1.32(0.99)
0.68(0.05)
0,38(0.02)
1.08(0.07)
ToT
186(0.07)
191(0.15)
1.33(0.12)
1.35(0.11)
1.8(0.00)
0.98(0.10)
0,97(0.11)
1.3
RLoT (ours)
2.75(0.21)
2.48(0.22)
0.57(0.2)
0.61(0.05)
0.95(0.05)
0.53(0.03)
0.53(0.04)
1.20(0.09)
Our navigator can directly generate a task-specific logical trajectory, reducing the LLM interaction cost. Also, our lightweight navigator itself brings almost negligible cost.
20

---

Published as a conference paper at ICLR 2026
D PERFORMANCE COMPARISON WITH LARGER LLMS
Table 9: Performance comparison between sub- 10B LLMs enhanced by RLoT, and larger LLMs with
several times of parameters.
LLM
Size
Method
MATH
GSMBK
GPQA
MMLU-STEV
StrategrQA
Average
Gap
14B
Few-shof CaT
80.00
94.80
45.50
85.06
78.60
76.79
..13
Qwen2.5 Instruct
RLoT (ours)
$0.38
94.16
51.34
88,93
81,22
79.21
-1.1
72B
Fow-shot CoT
83.10
95.80
49.00
89.80
86.90
80.92
Fow-shot CoT
48.52
61.75
Llama3. 1.1...iiioi I-Instract
RLoT (ours)
56,
90.07
46.88
80.56
84.42
71.70
-2.56
70B
Few-shot CaT
68.00
95.10
46.70
84,81
76.71
74.26
Tew CaT
75.46
91.48
35,94
85.82
80.06
74.15
.4.4
GPT-4o
mini (8B)
RLoT
(200B)
Fow-shot CoT
76.60
93.73
53.60
87,90
81.10
78.59
In table we show the performance of the sub-10B LLMs after enhancement with RLoT. The results indicate that our RL-based navigator, which contains fewer than 3,000 parameters, significantly enhances the performance of sub-10B LLMs, making them comparable much larger counterparts with around 10x more parameters. Specifically, our RLoT method empowers the sub- 10B LLMs comparable to, compensating most of the performance gap, or even surpassing their larger counterparts, demonstrating remarkable efficiency.
21

---

Published as a conference paper at ICLR 2026
E EXTENDED BASELINE COMPARISONS
E TEST-TIME SCALING BASELINES
In addition to math reasoning models, also compare RLoT with multiple models which cover different categories and designs. The details of baselines are listed below. Fixed reasoning patterns. Baselines from this category adopts fixed workflow to obtain the answer, which
Self-Refine (Madaan et al. 2023 : This method adds a simple refine step after every
reasoning step correct potential mistakes.
Least-to-Most (Zhou al. 2023): Ask LLMs to break down complex problem into series of simpler subproblems and then solve them in sequence.
SelfCheck (Miao et al. 2024): Utilize LLMs to check their own outputs and use the results of these checks improve question-answentne performance by conducting weighted voting on multiple solutions to the question. Here we adopt voting with candidates.
DeAR (Xue al., 2024): The problem is consequently decomposed, analyzed and refined in this workflow. The workflow follows a recurrent manner to detailed decomposition of original problem.
Tree search.Tree-bare Tree-based models divide the reasoning task into sub-steps, and searches for best path to the final answer. Tree-based methods usually requires multiple rounds Q/A, resulting in high reasoning cost.
One-step-greedy: Using PRM model to greedily select the best thought at each step.
Basic Monto-carlo Tree Search: An effective tree-searched method that explore the tree via multiple roll-outs. The MCTS is also supervised by the PRM.
Litesearch (Wang al. 2024a): An low-cost tree search method that assigns larger search budget to more promising nodes.
Q" (Wang et al. 2024b): Q* adopts a value function that estimate the "distance" between the current thought and the correct answer. Then A algorithm, which is widely applied in shortest path problem, is used to finish tree-search.
rStar (Qi et al. 2024) : Use Monte Carlo Tree Search during inference to select actions.
AFlow (Zhang et al. 2024b): An automated framework that uses MCTS to efficiently explore LLM agentic workflow.
Others.
DSPy (Khattab al., 2023): ): programming model that can express and optimize sophisti- cated LM pipelines.
Graph of Thoughts (GoT) (Besta et al.
2024): A graph-based model which allows arbitrary combination of thoughts according the manually designed graph architecture. We carefully
design graphs for each specific dataset.
Buffer of Thoughts (BoT) (Yang et al. 2024d): A Retrieval-Augnee -uameetl--ugmennted Generation (RAG) model that enhance reasoning via reasoning patterns in the buffer. These patterns continuously updated during inference time.
As shown in Table 10 we compare these methods on representative datasets, GSM8K, GPQA, and StrategyQA, which covers three different domains. Results show that our model outperforms all baseline methods among all datasets.
By flexibly organizing logic blocks, design outperform these baselines. To compare the efficiency and effectiveness, we show the token consumption per question for representative baselines:
Unlike search-basee methods that require multiple explorations, design directly generates task- specific logical trajectories, allowing for better performance with a similar or lower cost.
22

---

Published as a conference paper at ICLR 2026
Table 10: Extended baseline comparisons. The bold numbers indicate the best performance in each group of experiments, and the underlined numbers indicate the best baseline method. All results are averaged across repeated runs with standard deviation.
LLM
Method
MATH-550
GSMHK
GPQA
StrategyQA
Average
DirectQA
74.96=1.37
90.95=1.06
31.1640.20 23
68.53+0.53
66.40=0.46
Self-Refine
72.32=1.20
88.39+0.38
32.23+0.61
74.06+0. 79
66.75=0.33
Least-to-Most
73.28=0.84
89.10=0.82
31.79+0.54
73,60=0
66.94=0.25
SeliCheck
74.36=1.60
90.24+1.09
33.8440.61
72.87+0.77
67.83+0.35
DeAR
69.92=0.88
87.84=0.54
35.71#0.37
71.03=0.85
66. 06.139
One-step-greedt
71.24+1.43
89, 49.4990.99
32.63+0.50
72.43+0.55 56
66.45+0.54
Basic MCTS
72.40=0.79
89.51
35.13+0.59
75.72+0.85
68. 19.0.99
Tree search
Litesearch
72.52=1.29
89.8620.88 8.6
33.1740.30 330
76.0420.66
67.90=0.552
Q*
73.76=0.81
92.28+0.61
34.38+0.77 76
69.30=0.50
rStar
75.88=1.02
92.05=0.89
38.1740.32
77.09+0.83
70.80+0.68
AFlow
75.72=1.16
90.95e0.998
37.37+0.63
77.09*0.56
70.28=0.1 1
DSPy
72.48=0.86
88. 88.7520.77
34.1110.66
73.8910
67.31=0.49
Graph Thoughts (GoT)
73.24=1.06
88.76#
33.2640 66
75.69 25.69900.1
67.74+0.39
Buffer of Thoughts (BoT)
75.1620.74
91.98=0.64
34.9120.3 58
74. 74.38=0.44
69.1 19.1120.33
RLoT (ours)
92.95=0.63
44.78=0.71
79.56+0..4
73.54=0.16
Table 11: Token consumption, solving time (minutes) per question, and performance scores for representative baselines.
LLM
Method
GSMBK
GPQA
Strategy
Average
Score
Time
Score
DirecrQA
91.58
240
0.05
31.25
705
0.17
68.85
301
0.05
63.89
415
0.09
Self-Reline
88.
0.23
Qwen2.5- I-nsttcctt
Litescarch
89.76
912
0.20
33.01
2139
0.55
75.84
1228
0.24
66.21
0.33
iSir
91.96
2636
0.77
38.62
4127
1.44
77.15
2987
0.82
69.24
3483
1.01
RLoT (ours)
92.87
1759
0.61
3001
0.95
79.04
2341
0.53
72.18
2367
0.70
23

---

Published as a conference paper at ICLR 2026
E.2 MATH DOMAIN BASELINES
To better demonstrate the effectiveness of our design, we compare RLoT with more baselines that have been specifically designed for math reasoning tasks. We involve fine-tuning designs which computationally expensive.
AceMath (Liu et al. 2024): A supervised fine-tuning model designed specifically for mathematical reasoning. It first develops math-specialized reward model using public datasets and then performs fine-tuning and reasoning guided by this reward model.
PFPO (Jiao et al. 2024): A supervised fine-tuning approach guided by pseudo reward feedback. The feedback is generated either through self-consistency mechanism with the assistance of more powerful LLMs.
We also include inference-time designs that hire complicated algorithms like Monte-Carlo Tree Search (MCTS).
LLM2 (Yang et al. 2024ch: A lightweight model aimed at enhancing LLM reasoning during inference. It achieves this by training a verifier to distinguish and prioritize better LLM A.M--genrrted responses.
LLaMA-Berry (Zhang et al. 2024a): An inference-time Monte Carlo Tree Search (MCTS) method that explores reasoning paths using trained reward model.
HiAR-ICL (Wa et al. 2024): A method that matches problems with multiple reasoning templates inference time. These templates are previously generated using MCTS on subset of the dataset.
Table 12: Performance comparison of RLoT's with more baselines. The bold numbers indicate the best performance in each category.
Category
Method
MATH
GSM8K
Average
AceMath (Liu
2024
64.42
90.45
77.44
Fine-tuning
PFPO (Jiao et al.
2024)
57.80
89.60
73.70
LLM2 Yang et al. 2024c
48.60
88.00
68.30
LLaMA-Bemt -erryy Zhang el al. 2024a)
54.80
89.80
72.30
Inference-timet
HiAR-ICL
Wu el al. 2024)
55.00
90.70
72.85
(ours)
56.56
90.07
73.32
MATH and GSM8K benchmarks and show the results in Table 12] For the baselines, we use the performance reported in the original papers, and for RLoT, we test with the same navigator model as Section 4.3 in the main text. From the results, we can observe that our method outperforms all inference-time baselines. Meanwhile, with substantially lower computational consumption, reaches comparable performance to some fine-tuning methods that require modification on the parameters of LLMs.
We test with Llama3. 1-8-a33.--B-I-sttuuue
24

---

Published as a conference paper at ICLR 2026
F ANALYSES ON THE SELF-EVALUATION STATE
F.1 RELIABILITY
Previous work shows that LLMs can evaluate and correct their own outputs (Han et al. 2024). Meanwhile, multiple works (Madaan al. 2023 Xue et 2024) have reached success with self- conrecting, Based on the literature, we design the structured self-evaluation that prompts the LLM across granular aspects (Table I including the correctness of modeling and calculation. This design aims to be more specific and reliable than the single judgment.
To validate the robustness, we randomly sample 100 intermediate reasoning texts and manually check the self-evaluation. We find that the LLM's assessment was accurate in 82 out of 100 cases. Also, we deliberately introduced modeling and calculation errors in reasoning path, and test whether our self-evaluation mechanism is able correctly identify. As shown in Table [3] the problem is from GSM8K (Cobbe et al.) 2021), which includes the relationship of three variables. In addition to the correct reasoning step, modify it to obtain two wrong reasoning steps, which respectively makes mistake on modeling and calculation. The result shows that the L.LM is able to identify and classify different kinds of mistakes, demonstrating the effectiveness of self-evaluation state extraction.
Table 13: An example of self assessing of intermediate states.
Self-evaluated correctness
Question
Reasoning step
Modeling
Calculation
In this step, we aim calculate the number of purple flowers, which is 80% more than the yellow ones. We calculate by 10 \times (1 0.8) 18
Mark a garden with flowers. He planted plants of three different colors in it.
True
True
Ten of them are yellow, and there are 80% more of those purple. There are only 25% as many green flowers as there are yellow and purple flowers. How many flowers does Mark have in his garden?
In this step, we aim at calculate the number of purple flowers, which is 80% more than the yellow ones.
False
True
We calculate by \times 0.8 =
In this step, we aim calculate the number purple flowers, which is 80% more than the yellow ones.
True
False
We calculate by 10 times (1 0.8) = 17
Based on the validation, we consider our structured self-evaluation provides sufficiently reliable state representation for training the navigator.
F.2 NOISE IMPACT
To investigate the navigator's reliance on accurate state perceptions. we introduced synthetic noise into the self-evaluation state vectors during inference. As shown in Table [4 replacing the state with random values results in a significant performance drop, falling even below DirectQA. This confirms that the navigator actively utilizes the state information for decision decision-making rather than following fixed, state-agnostic policy. However, the framework exhibits commendable robustness. Under extreme conditions with 50% noise (where half of the state bits are flipped). RLoT still maintains an average accuracy of 64.9999 14,99 being comparable to the DirectQA baseline. As the noise decreases to 30%, performance recovers rapidly to 69.11%.
Meanwhile, to minimize potential noise inherent in self-evaluation, we deliberately designed the eval- uation criteria as a binary classification task rather than multi-grade continuous scoring. Continuous or fine- -ra-gna scoring (c.g., 1-10 scales) requires precise calibration, which challenging for LLMs. In contrast, binary standard simplifies the decision boundary, significantly reducing the cognitive load on the LLM. This design choice makes the evaluation process more robust, ensuring that the state vector remains stable and reliable signal for the navigator even with less capable L.LMs.
25

---

Published as a conference paper at ICLR 2026
Table 14: Impact of noise in self-evaluation states.
LLM
State
GSM8K
GPQA
Strategy
Average
DirectQA
91.58
31.25
68.85
63.89
Random
89.76
28.13
65.50
61.13
Owen2.5-Instruct
50% noise
91.74
34.82
68.41
64.99
30% noise
92.27
43.75
71.32
69.11
Normal
92.87
44.64
79.04
72.18
F.3
ALTERNATIVE DESIGNS
Furthermore, we explore alternative designs of self-evaluation:
MLP. Use the self-evaluation vector from fixed L.LM input, only train an ML. the navigator.
LLM (fixed). Directly prompt the LLM to select the action based on the raw reasoning trajectory.
LLM (finetune). Use the raw reasoning trajectory as input, train an LLM backbone with an MLP classification head as the navigator.
Table 15: Comparison of alternative self-evaluation designs.
LLM
Navigator
GSM8K
GPQA
Strategy
Average
DirectQA
91.58
31.25
68.85
63.89
LLM (fixed)
89.16
37.95
74.38
67.16
Qwen2.5 Innttutttt
LLM (finetune)
88.32
34.15
70.01
64.16
MLP (ours)
92.87
44.64
79.04
72.18
As the results in Table 15 show, LLM (fixed) performs badly since pre-trained LLMs for the task. LLM (finetune) even performs worse. Our method injects valuable human knowledge for explicit, structured self-evaluation. In contrast, LLM (finetune) requires implicit evaluation from raw text, which is much more complex and has more parameters to be tuned. This requires substantially more consumption and specialized design to overcome the complexity and uncertainty. Therefore, our method is comprehensive design at a low cost. While some self-evaluation cases may be inaccurate, the overall improvement demonstrates the general benefit.
specified
26

---

Published as a conference paper at ICLR 2026
G ANALYSES ON THE ROLE OF RL IN TRAINING THE NAVIGATOR
To illustrate the necessity of using RL in training the navigator, we compare the RL navigator with several other decision methods for deciding the next reasoning block. The methods includes the
Fixed logic sequence. Fixed Deccoppse-Reason-RReee
sequence like human thinking.
supervised-trainee navigator. We first collected 20K state-action-scon samples with random actions. Then, we trained model predict the outcome for each action given state, and select the action with the highest predicted score during inference.
LLM as navigator. Directly prompt another LLM to select the action based on the reasoning context.
We also add:
Repeated strong blocks. Repeat Debate or Refine until reaching the answer.
Table 16: Comparison with alternative navigator methods on various benchmarks.
LLM
Method
GSMSK
GPQA
Strategy QA
Average
DirectQA
91.58
31.25
68.85
63.89
Repeated refine
87.72
32.37
72.34
64. 14
Repeated debate
89.99
33.04
72.05
65.03
Qwen2.5-
Fixed logic sequence
88.38
36.16
71.13
65.22
Supervise-trained navigator
89.84
36.83
70.31
65.66
LLM as navigator
89.16
37.95
74.38
67.16
RL-trained navigator (ours)
92.87
44.64
79.04
72.18
The results in Table 16 show that these methods
than RLoT:
Fixed logic sequence. Rigid, unable to design flexible logical structures for specific tasks.
Supervise-trained navigator, The randomly sampled state-action pairs lack the ability of RL to purposely exploit effective parts of the sample space.
LLM navigator. Pre-trained LLMs are not specified for the logic selection task.
These show the importance of using RL for our navigator to flexibly select logic blocks.
27

---

Published as a conference paper at ICLR 2026
H ANALYSES ON THE PROCESS REWARD MODEL (PRM)
H. THE ROLE OF PRM
To investigation the role of PRM, we also test ORM train the navigator.
Table 17: Comparison of RLoT using an Outcome Reward Model (ORM) vs. Process Reward Model (PRM).
LLM
Method
GSM8K
GPQA
StrategyQA
Average
DirectQA
91.58
31.25
68.85
63.89
Sien2.5-Instruct-i
RLoT with ORM
91.96
41.52
73.94
69.14
RLoT with PRM (ours)
92.87
44.64
79.04
72.18
The results in Table 17 show that the PRM outperforms the ORM, indicating that the step reward signal from PRM crucial.
H.2 GENERALIZATION CAPABILITY OF PRM
Recent studies suggest that process reward models (PRMs) trained mathematical data exhibit strong transferability to related domains such as STEM (Zhang et al. (025). To empirically verify this and justify reward design, conducted PRM calibration analysis. We calculated the correlation between the intermediate PRM scores assigned at various reasoning steps and the correctness of the final answer.
Table 18: PRM calibration.
LLM
Correlation with Correctness
MATH
GPQA
Strategy QA
Step1 PRM
0.199
0.184
0.157
Step2 PRM
0.222
0,213
0.195
Qwen2.5-Innsrrctt77
Step3 PRM
0.236
0,215
0,202
Step4 PRM
0.240
0.228
0.218
Average
0.224
0.210
0.193
As presented in Table 18 the results reveal two insights. First, there is consistent positive correlation between intermediate PRM scores and the final outcome across all tested benchmarks. This confirms that the PRM effectively gauges the quality of intermediate steps, validating its suitability as dense reward signal for training the navigator. Second, although the PRM was trained primarily on mathematical data, its scores exhibit notable correlation with solution correctness on out-of-dommin tasks, including GPQA (STEM) and StrategyQA (commonsense). This demonstrates the PRM's intrinsic ability to generalize its verification logic across different domains.
This calibration analysis explains the underlying mechanism of our method's transferability. Because the math-trained PRM provides reliable feedback even unseen tasks like GPQA and StrategyQA. the navigator trained with these rewards can successfully learn effective reasoning policies across diverse domains.
H.3
IMPACT OF PRM QUALITY
Further, to explore the impact of PRM quality, we test:
ORM: As a minimal quality PRM.
Degraded PRM: Add a std=0.1 Gaussian noise to our raw PRM.
Better PRM: Owen2255MMth-PRRM-T7 stronger PRM.
These results in Table 19 show that higher-quality PRM is beneficial. However, the navigator is robustly beneficial with lower-quality PRM, as long as the reward signal is directionally meaningful.
28

---

Published as a conference paper at ICLR 2026
Table 19: Impact of PRM quality navigator performance.
LLM
PRM
GSMSK
GPQA
Strategy QA
Average
-one-DirectQAA
91.58
31.25
68.85
63.89
ORM
91.96
41.52
73.94
69.14
MathShhepher--Mittaall-TT disturb
92.49
42.41
77.15
70.68
MathShepherdk-isttal---1
92.87
44.64
79.04
72.18
Qwen2.5-Math-PRM-75
93.10
45.31
80.06
72.82
29

---

Published as a conference paper at ICLR 2026
I ANALYSES ON THE DESIGN OF LOGIC BLOCKS
1.1 EFFECT OF LOGIC BLOCK NUMBER
We first check the test-time scaling for the logic blocks. Results in Figure sequence of logic blocks brings performance gain.
show that a longer
GSMBK
GPQA
StrategyQA
3300
Taken Cimptoo
Token cons: consum tion
42
Accuracy
Input OuEQuE
Input Ousput
Input Ourpur
1000
Faw
logic
lagic
lagic
Figure Test-time scaling with different number of logic blocks.
1.2 ROLE OF DIFFERENT LOGIC BLOCKS ON DIFFERENT DATASETS
We also analyze the impact of different logic blocks on different datasets. Results in Figure show the average PRM reward gain each logic block. The most useful block varies in different tasks, specifically Debate for math tasks, Decompose for STEM tasks, and refine for common-sense reasoning.
MATH
GSMBK
GPOA
MMLU-STEM
StrategyOA
Relative roward gain
Rele reward gain
0.4
Relativve rewerd
02
Decompote Drbate Refne
0/
Secompose Bebate
Refne
becempops Debate Logic block
Rnfine
"ecompase Debote Refne
00
Dessmppeee Debate Rehre
Lagic block
Logic block
Logic block
Logic black
Figure 9: Contribution of each logic block in the reasoning sequence.
1.3
ALTERNATIVE BLOCKS
Further, test possible new blocks:
Follow-up-question: Propose and answer sub-question based on the current reasoning.
Rephrasing: Reorganize the current reasoning.
Compression: Condense the current reasoning.
Table 20: Performance with additional logic blocks.
LLM
Logic blocks
MATH
GPQA
Strategy QA
Average
Original in RLoT
76.70
44.64
79.04
66.79
Original Follow-up-question
76.78
43.97
79.24
66.66
Swen2.5-Instrucch
Original Rephrasing
77.30
45.54
80.20
67.68
Original + Compression
76.06
43.53
77.87
65.82
30

---

Published as a conference paper at ICLR 2026
The results in Table 20 showed that "Rephrasing" is beneficial, while "Follow-up-oe uu-eusstinuttio" (perhaps sulf-ask-and-answee has limited effect) and "Compression (likely due to information loss) are not. This shows the extensibility of RLoT: it provides framework for the community to design more reasoning blocks to enhance the performance.
31

---

Published as a conference paper at ICLR 2026
J IMPLEMENTATION DETAILS
J.1 SETTINGS
In this section, provide the main implementation settings for reproducibility in Ta-
ble 21 Please refer to our source code at /https://anon/:: 4open. secnnsccen scciencerr)
RL Rew.---Reannnnn----1 1A30 for the exact usage of each hyper-parametters and more details.
Table 21: Implementation details.
Module
Element
Detail
OS
Ubuntu 22.04.2
System
CUDA
11.7
Python
3.11.4
0.9
Number episodes
3000
Batch Size
64
Interval of target network updating
50
Optimizer
Adam
Double-Dueling DQN
Learming
0.01
Learning rate decay
0.5 per 1000 episodes
Replay buffer size
500
Start epsilon
Min epsilon
0.0
Epsilon decay
0.9995 per step
Maximum number of actions
RLoT framework
Number of trails for self-consistency Temperature for LLMSs
1.0
J.2 ILLUSTRATION OF THE FULL PIPELINE
Here we illustrate how every parts our design work together as an whole pipeline.
DATA CONSTRUCTION
We pick out "hard problems", namely those that the LLM wrongly answers when directly prompted, for training the navigator. Here are more details: we used Qwen2.5-14B-Inn Innt-ntrtt as reference model select "hard problems" from the training sets. The direct prompts are:
Prompts:
(Problem) Please generate the answer for the problem. (MATH) Wrap the answer with boxed [answer) (GPOA) End the answer with The answer is (CHOICE) (StrategyQA) End the answer with YES/NO'.
The statistics on benchmarks involved in our training are shown in Table 22
Table 22: Statistics on benchmarks involved in our training.
Dataset
Total
Hard
Proportion (%)
MATH
7500
1736
23.15
GPQA
448
255
56.92
StrategyQA
1600
341
21.31
By selecting "hard problems". our navigator dedicates to strategies that enhance LLMs on problems that they cannot solve directly. For our RL approach, we do not construct (input, expected_output) pairs like supervised learning. The training data consists of individual questions. During training, the model fed with questions, and the outputs are evaluated to generate rewards, which are then fed back train the model. We will detail the key components and entire process of training below.
32

---

Published as a conference paper at ICLR 2026
2 STATE BELF-EVALUATION (STATE SPACE)
We obtain the 7-aspect state vector through prompted self-evaluation mechanism. We feed the problem and the existing reasoning into the LLM with the prompt in Appendix|K. This design compresses the textual reasoning trajectory into a low-dimensional state vector, making the policy learning process more efficient and tractable.
Here an input-output example of state self-evaluation, as seen our case study in Appendix
Input
(Problem) Mark has a garden with flowers. He planted plants of three different colors in it. Ten of them are yellow, and there are 80 more of those in purple. There are only 258 many green flowers as there are yellow and purple flowers. How many flowers does Mark have in his garden?
(Existing reasoning) In this step, we aim calculating the number of purple flowers, which is 80% more than the yellow ones. We calculate by 10 * (1 0.8) = 17
Output (State vector) (Modelling: True, Calculation: False, ...)
3 LOGICAL BLOCKS (ACTION SPACE)
First, these logic blocks are inspired by well-established cognitive strategies in human problem solving. "Decompose." "Debate." and "Refine" the core actions, supplemented by "Reason one step" and "Terminate" to ensure complete reasoning flows.
For instance, "Decompose" reflects the widely studied divide-and-conquet approach, while "Debate" aligns with strategies involving comparative evaluation (Wang & Chiew 2010). Also, they are empirically effective. Prior studies (Madaan al. 2023 Xue et 22.24) have demonstrated that "Decompose" and "Refine" can significantly enhance the performance of LLMs on complex tasks.
Second, while preserving the high-level cognitive intuition, have operationalized them into specific primitives that the L.M can execute. The key differences are:
Formalization with prompts: Each block corresponds to a prompt template that instructs the LLM to perform specific function (Appendix K.2).
Standard Input/Output: All blocks operate on unified interface. The input is the original problem and the existing reasoning, and the output is a new segment of reasoning. This allows flexible composition of logie blocks.
Third, here is input-output example for the 'Debate' block (as Appendix B.1
Input
(Problem) How many square units are in the region satisfying the inequalities
$y \ge |x|s and $y Ale -|x|+3$? (Existing reasoning) Let's s start by analyzing the given inequalities. The next step is to determine the points of intersection... Now know the points of intersection are...
Output
(New reasoning) The most promising plan is: Given the diagonals of the rhombus, can new calculate the area.
The contributions of the logic blocks are discussed in Appendix
4 PROCESS REWARD
The PRM takes in the problem and existing reasoning and outputs numerical score evaluating its quality.
Here is an input-output example for the PRM:
33

---

Published as a conference paper at ICLR 2026
Input:
(Problem) How many square units are in the region satisfying the inequalities $y Age |x|s and $y \le -|x|+3$?
(Existing reasoning) Let's s start by analyzing the given inequalities. The next step is to determine the points of intersection... Now know the points of intersection are...
Output: (Reward) 0.765
DEFINITION OF MDP
Combining the above components, our MDP is defined as follows:
State: A low-dimensional vector generated from the L.M's self-evaluation of the current reasoning step.
Action: One of five logic blocks that guide the next step reasoning.
A score from PRM that evaluates the quality of the reasoning step after action is taken.
Reward:
TRAINING PROCESS
Follow the MDP framework, we train the navigator with Deep Q-Learning algorithm. The training process
For training ng episodes:
Randomly sample a Problem. Existing reasoning=().
WHILE True:
Problem Existing reasoning -- selll--vvllaaion--- State vector
State vector (MLP navigator] --> Logic block (action)
Problem Existing reasoning -- [Logic block]--3 New reasoning
Existing reasoning
Existing reasoning + New reasoning
Problem Existing reasoning -- [PRM] --> Reward
Use the reward to train the MLP navigator
IF reach the answer: BREAK
7 INFERENCE PROCESS
With the trained navigator, the inference process is:
Existing reasoning=(). Answers=(). For self-consistency candidates number:
Problem.
WHILE True:
Problem Existing reasoning -- [Sel walll--llaaiion--- State vector
State vector (MLP navigater]--> --> Logic block (action)
Problem Existing reasoning -- [Logic block]--3 --> New reasoning
Existing reasoning
Existing reasoning New reasoning
IF reach the answer: BREAK
Answers Answers New answer
Answers -- [Self-consi coinssstenncy---1 Final answer
8 SELF-CONSISTENCY EVALUATION
To enhance the robustness of our final answers, employ self-consistency mechanism. Since the final answers can in different formats, we use sympy.parsing.late library to parse the answers and determine if they mathematically equivalent. We select the final answer using majority voting. If no majority exists, one answer selected at random.
Here an input-output example for self-consistency:
34

---

Published as a conference paper at ICLR 2026
Input:
(Answers) (0.5) (0.7) (1/2)
Output:
(Final answer) 0.5
35

---

Published as a conference paper at ICLR 2026
K PROMPTS
In this section, we provide the prompts for obtaining states and performing actions to ensure repro- ducibility. In the following prompt blocks, the text enclosed in braces "()" denotes problem-specific content, such as intermediate reasoning steps, while the remaining texts serve as fixed templates.
K. PROMPT FOR OBTAINING STATES
We the following prompt extract state vector from an intermediate reasoning step. This prompt guides the LLM in systematically evaluating the current step across multiple aspects. During our experiments, we observed that LLMs effectively identify these aspects and provide detailed scores for each aspect.
Listing I: Prompts for obtaining states
(Problem and reasoning steps
2 Please evaluate the current step from the following aspects.
3 A) Correctness
4 Al: Correctness of modeling:
Whether the current step correctly derived from the origin problem.
6 A2: Clarity for further reasoning:
7
Whether the current step is clearly presented, without ambiguity to support further reasoning.
A3: Correctness calculation:
9
Whether the numerical computation in the current step is performed correctly. Complexity
10
Complexity to reach the final answer:
Whether still requires complex reasoning or calculation to reach the final answer from the current step. B2: Alternative methods in further reasoning:
13
14
Whether there exist multiple alternative methods to solve the problem in the current step.
C)
Completeness
16
C1: Closeness to the final solution:
17
Whether the current step is close enough to directly reach the final answer.
C2: Completeness within the step:
19
Whether all necessary elements within this specific step are known from the problem previous steps.
20
For each aspect, please score 1 for False, for Unsure, and 3 for True, and score if the current step does not involve this aspect. Please attach the reason for each score. 21 Use the format 'A1 (core=(ScORE] reason= [REASON]]
22 Only score the current reascning step here, and DONCT conduct further reasoning.
K.2 PROMPTS FOR ACTIONS
Reason one step: The prompt below is designed to conduct the action of "Reason one step". We emphasize that it should focus one step at time, which better controls the output and prevents mistakes long reasoning paths.
Listing Prompts for action "Reason one step"
Here is problem and several reasoning steps. 2 (Problem and previous steps)
Please reason exactly ONE more step based on the current step here, and DONCT reason too many steps at once.
Decompose: A "Decompose" action consists the following three prompts. First, we use prompt B to break down the current problem into multiple subtasks. Next, prompt 4 applied sequentially
36

---

Published as a conference paper at ICLR 2026
each subtask to complete its execution. Since the current subtask may depend previous ones, the results of earlier subtasks are included (see line of the prompt). Finally, the execution results are summarized using prompt which captures the key steps and outcomes. Only the summarized result utilized for subsequent reasoning actions.
Listing 3: Prompts for obtaining subtasks in action "Decompose"
Here is problem and several reasoning steps.
(Problem and previous steps)
3 Please decompose the current task into subtasks, where we can sol ve the original problem em by combining these results of subtasks.
4 Only provide subtasks decomposition here, and DONOT conduct speci fic reasoning calculation.
Use the format *## Subtaskl: subtask1'.
Listing 4: Prompts for executing subtasks in action "Decompose"
Here a problem and several reasoning steps. 2 (Problem and reasoning steps before decomposition)
3 For the next step, the task is decomposed into subtasks, here are the reasonings in the first few subtasks.
4 (Executing results of previcus subtasks) Please conduct the following Subtask subtask to continue the reasoning.
6 DONOT conduct more detailed decomposition for the subtask.
Listing 5: Prompts for summarize subtasks in action "Decompose"
Here are a few detailed reascning subtasks of a problem.
2 ( (Executing results of subtasks)
3 Please give clear and concise summary these subtasks, keeping the key reasoning
results each subtask.
4 Only provide the summary here, and DONOT conduct more reasoning or calculation.
Debate: A "Debate" action involves multiple rounds of quesstion-answentn First, the LLM generates different plans for the task using prompt 6 Next, the plans are compared, and the most promising one is selected using prompt 7 mimicking how human experts debate and discuss to reach solution. Based on the chosen plan, reasoning is advanced by one step through prompt Similar to the "Decompose" action, only the results of the final one-step reasoning (output of prompt are retained for subsequent reasoning processes.
Listing Prompts for obtaining various plans in action "Debate"
1 Here is a problem and several reasoning steps.
2 (Problem and previous reasening steps]
3 Please propose three different alternative plans for solving the problem in the current step.
4 Only provide plans here, and DONCT conduct specific reasoning or calculation.
5 Use the format *## Planl: planl'.
Listing Prompts for analysing and comparing plans in action "Debate"
1 Here is problem and several reasoning steps. (Problem and previous reascning steps)
3 Currently, we have several alternative plans for solving the problen in the current step. 4 (Generated Plans)
5 Flease review and compare these plans carefully, and tell which mest promising for further reasoning. Only compare the plans here, and DONOT conduct further reasoning or calculation.
6 Use the format 'The most premising plan is Plan(INDEX): [REASON] where [INDEX] is an integer index of the plan and [REASON] is detailed analysis.
37

---

Published as a conference paper at ICLR 2026
Listing 8: Prompts for executing the plan in action "Debate"
1 Here is a problem and several reasoning steps.
2 (Froblem and previous reasening steps)
3 For the (Plan)
step, have decided the most promising plan:
5 Please reason **eaactly one** more step according to the plan here, and DONCT reason too many steps at once.
We use prompt 9 to perform the "Refine" action, which instructs the LLM to review and improve the reasoning steps for clarity and correctness,
Refine:
Listing 9; Prompts for action "Refine"
Here is problem and several reasoning steps 2 (Problem and previous reascning steps)
Please check and refine the current thought here, and DONOT conduct further reasoning or calculation.
Terminate: The prompt for the "Terminate" action concludes the reasoning process by generating the final answer. The only variation lies in the output format, which is adapted to the specific requirements of each dataset.
Listing 10: Prompts for action "Terminate"
Here a problem and several reasoning steps 2 (Froblem and previous reasoning steps)
3
4 ## GSM8K
5
Please generate the answer for the problem. Please end the with 'The answer is numerical_answer'.
6
7 # MATH
8
P Please generate the answer for the problem. wrap the answer with W (andd
9
10 # MMLU-STEM and GPQA
11 End the answer with 'The answer is (CHOICE)'.
12
44 StrategyOA
Flease generate the answer for the problem. At the end of your answer, conclude the answer with The answer is yos' or 'The answer is no'.
K.3 SENSITIVE TO ACTION-PROMPT PHRASING.
To evaluate whether the performance RLoT is sensitive to the specific wording of the action prompts, conducted robustness test by rephrasing the natural language instructions for the Decompose, Debate, and Refine actions.
Decompose: We rephrase the prompts for the action of Decompose as:
Listing 11: Prompts for obtaining subtasks in action "Decompose"
This the problem
the reasoning progress sc far.
2 (Problem and previous steps)
3 Break down the current problem into smaller, manageable sub sub-problens
4 Solving these sub-problems sequentially should lead to the solution of the original question.
5 Only provide subtasks decomposition here, and DONOT conduct speci fic reasoning calculation.
6 Format: *## Subtaskl: subtaskl'.
38

---

Published as a conference paper at ICLR 2026
Listing 12: Prompts for executing subtasks in action "Decompose"
This is the problem context and the steps taken.
2 [Problem and reasoning steps before decomposition)
3 We have divided the main task into several parts. Below is the reasoning progress for the initial parts.
4 (Executing results of previcus subtasks)
5 Now, please execute the speciřic instruction for Subtask subtask
to advance the solution.
DONOT conduct more detailed decomposition for the subtask.
Listing 13: Prompts for summarize subtasks action "Decompose"
Below are the detailed execution steps for the problem's s subtasks.
2 (Executing results of subtasks)
3 Synthesize the steps above into brief and coherent overview,
ensuring that the essential logic and final outcomes of each subtask are preserved.
Only provide the summary here, and DONOT conduct more reasoning or calculation.
Debate: We rephrase the prompts for the action of Debate as:
Listing 14: Prompts for obtaining various plans in action "Debate"
This a problem and existing reasoning steps.
(Froblem and previous reascning steps)
3 Formulate three distinct strategies to address the problem from the current state.
4 Only provide plans here, and DONOT conduct specific reasoning calculation.
5 Use the format *Hf Planl: planl'.
Listing 15: Prompts for analysing and comparing plans in action "Debate"
This is the problem context and the steps taken.
2 (Froblem and previous reasening steps)
3 We have generated set of petential strategics to address the problem at this stage. (Generated Plans)
4 Critically evaluate these options and identify the most viable strategy to advance the reasoning.
5 Only compare the plans here, and DONOT conduct further reasoning calculation.
Use the format 'The most premising plan Plan INDEX) [REASON] where [INDEX is an integer index of the plan and [REASON] is a detailed analysis.
Listing 16: Prompts for executing the plan in action "Debate"
Here is problem and several reasoning steps.
2 (Problem and previous reascning steps)
3 We have selected the optimal strategy to proceed: (Pian)
4 Carry out the immediate next logical step following this strategy, and DONCT reason too many steps at once.
Refine: We rephrase the prompts for the action of Refine
Listing 17: Prompts for action "Refine"
Here a problem and several reasoning steps
2 (Problem and previous reascning steps)
3 Flease examine the current reasoning for potential improvements and revise it accordingly, and DONOT conduct further reasoning calculation.
39

---

Published as a conference paper at ICLR 2026
Table 23: Impact of prompt rephrasing.
LLM
Method
GSM8K
GPQA
StrategyQA
Average
DirectQA
91.58
31.25
68.85
63.89
@wee2.5-IInsruuct
RLoT (original)
92.87
44.64
79.04
72,18
RLoT (rephrased)
92.49
45.09
79.91
72.42
The results, presented Table 23 demonstrate that RLoT exhibits high robustness to prompt variations. The model with rephrased prompts achieves an average accuracy very similar to the original one. This consistency indicates that the effectiveness of RLoT stems from the adaptive logical structures constructed by the navigator, rather than overfitting specific phrasing or reliance on extensive prompt engineering.
40

---

Published as a conference paper at ICLR 2026
L
USE OF LLMS
The authors used LLMs to aid or polish paper writing, but all content has been carefully reviewed by the author. The authors used LLMs for literature retrieval and discovery, but all related works have been carefully reviewed and organized by the author. The research ideation in this work was entirely completed by the author and does not involve the use of LLMs.
41


Reviews:
[{"review_id": "5POfgG4lCx", "reviewer": "Reviewer_oJop", "summary": "The authors hypothesise that inference-time reasoning frameworks such as Chain of Thought (CoT), Tree of Thoughts (ToT), and Graph of Thoughts (GoT) are overly rigid and fail to adapt dynamically to different reasoning problems. To address this, they propose a reinforcement learning–based controller (“navigator”) that, given the current reasoning state representation, selects among five “logic blocks”: Reason, Decompose, Debate, Refine, or Terminate.\n\nReasoning is formalised as a Markov Decision Process (MDP) where states are defined by the LLM’s self-evaluation, actions correspond to logic blocks, and rewards are provided via a process reward model (PRM). The navigator is trained using a Double-Dueling DQN and subsequently used at inference time to guide the LLM’s reasoning adaptively.", "questions": "- What happens if the self-evaluation state is noisy or replaced with random values — does the navigator still help?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of viewing reasoning as an MDP with composable cognitive actions is original and conceptually appealing. The method is explained clearly.\n\n- The proposed navigator has fewer than 3K parameters, making it lightweight, computationally inexpensive, and compatible with frozen LLMs.\n\n- The authors experimented across reasoning, math, STEM, and commonsense domains, and also covered both small- and medium-sized models.\n\n- The authors show that their method can perform cross-model and cross-task generalisation.", "weaknesses": "- There’s no ablation on reward signals or PRM accuracy, so it’s unclear whether improvements come from RL navigation or simply from PRM-guided prompting.\n- Some parts of the paper are not detailed enough, for example, section 4.5. How many instances were considered? The authors do not justify why Double-Dueling DQN is used. There is no exploration of policy stability, convergence, or sample efficiency. \n- The paper majorly lacks qualitative analysis. The paper presents only anecdotal examples of reasoning patterns (“Reason–Refine–Debate”) and no failure-case analysis, error typology, or robustness study. We don’t know when or why RLoT fails. \n- Minor point: The paper over-relies on the appendix for critical methodological and experimental details. The main text should ideally be more self-contained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qzVobCdCPt", "reviewer": "Reviewer_TZiL", "summary": "The paper proposes RL-of-Thoughts (RLoT): treat multi-step reasoning as an MDP where a tiny navigator (Double-Dueling DQN, <3k params) learns to select among five logic blocks (Reason one step, Decompose, Debate, Refine, Terminate) while the base LLM stays frozen. States are 7-aspect self-evaluations the LLM generates via a fixed prompt; rewards come from a Process Reward Model (PRM), e.g. math shepherd. The navigator is trained on \"hard\" questions only; at test time they use self-consistency to pick the final answer.", "questions": "1. Could you report mean ± 95% CI over ≥3 seeds for the results (especially Table 9) (GSM8K/GPQA/StrategyQA) and key math sets? If not, please justify why single-seed is sufficient given self-consistency randomness.\n2. Is there a reason behind fix training at 3,000 episodes? Do curves show continued improvement (or overfitting) at 1k/5k?\n3. Table 10 reports tokens/question, but wall-clock or FLOPs can be very informative as different methods assume different number of model. Can you add runtime and PRM calls to enable compute-fair comparisons vs. search-heavy methods (e.g., r\\*, LiteSearch)?\n4. Did you think of including SelfCheck[1] and Least-to-Most prompting [2] to experiments as baselines? Even a lightweight SelfCheck (PRM as a one-shot checker without RL) would clarify how much the RL policy contributes beyond access to a PRM.\n5. You ablate −Decompose/−Debate/−Refine; can you also show fixed schedules (e.g., Decompose→Reason→Refine cycles à la DeAR) under the same token budget to separate \"RL policy\" gains from \"having blocks at all\"?\n\n[1] Miao et. al.. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. ICLR, 2023.\n\n[2] Zhou,et al. Least-to-most prompting enables complex reasoning in large language models. ICLR, 2022.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Reasoning as an MDP with a standardized action set and state interface; prompts for states/actions are fully specified and the MDP is clearly defined. The navigator adds negligible runtime overhead; training cost is amortized and reported with token accounting.\n- Good comparison with baselines (fixed workflows like DeAR, search methods like GoT, r\\*, LiteSearch, DSPy, ...) with an \"extended table\" across GSM8K/GPQA/StrategyQA; The authors also report token usage vs. accuracy.\n- Proper ablation:  Remove-one-block studies (for Decompose, Debate, and Refine),", "weaknesses": "Major problem is the insignificance of many results: For instance in Table 9, RLoT = 92.87 while several baselines cluster ≥92% (e.g. Buffer-of-Thoughts 92.35,  ...). The paper text claims overall wins, but GSM8K specifically looks like parity rather than a clear lead; without error bars the <1% differences are hard to interpret and they are rather frequent in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rqzAhTpzsQ", "reviewer": "Reviewer_HceF", "summary": "The paper proposes RLoT, a new inference-time technique to enhance the complex reasoning capabilities of LLMs. The core problem addressed is that existing inference-time methods use predefined, task-agnostic logical structures that lack adaptability. RLoT's key idea is to train a lightweight \"navigator model\" using Reinforcement Learning to dynamically construct task-adaptive logical structures during inference.", "questions": "1. The quality of the 5 logic blocks seems highly dependent on the quality of their prompts (listed in Appendix K.2 ). How much prompt-engineering was required to make these actions (especially Debate and Refine ) work reliably? Is the framework sensitive to the phrasing of these action-prompts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The primary strength is the RLoT framework itself. It reframes the problem from \"how to build a better reasoning LLM\" or \"what is the best fixed reasoning structure\" to \"how to learn a policy that navigates a frozen LLM's reasoning space.\" This is a clever and effective conceptual leap.\n\n2. The results show that a sub-10B model with a 3K-parameter navigator can close most of the performance gap to a 70B model on tasks like GPQA and StrategyQA is a massive win.\n\n3. The method is tested on multiple SOTA LLMs and a wide array of difficult reasoning benchmarks.", "weaknesses": "1. The state space, which is the sole input to the RL agent, is based on the LLM's own self-evaluation. The authors provide a validation (82% accuracy) in Appendix F and show it works better than alternatives. However, this remains a potential point of noise or failure. One might question if an LLM that is failing at a reasoning task (the \"hard questions\" used for training ) can simultaneously be a reliable evaluator of its own failing state. This could be particularly problematic for smaller, less-capable models.\n\n2. The five logic blocks are \"inspired from human cognition\"  and the quality of the entire system is upper-bounded by the quality and expressiveness of this predefined action set. However, they are hand-engineered component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9cOYBRXgiE", "reviewer": "Reviewer_VvQn", "summary": "The paper proposes the technique RL-of-Thoughts (RLoT), an inference-time method that trains a tiny “navigator” with reinforcement learning to compose task-adaptive reasoning structures for LLMs. Long-sequence reasoning is cast as an MDP, where the state is a self-evaluation of the current reasoning, and 5 possible actions: Reason one step, Decompose, Debate, Refine, and Terminate. A process-reward model scores intermediate steps to provide the reward signal. Experiments show consistent accuracy gains and the authors also claim transferability of a single navigator across LLMs and tasks.", "questions": "1. How does the number of tokens generated to achieve the solution compare between RLoT and the baselines?\n2. How does the number of actions affect the results? Why are these actions selected?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. It presents a method that reframes reasoning as an MDP over a small set of actions, avoiding deep tree searches.\n2. Results show performance gains across different tasks and models. The authors have analyzed a wide range of models.\n3. According to the authors, the method is transferable and generalizable to other models and datasets.", "weaknesses": "1. The method repeats inferences with self-consistency filtering. Including the results of token budgets per method and normalize across baselines can help understanding the budget costs in a fair way. \n2.PRM calibration. How well do PRM scores (Math-Shepherd) correlate with eventual correctness on non-math tasks?\n3. The paper fails to include confidence intervals of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)