THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning
Authors: Qikai Chang
Zhenrong Zhang
Pengfei Hu
Jun Du
Jiefeng Ma
Yicheng Pan
Jianshu Zhang
Quan Liu
Jianqing Gao
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Mathematical Problem Solving
Tool-Integrated Reasoning
Reinforcement Learning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We introduce THOR, a tool-integrated framework that combines hierarchical reinforcement learning with self-correcting inference to achieve SOTA mathematical reasoning.

Abstract: Large Language Models (LLMs) have made remarkable progress in mathematical reasoning, but still continue to struggle with high-precision tasks like numerical computation and formal symbolic manipulation. Integrating external tools has emerged as a promising approach to bridge this gap. Despite recent advances, existing methods struggle with three key challenges: constructing tool-integrated reasoning data, performing fine-grained optimization, and enhancing inference. To overcome these limitations, we propose THOR (Tool-Integrated Hierarchical Optimization via RL). First, we introduce TIRGen, a multi-agent based pipeline for constructing high-quality datasets of tool-integrated reasoning paths, aligning with the policy and generalizing well across diverse models. Second, to perform fine-grained hierarchical optimization, we introduce an RL strategy that jointly optimizes for both episode-level problem solving and step-level code generation. This is motivated by our key insight that the success of an intermediate tool call is a strong predictor of the final answer's correctness. Finally, THOR incorporates a self-correction mechanism that leverages immediate tool feedback to dynamically revise erroneous reasoning paths during inference. Our approach demonstrates strong generalization across diverse models, performing effectively in both reasoning and non-reasoning models. It further achieves state-of-the-art performance for models of a similar scale on multiple mathematical benchmarks, while also delivering consistent improvements on code benchmarks. Our code will be publicly available at https://github.com/JingMog/THOR.

Published as a conference paper at ICLR 2026
THOR: TOOL-INTEGRATED HIERARCHICAL OPTI- MIZATION VIA RL FOR MATHEMATICAL REASONING
Qikai Chang', Zhenrong Zhang Pengfei Hu'.Jun Du'; Mat.?, Yicheng Pan Jianshu Zhang", Quan Liu?, Jianqing Gao²
University of Science and Technology of China, 2iFLYTEK Research
ABSTRACT
Large Language Models (LLMs) have made remarkable progress in mathematical reasoning, but still continue to struggle with high-precision tasks like numerical computation and formal symbolic manipulation. Integrating external tools has emerged a promising approach to bridge this gap. Despite recent advances, existing methods struggle with three key challenges: constructing tool-integrated reasoning data, performing fine-grained optimization, and enhancing inference. To overcome these limitations, we propose THOR (Tool-Integraal Hierarchical Optimization via RI.). First, we introduce TIRGen, multi-agent based pipeline for constructing high-quality datasets of tool-integrated reasoning paths, aligning with the policy and generalizing well across diverse models. Second, to perform line-grained hierarchical optimization, we introduce an RL strategy that jointly optimizes for both episode-level problem solving and step-level code generation. This is motivated by our key insight that the success of an intermediate tool call is a strong predictor of the final answer's 's correctness. Finally, THOR incorporates a self-correction mechanism that leverages immediate tool feedback to dynamically revise erroneous reasoning paths during inference. Our approach demonstrates strong generalization across diverse models, performing effectively in both rea- soning and non-rease non-rcasoning models. It further achieves state-of slate-of-the-ar the-art performance for models of similar scale on multiple mathematical benchmarks, while also de- livering consistent improvements on code benchmarks. Our code will be publicly available https: hwtptb.t.bbbgggggggggggggg.bgh.b com/ wtt. JingMog///////// /THOR.
Current TIR Data
TIRGen: TIR data construction pipeline
Results
Generator
Natural language recsoning
Act
Choserv
Limitations
problem
Tool noi-iintgratted
dataset
Style mismetch
Refiner
Unapplicable to mensoning models.
Nan Non-Reasioninn Models
AMC23
THOR: Tool- Dote-rnneeeeteed Hierarchical Optimization via RL
Our Key Insight
Esciode level Optimaztionn in
Matth melving trok
Accurony Reavor
predictor
Cold Start
Self-cerrettion Inferance
Optiiization
Code is sttonng predictor for arewer tress in TIR
5tagel:5FT
Hiraachiee RL
Inf freenc
22
ASME24
Reasoning Models
ADMES5
AMC23
An overview of our method. The left panel depicts the motivation and challenges, the middle highlights our proposed solution with the TIRGen data construction pipeline and the THOR hierarchical RL framework, and the right panel reports experimental results.
Figure I:
"Corresponding author; Jun Du (jundu @ ustc.edu.cm).

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Large Language Models (LLMs) have achieved remarkable progress, increasingly exhibiting human-like capabilities such as thinking, reflection, and self-correction. They have shown signif- icant improvements in mathematical reasoning, code generation, and autonomous agent tasking (Jaech et al., 2024; Guo et al., 2025; Yang et al., 2025; Team et al., 2025).
Recent tool-free methods for enhancing LLMs' mathematical reasoning can be broadly categorized into march-basee methods (Besta et al., 2024; Zhang et al., 2025; Hu et al., 2025) and reinforce- ment learning (RL) paradigms (Yu ct al., 2025; Yue et al., 2025b; Luo et al., 2026). Despite notable progress, these approaches remain constrained by fundamental weakness of LLMs. As probabilis- tie next-token predictors, they inherently struggle with high-precision tasks (Chen et al., 2022), such as numerical computation, equation solving, symbolic manipulation (Pan et al., 2025), and formal proofs (Lewkowycz et al., 2022). In contrast, programmatic reasoning excels in these domains. Therefore, integrating the semantic reasoning capabílities of LLMs with the precise and verifiable execution of external code-based tools provides a promising pathway to overcome these limitations.
Tool-Integrated Reasoning (TIR) has emerged as a powerful paradigm for enhancing L.MM reasoning by enabling them to leverage external tools to augment reasoning (Gou et al., 2023; Li et al., 2025a). Despite considerable efforts, three core challenges remain: constructing TIR data, performing fine-grained optimization, and enhancing inference. (1) For constructing TIR data, common approach is to synthesize tool-use data by prompting external powerful models (e.g.. GPT-4o) (Gou et al., 2023). However, for reasoning models such as DeepSeek-R1, prompting alone often fails to elicit effective tool use (Guo et al., 2025; Li et al., 2025a). While techniques like START (Li et al., 2025a) explicitly inject code prompts into the thinking process, purely rule-based approaches struggle to identify suitable insertion positions. Therefore, existing TIR data construction methods suffer from style mismatches and poor applicability to reasoning models. (2) For performing fine- grained optimization, current research primarily employs either SFT or RL. SFT-based methods, like Toolformer (Schick et al., 2023) and Aimo-2 (Moshkov et al., 2025), require large-scale, high- quality demonstration data and often suffer from poor generalization. Existing RL methods (Mai al., 2025; Li al., 2025b; Feng et al., 2025) typically optimize the episode-level, overlooking fine-grained updates on specific steps. Although RL is a more scalable alternative, it faces severe sparse reward problems, particularly in long reasoning chains. (3) For enhancing inference, existing methods typically interleave tool calls directly with natural language reasoning in a single pass, therehy overlooking the role of immediate tool feedback in reasoning.
To address these challenges. we propose THOR, ioo-rinter framework designed to enhance the reasoning ability of LLMs. (1) For constructing TIR data, in order to efficiently generate policy-aligned TIR data, we propose TIRGen, generator-refiier-base data construction pipeline. The generator is responsible for generating natural language reasoning steps, while the refiner eval- uates whether steps can be transformed into executable code and interacts with an external exccutor to refine the reasoning. This iterative process yields a TIR dataset that is naturally aligned with the generator's 's policy and broadly applicable across diverse models and tools. (2) For performing fine-grained optimization, we are motivated by the key insight that the success of an intermediate tool call is a strong predictor of the final answer's correctness. Our experiments later confirm this insight. Based on this, we introduce a hierarchical RL strategy that combines episode-level and step- level optimization. At the episode-level, we directly optimize for the correctness of the final answer. Concurrently, at the step-level, we apply fine- ine-grained optimization to execution failure steps, specif- ically enhancing the model's code generation ability. (3) For enhancing inference, we propose a self-correction mechanism that leverages immediate feedback from tools to dynamically revise its CoT during inference. When code invocation fails, it backtracks and explores alternative reasoning paths, thereby significantly enhancing the model's 's reasoning robustness and overall performance.
We evaluate our method on diverse challenging and widely-used benchmarks, including MATH500 (Hendrycks et al., 2021), I), AIME 2024 & 2025, AMC, Minerva Math (Lewkowycz et al., 2022), and Olympiad Bench (He et al., 2024). THOR establishes a new of (SOTA) result among models of comparable size across architectures and scales, while reducing inference overhead. It further improves performance on code generation benchmarks HumanEval, MBPP (Liu et al., 2023), and LiveCodeBench, validating the effectiveness and generalizability of our approach.
2

---

Published as a conference paper at ICLR 2026
Our primary contributions troduce TIRGen, a pipeline for generating TIR data, applicable across diverse models, and better aligned with the preferences of the policy model. 2) Hierarchical Optimization. We propose hier- archical reinforcement learning approach that combines episode-leve and step-level optimization. 3) Self-correction Inference Enhancement. We introduce self-correction mechanism that lever- ages immediate tool feedback to revise reasoning steps during inference. 4) Superior Performance and Broad Generalization. Our approach generalizes across reasoning and non-reasoning models, achieving competitive results on mathematical benchmarks and consistent gains on code tasks.
follows: 1) Tool-Integrated Data Construction Pipeline. We in-
2
METHODOLOGY
2.1 PROBLEM FORMULATION
In the context of tool-integrated reasoning, an LLM solves mathematical problems by interleaving natural language reasoning with tool invocations. Specifically, we formulate an LLM, parameterized by 0, as policy k.. Given problem q and a corresponding instruction I, this policy Ty autore- gressively generates an entire interaction trajectory т, which is an alternating sequence of thoughts, actions, and observations:
T= (r
(1)
where r² is a step of natural language reasoning, a' an action of tool call, of the observation returned by the external execution environment after executing action a', and is the number of reasoning steps. This process is formulated as an iterative think-act-observe loop. The model incor- porates the new observation o' into its context to inform the generation of the subsequent thought p++ and action al+1, This cycle continues until the model produces the final answer within its last thought r", thereby concluding the trajectory. Formally, the likelihood of generating a specific trajectory is factorized as:
(ao q, Pro (r" 1,,, 1m-11
Pro (r' gg 1,1, Poo (a' r', prr.gg,, 1,
Thought
Action
, a o denotes the history of the previous interactions. Each term is modeled as product of token-level probabilities.
where H fr- al, ,.....,
2.2 TIRGEN: TIR DATA GENERATION PIPELINE
Generator
Refiner
Code Solvable?
Reasoning Step
LEF
dataset
reavoning
Multi-stage Filter
Format Format Consistency
Sampled trajectory T
Code Quality
Think Step
Difficulty code Balancing
Comvert code
solvable
Think Act
Exscuste
Observe
Observation
Action
dotoset SeT
Figure 2: The TIR data construction pipeline. In this pipeline, the Generator agent generates reason- ing steps. The Refiner agent identifies tool-executable steps and converts them into augmenueedteed reasoning steps. After multi-stage filtering, obtain the cold start dataset DEFT-
Existing methods for TIR highlight a significant need for high-quality training data. Most ap- proaches rely on simply prompting external large models to synthesize TIR data for non-reasoning models (Gou et al., 2023; Li et al., 2025b), but these approaches fail to generalize effectively to reasoning models such as DeepSeek-RR Although START (Li et al., 2025a) constructs long-CoT CoT TIR data using a rule-based prompt-hint approach, the resulting trajectories often contain redundant
3

---

Published as a conference paper at ICLR 2026
Algorithm TIRGen: TIR Data Generation Pipeline
Input: Generator model genn Refiner model Tiefiners Dataset Dg. Code interpreter sandbox S.
2: Initialize: Raw cold start dataset Draw f- 0
3: for question q E Dg do
4: Initialize trajectory T +- (q)
5: while not IsSolved(T) do
6:
r g- (- r); |r'1 < Latep JudgeCodeSotvable Tugs + ExtractLogic, Tr Tiether a' - ConvertToCodee Meetoa
Generator Tgen generates reasoning step Identify operation solvable with code by Trefiner > Step 1: Extract the pure reasoning part Step 2: Convert calculation part code
7:
then
8:
9:
Mlgicc
10:
+ S(a')
> Step 3: Execute code to get observation
11:
+ T (Tiogios at, o')
12:
else
13:
7 (r')
14:
Draw t D Draw fr}
15:
DEFT & MuliStageFFller(Drraal
Filter for format, code quality and difficulty
Return DSFT
code invocations. Therefore, existing TIR data construction methods face critical shortcomings, in- cluding style mismatches between the generated data and policy models, well poor scalability to reasoning models. To overcome these limitations, we introduce TIRGen, an automated TIR data synthesis pipeline based on an generator-refinet framework, illustrated in Figure 2.
In this framework, the Generator generates a natural language reasoning step with a maximum length of Luep- The Refiner agent then evaluates this step to identify operations that can be more efficiently solved with code, such as numerical calculations or equation solving. Upon identifying such an operation, the Refiner converts it into executable Python code snippet while preserving the Gen- erator's original reasoning logic. The Refiner then interacts with code interpreter to obtain a precise exccution result, which replaces the original operation and produces code-augmented reasoning step. The Generator resumes reasoning from this revised step, and the iterative cycle continues until a complete solution reached, as described in Algorithm 1. This design yields two key advantages:
Policy Alignment. Because the Refiner observes only isolated reasoning steps without the full problem statement and final answer, the synthesized data directly reflects the Gener- ator's intrinsic reasoning preferences. This ensures that the data remains in-distribution. thereby mitigating the performance degradation commonly caused by training on out-of- distribution data (Gudibande et al., 2023; Chen al., 2024).
Reduced Reliance on Large-scale Models. The Generator is responsible for high-level mathematical reasoning, while the Refiner requires only basic cnsruction-followiin and code-generation skills. This division of labor decomposes the complex task into subprob- lems that are collaboratively solved by the two agents.
After sampling, we apply a multi-stage filtering procedure: (1) Format Consistency. We remove samples with erroneous tool calls or incorrect formats, and enforce that final answers are wrapped in \boxed(), ). (2) Code Quality. We discard code that fails to execute or contains only trivial operations. Retained examples must include library invocations (e.g., sympy, numpy) or control flows (loops or branches).
(3) Difficulty& & Call-Round Balancing. To ensure diversity problem complexity and tool invocation rounds, we stratify samples by the number of code calls and apply moderate down-sampling within each subset. Additionally, we remove instances solvable by a pure CoT baseline, ensuring that all retained examples require tool integration. This comprehensive procedure yields the final dataset for the cold start phase, denoted as DSFT-
2.3
HIERARCHICAL RL TRAINING STRATEGY
2.3.1
COLD START
We initialize our model with a cold start procedure (Guo et al., 2025), utilizing the dataset DSFT generated by TIRGen. The primary objective of this stage equip the model with the fundamental
4

---

Published as a conference paper at ICLR 2026
(a) Episode-level Optim.
Acouracy Reward
Rg
(moth tosk)
01
T2
F2
R2
Datoset
Policy model
...
Execution folluns
Fo
rd
0
42
og
3Rg
(b) Step-level Optim.
o2
(code task)
4a
02
Step Dataset
4G
Tougen
Esccutiio failed steps A...
Splitting Steps
Regerierate
Pass Reword
Self Correction Infer Enhancement.
Policy Codrection
Cornection and regenerate
Figure 3: A hierarchical optimization framework comprising (a) episode-level RL for mathematical problem solving and (b) step-level optimization for code generation. In addition, we introduce (c) self-correction mechanism for online error correction during inference.
patterns of tool invocation. This initial training is crucial, as reasoning models often struggle to invoke code effectively before cold start. To this end, we perform supervised fine-tuning on the base model 7e with the loss function follows:
SSFT B(g.
Σ log (o I wo.--))
where 7' is the length of response. In this way, we obtained the policy model after the cold start.
2.3.2 HIERARCHICAL REINFORCEMENT LEARNING
In the hierarchical optimization phase, we employ RL to refine the model's policy for invoking tools solve complex problems. Conventional RL methods perform optimization the episode-level, relying a sparse reward signal derived solely from the correctness of the final outcome (Mai al., 2025; et al., 2025b; Feng al., 2025; Tao et al., 2025). Our empirical analysis reveals crucial insight: the success of an intermediate tool call is strong predictor of the final answer's correctess. Motivated by this finding, we propose a hicrarchical reinforeement learning framework that combines coarse-gre grained,, episode-level optimization to enhance the model's problem-solvvee ability with fine- fine-grained, grained, step-level optimization to improve code generation for individual tool calls, shown in Figure 3. The observation validated in Appendix C.I.
Episode-level Optimization. At the episode-leve optimization, we adopt GRPO (Shao et al., 2024) and use the correctness of the final answer the reward to enhance the tool invocation strategy of the model, as shown in Figure 3(a). The complete episode-level objective is:
Cepis (0) =E(q DRL {sile (o( T
Σ
Σ
S (Si (=d()))))==
(4)
(o(silu) Tea (s:(g)
To (8:(g)
min
A,, clip(
Clow + Eligh) A.) (x)
To.u (stlq)
where G denotes the group size, and A, is the advantage of response s;, defined as A, R; mean (
std(
The reward is given by R; if answer is correct, and 0 otherwise. The indicator function
5

---

Published as a conference paper at ICLR 2026
(ss.c) 1 indicates that the token St, is generated by Te rather than observed from external executor. Additionally, to better exploit successful trajectories, we incorporate a language modeling loss LNLL. weighted by coefficient a, applied to examples with positive advantage Tpas. directly reinforeing the likelihood correct samples during RI. training (Yue et al., 2025b).
(N) Ex.ET
log (8(6))))
(5)
BLET
To stabilize training, we filter out trajectories with failed code executions during episode-level opti- mization to avoid inappropriate penalties. Since execution failures may arise not only from model- generated errors but also from environment issues or sandbox limitations.
Step-level Optimization. After sampling full trajectories, perform step-level optimization to correct code errors using execution feedback, as illustrated in Figure 3(b). This stage focuses specif- ically reasoning steps that resulted in failed actions A First, we construct step-level optimiza- tion dataset D Datep. For each failed step, we treat it as query and generate group rollouts, where the exccution correctness of the generated code provides the reward signal. Consequently, it is crucial to ensure that the rollouts within each group cover diverse execution outcomes. Next, we describe the construction of D step and its corresponding optimization method.
Within a think-act-observe tuple (r",a', a'),",, a failed execution inherently indicates an error exists in the action at, Moreover, since a' is conditioned actions. To overcome this problem, we implement a backtracking procedure. We partition the erroneous thought r into prefix rfee and suffix r'ar of length L Subsequently, we condition the model on the history up fiee and regenerate a new reasoning suffix and action. This procedure produces the dataset D step for fine-graine step-level optimization:
r', fixing rf limits the diversity of generated
D Detep (p,ef(T,t) t) a' A T) (r. (g,r), r a=, ,o',,.,,''" ,-_ a ,0--1 ppp))
(6)
pre r suf+
This formulation defines fine-grained code generation task: given reasoning prefix pref (), the model must correctly regenerate the subsequent reasoning suffix f and action â'. At this stage, each sample contains a single think-act-observe loop, and the reward is computed from execution correctness. We optimize the policy with the following step-level loss:
Cstep (0) = =Elp- D Dsteps (01 mo(S
Σ
2t=l I(S a:I(e);))=1
==1
To (s(pp))) K0 (s)
A', clip( (o(3 p) W0 (6/1p)
min
Slow, + tnigh) ) + (lC.
where sf and A denote the responses and advantages at the step-level. In step-level optimization, each sample contains only single think-an act- act-observe loop, and its reward is directly determined by whether the corresponding code execution succeeds. For a group of step-level rollouts sampled from the same prefix ( the advantage is computed
r mean(r")
A
sld(r')
where r' denotes the code-execution reward within the group. The final training objective combines both episode-level and step- losses:
((0)=
Capis (8) +
Cstep (8). su
(10)
2.4 SELF-CORRECTION DURING INFERENCE
During inference, our model follows the think-act-observe loop. To exploit immediate feedback from tool execution, we introduce a self-correction mechanism that dynamically corrects erroneous reasoning steps, as shown in Figure 3(c). Specifically, when action of fails to execute, it indicates that both the action a' and its associated reasoning step r' are likely incorrect. To explore alterna- tive solving paths, the model backtracks pe and partitions it into prefix rfc and suffix r uu as

---

Published as a conference paper at ICLR 2026
Table I: Comparison with late-of-the-art methods mathematical benchmarks, the best results in bold and the second-best best are underlined. Code use indicates whether code tools are employed. denotes our reimplementation results of Avg @ 4, indicates results from their official relcases.
Code Use
MATH
AIME
AIME
AMC
Minerva Math
Olympiad Bench
Model
Avg.
500
2024
2025
2023
Won-Reasoning Models (Lightweight)
Qwen3-1.7B
72.0
15.8
7.5
41.9
39.0
42.2
36.4
Owen2.5-Math-1155
40.0
7.1
5.4
22.7
15.7
19.7
18.4
THOR-1.5B
78.2
36.7
20.0
67.5
38.2
54.0
49.1
Nea-Rnasoning Models (Standard
GPT-4o- 1120
77.2
11.1
7.6
60.0
53.4
43.9
42.2
rStur-Math-7B
78.4
26.7
47.5
47.1
Eurus2-PRIMME-7B
79.2
26.7
13.3
57.4
38.6
42.1
42.9
AKT1ST-7B
67.6
15.6
47.0
37.9
TATA-7B $
73.0
35.9
TORL-7B
82.2
40.8
29.2
73.8
53.8
57.3
56.2
AutoTIR-7B-
62.6
33.3
16.7
ZTRL-7B
80.2
50,0
26.7
wwen2.5-Muth77-lImttee
79.8
10.8
11.7
54.4
44.8
43.1
40.9
Qwen2.5-Math-7B
51.5
8.3
5.8
33.1
26.7
22.9
24.7
Qwen2.5-Math-7B
64.7
20.6
13.1
49.7
28.1
37.9
35.7
THOR-7B
87.5
50,0
33.3
81.3
53.9
61.1
61.2
Reasoning Models (Lightweight)
DeepSeek-RI ---Distill--wen--....
82.8
28.9
23.3
62.9
26.5
43.3
44.6
DeepScaleR-1.. 1.5.-----1B--Piei Qwen3-1.7B THOR-Thie Thinking-1.k68 1.78
87.8
43.1
30.0
73.6
30.2
50.0
52.5
91.0
45.0
31.7
80.6
52.7
65.7
61.1
92.8
60.0
33.3
82.5
54.4
68.8
65.3
Reasoning Models (Standard-scale)
QwQ-32B-Preview
90.6
50.0
40.0
80.0
39.0
58.5
59.7
START-32B
94.4
66.7
47.1
95.0
OpenMath-Nemotton-11
72.9
57.5
DeepSeek- RI-Disiill-Qeen-75
93.5
55.8
40.0
86.8
57.7
71.0
67.5
Qwen3-8B
95.5
64.2
54.2
91.2
64.4
77.7
74.5
THOR SROR--hinking-88
96.8
77.5
62.5
96.8
65.6
79.7
79.8
previously described. Conditioned on the history up rf the model then regenerates rea- soning suffix At and revised action â', thereby enabling online error correction during inference. The correction procedure can be repeated for up to Neor attempts. Importantly, each attempt only requires regenerating the suffix ft., and the corresponding action à', rather than recomputing the entire trajectory. Thus, the additional computational cost is minimal compared to the total cost.
3 EXPERIMENTS
3.1 DATASET
We evaluate the effectiveness of THOR on diverse set representative and challenging bench- marks for both mathematical reasoning and code generation. For mathematical reasoning, our evalu- ation covers the high school-level MATH 500 (Hendrycks et al., 2021). as well competition-leve benchmarks including AMC 2023, AIME 2024, AIME 2025, MinervaMath (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). These benchmarks span geometry, algebra, and num- adopt the LLM-as-Judge approach, using Qwen3-
ber theory. To evaluate answer correctness, 32B as the judge model to compare model predictions against the ground truth. For code generation, adopt EvalPlus (Liu et al., 2023) on HumanEval" and MBPP" to assess basic programming skills, and LiveCodeBenchh v (Jain et al., 2024) for competition-level programming tasks.
3.2 COMPARISON WITH STATE-OF-THE-ARR METHODS
To assess THOR's s effectiveness and generalization, we conduct extensive experiments on both non- reasoning and reasoning models. For the non-reasoning setting, use 0wen2.5-Math- (Yang

---

Published as a conference paper at ICLR 2026
Table 2:
Comparison of test-time scaling methods under equal compute budget (N 8). The
proposed self-rewarded strategy relies solely on execution pass rate as an intrinsic signal and does use any external PRMs.
MATH
AIME
AIME
AMC
Minerva Math
Olympiad Bench
Model
Avg.
500
2024
2025
2023
Non-Reasoonig Model
THOR-7B
87.5
50.0
33.3
81.3
53.9
61.1
61.2
Cell-Consistency
89.8
50.0
36.7
80.0
56.3
64.8
62,9
Interni .M2-RM
88.2
56.7
36.7
77.5
55.9
62.8
63.0
Swee2.5-Math-PPEr
89.6
53.3
33.3
77.5
57.7
66.5
63,0
Self-rewarded
87.7
53.3
38.3
83.8
54.9
61.5
63.31
Internl M2-RM Self-rewarded
88.0
60.0
33.3
85.0
56.3
63.5
64.47 3.24
Reasoning Model
THOR-Thinking-88
96.8
77.5
62.5
96.8
65.6
79.7
79.8
Self-Consistency
97.4
90.0
66.7
95.0
63.6
81.6
82.4
InternLM2-Re
73.3
97.5
81.5
83.2
+
-wwen.55-MthhPPPP
97.0
73.3
63.3
95.0
64.7
80.4
79.0
Self-rewarded
97.2
86.7
70.0
97.5
65.8
82.0
83.21
Internl .22-MM Self-rewarded
98.0
86.7
73.3
100.0
65.4
82.1
84.31 4.5%
et al. 2024) to obtain THOR-7B. For the reasoning setting, we adopt Qwen3-8B (Thinking Mode) further test generalization on the correspond-
(Yang et al., 2025) to obtain THOR- SHOR-Thinkinng-BB ing lightweight models, Qwen2.5-Math- 1.B and Qwen3-1.7B. To mitigate randomness, adopt Avg@4 as the evaluation metric. The maximum context length is 16,384 tokens for reasoning mod- els and 4,096 tokens for non-reasoning models. See Section D for implementation details.
For comparison, we evaluate THOR against diverse set TIR and CoT-based methods, including ARTIST-7B (Singh al., 2025), TATA-7B (Xu al., 2025), AIRoTIR (Wei al., 2025), TORL-7B (Li et al., 2025b), Surus-2-PRIME--7 (Cui et al., 2025), rStart-Math-78 (Guan et al., 2025), ZTRL- 7B (Mai al., 2025), and GPT-4o (Hurst et al., 2024). We also include long CoT methods include (Guo et al., 2025), DeepScaleR (Luo et al.,
START (Li et al., 2025a), DeepSsek-RI-Disiill-Qeee 2025), QwQ (Team, 2024) and Nemotron (Moshkov al., 2025). As shown in Table 1, THOR achieves substantial improvements on both non-reasoning and reasoning models, demonstrating its effectiveness in enhancing mathematical reasoning capabilities, Moreover, despite relying only on small policy models, THOR remains competitive with state-of-the-art systems and even surpasses many larger models, while maintaining low inference cost. Detailed inference costs are provided in Appendix C.2.3.
3.3
INFERENCE ENHANCEMENT
Traditional test-time scaling computation approaches, such Best-of-N (BoN), often rely exter- nal Outcome Reward Models (ORMs) to evaluate the trajectory quality. We propose an ORM-free search method that exploits intermediate code execution feedback as an intrinsic reward signal. Specifically, we sample N independent candidates and select the one with the highest execution pass rate, thereby removing the need for an external reward model. Each candidate does not employ the self-correction mechanism. As shown in Table 2, the self-rewardes BoN strategy significantly outperforms single path reasoning by exploring larger search space, indicating that code execution success can serve as reliable reward signal for assessing reasoning quality. Notably, the perfor- mance gains are more pronounced on challenging datasets such as AIME 2024 and 2025, indicating that complex reasoning tasks benefit more from precise code execution.
We compare our self-rewarded selection with self-consistency (Chen et al., 2023) and external PRMs, including M22emLLMM-1.8B-RN (Cai et al., 2024) and Swen2.5-MMathPRM (Zhang et al., 2025). We further evaluate a hybrid strategy that combines PRMs with our intrinsic signal. When N is large, multiple candidates often share identical code-pass rates; in such cases, PRMs provide an additional preference score that helps separate candidates with the same score. As shown in Ta- surpasses PRM-based selection.
ble 2, the self-rewarded inference method already matches
8

---

Published as a conference paper at ICLR 2026
Table 3: Results of the ablation each component. Cold start the data generated by TIRGen in Section 2.2. EpisRL and StepRL correspond to episode-level and step-level optimization defined
Section 2.3. SelfCorr denotes self-correction during inference in Section 2.4.
Code
Cold
Epis
Step
Self
MATH
AIME
AIME
AMC
Minerva Math
Olympiad Bench
Avg.
Use
Start
RL
RL
Corr
500
2024
2025
2023
Non-Reasoning Model
$1.5
8.3
5.8
33.1
26.7
24.7
72.9
30.0
11.7
53.8
41.5
38.3
41.4
64.7
20.6
13.3
49.7
28.1
35.7
T4
86.9
42.7
30.8
77.5
52.2
58.2
58.1
T5
87.3
45.0
31.7
80.0
53.4
60.5
59.7
T6
87.5
50.0
33.3
81.3
53.9
61.1
61.2
Reasoning Model
T1
95.5
64.2
S4.2
91.2
64.4
77.7
74.5
T2
95.7
65.8
52.5
93.1
64.4
78.0
74.9
d
92.9
60.8
51.7
88.8
61.4
71.4
71.7
60.0
95.0
64.5
78.9
77.7
74.2
60.0
95.6
65.4
78.5
96.8
77.5
62.5
96.8
65.6
79.7
79.8
Moreover, combining with PRMs yields further improvements, indicating that the two types of signals are complementary. In addition, our reward is model-agnostic and benefits both reasoning and non-reasoning models, whereas PRMs are fundamentally limited by the coverage and biases of their training data, which often restricts generalization.
3.4 ABLATION STUDY
To analyze the contribution of each component in THOR, we conduct an ablation study by selec- tively removing key modules. This results in six system variants T1-T6, built upon Qwen2.5-Math- 7B and Qwen3-8B, as summarized in Table 3.
Impact of Cold Start. The cold start data generated by TIRGen provides a foundation for subse- quent RL. The goal of RL is to refine the model's policy towards its capability frontier (Yue et al., 2025a), which can be estimated using pass@k (Chen et al., 2021). Consequently, we evaluate cold start by its impact on pass @ 66 and code invocation ratio. We further compare with other TIR dataset, including the Long TIR data generated by Nemotron (Moshkov et al., 2025) and the Short CoT TIR data from ReTooo (Feng et al., 2025). As shown in Figure 4, TIRGen substantially improves both metrics, effectively expanding the capability fronticr. Compared with other datasets, the data generated by TIRGen effectively mitigates performance degradation arising from out-of-distribution samples. More critically, it actively encourages reasoning models to utilize tools and dramatically increases the code ratio, behavior rarely in the baseline.
B Nemotio
G Raxeline (unG
Refooel
MATH
Reseline
TERGesHOuns)
MATIL
Nemetron
Reasoning Models
Olym
AIME 2024
AIME 2024
60
Reasoning Modele
Minerva
AIME 2025
Code Ratlo%)
20
40
60
80
100
AMC 2023
(1) Code invocation ratio
Pass reasoning model
Passúi 16 for non-reasoning model
Figure 4: Ablation on cold-start efficiency. We compare our TIRGen against other TIR datasets, including Long CoT from Nemotron and Short CoT from ReTool. Results are reported code ratio (a) and pass@1 16 (b) and (c), demonstrating the effectiveness TiRGen and cold start.

---

Published as a conference paper at ICLR 2026
Impact of Tool-Integrated RL. To evaluate the effectiveness of tool-integrated RL, apply two different RL strategies to the baseline model (T1): a standard CoT-based RL and episode-level TIR-based RL, which yield T2 and T4, respectively. While both outperform the baseline, T4 achieves substantially greater improvements than T2, validating the effectiveness of TIR RI..
Impact of Hierarchical RL. By incorporating step-level optimization into episode-level RL (T4), we obtain T5. T5 achieves further performance gains across most datasets, underscoring the impor- tance of fine-grained optimization for enhancing code generation capabilities TIR setting.
Impact of Coff-Correction By leveraging step-level feedback from intermediate code, we con- struct self-correction mechanism, yielding variant T6. We set the maximum number of correction attempts Nonr = 4, which leads to substantial performance gains. This result highlights the critical importance of successful code generation and execution for the final outcome.
3.5 GENERALIZATION ON CODE BENCHMARRS
We also evaluate THOR's code gen- eration abilities using the pass metric on HumanEval" .BBPP* and LiveCodeBench." As illustrated in Figure 5, our models achieve consis- tent improvements across all bench- marks. Notably, these gains are re- alized in zero-shot setting without any fine-tuning on code generation data. These results confirm that method strengthens both mathemat- ical reasoning and code generation, underscoring THOR's s robustness and versatility across distinct reasoning domains.
[fuman Eval"
Uluman
Lode
Live Clod Bends
Qwer?
STHOR Trwcku-
c cTHOk-18
75 (a) Reasoning Models
60
(b) Nou-reasoning Models
Figure 5: Pass@I accuracy on code generation benchmarks.
4 CONCLUSION
In this work, we address three core challenges in tool-integrated reasoning: construction of TIR data, hicrarchical optimization, and inference enhancement. We propose THOR (Tool-Integrated Hierarchical Optimization via RL), a novel hierarchical RL framework for TIR that fully leverages step-level feedback. First, mitigate the scarcity of TIR data, introduce TIRGen, multi-agent based TIR data construction pipeline. For model training, THOR integrates coarse-grained episode- level optimization for overall reasoning ability with fine-grained step-level optimization for code generation ability. During inference, tool feedback is used to dynamically adjust the reasoning and perform self-correction. Experiments demonstrate that THOR achieves substantial improvements across diverse models and benchmarks while maintaining a low inference cost.
REPRODUCIBILITY STATEMENT
For the reproducibility of our results, we have provided a detailed description of our method in Section and experimental setups in Appendix In addition, to further facilitate the reproduction, we will release our codes and datasets.
ETHICS STATEMENT
By integrating precise tool execution with hierarchical reinforcement learning, THOR significantly enhances the mathematical reasoning capabilities of LLMs. This advancement holds substantial promise for education and scientific research by offering reliable, automated assistance for complex problems mathematics, engineering, and the formal sciences. However, like any powerful LLM- based system, THOR carries risk of misuse, such generating misleading harmful content if deployed without proper oversight, Consequently, the development of robust ethical safeguards and responsible deployment protocols is important for its application in real-world scenarios.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant No. U25A20409.
REFERENCES
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 17682-17690, 2024.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui. Aijia Guo. Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv,, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang. Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang. Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang. Ziyi Wang, Xingjian Wei, Qizhen Weng. Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang. Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang. Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint vrXiv:2/07.03374 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. 2022.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.133:
2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint urXiv:2401.013:: 2024.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.00456, 2025.
Jiazhan Feng. Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms. arXiv preprint arXiv:2504.1 1536, 2025.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.
2023.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang- rstar-math: Small lls can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519 2025.
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary 1lms. arXiv preprint arXiv:2305. 15717, 2023.
11

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in llms through reinforce- ment learning. Nature, (45(8081):633-40 638. 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang. al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. /4008, 2024.
Zhiwei Hc, Tian Liang. Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath- 103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11556, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Pengfei Hu, Zhenrong Zhang, Qikai Chang, Shuhang Liu, Jiefeng Ma, Jun Du Jianshu Zhang, Quan Liu, Jianqing Gao, Feng Ma, et al. Prm-bas: Enhancing multimodal reasoning through prm-guided beam annealing search. arXiv preprint arXiv:2504.10222 2025.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16726. 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0799 079744 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sone Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. Start: Self-taught reasoner with tools. arXiv preprint 2553: 25X::22503.04655, 2025a.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025b.
Junfeng Liao, Qizhou Wang, Shanshan Ye, Xin Yu, Ling Chen, and Zhen Fang. Explainable Ilm unlearning through reasoning. In The Fourteenth International Conference on Learning Repre- sentations, 2026.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558-21572. 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Meng Luo, Bobo Li, Shanqing Xu, Shize Zhang, Qiuchan Chen, Menglu Han, Wenhao Chen, Yanxiang Huang, Hao Fei, Mong-Li Lee, and Wynne Hsu. Unveiling the cognitive compass: Theory-of-mint guided multimodal emotion reasoning, 2026. URL https: /aaxii.oorg. abs/2602. 00971.
12

---

Published as a conference paper at ICLR 2026
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
Xinji Mai, Haotian Xu, Weinong Wang, Yingying Zhang, Wenqiang Zhang, et al. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.0711 077773, 2025.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art math- ematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504. 16891, 2025.
Yicheng Pan, Zhenrong Zhang, Pengfei Hu, Jiefeng Ma. Jun Du, Jianshu Zhang, Quan Liu, Jianqing Gao, and Feng Ma. Enhancing the geometric moblem-solving ability of multimodal Ilms symbolic-neural integration. arXiv preprint arXiv:2504. 12773, 2025.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- -ür, Gokhan Tur, and Heng Ji. Toolrl: Reward all tool learning needs. arXiv preprint arXiv:2504.1322 2025.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Timo Schick, Jane Dwivedi-Yo Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke 7ettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539- 68551, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning. arXiv preprint arXiv:2505.01444 2025.
Zhou Tao, Shida Wang. Yongxiang Hua, Haoyu Cao, and Linli Xu. Dig: Differential ground- ing for enhancing fine-grained perception multimodal large language model. arXiv preprint arXiv:2512. /2633, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face, 2024.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL
https: /ttp///wwwmmmmmm/m//e//.../..... mariill.....itt github. /toii///lll hlwl://gg/322222
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint
2023a.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture- offture-of-experts. arXiv preprint arXiv:2406.1288 2024.
Peiyi Wang. Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce 1lms step-by-step without human annotations. arXiv preprint arXiv:2312.089.. 2023b.
13

---

Published as a conference paper at ICLR 2026
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. ofain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 01124424-248335 2022.
Yifan Wei, Xiaoyan Yu, Yixuan Weng, Tengfei Pan, Angsheng Li, and Li Du. Autotir: Autonomous tools integrated reasoning via reinforcement learning, arXiv preprint arXiv:2507.2 2025.
Xin Xu, Yan Xu, Tianhao Chen, Yuchen Yan, Chengwu Liu, Zaoyu Chen, Yufei Wang, Yichun Yin, Yasheng Wang. Lifeng Shang, al. Teaching Ilms according their aptitude: Adaptive reasoning for mathematical problem solving, arXiv preprint arXiv:2502. /0022, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self- reff-improvement. arXiv preprint arXiv:2409. /2122, 2024.
An Yang. Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.099 09388, 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:1 18099111188 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023b.
Fei Yu, Anningzhe Gao, and Benyou Wang. Ovm, autcome--supervised value models for planning in mathematical reasoning. arXiv preprint arXiv:2311.09724, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 2rXiv:2503..44777 2025.
Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.01491, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.11 13837, 2025a.
Yu Yue, Yufeng Yuan, Qiying Yu. Xiaochen Zuo, Ruofci Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05//8. 2025b.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mchran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.0733 2025.
Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. Marco-ol: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411.14400 2024.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao. Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507. 18071, 2025.
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE
We used LLMs, including GPT-5 and Gemini 2.5 Pro, only polish grammar and improve the clarity of the manuscript. All research ideas, experiments, and analyses were conducted by the authors.
B RELATED WORKS
LLMs have shown remarkable progress in mathematical problem solving. This section reviews relevant works, which we categorize into two main groups based on whether they integrate external tools: tool-free reasoning and tool-integrated reasoning.
B. TOOL-FREE REASONING
Search-based methods improve LLM reasoning by systematically explor- ing a large space of potential solutions. Early approaches leveraged prompting strategies like Chain- of-TThouhht (CoT) (Wei et al., 2022) and its search-oriented extensions, including Tree-of-Thought (ToT) (Yao et al., 2023a) and Graph-of-Thoughe guide this exploration, a prominent line of work integrates explicit Reward Models (RMs), includ- ing Outcome Reward Models (ORMs) (Yu et al., 2023; Zhang et al., 2024) and Process Reward Models (PRMs) (Wang et al., 2023b; 2024; Zhang et al., 2025) with search algorithms like BoN, step-level BoN, MCTS, and beam search. For instance, Marco-ol (Zhao et al., 2024) and rStar- Math (Guan et al., 2025) employ MCTS for systematic exploration, while PRM-BAS (Hu et al., 2025) uses Beam Annealing Search to balance search breadth and efficiency. Although these meth- ods yield significant performance gains, they have two key limitations. First, the large-scale search incurs substantial computational overhead at inference time. Second, and more critically, they do not directly optimize the model's ' internal reasoning policy, thereby constraining its ultimate capability.
Search-base baare Methods.
(GoT) (Besta et al., 2024). To more effectively
RL-based Methods. RL represents another dominant paradigm for enhancing LLM reasoning, where policy gradient (PG) methods have become a core technical route (Guo et al., 2025; Team al., 2025; Yang al., 2025). These methods can be categorized into value-model-baase and value-model-free approaches. Value-model-basee approaches are exemplified by Proximal Policy Optimization (PPO) (Schulman et al., 2017), which stabilizes training via policy probability ratio clipping. Its variants include VC-PPO (Yuan et al., 2025) with a decoupled-GAE to mitigate value bias and reward decay, and VAPO (Yue et al., 2025b) with a length-adaptive GAE address the bias-variance trade-off. Value-model-free methods bypass the need for explicit value critic. For instance, GRPO (Shao et al., 2024) estimates the baseline from group scores. This is enhanced by DAPO (Yu et al., 2025) with techniques like dynamic sampling and token-level loss, while GSPO (Zheng et al., 2025) performs optimization based on sequence-level likelihood ratios. A key limita- tion of these approaches is their primary reliance on sparse, episode-level reward signals. For tasks involving long reasoning chains, this reward sparsity impedes efficient policy learning.
B.2 IT REASONING
Integrating external tools, such as code executors, search engines, databases, and external APIs, has become a prominent strategy for augmenting the reasoning capabilities of LLMs while enhancing their interpretability (Liao et al., 2026). Early approaches focused on prompting methods, without integrating tool use into model optimization. For example, ReAct (Yao al., 2023b) demonstrated the use of prompting to invoke the Wikipedia API for question answering and fact verification. VOYAGER (Wang et al., 2023a) explored in-context learning to leverage predefined tools within Minecraft. Subsequent studies incorporated human-labeled or synthetic tool-integrated data during fine-tuning (Schick et al., 2023; Yang et al., 2024; Moshkov et al., 2025; Li et al., 2025a). How- ever, while effective in specific domains, the generalization is often constrained by the scope and quality the synthetic data. More recently, RL has been employed to learn dynamic tool- inttegrateed policies for mathematics reasoning (Mai et al., 2025; Li al., 2025b; Feng et al., 2025) and other tasks (Qian et al., 2025). Nevertheless, existing RL-based approaches often rely on prompt-base based triggers to initiate tool invocation, which limits their applicability models not previously exposed
15

---

Published as a conference paper at ICLR 2026
to tool-integrated training data, such DeepSeek-RI R1 (Guo al., 2025) and QwQ (Team, 2025). Furthermore, the step-level feedback provided by tools remains unexplored.
C MORE EXPERIMENTS
C.I STATISTICAL VALIDATION
To examine the relationship between code execution success and answer correctness, analyzed their joint distribution on the test set, as shown in Table 4. We then applied chi-square test of independence, which yielded a highly significant result (x 336.3, p 4.09 10-79), thereby rejecting the null hypothesis of independence and confirming statistical association between the two variables. These findings verified our research motivation: the success of an intermediate tool call is strong predictor of the final answer's correciness.
Table 4: Joint distribution between code execution result and answer correctness,
Code True
Code False
Answer True
3950
139
Answer False
1549
318
C.2 MORE ANALYSIS ON SELF-CORRECTION AND INFERENCE COST
C.2.1 ABLATION ON EXECUTION FAILURE REPAIR STRATEGIES
In Section 2.4, we introduced self-correction mechanism that backtracks to the most recent reason- ing step and regenerates suffix when code execution fails. While intuitive, execution failures can arise from different sources, including (i) syntax/runtime errors within the generated code and (ii) deeper logical inconsistencies in the preceding reasoning step. To disentangle these cases and evalu- ate the necessity suffix-level regeneration, we conduct a controlled ablation in Table 5, comparing three alternative repair strategies: (a) Action-only repair, (b) Step-suffix repair, (c) Full re-repair.
Table 5: Ablation of different repair strategies for self-correction during inference.
MATH 500
AIME
AIME
AMC
Minerva Math
Olympiad Bench
Model
Avg.
2024
2025
2023
Non-Reasen Reasoning Model
THOR-7B
87.3
45.0
31.7
80.0
53.4
60.5
59.7
THOR-7B action-only repair
87.0
46.3
32.1
82.5
53.3
60..
60.3
THOR-7B step-suffix repair
87.5
33.3
81.3
53.9
61.2
THOR-7B full re-plan
86.6
34.2
81.3
52.2
61.1
60.4
Reasoning Model
THOR T
96.6
74.2
60.0
95.6
65.4
79.0
78.5
THOR-Thinking action-only repair
96.8
75.0
60.0
95.6
64.0
79.7
78.5
THOR-Thinking-S step-suffix repair
96.8
77.5
62.5
96.8
65.6
79.7
79.8
THOR-Thinking-88 -B full re-plan
96.2
75.8
62.5
98.1
63.4
79.3
79.2
C.2.2 ANALYSIS OF SELF-CORRECTION MECHANISMS
To further evaluate effect explicit self-correction, we compare three settings: the baseline model without any correction, THOR-7B relying solely on its emergent self-correction behavior, and THOR-7B equipped with our explicit self-correction mechanism. We report results across accuracy, code ratio, the number of failed and successful tool calls, and the overall code pass rate. As shown in Table 6, the explicit mechanism markedly reduces failed executions, significantly increases the code pass rate, and yields consistent improvements in accuracy.
16

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of emergent and explicit self-correctinn across accuracy and code pass metrics.
Code Ratio
Failed Code
Success Code
Code Pass
Model
Acc
Num
Num
Ratio
Ren-RRasoning Model
Qwen2.5-Math-7B
35.7
72.7
858
5806
87.1
THOR-7B
59.7
96.6
685
5900
89.6
THOR-7B self-correction
61.2
96.2
82
6050
98.7
Reasoning Model
Qwen3-8B
74.5
8.5
260
1071
80.5
THOR-Thinkinge Thinking-81
78,5
91.3
2311
10535
82.0
SHOR--hinking--85 self-correction
79.8
90.6
155
12532
99.0
C.2.3 INFERENCE COST ANALYSIS
We evaluate the inference efficiency of THOR by analyzing its token consumption and runtime performance. Our data construction process, guided by TIRGen, identifies redundant computational steps within reasoning chains and transforms them into executable code. By learning from this data, THOR is trained to generate computations at inference time.
concise solutions, effectively leveraging tools simplify
In addition to token usage, further evaluate the total inference seconds and frames-per-second (FPS) throughput under a unified vLLM inference environment. As reported in Table 7, THOR consistently generates fewer tokens than the baseline models, while also reducing overall inference latency. These reductions directly translate into improved FPS. These results already include the overhead of self-correction.
Table 7: The number of tokens consumed during inference across different benchmarks.
MATH 500
AIME 2024
AIME
AMC
Minerva Math
Olympiad Bench
Avg Tokens
Infer Seconds
Model
FPS
2025
2023
Non-Reasoning Model
Swen2.5-Math--7
866
1283
1325
1132
802
1090
1083
1379
4.48 4.671 4%
THOR-7B
705
1351
1420
928
729
981
1019 6
1324
Reasoning Model
Qwen3-8B
5102
11986
13022
7989
6906
9238
9041
7244
0.85
THOR-Thinkings THOR--hinnkng-8B
4506
10338
11807
6749
5444
8205
7841
6280
0.98 15%
D IMPLEMENTATION DETAILS
D.1 TIR DATA CONSTRUCTION & COLD START
In the cold start stage, for data source construction, we collected a large set of question-answer pairs from various public datasets, including DAPOI7k (Yu et al., 2025), DeepMath 103k (He et al., 2025), and Deepscaler40k (Luo et al., 2025), which cover mathematical problems of diverse difficulty levels. For sampling, set the hyperparameters to temperature of 0.6. top-k of 50, and top-p of 1.0. After processing with TIRGen, we obtain 29,217 short CoT TIR samples from Qwen2.5- Math-7B and 57,598 long CoT TIR samples from Qwen3-8B. The distribution of code invocation counts in the final cold start dataset DSFT is shown in Figure 6. In TIRGen, the Generator agent uses the corresponding policy model, while the Refiner agent uses Qwen3-32B (Non-thinking) for its strong instruction-folloviii
capability. For non-reasoning models, we set Ssep 512, whereas code execution environment.
for reasoning models we set Lasp = 4096. Our experiments utilize SandboxFusion as the external
For cold start, models are full-parameter fine-tuned for epoch with global batch size of 256. We use AdamW optimizer (Loshchilov & Hutter, 2017) with fixed learning rate of 10-6
17

---

Published as a conference paper at ICLR 2026
Reasoning Models
Non-Rcasoning Models
27141
30910
20000
16732
20000
1736
6685
2000
3271
280
200
2000
20
200
2
3
41
1
3
4
Code call rounds
Code call rounds
Figure 6: The distribution of code call rounds in the cold start dataset DSFT.
ZeRO (Rajbhandari et al., 2020) is adopted for memory-effiiiett training. For reasoning models, the maximum context length is 20,000 tokens, while for non-reasoning models it is 4,096 tokens.
D.2 HIERARCHICAL REINFORCEMENT LEARNING
During the RL stage, training. we adopt an off- ooflic variant of GRPO and employ dynamic data filtering (Yu et al., 2025) accelerate convergence. In the early phase step-level optimization, the code execution success rate is relatively low, resulting in many failed steps. We downsample these failed steps to ensure that the number of step-level optimization queries does not exceed that of episode-level optimization, therehy prioritizing the model's overall problem-solving ability.
the publicly available dataset ToRL28k (Li al., 2025b). To stabilize
16, the group sampling temperature to 1.0, the weight coefficient on the loss LNLL to 0.01, and the learning rate to x 10-6, The KL-divergence term is omitted. Clipping coefficients are configured as Ehigh 0.28 to encourage diversity and 6low 0.2. During rollout, use the maximum sampling lengths of 4,096 tokens for non-reasoning models and 16,384 tokens for reasoning models, with up to rounds code interaction. In the step-level optimization, the backtracking length Touf is 500 for reasoning models and 100 for non-reasoning models. rule-based reward function is used to mitigate reward hacking. All experiments are conducted on 16 NVIDIA H200 GPUs.
We set the group size
D.3 INFERENCE WITH DELF-CORRECTION
During inference, we set the sampling parameters to a temperature of 0.6, top-p of 1.0, and top-k of 50. The maximum number of self-correction attempts Neor is set to 4. For non-reasoning models, the backtracking length Lut set to 100, while for reasoning models is set to 500.
D.4 PROMPT SETTING
In this subsection, we provide the complete prompt settings used in our framework. Figures 7 and illustrate the prompts designed for the Generator and Refiner agents in the TIRGen data construc- tion pipeline, respectively. Figures 9 and 10 present the prompts for tool-integrated reasoning in reasoning models and non-reasoning models.
E LIMITATION AND FUTURE WORK
In this work, we systematically investigate the effectiveness of tool-integrated reasoning. focusing specifically on code integration for mathematical problem solving. Although we have verified the effectiveness of code-integrated reasoning, other types of tools, such search engines, symbolic systems remain to be explored. Due to computational constraints, we did not experiment with larger- scale models such 32B or 72B. Nevertheless, we have validated the effectiveness of THOR across
18

---

Published as a conference paper at ICLR 2026
multiple model sizes ranging from 1.5B to 8B, which demonstrates that our approach generalizes well across different scales. In the future, we will explore larger models and conduct a deeper investigation into multi-tool joint optimization.
F
CASE STUDY
In this section, present case study to illustrate how THOR performs tool-integrated reasoning, including non-reasoning models in Figure 11, 12, and reasoning models in Figure 13.
19

---

Published as a conference paper at ICLR 2026
You are scientist proficient in computer science and mathematics. will provide you with detailed thought process (chain of thought) from powerful model for mathematical problem. Your task is to revise this thought process.
## Revision Goal:
Without altering the original model's reasoning flow and methods, identify any steps in the chain of thought that can be assisted by code for numerical calculations, equation solving, hypothesis testing, data processing, etc., and replace these natural language descriptions of computational processes with corresponding code execution and predicted results.
#Н# You need complete the following steps:
1. Identify Codable Parts: Carefully analyze the original chain of thought to pinpoint any parts involving specific numerical calculations, algebraic operations, set operations, logical verifications, etc., that can be precisely execuled using Python code.
2. Write Python Code: For the identified parts, write clear, concise Python code blocks that can accomplish the corresponding computational tasks. Each piece of your code should be carefully considered, not just performing simple arithmetic operations like addition, subtraction, multiplication, or division. Every code block should return the output with prrnt()' function.
3. Predict Code Output: Provide the expected output of the Python code you have written. 4. Embed in Chain Thought: Embed the Python code and its exccution results into the chain of thought, ensuring that the revised thought process remains logically correct and complete, Keep the parts that were not modified exactly as they were! Also, only modify the parts included in the original chain of thought; do not extend or continue solving parts not covered by the chain of thought.
### Note:
1. Multiple Code Blocks Supported: You can output multiple code blocks as needed to assist different computational steps in the reasoning process.
its own, does not depend on any previous variables, and imports any required librarics independently.
2. Independent Code Blocks: Each of your code blocks is independent,
3. Maintain Consistency of Thought: Do not alter the core reasoning logic, step order, or basic methods of the original chain of thought. Your task is solely to convert specific computational and verification processes described natural language into equivalent code execution and output.
4. Accuracy: Ensure that the code you provide is correct and that the predicted output is accurate.
5. No Problem Solving: Your is merely to revise the chain of thought. Parts that not need modification should remain unchanged and be outputted as such.
If Python code can used assist in solving, please strictly follow the format below to ensure your revised content is easily parsable by machines and understandable by humans:
Original reasoning step, ppython
"output Code output
Original reasoning steps, ...
pythoon Python block 2
''output Code output 2
Continue reasoning...
Figure 7: The prompt used by the Refiner agent in TIR data construction pipeline, TIRGen.
20

---

Published as a conference paper at ICLR 2026
Please reason step by step and put your final answer within [boxed Anin [question]
Figure 8: The prompt used by the Generator agent in our TIR data construction pipeline, TIRGen.
You are scientist skilled in mathematics and computer science. Please integrate natural language reasoning and Python code to solve mathematical problems. You can use Python code during the thinking process for numerical calculations, equation solving, hypothesis verification, data processing, and any other steps to assist your reasoning. The final answer should be placed within \boxed!).
Note:
1. Independent Code Blocks: Each of your code blocks is independent, can run on its own, does not depend on any previous variables, and imports any required libraries independently.
2. Multiple Code Blocks Supported: You can output multiple code blocks as needed to assist different computational steps in the reasoning process.
3. Careful Thinking: Each picce of code should be carefully considered and go beyond
simple arithmetic operations such as addition, subtraction, multiplication, or division. Each code block should include a 'print()' function to display its output.
Code Format:
python
code blocks
output Code output
Question: [question]
Figure 9: Prompt for tool-integrated reasoning in reasoning models.
A conversation between User and Assistant. The user asks question, and the Assistant solves it. Please integrate natural language reasoning with programs to solve the problem above, and put your final answer within \bboxed
Question: [question]
Figure 10: Prompt for tool-integrated reasoning in non-reasoning models.
21

---

Published as a conference paper at ICLR 2026
Question:
Find the largest possible real part of (75 1173)x+ 1446 where is complex number with
Answer:
540
Response:
Let's solve the problem step step using Python and SymPy, We need find the largest possible real of the expression
117))s 96 + 144i where 4.
Let = 4e (cose ssin
Substitute into the expression and simplify.
Find the Real part of the simplified expression. Maximize the real part with respect to e.
Let's implement this Python using SymPy.
"python
import sympy
Defore the variable thetto
theta sp sp.symbols( ('hhett" real-True)
Define the complex number
(sp.cos(theta)
sp.1 si.sin())eta)
Define the expression
expression = (75 117
xp 1) z 796 + 144 sp.1)/
Simplify the expression
simplified expression sp pp.simplifyeespressinn
Extract the real part the expression
real part rimpliiiii expression)
Find the maximam volue of the real part with respect to theta
max real part sp sp.maximum(ten real, part, theta, domain Interval(O, 2 sp.pi))
Outpur
print(max real__eaaa part)
"'output
-72/5 co2""c222aaaa4411 2604 "sin(("stannn2]]]
seams output complicated fully simplified. simplify the expression further numerical value the maximum real part.
We use SymPy's "evalf" function to get numerical approximation of the maximum real part. Let's update the code include this step.
"python
import sympy as sp
Define
theta sp symbols 'theta, real=True)
Define the complex mumber
4 (sp.cos sp.ceosthetaa) + sp.1 sp.sin(theta)
Define the expression expression (75
sp.I) z (96 sp.1)/
Simplify the expression
simplified expression sp (xxprsifyeeprrsssoon)
Extract the real part of the expression
real, part ressipiiiieed expression)
Find the maximum volue of the real part with respect to thete max real part sp maximum(real part, theta, domain=sp Interval(O, sp.pi)) Evaluate the maximmm real eval real pvrt.eval(() print(max real part eval)
part monerically
output
000000000000000000
The largest possible real part (75 117;); 96 144;
where is complex number with
540
Figure 1::
Example of tool-integrated reasoning by THOR-7B on an AIME 2024 problem, where external tools facilitate precise numerical computation.
22

---

Published as a conference paper at ICLR 2026
Question:
The parabola with equation p= retated counterrcooccs around the origin. The unique point the fourth quadrant where the original parabola and image intersect has a-coordinate (o -B-v/, where b, and positive integers, and are relatively prime. Find t-to....
Answer:
Response:
To salve this problem, we need to follow these steps: point (x.a)) on the parabola ym 60' coumterclockwise origin.
the
60" counterclockwise rotation
v3
Foeedu" bitudtr
-ath 50
So, the coordinates (e/.y)
after rotation
Since p p= 4, we substitute the rotation cquations:
x/
v
V82
2. Find the intersection: We need find the points where the original parabola y and rotated image 3'= (y=3/22/22 (x*-4) Pintersect
set y of:
vāz
2²-4
Simplify equation:
x- v3x 4--
Solve this quadratic equation using the quadraric formula == (-à 6 4a0)) /2a:
v3 1/ v 1
We are looking for the point the fourth quadrant, take the negutive
v v
Substituic
original parabola equation y- 2y
(
2/577
y-coordinate ya Here, 3, and
are relatively prime,
Let's confirm with Python code cusure accuracy.
"python
from import symbols, solve, sqrt
Define variahies
symbols(X')
Originul paraboll equation
y x=*2
8 Rotated parabola equaation
y prime- (x//t(3)*x\2)) (yy2
# Set the original and rotated equations equal to each other
equation yy-_ppeiin
# Solvee far:x
x solutionss solvei cabbeeequution,, x)
andution the fourth givadrant
x fourth quadrant - (sol.ovalff) for sol solutions if evallf) x 0][O]
Subotitate the
Into original panabola equation to find
value- subels, Fourth quadrant) evail()
Extract c from the valse
b-57
result -a+b+a b
print((x fourth quadrant, y value, a, b, result))
""ontpet
(1.31342406798590, 22..::00--22777912212252555
comesponding y-coordinate approximately- 2.2799121763337. confimms y-coordinatc indecd (3 (5-) valucs and respectively.
x-coordinane point quadrant
Thcrefore, b- + -262. The final answor is
Example of tool-integrated reasoning by THOR-7B on an AIME 2025 problem, where complex equations efficiently solved and verified with tool support.
Figure 12:
23

---

Published as a conference paper at ICLR 2026
Question:
be the greatest four-digit positive integer with the property that whenever its digits changed to resulting number divisible by 7. Let Q and R be the quotient and remainder, respectively, when N is divided by 1000. Find Q R.
Answer:
699
Response:
<think-
Okay, so need to find the greatest four-digit mimber N such that if you change any one of its digits to 1, the resulting number is divisible by 7. Then, have divide by 1000 get Q quotient) R (the remainder), down step by step.
add and together. Alright, let me bresk
More reasoming content are omittedd
To verify this, me write Python codk snippet confirm that N-5694 satisfies the conditions.
"python
#.Delise N
Chheck all four condittions
def is divisible by 7inum):
Chunge cack digit
disitrbility
def cheek condittios(NN):
# Change first dight to
= 1000 + (N 1000)
# Change second dight
mum2 = 1000 (N 1000) 100 + (N 100)
# Chhange dated digit
num3 - 1000 (N 1000) 100 100y- 10+
# Change Rourts digit
num4 N- (N 10)+1
return is divisible by_ 7(ouml) and ls_ divisible by 7tmm22) and is divisihle by ((uumn) and Is divisible by 7(mmm4)
Cloes
sutiyfir ronuitioos
check conditions(N) (N)
outpet
False
Wair, code False, which suggests that there my rensoning, This discrepancy indicates that there's mistake in the logic code. Let me debug this.
compute four numbers manually N-5694:
More reasoning content omáttedl here-
Similarly for other digits. Therefore, the code was wrong in the way it was calculating num3, Let me correct the code. Let me rewrite the code:
" "'yython
#DDlliee N
N 3694
C.hees condittions
def'ls divisible by 7(mm)):
return num
Clange wach dight and cleck divitrhillty
def check (on
# Change first digit
= N:: 1000
b- If 100) 10
numl - 1000*1 1 100% 10
8 Change wermnd digi
1000 100*1
10
8 Change dhird digit no
num3 - 1000% 100%b 10 +
# Change Sourth digit to
10"c
return divisible by_ (0__[) 1) divisible by. 70mm72) is_ divisible by 7(mmm)) and is divisiiibee by: 7(mm44)
Cheek of 5695 mitiffir the conditions
print(check Beeckkoooe comditions(N))
outpul
True
With this corrected code, the finction returns 'Truc', confirming that N-5694 indeed satis all the conditions. Therefore, our carlier conclusion was
correct, 699
was code's logic calculating numbers digits changed. Thus, final
Final Answer:
<ithink>
To solve the problem, given four-digit number N such that changing any of digits results number divisible by to find the greatest such number N. and then compute Q R. where Q the quotient and R is the remainder when N is divided by 1000. ressoning content amatred
All conditions are satisfied. Now, divide N 5694 by 1000:
[6694/1000] =5, 6894 mnd 1000 - 694. q R- 6 694 600
Minal Answer: 699
Figure 13: Example of tool-integrated reasoning by THOR-Thinkings on an AIME 2024 problem, where the use of the tool enables the self-validation of reasoning steps.
24


Reviews:
[{"review_id": "CqQpvW8Dol", "reviewer": "Reviewer_ybnx", "summary": "This paper proposes a tool-integrated reasoning framework with three components: (i) a multi-agent pipeline that builds tool-integrated reasoning (TIR) trajectories for cold-start SFT; (ii) hierarchical RL that jointly optimizes episode-level answer accuracy and step-level code-pass rewards; and (iii) an inference-time self-correction mechanism that backtracks to the last failed tool-use step.", "questions": "1. How often do failures stem from action errors versus reasoning errors? Please report an ablation: (a) action-only repair, (b) step-suffix repair, (c) full re-plan.\n2. In Table 2, how does your self-rewarded test-time selection compare with BoN/self-consistency and PRM/verifier-based reranking under the same (N), temperature, and compute?\n3. Does optimizing code-pass reward improve code-generation ability itself, or only downstream answer accuracy?\n4. Is the training data comparable to baselines (tokens, steps, RL updates)?\n5. Could you add more TIR baselines on the same backbone and training setup to isolate method effects?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The writing is clear.\n2. The method shows gains over recent TIR/CoT baselines.", "weaknesses": "1. The method assumes a failed execution implies both the action and its preceding reasoning step are wrong, and always backtracks/regenerates the suffix. This coupling can be brittle when failures arise from syntax/runtime issues; action-only repair may suffice.\n2. Optimizing only code-pass is not a promising step for fine-grained TIR reward shaping; analyses of code-trigger timing, necessity/helpfulness, and over/under-calling would be more informative.\n3. The self-correction trigger is hand-coded, whereas prior work such as ReTool [2] reports emergent self-correction behavior during training.\n4. TIR data curation and RL for tool use have seen substantial prior exploration[1,2,3], while the experiments and design here are solid, the technical novelty appears incremental relative to recent work.\n\n[1] ToRL: Scaling Tool-Integrated RL\n\n[2] ReTool: Reinforcement Learning for Strategic Tool Use in LLMs\n\n[3] ToolRL: Reward is All Tool Learning Needs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YMT6biYpyO", "reviewer": "Reviewer_5muf", "summary": "This paper tackles the poor computational and symbolic reasoning of LLMs in math problems. It introduces THOR, a framework built on three key parts. 1) TIRGen: A data pipeline described as a multi-agent \"actor-critic\" system, for generating high-quality examples of tool-integrated reasoning. 2) Hierarchical RL: A two-level optimization strategy. It combines a sparse, \"episode-level\" reward for the final answer's correctness with a denser, \"step-level\" reward for the success of intermediate code generation. 3) Self-Correction: An inference-time mechanism that lets the model backtrack and revise its reasoning path when a tool call fails.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel and Effective RL Framework: The paper's core strength lies in its hierarchical reinforcement learning strategy, which decisively tackles the sparse-reward problem in complex reasoning by introducing a denser, intermediate reward signal (code execution success), all while maintaining the focus on the primary objective (final answer correctness). This approach is both elegant and purposeful.\n2.  Strong Empirical Performance: The paper achieves state-of-the-art results on several challenging math benchmarks (Tab 1). These gains, particularly against the non-reasoning model baseline, underscore the system's high efficacy.\n3. Comprehensive Ablation Study: The ablation analysis in Table 3 systematically exposes each component's significant and additive impact—TIRGen dataset (T3 vs T1), hierarchical RL (T5 vs T3), and self-correction (T6 vs T5). This compellingly supports the paper's claims.", "weaknesses": "1.  Could the authors please clarify the choice of the term \"Actor-Critic\" for TIRGen? Since the \"Critic\" model converts reasoning steps into code, I am curious whether there are reasons to favor this RL-inspired terminology over alternatives such as \"Generator-Refiner\" or \"Proposer-Converter.\"\n\n2.   In the context of model evaluation, the Appendix indicates that the \"Critic\" is a 32B model, whereas the \"Actor\" is an 8B or 1.5B model. Could the authors comment on the importance of this size difference? Additionally, how might the quality of the TIRGen dataset ($\\mathcal{D}_{SFT}$) be affected if the Critic model were the same size as (or smaller than) the Actor model?\n\n3.  The self-correction mechanism (Section 2.4) is triggered when code execution fails. I am interested in how the framework deals with logically incorrect code that still executes successfully (e.g., producing a number from the wrong equation). Would an episode-level RL objective suffice to address this, or is an additional mechanism required?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xQuXLSQ6vg", "reviewer": "Reviewer_RCoA", "summary": "THOR is a technically competent and empirically thorough paper that explores hierarchical optimization for tool-integrated reasoning—a topic of rising importance in post-RLHF LLM training. The paper’s main contributions, especially the combination of TIR data generation (TIRGen) and dual-level GRPO optimization, are sound and moderately novel. The experiments are comprehensive, covering both reasoning and code tasks, and the observed gains are consistent.\n\nHowever, while the framework is methodically interesting, it does not radically depart from prior lines such as ToRL (Li et al., 2025b) or (Lin et al., 2025). The hierarchical decomposition is intuitively motivated but somewhat incremental relative to the existing literature, e.g., GiGPO (Feng et al., 2025) , and several details are under-specified.", "questions": "How to compute advantages at the step-level in Eq. 8? I think each sample contains a single think-act-observe loop, and the reward is computed from execution correctness. Do you mean the advantage is estimated by sampling a single response?\n\nMoreover, does this method cause efficiency issue due the correctness evaluation and regeneration?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "• Clear Problem Motivation and Scope: The paper is grounded in well-known issues of reasoning—error propagation in long CoT traces and failures in symbolic computation—and formulates these in the RL framework (Sec. 2.1). The empirical correlation between code success and final correctness offers solid motivation for dual-level credit assignment.\n\n• Hierarchical RL Formulation: The two-stage optimization (Eq. 4–9) represents a practical way to address sparse reward signals. Step-level RL on failed tool invocations is technically sensible and extends GRPO with fine-grained local feedback, offering a pragmatic, if not groundbreaking, advance.", "weaknesses": "• Data Generation Pipeline (TIRGen): The actor–critic construction (Alg. 1, Fig. 2) potentially yields a more noisy dataset than rule-based data due to the uncertainty introduced by LLMs. Althought multi-stage filtering avoids trivial tool use errors, I think human-in-the-loop verification and double-check is needed.\n\n• Reproducibility and Transparency: Appendix D only offers implementation details, including batch sizes, optimizer settings, and prompt templates. But data and code are critical to support reproducibility and checking.\n\n• Missing Analysis of Failure Modes: The paper does not report how often self-correction succeeds or fails, nor how many inference steps are affected by code errors or sandbox limitations (Sec. 2.4).\n\n• Restricted Generalization Domain: All evaluations focus on mathematical or code generation. No evidence is given that THOR generalizes to broader tool environments (e.g., multi-turn search and agentic tasks).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)