THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning

## Abstract

Abstract Large Language Models (LLMs) have made remarkable progress in mathematical reasoning, but still continue to struggle with high-precision tasks like numerical computation and formal symbolic manipulation. Integrating external tools has emerged as a promising approach to bridge this gap. Despite recent advances, existing methods struggle with three key challenges: constructing tool-integrated reasoning data, performing fine-grained optimization, and enhancing inference. To overcome these limitations, we propose THOR ( T ool-Integrated H ierarchical O ptimization via R L). First, we introduce TIRGen, a multi-agent actor-critic-based pipeline for constructing high-quality datasets of tool-integrated reasoning paths, aligning with the policy and generalizing well across diverse models. Second, to perform fine-grained hierarchical optimization, we introduce an RL strategy that jointly optimizes for both episode-level problem solving and step-level code generation. This is motivated by our key insight that the success of an intermediate tool call is a strong predictor of the final answer’s correctness . Finally, THOR incorporates a self-correction mechanism that leverages immediate tool feedback to dynamically revise erroneous reasoning paths during inference. Our approach demonstrates strong generalization across diverse models, performing effectively in both reasoning and non-reasoning models. It further achieves state-of-the-art performance for models of a similar scale on multiple mathematical benchmarks, while also delivering consistent improvements on code benchmarks. Our code will be publicly available at https://github.com/JingMog/THOR . Figure 1: An overview of our method. The left panel depicts the motivation and challenges, the middle highlights our proposed solution with the TIRGen data construction pipeline and the THOR hierarchical RL framework, and the right panel reports experimental results.

## 1 Introduction

Large Language Models (LLMs) have achieved remarkable progress, increasingly exhibiting human-like capabilities such as thinking, reflection, and self-correction. They have shown significant improvements in mathematical reasoning, code generation, and autonomous agent tasking (jaech2024openai; guo2025deepseek; yang2025qwen3; team2025kimi).

Recent tool-free methods for enhancing LLMs’ mathematical reasoning can be broadly categorized into search-based methods (besta2024graph; zhang2025lessons; hu2025prm) and reinforcement learning (RL) paradigms (yu2025dapo; yue2025vapo). Despite notable progress, these approaches remain constrained by a fundamental weakness of LLMs. As probabilistic next-token predictors, they inherently struggle with high-precision tasks (chen2022program), such as numerical computation, equation solving, symbolic manipulation (pan2025enhancing), and formal proofs (lewkowycz2022solving). In contrast, programmatic reasoning excels in these domains. Therefore, integrating the semantic reasoning capabilities of LLMs with the precise and verifiable execution of external code-based tools provides a promising pathway to overcome these limitations.

Tool-Integrated Reasoning (TIR) has emerged as a powerful paradigm for enhancing LLM reasoning by enabling them to leverage external tools to augment reasoning (gou2023tora; li2025start). Despite considerable efforts, three core challenges remain: constructing TIR data, performing fine-grained optimization, and enhancing inference. (1) For constructing TIR data, a common approach is to synthesize tool-use data by prompting external powerful models (e.g., GPT-4o) (gou2023tora). However, for reasoning models such as DeepSeek-R1, prompting alone often fails to elicit effective tool use (guo2025deepseek; li2025start). While techniques like START (li2025start) explicitly inject code prompts into the thinking process, purely rule-based approaches struggle to identify suitable insertion positions. Therefore, existing TIR data construction methods suffer from style mismatches and poor applicability to reasoning models. (2) For performing fine-grained optimization, current research primarily employs either SFT or RL. SFT-based methods, like Toolformer (schick2023toolformer) and Aimo-2 (moshkov2025aimo), require large-scale, high-quality demonstration data and often suffer from poor generalization. Existing RL methods (mai2025agent; li2025torl; feng2025retool) typically optimize at the episode-level, overlooking fine-grained updates on specific steps. Although RL is a more scalable alternative, it faces severe sparse reward problems, particularly in long reasoning chains. (3) For enhancing inference, existing methods typically interleave tool calls directly with natural language reasoning in a single pass, thereby overlooking the role of immediate tool feedback in reasoning.

To address these challenges, we propose THOR, a tool-integrated framework designed to enhance the reasoning ability of LLMs. (1) For constructing TIR data, in order to efficiently generate policy-aligned TIR data, we propose TIRGen, an actor-critic-based data construction pipeline. The actor is responsible for generating natural language reasoning steps, while the critic evaluates whether steps can be transformed into executable code and interacts with an external executor to refine the reasoning. This iterative process yields a TIR dataset that is naturally aligned with the actor’s policy and broadly applicable across diverse models and tools. (2) For performing fine-grained optimization, we are motivated by the key insight that the success of an intermediate tool call is a strong predictor of the final answer’s correctness. Our experiments later confirm this insight. Based on this, we introduce a hierarchical RL strategy that combines episode-level and step-level optimization. At the episode-level, we directly optimize for the correctness of the final answer. Concurrently, at the step-level, we apply fine-grained optimization to execution failure steps, specifically enhancing the model’s code generation ability. (3) For enhancing inference, we propose a self-correction mechanism that leverages immediate feedback from tools to dynamically revise its CoT during inference. When code invocation fails, it backtracks and explores alternative reasoning paths, thereby significantly enhancing the model’s reasoning robustness and overall performance.

We evaluate our method on diverse challenging and widely-used benchmarks, including MATH500 (hendrycks2021measuring), AIME 2024 & 2025, AMC, Minerva Math (lewkowycz2022solving), and Olympiad Bench (he2024olympiadbench). THOR establishes a new state-of-the-art (SOTA) result among models of comparable size across architectures and scales, while reducing inference overhead. It further improves performance on code generation benchmarks HumanEval, MBPP (liu2023your), and LiveCodeBench, validating the effectiveness and generalizability of our approach.

Our primary contributions are as follows:
1) Tool-Integrated Data Construction Pipeline. We introduce TIRGen, a pipeline for generating TIR data, applicable across diverse models, and better aligned with the preferences of the policy model.
2) Hierarchical Optimization. We propose a hierarchical reinforcement learning approach that combines episode-level and step-level optimization.
3) Self-correction Inference Enhancement. We introduce a self-correction mechanism that leverages immediate tool feedback to revise reasoning steps during inference.
4) Superior Performance and Broad Generalization. Our approach generalizes across reasoning and non-reasoning models, achieving competitive results on mathematical benchmarks and consistent gains on code tasks.

## 2 Methodology

### 2.1 Problem Formulation

In the context of tool-integrated reasoning, an LLM solves mathematical problems by interleaving natural language reasoning with tool invocations. Specifically, we formulate an LLM, parameterized by $\theta$, as a policy $\pi_{\theta}$. Given a problem $q$ and a corresponding instruction $I$, this policy $\pi_{\theta}$ autoregressively generates an entire interaction trajectory $\tau$, which is an alternating sequence of thoughts, actions, and observations:

$$ $\tau=(r^{1},a^{1},o^{1},...,r^{t},a^{t},o^{t},...,r^{n-1},a^{n-1},o^{n-1},r^{n}),$ (1) $$

where $r^{t}$ is a step of natural language reasoning, $a^{t}$ is an action of tool call, $o^{t}$ is the observation returned by the external execution environment after executing action $a^{t}$, and $n$ is the number of reasoning steps. This process is formulated as an iterative think-act-observe loop. The model incorporates the new observation $o^{t}$ into its context to inform the generation of the subsequent thought $r^{t+1}$ and action $a^{t+1}$. This cycle continues until the model produces the final answer within its last thought $r^{n}$, thereby concluding the trajectory. Formally, the likelihood of generating a specific trajectory $\tau$ is factorized as:

$$ $\displaystyle P_{\pi_{\theta}}(\tau\mid q,I)=P_{\pi_{\theta}}(r^{n}\mid q,I,\mathcal{H}^{1:n-1})\prod_{t=1}^{n-1}\underbrace{P_{\pi_{\theta}}(r^{t}\mid q,I,\mathcal{H}^{1:t-1})}_{\text{Thought}}\underbrace{P_{\pi_{\theta}}(a^{t}\mid r^{t},q,I,\mathcal{H}^{1:t-1})}_{\text{Action}},$ (2) $$

where $\mathcal{H}^{1:t-1}=\left\{r^{1},a^{1},o^{1},\ldots,r^{t-1},a^{t-1},o^{t-1}\right\}$ denotes the history of the previous interactions. Each term is modeled as a product of token-level probabilities.

### 2.2 TIRGen: TIR Data Generation Pipeline

Figure: Figure 2: The TIR data construction pipeline. In this pipeline, the Actor agent generates reasoning steps. The Critic agent identifies tool-executable steps and converts them into tool-augmented reasoning steps. After multi-stage filtering, we obtain the cold start dataset $\mathcal{D}_{SFT}$.
Refer to caption: x2.png

Existing methods for TIR highlight a significant need for high-quality training data. Most approaches rely on simply prompting external large models to synthesize TIR data for non-reasoning models (gou2023tora; li2025torl), but these approaches fail to generalize effectively to reasoning models such as DeepSeek-R1. Although START (li2025start) constructs long-CoT TIR data using a rule-based prompt-hint approach, the resulting trajectories often contain redundant code invocations. Therefore, existing TIR data construction methods face critical shortcomings, including style mismatches between the generated data and policy models, as well as poor scalability to reasoning models. To overcome these limitations, we introduce TIRGen, an automated TIR data synthesis pipeline based on an actor-critic framework, as illustrated in Figure [2](https://arxiv.org/html/2509.13761v2#S2.F2).

In this framework, an Actor agent generates a natural language reasoning step with a maximum length of $L_{\text{step}}$. A Critic agent then evaluates this step to identify operations that can be more efficiently solved with code, such as numerical calculations or equation solving. Upon identifying such an operation, the Critic converts it into an executable Python code snippet while preserving the Actor’s original reasoning logic. The Critic then interacts with a code interpreter to obtain a precise execution result, which replaces the original operation and produces a code-augmented reasoning step. The Actor resumes reasoning from this revised step, and the iterative cycle continues until a complete solution is reached, as described in Algorithm [1](https://arxiv.org/html/2509.13761v2#alg1). This design yields two key advantages:

- •
Policy Alignment. Because the Critic observes only isolated reasoning steps without the full problem statement and final answer, the synthesized data directly reflects the Actor’s intrinsic reasoning preferences. This ensures that the data remains in-distribution, thereby mitigating the performance degradation commonly caused by training on out-of-distribution data (gudibande2023false; chen2024self).
- •
Reduced Reliance on Large-scale Models. The Actor is responsible for high-level mathematical reasoning, while the Critic requires only basic instruction-following and code-generation skills. This division of labor decomposes the complex task into subproblems that are collaboratively solved by the two agents.

Figure: Algorithm 1 TIRGen: TIR Data Generation Pipeline

After sampling, we apply a multi-stage filtering procedure: (1) Format Consistency. We remove samples with erroneous tool calls, incorrect code or output formats, and enforce that final answers are wrapped in \boxed{}. (2) Code Quality. We discard candidates whose code either fails to execute or contains only trivial operations. Each retained example must include at least one of the following: (i) a library invocation (e.g., sympy, numpy) or (ii) a control flow (loop or branch). (3) Difficulty & Call-Round Balancing. To ensure diversity in problem complexity and tool invocation rounds, we stratify samples by the number of code calls and apply moderate down-sampling within each subset. Additionally, we remove instances solvable by a pure CoT baseline, ensuring that all retained examples require tool integration. This comprehensive procedure yields the final dataset for the cold start phase, denoted as $\mathcal{D}_{SFT}$.

### 2.3 Hierarchical RL Training Strategy

Figure: Figure 3: A hierarchical optimization framework comprising (a) episode-level RL for mathematical problem solving and (b) step-level optimization for code generation. In addition, we introduce (c) a self-correction mechanism for online error correction during inference.
Refer to caption: x3.png

#### 2.3.1 Cold Start

We initialize our model with a cold start procedure (guo2025deepseek), utilizing the dataset $\mathcal{D}_{SFT}$ generated by TIRGen. The primary objective of this stage is to equip the model with the fundamental patterns of tool invocation. This initial training is crucial, as reasoning models often struggle to invoke code effectively before a cold start. To this end, we perform supervised fine-tuning on the base model $\tilde{\pi}_{\theta}$ with the loss function as follows:

$$ $\displaystyle\mathcal{L}_{\text{SFT}}(\theta)=\mathbb{E}_{(q,I,y)\sim\mathcal{D}_{SFT}}\Big[-\sum_{t=1}^{T}\log\tilde{\pi}_{\theta}(y_{t}\mid q,I,y_{0:t-1})\Big],$ (3) $$

where $T$ is the length of response. In this way, we obtained the policy model $\pi_{\theta}$ after the cold start.

#### 2.3.2 Hierarchical Reinforcement Learning

In the hierarchical optimization phase, we employ RL to refine the model’s policy for invoking tools to solve complex problems. Conventional RL methods perform optimization at the episode-level, relying on a sparse reward signal derived solely from the correctness of the final outcome (mai2025agent; li2025torl; feng2025retool). Our empirical analysis reveals a crucial insight: the success of an intermediate tool call is a strong predictor of the final answer’s correctness. Motivated by this finding, we propose a hierarchical reinforcement learning framework that combines coarse-grained, episode-level optimization to enhance the model’s problem-solving ability with fine-grained, step-level optimization to improve code generation for individual tool calls, as shown in Figure [3](https://arxiv.org/html/2509.13761v2#S2.F3). The observation is validated in Appendix LABEL:appendix:statistical_validation.

Episode-level Optimization.
At the episode-level optimization, we adopt GRPO (shao2024deepseekmath) and use the correctness of the final answer as the reward to enhance the tool invocation strategy of the model, as shown in Figure [3](https://arxiv.org/html/2509.13761v2#S2.F3)(a). The complete episode-level objective is:

$$ $\begin{split}\mathcal{L}^{\text{epis}}_{\pi_{\theta}}(\theta)=&\mathbb{E}[q\sim\mathcal{D}_{RL},\{s_{i}\}_{i=1}^{G}\sim\pi_{\theta}(S|q)]\frac{1}{G}\sum_{i=1}^{G}\Big(\frac{1}{\sum_{t=1}^{|s_{i}|}I(s_{i,t})}\sum_{t:I(s_{i,t})=1}^{|s_{i}|}\\ &\min\big(\frac{\pi_{\theta}(s_{i}|q)}{\pi_{{\theta}_{\text{old}}}(s_{i}|q)}A_{i},\text{clip}(\frac{\pi_{\theta}(s_{i}|q)}{\pi_{{\theta}_{\text{old}}}(s_{i}|q)},1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}})A_{i}\big)\Big)+\alpha\mathcal{L}_{\text{NLL}}(\theta),\end{split}$ (4) $$

where $G$ denotes the group size, and $A_{i}$ is the advantage of response $s_{i}$, defined as $A_{i}=\frac{\mathcal{R}_{i}-\text{mean}(\mathcal{R})}{\text{std}(\mathcal{R})}$. The reward is given by $\mathcal{R}_{i}=1$ if answer is correct, and 0 otherwise. The indicator function $I(s_{i,t})=1$ indicates that the token $s_{i,t}$ is generated by $\pi_{\theta}$ rather than observed from an external executor. Additionally, to better exploit successful trajectories, we incorporate a language modeling loss $\mathcal{L}_{\text{NLL}}$, weighted by a coefficient $\alpha$, applied to examples with positive advantage $\mathcal{T}_{\text{pos}}$, directly reinforcing the likelihood of correct samples during RL training (yue2025vapo).

$$ $\displaystyle\mathcal{L}_{\text{NLL}}(\theta)=-\frac{1}{\sum_{s_{i}\in\mathcal{T_{\text{pos}}}}|s_{i}|}\sum_{s_{i}\in\mathcal{T}_{\text{pos}}}\log\pi_{\theta}(s_{i}|q).$ (5) $$

To stabilize training, we filter out trajectories with failed code executions during episode-level optimization to avoid inappropriate penalties. Since execution failures may arise not only from model-generated errors but also from environment issues or sandbox limitations.

Step-level Optimization.
After sampling full trajectories, we perform step-level optimization to correct code errors using execution feedback, as illustrated in Figure [3](https://arxiv.org/html/2509.13761v2#S2.F3)(b). This stage focuses specifically on reasoning steps that resulted in failed actions $\mathcal{A}_{\text{err}}$. First, we construct a step-level optimization dataset $\mathcal{D}_{step}$. For each failed step, we treat it as a query and generate group rollouts, where the execution correctness of the generated code provides the reward signal. Consequently, it is crucial to ensure that the rollouts within each group cover diverse execution outcomes. Next, we describe the construction of $\mathcal{D}_{step}$ and its corresponding optimization method.

Within a think-act-observe tuple $(r^{t},a^{t},o^{t})$, a failed execution inherently indicates an error exists in the action $a^{t}$. Moreover, since $a^{t}$ is conditioned on $r^{t}$, fixing $r^{t}$ limits the diversity of generated actions. To overcome this problem, we implement a backtracking procedure. We partition the erroneous thought $r^{t}$ into a prefix $r^{t}_{\text{pre}}$ and a suffix $r^{t}_{\text{suf}}$ of length $L_{\text{suf}}$. Subsequently, we condition the model on the history up to $r^{t}_{\text{pre}}$ and regenerate a new reasoning suffix and action. This procedure produces the dataset $\mathcal{D}_{step}$ for fine-grained step-level optimization:

$$ $\displaystyle\mathcal{D}_{step}=\{\operatorname{pref}(\tau,t)\mid a^{t}\in\mathcal{A}_{\text{err}},\tau\in\mathcal{T}\},$ (6) $\displaystyle\operatorname{pref}(\tau,t)=(q,r^{1},a^{1},o^{1},...,r^{t-1},a^{t-1},o^{t-1},r^{t}_{\text{pre}}),r^{t}=r^{t}_{\text{pre}}\oplus r^{t}_{\text{suf}},$ (7) $$

This formulation defines a fine-grained code generation task: given a reasoning prefix $\operatorname{pref}(\tau,t)$, the model must correctly regenerate the subsequent reasoning suffix $\hat{r}^{t}_{\text{suf}}$ and action $\hat{a}^{t}$. At this stage, each sample contains a single think-act-observe loop, and the reward is computed from execution correctness. We optimize the policy with the following step-level loss:

$$ $\begin{split}\mathcal{L}^{\text{step}}_{\pi_{\theta}}(\theta)=\mathbb{E}[&p\sim\mathcal{D}_{step},\{s_{i}^{\prime}\}_{i=1}^{G}\sim\pi_{\theta}(S|p)]\frac{1}{G}\sum_{i=1}^{G}\Big(\frac{1}{\sum_{t=1}^{|s^{\prime}_{i}|}I(s^{\prime}_{i,t})}\sum_{t:I(s^{\prime}_{i,t})=1}^{|s^{\prime}_{i}|}\\ &\min\big(\frac{\pi_{\theta}(s_{i}^{\prime}|p)}{\pi_{{\theta}_{\text{old}}}(s_{i}^{\prime}|p)}A_{i}^{\prime},\text{clip}(\frac{\pi_{\theta}(s_{i}^{\prime}|p)}{\pi_{{\theta}_{\text{old}}}(s_{i}^{\prime}|p)},1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}})A_{i}^{\prime}\big)\Big)+\alpha\mathcal{L}_{\text{NLL}}(\theta).\end{split}$ (8) $$

where $s^{\prime}_{i}$ and $A^{\prime}_{i}$ denote the responses and advantages at the step-level.

The final training objective combines both episode-level and step-level losses:

$$ $\displaystyle\mathcal{L}(\theta)=\mathcal{L}^{\text{epis}}_{\pi_{\theta}}(\theta)+\mathcal{L}^{\text{step}}_{\pi_{\theta}}(\theta).$ (9) $$

### 2.4 Self-Correction During Inference

During inference, our model follows the think-act-observe loop. To exploit immediate feedback from tool execution, we introduce a self-correction mechanism that dynamically corrects erroneous reasoning steps, as shown in Figure [3](https://arxiv.org/html/2509.13761v2#S2.F3)(c). Specifically, when an action $a^{t}$ fails to execute, it indicates that both the action $a^{t}$ and its associated reasoning step $r^{t}$ are likely incorrect. To explore alternative solving paths, the model backtracks to $r^{t}$ and partitions it into a prefix $r^{t}_{\text{pre}}$ and a suffix $r^{t}_{\text{suf}}$, as previously described. Conditioned on the history up to $r^{t}_{\text{pre}}$, the model then regenerates a new reasoning suffix $\hat{r}^{t}_{\text{suf}}$ and a revised action $\hat{a}^{t}$, thereby enabling online error correction during inference. The correction procedure can be repeated for up to $N_{\text{corr}}$ attempts. Importantly, each attempt only requires regenerating the suffix $\hat{r}^{t}_{\text{suf}}$ and the corresponding action $\hat{a}^{t}$, rather than recomputing the entire trajectory. Thus, the additional computational cost is minimal compared to the total cost.

## 3 Experiments

Table: Table 1: Comparison with state-of-the-art methods on mathematical benchmarks, the best results are in bold and the second-best are underlined. Code use indicates whether code tools are employed. † denotes our reimplementation results of Avg@4, ‡ indicates results from their official releases.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. How often do failures stem from action errors versus reasoning errors? Please report an ablation: (a) action-only repair, (b) step-suffix repair, (c) full re-plan.
2. In Table 2, how does your self-rewarded test-time selection compare with BoN/self-consistency and PRM/verifier-based reranking under the same (N), temperature, and compute?
3. Does optimizing code-pass reward improve code-generation ability itself, or only downstream answer accuracy?
4. Is the training data comparable to baselines (tokens, steps, RL updates)?
5. Could you add more TIR baselines on the same backbone and training setup to isolate method effects?
soundness: 2
strengths: 1. The writing is clear.
2. The method shows gains over recent TIR/CoT baselines.
summary: This paper proposes a tool-integrated reasoning framework with three components: (i) a multi-agent pipeline that builds tool-integrated reasoning (TIR) trajectories for cold-start SFT; (ii) hierarchical RL that jointly optimizes episode-level answer accuracy and step-level code-pass rewards; and (iii) an inference-time self-correction mechanism that backtracks to the last failed tool-use step.
weaknesses: 1. The method assumes a failed execution implies both the action and its preceding reasoning step are wrong, and always backtracks/regenerates the suffix. This coupling can be brittle when failures arise from syntax/runtime issues; action-only repair may suffice.
2. Optimizing only code-pass is not a promising step for fine-grained TIR reward shaping; analyses of code-trigger timing, necessity/helpfulness, and over/under-calling would be more informative.
3. The self-correction trigger is hand-coded, whereas prior work such as ReTool [2] reports emergent self-correction behavior during training.
4. TIR data curation and RL for tool use have seen substantial prior exploration[1,2,3], while the experiments and design here are solid, the technical novelty appears incremental relative to recent work.

[1] ToRL: Scaling Tool-Integrated RL

[2] ReTool: Reinforcement Learning for Strategic Tool Use in LLMs

[3] ToolRL: Reward is All Tool Learning Needs

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 2
strengths: 1. Novel and Effective RL Framework: The paper's core strength lies in its hierarchical reinforcement learning strategy, which decisively tackles the sparse-reward problem in complex reasoning by introducing a denser, intermediate reward signal (code execution success), all while maintaining the focus on the primary objective (final answer correctness). This approach is both elegant and purposeful.
2.  Strong Empirical Performance: The paper achieves state-of-the-art results on several challenging math benchmarks (Tab 1). These gains, particularly against the non-reasoning model baseline, underscore the system's high efficacy.
3. Comprehensive Ablation Study: The ablation analysis in Table 3 systematically exposes each component's significant and additive impact—TIRGen dataset (T3 vs T1), hierarchical RL (T5 vs T3), and self-correction (T6 vs T5). This compellingly supports the paper's claims.
summary: This paper tackles the poor computational and symbolic reasoning of LLMs in math problems. It introduces THOR, a framework built on three key parts. 1) TIRGen: A data pipeline described as a multi-agent "actor-critic" system, for generating high-quality examples of tool-integrated reasoning. 2) Hierarchical RL: A two-level optimization strategy. It combines a sparse, "episode-level" reward for the final answer's correctness with a denser, "step-level" reward for the success of intermediate code generation. 3) Self-Correction: An inference-time mechanism that lets the model backtrack and revise its reasoning path when a tool call fails.
weaknesses: 1.  Could the authors please clarify the choice of the term "Actor-Critic" for TIRGen? Since the "Critic" model converts reasoning steps into code, I am curious whether there are reasons to favor this RL-inspired terminology over alternatives such as "Generator-Refiner" or "Proposer-Converter."

2.   In the context of model evaluation, the Appendix indicates that the "Critic" is a 32B model, whereas the "Actor" is an 8B or 1.5B model. Could the authors comment on the importance of this size difference? Additionally, how might the quality of the TIRGen dataset ($\mathcal{D}_{SFT}$) be affected if the Critic model were the same size as (or smaller than) the Actor model?

3.  The self-correction mechanism (Section 2.4) is triggered when code execution fails. I am interested in how the framework deals with logically incorrect code that still executes successfully (e.g., producing a number from the wrong equation). Would an episode-level RL objective suffice to address this, or is an additional mechanism required?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: How to compute advantages at the step-level in Eq. 8? I think each sample contains a single think-act-observe loop, and the reward is computed from execution correctness. Do you mean the advantage is estimated by sampling a single response?

Moreover, does this method cause efficiency issue due the correctness evaluation and regeneration?
soundness: 3
strengths: • Clear Problem Motivation and Scope: The paper is grounded in well-known issues of reasoning—error propagation in long CoT traces and failures in symbolic computation—and formulates these in the RL framework (Sec. 2.1). The empirical correlation between code success and final correctness offers solid motivation for dual-level credit assignment.

• Hierarchical RL Formulation: The two-stage optimization (Eq. 4–9) represents a practical way to address sparse reward signals. Step-level RL on failed tool invocations is technically sensible and extends GRPO with fine-grained local feedback, offering a pragmatic, if not groundbreaking, advance.
summary: THOR is a technically competent and empirically thorough paper that explores hierarchical optimization for tool-integrated reasoning—a topic of rising importance in post-RLHF LLM training. The paper’s main contributions, especially the combination of TIR data generation (TIRGen) and dual-level GRPO optimization, are sound and moderately novel. The experiments are comprehensive, covering both reasoning and code tasks, and the observed gains are consistent.

However, while the framework is methodically interesting, it does not radically depart from prior lines such as ToRL (Li et al., 2025b) or (Lin et al., 2025). The hierarchical decomposition is intuitively motivated but somewhat incremental relative to the existing literature, e.g., GiGPO (Feng et al., 2025) , and several details are under-specified.
weaknesses: • Data Generation Pipeline (TIRGen): The actor–critic construction (Alg. 1, Fig. 2) potentially yields a more noisy dataset than rule-based data due to the uncertainty introduced by LLMs. Althought multi-stage filtering avoids trivial tool use errors, I think human-in-the-loop verification and double-check is needed.

• Reproducibility and Transparency: Appendix D only offers implementation details, including batch sizes, optimizer settings, and prompt templates. But data and code are critical to support reproducibility and checking.

• Missing Analysis of Failure Modes: The paper does not report how often self-correction succeeds or fails, nor how many inference steps are affected by code errors or sandbox limitations (Sec. 2.4).

• Restricted Generalization Domain: All evaluations focus on mathematical or code generation. No evidence is given that THOR generalizes to broader tool environments (e.g., multi-turn search and agentic tasks).