Teaching Language Models to Reason with Tools
Authors: Chengpeng Li
Zhengyang Tang
Ziniu Li
Mingfeng Xue
Keqin Bao
Tian Ding
Ruoyu Sun
Benyou Wang
Xiang Wang
Junyang Lin
Dayiheng Liu
Venue: NeurIPS 2025 poster ()
Keywords: Tool-integrated Reasoning
Large Reasoning Model
Long Chain-of-Thought
Primary area: applications
TL;DR: This paper introduce CoRT, a post-training framework for teaching large reasoning LLMs to leverage CI effectively and efficiently.

Abstract: Large reasoning models (LRMs) like OpenAI-o1 have shown impressive capabilities in natural language reasoning. However, these models frequently demonstrate inefficiencies or inaccuracies when tackling complex mathematical operations. While integrating computational tools such as Code Interpreters (CIs) offers a promising solution, it introduces a critical challenge: a conflict between the model's internal, probabilistic reasoning and the external, deterministic knowledge provided by the CI, which often leads models to unproductive deliberation. To overcome this, we introduce CoRT (Code-Optimized Reasoning Training), a  post-training framework designed to teach LRMs to effectively utilize CIs. We propose **Hint-Engineering**, a new data synthesis strategy that strategically injects diverse hints at optimal points within reasoning paths. This approach generates high-quality, code-integrated reasoning data specifically tailored to optimize LRM-CI interaction. Using this method, we have synthesized 30 high-quality samples to post-train models ranging from 1.5B to 32B parameters through supervised fine-tuning. CoRT further refines the multi-round interleaving of external CI usage and internal thinking by employing rejection sampling and reinforcement learning. Our experimental evaluations demonstrate CoRT's effectiveness, yielding absolute improvements of 4\% and 8\% on DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Qwen-1.5B, respectively, across five challenging mathematical reasoning datasets. Moreover, CoRT significantly enhances efficiency, reducing token usage by approximately 30\% for the 32B model and 50\% for the 1.5B model compared to pure natural language reasoning baselines. The models and code are available at: [this url](https://github.com/ChengpengLi1003/CoRT).

Teaching Language Models to Reason with Tools
Chengpeng Li Zhengyang Tang*2:. 223, Ziniu Li*3A, Mingfeng Xue², Keqin Bao!? Tian Ding', Ruoyu Sun Benyou Wang', Xiang Wang', Junyang Lin", and Dayiheng Liu
University of Science and Technology of China "Qwen Team, Alibaba Inc.
The Chinese University of Hong Kong, Shenzhen *Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen Research Institute of Big Data
Abstract
Large reasoning models (LRMs) like OpenAl-ol have shown impressive capa- bilities in natural language reasoning. However, these models frequently demon- strate inefficiencies inaccuracies when tackling complex mathematical opera- tions. While integrating computational tools such as Code Interpreters (CIs) offers a promising solution, introduces a critical challenge: a conflict between the model's 's internal, probabilistic reasoning and the extemal, deterministic knowl- edge provided by the CI, which often leads models to unproductive deliberation. To overcome this, we introduce CoRT (Code- Sode-Oppimizee Reasoning Training), a post-training framework designed to teach LRMs effectively utilize Cls. We propose Hint-Ennineet Siiineiiinne new data synthesis strategy that strategically injects diverse hints at optimal points within reasoning paths. This approach generates high-quality, code-integrated reasoning data specifically tailored optimize LRM- interaction. Using this method, we have synthesized 30 high-quality samples to post-train models ranging from 1.5B to 32B parameters through supervised fine-tuning. CoRT further refines the multi-round interleaving of external CI usage and internal thinking by employing rejection sampling and reinforcement learn- ing. Our experimental evaluations demonstrate CoRT's effectivvnesss, yielding absolute improvements of 4% and 8% DeepSeek-RR - I-Distill -atiiill-wee-22BB and -iinseee--R--issil---------
1.5B, respectively, across five challenging mathemati- cal reasoning datasets. Moreover, CoRT significantly enhances efficiency, reducing token usage by approximately 30% for the 32B model and 50% for the 1.5B model compared to pure natural language reasoning baselines. The models and code are available at: https:// :/gi github.... com (ChengenngLi1003..
Introduction
Benefiting from advancements in reinforcement learning (RL) techniques [1-4], Large Reasoning Models (LRMs) such as OpenAl-ol [5], Qwen-3 [6], and DeepSeck- R1 [7] have achieved break- through progress in complex reasoning tasks [8, 9]. These models exhibit numerous human-liie like cognitive strategies with long chain of thought (CoT) [10, 11] reasoning, including self-refinement, self-reflection, and multi-strategy exploration. However, LRMs still demonstrate limitations in accu- racy and efficiency when handling complex mathematical operations, such precise computation
*: Equal contribution. Email: chengpengli imaall. astc.edu zhengyangtangol ink, cuhk. edu. iimiull@link cuhk. edu, Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

AIME24 (Passe1)
Best performance/ avg tokens
Hint Enginneinnneeeeeee RFT-328
Premega Min---inSS SFT-33B STIL-3 TOOL-22
Hinn-Enginearing SFT-328
Seep 11-328
RaTooll 1-328
72%
START-328
avg takens
50%
7K
8K
10K
14k
Figure I: Performance vs. token efficiency on AIME24. The x-axis represents average token usage while the y-axis shows Pass accuracy. Siint-Engineerin-RRFTA achieves comparable accuracy to other frontier models while using significantly fewer tokens.
and complex equation solving [12, 13], which are better suited for code interpreters (CIs). Leveraging CIs, LRMs like o3 and 04-mini [14] have substantially enhanced their mathematical reasoning capa- bilities. However, these models are proprictary, the methods for achieving this effective LRM-CI integration remain undisclosed.
A key open challenge is teaching LRMs when and how to effectively and efficiently use CIs to generate structured reasoning. This is a scientifically new problem because, unlike pure natural language reasoning, CIs introduce external deterministic knowledge that exists beyond the model's internal representations. This raises critical questions: (1) How can we synthesize high-quality training data when models like o3 and 04-mini do not expose their detailed reasoning traces? (2) How can the model effectively coordinate a CI's computational precision with its abstract CoT reasoning capabilities? (3) How can the self-reflection mechanisms inherent to LRMs be reconciled with the exact external knowledge provided by CIs?
This paper explores to answer the above questions. We begin by tackling the data synthesis chal- lenge, which forms the foundation for post-training via supervised fine-tuning (SFT) [15]. rejection fine-tuning (RFT) [16] and RL [7]. Based on the open-source LRM DeepSeek-R1, we investi- gate direct prompting methods like [17] for CI integration. Our key discovery that inserting simple hint-"Okay, let's try to solve this problem step by step using multiple python code inmeeii-iyyyyaaellll after the model's thinking token <think> improves code triggering rates from 50% to 90% (on 100 problems from [17]). We term this straightforward approach the prompt-hil hins method. This confirms that LRMs possess the latent capability to leverage CIs for reasoning despite being primarily trained natural language. However,
also find that they struggle with efficient tool utilization. The two most prominent inefficiencies are delayed code computation (preferring text reasoning before utilizing CI) and code result distrust (unnecessarily verifying CI outputs manually). To address these limitations, we have designed a more sophisticated approach we call hinr- engineering. The key idea is to strategically insert different hints at appropriate positions throughout the reasoning process. These hints are specifically designed to teach the LRM to understand the outputs of Cls, mitigating meaningless reflection behaviors. A notable feature of this approach is that the resulting reasoning traces become significantly shorter and more efficient.
Following the principle that data quality outweighs quantity (less is more) [18-20], manually generate 30 high-quality samples with human verification. Using these samples, we post-train models of varying sizes based on available computational resources. For large 32B parameter models, we conduct SFT and RFT, while RL remains computationally infeasible within our infrastructure. How- ever, successfully implement the complete SFT-RFT-RI pipeline for smaller models. Moreover, we carefully design outcome rewards to encourage correct code generation behaviors.
Our experiments confirm the effectiveness of the above approaches. Results across five challenging mathematical reasoning datasets demonstrate that Hint-Engineering models achieve significant improvements: 4% absolute accuracy gain for DeepScck-RI -il--iilll--wen------- and 8% for DeepSeek-

---

Base Models
Cold Start
Stage 1
Stage 2 Rejection Fine-tuning
Hit-Engineering FF 1-328
Frontier Models
Lightwelght Models
Strong-toWWeak Distilation
Stage
Hint-Engineering
Base Models
code-integrated Rt.
RL-1 .588
Figure 2: The training framework of CoRT.
R1-Distill- Pessill-Qwen-l.e 1.5B.. Moreover, the most challenging AIME benchmarks, approach reduces token consumption by 30% for the 32B model and 50% for the 1.5B model.
To summarize, our key contributions include:
A new data synthesis framework specifically engineered for code-integrated reasoning that effec- tively addresses the critical data scarcity challenge in this emerging domain.
An efficient and scalable training pipeline that enables LLMs to acquire sophisticated code- integrated reasoning capabilities through targeted post-training procedures.
Comprehensive empirical evaluations demonstrating significant performance and token efficiency improvements across 5 challenging mathematical benchmarks. We further validate the generaliz- ability of our approach with an out-of-distribution evaluation on chemistry problems (Appendix I).
Methodology
This section introduces the CoRT framework (Figure 2), which involves three key stages: modeling code-integrated reasoning, training 32B models (Cold Start, SFT, RFT), and developing 1.5B models through strong-to- toennkk distillation and RL.
2.1 Task Formulation
By leveraging executable programs, LRMs can now perform precise calculations and complex logical operations. The framework comprises three essential components: a problem input P, a language model π, and an executor environment E. During the reasoning process, the system constructs sequence Tr at time step t, which can be represented as:
T: (11111 o.,
oz)3.
(I)
Here, n, represents the textual reasoning step, p; denotes the program snippet generated by the model, 0; indicates the execution output, with indexing the sequential interactions between the language model and the execution environment. The sequential reasoning process follows these steps:
(1,, Pt) TPP @ Tr-1), {p:),
T = T--1 e n, + pt + ot.
This iterative mechanism establishes dynamic feedback loop, where each reasoning step is informed by previous computational results. The process continues until the model reaches a definitive answer.
2.2
Cold Start Methods
2.2.1
Prompt-hint
This section explores straightforward approach, referred to as prompt-hint, for constructing cold-start dataset for teaching LRMs. Specifically, initiate our data generation process, we carefully crafted prompt (detailed in Appendix E) designed to instruct R1 [7] to leverage both natural language reasoning and interactive Python code execution during inference. We integrated code interpreter that enables R1 to perform real-time interactive reasoning, as outlined in Section 1. To encourage generating reasoning trajectories that incorporate code, inspired by [21], we enhanced the generation process by introducing a strategic hint-"Okay, let's try to solve this problem step by step using multiple python code calls" "alls"=fff -ollowwinn the model's initial thinking token <think>. An example shown in Figure 3 (a).

---

Each Bice two nescomgrant purallelepipec's is chombus whose disgonals have lengths v2lund v31. The ratio the volume of the lorger of polyhedra volume the m/in, Sms and relatively prime positive integers, wo
<think>
<think>
Okey, let's try salve this problem
Okay, let's solve this prablem stop
step by step using multiple
using multiple python code calls.
pythen calls.
[skip
Let's compute V1 and V2 with Python for acturacy.
Without Hint-1
[skip 8524 tokens]
Let's compute V1 and for these two cases.
Thus,
import sympy sp
Hint-1: 1:
Complex calculations
[skip 5057 takens]
Now, to confirm using Python.
VI= 63sqrt(2/22222 31sqrt(2)
60~s2rt222 Valume 31vsgri22
import numpy
Without Hint-2
We don't
doubt accuracy of python
125
calculations.
[skip tokens]
Hint-2:
These calculation...
Wait, check
So m=63, n=62, m-n=125. </think>
</thinko...
Confirm python calculation
[kkpp 2215 tokens] </thinko...
answeris is \boved[125).
answer \1oxed(125).
The sum n 63 62 (2oked(125).
(+) Prompt-hint
(b) Hint-engineering
Figure 3: Comparison between prompt-hint and hint-engineering approaches using Problem 13 from AIME23 I case study (prompt prefix omitted for brevity).Boll methods begin with a general hint (in green) after <think> to encourage code usage. While prompt-hint (a) allows natural interaction between RI and the Code Interpreter (CI), leading to inefficient token usage, bint-engineering (b) introduces strategic hints at key decision points. Hint-1 is inserted when the model begins manual calculation of complex volumes (VI and V2), redirecting Python computation. Hint-2 is added to prevent unnecessary verification of Python calculations. Through these targeted interventions, hint-engineering achieves approximately 5000 tokens reduction while maintaining solution accuracy. More examples are provided in Appendix F.
Leveraging the publicly available STILL3 [17] dataset comprising 820 math problems and R1, we employed our prompt-hint annotation method generate 800 training instances, denoted as D Prompt-Aint. We then performed SFT with diversity preserving [15] on Soapscck-R--Distll---eeal 32B using this dataset, resulting in our Prompt-Hint-SFT-322 model.
2.2.2 Hint-engineering
Despite the simplicity of the prompt-hint approach, where LRMs autonomously decide when and how to utilize the Code Interpreter (CI), we find it does not fundamentally address the challenge of interleaving internal probabilistic reasoning with external deterministic knowledge. Specifically, we identified several inefficiencies and instances of overthinking. These limitations can be categorized into two main issues:
Delayed code computation: When handling complex mathematical operations, models tend to first engage in text-based reasoning before writing code and using CI for verification. This pattern often results in redundant computational steps.
Code result distrust: Upon receiving CI execution results, models frequently display lack of trust in the output, leading to unnecessary manual verification and redundant calculations.
These behavioral patterns significantly impact the model's 's reasoning efficiency, particularly in terms of the number of tokens required for problem-solving.
To address these inefficiencies, we implement a targeted approach named linttengineeiing. When delayed computation is detected-sspeifffclllyy, at the point where the model begins to manually calculate complex mathematical operations we insert a strategic hint, such as, "It looks tedious, and we can use python code to simplify the reasoning". followed immediately by Python code trigger < pyythnn. Similarly, when code result distrust emerges, we introduce hint such as "We don't 't need to doubt the accuracy of python calculations". This intervention redirects the model's focus back to core problem rather than engaging unnecessary verification computational accuracy. is important to note that while we discourage the verification of Python's 's numerical calculations, we maintain the model's ''s behavior to verify the logical correctness of the code structure. Figure 3 (b) illustrates concrete example. We formulate this process in Appendix B.

---

A critical challenge in our approach was identifying suitable positions for hint insertion. While we initially attempted to automate this process using DeepSeek-V3 and R1, we found the results to be insufficiently precise. Hence, we opted for manual hint insertion on 30 problems from AIME (pre-2024) to create the onginernngn SPT dataset and train the Siitt-nginneerin-SHH -FF--322 model.
To further enhance model performance, we conducted rejection fine-tuning(RFT) using the Hint- Enginecring-SFFF SFFT332B model on the 820 problems from STILL3. Specifically, we performed multiple sampling iterations on each problem and implemented filtering process to eliminate trajectories with incorrect final answers, as well as those exhibiting delayed code computation or code re- sult distrust behaviors. We combined the filtered trajectories with Dmint-ence -iiiit-eniinerinn--a to create Siiatt-eniineriing-ae a dataset of 830 examples. This curated dataset was then used fine-tune DeepSeck-RI-DDe Sel-Distill--we----- resulting in our Hint- Sint--nninneringg -ateerrinnn----222B RFT-32B model.
2.3 Strong-to-weal Distillation
We aim use RL refine the multi-round interleaving of CI usage and internal thinking, as RL allows models to interact directly with the CI. However, computational constraints made infeasible to perform RL on 32B-parameter models. We therefore opted for strong-to-weak distillation approach. We distilled both the Hommp--iint-FF--322 and Hint-Enginnering-RTTT -Frrrnnn--RF-332 models into the DeepSeek-R1- --ieee-----iiss-eeeeee 1.5B- architecture. This process yielded smaller models, Prompt-Hint-I.5Be
and Sinn--nniieeringSSFTR -FF.-.... which served as the foundation for our RL experiments. We selected and preprocessed 10k examples from publicly available datasets for this distillation, with detailed procedures provided in Appendix G.
2.4 Code-integrated Reinforcement Learning
To further refine the models' code-integrated reasoning. we apply RL to the 1.5B models. Following recent trends value-fre frre methods [2]. we chose the GRPO algorithm [3], as it is well-suited for scenarios with sufficient rollouts. In applying the GRPO algorithm to models, introduced several modifications to the standard text-based GRPO framework:
Rollout with Code Interpreter: We enable multiple model-CI interactions during the RL rollout process, as described in Section 2.1. To manage computational overhead during rollouts, we implement maximum tool usage limit T', Once this limit reached, we append hint informing the model to proceed without further Python usage.
Persistent Execution Environment: Unlike traditional TIR environments that execute each Python block independently, we construct Jupyter-like environment where variables, environments, and functions persist across code blocks, enhancing code efficiency and reducing errors.
To ensure training stability, we implement execution result masking, significantly reducing model collapse probability during training. This crucial modification prevents potential training failures that would otherwise occur without such masking.
Output Masking:
Reward Design: We implement a dual reward system comprising accuracy reward and code execution reward as defined in Equation (3). For accuracy assessment, we require models to present final answers in a specified format (c.g., within boxed 1). enabling reliable rule-based verification against ground truth answers. To prevent infinite loops resulting from repeated code failures, implement a code execution penalty for responses where all code execution attempts fail. The weighted sum of these two components, where controls the
total reward R is computed contribution of the code execution penalty.
if answers match otherwise
{
-1
if all codes fail otherwise
R₂
R. Ro=1
R R2 + w R.
(3)
3 Experiments
In this section, we evaluate the effectiveness of proposed CoRT framework through comprehensive experiments on five challenging mathematical reasoning benchmarks: (1) AIME24, (2) AIME25.(3) AMC23, (4) MATH500, and (5) OlympiadBench. Comprehensive deseriptions of the evaluation datasets are provided in Appendix C. For space saving, our implementation details of SFT, RFT, RL and inference are listed in Appendix A. Due space constraints, we present only representative experimental results in the main text, with comprehensive results available in Appendix D. Moreover, the baseline models are described in Appendix H.
5

---

3.1 Main Results
Table Performance comparison of different math reasoning models across benchmarks. For each section, best results are shown in bold and second- best results are underlined. During inference, we set temperature 0.6 and topp 0.95. Results for AIME24, AIME25, and AMC23 are averaged over 16 samples, while MATH500 and Olympiad results are averaged over 4 samples. All experiments use maximum sequence length of 32.768 tokens and limit tool usage to 15 calls.
Model
Tool-Use
Stage
AIME24
AIME25
AMC23
MATT1500
Olympiad
Avg
SOTA Models
ol
RL
74.3
79.2
96.4
03
RL
95.2
98.7
04-mini
RL
98.4
99.5
DeepSeck-RI
RL
79.8
70.0
97.3
QwQ-32B
RI.
79.5
65.3
94.3
92.3
79.7
82.2
Frontier Models(328)
DeepSeck-R1-33
SFT
72.9
59.0
88.8
94.3
72.5
77.5
START-32B
SFT
66.7
STILL-3-TOOL-32B
SFT
76.7
64.4
91.3
96.6
75.9
81.0
DeThol-R1-32F
RT.
72.5
54.3
92.9
94.3
69.2
76.6
Commpt-Hint-SFT--33
SFT
77.3
65.0
95.0
96.6
75.1
81.8
Himt-- Somt-EEngineering-STT...
SFT
72.1
60.2
91.3
94.4
71.2
77.8
Biit-Enginnering-RRe RFF--33
RFT
76.7
94.4
95.1
73.4
81.3
Lightweight Models (1.58)
DeepSeck-R1-1.55
SFT
28.8
21.8
62.9
83.9
43.3
48.1
DeepScaleR-1.5B-Peee
RL
40.0
30.0
73.6
87.8
50.0
56.3
ToRL-1.5B
RL
26.7
26.7
67.5
77.8
48.5
Prompt-Hint- Soopp--Hint---B--- -55-SFT
SFT
30.6
25.0
63.1
83.3
50.4
50.5
Prumpp--fnttt1.5 Monpp--Iintt1.SB-BL
RL
43.1
30.2
73.8
87.3
57.1
58.3
Hint- Somn--nnineeringg- 1. 1.5
SFT
34.0
23.5
64.6
84.2
49.8
51.2
SiittEnginnerrig-15555 1.5B-RL
RL
41.0
29.4
70.0
85.8
55.6
56.4
Table presents our main results, comparing our models with state-of-the-art baselines across multiple mathematical reasoning benchmarks. We organize the results into three sections: SOTA Models, Frontier Models (32B), and Lightweight Models (1.5B).
For 32B models, observe that after SFT, our models achieve performance comparable existing tool-integrated models, with Prompt-Hint-SFT-322 slightly outperforming others with an average accuracy of 81.8% across benchmarks. Notably, Engineenginerinng- RFT-32B, despite being trained on just 30 manually annotated examples initially, achieves competitive performance with an average accuracy of 8.33%. This highlights the effectiveness of our rejection fine-tuning approach and the importance of high-quality data over quantity.
Enginneerng-RRF--22...
For 1.5B models, the reinforcement learning stage brings substantial improvements. Prompt- -int- 1.5--L achieves state-of oftte-of-theart performance among lightweight models with an average accuracy of 58.3%, outperforming the non-tool-using DeepScaleR-I.5B--Pie
Similarly, Hint-Engineering- 1.5B-RL shows strong performance at 56.4%. The dramatic improvement from SFT to RL stages (approximately 8% absolute gain) demonstrates the effectiveness of reinforcement learning approach for tool-integrated reasoning.
3.2 Token Efficiency Analysis
Beyond raw performance, analyze the token efficiency of models. Token efficiency can be roughly estimated by dividing the model's accuracy by its average token consumption. Figures and 4 illustrate this analysis, revealing several key insights:
Superior Efficiency of Binn--nniiner::g: At equivalent performance levels, Hint-Engitn fint--Eninnering series models demonstrate the highest token efficiency. For example, as shown Figure 1, Hint- Saitneering-FFT-330 achieves the same performance as QwQ-32B while using 50% fewer tokens (7K vs 14K). Comparing
with R RI-distill-32B, with just training examples for fine-tuning, the model reduces token consumption by approximately 30% while maintaining comparable performance. observed from the inference token budget analysis in Figure (a), Hint fiitt-Eninneringg achieves superior performance compared to Prompt-Hin Hint under

---

Aug Tokens Correct Resporses
60
20000
40%
70
9428
Averige Theeess
3323
Pasz
73.9%
75.3%
76.7%
72.99
77.39
72-1%
77.3%
10-
-10000
12207
22000
Aog Tikens for incorrett Resgonses
Premert
Hen Ar7
Hare
(a) Performance on AIMEZ4 with different token budget
(b) Average token usage on A1ME24
Figure 4: Token efficiency analysis on AIME24. (a): Token efficiency comparison showing Hint- Engineering- RFFT-322 achieves comparable accuracy with significantly fewer tokens (40-50% token saving) compared to Prompt-Hint-SFT- sponses across different models, with Hint-Engineering models maintaining lower token consumption while achieving competitive performance.
(b): Average token usage for correct and incorrect re-
limited token budgets, while Prompt-Hint shows significant advantage over CoT in these constrained conditions.
Low Token Usage for both Correct and Incorrect Responses: As shown in Figure 4 (b), compared CoT and Prompt-Hint approaches. Engineniine reduces token consumption in both correct and incorrect responses, indicating that it not only solves problems efficiently but also minimizes token waste during unsuccessful attempts. This improvement stems from Hint-Engineering's fundamental design: increasing the utilization of code for computations and enhancing confidence in code execution results.
As shown in Figure 1, when plotting performance against token usage, Seant--ngineeiin-RRRA RFF-332B sits in the optimal region with high performance and low token consumption, demonstrating the best cerpmmanncett-fffcciee ratio among all compared models.
3.3 Code Behavior Analysis between Prompt-Hin Hint and Hint- Siint-nnineering
We first establish taxonomy for Python code usage based on two dimensions. From the perspective of reasoning relationship, we categorize code usage into Calculation (computing results not present in the current reasoning chain) and Verification (validating results derived from chain-of-thought reasoning). In terms of specific functionality, we classify code into categories including Solving Equations, Numerical Approximation, Pattern Recognition, Combinatorial Enumeration and
We conduct a comprehensive analysis of Python code usage patterns across all test sets, comparing Hoonpt--int-FFT-322 and Hint fintt-nniinerring MaarinngRFF33222
RFT-32B, two models with comparable overall per- formance. We employ DeepSeek- V3 classify Python code functionality, with the corresponding classification prompts detailed in Appendix E.
Figure 5 provides a qualitative analysis of the code behavior patterns in our different approaches. The most striking difference is in how the models utilize code:
Code is predominantly used for verification purposes (68.2%), with only 31.8% dedicated to actual computational tasks. This indicates an inefficient utilization pattern where the model performs calculations natural language and then uses code primarily to verify these calculations.
Prompt-Hint Approach:
Hint- Eigineiiinn Approach: Shows a much more balanced usage, with 51.1% of code dedicated to direct calculation and 98.9% for verification. This more optimal distribution reflects the model's understanding of when leverage computational tools versus when rely on reasoning.

---

Prompt-Hiis
Pires fer incoratie
125
Potiise
1b)
Figure Analysis Python code usage patterns. (a) Distribution of code usage types: Hint- Engineering shows preference for calculation while Prompt-Hint favors verification tasks. (b) and (c) Function-specific distribution: Hint-Engineering demonstrates more balanced usage across different Python functions compared to Prompt-Hint.
Training vs. Code Reward AIME24
Prompt-Hint Approach
Hint- Hint-Enn neering Approach
(44
36-
Forrutriee 13-66
Frongterat--.22 L-3-44
1ir wwwiingggnnnn..... 58-14
60
100
333 140
42
330
120
340
Taningg
Figure 6: Ablation study the impact of code execution reward during RL training on AIME24. with and without code reward. Right: Performance of Hint-Engineeringg- -.5eringg---BBRRU 1.5B-RL with and without code reward. Both approaches show consistent performance improvements when trained with the additional code execution reward.
Left: Performance of Hoompt-Hint-L.S---
Additionally, the Bint-Engineering approach shows greater diversity the types of computational operations performed, including symbolic mathematics, equation solving, and combinatorial enu- meration. This suggests that the model has developed more sophisticated understanding of the appropriate use cases for different types of code operations.
As shown in Figure 5, Prompt-Hint and Hint-Engineering exhibit distinct code-integrated Rea- soning patterns. Prompt-Hint demonstrates a strong preference for verification (82.4%), while Hint-Engineering maintains relatively balanced distribution between calculation and verification (approximately 50% each). This balanced distribution emerges from the interplay between our Engineeninn design, which encourages computational efficiency, and the model's inherent tendency toward verification in long chain-of-the chain-of-thought reasoning. Regarding specific mathematical functionalities. we observe that Property Verification and Theorem Checking dominates Prompt- Hint's 's code usage (51%), whereas Bint-Engineering exhibits a more uniform distribution across different functions. Interestingly, both approaches share the same top-5 most frequently used Python functions, suggesting the influence of the test sets' mathematical domains. Moreover, we present representative examples in Appendix F.
3.4 Impact of Code Reward in RI.
Figure presents an ablation study the effect of incorporating code execution reward into the RL process. We set the code reward ratio w = here. The results demonstrate that incorporating this code reward consistently improves performance for both approaches:

---

Passigk Analysis
Prompt Hint Approach: Passgk on AIME24
Hint Engineering Approach: Fasogi on AIMEZ4
Desci R2
10 15041
Har wig.asin 13
(Mumbee Sumphol
k-Numbber Sumple)
Prompt-Atint Approach: Pass on NATH500-Level5
Hint-Engineering Approach: Pass@k on MATH500-LevelS
Smeptork 11 1.10 Nompo-maa 5841
Derr #11 Prghnemn 189 1.5.8L
(Hummee
L.Wkmbber Samples)
Figure 7: Pass@k performance on AIME24 (top) and MATH500-Level5 (bottom) for both Prompt- Hint (left) and Hint-Engineering (right). The analysis compares the base model DeepSeek-R1-i R1-1.5B with SFT and RL variants. While SFT does significantly improve the Pass k upper bound, RL substantially elevates performance across all k values, particularly at lower sampling budgets.
For Prompt-Hint: Models trained with code reward achieve up 5% higher accuracy than those without, reaching a peak of 43.1 versus 37.9%.
For Miint-Engnneering: A similar pattern emerges with approximately 3% performance improve- ment, reaching 41.0% versus 38.3%.
Notably, we found that the magnitude this reward is crucial: a modest code reward ratio w = 0.1 provides optimal results, while stronger penalties (e.g., 0.5) degraded performance. This suggests that while encouraging code correctness is valuable, overly penalizing experimental code attempts can inhibit the model's exploration and learning. Additional experimental results can be found in Appendix D.
3.5 Pass@ K Analysis
Figure illustrates the performance of our models as function of sample size (k) on both AIME24 and MATH500-Level5 datasets. Several important patterns emerge:
Supervised fine-tuning alone does not significantly raise the Pass @k upper bound for either approach. This suggests that while SFT can teach the model format and basic tool usage, doesn't fundamentally enhance the model's reasoning capabilities for 1.5B size model with the selected 10k probelms.
SFT Impact on Reasoning Ceiling:
RL Significantly Raises Performance Ceiling: Both Prompt-Hint-I.53 1.5--RLL and Hint- Ennineeiinne 1.5B-RL show substantially higher Pass@k curves than their SFT counterparts, particularly lower k values. This indicates that reinforcement learning successfully improves not just the average performance but the model's ability to consistently arrive at correct solutions with fewer attempts.
These observations confirm that RL effectively amplifying the benefits of the more optimal code usage patterns established during the Hint- Eiginnengin training.
Additional interesting findings, such as the impact of problem difficulty on RL performance and the evolution of code behavior during RL training, are documented in Appendix D.

---

Related Work
4.1
Reasoning in LLMs
The evolution of reasoning capabilities in LLMs has progressed rapidly in recent years [22-24, 3, 7. 6, 14]. For comprehensive coverage of this field, we direct readers to recent surveys [8. 9, 25-27]. pivotal technique in this field is Shain-of-TThughh (CoT) reasoning [10], which, when combined with Transformer architectures [28], enables models to perform complex computational tasks [29].
This field has benefited substantially from scaling up synthetic data [30-32]. These advances leverage various approaches, including preference optimization [33], tree search [34, 35], and advanced exploration strategies [36, 37].
Notably, following OpenAl's ol [38], there has been significant trend towards long-form CoT rea- soning, incorporating human-like cognitive patterns such as multiple reflections, task decomposition, and strategic exploration and these LLMs are often called large reasoning models (LRMs). This integration of iimannisspired reasoning approaches has led to substantial improvements in complex reasoning tasks, particularly in coding and mathematics, where models have demonstrated unprece- dented capabilities in systematic problem-solving [39-42, 36]. While natural language remains the primary medium for reasoning in this domain, there is growing interest in integrating external tools such as code interpreters [43, 44, 12] and automatic verification systems [45-47] overcome the inherent limitations of natural language reasoning, such as accurate computation.
4.2
Code-Integrated Reasoning for LLMs
Recent research has explored integrating external tools with language models to enhance their reasoning capabilities [48-53]. ToRA [12] pioneered code-integrated reasoning specifically for mathematical problem-solving. demonstrating that offloading complex calculations to specialized systems significantly improves performance. Since then, COA [54] trains LLMs to decode reasoning chains with abstract placeholders, and then call domain tools to reify each reasoning chain by filling in specific knowledge. rStar-Math [55] introduced a code-augmented Chain of Thought data synthesis method through Monte Carlo Tree Search. Recent works [56-59] have initiated investigations into frameworks that enable base models to autonomously develop code code-integrated reasoning capabilities for mathematical problem-solving.
With the advancement of LRMs, the organic integration of code interpreters within long-form CoT reasoning has emerged as crucial research challenge. Several concurrent works have explored this direction. START [21], which shares similarities with our approach, introduces hints to guide code generation within large reasoming modes; however, their random hint insertion may lead to suboptimal utilization of code interpreters. STILL3 [17] employs prompting techniques to construct integrate data, similar to our proposed Prompt-
method, but does not address reasoning efficiency. Retool [60] attempts to bootstrap training data by rewriting long CoT reasoning, yet shows limited performance improvements when based on DeepSeek- DeepSeck-RI-Distieee Sessiill-wee-Bee
While OTC [59] considers efficiency from the perspective of tool call frequency, it does not explore methods for enhancement building upon existing LRMs. CoRT proposed highly sample-efficient approach that achieved both performance breakthroughs and significant improvements in reasoning efficiency. Through numan-in-the-logy annotation of 30 high-quality samples, combined with techniques such as RFT [16, 61] and RL, they demonstrated that substantial improvements in both reasoning capabilities and efficiency can be achieved with minimal high-quality training data.
Conclusion
Our experiments reveal that properly integrated code tools enhance mathematical reasoning across model scales. High-quality data with optimal code behavior patterns can match exceed the performance of larger datasets, while reinforcement learning significantly improves performance beyond SFT, particularly for smaller models. The Hinii-engneerinn approach achieves remarkable efficiency, reducing token usage by 30-50% while maintaining competitive performance. Moreover, RL shapes code usage behavior toward either efficiency or increased integration. These findings demonstrate that combining high-quality data curation, targeted fine-tuning, and reinforcement learning with carefully designed rewards effectively enhances mathematical reasoning capabilities through tool integration.
10

---

Acknowledgments
This work of Xiang Wang is supported by the National Natural Science Foundation of China (62572449). This work of Benyou Wang is supported by NSFC grant 72495131, the Shenzhen Science and Technology Program (JCYJ202208) 18 1000000002), Shenzhen Doctoral Startup Funding (RCBS2022 100809333065), Shenzhen Science and Technology Program (Shenzhen Key Laboratory Grant No. ZDSYS2023062600 130200)),,, and Shenzhen Stability Science Program 2023. The work of Tian Ding is supported by Hetao Shenzhen- thenzhen-Hong Kong Science and Technology Innovation Cooperation Zone Project
The work of Ruoyu Sun is supported by NSFC (No. 12326608); Hetao Shenzhen- Hong Kong Science and Technology Innovation Cooperation 6); University Development Fund LDFF1001499 the Chinese University of Hong Kong, Shenzhen; Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023 (21200001000001).
Zone Project
References
[1] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1XXi:1707.00347 2017.
[2] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, 2024.
[3] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[4] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. TV, ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 2024.
[5] OpenAl. Learning to reason with Ilms. https//opeeii.comiinneeeeemmiii 3-to-reasonwwith-llme 2024.
[6] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprins arXiv:2505.09388, 2025.
[7] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. CoRR, abs/2501. 2025.
[8] Peng- Yuan Wang, Tian-Shuo Liu, Chenyang Wang, Yi-Di Wang, Shu Yan, Cheng- Xing Jia, Xu-Hui Liu, Xin-Wa Wei Chen, Jia-Cheng Xu, Ziniu Li, et al. A survey on large language models for mathematical reasoning. arXiv preprint arXiv:2506. 08446, 2025.
[9] Ziniu Li, Pengyuan Wang, Tian Xu, Tian Ding, Ruoyu Sun, and Yang Yu. Review rein- forcement learning for large language models: Formulations, algorithms, and opportumities. 2025.
[10] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of -f-thhughtt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24888 2022.
[11] Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, et al. A comparative study reasoning patterns of openaï's ol model. arXiv preprint arXiv:2410.13633 2024.
[12] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujuu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint 23X9v 2023.
11

---

[13] Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, et al. Can large language models detect errors in long chain-of-thoughh reasoning? arXiv preprint arXiv:2502. /9361, 2025.
[14] OpenAl. Introducing openai o3 and 04-mini, 2025. URL https: wtt.:/cccooooc/i/ee/eeeei roducing-o3- -4----03-ad-------i mini//
[15] Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[16] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship learning mathematical reasoning with large language models, 2023. URL https:tt////t //axivv org/ abs abs/a308.....////// 01825.
[17] Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving rl -like reasoning models. arXiv preprint arXiv:2503.
2025.
[18] Chunting Zhou, Pengfei Liu, Puxin Xu. Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu. Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-55022 2023.
[19] Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zeutlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint 2x..::25001199933 2025.
[20] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chem, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03388, 2025.
[21] Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang. Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. Start: Self-taught reasoner with tools. arXiv preprint arXiv:2503.0466 2025.
[22] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110, 14168, 2021.
[23] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.038874 2021.
[24] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilaa Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[25] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212. 2022.
[26] Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, survey. arXiv preprint 1111,
[27] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.099 096666, 2025.
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[29] Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances Neural Information Processing Systems, 36:70757-70798, 2023.
12

---

[30] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arX5v:2308.09583, 2023.
[31] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:23 10631, 2023.
[32] Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2023.
[33] Richard Yuanzhe Pang, Weizhe Yuan. He He. Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:116617-116677, 2024.
[34] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai. Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.005: 08935, 2023.
[35] Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, al. Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling. arXiv preprint arXiv:2508. 1 17445, 2025.
[36] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Wei-Ying Ma, Ya-Qin Zhang. Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-souree Ilm reinforcement learning system scale, 2025. URL https: //arxiv. org/ 22rg//abs/2503. 14476.
[37] Ziniu Li, Congliang Chen, Tianyun Yang, Tian Ding, Ruoyu Sun, Ge Zhang, Wenhao Huang, and Zhi-Quan Luo. Knapsack rl: Unlocking exploration of lims via optimizing budget allocation. arXiv preprint arXiv:2509.25849, 2025.
[38] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card, arXiv preprint arXiv:2412.167720, 2024.
[39] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https:
220.. 19393.
[40] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Shejie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025. URL htt.tppp//aakiw...t////////0... 03387.
[41] Huggingface. Open rl, 2025. URL heu.uttgge/ggtaaabb.ooo/ugggggge
[42] Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. ol-coder: an ol replication for coding, 2024. URL https: :/axxiv. htrg/abs/2412... 0154.
[43] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint 2xXi:2221..15588 2022.
[44] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
13

---

[45] Ruida Wang, Jipeng Zhang, Yizhen Jia, Rui Pan, Shizhe Diao, Renjie Pi, and Tong Zhang. Theoremllama: Transforming general-purpee llms into lean4 experts. arXiv preprint arXiv:2407.03203. 2024.
[46] Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. srepseek-prover, Advancing theorem proving in Ilms through large-scale synthetic data. arXiv preprini urXiv:2405. /4333, 2024.
[47] Kefan Dong and Tengyu Ma. Beyond limited data: Self-play Ilm theorem provers with iterative conjecturing and proving. arXiv preprint rrXi::2502.00212, 2025.
[48] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:230.. 2023.
[49] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu. Yihao Feng, Rithesh RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-callinn calling datasets. Advances in Neural Information Processing Systems, 37: 54463-54482, 2024.
[50] Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343, 2025.
[51] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement abs/ 05592.
learning, 2025. URL htttps//aaree
[52] Xiaoxi Li, Guanting Dong. Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models, 2025. URL attps://arme org/ abs absb/5511.. 05366.
[53] Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL /atttss://axiii prg/abs/22e /2002. 04761.
[54] Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating on long-context tasks. Advances in Neural Information Processing Systems,
2024.
[55] Yao Zhang, Hongxiao Zhang, Jiacheng Zhang, Jingcheng Zhao, Rui Yan, Xiaoqing Liu, Jiahuan Wang, Min Zhang, Houfeng Wang, and Zhengguang Guo. rStar-Math: Small LLMs can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2403.01707, 2024.
[56] Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, and Fangzhen Lin. Learning autonomous code integration for math language models, 2025. URL https://arxx:
v. org/ aa////b///// /5020 00691.
[57] Xinji Mai, Haotian Xu, Xing W. Weinong Wang, Yingying Zhang, and Wenqiang Zhang. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving,
2025. URL https: /arxiv.iivv.... wwt.oogg//////...oooo ab20055. 07773.
[58] Kezhou Wang, Ruijie Wu, Qinlin Zeng. Huao Lu, Hanye Wu, Qingfeng Cui, Haichao Lin, Yujia Liu, Xiaoyan Huang, Qingpeng Guo, Songtao Jian, Kaiyuan Lu, Shiyu Li, Hao Tian, Yongqin Sun, Xue Yang, Libin Song, Zejun Ou, and Guoqing Wang. ToRL: Scaling tool-integrated RL. for LLMs. arXiv preprint 23 2023.
[59] Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Ote: Optimal tool calls via reinforcement learning. arXiv preprint 2504v:2504. /4870, 2025.
[60] Jiazhan Feng Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in 1lms, 2025. URL https: //ops::/aaan /awxiviiivv...//i org/ abs/2504. 11536.
14

---

[61] Hanze Dong. Wei Xiong. Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: reward ranked finetuning for generative foundation model alignment. Trans. Mach. Learn. Res., 2023.
[62] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu, Hybridflow: flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
[63] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. com/project http:/proecttnminaaaaemmea nwminummaaaaaaamaa aimo- pimaa-rgree-ppssees
htttps://huggiggeee
co CooAAI-M0/MummmmMathhh 1. 5] aattps://githen --prrize/blob/mmii/rreppoo wwwprriattmaaaaaaaaeeaaaaaaaaaae _ataaiaadaatttttaaaaaaaaaaaaeeee pdf), 2024.
[64] Hynek Kydlíček. Meth-verify: Math verification library, 2024. URL https:/s:///iia com/h uggingface/matiin vetth-
[65] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with 1.5b model by scaling rl. https / radeptttttrrriooooo ra5 notion site/D PesScaleR-Surpassene 1aets---1Preeeeee www--Peeeeeiwidlae a 1B-1--5---0 M--B-Model-Daa Soy--Saainnn-aae- RL-19681 900c14466655ddd8c000000334444443 2025. Notion Blog.
[66] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
[67] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurlPS Datasets and Benchmarks, 2021.
[68] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https: / gi tituub. com/pro prw.ress--rreeeeeeeeeee----- prize] thttps://githii com/projece ject- http://ooccttuuiiaa//a/aaaaaaaaa aimo pptggeesssrrtee////o/////aprprpp dataset pdf), 2024.
ject--nmina/ aainnnaaaaaaaaaaaaaaaaaaaa progress-rreeeseeeeeeo
[69] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenln. github. ho/bl/gg//gg/3/////
[70] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https: //openreview. hw/.fforr///111111555555..
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contributions are clearly articulated in the Abstract, while the concluding paragraphs of the Introduction provide detailed scope and enumerate specific technical innovations that accurately represent the paper's content and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix K.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
Our work is an empirical exploration.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all of the evaluation and training hyperparameters in the Appendix and we plan to release the full code and model.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will submit our code in Supplemental Material and will open-source code and models GitHub in the final version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (attps://nips cc/pu
blic/gu iewudd///odeeeaeeeeeeeeee wwweeesuuiiissisnnnni. ssssssssiccyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /ttpps.cc/pppiiiicc htt.cccpppllllcccc honollccgguddes.. SeguddeeeCoeeuubbni
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We will specify all details regarding training and test the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide statistical significance for our main claims in two ways. First, as detailed in the caption of Table 1, the primary performance metries reported for the AIME24, AIME25, and AMC23 benchmarks are averages over 16 stochastic samples (avg@ 16), and for MATH500 and Olympiad, they are averages over samples (avg@ @4). This averaging process inherently reduces the impact of random variance and provides stable estimate model performance. Second, to further strengthen our claims, we conducted pairwise Wilcoxon signed-rank tests to formally assess the statistical significance of the improvements in both accuracy and token efficiency. The detailed methodology and results of these tests, which confirm the statistical significance of our findings, are provided in Appendix D.6.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We will claim our computing resources in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ////nnurippp ipsuipp..oo hwtppalilc/hcccaciieeeee.eeeeeee
Answer: [Yes]
Justification:
We have read this code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the limitations in Appendix J.
19

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Our study is an empirical exploration using open-source mathematical problem sets. The models we release are intended only for academic research purposes and not designed for industrial applications, posing minimal risk for misuse.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have rigorously credited all creators and original owners of assets used in our research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datattssets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All datasets, code, and models developed in our research will be comprehen- sively released under the CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We will provide detailed instructions in the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification:
The annotators involved in our research were fully informed about all details of the annotation task and its purpose. We obtained proper institutional approval before beginning the annotation process
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We use LLMs solely for grammatical checks in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: /ttpas:://ueuppspeeeeee http://uuurss.ceeeeeeeeeeeeeeeee httpa///oooeeeeeee//////0/0 /2tp:/cceee/////////e/ for what should or should not be described.
22

---

A Experiment Implementation
A.1 Training Implementation
For our experiments, we implemented several model variants with different training stages and architectures:
32B Models:
Prompt-Hint-SFT--22: Starting from the DeepSeck-R113222 base model, we fine-tuned using 800 data samples with learning rate of x 10 runninng for 17 epochs with a batch size of 96.
fine-tuned using only 30 high-quality, human-annotated data samples with a batch size of 96, learning rate of x 10-5, and 40 epochs.
Soitt-Egiineerig-STTe SFFT333B. Based PeepSeck-R1-325
Hint-Enginee Settineeiinn-FF-32222 Building upon Hint-Engineering-SFF SFT-32R, we further fine- tuned using 800 filtered data samples with a learning rate of 1 x 10-5, 5, I7 epochs, and batch size of 96.
1.5B Models:
We distilled both Hint----int-SFT-332B and Hint--nnginering--RFT-- down to the DeepSeck-R1-A -1--1-BB architecture using 10k data samples with a learning rate of 7 x 10-6, epochs, and batch size of 128.
For reinforcement learning, we adapted the veRL framework [62] to implement our special- ized design outlined in Section 2.4. We further trained these 1.5B models with a learning rate of x 10-6, maximum response length of 16,000 tokens, rollouts per problem, and maximum function calls limited to 15 per response, with each function call having maximum length of 16,000 tokens.
The RL training data was carefully selected by computing the average accuracy over samples (avg@8) on 20k randomly selected problems from the NuminaMath-1.5 [63] dataset, then selecting only Ik challenging problems where avg@8 1/8 for focused training. This selective approach is motivated by our data ablation studies (Appendix D.2), which demonstrate that training on hard queries, while requiring longer convergence time, ultimately yields superior performance compared easy uniformly distributed queries.
A.2 Evaluation Methodology
To ensure comprehensive and fair comparisons across different approaches, we implemented the following evaluation protocol:
Fair Comparison: For publicly available models, we re-evaluated them on our local infras- tructure using their original evaluation scripts ensure consistent comparison conditions across all models.
For all datasets, we extract the final answer from each model response and compare it directly to the ground truth using Math-Verify [64], considering problem correctly solved only when Math- Math-Verify returns True.
Evaluation Protocol:
base metric. Concretely, employed @ 16 (average accuracy over 16 samples) as pass@1 I for AIME24, AIME25, and AMC23 datasets. For MATH500 and OlympiadBench datasets, used avg as pass@I @ due to their significantly larger test sizes.
Evaluation Metries: We primarily used pass @ 1
Inference Setting: Across all evaluations, we standardized inference parameters with maxi- mum sequence length of 32,768 tokens, maximum function calls limited to 15, maximum tokens per function call set to 32,768, temperature of 0.6, and top-p sampling parameter of 0.95.
23

---

A.3 Function Call Limiting Strategy
During both evaluation and reinforcement learning rollout sampling with Python usage, we imple- mented mechanism to handle scenarios where models reached the maximum allowed function calls. When this limit was reached, we appended the following system message to guide further reasoning:
[SYSTEM]
You have exceeded the allowed number of code executions. You can no longer write code. Please continue solving the problem using your reasoning and amalytical skills.
This approach ensures that models can still complete their reasoning process without unlimited computational resources, better reflecting real-world usage constraints.
A.4 Computational Hardware
Our experiments utilized the following hardware:
All training procedures, including supervised fine-tuning (SFT), rejection fine- tuning (RFT), and reinforcement learning (RI.), were conducted on servers, each equipped with 8 NVIDIA A100 GPUs.
Training:
Evaluation: All model evaluations were performed single servers, each equipped with 8 NVIDIA A100 GPUs, ensuring consistent measurement conditions across all compared approaches.
B Hint-engineering Process
The Iterative Bint-ennine Loop Hint-engineering implements an iterative refinement proce- dure that converts imperfect reasoning trajectories into expert-aligned ones. For problem instance P, let (() denote the trajectory produced at iteration i. The loop operates as follows:
1. Initial generation (i = 0). The reasoner produces an initial trajectory 7(0) for P.
2. Annotation and evaluation. A human annotator reviews 7(9) If no deviation from the desired reasoning detected, the procedure terminates with the final trajectory =
Otherwise, the annotator localizes the erroneous step /. and its associated action a, and formulates corrective hint h;-
3. Localized revision and resumption. The context step is augmented with h;, yielding an updated state. From this state, the reasoner resumes its computation and produces the refined trajectory ((i+1),
The process iterates until (()) meets the acceptance criteria or the allotted budget is exhausted.
C Evaluation Dataset Description
To comprehensively assess the mathematical reasoning capabilities of our models, we utilize several challenging benchmarks that span diverse difficulty levels and mathematical domains:
AIME24 and AIME25. The American Invitational Mathematics Examination (AIME) represents significant advancement beyond standard high school mathematics competitions, featuring problems that demand sophisticated reasoning techniques. We employ AIME24 and AIME25 as our primary benchmarks for evaluating advanced mathematical reasoning capabilities our models.
AMC23. Following DeepScaleR [65], we utilize their American Mathematics Competition (AMC) test set, which presents problems of moderate yet substantial difficulty, requiring considerable mathematical insight to solve. This dataset enables us to evaluate our models' proficiency in addressing a wider spectrum of mathematical challenges typically encountered in standard high school competitions.
24

---

MATH500. Curated from the test split of OpenAl's PRM800K dataset [24], MATH500 encom- passes 500 carefully selected problems that represent a diverse range of mathematical challenges. We utilize this dataset to evaluate our models' ability to generalize across varied mathematical topics and problem structures.
Following DeepScaleR [65], we incorporate their OlympiadBench [66] into our evaluation framework. This benchmark comprises 675 Olympiad-level problems sourced from elite mathematics competitions, providing exceptionally rigorous test of advanced mathematical reasoning.
Olenp.
D
Extra Experiments
D.1
Code Behavior Evolution During RL
Code Behavior Metrics During RL. Training (AIMF24)
Code Usage Rate
Code Success Rate
Ang Code Blocks
6.00
6.-
Code 0.7
0.3
0.0
Taining
Stazea
Figure 8: Evolution of code behavior metrics during RL training AIME24.
Figure tracks six key metrics of code behavior throughout the RL training process:
Code Usage Rate: The percentage of responses containing Python code out of responses.
Both approaches show increasing code usage rates during training, with Hint-Engined Eigineennnnninng consistently maintaining higher rates, starting at 86% and quickly rising above 95%.
Code Success Rate: The percentage code blocks that execute without errors. The success rate shows different patterns between the two approaches. Prompt- Hint maintains relatively stable success rates around 78%, while fintt-nnineering shows more variability but achieves higher peaks.
The average number of Python code blocks per response. Interest- ingly, Hint-Engineering shows a steady decrease the average number code blocks from 4.4 to around 3.5 at peak performance, while Prompt-Him increases from 1.9 around 2.7. This divergence reveals a fundamental difference evolution: Hint- fenn--Egineering evolves toward more efficient code usage (fewer but more effective blocks), while Prompt-Hint develops more code integration capabilities from its lower starting point.
Average Code Blocks:
These patterns reveal that reinforcement learning not only improves raw performance but actively shapes code usage behavior, with Hint Engineerig evolving toward an (less code but more effective) while Prompt-Hin Hint evolves toward increased code integration (more code with improving effecttiveness.
pattern
D.2 RL Training Data Ablation
To understand the impact of training data characteristics on RL performance, we conduct three ablation studies examining data volume, query difficulty distribution, and topic distribution effects.
Data Scaling Ablation (Figure 9, left): We investigate whether increasing training data volume directly improves optimal performance by comparing 1K, 5K, and 20K training samples while maintaining uniform difficulty and topic distributions. Surprisingly, we find that simply scaling up RL training data does not lead better optimal performance. This finding suggests that the "less is
25

---

RL "raining Data Ablarion Studies on AIME24
Data Volume [Unform Oimicuty & Tepici
Difficulty Level (IK Samples. Uniform Topic]
Toolc Distribution Abition (IK Sampless Unioe Difiuutyl
Figure 9: RI training data ablation studies on AIME24. Left: Data scaling ablation with 1K, 5K, and 20K training samples under uniform difficulty and topic distributions. Middle: Query difficulty uniform, and hard (avg 8-1/8) difficulty distributions with
ablation comparing easy IK samples. Right: Topic distribution ablation comparing uniform and hard topic distributions with IK samples under uniform difficulty.
more" principle still holds in large reasoning model (LRM) training, and data quality may be more important than quantity for RI. fine-tuning.
Query Difficulty Ablation (Figure 9, middle); We examine how query difficulty distribution affects learning dynamics by comparing three settings with IK samples: easy queries (avg@8-7/8), uniform difficulty distribution, and hard queries 11/8) Our results reveal distinct learning patterns: easy queries achieve optimal performance earliest but with lower peak accuracy, uniform distribution shows intermediate behavior, while hard queries take longer to reach optimal performance but ultimately achieve the best results. This suggests that training on challenging examples, though slower to converge, leads to superior final performance in mathematical reasoning tasks.
Topic Distribution Ablation (Figure 9, right): We investigate whether aligning training topic distribution with hard query topics improves performance by comparing uniform topic distribution against distribution matching hard queries (avg@ 8-1/8). The results show minimal differences between the two distributions, indicating that topic distribution changes do not significantly impact optimal performance. This suggests that the model's reasoning capabilities generalize well across different mathematical topics, and the difficulty level is more crucial than specific topic coverage.
D.3 Code Reward Ablation
To investigate the effectiveness of our code reward mechanism and determine the optimal penalty strength, we conduct ablation studies comparing different penalty values the code reward function.
Code Reward Penalty Ablation Study on AIME24
Cade Reward Penalty 0.1
Code Reward Penalty 0.5
32
Prmptii code Promkee eod.50 Eu aetinng panally 0.1
Fooegti 53-88
Doonstrare----- 31-7----ttttt sutting panalty
128
3.0
160
Figure 10: Code reward penalty ablation study on AIME24. Left: Comparison between training with and without code reward using p.n Right: Comparison between training with and without code reward using pen on different SET model.
26

---

Code Reward Effectiveness: Our results demonstrate that incorporating code reward significantly improves model performance compared training without In the penalty=0.1 setting (Figure 10, left), the model with code reward achieves a peak accuracy of 43.1 1% at step 100, substantially outperforming the baseline without code reward (37.9 97%9% peak). This improvement persists throughout most of the training process, indicating that the code reward provides consistent learning signals that guide the model toward better reasoning strategies.
Penalty Strength Analysis: Comparing different penalty values reveals that penalty=0.1 yields superior and more stable performance than penalty=0.5. The penalty=0.1 configuration shows smoother convergence and maintains higher accuracy levels across training steps. In contrast, penalty=0.5 (Figure 10, right) exhibits more erratic behavior, with a notable spike step 60 (41.4%) followed by sharp decline. This suggests that excessive penalty strength may introduce instability in the training process, potentially causing the model to over-correct its behavior when generating incorrect code.
D.4 Token Efficiency Analysis for Lightweight Models
We conduct a detailed analysis of token efficiency in 1.5B parameter lightweight models, examining how different approaches impact computational resource utilization while maintaining mathematical reasoning capabilities.
Performance AIMEZ4 with Different Token Budgets
Token Usage Analysis AIME24
Ang. Tikens for Conrect Responses
60-
7635
6812
Average Tokkens
20 0%
AO 0%
7%%%%
30.0%
43.2%
34.8%
43.0%
30
2054
-1000
8435
20
12352
-20000
Premut-iip Haw.ou-oot--------00
21:99
Hit enginering 1 58
Aug Taklens Incorrect Responses
10
12
Inference Token Budget (k)
10
Deriitel R1
Pronpp Mesk Spp
Promist dlea
drop
Wes
Figure 11: Token efficiency analysis for 1,5B parameter models on AIME24. Left: Performance comparison across different token budgets. Right: Detailed token usage breakdown for each model, showing average token consumption for both correct (above axis) and incorrect (below axis) responses, with Pass @ accuracy displayed the center.
Our token efficiency analysis reveals two key insights about the computational efficiency of lightweight mathematical reasoning models:
Superior Performance with Limited Token Budgets: As shown in Figure 11 (left), Hint- Engineering-SFr consistently outperforms both the base model (DeepSeek-RR-1.55) and Prompt- Hint-SFT across all token budget constraints. Most notably, with just 4k token budget, Hint- Engineering-SFT achieves 22.7% accuracy, nearly double the performance of DeepSeck-Rl-Les R1-1.5 (12.1%) and Prompt-Hing Hint-SFT (1 (.0%). This indicates that Hint- Sinun--nnnnerin''s 's structured approach to problem-solving enables more efficient reasoning within constrained computational environments.
Substantial Token Savings Across All Response Types: Figure 11 (right) demonstrates that Hint-Engineering models maintain significantly lower token usage compared to alternatives. For correct responses, Siint-Engineerin-SSFFT uses 47% fewer tokens than Prompt-Hint-She 8,862), while for incorrect responses. the savings are even more dramatic, with Hint-Enginneeing--P consuming 31% % fewer tokens than Prompt-Hint- (8,485 vs. 12,292). Overall, Hint-Enninnerrin: RL achieves 32% reduction in total token consumption compared to Hompt-Hint-Ree (6,684 vs.
(4,711 vs.
27

---

9,891) while maintaining comparable accuracy (41.0% vs. 43.1%). Furthermore, Hint- Sint--nninneering models use about 50% fewer tokens for the 1.5B model compared with the natural language models.
D.5 @ass@ K Analysis for Frontier Models
To understand how approaches scale with sampling budget and model size, we conduct comprehensive Pass @k analysis on 32B parameter frontier models.
Pass@jk Analysis
Prompt Hint Approach: Pass@k on AIMEZ4 (328)
Hint: Engineering Approach: Parsgk on AIME24 (328)
DNamber of Semplest
Otunbeer Samples
Prompr-Hint Approach: Passgk on MATH500- Levels (328)
Hint Hint-Engineering Approach: Pasegk on MATH500-Levet5 (328)
Hom Mompooe
1lNumber Sumphol
X.uumbrrr Sumpto))
Figure 12: Pass @k analysis for 32B parameter models. Top row: Performance on AIME24 comparing Prompt-Hin Hoitt baseline. Bottom row: Performance on MATH500-Level5 showing similar patterns.
Hint (left) and Hint-Engineeni Engineering (right) approaches against the DeepSeek-R1-322
Figure illustrates the performance our 32B parameter models as a function of sample size (k) on both AIME24 and MATH500-Leveel datasets. Our analysis reveals two key patterns in the scaling behavior of these models.
SFT Provides Modest Gains at Lower Sampling Budgets: Both Honnptt-intt32B-B-5 and Hint- Engineering-322 --B- 32B-SFT show improvements over the baseline DeepSeek-Ri- -1-32B,, particularly lower k values. At k=1 on AIME24, baseline's 72.9%. However, all models eventually converge similar maximum performance levels (93.3% for AIME24 and 99.2% for MATH500- as k increases, suggesting that SFT primarily enhances the model's efficiency rather than raising its reasoning ceiling.
achieves 77.3% accuracy compared to the
RFT Significantly Enhances Sample Efficiency: Hinn-Enginnerrng-g22--- --2--RFF-F2RFTR demonstrates remark- able efficiency, reaching 93.3% accuracy with just k-8 samples on AIME24, while other approaches require 2-4 times more samples to achieve comparable results. This indicates that reinforcement fine-tuning successfully optimizes the model's ability to consistently arrive correct solutions with fewer attempts, making it particularly valuable in scenarios where computational efficiency is critical.
D.6 Statistical Significance Testing
To rigorously validate our empirical results and ensure that observed improvements are statistically meaningful, we conducted pairwise Wilcoxon signed-rank tests. This parameatettiic statistical test is well-suited for comparing the performance of two models on per-problem basis, without assuming normal distribution of performance differences.
28

---

We followed the procedure outlined in our rebuttal: we pooled the results across all problems from the five mathematical benchmarks create a sufficiently large sample for robust analysis. The was performed separately for our 32B frontier models and 1 1.5B lightweight models to assess statistical significance different scales. We evaluated two key dimensions: accuracy improvements and token reduction.
The results of our statistical tests are presented Table 2.
Table 2: Statistical significance of performance gains, evaluated using pairwise Wilcoxon signed-rank tests pooled results from all benchmarks. The p-values confirm that our efficiency gains are highly significant, and accuracy improvements are either significant or show a strong positive trend.
Model Scale
Model A (Ours)
Model (Baseline)
Acc. Improv.
p-value (Acc.)
Token Reduction
p-value (Tokens)
Hint-Eng- RFF
R1-disti R1-distill-1.BB
+3.80%
36.1%
1.5
Hint-Eng-RL
+7.41%
0.013
59.1%
0.001
The analysis confirms the following:
Token Efficiency Gains Highly Significant: For both the 32B and 1.5B model scales, the reduction in token consumption statistically highly significant, with < 0.001. This provides strong evidence that the CoRT framework induces fundamentally more efficient reasoning process.
Accuracy Improvements are Statistically Meaningful: For the 1.5B model, the accuracy improvement of +7.41% is statistically significant (p 0.013). For the 32B model, the +3 80% improvement shows a strong positive trend that approaches statistical significance (p 0.055).
These results, combined with the fact that our reported metrics in Table are averages over multiple stochastic samples (16 for AIME/AMC and for MATH/Olympiad), robustly support our claim that the CoRT framework leads to meaningful and reliable improvements in both reasoning accuracy and efficiency.
E
Prompts
Hint-Prompt and Sint-Engineering Prompt
Given a mathematical problem, follow the instructions below to solve it.
Instructions:
When solving mathematical problems, you should leverage both natural language reasoning and interactive Python code execution. Your goal is to provide clear, detailed explanations while utilizing Python to perform complex calculations, symbolic manipulations, data analysis, any other tasks that can aid in problem-solving. Follow these guidelines to ensure a coherent and effective response:
Natural Language Reasoning:
Provide comprehensive, step-by-step explanations of your thought process. Ensure that each step logically follows from the previous one, maintaining clarity and coherence.
Use appropriate mathematical terminology and notation where necessary. Planning, Modeling, and Analysis:
Use natural language outline the overall approach the problem.
Develop mathematical models representations as needed.
Analyze the problem to determine the best strategies for finding solution.
Inserting Python Code Blocks:
When Python code snippet can aid in analysis, computation, symbolic manipulation, insert a Python code block.
Use triple backticks with 'python' ' to denote the start of Python code block and triple backticks to close it.
29

---

Example:
"python
3. Displaying Code Output:
-Immediately after Python code block, present the output generated by the code.
Use triple backticks with "output" to denote the start of the output block and triple backticks close it.
Exammpe: "outppt
".
Encouraging Multiple Python Calls and Diverse Functionality:
Utilize Python multiple times throughout your solution to handle different aspects of the problem.
Take advantage of various Python libraries and functionalities such as:
"numpy' for numerical computations
"scipy" for scientific computing and advanced mathematical functions
'sympy" for symbolic mathematics
'pandas' for data manipulation and analysis
'math' for fundamental mathematical operations
"statisties' for statistical computations
'frcction'' for rational number calculations
Ensure that each Python snippet is purposeful and enhances the understanding resolution of the problem.
Specific Calculations and Complex Operations:
Use Python to perform detailed calculations that would be cumbersome by hand. Implement complex algorithms data processing tasks that facilitate the solution.
Handle any intricate operations that support the overall analysis and modeling of the problem
Problem:
Code Behavior Analysis Prompt
You expert in analyzing code and understanding its purpose, especially within the context of mathematical wroblem-solving. Your task is to analyze Python code snippets within a solution to mathematical problem and classify each snippet based on its purpose.
You will be given a problem/solution pair. The solution may contain multiple Python code snippets. For each Python code snippet, you must determine:
1. Is it Verification or Calculation?
Verification: The Python code verifies a result or conclusion that was already reached through reasoning in the solution. The Python code confirms pre-existing answer or property.
Calculation: The Python code calculates result that NOT explicitly present in the solution's reasoning up to that point. The Python code derives a new, previously unknown answer or intermediate value.
2. What is the specific function of the Python code snippet? Choose the following list functions (be as specific as possible). If none of these functions are appropriate, provide brief (one sentence) description of the function of the code.
more from
Solving Equations and Systems of Equations Finding numerical or symbolic solutions to algebraic, differential, and other types of equations.
30

---

2. Symbolic Mathematics and Manipulation Performing algebraic operations such as differentiation, integration, simplification, and expansion using symbolic math libraries like SymPy.
3. Numerical Approximation Methods
Approximating solutions for problems that lack analytical solutions, including numerical integration, root finding, and solving differential equations.
Data Visualization and Plotting
Creating graphs, charts, and other visual representations using libraries like Matplotlib and Seaborn to illustrate mathematical concepts and data patterns.
Pattern Recognition and Analysis
Identifying and analyzing patterns or relationships in data using statistical and machine learning techniques.
Optimization and Solution Searching
Implementing algorithms to find optimal solutions to problems, including linear program- ming, integer programming, and heuristic methods.
Property Verification and Theorem Checking
Verifying mathematical properties and theorems for given inputs using computational methods.
Modular Arithmetic and Number Theory Operations
Performing calculations involving modular arithmetic, such finding inverses, solving congruences, and applying the Chinese Remainder Theorem.
9. Prime Number Testing and Factorization Determining the primality of numbers and performing prime factorization using efficient algorithms.
10. Geometric and Computational Geometry Calculations Calculating areas, volumes, distances, angles, convex hulls, intersections, and performing geometric transformations.
11. Probability, Statistics, and Simulations Computing probabilities, expected values, variances, and running Monte Carlo simulations to model random processes.
12. Linear Algebra: Matrix and Vector Operations
Performing matrix multiplication, inversion, eigenvalue decomposition, and other linear algebra operations using libraries like NumPy and SciPy.
13. Data Generation and Simulation Creating synthetic data sets and simulating mathematical models to explore and analyze behaviors.
14. Combinatorial Enumeration and Game Theory
Counting permutations, combinations, and analyzing combinatorial games to determine winning strategies.
15. Graph Theory Algorithms
Implementing algorithms for graph traversal (DFS, BFS), shortest paths (Dijkstra's, Floyd-Wor Warshall,, and finding minimum spanning trees (Kruskal's. ss Prim's).
31

---

16. Dynamic Programming and Recurrence Relations Designing dynamic programming solutions and solving linear and non-linear recurrence relations find closed-form expressions.
17. Fast Fourier Transforms (FFT) and Signal Processing
Utilizing FFT for problems involving polynomial multiplication, number-theoretic transforms, and analyzing frequency components.
18. Boolean Algebra and Logic Operations Manipulating and simplifying logical expressions, constructing truth tables, and solving Boolean equations.
19. Big Integer and Arbitrary-Precision Arithmetic Handling calculations with very large integers beyond standard data type limits using Python's 's arbitrary precciiioo capabilities.
Symbolic Integration, Differentiation, and Proof Verification
Perforring advanced calculus operations and assisting in verifying mathematical proofs using symbolic computation librarics.
21. Linear Programming and Optimization Techniques FForuulating and solving linear optimization problems using libraries like PuLP and SciPy.
22. Algorithm Optimization and Numerical Stability
Enhancing algorithm performance by improving time and space complexity and ensuring numerical stability for accurate results.
23. Automated Theorem Proving and Symbolic Logic
Utilizing tools and libraries to automatically prove mathematical theorems and manipulate symbolic logic statements.
24. Data Structures Implementation and Management Creating and utilizing advanced data structures such as trees, graphs, and heaps to efficiently solve competition problems.
25. Fractal and Recursive Pattern Generation
Creating and analyzing fractals and other recursive patterns relevant to geometry and combinatorics problems.
You will be given a "Problem" and a "Solution". The solution may contain one or more "Python code" snippets. For each Python code snippet, provide the following output:
Python Code idx: [The idx of Python cfor example 1,2.3)
Classification: !Verification or Calculation] Function: [one more specific functions from the list above OR one-sentence descriptions]
Problem:
Solution:
32

---

Case Study
F1
Samples of D eint--nninneeinngSS
Sample
Problem:
Find the least positive integer n for which 2" + 5" - is a multiple of 1000.
Solution:
Let's solve this step by step. The problem is to find the least positive integer such that 2n + 5 n = 0 (mod 1000).
Since 1000 = 8 x 125, by the Chinese Remainder Theorem, this is equivalent to solving:
2 +5" = (mod 8)
2 + 5thh
0 (mod 125)
For modulo 8: When 3: 2 (mod 8) 5" (mod 8) cycles every 2: 1, 5, After analysis, we find n = (mod 8)
For modulo 125, let's first find the period of 2 mod 125:
Python Code
def pand_period_mdd15 seen_values uurrent_valus = 1 position sequence []
while True:
if current_valle value in seen_values:
period_length position seen_values [current_value] period_sequence sequence seeen_values [current_value]
return period_lengte period_sequence
seen_ goen_values [current_aalll value] position
sequence. a (current__aauee)e)
perrent_value (current_vall) value 2) % 125
position +=
period_length, period_sequence = find_period_mod print (finerrd length: (period_length)""
print("\nxxmmple cases:") test_numbers [0, 1, 2, penioh_ for n test_numbers:
result = pow (2, n, 125)
print (f) mod 125 (result}")
Output
Period length: 100
Example cases:
2-0 mod 125 1
2-1 mod 125 2
2n2 mod 125 4
2-99 mod 125 63
33

---

Similarly, let's check the period 6* mod 125:
Python Code 2
def
()
seen_values 0
current_value 1
position 0
sequence
while True:
if current_ in seen_val seen_values:
period_length position seen_values Courent_valus
period_sequence = sequence [seen_alues [current_valuee
return period_length, period_sequence
seen_ values sequence (pponncuuttot__alll)
position
current_valus (current_value 6) 125 position += 1
period_length, period_sequence find_ madd_pprrod_mdd122 print length: (period_lengthh"")
print(") cases:")
test_numbers = [0, 1, 2, period_length-1
for n in test_ test_numbers:
result = (o
125)
print (f"6 (n) mod 125 (result}")
Output 2
Period length: 25
Example cases:
600 mod 125 1
6-1 mod 125 6
6-2 mod 36
64 mod 125 21
[...detailed analysis omitted...] After analysis, we need to solve:
32 6 = 8h +
(mod 125)
Let's find the solution:
34

---

Python Code
def find find_k():
for k in range (0, 150):
mod_ 6k pow (6, k, 125)
1hs (32 * mod_ 6k) % 125
rhs (8*k
% 125
if 1hs == rhs:
print (fund k=(k), (8 return 8*k+5
return None
result find_k(
print (result)
Output 3
Found k=99, n=8+99+5=797
797
Therefore, 99 gives n 8 99 + 797.
Verification: 5797 = 125 (mod 1000) (since 5" cycles 125, 625 for odd/even n 2 3)
2797 672 (mod 1000) Therefore, 2797 5777 797 0 (mod 1000)
The answer is 797
Sample 2
Problem:
Find the largest prime number p < 1000 for which there exists a complex number & satisfying: (1) the real and imaginary part of are both integers; (2) |x] vp. and (3) there exists triangle whose three side lengths p, the real part of 23 and the imaginary part of 23. Solution:
Let's try to solve this problem step by step. The goal
find the largest prime number < 1000 such that there exists complex number = with integer real and imaginary parts, [2| = vP, and there's triangle with side lengths p. Re(=²), and Im(=²).
First, let's understand the problem. We need to find a prime p < 1000. The complex number bi where and b are integers. The magnitude |z| Va² 12 vP. which implies that a² 6² p. Therefore, p must be prime that can be expressed as the sum of two squares. That's 's key point. [...detailed analysis omitted...]
must have integer real and imaginary parts, so
So the algorithm we will implement is:
For each prime p (starting from largest smallest):
Find (a, b) using find a b(p)))
Generate all variations:
(a, b), (-a, b), (a, -b), (-a, -b), (b), (b, -0), -a)]
For each variation (I, y):
Compute Re 3 3ry
Compute Im 3xz - y²
If > 0 and Im > 0:
* Check triangle inequalities:
-p+ Re > Im
-p+ Im Re
Re + Im p
35

---

If all three true, then return valid solution
Let's implement this in Python:
Python Code
import math
def sieve (n)
sieve [True] (n + 1)
sieve [0] sieve [1] False
for i in range(2, int (math.sqrt + 1): if sieve
sieve [i :inn:
i]
primes = [i for i, is_prime in enumerate(sieve) if is-prime] return primes
def find_a_b(p)
max_a int (math, (ssnt(p))))(())) for a in Lange(max__ 1): b_sq p if b_sq < continue
b int (math. (b_s((b_s))) if b * b_sq:
return (a, b)
return None
def is is_triangle(p. (p, re, im): return (re im > p) and (p > im) and (p + im > re)
def check Beocck_prime_fteee foreees__oo__eee for_triangle():
oligible_primes
eligible_primes sort (reverse=True)
for p in eligible stigible_primes:
ab = find_a_b(p)
if ab is None:
continue
a, b = variations
(a, b), (-a, b), (a, -b), (-a, -b),
(b, a), (-b, a), (b, -a), (-b, -a)
for x, y in variations:
= x ** 3
** 2
im 3 * x ** 2 y ** 3
if > 0 and im 0:
if is_triangle(p, re, im)
print Fouuud valid prime ...")
print Chehhinnn triangle inequalities: ...")
print (f" Answer: (p)")
return
36

---

Output
Found valid prime 349 with (=,))=(-5,118)) re=4735, im=4482 Checking triangle inequalities: True, True, True Answer: 349
Therefore, the largest prime that satisfies all conditions is 349
Sample
Problem:
Each face of two noncongruent parallelepipeds is a rhombus whose diagonals have lengths v2T and V3T. The ratio of the volume of the larger of the two polyhedra to the volume of the smaller m. where m and are relatively prime positive integers. Find m + n. parallelepiped is solid with six parallclogram faces such as the one shown below. Solution:
Let's try solve this problem step by step. The problem is about two noncongruent paral- lelepipeds where each face is rhombus with diagonals of lengths v 21 and VII. We need to find the ratio of their volumes (larger to smaller) and then compute m n where the ratio in simplest terms.
[...detailed analysis omitted...]
Therefore, the two possible rhombohedrons would have all angles equal arccos(5/200 or arccos( -5 5/26). Thus, these are two distinct rhombohedrons with the same edge length but different angles, hence noncongruent.
Let's s compute the volumes using Python:
Python Code
import sympy as sp
#
Def one the square root of 13 symbolically sp. sqrt
Define angles symbolically using arccos theta sp. acos (sp. Rational 26)) theta_prime sp. acos acp.ssppppp.......cc............ Rationat (-5, 26))
(sp..Riinnl
Define cos (theta) and cos (theta_prime) as symbolic fractions cos_ theta sp. Rational (5, 26)
cos cos_ tettthetttiimeeeeeeee sp. RatiRnaioalllll (-5, 26)
Compute volume for theta using the symbolic expression vol1, voli_expr expr s**3 sp. sp. sqrt(1 3*cos_ theta*#2 + A4cos_thetaxxe vol1 = sp
Compute volume for theta_prime using the symbolic expression vol2_expr s**3 sp. sqrt(11 3*cos_ thaas_tttt_ppiieett vol2 = sp. ppp.simllfyyovl22_epp
# Print the symbolic results orVollue 1:", vol1) print("Volura 2:", vo12)
37

---

Output
Volume 1: s (2) /2
Volume 2: 31 31*sqrt(C (2)
The two volumes are 6 and 31 Since 2 31.5 31, 63y2 larger than 31 Therefore, m and + 125. Therefore, the answer
2
2 62 Since 63 7 x and 62 x 31, they are coprime, So m 63, n 62.
125
Sample
Problem:
Let AABC be equilateral triangle with side length 55. Points D, E, and F lie on BC, C.A. and AB, respectively, with BD = 7, CE = 30, and AF = 40. Point P inside AABC has the property that
LAEP = 1BFP=2CDE
Find (ZAPP). Solution:
Let's try to solve this problem step by step. We have an equilateral triangle ABC with side length 55. Points D, E, F are on BC, CA, AB respectively, with BD=7, CE=30, AF=40. There's a point P inside the triangle such that the angles LAEP, LBFP, and LCDP are equal. We need find the square of the tangent of this angle. First, let's place the triangle ABC in coordinate system with A(0, 0), B(55, 0), and C( 55y3). ). Then we can calculate the coordinates of points E, and F:
Point D: D 103
2 ) PPint E: ( 25y3
Point F: F(40, 0)
detailede analysis omitted...]
For each angle 0,
write equations involving the slopes of the lines:
For LAEP:
y
20:3
Can(O)
25 x-
v3.
25
For LBFP:
(an y
x 40
3. For LCDP:
y Iv
2- 1o + v3
tan (0)
v3 Iv1
Let's solve this system using Python:
38

---

Python Code
import sympy sp
Define variables x, = sy.symbols ('x ('x y*)
# Coordinates of points
E_x sp.Rational (25, 2)
E_y = sp. Rat.iiooall (25, 2) sp.sqrt(3)
F_x sp. sp.Rational (40, 1)
F_y sp.Rational (0, 1)
D_x sp.Rational (103, 2)
D_y = sp. Rational (7, 2) * sp.sqrt(3)
# Slopes and tan(th)ta) expressions m_ sp.sqrt
m_EP (y E_y) / (x E_x)
tan tan_thentteeeeteeeee theta_E
Solve the system of equations solution sp. solve [eql, eq2], (x, y)) ("Soluttit for x and y:")
(solutioio))))
Output
Solutions for x and y: [(1525/38, (3+))r((3)338)
With these coordinates, we can compute t@n(
25./3
Lan(0) T328 40
5v3
Therefore: (0) = (5 3 2 75
Therefore, the answer is
75
Sample
Problem:
In AABC with side lengths AB = 13, BC = 14, and CA = 15, let M be the midpoint of BC. Let P be the point on the circumcircle of AABC such that M is on AP. There exists unique point Q on segment AM such that LPBQ = LPCQ. Then AQ can be written as Vn: where and are relatively prime positive integers. Find m Solution:
Let's solve this step by step. First, let's establish coordinate system: Place B (0,0)
Place C at (14.0) Then M (7,0) Using distances AB=13 and AC=15, we can find A at (5.12)
[..detailed analysis omitted...] First, let's find the circumcirele of triangle ABC:
39

---

Python Code
import sympy sp
Define symbols for circle's center (h, k) and radius r h, K, = sp. syp.olbols("h k r²)
# Points A(,12), B),,0), C(14,0)
Equations for the circle passing through each point
eq1 (0 h)*+2 + (0 x+*+2 r**2 B(0,0)
eq2 h)*+2 (12 k)*+2 r**2 A(),2)
eq3 = (14 ++++2 + (0 k)*+2 r**2 C(14,0)
# Subtract equations to eliminate r-2
eq21 sp. simplify eq1)
eq31 = ap simplify(eqq eq1)
# Solve the system of equations solution sp. sp.solve ( eq3i], (h, k)) h_value, k_value = solution[h], solution [k] print ( ("Center (h, k):", (h_value, k_value))
radius_edt _q eq1 subs ((h: h h_value, k: k_value}) radius_ solbuus sp. wwwveeeaaaiuuuu.eeaaaaaaaaaaaale r_value = [r_val for r_val in radius_solution if r_val > 0] print("Radion r:", r_value)
r)
Output
Center (h, k): (7, 33/8)
Radius r: [65/8]
[...detailed analysis omitted...]
Next, let's find point P by finding where line AM intersects the circumcircle:
40

---

Python Code
from sympy import symbols, solve, Rational
Define the variable
t symbols(?t))
# Define the parametric equations for and
x
12 12*t
# Define the equation of the circle circle_eq (x 7)++2
Solve for expanded_eq eq circle_eq ex.pand
solution_t solve(expanded_eg) t)
Calculate the coordinates for each solution t points = []
for t_solution in solution_t:
x_value subs (t, t_solution)
y_value = y subs t_solution) Seints.appents ((x_ value, y_value)
print ("Solutions for t:", solution_t) for idx, point in enumerate (points):
"Fioof"CCorr of point for t=sslutionn [idx]]][]x]]..__
Output 2
Solutions for t: [0, 197/148]
Coordinates of point for t=0: (x, y) (5, 12) Coordinates of point for t=197/148: (x, y) (567/74, -147/37)
[...detailed analysis omitted...] Now, let's find point using the condition that LPBQ LPCQ:
41

---

Python Code
from sympy import symbols, Eq, solve, sqrt, simplify, fraction from sympy import Rational
t = symbols (?t²)
# Coordinates of Q
Qx 5 2+t
Qy 12 12*t
# Coordinates of P
Px Rational (567, 74)
Py Rational (-147, 37)
Vectors BP. BQ, CP, CQ
BPx Px 0
BPy Py 0
BQx Qx 0
BQy Qy 0
CPx Px 14
CPy Py 0
CQx Qx 14
CQy = Qy 0
Compute dot products and magnitudes
dot_BP_EB BQ BPx BQx
BPy BQy
dot_ CPx CQx
CPy cqy
magnitude_Bt __
sqrt (BPx**2 + BPy*+2)
magnitude__. BQ sqrt (BQx**2 BQy**2)
magnitude___ sqrt (CPx**2 + CPy*+2)
magnitude_co CQ sqrt (CQx**2 CQy**2)
Set up and solve the equation
equation [g(dot_B__BQ/ (magnitude BP magnitude_BQ)
dot_CP_CO/ (magnitude__om magnitude_CQ)
solution solve (equation, t)
print ("ioouuttto for t:", solution)
Output 3
Solutions for [99/148, 197/148)
[...ettaieed analysis omitted...]
Finally, let's compute AQ and verify our answer:
42

---

Python Code
from math import ged
First compute AQ distance
Qx_val Qx subs (t, Rational (99, 148))
Qy_val Qy subs (t, Rational (99,14)) 148))
AQ_distance = cqrt((Qx_val)
5)++2 (Qy_val 12) +2)
simplified_AQ AQ_distance. simplify( print("AQ =", simplified_AA)
# Verify and are coprime 99
148
god_val ged (m, n) print ("GCD of 99 and 148:", ged_val)
Output 4
AQ s (37) /74
GCD of 99 and 148:
After manipulation, we find AQ integers.
99
where
99 and n = 148 are coprime positive
v148
Therefore, m 99 + 148 247
F.2
Test case of Prompt-Hlint-SFT-322
A case from AIME24
Problem:
Let B be the set of rectangular boxes with surface area 54 and volume 23. Let be the radius of the smallest sphere that can contain each of the rectangular boxes that are elements of B. The value of 2 can be written as P, where and are relatively prime positive integers. Find
p+ q. Solution:
The problem is about finding the smallest sphere that can contain any rectangular box from the set B, where each box has surface area of 54 and volume 23. The radius of this minimal sphere would be half the maximum space diagonal among all possible boxes.
For box with side lengths a, b, e, we have: Space diagonal
v + + c Surface
area: 2(ab + be + uc) = 54 3. Volume: abc- = 23
Therefore, 2 a t2² and we need maximize a² + b2 + c² subject the constraints. [...detailed analysis omitted...]
Let's approach this using Lagrange multipliers. Our Lagrangian is:
L a² + b2 + o² X(2(ab be ac) 54) p(abe 23)
Taking partial derivatives:
aL
2a - X(2b + 2c) - ubc = 0
dn
0L
2h X(2a + 2c) pacc
db
0L
2c X(2a + 26) 4ab
dc
43

---

[...detailed analysis omitted...] After analysis, we find that the maximum occurs when: a = 2 Let's verify this solution:
Python Code
from sympy import symbols, Eq, solve, sqrt from sympy import Rational
# Define variables
b 2
Rational (23,
Check surface area surface_aree area 2 * (a*b + b*c + a*c) print (f Surffcee area (eurface_area)"")
Check volume volume * b (f"Voumee (volume}")
Calculate space diagonal squared
aiaaonal_squared squared a**2 + b**2 c**2
print (p"ee diagonal squared (diagonal_squared))
Calculate squared
r_squared diagonal_squared
print (f"r squared fr_squared)")
Output
Surface area 54
Volume
Space diagonal squared 657/16 r squared 657/64
Therefore: Space diagonal squared
657
r
657
16
Since 657 = 3 73 and 64 20, these numbers coprime. Therefore, p 657 and q= 64.
The answer is + 657 + 64
721
A Case from AIME25
Problem:
Six points A, B, C, D, and lie straight line that order. Suppose that a point on the line and that AC 26, BD 22, CE 31, DF 33, AF 73, CG 40, and DG = 30. Find the area of ABGE.
Solution:
Let's solve this step by step:
First, let's assign coordinates to the points on the line, placing origin: A: (origin)
AF 73, so F: 73- AC 26, so C: 26 CE = 31, 1, so E: 26 31 = 57 DF = 33, so D: 73 33 400 BD 22, so B: 40-40011 00-22-2 18 [...detailed analysis omitted....
2) Next, let's find coordinates of using distances CG and DG 30: From C(26,0):
(cc 26) + 2 = 1600 ...(1) From D(40,0): (xg 40) + n/ 900 (2)
44

---

Let's verify this using Python:
Python Code
from sympy import symbols, Eq, solve
x_G, y_G
symbols ('xbo y_G')
Equations based on CG 40 and DG 30 eq1 Eq (x_G 26) +22 + y y_ 1600) eq2 Eq((x_G 40) y y_ 900)
Subtract equations find x_G subtracted_et eq = [x[ 26) ++2 (x_G *+, 700)
x_G_val solve (subtracted_egg x_G) [o]
Substitute x_G into eq2 to find y_G y_G_val = solve subs (x_,, x_G_val), y_G) [o]
print (f"a coordinates:
([x_G_val), (y_G_val))")
Output
coordinates: (58, 24)
Therefore, the coordinates of the triangle vertices are: B(18,0) - 3) Calculate the area using the shoelace formula: Area = ] 21 (y2 ya) + 22 21) +
T3(311 y2)l
Where: (I1, vi) = (18,0) 0) (x2, 1/2) = (58, 24) (I3, Us) (57, 0)
Substituting:
Area
[18(24 - 0) + 58(0 0) + 57 24)
2
1
432 + 1368
2
1
2 9366
468
Therefore, the area of triangle BGE is 468
F.3
Test case of Hint- Hiint-Engineering-EEE
Case form AIME24
Problem:
Let B be the set of rectangular boxes with surface area 54 and volume 23. Let be the radius of the smallest sphere that can contain each of the rectangular boxes that are elements of B. The value of 2 can be written f. where p and relatively prime positive integers. Find
+ g. Solution:
The radius of the smallest sphere that can contain any rectangular box must be half of the maximum space diagonal. For a box with dimensions c, have;
Space diagonal Va² 62 d 2. Surface area: 2(ab be ac) 54 3. Volume:
abc = 23
Our goal to maximize a² b2 + subject to these constraints, then p max(a +63+
45

---

Let's try assuming 54 + a² + 2ac 27 Volume becomes: a²c 23 + 23
to simplify the problem: Surface area becomes: 2(a² + 2ac)
[...detailed analysis omitted...]
Let's solve the resulting cubic equation:
Python Code
from sympy import symbols, solve, Eq
symbols (*a²)
equation [q 27+a 46, 0) solutions solve (equation, a) print ("aooluttonss:"))
for sol in solutions:
print (sol. simplify())
Output
Solutions:
-1 2*sqrt (6)
-2+sqrt(6)
The only valid solution is 2, giving b c 5.75
Let's verify this is indeed the maximum using numerical optimization:
Python Code 2
from scipy optimize import minimize
def objective(x):
a, b, = return (a*+2 + b**2 + c*+2)
def constraint1xx): a, b, c =x return (*(a + b*c + a*c) 54
def constraint2xx): a, b, =x
return a#b*c 23
x0 [2, 2, 5.75]
cons = [['type': 'eq', 'fun': constraint1); ('type': 'eq', 'fun'; constraint2]] bounds [(0.0001, None), (0.0001, None), (0.0001, None)]
result minimize x0, m bounds=bounds, constraints=cons)
print ("Optimization result:", result.sue success)) ('intti"mmm b, c:", result.x)
print ("Maximum a ba + ca:", -result.
46

---

Output 2
Optimization result: True Optimal a, b, c: [2. 2. 5.75]
Maximum a ba c²: 41 0625
Therefore: 1. Maximum space diagonal squared 2. x2 3 3 73 and 64 2 coprime Thus, p 657 and g 64
/4
3. 657
16
64
The answer is p + 657 + 64 721
Case from AIME25
Problem:
Six points A, B, C, D, E and F lie in straight line in that order. Suppose that G is point not on the line and that AC = 26, BD = 22, CE = 31, DF 33, AF = 73, CG 40, and DG = 30. Find the of AGG..
Solution:
First, let's assign coordinates to all points the line, placing A origin:
Given distances: AC' = 26 BD 22 CE 31 DF = 33 AF = 73
This allows determine coordinates: (0, 0) B: (18, 0) C: (26, 0) D: (40, 0) E: (57,0) F: (73, 0)
[...dttaieed analysis omitted...]
To find G's coordinates, we use distances CG 40 and DG 30: From C (26, 0): (ra (==)-6))2- v 1600 ...(2)
1600...(1) ...(()) 2) From D (40, 0): (zc -0) v2 900
Let's solve this system using Python:
Python Code
from sympy import symbols, Eq, solve
x_G, y_G symbols ('x_G y_G')
Equations based on CG = 40 and DG 30
eq1 Eq ( (x_G 26) *22 y 2_ 1600) eq2 = Eq (x_G 40) #+2 + G_ 900)
Solve for x_G and y_G
subtracted_eg Eq (x_ **2 (x_G 40) 700)
x_G_val = solve (subtracted_eq, x_G) [0]
y_G_val solve (eq2. subs (x_G, x___val), y_G) [o]
print(f"C coordinates: G[1_)_val), (y_G_val))
Output
coordinates: (58, 24)
Therefore, we have all coordinates: B: (18, 0) G: (58, (57.0)
Let's calculate the area using the shoelace formula:
47

---

Python Code
Calculate using shoelace formula B_coord = (18, 0)
G_coord (58, 24)
E_coord = (57, 0)
area = 0.5 * abs
[_] [G_coord[1] [_oorr
G_coord [o] (E_coord[1] B_coord[il) +
E_coord [o] [B_coord[1] G_coord[il)
)
print (f"Area of triangle BGE: (area)")
Output 2
Area of triangle BGE: 468.0
Verification using Base x Height method: Base (BE) 57 18 39 Height 24 Area 24
468
2
Therefore, the area of triangle BGE is 468
G Strong-to-Weak Distillation Data
For the strong-to-weal distillation process, we constructed a comprehensive mathematical problem dataset of approximately 10,000 problems from three primary sources. The dataset comprises 800 problems from AIME (American Invitational Mathematics Examination) competitions prior 2024, 2,280 problems from the MATH [67] dataset's 's training (Level 5), and 7,000 problems sampled from the Numina-man math [68] collection. For the Numina-math sampling, we implemented several filtering criteria: problems containing figures were excluded, were proof-based problems, multiple-choice questions, and problems where answers appeared within the problem statements. After applying these filtering criteria, we randomly sampled 7, 7,000 problems ensure clean and standardized dataset suitable for our distillation training process.
H
Baselines
We compare our approaches against range of state-of-the-art mathematical reasoning models across different parameter scales:
SOTA Models. We benchmark against the most advanced large reasoning models currently avail- able: OpenAl's ol [5], QwQ-32B [69], and DeepSeek-RI R1 [7]. These models represent the frontier of mathematical reasoning capabilities and serve as an upper bound for performance comparison.
Frontier Models (32B). For the 32B parameter scale, we use DeepSeek-B 1-32B [7] as our founda- tion model since it serves as the starting point for many open-source models in this size range. We also compare against contemporary tool-integrated reasoning (TIR) models, including START-32B [21], STILL-3-TOOL-322 [17], and reinforcement learning for tool use.
[60], the latter being concurrent work focusing
Lightweight Models (1.5B). the lightweight category, we use DeepSeek-R1-1.5B [7] as our base model for the same reason as its 32B counterpart. We benchmark against DeepScaleR-11. Preview [65], the current state-of-the-art reinforcement learning model this scale, and ToRL.- 1.5B [58], concurrent work on tool-integrated reasoning with reinforcement learning.
48

---

Our selection of baselines spans different model sizes, training methodologies (SFT, RL, and RFT), and approaches to mathematical reasoning (with and without explicit tool use). This comprehen- sive comparison allows to evaluate the effectiveness of our Prompt-Hint and Hint-ennineeeing approaches relative to both established benchmarks and recent innovations in the field.
I Out-of-Distribution Generalization on Chemistry Problems
To rigorously evaluate the generalization capabilities of our CoRT framework, we conducted a challenging out-of-distribution (OOD) experiment domain far removed from mathematical training data: chemistry. We utilized a subset of problems from the GPQA benchmark [70], which requires deep domain knowledge and specialized computational tools.
This experiment serves as a stringent test for several reasons:
Domain Shift:
The problems involve reasoning about molecular structures and properties, a domain conceptually distinct from the algebra, geometry, and number theory problems used in training.
2. Tool Shift: Standard mathematical libraries like NumPy or SymPy inadequate for these tasks. The gold-standard tool for cheminformatics the RDKit library.
3. Zero-Shot Tool Discovery: Crucially, the RDKit library was never included in any of our training data or prompts. The model's ability to successfully solve these problems hinges on its capacity to autonomously identify the need for new tool, discover the correct library (RDKit), t), and learn to use its functions on the fly.
We compared our Hint Saan-EEggine Engineering-RFT a set of chemistry problems from GPQA. The evaluation focused on final answer accuracy, token efficiency, and, most importantly, the model's ability spontaneously utilize the unseen RDKit library.
model against the baseline DeepSeek-R1 -R---eekk-R3322 on
1.1 Results and Analysis
The results, summarized in Table 3, demonstrate that our CoRT framework imparts generalizable reasoning ability that successfully transfers across domains and tools.
Table 3: Performance on the OOD Chemistry (GPQA) Benchmark. Our model not only improves accuracy and efficiency but also spontaneously discovers and utilizes the unseen RDKit library, demonstrating true generalization.
Model
Accuracy
Avg. Tokens
RDKit Usage Rate
DeepSeek-R1- R1-22B (Baseline)
40.6%
5.947
0%
47.5%
4,220
81.3%
Improvement
+6.9 pts
-29.0%
+81.3 pts
Our key findings from this experiment are:
Robust Cross-Domain Performance: Our model achieved a 6.9 absolute point accuracy gain over the baseline in the unseen chemistry domain. This provides powerful evidence that CoRT teaches a fundamental problem-solving methodology rather than domain-specific memorization.
The most striking result is that our model spontaneously discovered and correctly utilized the RDKit library in 81.3% of cases, whereas the baseline model completely failed to do so. This showcases a sophisticated, emergent ability to adapt to new problem contexts, feat impossible for systems reliant on fixed, pre-defined API sets.
Spontaneous Tool Discovery and Utilization:
Sustained Efficiency Gains: Consistent with our in-domain results, the model also demon- strated significant efficiency improvements, using 29.0 % fewer tokens on average. This indicates that the learned behavior of prioritizing computation via code is generalizable strategy.
49

---

In conclusion, this OOD experiment provides strong evidence that the reasoning patterns instilled by our Hint-Engineering and post-training pipeline are brittle but foster a flexible and adaptive problem-solving capability, which is a critical step towards more general-purposs Al reasoning systems.
Broader Impacts
Our work enhancing code-integrain reasoning capabilities in LRMs offers several potential benefits to society. The improved efficiency and accuracy in mathematical reasoning could enhance educational applications and academic research, while reduced token usage contributes to environ- mental sustainability through decreased computational resource consumption. The integration of precise computational tools with natural language reasoning may also improve the reliability of AI systems in applications requiring accurate calculations.
However, as with any advancement in Al capabilities, potential risks should be considered. While our models are trained on public mathematical datasets and intended for educational and research purposes, the enhanced reasoning capabilities could potentially be misused if not properly secured. We recommend implementing appropriate access controls and maintaining transparency about the system's s limitations and intended use cases.
We encourage future work to continue exploring responsible deployment strategies that maximize beneficial impacts while minimizing potential risks.
K
Limitation
Our work presents a novel framework for enhancing code-integrated reasoning LRMs through hint- engineering and efficient post-training strategies. While we demonstrate significant improvements in both performance and efficiency, particularly in mathematical reasoning tasks, several directions remain for future exploration. Due 1.5B), the computational requirements for training and inference may affect broader accessibility. While we utilize publicly available datasets and models, ensuring work maintains transparency and reproducibility, the rapid evolution of LRM capabilities suggests opportunities for continued enhancement new architectures emerge. Furthermore, while current work emphasizes the integration of code interpreters within long-form reasoning, there remain opportunities to investigate additional synergies between external tools and language models' inherent reasoning capabilities. As the field of large reasoning models continues to advance rapidly, we anticipate further developments in optimizing these interactions.
of SeepSeee-RR-Distillle
models (32B and
50


Reviews:
[{"review_id": "0SUunXBlD0", "reviewer": "Reviewer_qM62", "summary": "This paper proposes a solution to a technical challenge: how to combine a code interpreter for Large Reasoning Models. The authors identify inefficiencies in direct use of LRMs with CIs (such as excessive token usage and manual verification of code output), and propose a multi-stage training process based on hint-engineering to address these issues. Training methods use supervised fine-tuning, rejection fine-tuning, and reinforcement learning on both large (32B) and smaller (1.5B) models. Experiments across five challenging math benchmarks show the proposed method improves accuracy and reduces token use, with empirical analysis supporting the main claims.", "questions": "(1) Baselines: The key difference between this approach and traditional tool-augmented LLMs is not obvious. In the examples provided in the main part and the appendix, the Python code being generated is relatively simple—for instance, just calling packages like sympy or computing the gcd. In these cases, it seems entirely feasible to build a toolLLM to retrieve related functions and call them. We do not need to write code and execute it. Can you explain more why a toolLLM with function calling is not enough and why it is needed to write a \"code wrapper\" for the usage of some API? Maybe such a simple function calling with all APIs available in a famous math package should be conducted using the same training process as a baseline.\n\n(2) The reason why the models rely heavily on text reasoning instead of code reasoning is potentially because the base model is R1-style.  Is it possible to start from the Instruct model and then just run RL based on it? The ideal solution for efficient CI-augmented reasoning is that the LLM generates computing programs first and then runs the text reasoning. The existence of verification and the existence of heavy text reasoning can be due to the instructional data and base reasoning model.\n\n(3) Can we simply prompt LLMs to skip generating or executing verification code altogether, focusing only on code that performs the actual computation? This might further improve overall efficiency and response time.\n\n(4) Small LLMs are designed for on-device use, but enabling them to run and execute code often requires Docker or similar environments. This adds extra overhead, potentially making the deployment heavier than intended for lightweight, on-device applications.\n\n(5) There is also a formatting error in Line 119—the code block should start with python, not python''.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nClarity of Writing: The paper clearly presents its main ideas and objectives. The problems addressed are well-motivated and easy to follow.\n\nComprehensive Experiments: The authors provide thorough experimental results and analyses across several well-known benchmarks, including AIME24, AIME25, AMC23, MATH500, and Olympiad datasets. This demonstrates the effectiveness of prompt-hint and hint-engineering training.\n\nData Quality over Quantity: The work shows that small, carefully curated post-training datasets can yield substantial improvements. This finding aligns well with ongoing discussions in the alignment and LLM training communities.\n\nWeakness: \n\nLimited Scale of Hint-Engineering Data: The manual annotation for hint-engineering is limited to just 30 problems for the 32B models. While this produces promising results, it raises concerns about scalability and generalization. The main text should discuss in more detail (not just in the appendix) how performance is expected to transfer to less curated or broader settings. Many recent works question whether such small data is even needed for this type of training [1].\n\nLack of Statistical Significance Testing: The paper does not provide significance tests (e.g., p-values) to demonstrate that the improvements are statistically meaningful. Adding such analyses would strengthen the claims.\n\nInference Time Analysis Missing: Although the paper claims better token efficiency, the actual inference time per task is not reported. Since real-world applications care about wall-clock inference time when given a problem, more evidence is needed to show that hint-engineering is actually faster than prompt-hint methods. My main concern is that maybe token efficiency is not the bottleneck of inference speed. Execution time can be the inference speed bottleneck, potentially making reducing the token number not meaningful. \n\nAlternative baselines for hint-engineering: Instead of complicated RL with CI for hint-engineering, could simple length-based penalty rewards achieve similar token efficiency without the complexity of code-integrated RL? An analysis or discussion on this would be valuable.", "was_revised": true, "final_justification": "My concern is well answered and I choose to keep my score.", "rebuttal": "We sincerely thank the reviewer for their insightful and constructive feedback. We are particularly encouraged that the reviewer recognized and appreciated the **clarity of our writing**, the **comprehensiveness of our experiments**, and the core finding that **high-quality data is more impactful than sheer quantity**. These acknowledgments affirm the central contributions of our work.\n\nWe are grateful for the opportunity to address the remaining concerns and further clarify our contributions:\n\n---\n\n**W1: Limited Scale of Hint-Engineering Data, Scalability, and Necessity of Small Data**\n\nWe would like to clarify that achieving significant improvements with a small, manually curated dataset is not a limitation, but rather **a key strength and central finding of our work**, demonstrating the high sample efficiency of our CoRT framework.\n\n*   **A High-Quality Seed for a Scalable Pipeline:** The 30-sample dataset serves as a strategic \"seed\" to efficiently elicit the base model's latent capability to interact effectively with a Code Interpreter. The manual effort is minimal (<2 hours), far more efficient than the extensive annotation required in works like InstructGPT. Our approach then scales through **automated data synthesis via Rejection Fine-Tuning (RFT)**, where the model generates large volumes of candidate solutions which are then filtered for quality, creating a virtuous self-improvement cycle.\n\n*   **Necessity of Seed Data:** To demonstrate that this small-scale data is essential, we conducted zero-shot experiments on the **DeepSeek-R1-32B**. The results below show that prompting alone cannot achieve the same fundamental shift in reasoning behavior that our method enables.\n\n| **Model** | **Dataset** | **Avg. Accuracy** | **Avg. Tokens** | **Avg. Code Blocks** |\n| :--- | :--- | :---: | :---: | :---: |\n| TIR Prompt-hint on DeepSeek-R1-32B | AIME24 | 67.9% | 9,382 | 0.89 |\n| | AIME25 | 50.4% | 11,902 | 0.68 |\n| Hint-Engineering-RFT-32B | AIME24 | **76.3%** | **7,260** `(-22.6%)` | **2.10** |\n| | AIME25 | **65.2%** | **8,532** `(-28.3%)` | **2.64** |\n\nOur **Hint-Engineering** model achieves significantly higher accuracy while using 23-28% fewer tokens. Critically, the \"Avg. Code Blocks\" metric shows a dramatic behavioral shift: our model uses **2-4 times more code blocks**, indicating a learned, efficient strategy of proactive computation. This aligns with recent findings [1] on the power of small, high-quality data.\n\n[1] \"The Challenge of Teaching Reasoning to LLMs Without RL or Distillation.\"\n\n---\n\n**W1: Generalization to Broader Settings**\n\nTo directly address transferability, we conducted a comprehensive **out-of-distribution (OOD) evaluation** on **Chemistry problems from GPQA**. This stringent test assesses if the model can generalize its reasoning and tool-use skills to a completely new domain, including discovering and using a domain-specific library (`RDKit`) that was **never seen during training**.\n\n| **Model** | **Accuracy** | **Avg. Tokens** | **RDKit Usage Rate** |\n| :--- | :---: | :---: | :---: |\n| DeepSeek-R1-32B | 40.6% | 5,947 | 0% |\n| Hint-Engineering RFT-32B | **47.5%** | **4,220** | **81.3%** |\n| Improvement | **+6.9 pts** | **-29.0%** | **+81.3 pts** |\n\nThe results show robust cross-domain transfer: our model achieves a **6.9-point accuracy gain**, uses **29% fewer tokens**, and spontaneously discovers and utilizes the novel `RDKit` tool in **81.3%** of cases. This provides strong evidence that our framework teaches generalizable reasoning capabilities, not domain-specific memorization.\n\n---\n\n**W2: Lack of Statistical Significance Testing**\n\nWe thank the reviewer for this constructive suggestion.\n\nWe must clarify that our results are **statistically meaningful** as each reported accuracy is a stable estimate averaged over multiple stochastic samples (4 or 16 per problem), not a single-shot result, thus significantly reducing the impact of randomness as described in caption of table 1. Then we have now conducted pairwise **Wilcoxon signed-rank tests** by pooling results across all problems from the benchmarks to assess statistical significance.\n\n| Model Scale | Model A | Model B | Acc. Improv. | p-value | Token Reduction | p-value |\n|:---:|:---|:---|:---:|:---:|:---:|:---:|\n| 32B | Hint-Eng-RFT | R1-distill-32B | +3.80% | 0.055 | 36.3% | < 0.001 |\n| 1.5B | Hint-Eng-RL | R1-distill-1.5B | +7.41% | 0.013 | 59.1% | < 0.001 |\n\nThe analysis confirms that our efficiency gains are **highly significant (p < 0.001)**. Accuracy improvements are also statistically significant for the 1.5B model (p=0.013) and approach significance for the 32B model (p=0.055).\n\n---\n\n**W3: Inference Time Analysis Missing**\n\nWe thank the reviewer for this critical question. To confirm that token efficiency translates to real-world speed, we conducted a comprehensive latency analysis on AIME24, decomposing wall-clock time into **Generation Time** and **Execution Time**.\n\n| Model / Method | Avg. Wall-Clock Time (s) | Avg. Generation Time (s) | Avg. Execution Time (s) | Generation Time as % of Total |\n| :--- | :---: | :---: | :---: | :---: |\n| Prompt-Hint-SFT-32B | 187.35 ± 107.74 | 182.39 | 4.96 | 97.3% |\n| Hint-Engineering-RFT-32B | 101.17 ± 98.03 | 97.84 | 3.32 | 96.7% |\n| Improvement | **46.0% Faster** | **46.3% Faster** | **33.1% Faster** | - |\n\nThe results are definitive:\n1.  **Generation time is the bottleneck**, accounting for over 96% of total latency.\n2.  Our method achieves a **46.0% reduction in total wall-clock time**, a substantial real-world speedup.\nThis confirms that token efficiency is the primary driver of performance gains in this setting.\n\n---\n\n**W4 & Q1: Alternative Baselines**\n\n*   **Length-based Penalty:** We agree this is an intuitive baseline. However, a simple length penalty is a \"black box\" method that cannot distinguish between **efficient conciseness** and **logical incompleteness**. Our experiments show that simply finetuning on the shortest responses (`prompt-hint-Shortest`) leads to worse performance than random sampling, while our `Hint-engineering` approach achieves the highest performance with the shortest generation length, demonstrating it learns true efficiency.\n\n| Model Size | Method | Performance on AIME24 & 25 | Avg. Generation Length |\n| :--- | :--- | :--- | :--- |\n| **1.5B** | prompt-hint-Shortest | 25.6 | 18290 |\n| | Hint-engineering | 28.8 | 16263 |\n| **7B** | prompt-hint-Shortest | 40.6 | 14206 |\n| | Hint-engineering | 42.2 | 11043 |\n| **14B** | prompt-hint-Shortest | 63.2 | 11770 |\n| | Hint-engineering | 65.4 | 8702 |\n\n*   **ToolLLM with Function Calling:** We agree that function-calling is effective for simple tasks, but argue that for complex mathematical reasoning, **code generation offers indispensable advantages in flexibility, expressiveness, and emergent problem-solving**.\n\n    *   **Flexibility & Composability:** Pre-defined APIs are rigid. They cannot handle novel combinations of functions or tasks requiring intermediate logic (e.g., loops, conditionals). In contrast, code generation empowers the model to **dynamically orchestrate tools** using loops, variables, and conditional logic, acting as a powerful \"computational glue\" for complex, multi-step reasoning. This is essential for problems like iterative approximation that have no single, pre-defined tool.\n    *   **Expressiveness & Generalization:** An API toolkit is inevitably incomplete. It cannot cover the \"long tail\" of thousands of functions in libraries like `SymPy`. Code generation provides universal access to the entire library ecosystem. Our **Out-of-Distribution (OOD) experiment** provides the strongest evidence: the model **spontaneously discovered and used the unseen `RDKit` library** to solve chemistry problems, a feat impossible for a system with a fixed API set.\n\n---\n\n**Q2: Impact of R1-style Base Model**\n\nOur goal was to augment a SOTA LRM, leveraging its sophisticated reasoning patterns (e.g., self-reflection, hypothesis testing) and combining them with computational precision. This hybrid approach, also seen in proprietary models like OpenAI's o3, is a key research gap in the open community. Our work aims to fill this gap.\nTo validate this design choice, we compared our LRM-based models with a strong code-centric baseline.\n\n| **Model (1.5B scale)** | **Base Model Style** | **Avg. Accuracy** |\n| :--- | :--- | :---: |\n| Qwen-2.5-Math-1.5B-RL | Short-CoT, Code-Centric | 47.3% |\n| Prompt-Hint-1.5B-RL | LRM (R1-style), Hybrid | **58.3%** |\n| Hint-Engineering-1.5B-RL | LRM (R1-style), Hybrid | **56.4%** |\n\nOur LRM-based models substantially outperform the code-centric baseline by **~10 absolute percentage points**. This suggests that the \"heavy text reasoning\" from the LRM base provides strategic advantages like problem structuring, strategic tool deployment, and self-verification, which are crucial for top-tier performance on complex tasks.\n\n---\n\n**Q3: Prompting to Skip Verification**\n\nThis is an astute suggestion. We experimented on D_{Hint-engineering-SFT} and found that explicitly prompting the model to skip verification does reduce token usage, but our **Hint-Engineering** method is far more effective.\n\n| Method | Average Response Length |\n| :--- | :---: |\n| Prompt-Hint | 7867 |\n| Prompt-Hint (w/ skip verification prompt) | 6978 (`-11.3%`) |\n| Hint-Engineering | **4620 (`-41%`)** |\n\n---\n\n**Q4: Overhead of Code Execution for Small LLMs**\n\nWe clarify that our work focuses on the fundamental challenge of integrating systematic reasoning with natural language reasoning, a paradigm applicable across model scales. We note that commercial products like ChatGPT successfully integrate code execution without prohibitive overhead, and we believe deployment challenges are addressable through engineering advances.\n\n---\n\n**Q5: Formatting in Line 119**\n\n'''python is a special token that starts with the code block, not a typo, we will explain it further."}, {"review_id": "c2VuHcgx4X", "reviewer": "Reviewer_xQk3", "summary": "This paper introduces CoRT, a framework designed to teach LRMs how to effectively use a Code Interpreter (CI) for mathematical reasoning. The authors identify key inefficiencies in existing models, namely \"delayed code computation\" and \"code result distrust.\" To address this, they propose \"Hint-Engineering,\" a method of manually inserting strategic hints into reasoning traces to guide the model toward more efficient use of the CI. Based on a small set of 30 manually crafted examples, they fine-tune models of various sizes and demonstrate absolute accuracy improvements and significant reductions in token consumption (up to 50%) on several mathematical benchmarks.", "questions": "1. The abstract and main results table highlight specific absolute performance gains of 4% and 8% for your models. Given that the paper acknowledges these results are from single experimental runs due to computational costs, how can you ensure these improvements are statistically significant and not a result of random factors like initialization? \n\n\n\n2. Your core contribution, \"Hint-Engineering,\" relies on manually creating 30 high-quality samples and manually inserting hints, because you found that attempts to automate this process were \"insufficiently precise\". Can you elaborate on the specific failure modes of your automation attempts and explain how this framework could be scaled to new or larger problem domains without requiring prohibitive amounts of expert manual labor?", "limitations": "See the questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Minor Concerns\n- Overstated Novelty: The paper frames CoRT as a new framework. However, the constituent parts—SFT, RFT, strong-to-weak distillation, and RL on curated data—are all established techniques. The primary novelty lies in the manual \"Hint-Engineering\" data curation strategy, whose limitations are discussed below.\n\n\n- Token Efficiency Framing: The paper highlights a 30-50% reduction in token usage. While this is a valuable practical outcome, it is a direct and intended result of the hint design, which explicitly discourages textual reasoning in favor of code execution. This is more of a behavioral shift guided by data curation than a fundamental improvement in the model's innate reasoning efficiency\n\n## Major Concerns\n\n- The most significant flaw in this paper is the absence of any statistical validation for the empirical results. The authors explicitly state in the checklist (Question 7) that they did not report error bars or conduct multiple runs due to the high computational cost of LLM inference. While the resource constraints are understandable, a single experimental run is insufficient to conclude that the observed gains are not the result of random factors such as initialization seed or other sources of noise. This lack of rigor undermines the paper's central claims, such as the 4% and 8% absolute improvements cited in the abstract. For instance, the average accuracy difference between the main proposed model, Hint-Engineering-RFT-32B (81.3%), and the Prompt-Hint-SFT-32B model (81.8%) is small and could easily fall within a standard margin of error.\n\n- Limited Scalability and Generalizability of \"Hint-Engineering\": The core methodological contribution, \"Hint-Engineering,\" is described as a process of manually creating 30 high-quality samples and manually inserting hints. The authors concede that their attempts to automate hint placement were \"insufficiently precise\". This heavy reliance on manual, expert-driven intervention raises serious questions about the scalability and generalizability of the proposed framework. The contribution appears to be less of a novel, scalable training method and more of a highly specific, labor-intensive data curation effort. It is unclear how this approach would be applied to new domains or larger, more diverse datasets without a similar, prohibitive level of manual effort. The paper fails to provide a path toward automating this critical step, which is a significant limitation.", "was_revised": true, "final_justification": "The authors have addressed my concerns in the rebuttal.", "rebuttal": "We sincerely thank the reviewer for their detailed and insightful feedback. We are encouraged that the reviewer recognizes the value of our work, particularly the significant practical outcome of achieving a **30-50% reduction in token consumption** while improving accuracy, and acknowledges that our primary novelty lies in the **\"Hint-Engineering\" data curation strategy**. We appreciate the opportunity to clarify our contributions and address the concerns raised.\n\n---\n\n**1. On the Novelty of the CoRT Framework**\n\nWe acknowledge that the individual techniques—SFT, RFT, strong-to-weak distillation, and RL—are well-established. Our novelty lies not in inventing these components, but in **systematically combining them into a data-centric framework specifically engineered for the unique challenges of code-integrated reasoning.**\n\n*   **Addressing a Core Weakness:** Our work tackles a fundamental limitation of LRMs: their struggle with precise numerical calculations and complex, multi-step logic. By integrating a Code Interpreter (CI), we augment the model's internal reasoning with external computational power.\n*   **Unique Challenges of Tool-Integrated Reasoning:** This interactive, tool-augmented reasoning is fundamentally different from tasks that rely solely on a model's internal knowledge. It introduces new challenges, such as deciding *when* to use a tool, *how* to formulate the query, and *how* to interpret the feedback.\n*   **Principled Data-Centric Approach:** Through systematic data curation, we identified specific failure modes—such as \"delayed code computation\" and \"code result distrust\"—and developed targeted data patterns to address them. This principled approach to designing data for tool-integrated reasoning provides actionable insights for this emerging domain.\n\nOur contribution is demonstrating *how* to effectively orchestrate existing techniques to teach a model a new, more efficient reasoning paradigm, a critical step toward building more capable and practical AI systems.\n\n---\n\n**2. On the Framing of Token Efficiency and the Necessity of Data**\n\nWe agree with the reviewer that the token reduction is a direct and intended result of our hint design. However, we respectfully argue that this guided \"behavioral shift\" is a significant contribution. **Achieving this shift is non-trivial, as simply instructing a model to \"use code more\" is often ineffective.**\n\nTo substantiate this, we conducted an experiment demonstrating that our small-scale, high-quality data is **essential** for teaching this superior reasoning paradigm. We compared our fine-tuned `Hint-Engineering-RFT-32B` model against the `DeepSeek-R1-32B` model using only zero-shot prompting (without any fine-tuning).\n\n| **Model / Method** | **Dataset** | **Avg. Accuracy** | **Avg. Tokens** | **Avg. Code Blocks per Response** |\n| :--- | :--- | :---: | :---: | :---: |\n| TIR Prompting on DeepSeek-R1-32B | AIME24 | 67.9% | 9,382 | 0.89 |\n| | AIME25 | 50.4% | 11,902 | 0.68 |\n| Hint-Eng-RFT-32B | AIME24 | **76.3%** | **7,260** `(-22.6%)` | **2.10** |\n| | AIME25 | **65.2%** | **8,532** `(-28.3%)` | **2.64** |\n\n**Key Findings:** Our method significantly outperforms the zero-shot baseline in accuracy (**up to +14.8%**) and token efficiency (**-23-28%**). The model learns a new reasoning strategy, increasing code usage by **2-4x** for proactive computation. This demonstrates that fine-tuning on our data instills a superior, more efficient behavior unattainable through prompting alone.\n\n---\n\n**3. On Statistical Validation for Performance Gains (Major 1 & Q1)**\n\nWe thank the reviewer for highlighting this critical point and apologize if our initial presentation was unclear. To directly address your question about ensuring the **4% and 8% absolute improvements** are statistically significant, we provide the following clarifications.\n\n*   **Multiple Runs Are Incorporated:** Our experimental setup already incorporates multiple runs for statistical validity, following best practices of DeepSeek-R1. As stated in the caption of Table 1 of the paper, results are indeed averages:\n    *   **AIME24, AIME25, and AMC23:** Averaged over **16 samples** per problem.\n    *   **MATH500 and Olympiad:** Averaged over **4 samples** per problem.\n\n*   **Statistical Significance Testing:** To further validate our claims, we conducted **pairwise Wilcoxon signed-rank tests** on a per-problem basis by pooling results across benchmarks. This provides a robust micro-average statistical analysis.\n\n| Model Scale | Model A (Ours) | Model B (Baseline) | Acc. Improv. | p-value (Acc) | Token Reduction | p-value (Tokens) |\n|:---:|:---|:---|:---:|:---:|:---:|:---:|\n| 32B | Hint-Eng-RFT | DeepSeek-R1 | +3.80% | 0.055 | 36.3% | < 0.001 |\n| 1.5B | Hint-Eng-RL | DeepSeek-R1 | +7.41% | 0.013 | 59.1% | < 0.001 |\n\n**This analysis confirms:** The accuracy gains are statistically significant for the 1.5B model (**+7.41%, p=0.013**) and show a strong positive trend for the 32B model (**+3.80%, p=0.055**). Token reduction is highly significant (**p < 0.001**) for both, providing robust empirical support for our claims.\n\n---\n\n**4. On Scalability and Generalizability of \"Hint-Engineering\" (Major 2 & Q2)**\n\nWe appreciate the reviewer's concern. To address your question about scalability, we clarify (1) why manual curation was initially necessary and (2) how our framework is designed to be scalable and generalizable.\n\n**Why Manual Curation? Elaboration on Automation Failures**\n\nOur initial attempts to fully automate hint insertion using powerful LLM judges failed due to a lack of precision. On a test set of 100 AIME problems, we identified two primary failure modes:\n\n| Failure Mode | Description | Example Failure | Occurrence |\n| :--- | :--- | :--- | :---: |\n| **1. Distinguishing Planning from Inefficient Calculation** | The judge struggled to differentiate between necessary high-level planning and low-level calculations that should be offloaded to the CI. | It would insert a \"use code\" hint too early, disrupting planning, or too late, after token waste had already occurred. | 74/100 |\n| **2. Distinguishing Logical Verification from Numerical Distrust** | The judge couldn't distinguish between a valid check of code logic versus unnecessarily re-calculating a trusted CI result. | It would incorrectly suppress valid self-correction on code logic by inserting a \"don't doubt python\" hint at the wrong moment. | 19/100 |\n\nThese subtle but critical distinctions led us to use a small, high-precision, manually-annotated seed dataset to bootstrap the process effectively.\n\n**How Our Framework Scales and Generalizes**\n\nThe manual step is a highly efficient enabler, not a bottleneck. We argue that achieving significant improvements from a small, manually curated dataset is not a limitation, but rather a **key strength and central finding of our work**, demonstrating the high sample efficiency of our CoRT framework.\n\n1.  **High Sample Efficiency of a \"Seed\" Dataset:** The 30-sample dataset serves as a high-quality \"seed\" for our scalable pipeline. These manually annotated samples are not the final training set, but a strategic intervention designed to provide focused demonstrations of an efficient reasoning paradigm (i.e., using code for computation, not just verification). This process efficiently \"unlocks\" the model's latent capabilities. Importantly, this manual curation is **not a scalability bottleneck**—the human effort required is less than **2 hours**, which is significantly more efficient than the extensive human annotation required in other domains like InstructGPT.\n\n2.  **Scalability via \"Seed-then-Scale\" Pipeline:** Once the model grasps these core concepts from the seed data, we employ a scalable, self-improving synthesis pipeline. As described in Lines 129-135 of our paper, the SFT model generates large volumes of candidate solutions. These are then **automatically filtered** for Rejection Fine-Tuning (RFT) based on final answer correctness and the absence of inefficient reasoning patterns. This creates a virtuous cycle that scales our dataset to 830 examples with minimal further manual effort, directly addressing scalability concerns.\n\n3.  **Generalizability via Out-of-Distribution (OOD) Evaluation:** To directly address transferability, we conducted comprehensive **out-of-distribution (OOD) evaluation** on **Chemistry problems from the GPQA benchmark**. This experiment, conducted in a domain **far removed from our mathematical training data**, provides compelling evidence that both performance improvements and reasoning behaviors generalize to broader settings.\n\n    **Our OOD evaluation presents a stringent test of generalization.** Advanced chemistry problems require analyzing molecular structures and properties—tasks for which standard Python libraries like `numpy` or `sympy` are inadequate. The gold-standard tool for such analysis is `RDKit`, a powerful open-source cheminformatics library. **Crucially, `RDKit` was never included in any of our training data.** The model's ability to spontaneously discover and effectively use this domain-specific library serves as a rigorous test of its generalization capabilities.\n\n| **Model** | **Accuracy** | **Avg. Tokens** | **RDKit Usage Rate** |\n| :--- | :---: | :---: | :---: |\n| DeepSeek-R1-32B (Base) | 40.6% | 5,947 | 0% |\n| Hint-Eng-RFT-32B (Ours) | 47.5% | 4,220 | 81.3% |\n| Improvement | **+6.9 pts** | **-29.0%** | **+81.3 pts** |\n\n**Key OOD Findings:** Our model achieves a **+6.9 point** accuracy gain in the unseen chemistry domain. Crucially, it **spontaneously discovers and uses the correct, domain-specific `RDKit` library (81.3% usage)**, demonstrating true generalization rather than memorization.\n\n---\n\nThank you again for your valuable and encouraging feedback. Your comments have helped us significantly strengthen the paper's rigor and clarity. Please let us know if we can provide any further information."}, {"review_id": "7CbAKKBYqR", "reviewer": "Reviewer_mxrG", "summary": "This paper proposes CoRT, a post-training framework for teaching LRMs to effectively integrate code interpreters for improving the model's ability in complex mathematical reasoning. The method introduces prompt-hint and hint-engineering strategies to synthesize high-quality code-integrated reasoning data, followed by supervised fine-tuning, rejection fine-tuning, and reinforcement learning. Experiments on multiple math benchmarks demonstrate the effectiveness of CoRT in improving both accuracy and efficiency in math reasoning.", "questions": "1. Will the proposed method cause the fine-tuned model to overuse the Code Interpreter, even when it is not needed?\n2. What are the code triggering rates of the fine-tuned models?\n3. In the abstract and introduction, the author mentioned that they manually created 30 high-quality examples to post-train models. But in line 134, it mentions that 830 examples are created to obtain the Hint-Engineering-RFT-32B model. Is it a typo here?\n4. The proposed hint-engineering method encourages models to avoid reflecting on code execution results in order to reduce redundant verification and improve token efficiency. However, in cases where the model generates incorrect or semantically flawed code, would this suppression of reflection hinder the model's ability to detect and correct its own code errors?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper focuses on an important and timely topic, which is how to effectively integrate external tools like Code Interpreters into LRM to improve mathematical reasoning.\n2. In general, the writing of the paper is clear, well-structured and easy to follow.\n3. The proposed method is simple yet effective, using well-designed prompt strategies and fine-tuning techniques to enhance both accuracy and efficiency.\n\nWeakness:\n1. The paper does not provide comparisons with state-of-the-art tool-using models such as o3 and o4-mini. While these models are mentioned as motivation, it would strengthen the empirical analysis to include their performance as a reference.\n2. While the paper emphasizes the goal of teaching LLMs when and how to use tools, the proposed method mainly relies on prompt-based heuristics and fine-tuning to induce tool usage. It lacks an explicit mechanism for learning the decision of whether to invoke the external tool, such as a gating function or planner module. As a result, the model may not develop a genuine understanding of tool invocation timing, limiting its flexibility and generalization to unseen scenarios.\n3. There is a typo in line 267 \"probelms\"-->\"problems\".", "was_revised": true, "final_justification": "I recognize the paper’s main contribution and the strength of its comprehensive experiments. Since the authors have adequately addressed the main concerns raised in the initial review and discussion, I maintain my positive score.", "rebuttal": "We sincerely thank the reviewer for their insightful and constructive feedback. We are particularly grateful that the reviewer recognizes the key strengths of our work, including the **importance and timeliness of the topic**, the **clarity and structure of our writing**, and the **simplicity yet effectiveness of our proposed CoRT framework**. Your comments have been invaluable in helping us refine our paper.\n\nWe are grateful for the opportunity to address the remaining concerns and further clarify our contributions:\n\n---\n\n**W1: Comparison with SOTA Models**\n\nWe thank the reviewer for this excellent suggestion. We agree that including the performance of SOTA models like o3 and o4-mini provides a crucial point of reference.\n\nWe will update Table 1 in our revised manuscript to include these comparisons. The updated section of the table will be as follows:\n\n| Model                                         | Tool-Use | AIME24 (Pass@1) | AIME25 (Pass@1) |\n| :-------------------------------------------- | :------: | :-------------: | :-------------: |\n| **Hint-Engineering-RFT-32B** |    ✓     |    76.7%    |    67.1%    |\n| o3                                            |    ✗     |      91.6%      |      88.9%      |\n| o3                                            |    ✓     |      95.2%      |      98.4%      |\n| o4-mini                                       |    ✗     |      93.4%      |      92.7%      |\n| o4-mini                                       |    ✓     |      98.7%      |      99.5%      |\n\n---\n\n**W2 & Generalization: Lack of an Explicit Planner**\n\nWe thank the reviewer for this excellent and thought-provoking question about our method's approach to learning tool invocation. The reviewer correctly notes that we do not use an explicit, separate module like a planner or a gating function. This is a deliberate design choice, and we believe our integrated, end-to-end approach fosters a more genuine and flexible understanding of tool use.\n\n1.  **An Integrated, End-to-End Decision Process:** Instead of offloading the decision to a separate module, our framework teaches the LRM to make the tool-use decision as an intrinsic part of its own reasoning process. The model learns to generate the special `'''python` token at the appropriate time, based on the entire context of the problem and its reasoning chain. This is achieved through fine-tuning on high-quality exemplars (our \"Hint-Engineering\" data) where the timing of tool invocation is optimal. We argue this internalization is a more powerful form of learning than relying on a simpler, potentially brittle, external gate.\n\n2.  **Flexibility and Contextual Nuance:** An explicit gating function often reduces the decision to a simplified binary choice. Our integrated approach is far more flexible. The decision to invoke the Code Interpreter is made by the full LRM, which can consider the nuance of the entire conversation history. It's not just a decision of *whether* to use a tool, but *which specific sequence of computations* is needed at that exact moment.\n\n3.  **Empirical Evidence of Generalization:** Our framework's ability to develop a flexible and generalizable understanding is empirically supported by our results.\n    *   **Diverse Code Usage:** Our Code Behavior Analysis (Section 3.3, Figure 5) shows that the fine-tuned models learn to use a wide variety of Python functions for complex tasks like \"Solving Equations,\" and \"Symbolic Mathematics.\" This demonstrates a sophisticated, context-dependent application of the tool.\n    *   **Out-of-Distribution (OOD) Generalization:** To further address transferability, we conducted an OOD evaluation on **Chemistry problems from the GPQA benchmark**. This domain is far removed from our mathematical training data. Crucially, the standard tool for this domain, the `RDKit` library, **was never included in our training data**. The model's ability to spontaneously discover and effectively use this domain-specific library serves as a rigorous test of its generalization.\n\n| Model | Accuracy | Avg. Tokens | RDKit Usage Rate |\n| :--- | :---: | :---: | :---: |\n| DeepSeek-R1-32B (Base) | 40.6% | 5,947 | 0% |\n| Hint-Engineering-RFT-32B | 47.5% | 4,220 | 81.3% |\n| Improvement | **+6.9 pts** | **-29.0%** | **+81.3 pts** |\n\nThese results strongly suggest that the model has generalized the decision-making process of *when* and *how* to invoke tools, rather than simply memorizing triggers.\n\n---\n\n**Q1: Will the method cause tool overuse?**\n\nThank you for this crucial question. Our analysis on the MATH500 dataset, stratified by difficulty, shows that our framework does not lead to indiscriminate overuse. Instead, it fosters an **adaptive and intelligent tool-use policy**, where code usage is modulated based on problem difficulty.\n\nThe table below for our `Hint-Engineering-1.5B` model demonstrates this clearly:\n\n| MATH500 Level (Difficulty) | Avg. Code Blocks (after SFT) | Avg. Code Blocks (after RL) |\n| :--- | :---: | :---: |\n| **Level 1 (Easiest)** | 2.05 | 1.63 |\n| **Level 2** | 1.94 | 1.89 |\n| **Level 3** | 2.63 | 2.01 |\n| **Level 4** | 2.70 | 2.25 |\n| **Level 5 (Hardest)** | 2.71 | 2.58 |\n| **Trend** | **Usage increases with difficulty** | **Usage increases with difficulty** |\n\nAs shown, the model consistently uses **fewer code blocks for simpler problems and more for complex ones**. This intelligent, demand-driven usage pattern is the antithesis of \"overuse\" and a hallmark of an efficient problem-solver.\n\n---\n\n**Q2: What are the code triggering rates?**\n\nOur models achieve high and stable code triggering rates after training, demonstrating they have robustly learned to invoke the Code Interpreter.\n\nThe code usage rate after Supervised Fine-Tuning (SFT) is already high, and it is further reinforced and stabilized during Reinforcement Learning (RL). The table below details the code usage rates on the AIME24 dataset at different stages of RL training for our 1.5B models.\n\n| Training Stage | Prompt-Hint-1.5B | Hint-Engineering-1.5B |\n| :--- | :---: | :---: |\n| After SFT (Step 0) | 76.7% | 86.9% |\n| RL Step 20 | 86.9% | 95.0% |\n| RL Step 40 | 94.4% | 97.7% |\n| RL Step 60 | 94.6% | 99.0% |\n| RL Step 80 | 97.5% | 99.0% |\n| RL Step 100 | 96.5% | 99.6% |\n| RL Step 120 | 97.1% | 97.9% |\n\nAs the table shows, the **Hint-Engineering** approach achieves a very high triggering rate of **86.9%** immediately after SFT, which then quickly climbs to and stabilizes **above 95%** during RL. This indicates that the model has effectively learned to utilize the tool consistently. A full visualization of this trend, along with other code behavior metrics, is available in **Appendix D.1, Figure 8**.\n\n---\n\n**Q3: Clarification on 30 vs. 830 examples.**\n\nWe thank the reviewer for this question and for the careful reading. This is not a typo; the two numbers refer to two distinct datasets used in our sequential, multi-stage training pipeline designed to scale up high-quality data.\n\nOur process involves two main fine-tuning stages, where the output model of one stage becomes the input for the next:\n\n1.  **Stage 1: Bootstrapping with SFT.** We start with the base `DeepSeek-R1-32B` model. We perform Supervised Fine-Tuning (SFT) on it using a small, manually created set of **30 high-quality samples** (`D_Hint-engineering-SFT`). The goal is to \"bootstrap\" the model, teaching it the fundamental principles of our efficient Hint-Engineering reasoning pattern. The output of this stage is the `Hint-Engineering-SFT-32B` model.\n\n2.  **Stage 2: Scaling with RFT.** We then use the improved `Hint-Engineering-SFT-32B` model from Stage 1 to generate solutions for a larger set of 820 problems. After automatically filtering these generated solutions for correctness and quality, we obtain approximately 800 new high-quality trajectories. We combine these with our original 30 manual samples to create a larger, scaled-up dataset of **830 examples** (`D_Hint-engineering-RFT`). This dataset is then used for Rejection Fine-Tuning (RFT) on the `Hint-Engineering-SFT-32B` model, producing our final, most capable model: `Hint-Engineering-RFT-32B`.\n\nThe full pipeline is summarized in the table below:\n\n| Fine-Tuning Stage | Input Model | Training Data Used | Data Size | Data Origin | Output Model |\n| :--- | :--- | :--- | :---: | :--- | :--- |\n| **SFT** | DeepSeek-R1-32B | `D_Hint-eng-SFT` | 30 | Manually created & annotated | Hint-Eng-SFT-32B |\n| **RFT** | Hint-Eng-SFT-32B | `D_Hint-eng-RFT` | 830 | 30 manual + 800 generated & filtered | **Hint-Eng-RFT-32B** |\n\n---\n\n**Q4: Does suppressing reflection hinder self-correction?**\n\nWe thank the reviewer for this crucial question. Our method is carefully designed to avoid this exact problem by making a critical distinction between two types of reflection:\n*   **Unnecessary Numerical Verification:** Doubting the computational accuracy of the Code Interpreter (e.g., checking if `2+2` is `4`). Our hints are designed to suppress this inefficient behavior.\n*   **Essential Logical Verification:** Checking the semantic correctness of the model's *own generated code* to ensure it aligns with the problem-solving plan. This is a vital self-correction skill that we preserve and encourage.\n\nAs we explicitly state in the paper on lines 122-124, *\"while we discourage the verification of Python’s numerical calculations, we maintain the model’s behaviour to verify the logical correctness of the code structure\"*. Therefore, our approach improves efficiency without hindering the model's ability to detect and correct its own logical code errors.\n\n---\n\n**Typo Correction:**\n\nThank you for catching the typo in line 267. We will correct this to \"problems\" in the revised manuscript.\n\n---\n\nThank you again for your valuable and encouraging feedback. We believe that incorporating these changes will significantly strengthen our paper. Please let us know if there is anything else we can do to help you better understand and recommend our work."}, {"review_id": "LSc6SOYBiO", "reviewer": "Reviewer_jUL7", "summary": "This paper introduces prompting techniques to make the best use of code interpreter tools during the reasoning phase of modern LLMs.\n\nThe authors identify two inefficiencies in reasoning traces of modern LLMs: (1) they tend to prefer natural language computation and using Python as a verification tool only, (2) they verify again in natural language the code execution results. To mitigate these two limitations, the authors try to identify these moments in the reasoning sequence and inject “hints” to redirect the LLM towards (1) executing Python computations or (2) not verifying the accuracy of the Python calculation, respectively.\n\nThey generate multiple samples on the STILL3 dataset and filter out samples with incorrect answers or samples that don’t adhere to the above hints to rejection finetune a deepseek-r1-distill-qwen-32B with this data (resulting in the hint-engineering-rft-32B model).\nThey also use another prompt at the beginning of each reasoning sequence to tell the model to use multiple Python code calls and finetune a deepseek-r1-distill-qwen-32B with similar data (resulting in the prompt-hint-sft-32B model).\n\nTo conduct RL experiments, the authors distill their finetuned 32B models into 1.5B models by sampling from them on a diverse set of 10,000 examples.\n\nFinally, the result section shows that the specially prompted models achieve comparable performance to other baselines of similar sizes but generates less tokens, hence is more efficient.", "questions": "### Questions\n\nQ1: How do you manually inject hints to mitigate the two inefficiencies mentioned on page 4? Do you mean that you have a heuristic computation that checks if either of the two situations arises? \n\n> A critical challenge in our approach was identifying suitable positions for hint insertion. [...]. Consequently, we opted for manual hint placement and insertion.\n\nThis sentence should be much more detailed, especially since this is a big part of the contribution of this paper.\n\nQ2: At the end of section 2.2.2, you mention the “Hint-Engineering-SFT-32B model” and the D_{hint-engineering-sft} dataset, but these are not described before. What are these?\n\nQ3: Why did you distill your fine-tuned 32B model into 1.5B models instead of finetuning directly the 1.5B models with the same data you used to train the 32B models?\n\n### Typos\n- Line 60: [...] to encourage writes the codes correctly.\n- Line 75: [...] as illustrated in Figure 2.1 – there is only figure 2.\n- Line 145-150: duplicate wording in line 145 & 149. Not coherent for line 145 but ok for line 149.", "limitations": "No limitation discussion.\nOne limitaion of this work is the focus on math problems and only the python interpreter tool. Future work can explore more diverse domains.\nA brief paragraph on the societal impacts of having better llms at math and overall tool use could be included.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "## Strengths\n\nThis paper raises an important point as modern LLMs having access to a panoply of external tools should be able to use them efficiently. Experiments are well made and empirical analysis show how efficient the proposed prompt can be.\n\n## weaknesses\n\nW1: The paper omits several important aspects in the main text and instead sends the reader in the appendix. While it is acceptable to save space by puting **additional results** in appendices, the main experimental setup (baselines, training objectives, data splits, metrics, etc…) should be part of the main text. Similarly for the literature review, while a longer version can be in the appendix, a short version should still be in the main text.\nIn its current form, the paper is missing important details making it hard to understand (see questions in the Questions section).\n\nW2: The data synthesis framework is presented as a core contribution of this paper; however it is not clear what it is? The authors generated 30 samples manually and prompted LLMs to generate more samples on STILL3. Is the “framework” how the 30 example were generated? If so, then while quality is indeed more important than quantity, a data synthesis framework should still be scalable to more than 30 examples. Is the “framework” related to how the LLMs are prompted? If so, calling it a prompt rather than a framework would be more accurate. \n\nW3: In the RL experiments, injecting a “hint” to say to the model to **not** use Python anymore contradicts previous hints in the prompt. In addition, this adhoc intervention will directly impact the conclusion you will observe to the research question of _“How the models’ interaction patterns with the Code Interpreter evolve during the RL process?”_", "was_revised": true, "final_justification": "Based on the provided explanations and additional studies, I will raise my score from 2 to 4, however, I request that the authors must include (1) clarification about the data synthesis framework (2) clarification about the usage of the RL safeguard hint, and (3) clarification about the manual annotation workflow in the main text of the paper, along with a literature review.", "rebuttal": "We sincerely thank the reivewer for their thorough review and constructive feedback. We are particularly grateful that the reviewer recognizes the importance of our research direction—**enabling LLMs to use external tools efficiently**—and acknowledges that our **\"experiments are well made and empirical analysis show how efficient the proposed prompt can be.\"** This encouragement is invaluable. \n\nWe are grateful for the opportunity to address the remaining concerns:\n\n---\n\n**W1: Details in Main Text**\n\nThank you for this crucial feedback. We agree completely. Due to strict page limits, we initially prioritized presenting our key findings in the main text while moving detailed setups to the appendix. In the revision, we will integrate these essential details and a concise literature review into the main text, relocating less critical material to the appendix.\n\n**W2: The Data Synthesis Framework**\n\nThank you for seeking clarification. We understand the confusion and appreciate the opportunity to elaborate on this core contribution. Our \"data synthesis framework\" is a **systematic, scalable pipeline** designed to teach models efficient tool-integrated reasoning. Its necessity and effectiveness are rooted in a two-stage process:\n\n1.  **Stage 1: Seed with High-Quality Data (Hint-Engineering & SFT).** We start by manually creating 30 high-quality \"seed\" examples. This small dataset is **essential** because, as we will demonstrate below, prompting alone **cannot** teach the model the desired efficient reasoning patterns. These 30 examples provide focused demonstrations of how to proactively use code for computation, not just verification. **In this stage, we fine-tune the base `DeepSeek-R1-32B` model on these 30 seed examples to produce our `Hint-Engineering-SFT-32B` model.**\n\n2.  **Stage 2: Scale with Automated Synthesis (RFT).** After SFT on the seed data \"unlocks\" the correct behavior, we use the now-improved model to generate thousands of candidate solutions. These are then automatically filtered based on correctness and reasoning quality to create a larger, high-quality dataset for Rejection Fine-Tuning (RFT). This creates a self-improving cycle that scales our data from 30 to 830 examples (with potential for further expansion given additional computational resources). **In this stage, the `Hint-Engineering-SFT-32B` model serves as the input, and after RFT on the scaled data, we obtain our final `Hint-Engineering-RFT-32B` model.**\n\nTo prove our seed data is necessary, we ran a zero-shot experiment on the base model using a strong TIR prompt without fine-tuning. The results below show our framework teaches a superior reasoning paradigm that prompting alone cannot achieve.\n\n| Model                     | Dataset | Avg. Accuracy | Avg. Tokens | Avg. Code Blocks |\n| :------------------------------------- | :---------- | :---------------: | :-----------------------: | :-------------------------------: |\n| TIR Prompt-hint on DeepSeek-R1-32B    | AIME24      |       67.9%       |           9,382           |             0.89             |\n|                                        | AIME25      |       50.4%       |          11,902           |            0.68              |\n| Hint-Engineering-RFT-32B | AIME24      |    **76.3%**      |   **7,260** `(-22.6%)`   |             **2.10**              |\n|                                        | AIME25      |    **65.2%**      |   **8,532** `(-28.3%)`   |             **2.64**              |\n\n**Key Findings:**\n*   **Superior Performance & Efficiency:** Our model achieves significantly higher accuracy (**+14.8 pts** on AIME25) with substantially fewer tokens (**-28.3%**).\n*   **Fundamental Behavioral Shift:** Our method increases code usage by **2-4x**, shifting the model from an inefficient \"verify-with-code\" pattern to an efficient \"compute-with-code\" strategy. This demonstrates our data is both necessary and effective.\n\n**W3: RL Safeguard Mechanism**\n\nWe appreciate this sharp observation. The hint to stop using Python is **not a contradictory instruction but a critical computational safeguard**, triggered only when a model exceeds the predefined 15-tool-call limit. This is a standard practice to prevent infinite loops and manage resources during large-scale RL, also utilized in works like the AIMO-2 winning solution by NVIDIA [1].\n\nCrucially, this safeguard has a **negligible impact on our conclusions** because it affects only a tiny fraction of failing cases. Our analysis of the RL training logs provides clear evidence: the limit was reached in only **0.2%** of all trajectories. Furthermore, among these rare instances, **85.6%** involved models already trapped in non-productive error loops.\n\nThis data shows the intervention is a rare mechanism for exceptional failure modes and does not influence learned behavior in the vast majority of trajectories.\n\n[1] \"AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset\"\n\n---\n\n**Q1: Manual Hint Injection**\n\nOur process relies on expert judgment guided by a replicable procedure—not on heuristics. We adopted this manual approach because automated methods failed to produce satisfactory results. Given the complexity of our task, we are unaware of any purely automated methods that can succeed without human insight.\n\n**Why Manual Curation was Necessary:** We initially used powerful LLMs as judges to automate hint insertion. However, they consistently failed on subtle but critical distinctions. On a test set of 100 AIME problems, we identified two primary failure modes:\n\n| Failure Mode                                    | Description                                                                                                                              | Occurrence |\n| :-------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------- | :------------: |\n| 1. Confusing Planning with Inefficient Calculation | The judge struggled to differentiate between necessary high-level planning and low-level calculations that should be offloaded to the CI.          |    74/100      |\n| 2. Confusing Logical Verification with Numerical Distrust | The judge couldn't distinguish between a valid check of code logic versus unnecessarily re-calculating a trusted CI output. |    19/100      |\n\nThese frequent and critical errors led us to conclude that a small, high-precision, manually-annotated seed dataset was the most effective way to bootstrap the learning process.\n\n**The Manual Workflow:**\nOur manual editing follows a clear, three-step procedure:\n\n1.  **Generate Baseline Trace:** An annotator prompts the base model to generate a complete reasoning trace for a problem.\n2.  **Identify Inefficiencies:** The annotator inspects the trace, looking for two specific, easily identifiable patterns of redundancy:\n\t*   **Delayed Computation (\"Thinking too hard\"):** The model first guesses an answer in text (e.g., solving `x^2 - 17x + 60 = 0` by sight) before using code (`sympy.solve(...)`) for a precise result. The initial guess is inefficient and error-prone.\n\t*   **Code Distrust (\"Not trusting the calculator\"):** After getting a precise code output (e.g., `>>> [5, 12]`), the model re-verifies it in text (e.g., plugging roots back into the equation). This is redundant.\n3.  **Surgical Correction:** Based on the identified inefficiency, the annotator performs a targeted edit. For \"thinking too hard\", they remove the text-based calculation and insert a \"use code\" hint earlier. For \"not trusting the calculator\", they remove the redundant verification and insert a \"trust Python\" hint.\n\nNote that prior works like InstructGPT also designed detailed annotation instructions. In the revision, we will clarify our own instructions for reference.\n\n**Q2: Model & Dataset Naming**\n\nWe apologize for the lack of clarity.\n*   **D_{hint-engineering-sft}:** This is the dataset of **30 high-quality reasoning traces** created through the manual \"Hint-Engineering\" process described in our response to Q1. It is first mentioned in Line 132.\n*   **Hint-Engineering-SFT-32B:** This is the model resulting from **Supervised Fine-Tuning (SFT)** the base DeepSeek-R1-32B model on the `D_{hint-engineering-sft}` dataset. It serves as the starting point for our Rejection Fine-Tuning stage and is mentioned in Lines 128-129.\n\n**Q3: Distillation vs. Direct Fine-tuning**\n\nThis is an excellent question. We chose distillation because direct fine-tuning the 1.5B model with our small dataset was **ineffective**, as it lacks the capacity to benefit from such subtle guidance. In contrast, the 32B model learns effectively and can then generate a larger dataset to teach the smaller model via distillation.\n\nThe table below confirms this. Direct fine-tuning shows negligible improvement over the baseline, while our distillation approach provides meaningful gains.\n\n| Model (1.5B)         | Training Method    | Avg. Accuracy |\n| :----------------------- | :--------------------- | :---------------: |\n| DeepSeek-R1-1.5B         | Base Model (None)      |       48.1%       |\n| SFT on DeepSeek-R1-1.5B | Direct Fine-Tuning     |       43.2%       |\n| Hint-Eng-SFT-1.5B (Ours) | Distillation from 32B |     **51.2%**     |\n\nThis approach effectively transfers capabilities and enables large-scale RL on smaller, computationally manageable models.\n\n--- \n\nWe sincerely thank you for your insightful review. We gratefully hope that you could re-evaluate our paper based on the clarifications provided above. If our responses have satisfactorily addressed your concerns, we would greatly appreciate it if you could consider updating the review score accordingly."}]

Author rebuttal:

Meta-review: (a) Summarize the scientific claims and findings of the paper.

This paper introduces CoRT, a framework to improve how Large Reasoning Models (LRMs) use Code Interpreters (CIs) for mathematical reasoning. The core method, "Hint-Engineering," uses a small, high-quality "seed" dataset to fine-tune models, teaching them to avoid inefficiencies like "delayed code computation" and "code result distrust." The paper claims this approach significantly improves accuracy while drastically increasing token efficiency by 30-50%.

(b) What are the strengths of the paper?

1.  It addresses the critical challenge of effectively integrating external tools with LRMs.

2. The paper compellingly shows that a very small, high-quality dataset can lead to significant performance and efficiency gains.

3. The claims are well-supported by thorough evaluations across multiple models and challenging math benchmarks.

(c) What are the weaknesses of the paper?

The initial submission's primary weaknesses, which were later addressed, included:

1. The method's reliance on only 30 manually created samples raised questions about its generalizability.

2. The original results lacked statistical significance tests, making the claims hard to validate.

3. Details of the data synthesis pipeline were initially not well-explained.

(d) Provide the most important reasons for your decision to accept.

The paper is recommended for acceptance primarily due to the authors' thorough and convincing rebuttal, which directly addressed all major concerns. The authors provided compelling new evidence during the discussion period, including statistical significance tests and a novel out-of-distribution (OOD) evaluation on chemistry problems, which successfully validated their claims and demonstrated the framework's scalability. This, combined with their commitment to incorporating clarifications into the final manuscript, led all reviewers to a positive consensus.

(e) Summarize the discussion and changes during the rebuttal period.

The rebuttal period was highly effective and pivotal. Reviewers raised major concerns about the method's scalability and the lack of statistical validation. The authors responded convincingly by conducting a new OOD experiment on chemistry problems, which demonstrated that the model could generalize its skills to an unseen domain and tool (RDKit). They also added pairwise Wilcoxon signed-rank tests to confirm their results were statistically significant. These additions, along with detailed clarifications of their "seed-then-scale" data pipeline, successfully resolved the reviewers' concerns. This led to a strong consensus, with multiple reviewers explicitly stating their issues were addressed and consequently raising their scores in support of acceptance.

Please improve the manuscript in the final version as recommended by the reviewers (especially Reviewer jUL7).
Decision: Accept (poster)