Title: Teaching Language Models to Reason with Tools
Venue: NeurIPS 2025
Accepted: yes
Keywords: Tool-integrated Reasoning

Large Reasoning Model

Long Chain-of-Thought
Abstract: Large reasoning models (LRMs) like OpenAI-o1 have shown impressive capabilities in natural language reasoning. However, these models frequently demonstrate inefficiencies or inaccuracies when tackling complex mathematical operations. While integrating computational tools such as Code Interpreters (CIs) offers a promising solution, it introduces a critical challenge: a conflict between the model's internal, probabilistic reasoning and the external, deterministic knowledge provided by the CI, which often leads models to unproductive deliberation. To overcome this, we introduce CoRT (Code-Optimized Reasoning Training), a  post-training framework designed to teach LRMs to effectively utilize CIs. We propose **Hint-Engineering**, a new data synthesis strategy that strategically injects diverse hints at optimal points within reasoning paths. This approach generates high-quality, code-integrated reasoning data specifically tailored to optimize LRM-CI interaction. Using this method, we have synthesized 30 high-quality samples to post-train models ranging from 1.5B to 32B parameters through supervised fine-tuning. CoRT further refines the multi-round interleaving of external CI usage and internal thinking by employing rejection sampling and reinforcement learning. Our experimental evaluations demonstrate CoRT's effectiveness, yielding absolute improvements of 4\% and 8\% on DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Qwen-1.5B, respectively, across five challenging mathematical reasoning datasets. Moreover, CoRT significantly enhances efficiency, reducing token usage by approximately 30\% for the 32B model and 50\% for the 1.5B model compared to pure natural language reasoning baselines. The models and code are available at: [this url](https://github.com/ChengpengLi1003/CoRT).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces prompting techniques to make the best use of code interpreter tools during the reasoning phase of modern LLMs.\n\nThe authors identify two inefficiencies in reasoning traces of modern LLMs: (1) they tend to prefer natural language computation and using Python as a verification tool only, (2) they verify again in natural language the code execution results. To mitigate these two limitations, the authors try to identify these moments in the reasoning sequence and inject “hints” to redirect the LLM towards (1) executing Python computations or (2) not verifying the accuracy of the Python calculation, respectively.\n\nThey generate multiple samples on the STILL3 dataset and filter out samples with incorrect answers or samples that don’t adhere to the above hints to rejection finetune a deepseek-r1-distill-qwen-32B with this data (resulting in the hint-engineering-rft-32B model).\nThey also use another prompt at the beginning of each reasoning sequence to tell the model to use multiple Python code calls and finetune a deepseek-r1-distill-qwen-32B with similar data (resulting in the prompt-hint-sft-32B model).\n\nTo conduct RL experiments, the authors distill their finetuned 32B models into 1.5B models by sampling from them on a diverse set of 10,000 examples.\n\nFinally, the result section shows that the specially prompted models achieve comparable performance to other baselines of similar sizes but generates less tokens, hence is more efficient.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThis paper raises an important point as modern LLMs having access to a panoply of external tools should be able to use them efficiently. Experiments are well made and empirical analysis show how efficient the proposed prompt can be.\n\n## weaknesses\n\nW1: The paper omits several important aspects in the main text and instead sends the reader in the appendix. While it is acceptable to save space by puting **additional results** in appendices, the main experimental setup (baselines, training objectives, data splits, metrics, etc…) should be part of the main text. Similarly for the literature review, while a longer version can be in the appendix, a short version should still be in the main text.\nIn its current form, the paper is missing important details making it hard to understand (see questions in the Questions section).\n\nW2: The data synthesis framework is presented as a core contribution of this paper; however it is not clear what it is? The authors generated 30 samples manually and prompted LLMs to generate more samples on STILL3. Is the “framework” how the 30 example were generated? If so, then while quality is indeed more important than quantity, a data synthesis framework should still be scalable to more than 30 examples. Is the “framework” related to how the LLMs are prompted? If so, calling it a prompt rather than a framework would be more accurate. \n\nW3: In the RL experiments, injecting a “hint” to say to the model to **not** use Python anymore contradicts previous hints in the prompt. In addition, this adhoc intervention will directly impact the conclusion you will observe to the research question of _“How the models’ interaction patterns with the Code Interpreter evolve during the RL process?”_\n\nQUESTIONS:\n### Questions\n\nQ1: How do you manually inject hints to mitigate the two inefficiencies mentioned on page 4? Do you mean that you have a heuristic computation that checks if either of the two situations arises? \n\n> A critical challenge in our approach was identifying suitable positions for hint insertion. [...]. Consequently, we opted for manual hint placement and insertion.\n\nThis sentence should be much more detailed, especially since this is a big part of the contribution of this paper.\n\nQ2: At the end of section 2.2.2, you mention the “Hint-Engineering-SFT-32B model” and the D_{hint-engineering-sft} dataset, but these are not described before. What are these?\n\nQ3: Why did you distill your fine-tuned 32B model into 1.5B models instead of finetuning directly the 1.5B models with the same data you used to train the 32B models?\n\n### Typos\n- Line 60: [...] to encourage writes the codes correctly.\n- Line 75: [...] as illustrated in Figure 2.1 – there is only figure 2.\n- Line 145-150: duplicate wording in line 145 & 149. Not coherent for line 145 but ok for line 149.\n\nLIMITATIONS:\nNo limitation discussion.\nOne limitaion of this work is the focus on math problems and only the python interpreter tool. Future work can explore more diverse domains.\nA brief paragraph on the societal impacts of having better llms at math and overall tool use could be included.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CoRT, a post-training framework for teaching LRMs to effectively integrate code interpreters for improving the model's ability in complex mathematical reasoning. The method introduces prompt-hint and hint-engineering strategies to synthesize high-quality code-integrated reasoning data, followed by supervised fine-tuning, rejection fine-tuning, and reinforcement learning. Experiments on multiple math benchmarks demonstrate the effectiveness of CoRT in improving both accuracy and efficiency in math reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper focuses on an important and timely topic, which is how to effectively integrate external tools like Code Interpreters into LRM to improve mathematical reasoning.\n2. In general, the writing of the paper is clear, well-structured and easy to follow.\n3. The proposed method is simple yet effective, using well-designed prompt strategies and fine-tuning techniques to enhance both accuracy and efficiency.\n\nWeakness:\n1. The paper does not provide comparisons with state-of-the-art tool-using models such as o3 and o4-mini. While these models are mentioned as motivation, it would strengthen the empirical analysis to include their performance as a reference.\n2. While the paper emphasizes the goal of teaching LLMs when and how to use tools, the proposed method mainly relies on prompt-based heuristics and fine-tuning to induce tool usage. It lacks an explicit mechanism for learning the decision of whether to invoke the external tool, such as a gating function or planner module. As a result, the model may not develop a genuine understanding of tool invocation timing, limiting its flexibility and generalization to unseen scenarios.\n3. There is a typo in line 267 \"probelms\"-->\"problems\".\n\nQUESTIONS:\n1. Will the proposed method cause the fine-tuned model to overuse the Code Interpreter, even when it is not needed?\n2. What are the code triggering rates of the fine-tuned models?\n3. In the abstract and introduction, the author mentioned that they manually created 30 high-quality examples to post-train models. But in line 134, it mentions that 830 examples are created to obtain the Hint-Engineering-RFT-32B model. Is it a typo here?\n4. The proposed hint-engineering method encourages models to avoid reflecting on code execution results in order to reduce redundant verification and improve token efficiency. However, in cases where the model generates incorrect or semantically flawed code, would this suppression of reflection hinder the model's ability to detect and correct its own code errors?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoRT, a framework designed to teach LRMs how to effectively use a Code Interpreter (CI) for mathematical reasoning. The authors identify key inefficiencies in existing models, namely \"delayed code computation\" and \"code result distrust.\" To address this, they propose \"Hint-Engineering,\" a method of manually inserting strategic hints into reasoning traces to guide the model toward more efficient use of the CI. Based on a small set of 30 manually crafted examples, they fine-tune models of various sizes and demonstrate absolute accuracy improvements and significant reductions in token consumption (up to 50%) on several mathematical benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Minor Concerns\n- Overstated Novelty: The paper frames CoRT as a new framework. However, the constituent parts—SFT, RFT, strong-to-weak distillation, and RL on curated data—are all established techniques. The primary novelty lies in the manual \"Hint-Engineering\" data curation strategy, whose limitations are discussed below.\n\n\n- Token Efficiency Framing: The paper highlights a 30-50% reduction in token usage. While this is a valuable practical outcome, it is a direct and intended result of the hint design, which explicitly discourages textual reasoning in favor of code execution. This is more of a behavioral shift guided by data curation than a fundamental improvement in the model's innate reasoning efficiency\n\n## Major Concerns\n\n- The most significant flaw in this paper is the absence of any statistical validation for the empirical results. The authors explicitly state in the checklist (Question 7) that they did not report error bars or conduct multiple runs due to the high computational cost of LLM inference. While the resource constraints are understandable, a single experimental run is insufficient to conclude that the observed gains are not the result of random factors such as initialization seed or other sources of noise. This lack of rigor undermines the paper's central claims, such as the 4% and 8% absolute improvements cited in the abstract. For instance, the average accuracy difference between the main proposed model, Hint-Engineering-RFT-32B (81.3%), and the Prompt-Hint-SFT-32B model (81.8%) is small and could easily fall within a standard margin of error.\n\n- Limited Scalability and Generalizability of \"Hint-Engineering\": The core methodological contribution, \"Hint-Engineering,\" is described as a process of manually creating 30 high-quality samples and manually inserting hints. The authors concede that their attempts to automate hint placement were \"insufficiently precise\". This heavy reliance on manual, expert-driven intervention raises serious questions about the scalability and generalizability of the proposed framework. The contribution appears to be less of a novel, scalable training method and more of a highly specific, labor-intensive data curation effort. It is unclear how this approach would be applied to new domains or larger, more diverse datasets without a similar, prohibitive level of manual effort. The paper fails to provide a path toward automating this critical step, which is a significant limitation.\n\nQUESTIONS:\n1. The abstract and main results table highlight specific absolute performance gains of 4% and 8% for your models. Given that the paper acknowledges these results are from single experimental runs due to computational costs, how can you ensure these improvements are statistically significant and not a result of random factors like initialization? \n\n\n\n2. Your core contribution, \"Hint-Engineering,\" relies on manually creating 30 high-quality samples and manually inserting hints, because you found that attempts to automate this process were \"insufficiently precise\". Can you elaborate on the specific failure modes of your automation attempts and explain how this framework could be scaled to new or larger problem domains without requiring prohibitive amounts of expert manual labor?\n\nLIMITATIONS:\nSee the questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a solution to a technical challenge: how to combine a code interpreter for Large Reasoning Models. The authors identify inefficiencies in direct use of LRMs with CIs (such as excessive token usage and manual verification of code output), and propose a multi-stage training process based on hint-engineering to address these issues. Training methods use supervised fine-tuning, rejection fine-tuning, and reinforcement learning on both large (32B) and smaller (1.5B) models. Experiments across five challenging math benchmarks show the proposed method improves accuracy and reduces token use, with empirical analysis supporting the main claims.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\nClarity of Writing: The paper clearly presents its main ideas and objectives. The problems addressed are well-motivated and easy to follow.\n\nComprehensive Experiments: The authors provide thorough experimental results and analyses across several well-known benchmarks, including AIME24, AIME25, AMC23, MATH500, and Olympiad datasets. This demonstrates the effectiveness of prompt-hint and hint-engineering training.\n\nData Quality over Quantity: The work shows that small, carefully curated post-training datasets can yield substantial improvements. This finding aligns well with ongoing discussions in the alignment and LLM training communities.\n\nWeakness: \n\nLimited Scale of Hint-Engineering Data: The manual annotation for hint-engineering is limited to just 30 problems for the 32B models. While this produces promising results, it raises concerns about scalability and generalization. The main text should discuss in more detail (not just in the appendix) how performance is expected to transfer to less curated or broader settings. Many recent works question whether such small data is even needed for this type of training [1].\n\nLack of Statistical Significance Testing: The paper does not provide significance tests (e.g., p-values) to demonstrate that the improvements are statistically meaningful. Adding such analyses would strengthen the claims.\n\nInference Time Analysis Missing: Although the paper claims better token efficiency, the actual inference time per task is not reported. Since real-world applications care about wall-clock inference time when given a problem, more evidence is needed to show that hint-engineering is actually faster than prompt-hint methods. My main concern is that maybe token efficiency is not the bottleneck of inference speed. Execution time can be the inference speed bottleneck, potentially making reducing the token number not meaningful. \n\nAlternative baselines for hint-engineering: Instead of complicated RL with CI for hint-engineering, could simple length-based penalty rewards achieve similar token efficiency without the complexity of code-integrated RL? An analysis or discussion on this would be valuable.\n\nQUESTIONS:\n(1) Baselines: The key difference between this approach and traditional tool-augmented LLMs is not obvious. In the examples provided in the main part and the appendix, the Python code being generated is relatively simple—for instance, just calling packages like sympy or computing the gcd. In these cases, it seems entirely feasible to build a toolLLM to retrieve related functions and call them. We do not need to write code and execute it. Can you explain more why a toolLLM with function calling is not enough and why it is needed to write a \"code wrapper\" for the usage of some API? Maybe such a simple function calling with all APIs available in a famous math package should be conducted using the same training process as a baseline.\n\n(2) The reason why the models rely heavily on text reasoning instead of code reasoning is potentially because the base model is R1-style.  Is it possible to start from the Instruct model and then just run RL based on it? The ideal solution for efficient CI-augmented reasoning is that the LLM generates computing programs first and then runs the text reasoning. The existence of verification and the existence of heavy text reasoning can be due to the instructional data and base reasoning model.\n\n(3) Can we simply prompt LLMs to skip generating or executing verification code altogether, focusing only on code that performs the actual computation? This might further improve overall efficiency and response time.\n\n(4) Small LLMs are designed for on-device use, but enabling them to run and execute code often requires Docker or similar environments. This adds extra overhead, potentially making the deployment heavier than intended for lightweight, on-device applications.\n\n(5) There is also a formatting error in Line 119—the code block should start with python, not python''.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}