Title: ToolRL: Reward is All Tool Learning Needs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Language Model Agent

Tool Use

Reward Design
Abstract: Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar or complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges: multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning.
In this work, we present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose a principled reward design tailored for tool use tasks and apply it to train LLMs using RL methods.
Empirical evaluations across diverse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving a 17\% improvement over base models and a 15\% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a detailed study on reward design for tool selection and application tasks within the RL paradigm. They investigate the impact of four dimensions of reward strategies: reward type, granularity, scale, and temporal dynamics. Based on these analyses, the authors propose a  principled reward design framework for Tool-Integrated Reasoning (TIR) tasks. Specifically, they introduce two types of rewards: format reward and correctness reward. The correctness reward is further decomposed into three components: tool name matching, parameter name matching, and parameter content matching. Using this reward design, the LLMs trained with reinforcement learning algorithms demonstrate notable performance improvements over base models and SFT models across varied benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The paper explores reward design for tool use within the RL paradigm, covering four aspects: reward type, granularity, scale, and temporal dynamics.\n2. Based on the reward exploration, the authors design a reasonable reward strategy framework.\n3. After reinforcement learning training, the model shows significant improvement over both the base model and the SFT model.\n4. The trained model demonstrates good generalization performance, not limited to improving the reward values during training.\n\n**Weaknesses**:\n\n1. The reward design does not take into account the completion of the final task. Although the authors claim that optimizing each step of tool usage aligns with achieving the overall goal, there remains a gap between the two. Introducing final-task rewards (e.g., answer correctness in question-answering tasks) for comparison would benefit the paper.\n2. The fine-grained correctness reward consists of three components (correctness of tool name, parameter name, and parameter content), yet the maximum values of these components differ. The reasonableness of the weight allocation among them lacks experimental validation.\n3. Qwen2.5 and Llama 3.2 offer native support for function/tool calling, but the paper does not utilize their native tool-calling capabilities. Notably, Llama-3.2-3B-Instruct using native tool calling achieves an overall accuracy of 45.86 on the BFCL V3 leaderboard (see https://gorilla.cs.berkeley.edu/leaderboard.html), which is higher than the RL-trained model reported in this paper.\n\nQUESTIONS:\n1. Given that the dynamic method shown in Figure 5 performs better on more models, why wasn’t it included in the methods section of the paper?\n\n2. Why not conduct a direct comparison with other RL frameworks specifically designed for TIR, such as ToRL or Search-R1?\n\n3. Lines 235–239 mention two examples in Table 4 that illustrate the agent’s ability to proactively refuse inappropriate tool use. Could the authors provide statistical data on how much this reasonable proactive refusal behavior increased after training?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a study on reward design for training LLMs to use tools through RL. The authors propose a reward framework that decomposes tool-calling formatting and correctness into multiple components and demonstrate improvements over SFT approaches approaches. The authors also go in-depth about various aspects of the reward - like scale, length, dynamic vs static etc. and perform experiments using GRPO and PPO algorithms on Qwen 2.5 and Llama 3.2 series of small LMs.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper is well-motivated and tackles an important problem in the domain. The study of reward design for tool use is novel and timely.\n2. If the claims hold true, this work could be very helpful for many downstream applications involving LLM agents. \n3. I particularly liked the comprehensive analysis presented in Section 4. The systematic exploration of reward design (length, scale, granularity, static vs dynamic reward, etc.) provides valuable insights. The finding that longer reasoning traces can actually degrade tool-use performance is particularly interesting.\n\n## Weaknesses\n\n1. There are several experimental details that are currently unclear (detailed in questions section below), which makes it difficult to properly verify the claims made. Key details about data selection and statistical significance are missing too.\n2. When the authors claim that their approach is agnostic to the RL algorithm used, this claim is not really supported by the results presented. So far it seems their reward design works well only with GRPO, and that too specifically with Qwen 2.5 models. The PPO results are consistently weaker, contradicting their broad applicability claims.\n3. Minor: There are several minor presentation issues, typos, and missing details that affect readability.\n\nQUESTIONS:\n## Questions\n\n1. Why is R_max set to 3? This seems like an important hyperparameter but no justification or ablation is provided for this choice.\n2. The authors mentioned that they removed KL penalty term in GRPO. In comparison with PPO baselines, do they also have KL term removed?.\n3. Line 157 states that removing KL penalty leads to faster convergence. Do the authors have any plot to support this claim? What are the performance differences between Qwen 3B Cold Start GRPO with and without KL divergence?\n4. How were the 400 datapoints selected? Randomly sampled out of 4K dataset or were the most difficult data points chosen? \n5. Regarding Line 208 \"We hypothesize that SFT initialization leads to memorization and overfitting, which reduces the impact of GRPO's effectiveness in generalization\": Isn't there any loss curve to establish this hypothesis? Otherwise could the authors verify this by seeing if SFT+GRPO plateau faster than cold-start GRPO? or maybe testing with tools that have identical functionality but different parameter names?\n6. What do the error bars represent in Figure 2? How many evaluations were conducted (how many random seeds per baseline)? Also how much smoothing and what type of smoothing has been done in these figures?\n7. If there is significant overlap in error bars, can we also report standard error over multiple evaluation runs in Tables 1, 2 and 3?\n8. Line 215 claims \"PPO tends to be less stable\" - where is stability shown empirically? In Table 1, all I can see is accuracy which shows SFT400+PPO > Cold Start PPO, but nowhere does it establish stability.\n9. Based on the PPO results, does this mean your framework is not widely applicable as you claim in the introduction? I would then want to see results on DPO or at least one more RL method, otherwise the authors should change their claims - especially the claim made in lines 53-56: \"While our reward design is algorithm-agnostic by nature.\"\n10. Line 221 states \"PPO benefits from our reward design\" - I don't think it's correct to say this when in most model sizes and evaluation datasets across Tables 1, 2 and 3, SFT400+PPO > PPO Cold Start. Therefore this line seems misleading.\n11. I have a curious question - do you believe that tool use didn't improve with extended reasoning could be due to smaller LM size? This phenomenon might not exist when using 7B parameters or beyond?\n12. Why no comparison with other recent RL methods like DPO, SimPO, or direct comparison with Search-R1/TORL on overlapping domains?\n\n\n## Suggestions\n\n1. Typos to fix:\n\nLine 105: \"out\" → \"our\"\nLine 109: \"indicates\" → \"indicate\"\nLine 152: \"employs\" → \"employ\"\nLine 153: \"later\" → \"latter\"\n\n2. It would be nice to add numbers to each equation in Section 2.3 for better readability.\n3. Please increase font size and legend size. Not visible when the paper is printed on A4 sheets - especially for Figures 3 and 7.\n4. Lines 258-259 state \"we introduce a length-based reward, motivated by prior findings that longer thinking traces can support deeper reasoning\" - please back this with appropriate citation.\n5. Include error bars and statistical significance tests for main results tables if you have run multiple seeds based on Figures 3 and beyond\n\nLIMITATIONS:\nThe authors have not discuss neither limitations nor societal impact of their work. The authors claim in the checklist that they discussed Limitations in Section 5 but I can only see future works. \n\nThe authors should add a dedicated \"Limitations\" section covering things like limited evaluation on smaller LMs as well as a comprehensive \"Broader Impact\" discussion addressing potential misuse scenarios, safety considerations since this paper is regarding tool-use in LLMs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of multi-turn tool use with LLMs. It argues that SFT is not a sufficient solution to enable good performance in this setting. RL can enable the model to generalize and act better. The key to making RL work they argue is a good reward design which they provide a reward design for in this work (the main contribution). They conduct extensive experiments and ablations on their reward and aspects of RL training vs SFT to help confirm their hypothesis (second contribution).\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The problem in this paper is well motivated. Multi-turn tool use is the template for modern agents. The paper experiments are comprehensive with sufficient baselines and on a good number of datasets with multiple models. The method is also sound. \n\nClarity: \nThe paper overall is clear but a bit hard to read in a few places:\n- I'm a bit confused about the terminology in 2.1, the choice of the LLM is a tool & its parameters jointly (not just the discrete tool), which would not fall into a discrete set of tools as the parameter set can be continuous. So the choice is tool + parameters. \n- I have only seen the term, Tool-Integrated Reasoning ,in [1] before, is there a reason for using this term instead of the more common terminology of multi-turn tool use, or LLM agent? I would suggest using the more traditional terminology to make the paper more friendly to readers. I don't see how this is different from multi-turn tool use i.e., agents.\n- I would avoid referencing an appendix figure and just include that in the main paper (Figure 9)\n- the experimental tables are too small and make it hard to read, while I appreciate reporting all the numbers, for the main paper I would pick a few of these and allude to the rest in the appendix. What does bold and underline mean?\n\nSignificance: The main contribution here is the reward design they have for tool use in addition to their experimental results. I believe this reward design has potential to be re-used by others.\n\nOriginality: \nThe main contribution is in the reward design, while this might not be flashy often times all it takes to make RL work is a good reward design. Empirically some groups might have implemented similar schemes but no published work exists with these formulas.\nSince the time of submission in May, there has been many numerous works on using RL for multi-turn tool use including frameworks for doing so, so that the claim \"Training LLMs for TIR tasks has predominantly relied on Supervised Fine-Tuning (SFT)\" is no longer applicable [2,3,4]. \n\n[1]: ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving https://arxiv.org/abs/2309.17452\n[2] StepTool: Enhancing Multi-Step Tool Usage in LLMs through Step-Grained Reinforcement Learning https://arxiv.org/abs/2410.07745\n[3] Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment https://arxiv.org/abs/2505.11821\n[4] https://sky.cs.berkeley.edu/project/skyrl/\n\nQUESTIONS:\n- Can you provide error bars for the accuracy numbers reported across the numbers? \n\n- Do you have any idea how the results might change if you use explicit tool calling mechanism of LLMs instead of the special tokens approach? Similarly if you adopt a ReACT style agent loop [1]\n\n[1] https://arxiv.org/abs/2210.03629\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper presents ToolRL, a framework for training LLMs to use tools via reinforcement learning with a structured reward design. The authors propose decomposing rewards into format and correctness components, where correctness is further broken down into tool name, parameter name, and parameter value matching. They apply GRPO for training and evaluate on BFCL, API-Bank, and Bamboogle benchmarks, claiming 17% improvement over base models and 15% over SFT models.

STRENGTHS AND WEAKNESSES:
**Strengths**  
- The reward design dimensions (type, scale, granularity, temporal dynamics) is good for tool-learning fields. 
- Presentation is clear. 

**Weaknesses**  

- Technical contribution is limited. Its proposed reward design seems not novel as it can be seen from many RL related works.

- Some statements are over-claimed. It names its title as "Reward is All Tool Learning Needs", however, its training still requires supervised data for initial format learning (400-4K examples), which is not consistent with their claim. 

- The training scale is limited as it only trains on only 4K examples across three dataset, which makes the generalization less convincing.

QUESTIONS:
- Are there any specific challenges that adapts RL to tool-learning will encounter? If no, it seems this work just adapts GRPO into the tool-learning scenarios without bringing any new insights.  
- What are contributions of each reward component to the end-to-end performance?
- The method requires ground truth tool calls during training but claims to generalize to "free-form" scenarios. What is the mechanism enabling this generalization? The Bamboogle results suggest limited transfer, as tool call patterns do not significantly differentiate.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-05