ToolRL: Reward is All Tool Learning Needs
Authors: Cheng Qian
Emre Can Acikgoz
Qi He
Hongru WANG
Xiusi Chen
Dilek Hakkani-Tür
Gokhan Tur
Heng Ji
Venue: NeurIPS 2025 poster ()
Keywords: Language Model Agent
Tool Use
Reward Design
Primary area: deep_learning
TL;DR: The paper proposes a principled reward design framework for training LLMs on tool use via reinforcement learning, leading to significant gains over SFT and baseline models in generalization and performance.

Abstract: Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar or complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges: multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning.
In this work, we present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose a principled reward design tailored for tool use tasks and apply it to train LLMs using RL methods.
Empirical evaluations across diverse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving a 17\% improvement over base models and a 15\% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research.

ToolRL: Reward is All Tool Learning Needs
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- Gokhan Tur, Heng Ji
University Illinois Urbana- Champainn {chengg9, acikgoz2, henggi//illlioiss.. edu
Abstract
Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges; multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such answer matching, fail offer the finegrained feedback required for effective learning. In this work, we present the first com- prehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose principled reward design tailored for tool use tasks and apply it to train LLMs using RL methods. Empirical evaluations across di- verse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving 17% improvement over base models and a 15% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research !
Introduction
Recent advances in Large Language Models (LLMs) have showcased remarkable capabilities in complex reasoning tasks [20]. Among the techniques that have significantly contributed to this progress, Reinforcement Learning (RL) has emerged as a powerful paradigm, enabling LLMs to develop emergent capabilities such as self-reflection, self-correction, and long-horizon planning [13 48]. These capabilities have been instrumental in the success of models like ol and RI, particularly mathematical and logical reasoning domains 36
23
Beyond traditional reasoning tasks, an increasingly important area is Tool-Integrated Reasoning (TIR). TIR involves LLMs interacting with external tools, such search engines [17
calcula- 37 or code interpreters 12 24]. in a multi-step. feedback-driven loop to arrive at solutions. tors
TIR particularly important it addresses core limitations of LLMs,, such as outdated knowledge and calculation inaccuracy. By integrating external tools that offer real-time access and specialized capabilities, TIR enables models tackle complex tasks in a more grounded and goal-directed way.
Unlike textual reasoning, which primarily involves deduction and inference from static text, TIR additionally demands the model's ability select appropriate tools, interpret intermediate outputs, and adaptively refine its trajectory on the fly. These dynamic and interactive reasoning skills position TIR at the core of the emerging paradigm of aggssass As such, TIR enables wide range
Data codes released at http:://ggtttt..coooaaaaaaaheegr
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

of applications, including scientific discovery 41 16 research automation [1.11] embodied task
completion 660 and everyday decision-maken making [34 59
Training LLMs for TIR tasks has predominantly relied on Supervised Fine-Tuning (SFT), wherein existing approaches typically generate these integrated reasoning steps offline, followed by subsequent SFT on these trajectories [3] 58 g D. While SFT is effective to some extent, it struggles with generalization, exploration, and adaptability O [3, failing to capture the strategic flexibility needed for optimal tool use. This motivates a fundamental research question: Can RL better equip LLMs with agentic tool-using capabilities, and if so, what the optimal RL. design for TIR?
Recent efforts such as Search-RI [77] and TORL [23] have begun to explore this direction. However, their focus is narrow: either constrained to search tools in question answering settings or code tools in math problem-solving. In contrast, our work aims to study RI--based training for general-purpost tool selection and application, across diverse and complex tool sets with different task types.
For an RL algorithm to be effective, a well-designed reward is essential. Unlike math tasks with a single correct answer, Tool- Tool-Integrated Reasoning (TIR) tasks introduce multiple layers of complexity: they often involve multi-step interactions where each turn may require invoking multiple tools, each with carefully specified parameters. Designing effective reward signals to guide learning through this complexity remains an open and underexplored challenge. In this paper, we address reward design for TIR by proposing framework applicable across RL algorithms. We validate it on Group Relative Policy Optimization (GRPO) 44 and Proximal Policy Optimization (PPO) 431, highlighting its effectiveness in improving tool use under GRPO.
We begin by formalizing the TIR task, and outlining general principles for effective reward design. Building on this foundation, we show how RL algorithm can be leveraged to train LLMs for robust and context-aware tool selection and application. Empirical results demonstrate that our approach outperforms base models by 17% and SFT models by 15% across multiple tool use and QA benchmarks. Moreover, the trained model exhibits strong generalization to unseen scenarios and task objectives, along with emergent behaviors such proactiveness and metacognitive reasoning.
To identify optimal reward strategies, we systematically explore broad spectrum of reward configu- rations across four key dimensions: (1) reward rype (what aspect to reward), (2) reward scale (how much to reward), (3) reward granularity (how detailed the reward signal is), and (4) reward dynamics (how rewards evolve over time). Through extensive experiments, we identify reward designs that best align with agentic tool use and uncover insights into what makes reward "useful" for tool invoking LLMs. We summarize the core insights we derive as follows:
Longer reasoning trace is not inherently better and length rewards can degrade performance.
Dynamic reward scale helps models transition smoothly from simple to complex behaviors.
Finegrained reward decomposition leads to more stable and effective learning.
We summarize the overall contributions of our paper as follows:
We present the first systematic study on RL-based training for general-purpose purpose tool selection and application in LLMs.
We propose a principled reward design framework for TIR and validate its effectiveness across RL algorithms, with particular strength demonstrated on GRPO.
We conduct extensive experiments analyzing the effects of various reward strategies and distill actionable insights for future research on LLM-agent training.
2
Method
SFT often suffers from overfitting to certain patterns and constrains the model's ability to learn optimal strategies for tool use. To address this, we introduce RL approach for enhancing TIR in LLMs. this section, we begin by defining the TIR task (Section 2.1). followed by our customized rollout strategy (Section 2.2 and reward design (Section 2.3). These components are then integrated into the RL framework
K4 to guide model training on general TIR tasks (Section 2.4).
2.1 Task Definition
Tool- hoo--nnteeetted Reasoning is the process of incorporating external tools into the reasoning trajectory of an LLM to solve user task. A typical TIR trajectory involves multiple tool invocations over several

---

Formal (ormat)
Name:
Parameters:
7e0
Kuseer le flying from ORD LAX cheaper than flying to SPOT (he7
Dialegue Hlltteyy (Trajectory)
shool outo
XW
IName: Get, Pries, Paranclees: [loe J: OR0I, Jr 3-001
Teat
Policy
chool calls
ONG, 2.0))
Buck sleel Sholl oad
yeanl Reservet farmat Match
Rad Nome
Porameter Nowe
Paramnter Contert
Toal Name March
Paramater Contont Match
Thel2
Dse, ORD, he SHO))
(Name: Parmmncters, (:c_2:200
Scuver
(Nane Parameters (From: SPO1)
(Name Price, Parameters [[a__ loc_ TAX)
Agge Stoug
Jownn
Trail
1: NName: Pricc Parsentey 7:00 2-40)
Teal In Price
Teal In Cat, Price
Parea Io Like
Periee
2
Conteat 0412
Correctness correct)
Figure Illustration of TIR rollout and calculation of format and correctness reward.
reasoning steps, with the final outcome determined by the cumulative success of these intermediate decisions. Formally, given a tool set T = (1,,22....,, (n containing available tools, and user query Q, the reasoning trajectory up to step is denoted as:
8k (r1, Tı, 01), (r2, 02, (rk, Tk, 0z);
where r; denotes the model's natural language reasoning at step i, T, S T denotes the set of tool calls invoked at step i, and o, denotes the observation received after executing tools in Tú, possibly including both environment and user feedback.
At each step k + the model must generate the next reasoning step Tk+11 select of tools Tk+1 S T. and formulate a grounded tool call (i.e., a parameterized invocation of each tool) make progress toward solving Q. The model's policy is defined as л : 8k - (Mk+1, Tk+1), where the model's 's objective at each step is to select a tool set Tk+1 that maximizes the reward:
T arg max TR+
R(sk, 1k41,0041).
where R(-)) represents the reward function that evaluates progress made by invoking the tools in T 1- While the immediate reward at each step is maximized, the model's policy is implicitly optimized to maximize the cumulative reward over the entire trajectory, formulated as:
K Σ
max E
R(sx, Te+1, Te+1,00-1)
This formulation is valid because our training data includes ground truth tool calls at each step, allowing step-wise reward signals to guide multi-step success. Unlike QA tasks that focus solely on the final answer, tool selection and application tasks provide dense intermediate feedback. Moreover, we later demonstrate that method enables the model to generalize to settings where tool calls free-form and only the final outcome matters. Therefore, our task setting encourages the model to optimize tool use each step while aligning with the overall task goal.
2.2 TIR Rollout
To enable the model to autonomously generate reasoning traces and tool calls, we instruct the LLM use special tokens <think>, <tool_calls, and <response> to indicate their thoughts, tool calls and responses in output. The full prompt content is shown in Figure g
As illustrated in Figure I when the model output includes <tool_call _alla, automatically parse the tool calls into individual invocations using the model-predicted parameters. The outputs from executions are then inserted into the <obs> field and appended to the dialogue history, serving as the model's 's interaction trajectory. Similarly, if the output contains <response>, the corresponding response is parsed and appended to the dialogue history. Please refer Figure and Figure 9 in the Appendix for instruction details.
It is important to note that <tool_call and <response> are not mutually exclusive; they may co-occur within a single output. The user's initial query Q is placed in the Initial User Input placeholder, and any subsequent user inputs are also appended to the dialogue history when present.
3

---

2.3 Reward Design
Rule-based reward mechanisms have demonstrated strong empirical performance and are commonly employed. In our training, we similarly adopt a reward formulation that combines structural and correctness-based components [77 23 52.. Specifically, the format reward assesses whether the model output adheres to the expected structure including thoughts, tool calls, and responses, while the correctness reward evaluates the accuracy of tool invocations. Formally, the overall reward Ranal decomposed into two components:
Riormat + Reurret, each described in detail below:
Format Reward. The format reward Riormar checks whether the model output contains all required special tokens in the correct order as specified by the ground truth:
Roomat 1 if all required fields appear and are in the correct order else 0
Correctness Reward. The correctness reward Reorreet evaluates predicted tool calls P {Pi,.. J against ground-truth calls G = {GI, G. It includes three components:
Tool Name Matching:
Na N
NG U. E [0,
Thume
where NG and Np are sets of tool names extracted from the ground-trute and predicted tool calls.
Parameter Name Matching:
n E |0,
param
GJEG keys(Pa) U keys
where keys( (G)) and keys( Pp) Parameter Content Matching:
are parameter names of ground-truth and predicted tool calls.
Tvalue
[[Pa Po[k]]
P E [o,
Σ [c))(G CjE0
C keleys(Gy)
where Parl [k] and [k[k] represent parameter contents of ground-truth and predicted tool calls.
The total match score for each match is derived as Tmatch Tname + ''param + Pvalue [0, Smax], where Smax + |G| Σc,ea [keys(G,)] denotes the maximum possible score. The total score is computed by finding the optimal matching between P and G to maximize the total match score:
Receett 2 2
Tmakch
Rmmx E { Rmax]
Sma
where Rmax denotes the maximum possible score after normalization, for which empirically set to in all experiments. More analysis and ablations of reward scale is presented Section
The final reward value Rima is finally derived as the sum of Rivemat and Reumeet Unlike prior works that often rely on binary or overly simplified reward signals, our design captures the nuanced structure of tool calls by evaluating multiple interdependent components including tool names, parameter schemas, and parameter values. This finegrained formulation better reflects the complexity of real- world tool use, where correctness cannot be reduced to single binary criterion. We further validate impact of this design through comprehensive analysis in Section
Overall, our reward design ensures balanced and interpretable evaluation signal by explicitly
separating structural compliance from semantic correctness. By aligning rewards with both format adherence and finegrained tool call accuracy, the model is guided to produce outputs that are not only syntactically valid but also semantically faithful, which crucial for final task success.
2.4 RL Training
To tune the model with structured rewards, we apply both PPO and GRPO algorithms and employ the latter one our main experiment setting. The details about GRPO algorithm tailored with TIR tasks is presented in Appendix
Unlike the original GRPO formulations, we omit the KL penalty term against a reference model. This design choice encourages the model to more freely adapt its behavior to our custom response format and structured reward signals. In practice, we observe that this leads to faster convergence and comparable performance, while also simplifying the training pipeline.

---

Experiments
3.1 Training Dataset
To support robust tool learning, construct mixed dataset spanning diverse tool use scenarios:
ToolACE 27 : A general tool use dataset where the model learns when to invoke tools versus
respond directly, improving decision makiing multi-step interactions.
Hammer (Masked) [25 A subset of Hammer with randomized tool and parameter names, forcing the model rely on descriptions rather than memorized labels, thus enhancing generalization and reducing overfitting to certain tools.
xLAM 61: A compositional dataset requiring one or multiple tool calls per turn, encouraging the model to reason about tool dependencies and plan diverse tool calling action actively.
Empirically, we sample 2K examples from ToolACE and IK each from Hammer and xLAM, creating balanced dataset spanning diverse levels of complexity and tool use. Multi-step trajectories are decomposed into single-step instances, with prior dialogue history injected into the user prompt to preserve context. This encourages strategic exploration and teaches the model to apply tools appropriately within each step.
3.2 Experiment Settings
Training. We conduct all RL experiments using the veRL framework [46]. For each training step. sample batch of 512, and generate 4 responses per query, training for 15 epochs in total (see Appendix [ for full configuration). To encourage policy exploration, we remove KI. regularization and apply temperature 1.0. We initialize our models with the Qwen-2.5-Instruut [49] series, which are further tuned under our customized reward design.
and I.lam-33.2
Evaluation. We evaluate our approach on the Berkeley Function Call Leaderboard (BFCL) 30),
comprehensive benchmark that spans diverse set of challenges, including single-step reasoning, multi-step tool use, real-time execution, irrelevant tool rejection, simultaneous multi-tool selection, and multi-tool application In addition, we present results API-Bank (2T) a three-level evaluation framework comprising 73 diverse and complex API tools. It assesses an LLM's ability to select and apply tools through natural multi-turn dialogues, across three levels of difficulty. We also evaluate on a representative QA benchmark Bamboogle (31]. which comprises variety of question-answering tasks where performance is measured based on the final answer accuracy rather than the correctness of tool use. These broad coverage makes our evaluation setting effective for evaluating real-world LLM tool use proficiency, All results are reported terms of accuracy.
Baselines. We GRPO cold start as our main setting and compare against several baselines: (1) Raw Instruct Model: the original model without any additional fine-tuning or RL. (2) SFT on RL Data: the instruct model fine-tuned using the full 4K selected 400 data points from the RL training set, providing a comparison point to assess whether RL training outperforms SFT. (3) RL on SFT Model: GRPO is applied to model that has undergone SFT on the selected 400 data points. This allows us to evaluate the impact of initializing RL. with format-aware model, in contrast to starting from the raw instruct model cold start manner. (4) PPO: We treat the PPO setting evaluate whether our reward design is effective beyond GRPO. We incorporate both the cold start and as GRPO ensure fairness. Please refer to
baseline to
SFT initialization setting with the same
Appendix D for more details and justifications.
3.3 Results
Main Results. We report BFCL and API-Bank results in Table Table Our primary setting trained from scratch on the Qwen2.5-Instruct series generally outperforms other baselines, achieving about 10% absolute gains over SFT trained on the same data volume. In contrast, LLAMA-3..-Insttu- shows less improvement, possibly due to the model's 's lower adaptability to GRPO-style generalization. Nevertheless, remains competitive and outperforms most basclines API- API-Bank.
https://gorilla.co www.eeew..eeee..e..oo....oo..ee. berkeley.edu/blogg. 33___fc__333_________________33_ mom_mult. tt_turrnBba html
5

---

Tahle 1: BFCL. V3 Benchmark Results, with GRPO cold start primary setting.
Model
Overall Ace
AST Are
NeweLave Eroc Ase
Live Acc
Mali Turn Ace
Recevance Detection
Inclosace Desettin
Ownnt Se1 bel--m- (Raw))
19 19.41%
16,0%%
18F
15. 1-.8%%
0.00%
64,445
62 40%
Qwee25 1,58. Iastact (STT41))
40.3 21
65.125
01.15
56,00%
1,00%
14.44%
60 14%
Qke225 5 1155811-1s00 1
00.71
59344
50.54%
$9.315
1,006
59,895
71 1.346
Qam2 bt-mmm- (SPT-36--PO)
42.91%
77.65
100.00%
4076
Owen2:: (8--0---00 Ionon (PPO Cold
100,075
Qwen2 (0.55555.8--ppp0 (Ours, GRPO Cold Start)
46.3%%
77 1956
76 76.905
00.758
2.25%
100,045
50 44444
O-1-b---1-- (Raw)
33.815
0wen2.5.B 1---BBBlstuu (SFT4))
02,8515
77.786
1216
Qwen2..38 lestrct (SF PPO)
45.50 65.00%
78.2946
71 (975
15. 766
5.136
16.12 14.1355
54 7016
0.m225-18 9-8-a5-1-- ISF GRPO
46.42%
76.2155
68 92
04.1 155
1.75%
71. 76:56
Col
11.15
18.52
67.88%
87%)
Cold Start)
Qwen2.57B Iastuct (Rew)
41.57%
66 66.8256
71.11%
$3.515
4,25%
70.475
66.666
Qk4125 Jastuct (SFT40)
54 SLBFG
09.2546
66.00%
41.4%
0.006
44.445
$.11%
QAm2 5-l--baalB- (SFT4L)
1955
51.5%
73.22%
T480-PPO)
0wen2.5-7B
1orn2.5-7BBlastra Ias-aam ISFT (SFT86--GRM))
251
9036945
76. 34/5
46.519
100.045
19%6
Qwen2.5 1a---aemtatt 7-B-aasaaa (PPO Cold Star)
40 01.815
74.3115
78 16/
63.175
0.385
50 39.9995
52. M775
Qwent 5-78- 1888-aommm (Oers, GRPD Cold Stort)
90, 96.17%
7N 259
749%
18.129
K3.375
0//
LJuno 38 Instrctt (537400
64.2716
66.675
12%
Llans 3.3 3B Instract (5FT4k)
44.36%
65.4256
67.00%
63.049
1.38%6
77.74%5
736 25
Llano 12 38 0SPT (SPT
68.196
$3.90%
3.006
04.13 14.1
2016
1imm--12-88 1e--8---ammmt1io CSPO)
59.49%
72.22%
9016
1llmn-33 1.030-33-88 10--8-Beettecreee (Ders, GRPO Cold Start)
44
74.3455
1859
1.37%
Table API-Bank Test Results.
Table Bamboogle Test Results
Model
Overail Are
Tavrt
1asr13
Tavet
Medel
Accurary
Ang Namn Thxil Fall
Owen2 Transt (SPT490
$1
17.165
1.75%
64.25%
Owee2: 1B-Svancc (SFT60)
0.76
Owen? 5: 58 Ioonur (SFT4)
47/07%
53 155
2/15
36.72%
Owee2 5-1 18- ful-aic (SF Tik:
25, 13.76-
1.25
Osen? 5-1 fratren (SFT-8--77))
$1.175
00.055
08:85%
Owee2 1-1 18-5ramec T9M--P9))
16.55
1.06
Own't 5t -5h-nrana (PPO Call
40.585
1/5
57.87%
Owre2 98-5vaneci (PPO Culd Start)
23,755
2:36
(wead5--1558 Inspust (Ours. GRPO Cold Start:
63.15%
70.89
61.19%
41.22%
Dave2 5-1 1R-hsamee (Ouns GRPO Cold Stari)
4401
1:1
Jnstnct
Qwen) (817)480)
52285
52.224
Owee2.5 58
(SPT490)
54.45
0.56
Qweat (SF14k)
50025
55.65
653205
Owee2. 38 Jssmxt -SFT4:
40 06
0.5
65115
67525
613395
Oxee2. 55 Tesmutt -
45 1.26
1.04
Qwes2 cill-tmmtzzti (PPO Cold Starii
57.25
06.899
55.15%
Unee2.3-8 -55 Isenat (PPO Cold Start)
4009%
1.54
(Ours, GRPP Cold Start)
67.00%
73.455
07.10%
47.50%
Uwee2 355 Isenat (Ours, GRPO Cold Start
6805
1.32
Own' s-Tn-nsm-m (817
50.955
55.89%
51.55
Owee2 5-155 Isonat (39T-433)
3.31
Qver2.5. --B-laoknxc (KT4k)
4700%
51.159
41 11.25%
Dnee2 5-155 honns (NFFT4)
50.45:
1.06
Qwerd 5-78 lestras
Okee2 5-722-5-78-- Irems -9F1-1--790)
45.5%
3.54
Ower2.5-78 CMPOO
54305
Qwen2.511 5-7 foolaas (PPO Cald Starti
$1,445
66.675
4.78%
43.5%%
Uwen2 1-1 Isenat (PP0 Cold Start)
48.9%
1.25
Qwar2 5-7--tssti (Ours, GRPO Cold Start:
61.6%
73.555
69.
38.17%
Usee255-18 1-5 Ioons (Ours, GRPO Cold Start)
72.0%
1.53
Liana
40.545
29.55%
Liana 3.2 18 fastrci (SFT410)
$1,75%
60.65%
35.82%
37.40%
Hime-3 16 Iranzz (SFT44))
44.09%
D:56
Liama-32222 18 Bel-aattse (SFTkk
43,8%%
S3.85%
20.55%
2061%
1mmp11 2-19 frenaz 2-10--r-azz (58744k
08.425
D:58
1imo-3 telmre
16:75
1.75
1ona-113 1- (SFT (BTTH)
$6.355
19%
4331
1.2. 1.2-111-aaarrerr (PPD Cald Start)
00
79%
1ime-3 2-18 Iranar (IPO Cold Star)
29:355
1.42
1.amm-12-11-bbumrr (Ouns GRTP Cal Start)
50135
85 065
12.79%
Llans-3.2 3.338 Iestudt (Ours, CRPO Cold Start
$2.0%
0.59
Plan OomeE Sahurd
N400 SeAMel
$00
(a) Format Reward (Init.) (b) Correct Reward (Init.) (c) Format Reward (Algo.) (d) Correct Reward (Algo.)
Figure 2: Reward trends across training for Dwwen.55-BBIInstuse (a)-(b): Different model initialization strategies
(Cold Start vs. SFT Initialization). (c)-(d): Different RL strategies (GRPO vs. PPO).
SFT Initialization Impacts. Interestingly, GRPO also improves models initialized with limited SFT, often outperforming full-scale SFT trained on 10 times more data. However, this setup still underperforms compared GRPO cold start. We hypothesize that SFT initialization leads to memorization and overfitting, which reduces the impact of GRPO's effectiveness in generalization. As shown in Figure 2 SFT-initialized models achieve higher training rewards due to distributional alignment between SFT and RL data, but empirically generalize worse on the two benchmarks. This further highlights that higher training rewards do not necessarily translate to better generalization.
Reward Design on PPO. We also evaluate PPO under both cold start and SFT-initialized settings to examine the effect of our reward design. The results show that while PPO with cold start can

---

Table 4: Cases of Swee2.5-3B-Inntreuee (GRPO Cold Start) generalize to irrelevant tool detection.
losiid buy mine ticket San Jose right.
User Whet dhe sloge the Iine wikh is peppodionla to the liee win the eguatin y +x+27 Tools Aviilable
_ticketa,
tickets
rangee(o.d. 3.41 fuaciio with vuiitle optionally
Moonie, PindMorrie (location,
Find criscal
Search movia kistton forme.
with
citinks Daplorraat
bay
lave The poutlem adong The nepaiie necipecal of the with coustion
shaw We seed inal movic tilees, slow louatio, time. the stow dasc Hewever the dil not specily musicc mame the sbow cait. We inter the show daz from the ornens dane but nead to lnow the movie rame procoed with prectass <thinie
copiled detals soch
tr that perurdicntar ansther onginal lines sdope Sia the skope of the lne perpendionio yo 3s woruld -18
The pives tod fiod_ori cnl.orition coritioa_.po poitt is relovuee finding the stepe of line and carzati be and to anlre this probiems, <attinks
gesponse> To assis in purcasing a movie tikke, need the same of dhe cooris and quasitt dasc of the dooe Dauld you pizzer provide thes information? comepeeeco
czespozaco The skops line perpadicala lize the regetion y
chrogenurs
outperform SFT in some cases, it tends to be less stable across different model settings. In contrast, GRPO consistently achieves higher rewards even from a cold start, suggesting that our reward design works best under the GRPO framework, which contributes
main experiment setting. As shown in Figure 2 GRPO not only achieves higher correctness rewards but also gains format rewards more rapidly during training. Interestingly, PPO benefits from SFT initialization, generally yielding better results than a cold start, whereas GRPO performs better when trained from scratch. These findings highlight that PPO's gain from our reward design is limited compared the more robust and consistent improvements observed with GRPO,
Generalization Studies. We ate the generalization ability of our trained model in two challenging settings: unfamiliar scenarios and novel task goals (both from BFCL benchmark subset). Specifically, we test the model's performance in tool use within unseen programming lan- guages and its ability to detect irrele- vant tools, neither of which were ex- plicitly included during RL training or in the dataset. As shown in Figure 3 wwen2.5-3B-Instruce. when trained from scratch with our GRPO- reward design, consistently achieves highest performance. Additionally, Table illustratee two qualitative cases where the model proactively avoids inappropriate tool use: in one, by clarifying an ambiguous intent, and in the other, by answering directly without invoking tools. These behaviors demonstrate emergent proactivity and metacognition, leading to greater efficiency, reduced hallucinations, and signs of agentic intelligence.
Sort
27%
SFTRc
Programming Language
Ireeevarcee Deteection
(a) Unfamiliar Scenario
(b) Unfamiliar Goal
2een2.5-3--nsttruee performance across unfamiliar programming languages and novel relevance detection task goals.
Figure
Free-form Inference Effectiveness. While our model is trained with fo- cus on tool call format and correct- ness, we further evaluate its ability to handle free-form tool use in QA set- ting. Unlike the structured tool selec- tion and application tasks, QA setting: (1) imposes no constraints on tool call parameters, and (2)
Table 5: Performance comparison of our results on Bamboogle vs. Search-RI under GRPO and PPO training.
Model
Ours (GRPO)
Ours (PPO)
Search-RI RI (GRPO)
Search-RI (PPO)
5-3B
60.00
40.00
23.20
26
Qwen2.5:7B
72.00
48.00
40,00
36.30
evaluates only the final answer, making it a "goal-oriented" rather than a "process-oriented" task.
Specifically, we use Bamboogle, multi-hop QA dataset, to assess this capability. The model is equipped with a web search tool, and we report both the answer accuracy and the number of tool calls for all baselines and our approach. As shown in Table our reward design achieves the highest performance, despite this setting not being explicitly seen during training. Notably, our GRPO cold start model surpasses others in accuracy without relying on excessive number of tool calls. This suggests that the model can flexibly invoke tools when needed, effectively leverage feedback, wisely and efficiently navigating toward the correct answer. In addition, to validate the effectiveness of our method, we compared our training result with Search-R1 [7 in Table 5 The results demonstrate that our approach achieves significantly better performance than Search-R R1, which further confirmis our RL training approach's effectiveness.
7

---

3.4 Ablation Studies
To validate our experimental design, we conduct ablation studies on three key factors: the scale RL training data, the relative weighting of correctness rewards, and the effect of including or removing the KL divergence term.
Ablation on Data Scale. To investigate the effect of training scale, conducted ablation experiments by varying the number of RL training examples from
Table 6: Ablation training data scale using -nen2.5-3B-nnstruct BFCT.
4K to 10K while keeping the distribution, epochs, and GRPO cold-start setting fixed. As shown in Ta- ble 6, the performance of Owen2.5-3B-Inst. Instruct on BFCL remains nearly unchanged. with gains within
Training Setting
Performance
Original 4K RL data
52.98
Scaling 6K RL data (same distribution) Scaling
53.02
(same distribution)
$3.31
0.5 despite more than doubling the training data. We further observed that the reward convergence curves exhibit nearly identical speeds and final values across all settings. These results suggest that under our reward design, generalization in tool learning is more influenced by reward shaping than the sheer scale of training data. This highlights both the data efficiency of our approach and its practical advantage: strong generalization with limited data, minimizing training cost while maintaining performance, Accordingly, we adopt the 4K dataset as our main RL setting.
A Ablation on Reward Scale. To justify our choice of relative scaling between format and correctness rewards, we further conduct ablation experiments on BFCL using Qwen2.5 - trained with
Table 7: Ablation on correctness reward scale us- ing Qwen2.5-3B 1ane2.5-----nsstrue on BFCL.
Correctness Reward Scale
GRPO. Following Logic-RI RL [52], we fixed the maxi- mum format reward at 1.0 and varied the scale of the
BFCL Performance
40.62
51.07
52.98
correctness reward. As shown in Table setting correctness equal to format (scale = 1) leads to slower convergence and substantially lower performance. Increasing the scale improves performance, with the best result obtained at scale 3. This indicates that emphasizing correctness more strongly than format is essential for effective RL training in tool use, consistent with prior work where correctness is given higher weight than intermediate format signals. Accordingly, we adopt a correctness reward scale of 3 in our main experiments.
Ablation on KL Divergence. We removed the KL penalty across all RL settings, including PPO, to en- courage more flexible exploration and better adap- tation to our custom tool-use format. As shown in Table the impact on BFCL minimal, with Qwen2.5-31 showing nearly identical performance and Qwen2.5-7B slightly improving without KL. Beyond performance, we observed faster conver- gence, with training reaching stable rewards about five steps carlier, well as improved efficiency, with total training time reduced by approximately 1.5 times while lowering GPU cost. These findings support our design choice of removing KL in the trainings of the main experiments.
Table 8: Effect of KL penalty on BFCL perfor- mance, which shows minimal impact.
Model
w/o KL
w/ KL
0wen2,553B-IInttae 5-3--nsttt $2.98
53.05
Owen2.5-7BB-nnstenn
58.38
57.21
4 Analysis
In this section, we perform ablation studies to identify the most effective reward design for tool use, examining reward type, scale, granularity, and temporal dynamics. The original (ablated) setting refers to GRPO cold start. All experiments use BFCL benchmark, and we only report overall accuracy for simplicity. Full results could be found in Table 14 Table [5] and Table I6
4.1 Effect of Length Reward
To encourage more elaborate reasoning, we introduce length-based reward, motivated by prior findings that longer thinking traces can support deeper reasoning [5]. We investigate whether simply promoting longer outputs leads better task performance in tool use scenarios.
We consider both static and dynamic length rewards, defined as:
Lothink Laurget"
1):
Lihink
Renghhh min
Raynamic min
Langea(1 + p)
8

---

(a) BFCL Results (Length)
(b) Response Length
(c) Length Reward
Model
Overall Ace
Qwen2.5 1-150 Instruct (Origima)
46,29 %
Vean Langhh Reserdd
Qwea2.5.
(w) Length Reward)
33.23%
Qwea2,5-1 1.58 (Dynumic)
28.51%
Owea2.5.38 B--Jestcttt (Oviginal)
Quend. 5-9--Inddee Length Reward)
Ques2.5. 5-B--lrstrct (Dynamic)
Ove.1.00
1oma-3 Enstrac (Original)
44 10%
Instruct Length Reward)
Instract numic)
Step
Figure 4: (a) Overall accuracy across models and length reward settings, (b) Response length and (c) correspond- ing reward trends over training steps under the dynamic length reward setting.
(a) BFCL. Results (Scale)
(b) Format Reward
(c) Correctness Reward
Overall
Mean Comectress Reward
Owen2.5-1.90 Instrct (Original)
Mean Format Reward
Owen2.5 1.58 Instuct EEquil
Owen2.5 Instuct stage)
Owen2.5 1.50 Instrct (Dynamki
45.71%
Neen2.S-38-hnoree (Original)
$2.00%
Owen2. 5-8--horrd
51.70%
Owen2. D-enh---38-ledrrac
Owen2.5-39. 3-B-berrc (Dymmiic)
Orig Seo Stags Dynones
Onsine
Liama 5.2 38
(Original)
10%0
Stag=
Liama-3 3.2 38
(Equal man))
42.4%%
Dyrame
Llama 3.2.BB Jastruct (Dynamic)
Llama-322 3-2-78-lnsimmn (Tho stage)
46.85%
35%
22
Step
60
26
40
Step
66
Figure 5: (a) Overall accuracy across models and reward scale settings. (b) Format reward and (c) correctness reward trends across Owen2.5-3B-Instruus training under different scale dynamics.
where Tahink is the length of the <think> segment, Lurget is empirically chosen target length, and p E [0, 1] denotes normalized training progress. The static reward encourages reasoning up to fixed target length, while the dynamic reward gradually raises this target the training progresses.
As shown in Figure 4 both strategies successfully extend reasoning traces. However, the BFCL performance reveal that longer thinking does not consistently improve task success and often degrades performance. This suggests that while extended reasoning appears beneficial, can introduce unnecessary complexity, leading to overthinking and reduced effectiveness in tool use.
4.2 Effect of Reward Scale
We investigate how scaling reward components influences learning, particularly the balance between correctness and format rewards. Prior work [52] [17] emphasizes correctness over format to avoid reward hacking, where models exploit superficial patterns without acquiring real task competence.
We test an Equal Max variant that equalizes the maximum scales of correctness and format rewards. As shown in 5 this leads a slight accuracy drop across models, confirming that correctness reward should remain dominant to guide learning toward core reasoning abilities in tool use tasks.
Motivated by the intuition that different learning stages benefit from different focuses, explore two dynamic scaling strategies: Two--Stae approach divides training into two phases: early steps prioritize format learning by downscaling correctness rewards, while later steps reverse this emphasis to focus on correctness; Dynamic approach continuously interpolates reward seales throughout training, smoothly shifting focus from format fidelity to correctness training progresses.
Formal scaling functions are detailed Appendix H Figure 5 shows that abrupt shifts in reward emphasis (Two-Stage) degrade performance, while gradual adjustments (Dynamic) improve it. This suggests that smoother transitions help models better navigate from simpler objectives to mastering complex reasoning and tool use.
4.3 Effect of Reward Granularity
We finally analyze how the granularity of correctness rewards affects learning. Unlike tasks with definitive answers (e.g., math reasoning). tool use involves multiple facets, making reward assignment more complex. Our original finegrained design decomposes correctness into matching tool names, parameter names, and parameter values, providing a detailed, process-oriented learning signal.

---

(a) BFCL. Results (Granularity)
(b) Correctness Reward Trend
Model
Overall Ace
25
Qwen225 1.58 Instact (Original)
46.3%%
Quen 58-Ineract Fimegraiuna)
40-7 10-11/5
Qwad 5..55rrrr (otermediatt))
10een2.511100 1o--o---- (oarse)
30.72%
Qwen2.5 16 Onttact/OOiggaall
52.9%%
Qwe2 Dwen2.5-1B-Im-il ---trcc Finegrained)
53 06%
Qween15 3B Intrict (intermediate)
S1 3666
Liama 3.2 lastroct (Criiginad)
30oen2.3-31--1 -II--tttmcc (CppictCConn Cearse)
46.10 10
51, 40%
Mean
13
brikel Fregraned
Ulmu3 2.38--aattsss Fingrainnl)
Course
(Intermedlate))
Llana 3.2.383assieee
Stup
Figure 6: (a) Overall accuracy across models and reward granularity settings. (b) Correctness reward trends
across Qwen2.5-3PP --n-trruucttt training under different reward granularities.
To assess the impact of granularity, we compare three reward aggregation levels. Relative to the original (most finegrained decomposition) setting: Finegrained applies strict match constraints to both tool and parameter names without awarding partial credit; Intermediate merges parameter name and value correctness, rewarding only when the entire parameter dictionary matches exactly; Coarse treats the entire tool call-including tool name, parameter names, and values-ar as a single unit, granting reward only for perfect match. Detailed reward formulations are provided in Appendix [
As shown in Figure 6 coarser reward formulations result in lower reward attainment and slower learning progress due sparse and less informative feedback. This highlights that effective policy optimization benefits not just from stronger rewards but from strategically structured signals that guide the model through complex reasoning processes.
5 Conclusion and Future Work
In this paper, we present a reward design tailored for RL training on tool use tasks. Empirically, our model trained from scratch using GRPO consistently outperforms both SFT-based and SFT-initializer RL baselines across a variety of held-out tool use benchmarks. Furthermore, we demonstrate that model generalizes well QA settings, exhibiting robust multi-turn interactions, emergent proactiveness, and metacognitive behaviors, all of which are key traits for efficient and adaptable tool use, lying at the of foundational agent capabilities.
Our in-depth analysis reward types, scaling strategies, granularity, and temporal dynamics offers valuable insights into how reward shaping influences both learning efficiency and behavioral outcomes. Building on this foundation, future research could explore: (1) the impact model scaling in relation to reward design, (2) the adaptation of our reward framework to embodied agents requiring tool use, and (3) the effectiveness of multi-modal integrated tool use within our proposed ToolRL framework. We hope findings provide clear roadmap for advancing the application of reinforcement learning in tool-use scenarios. Ultimately, we envision that reward is all tool learning needs, positioning reinforcement learning powerful pathway toward developing agents capable of generalizable and creative behaviors.
Limitations and Broader Impact
While our study highlights the effectiveness of finegrained reward design in enabling tool learning, several limitations remain. First, our experiments primarily focus on policy-gradient methods such as GRPO and PPO, which are naturally compatible with structured, trajectory-level rewards; the generality of our approach other RL paradigms (e.g., preference-based methods like DPO or SimPO) has not been fully explored. Second, our evaluation is constrained to limited range of model sizes (up to 7B), leaving open questions about scalability both smaller lightweight models and much larger foundation models. Third, although our framework demonstrates promising gains in structured reasoning and grounded tool use, it also raises broader social considerations. More capable agents may enhance reliability, interpretability. and user alignment, but the same techniques could be misused if deployed without oversight, particularly in sensitive domains requiring trustworthy decision- decision-making. We view these risks as an important area for ongoing discussion and emphasize the need for responsible application. Overall, our contribution should be seen as an early timely step toward reward-driven agent training: by dissecting how specific reward components shape learning and generalization, we aim to provide both practical methodology and conceptual insights that can guide future work in building safe, adaptive, and grounded LLM-based agents.
10

---

Acknowledgement
This research is based upon work supported by DARPA ITM Program No. FA8650-23-C-77
the AI Research Institutes program by National Science Foundation and the Institute of Education Sciences, U.S. Department of Education through Award # 2229873 - AI Institute for Transforming Education for Children with Speech and Language Processing Challenges. Cheng Qian is supported by PhD Fellowship from the CapitalOne-Ilinois Center for Generative Al Safety, Knowledge Systems, and Cybersecurity (ASKS). The views and conclusions contained herein those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Govemmennt The U.S. Govermment is authorized reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
[1] Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang. William Zeng, Oussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani-Tür and Gokhan Tur. Can single model master both multi-turn conversations and tool use? coalm: A unified conversational agentic language model, 2025. URL httpp://////ttt httt:::://axxv.o/////e/// www/obg.///..................... haa.b//550208855
[2] Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07733, 2024.
[3] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:23 10.0599,, 2023.
[4] Nuo Chen, Hongguang Li, Baoyuan Wang, and Jia Li. From good to great: Improving math reasoning with tool-augmented interleaf prompting. arXiv preprint arXiv:2401.05530 2023.
[5] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint Siti::25030095661 2025.
[6] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv: 12588, 2022.
[7] Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint arXiv:2505.02338 023877 2025.
[8] Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-FLAN: Designing data and methods of effective agent tuning for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 9354-9366, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 18.18653/v 12224.findings--ee 557. URL
[9] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model trst-training. arXiv preprint arXiv:2501.17161, 2025.
[10] Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small 1lss: What works and what doesn't. arXiv preprint arXiv:2503 16219, 2025.
[11] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178, 2024.
[12] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint ar 2309:2309.14455 2023.
11

---

[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprint arXiv:2501 2025.
[14] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent 3d world. arXiv preprini ar3iv: 1.2871,, 2023.
[15] Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey-pan part 2: Surpassing ol op-preveww through simple distillation, big progress bitter lesson? arXiv preprint arXiv:2411.1648) 2024.
journey
[16] Yoshitaka Inoue, Tianci Song, and Tianfan Fu. Drugagent: Explainable drug repurposing agent with large language model-based reasoning. arXiv preprint M.Xi::2408.133788 2024.
[17] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Iims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025.
[18] Minki Kang, Jongwon Jeong, and Jaewoong Cho. T1: fool-integrated self-verification for test-time compute scaling in small language models. arXiv preprint arXiv:2504.047/8 2025.
[19] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 2023.
[20] Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming- Hasg- Yang, Phillip HS Torr, Salman Khan, and Fahad Shahbaz Khan. Lim tostnnraini A deep dive into reasoning large language models. arXiv preprint 2xXiv:2502.21221 2025.
[21] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102-3116 16, 2023.
[22] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling, arXiv preprint arXiv:2502.1 1886, 2025.
[23] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025.
[24] Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output-a reproducible pipeline. arXiv preprint arXiv:2401. S0t::2401.08/90 08/90, 2024.
[25] Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, et al. Hammer: Robust function-calling for on-device language models via function masking. arXiv preprint arXiv:2410.04588,, 2024.
[26] Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, al. Domain specialization as the key make large language models disruptive: A comprehensive survey. arXiv preprint 23Xv:23355.8800 2023.
[27] Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. Toolace: Winning the points of Ilm function calling, arXiv preprint arXiv:2409. 00920, 2024.
[28] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198-124255 2024.
[29] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint 2rxXi::3305.5334. 2023.
12

---

[30] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 1 126544-126565, 2024.
[31] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith. and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.033500 2022.
[32] Cheng Qian, Chi Han, Yi Fung. Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Tool creation for disentangling abstract and concrete reasoning of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6922-6939, 2023.
[33] Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao. Xiaocheng Yang, Denghui Zhang, et al. Escapebench: Pushing language models to think outside the box. arXiv preprint arXiv:2412.13549, 2024.
[34] Cheng Qian, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. Toolink: Linking toolkit creation and using through chain-of-solving on open-source model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Pupers), pages 831 831-854, 2024.
[35] Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Smart: Self-aware agent for tool overuse mitigation. arXiv preprint 1tt::2502.114335 2025.
[36] Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. 01 replication journey: A strategic progress report-part 1. arXiv preprint arXiv:2410.18986 2024.
[37] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint .30.05304 10, 2023.
[38] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, et al. Tool learning with foundation models. ACM Computing Surveys, 57(4):1-40. 2024.
[39] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models master 16000+ worldworld apis. In The Twelfth International Conference Learning Representations, 2024.
[40] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-5333 53741, 2023.
[41] Yusuf Roohani, Andrew Lee, Qian Huang, Jian Vora, Zachary Steinhart, Kexin Huang, Alexan- der Marson, Perey Liang, and Jure Leskovec. Biodiscoveryagent: An ai agent for designing genetic perturbation experiments. arXiv preprint arXiv:2405.17663) 2024.
[42] Timo Schick, Jane Dwivedi-Yi Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 68539-68551, 2023.
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXlv:1707.06347, 2017.
[44] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03332 03300, 2024.
13

---

[45] Haozhan Shen, Peng Liu, Jingcheng Li, Chumxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. VIm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025.
[46] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409. 10000922200 19256, 2024.
[47] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang. and Ji-Rong Wen. RI R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025.
[48] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
[49] Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm. gwt.tb.ttooooooo ho/blo//////ooooooooooooooooooot http//lll///we...50 qwen2.5/ 5/
[50] Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, et al. Freshlims: Refreshing large language models with search engine augmentation. arXiv preprint 2rXiv:2310.032) 2023.
[51] Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang. Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, et al. Autosurvey: Large language models automatically write surveys. Advances in Neural Information Processing Systems, 37:1151 19-111145, 2024.
[52] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong. Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with cule-based reinforcement learning. arXiv preprint arXiv:2502. /4768, 2025.
[53] Shunyu Yao, Jeffirey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
[54] Yining Ye, Xin Cong, Shizuo Tian, Yujia Qin, Chong Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Rational decision deeision-making agent with internalized utility judgment. arXiv preprint ArXiv:2308. 12519, 2023.
[55] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.1441 2025.
[56] Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang. Steptool: A step-grained reinforcement learning framework for tool learning in 1lms. arXiv preprint 4rX1v:2410.07745. 2024.
[57] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiiv2500..0111 2025.
[58] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. AgentTuning: Enabling generalized agent abilities for LLMs. In Lei-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 3053-3077, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1861386511122 fi00244.innigg--a...8. URL 2024. findininndnggs-.. acl. 181/
org/
[59] Yuanzhao Zhai, Tingkai Yang, Kele Xu, Feng Dawei, Cheng Yang, Bo Ding, and Huaimin Wang- Enhancing decision-making for Im agents via step-level q-value models. arXiv preprint arXiv:2409.093.2 2024.
14

---

[60] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2337..02445: 2023.
[61] Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv: 2409.03215, 2024.
[62] Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprini arXiv:2504.00 200.55044000000 03160, 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our main technical contribution on reward design is presented in Section 2.3, and our claims are further supported by the primary experiments reported in Section (Tables I, 2, and 3).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 5, we discuss the limitations of our current work, which also outline potential directions for future research building on our study.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not provide theoretical proofs, but demonstrates empirically effective models enabled by our proposed reward design for tool use.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We discuss the experiment settings in Section 3.2, with full details in Appendix
E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will publicly release all the code through Github. In the paper have already indicated the instructions we employ during RL training in Appendix D.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
www.limmssioooooiiio.ooooo.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. www.cc/pwwcccccccceeceeeoooooece http://wdiesssee www.eueiiissioonnooooooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the detailed experiment settings in Section 3.2 and Appendix E. The detailed hyperparameters settings are also provided in Table 5 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experimental figures include error bars represent variability during training. Most of the results presented in the main tables show that our proposed settings outperform the baselines, and statistical tests confirm the significance of these results. supporting our claims.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide our training GPU device and training steps in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp::///ourrrp..cc////llltttte
Answer: [Yes]
Justification: We have made sure to preserve the paper's anonymity and its conformation with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the paper's broader impact the RL and agent community at the end of Section I and 5, following our work's contributions.
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The RL training on tool use capability does not impose such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited the source of existing datasets and benchmark, RL training framework and RL algorithms across Section 3.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our work does not release any new dataset. The trained model through ToolRL framework will be released and its training details can be found in Section 3.1, 3.2 and Appendix E.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not include human subjects or crowdsourcing throughout the research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not include human subjects crowdsourcing throughout the research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core contribution of TooIRL reward design and analysis does not involve LLM as core component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) hett:s///eerrrps.ccc//ooerrreeee / for what should or should not be described.
22

---

Appendix
A Motivation Details
Training model's 's tool selection and application abilities using SFT often suffers from limited generalization. As shown on the left side of Figure 7 model trained with SFT on deep-thinking trajectories tends to over-interpret tool usage, failing reject inappropriate tools. Instead of engaging in genuine deep reasoning, it merely imitates superficial cues such as "but wait." This highlights the importance of using reinforcement learning (RL) guide the model toward more principled Decision-making and deeper reasoning capabilities.
Tark Loadl Iusieoal ie dl ingell
Whet's Fraecten
(teiiied Thrnght
Cranci in
of sllo beng
As toplu
sill Adn fae lorad Angiln
lnveles lovehe las
Figure 7: SFT on distilled deep-thinking trajectories leads to overthinking and poor generalization (left). In contrast, RL with our proposed reward design achieves consistently higher performance, with reward curves demonstrating rapid improvement during training (right).
B
Related Work
Tool-Integrated Reasoning of LLMs. Tool Tool-integrated reasoning (TIR) has emerged as a promising approach enhance the capabilities of LLMs. Early studies introduced the concept of equipping LLMS with external tools to overcome their inherent limitations [42
such as program ex- ecutors 6 and search engines [50]. To systematically assess these enhanced capabilities, several benchmarks have been proposed to evaluate tool use performance across various dimensions, includ- ing API selection, argument generation, and generalization [39 29 33). Building on this foundation,
38 53
subsequent research has focused on constructing high-quality tool use datasets [27 355, enabling models autonomously create and invoke tools [3 34 and applying these techniques problems spanning different modalities (45) and specialized domains [26). More recently, reinforcement learning (RL) has been explored as an effective framework further improve TIR, demonstrating
success in tasks such as information retrieval [17] and math computation [23]. These advances collectively highlight the growing potential of tool-augmented LLMs for general- ppurpoe reasoning
in open-domain settings.
Exploration of RL LLMs. Previous work has primarily relied on supervised fine-tuning (SFT) with carefully curated datasets to enhance LLM performance in tool use 42 39. Recently, reinforce- ment learning (RL) has gained traction as a more scalable and generalizable training paradigm. The development of RI methods for LLMS has evolved from reinforcement learning from human feed- back (RLHF) [19] and proximal policy optimization (PPO)
433 to more advanced techniques such direct preference optimization (DPO) (4)), SimPO 228 and group relative policy optimization (GRPO) 44. Extensions like dynamic sampling policy optimization (DAPO) (55) and the more recent value-based augmented proximal policy optimization (VAPO) [77 further improve training stability and efficiency.
Among these, GRPO 44 is specifically designed for LLMs, replacing the traditional critic with group-based evaluation strategy. has shown strong performance in enhancing reasoning abilities across range of tasks, including mathematical problem solving 44 52
search engine interac- tion [17 47]. and code generation [23]. Beyond task variety, recent studies have analyzed the
influence of dataset scale [22] and GRPO's effectiveness in smaller model settings [10]. GRPO's
23

---

flexible reward function enables adaptation to diverse objectives, such as assigning weights to sub- tasks [56] or constraining tool use frequency [3]. In this work, extend GRPO to enhance general tool use capabilities, improving LLMs' ability to select and interact with external tools across wide range of scenarios.
C Algorithm Details
We employ GRPO as our standard RL training setting, a variant of PPO that introduces advantage normalization within grouped samples. This normalization helps stabilize training by reducing variance across samples that share a common input context. We continue to use the symbols defined in Section 2 and further let To represent the current policy.
Normalized Advantage Across Query Groups. For each query Q, its responses derived from the rollout form group GQ consisting of multiple responses and their corresponding reward values:
Go {A, (81, ri), (82, (Sa, [n)]
where A denotes the ground-truth annotation for Q, and each reward ri is computed as the sum of the format and correctness rewards associated with response 8,, i.e., r; Rioma (8,, A) + Reorrea A). For each group, we calculate the mean and standard deviation of the rewards:
nQQ
QQ
Σ
( m2)²
Then, for each sample s; in the group, define the normalized advantage:
A. (Q) Ti P4Q
σQ + +
where 1) a constant avoid division by zero.
Policy Optimization Objective. The policy те is optimized using the standard clipped PPO objective, adapted with our group-wise normalized advantages:
To (s:
JaRrO = BLNDEE,N min
(s:()))
Tolal (s:(Q)
(a(s:1Q)
clip(
1, +,
1-
Told (ss QQ)
Overall, this objective guides the policy to generate structurally consistent and semantically accurate tool calls, while group-win wise normalization mitigates reward variance across queries, leading to more stable and sample-efficient alignment with task-specific response requirements.
D Prompt Details
The system prompt we employ during the rollout is shown in Figure 8 The user prompt is used to store the trajectory history, including intermediate thoughts, tool calls, environment observations, and any additional user commands. The complete user instruction is presented in Figure 9
E Experiment Details
Training Data Details. We empirically use 4K data points for training, as each dataset consists of samples drawn from the same distribution. Adding more data similar nature does not increase task diversity. Moreover, we observe that increasing the dataset size beyond 4K does not yield noticeable improvements the training convergence or final performance, suggesting diminishing returns from additional data under this setting.
24

---

Syslem Prompt for Training
are helpful dialogue assistant capable levernging
provide responses.
Available Tools
response, you the following lools:
((Tool List))
Steps for Each Turn
Think: relevani context analyze current user goal.
Decide on Tool Usage: Ia tool is needed, specify the tool and its parameters.
Respond Appropriately: response needed, generate one while maintaining consistency across user queries.
Output Format
<thinks Your thoughts and reasoning <ithink>
<tool_ calls
I"name"; "Tool name". "parameters" ("Parameter name": "Parameter content", " . ".... ("name": ".. "parameters"" :
[" ("...."
...]
<hool calls
<response>>
response </resqonse>
Important Notes
You always include the <think> field outline your reasoning. Provide least of <tool_call>> cresponse>, Decide whether to use <lool, call> (possibly multiple times), cresponses, or both. <tool calls fields. Fach tool call should JSON object with "name"
You invoke multiple tool calls simultaneassy field and a "parameters" field containing dictionary of parameters. If no parameters are needed, leave the "parameters" field an empty dictionary.
Refer the previous dialogue revords in the history, including the user's queries, previous callo, <response>, and any tool cobso exists).
Figure 8: The system prompt used for TIR's rollout.
User Prompt for Training
Dialogue History
<user>
[[ [nitial User Input
I| </user>
<thinks Round Model Thought </thinks
Round madel output stool, calls cresponse> ]]
<obs> Round Observation <lobs>
User Input <luser>-
Figure 9: The user prompt used for TIR's rollout.
GRPO Setting Details. For all the tool calls in the dataset, we all use JSON format represent tool call as it's easy to parse and is the most general and structure way of performing tool call. For the GRPO training, we 2 A100 (80G) GPUs per run with the hyper-parameters shown in Table
PPO Setting Details. We apply approximately the same parameter settings GRPO for the PPO training. Similarly, we use A100 (80G) GPUs per run with the hyper-parameters shown in Table T0
Baselines. The 400 selected data points used for SFT share the same distribution as the 4k data points used for RL training, but differ in content. For SFT, each data point includes <think> field, with thought content distilled from Deepseek-R1 trajectories. In contrast, GRPO does not require ground truth thought, only the tool calls are used to compute rewards in the GRPO setting.
We use 400 data points for SFT based on empirical observations that this amount is sufficient to help the raw model learn to follow our tool call format. This provides stronger initialization and reduces the burden of learning the format from scratch during RL training. However, we also find that relying solely SFT can lead to overfitting, which may ultimately degrade performance.
25

---

Table 9: Full configuration for GRPO training.
Table 10: Full configuration for PPO training.
Category
Hyperparameter
Category
Hyperparameter
Data Configuration
Data Configuration
Train Batch Size
512
'Train Batch Size
512
Validation Batch Size
128
Validation Batch Size
128
Prompt Length
Max Prompt Length
1024
Max Response Length
1024
Max Response Length
512
Optimization
Optimization
Leurning Rate
le-6
Learning
1e-6
PPO Mini Batch Size
128
Critic Learning Rate
1e.5
K1. Loss Used
False
PPO Mini Batch Size
PPO Micro Batch Size
Rollout Configuration
Rollout Configuration
Rollout Name
vllm
GPU Memory Utilizztionn
0.6
Rollour Name
vtlm 0.3
Number of Rollouts
GPU Memory Utilization
Training Logging
Training Logging
Save Frequency (Steps)
15
Save Frequency (Steps)
15
Test Froquency (Steps)
Test Frequency (Stops)
Total Epochs
15
Total Epochs
15
Table 11: BFCL v3 Benchmark Results (Additional Results with 4K SFT Initialization)
Model
Overall Nend Ace
Ive Farz Acc
Art
Malti Tur Aze Relecanee Detection
Innlecane Devasius
Qwen25.1 5- listuc (SFT PPO:
42.05%
77.65%
69.75%
55.73%
1.85%
100,00%
43.40%
Qwen? 5-1 S8-Irammel (SPT 4M0GGRP
40.955
70.54%
60.795
$6,559
1005
94.44%
Qwen2. 155 It---tsmeet (SFT4K-PPO)
40.24%
66.42%
02.02%
54.58%
2.50%
94.12%
55 (9.09%
Qwen2. 1.58 Instrcc GGPP)
42.67%5
66.00%
64.77%
60.15%
1,385
$5,555
07.98%
Qwend 5-W0-raimm (Raw))
13.1455
47.52%
40.015
$1.955
10%
61.11%
Qwen 5-38-Iraimss (SFT00-PPO)
78.29%
71/09%
51256
$4
Qwen2 lestrx (SP1300-GKPO)
64.15%
71.7688
0wen2.5.3--tramme
73.18%
5.25%
Qwen2 Itstrot GGPP)
82%
09.5256
77.7556
70.10%
Quen2.5 7B Iestrct (Ruw)
41.97%
66,62%
70.11%
53.51%
4 2.55
76.47%
02.06%
Qwend 5-78--ramme: (SFT (SFT4H-PPO)
42.02%
63.00%
72.62
$1.549
0.25%
1006
20 AMA
Quen2.5 5- Instrtt (SP1-00-GRPP)
2315
M0.99%
5455
25.51%
2316
1006
14.19%
Qwed 5-78-Iramms PPO)
42.67%
49.5019
$1.80%
77.78%
5-0-tt2-00 (SPI))-GRP))
77.78%
Liamo 3.2 Insnuct (Raw)
12.0915
17.45%
14.57%
43,85%
0.0055
77.754
66.07%
1011-122-B--ndmmett.m (SF PPO,
41.625
68.109
69.88
52.98%
3,005
14.12
56.29%
Liama 1.2198 Intruct (SF T-R0-GRPO)
42.5455
60.150 1155
05.9655
59.4055
0.3245
72.22 22.
65 A07
Hamm-3.2-38-fmmal (SF PPO)
45.4%%
73.71%
68.46%
62.27%
2.506
82.35%
08.72 75.75%
Liam 3,2 Ii-ariaruii (SF T4k.GGPOO
45.50%
70.69%
67.70%
64.73%
1.0056
77.78
78.85%
Table 12: API-Bank Test Results (Additional Results with 4K SFT Initialization)
Table 13: Bamboogle Test Results (Additional Re- sults with 4K SFT Initialization)
Model
Overall
Level
Level
Moneel
Nun Tool
Qwen 5-1 1aa....-..5B-nn4rrrr 58-ineraatt (Raw)
30.55
28.32%
35.825
35.11%
Owen2.5 5: 1.
(Raw)
0.61
Qwen2.5 1.55 Insract (SP740--PP:
5T.12%
60.9%
5UL73%6
48.35%
Owend 5-1 Eevirnc (SF 3ST4404PPO
36.8%
1.06
Qwen2.5-1 ins-ineract (SPT408-GKPU)
01.3116
56.159
55.2116
S4.219
Owen2 5-1 1. Basines (SF TH00&GRPO)
38.4%
0.96
0wen2.5-1.58 5--estt PPO)
61.31%
54.11%
56.72%
52.67%
Owen2 5-1 58--bemma (S) T4K+PPP)
$5.87
1.06
Qwent 5-1 38-Inersc (NPM))
59 59-4655
05/159
53.735
45/06%
Qwent 5-1 58-baemms (SF T4k+GRPO)
$4.45
1.02
Uwen2.5-J0 Instmeett
Orven2.5-11-- Mtl-mmttttt (Raw)
51.05
1.77
PPP
67.92%
55.22%
61.33%
Ower? 5-10-mmeree (SB PSPT))
41.29
1/06
Qwen2.5.3B 5-B-it5-3B--nttuuu1..... SPT400
58.21
45.50%
Qwea2.5.35 Insruct CSPP440-CRPO)
0.99
Qwen2.5-10-- (en72.5-11--ndmnn PPPO
6311
54.415
7856
$4.969
Owen2.5.30 Instuct (SPI4--PPO)
1.11
Quen2.5 3B 1i-.----nnnt///-- (nt.nnnttETTttecco. (GP4
60.00%
64.41%
56.72%
51.91%
9-90225--99-nssrruus (SPT4.-GRPO)
Qwen2.5-7B-Bh BrstT--IIsttctBBB. Raw)
62. 03.455
70.65%
49.25%
44,27%
Qwen2 5-78-Inaruct (Raw)
09.69
1.42
Qwend (-10--Inamcctt/PP (PP)
1516
77.45%
58.2156
17.4%
Qwen2 5-78-Inaruct (SPT-40)PPO)
45.67
3.54
Qwnn 5-78-Indmec1 (PP)))
54. 1056
61.4
$3.24%
50.42%
Qwen2.5 5 7B Instrnct (SF CSPP))
29.6%
3.70
Qwen2.5 (1ttct//P444:PPPP0
Q0en2.5-78-lnedeee (SPT4L+RRP))
59.30%
51.40%
34.33%
61.60%
Qwesd BeesB.5-7B-Gnstrurr (SB (SF14L-GRPO)
Owen2.5/70 Instrnct (SF ISK-
40.05%
$2.05%
1.25
1.35
Liuma 3.2 dastruct PSPT
JJme- 22Aii-aaeims (Raw)
40 5415
$7.79%
63.16R
44 13.85/
47.76%
79.8494
46.56%
33.42%
Limn 3.2.30 38 Instruci (Kam)
SFF4
54.5%
$9.25
1.25
1.35
Liama 3.2 23 lastrutt (SF1300-CRF))
56.79%
$4.10%
13 53.60R%
60 65/65
41.79%
40. 505
43.51%
41.22%
Mama-322-9--assmmm (SF400-CPPP
1.00
Liama 3.2 Instroot (SFIEE
500526
19.15%
34,33%
54.35%
Hama- 2-M8-Intmmt TRR-GG))
F Additional Result Details
We present additional results on three benchmarks, applying GRPO and PPO methods to models initialized with SFT on 4K data points. This setting serves as "theoretical" upper bound, since the same 4K data is first used for SFT and subsequently reused for RL training.
26

---

Table 14: BFCL V3 Benchmark Analysis Full Results (Length)
Live AST Aec
Live Esex
Tam
Relevaace Desccio
Erelerance Desection
Qwee2.5: 1.55 listn: (Original)
46.20%
77.96%
76.98%
60/73/c
1,25%
100000%
56 $6,44%
Qwes2.5 5-1.5 Instn owl Length Reward:
33,25%
70.58%
71.35%
35.63%
0.50%
94. 63.48%
4,52%
Quer? S-1 18--mmmm (bynamic)
24.1144
55-23%6
48.7V5
14.07%
0.005
56 5M6
35 0B9
Uwer2 fii-inaae (Original)
$1.W%
81 5416
75.355
71 187
3758
24146
14
Qver25.3B-lian S-1-iinnn Length reward))
43.90%
77
78,11%
63.56%
4,50%
Liams-32 Insract (Original)
44.00%
74. 55.35%
75.08%
56 86%
1.74
54 48%
62 23%
1Jme-112 38 intruct (wr Length reward)
46.919
78 1215
77.54%
56.55%
1,258
1000
63 7016
Llams-32 3B fnaruct (Dynaiic)
43.15%
75.306
71,64%
55,00%
1.00%
100,00%
$2.82%
Table 15: BFCL V3 Benchmark Analysis Full Results (Scale)
Overall
Non AST Acc
Nen Exec Acc
Live
Multi Tom
Reievance Detection
Inmlevancee Desection
Owea2.5 5: 1.5 (Original)
46.20%
77.90%
76.95%
50.731
2,25%
100,00%
56,44%
Owen2.5
(Equal maxi
39.
55.
75.50%
45.45%
100.00%
Owen2:5 5: 1.5
(Two
28.85%
999
13%
44.31%
2.25%
100.00%
10.01%
Quen2 2.1.225-5-5---.. 3B-iaiiii (Dynamic)
45.71%%
76.319
75.75%
$8.91%
2.505
100.00%
17.30%
Owes2.5 58 Insract (Original)
52 945
$1.55%
79 43%
73.78%
3.75%
$8,2416
$4,85%
Qwes2.5-3B-bne h-B8-b (Equal mox)
31 1672
81.500 505
50%
09.79%
4.25%
88,59 $8.996
78.07% 9
Qwer2 S-B--nnaa
Intruct (Tho stage)
50 66.66
80.628
78.82%
67.93%
3.50%
$3.996
76.42%
Qwes2 Inuruct (Dynamic)
53.51%
81.44%
$0.75%
75.43%
3,628
77.78%
$8.92%
Lhmn-11 1i-s-llmmml (Original)
10%%
75.18%
$6.869
Limn-3 22-1tmls EEqail max)
42.4795
75.05%
1.008
Laam-1
stage)
(Dynamic)
83.009
Table 16: BFCL V3 Benchmark Analysis Full Results (Granularity)
Model
Overall Ave
Naus-Live AST Aue
NunsLinc Essr Aue
Live Ace
Malti Tur Ace
Relevance Detection
Inclevance Detection
Qwen2.5 5: 1.58 lsstrect Orrigiaal)
46.20%
77 77,955
75, 15,95%
60.75%%
2.25%
100,00%
55,4415
Qwer2 5-1 S8-Jtaiieec (Finegrained)
40.719
78,00%
75.55%
48.91%
2,00%
100.00
24, 8455
Qwen2.5 5- 1,50 Instrett (Ineemmdiaate)
37 27.8815
77.94%
72.45%
430019:
1.62%
100,005
12.35%
Qwent 1.n.2.-1.35 1.50l--rsmmett (Corsse)
36, 76.7216
76 76.94%
10.30%
41.27%
2.12%
100,00%
12.24%
Uwen2.5 18 Instrct (Original)
81.65%
69.666
60.31%
5.50%
Owen2. 5:18
(Talirrmddiate)
ST, 36.
81.15%
50.07%
68.64%
4.25%
$8.89%
Qwen2.
(Cearse)
40%5
78.56%
88 85.89%
Liuma 32 3B Instrctt (Original)
48.10%
74.38%
75.18%
56.81%
1.3%
91, 91,1415
02 23%
Liama 12: 1-2-9B--0 Insrict (Pinegraimed)
39 8212
66.715
70.8%
$2.20%
0.25%
100,001
50.00 091661
Lhmu-312.33 1.2-3B-1aa B-strutt (lotermedistee
59.8355
71.36%
50.56%
0.25%
94.44
33.689
Llamx-3.2 3.2.BB Insruct (Coarse)
35.05%
$2.005
51.43%
48.96%
1.12%
83.33%
61.92%
The results are shown in Table 11 Table T2 and Table 13 for BFCL, API- and Bamboogle, respectively. We compare RL training initialized with models fine-tuned on either 400 or 4K SFT data points. Interestingly, our findings suggest that initializing from model finetuned on 4K data does not consistently outperform initialization from a model finetuned on only 400 data points. In the BFCL benchmark, we even observe cases where performance drops below that of the raw instruct model. This counterintuitive result may stem from overfitting during the SFT phase, which could restrict the model's exploration during RL. and lead to poorer generalization on held-out tasks.
G Length Reward Analysis Details
We examine the role of a length-basee reward. Prior work has demonstrated that the RI-like models can promote deeper reasoning, often reflected in longer thinking traces. To encourage this behavior, we introduce a reward term proportional to the length of the <think> field:
Lohink Liarget
Reength = min
where Luhink denotes the length of the thinking segment in model's output, and Luarer denotes the target output length, which empirically set to 512. We found that the raw model rarely generates responses longer than half this length, making 512 a reasonable and effective target for encouraging longer outputs. This length-based component is added to the overall reward, which now consists of format, correctness, and reasoning length.
As shown in Figure 4 both response length and the length reward generally increase throughout training, particularly for the Qwen model series. This indicates that the length reward effectively encourages longer reasoning. However, the downstream BFCL. results reveal that adding length reward does not consistently improve task performance, and smaller-scale models, it can even cause substantial degradation. These observations suggest that while extended reasoning may appear
27

---

desirable, it is not always beneficial for tool use tasks. In fact, excessive length may introduce unnecessary complexity, leading overthinking and reduced effectiveness.
Dynamic Length Reward. Since fixed-length rewards showed minimal impact and converged quickly, we explored dynamic length reward that adapts over training steps. Specifically, we define:
I Tuhink
Rdynamic = min
Liarget
(1 + p)
where S denotes the training steps and p
Samm
[0, 1] represents the normalized training progress.
Speal
This formulation gradually increases the target thinking length over time, aligning with model maturity.
As shown in Figure this approach yields steadier growth in thinking length, particularly for the Llama model. However, the BFCL performance results reveal that even scheduled rewards fail to improve performance. This further supports our hypothesis that extended reasoning may not benefit this task and can even have adverse effects.
Takeaway 1: While length rewards encourage longer reasoning traces, they do not consis- tently improve task performance and may even harm it in smaller models, highlighting that longer reasoning is not inherently better for tool use tasks.
H Reward Scale Analysis Details
We investigate the effect of reward scaling, specifically the relative weighting between correctness and format rewards. Prior work in RI-style RL commonly assigns higher weight to correctness reward than format reward, emphasizing the importance learning correct answer over superficial adherence to format. This strategy helps prevent reward hacking, where model might exploit formatting heuristics without learning task semantics.
To test the importance this design choice, we conduct an ablation where we equalize the maximum correctness and format rewards by setting the former's 's range to [-1, matching that of the format reward. This adjustment only affects the final normalization step of the correctness reward:
Recrrect 2 Rmax 1 [-1,]
Smax
where variables are defined as in Section 2.3
As shown in Figure 5 this equal-scaling variant, denoted as "Equal Max", results in slight drop in overall accuracy across most models, with the exception of Qwen2.5-3B, which maintains per- formance comparable to the original setting. These results underscore the importance of assigning greater weight to correctness reward: doing so helps steer the model toward mastering the core reasoning and tool use capabilities necessary for robust generalization.
Dynamic Reward Scaling. Building on the insight that correctness reward plays a more critical role, we are further motivated by the intuition that different reward components may benefit from being emphasized at different stages of training. This leads us to explore dynamically adjusting reward scales in accordance with training progress. Specifically, we hypothesize that in early training, the model should prioritize learning the correct output format, which entails an easier objective, before gradually shifting focus the more challenging goal of tool correctness. To test this hypothesis, we design two dynamic reward scaling strategies:
Two stage (Coarse) Setting: We divide training into two phases. In the first training steps, we downscale the correctness reward of its original scale while keeping the format reward at its original scale. After step s, we restore the correctness reward to its original scale and simultancously reduce the format reward to range [0, 0.5) ( of its original scale). Formally the reward scales are:
[0 0.5)
if Samen otherwise
Scalesomat
28

---

11,1] if Saureem <
Scalecorest
-33
otherwise
where Sturent denotes the current training step. In our experiments, we empirically set the switching point to 30 steps, as we observed that the format reward typically experiences significant increase within the first 30 steps. Therefore, it is more beneficial for later steps to shift focus toward optimizing correctness.
Dynamic (Finegrained) Setting: We apply continuous interpolation between the two reward scales throughout training. Initially, both the format and correctness reward scales are equally. Over time, the format reward scale linearly decays reward scale gradually increases to its original value, allowing the training to shift focus from format adherence to task correctness accordingly. Formally, the dynamic scaling is defined as:
original value, while the correctness
Scalenemar 1 p. p]:
Scale = 1-2 p.2
where E [0, 1] similarly represents the normalized training progress. This design ensures smooth shift of learning focus from format fidelity to correctness.
We present the reward dynamics of the original and two dynamic scaling strategies in Figure 5 The Two-stage (Coarse) reward setting unexpectedly leads to a drop in performance, whereas the Dynamic (Finegrained) scaling could improve model's benchmarking performance. These findings suggest that abrupt shifts in reward scale may negatively impact the training dynamics, In contrast, smoother and gradual transition from simpler objectives to more nuanced ones appears to better support the model's s learning trajectory and generalization during GRPO training.
Takeaway 2: Gradually adjusting reward scales during training, rather than abrupt changes, better supports model learning and generalization, highlighting the benefits of a smoother transition from simpler objectives more complex ones.
I Reward Granularity Analysis Details
We finally perform a detailed analysis of the effect of reward granularity, focusing specifically on the correctness reward. Tool calling, by nature, poses challenges for reward assignment, as it involves multiple facets beyond single definitive answer (e.g.. in contrast to math reasoning tasks). Our original reward design decomposes correctness into matching the tool name, parameter names, and parameter values, offering finegrained, "process-oriented" signal that reflects partial correctness in tool usage.
To assess the impact of this granularity, we evaluate three alternative reward formulations with progressively coarser levels of aggregation:
Finegrained: We apply strict exact-match constraints to both tool name and parameter name matching. Specifically, we define:
I Thame 1[No Nr] E [0,1]
Σ [[kys(PPc) Pe)s(PP)] e [G]]
T
GjEG
Intermediate: We combine the parameter name and value rewards into single term that enforces an exact match on the entire parameter dictionary. Formally:
Tpuas +
l'value
1[P 1 Pp] GG]
G,EG
Coarse: At the coarsest level, we fully entangle tool name, parameter names, and parameter values, treating the entire tool unit. Reward is given only if the generated tool exactly matches the ground truth:
Thmme
pamm
Fvalue
1G = P] E (0.
29

---

All other aspects of reward computation are kept identical to those described in Section 2.3 Starting from our original design, which is the most finegrained, we progressively entangle reward components to derive increasingly coarse-grand grained alternatives.
The reward dynamics across training steps, shown in Figure 0 demonstrate that as reward granularity becomes coarser, it becomes harder for the model to achieve higher reward values during RL training. This suggests that overly strict and entangled rewards may lead sparse learning signals, potentially hindering effective credit assignment.
Empirical BFCL test results further support this insight: original, most finegrained reward strategy performs well across models. In general, finer-grained reward decomposition leads to better training outcomes and higher final task performance, indicating its advantage in promoting more stable and effective policy learning.
Takeaway 3: Finegrained reward decomposition provides richer learning signals, high- lighting its role in enabling more effective training compared to coarse reward formulations, which can impede progress and degrade final performance.
J Additional Clarifications and Discussions
J.1 Scope of Our Claim
Our title, "Reward All Tool Learning Needs," " emphasizes the central role of reward design in enabling tool learning within reinforcement learning, rather than claiming the complete absence of supervised signals. We view RL reward itself as a form of supervision, providing structured and interpretable guidance that is particularly well suited to tool-use scenarios. While amall-scale supervised examples (e.g., 400-4K) can be used for initial format learning, our main method settings demonstrate that with the proposed format reward, models acquire correct tool-calling behavior purely through RI. training and, in many cases, surpass SFT-only baselines.
Furthermore, our reward design is algorithm-agnoste across different RL algorithms. In this paper, we adopt oolcy-grradeen methods such as GRPO and PPO, which naturally accommodate structured, multi-turn rewards and support trajectory-leve credit assignment. In contrast, preference- based approaches like DPO or SimPO are less compatible with this setting, they rely on pairwise comparisons that cannot easily capture finegrained signals such as tool name or argument correctness. Our choices align with prior work (e.g., Search-R R1 using further supporting their methodological suitability.
in principle, though its effectiveness may vary
GRPO/PPO, ToRL using GRPO) 17 23
Finally, while PPO achieves its strongest performance when combined with supervised initialization (e.g., SFT+PPO), our PPO experiments also demonstrate that the proposed reward signals provide meaningful improvements over SFT-only training. Taken together, these findings clarify the scope of our claim: reward design constitutes the foundation of effective tool learning by guiding both format and correctness, whereas supervised data can play a supportive but not fundamental role.
J.2 Choice of Customized Generation Template
In our experiments, we adopt a customized XML-tag-based generation template instead of relying on the native tool-calling mechanisms of L.MMs. Our preliminary comparisons on BFCL with Qwen2.5- 3B-Instruct showed that performance under the native tool-call format and our XML-based template was largely comparable. However, the XML design offers several important advantages that motivated our choice. First, it enables structured reasoning through additional fields such as <think>, which improves the model's 's capacity for explicit intermediate reasoning steps and aligns naturally with agentic paradigms such as ReAct (53), where reasoning and action are explicitly separated. Second, it provides flexibility and extensibility: XML tags can be easily adapted to incorporate new fields, supervision signals, or tool types, which crucial for generalizing to diverse domains and future research extensions. Third, this approach is consistent with recent works such as Search-R1 [17]. ToRL. (23), and RM-RI [7]. of which adopt customized generation formats rather than relying solely native tool-calls.
30

---

While models like LLaMA-3.2 -aastr-----nstruuet may exhibit strong zero-shot tool-use performance with native templates due to pretraining exposure, we emphasize that our XML-based template is model-agnostic and optimized for training-time flexibility rather than inference-ting execution. Native tool-call format, while convenient, are often rigid and less amenable to reinforcement learning setups that require finegrained, dynamic reward shaping. Our framework prioritizes consistency across models and adaptability to unseen tools and tasks, which we find more critical than exploiting architecture- specific optimizations. In this sense, the XML-based template supports both structured reasoning and broader applicability, making it better fit for our focus on reward-driven optimization in tool learning.
31


Reviews:
[{"review_id": "GVnGwBXybr", "reviewer": "Reviewer_j96L", "summary": "This paper presents ToolRL, a framework for training LLMs to use tools via reinforcement learning with a structured reward design. The authors propose decomposing rewards into format and correctness components, where correctness is further broken down into tool name, parameter name, and parameter value matching. They apply GRPO for training and evaluate on BFCL, API-Bank, and Bamboogle benchmarks, claiming 17% improvement over base models and 15% over SFT models.", "questions": "- Are there any specific challenges that adapts RL to tool-learning will encounter? If no, it seems this work just adapts GRPO into the tool-learning scenarios without bringing any new insights.  \n- What are contributions of each reward component to the end-to-end performance?\n- The method requires ground truth tool calls during training but claims to generalize to \"free-form\" scenarios. What is the mechanism enabling this generalization? The Bamboogle results suggest limited transfer, as tool call patterns do not significantly differentiate.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**  \n- The reward design dimensions (type, scale, granularity, temporal dynamics) is good for tool-learning fields. \n- Presentation is clear. \n\n**Weaknesses**  \n\n- Technical contribution is limited. Its proposed reward design seems not novel as it can be seen from many RL related works.\n\n- Some statements are over-claimed. It names its title as \"Reward is All Tool Learning Needs\", however, its training still requires supervised data for initial format learning (400-4K examples), which is not consistent with their claim. \n\n- The training scale is limited as it only trains on only 4K examples across three dataset, which makes the generalization less convincing.", "was_revised": true, "final_justification": "The additional experiments the authors provide about using scaling numbers of data and clarifications of the reward design mainly solve my concerns and I think it is good to be accepted to benefit tool-learning community.", "rebuttal": "> **Weakness: Technical contribution is limited. Its proposed reward design seems not novel as it can be seen from many RL related works.**\n\n**Contribution and novelty of reward design** \n\nThank you for raising this point. We would like to emphasize that our work is the first to **specifically address tool learning in RL from the perspective of reward design**. In particular, we propose a finegrained reward formulation across multiple dimensions of tool use and empirically demonstrate its effectiveness. Our contributions also go beyond the reward design itself, as we also provide in-depth analysis of how each component impacts model performance and generalization across diverse task types. We believe that the success of our RL-based tool learning paradigm can timely help advance the development of agentic LLMs, particularly in enhancing their tool use and reasoning capabilities.\n\nAs stated in Line 44-47, while concurrent works such as Search-R1 \\[1] and ToRL \\[2] explore RL for specific tools like search or code, their focus lies mainly in multi-turn rollout strategies rather than reward formulation. Additionally, these approaches are typically limited to narrow domains. In contrast, our method is designed for general-purpose tool learning, which requires a more versatile and principled reward design framework.\n\n\n\n***\n\n\n\n> **Weakness: Some statements are over-claimed. It names its title as \"Reward is All Tool Learning Needs\", however, its training still requires supervised data for initial format learning (400-4K examples), which is not consistent with their claim.**\n\n**The interpretation of our claim \"Reward is All Tool Learning Needs\"** \n\nWe'd like to clarify that our title emphasizes the central role of reward design in enabling tool learning within RL training. It does not claim the complete absence of supervision. In fact, we consider RL reward as a form of supervision signal. \n\nImportantly, our main method setting (Tables 1, 2, and 3) also **does not depend on SFT** (400 / 4K examples). With our designed format reward, LLMs can learn the correct tool-calling behavior purely through RL training and, in many cases, outperform the SFT baseline.\n\n\n\n***\n\n\n\n> **Weakness: The training scale is limited as it only trains on only 4K examples across three dataset, which makes the generalization less convincing.**\n\n**Training scale of ToolRL and its relationship with generalizability** \n\nThank you for raising this concern. From observations, we argue that generalization in tool learning is more influenced by the reward design than the sheer scale of training data. This is supported by the following:\n\n* **Performance:** Despite the reletively small training scale, our model generalizes well to held-out tasks and domains. API-Bank and BFCL (Tables 1 and 2) were not seen during RL training. Bamboogle (Table 3) represents a QA setting with different tools and task formulation, reinforcing our claim. \n\n* **Data Efficiency:** Current perfomance also inidcates our reward design's strong generalization with limited data, which we believe is a strength and a practical advantage instead of limitation.\n\n* **Empirical Trials:** In our initial trial experiments, training with 6K or 10K examples (of same distribution, all 15 epochs, GRPO cold start) yielded performance nearly identical to the 4K setting. Specifically, the results of Qwen2.5-3B-Instruct on BFCL is presented in the following table. With a performance gain within 0.5, this suggests that further increasing the training data offers minimal benefit under our current data and reward design setup.\n\n| Training Setting                               | Performance |\n|------------------------------------------------|-------------|\n| Original **4K** RL data                        | 52.98       |\n| Scaling **6K** RL data (same distribution)     | 53.02       |\n| Scaling **10K** RL data (same distribution)    | 53.31       |\n\nIn addition, we observed from the training time reward curves that convergence speed and final average reward value remain nearly identical across the 4K, 6K, and 10K training settings. This further supports that simply increasing the amount of data from the same distribution offers minimal gains. Given these empirical findings, and considering both time and computational efficiency, we selected the 4K dataset for RL as our main setting. We will clarify these details more explicitly in the next revision.\n\n***\n\n\n\n> **Question: Are there any specific challenges that adapting RL to tool-learning will encounter? If no, it seems this work just adapts GRPO into the tool-learning scenarios without bringing any new insights.**\n\n**Specific challenges in adapting RL in tool learning** \n\nWe empirically encountered multiple unique challenges in applying RL to tool learning, especially since our work is among the first to do so beyond specific tools like search or code: \n\n* **What to reward:** Tool use tasks are both intermediate-reward-rich (e.g., correctness per tool call) and final-goal-driven. Designing rewards that balance both was non-trivial. For instance, we found in initial trials that rewarding only final answers harms generalization of intermediate tool calls, whereas rewarding tool calls improves both tool use and final outcomes.\n\n* **How to reward:** Tools have structural components, including name, arguments, contents. Through multiple failed attempts, we found that sparse or coarse-grained rewards led to poor generalization. Our current fine-grained scheme emerged from extensive experimentation. \n\n* **When to reward:** RL allows dynamic reward scheduling, which poses challenge on what is the best reward dynamic along the full training process. This is addressed in Section 4.1 and 4.2 where we analyze how reward timing and weighting influence training dynamics and generalization. \n\nOverall, we offer a comprehensive, finegrained reward design tailored to tool learning, validated both empirically and analytically. The Section 4 in our paper also yeilds multiple insights into what is the most effective training strategy.\n\n\n\n***\n\n\n\n> **Question: What are contributions of each reward component to the end-to-end performance?**\n\n**The contribution of each reward component** \n\nWe present detailed ablations of reward's impact on BFCL performance in Section 4.2, 4.3, and Appendix H & I. Below is a concise summary: \n\n* **Format reward:** This component ensures that the model’s outputs adhere to valid and executable tool call formats. Without it, models often generate invalid tool calls, particularly in the early stages of RL training, which leads to sparse or ineffective reward signals. For example, in our initial trials, Qwen2.5-3B-Instruct achieved near-zero performance on BFCL when format reward was omitted, causing RL training to collapse. Incorporating format reward enables the model to quickly learn the correct output structure. Moreover, as discussed in Section 4.2, gradually decreasing the weight of format reward during training helps the model shift focus from structural correctness to functional correctness, improving overall performance.\n\n* **Correctness reward:** We present a finegrained correctness reward design as our main setting, which provides supervision across multiple components of the tool call, including the invoked tool name, the corresponding argument names, and the argument content. Each component delivers explicit guidance to the model during training. As shown in the ablation studies in Figure 6, progressively removing or coarsening these components results in a gradual and notable decline in performance. This empirical evidence highlights the importance of maintaining finegrained reward signals to support effective tool learning.\n\n\n\n***\n\n\n\n> **Question: The method requires ground truth tool calls during training but claims to generalize to \"free-form\" scenarios. What is the mechanism enabling this generalization? The Bamboogle results suggest limited transfer, as tool call patterns do not significantly differentiate.**\n\n**Generalization of tool call on Bamboogle** \n\nThank you for raising this question, and we would like to clarify that the \"free-form\" here refers to the *process* of tool use being unconstrained, not the format. Tool call format needs to remain fixed to ensure every tool call can be correctly parsed and ensure evaluation consistency. \n\nOur mechanism for generalization lies in training with reward-dense, ground-truth-guided tool call trjectories. This enables the model to generalize to unseen tools and looser supervision (no ground truth tool call, only final answer is evaluated), as in Bamboogle. \n\nBamboogle tests two kinds of generalization: \n\n* **Domain/Scenario:** QA setting with different task structure. \n\n* **Tool:** Use of tools (e.g., search) not explicitly trained during RL. \n\nOur model's ability to perform well in this setting validates the effectiveness of our reward-based approach in guiding tool learning in open-ended environments and customized tools.\n\n\n\n***\n\n\n\nWe hope our responses have addressed your concerns clearly and thoroughly. If there are any remaining questions or points you’d like to discuss further, we’d be grateful for the opportunity to continue the conversation. **If you find our clarifications satisfactory, we would sincerely appreciate your consideration in reflecting that in your overall rating.** Thank you very much for your constructive feedback.\n\n\n\n***\n\n\n\n**References**\n\n> \\[1] Jin, Bowen, et al. \"Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\" *arXiv preprint arXiv:2503.09516* (2025).\n>\n> \\[2] Li, Xuefeng, Haoyang Zou, and Pengfei Liu. \"Torl: Scaling tool-integrated rl.\" *arXiv preprint arXiv:2503.23383* (2025)."}, {"review_id": "MlmxA8aDG4", "reviewer": "Reviewer_gizX", "summary": "This paper tackles the problem of multi-turn tool use with LLMs. It argues that SFT is not a sufficient solution to enable good performance in this setting. RL can enable the model to generalize and act better. The key to making RL work they argue is a good reward design which they provide a reward design for in this work (the main contribution). They conduct extensive experiments and ablations on their reward and aspects of RL training vs SFT to help confirm their hypothesis (second contribution).", "questions": "- Can you provide error bars for the accuracy numbers reported across the numbers? \n\n- Do you have any idea how the results might change if you use explicit tool calling mechanism of LLMs instead of the special tokens approach? Similarly if you adopt a ReACT style agent loop [1]\n\n[1] https://arxiv.org/abs/2210.03629", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Quality: The problem in this paper is well motivated. Multi-turn tool use is the template for modern agents. The paper experiments are comprehensive with sufficient baselines and on a good number of datasets with multiple models. The method is also sound. \n\nClarity: \nThe paper overall is clear but a bit hard to read in a few places:\n- I'm a bit confused about the terminology in 2.1, the choice of the LLM is a tool & its parameters jointly (not just the discrete tool), which would not fall into a discrete set of tools as the parameter set can be continuous. So the choice is tool + parameters. \n- I have only seen the term, Tool-Integrated Reasoning ,in [1] before, is there a reason for using this term instead of the more common terminology of multi-turn tool use, or LLM agent? I would suggest using the more traditional terminology to make the paper more friendly to readers. I don't see how this is different from multi-turn tool use i.e., agents.\n- I would avoid referencing an appendix figure and just include that in the main paper (Figure 9)\n- the experimental tables are too small and make it hard to read, while I appreciate reporting all the numbers, for the main paper I would pick a few of these and allude to the rest in the appendix. What does bold and underline mean?\n\nSignificance: The main contribution here is the reward design they have for tool use in addition to their experimental results. I believe this reward design has potential to be re-used by others.\n\nOriginality: \nThe main contribution is in the reward design, while this might not be flashy often times all it takes to make RL work is a good reward design. Empirically some groups might have implemented similar schemes but no published work exists with these formulas.\nSince the time of submission in May, there has been many numerous works on using RL for multi-turn tool use including frameworks for doing so, so that the claim \"Training LLMs for TIR tasks has predominantly relied on Supervised Fine-Tuning (SFT)\" is no longer applicable [2,3,4]. \n\n[1]: ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving https://arxiv.org/abs/2309.17452\n[2] StepTool: Enhancing Multi-Step Tool Usage in LLMs through Step-Grained Reinforcement Learning https://arxiv.org/abs/2410.07745\n[3] Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment https://arxiv.org/abs/2505.11821\n[4] https://sky.cs.berkeley.edu/project/skyrl/", "was_revised": true, "final_justification": "The authors have adequately answered my questions from the initial review. I don't have further questions.\n\nI still recommend acceptance, I think this paper will be very valuable for practically do RL for tool calling", "rebuttal": "> **Clarity: I'm a bit confused about the terminology in 2.1, the choice of the LLM is a tool & its parameters jointly (not just the discrete tool), which would not fall into a discrete set of tools as the parameter set can be continuous. So the choice is tool + parameters.**\n\n**Clarification on tool use trajectory formulation and terminology**\n\nThank you for raising this important point. We agree that tool use decisions should be viewed as including both the discrete tool identity and its associated parameters, where the latter lies in a continuous space. Our intent in Section 2.1 was to express that T refers to the entire tool use decision (tool + parameters), rather than just the tool name. We acknowledge this could be clearer and will revise our explanation in the revisions to reflect this formulation more explicitly. \n\n\n\n***\n\n\n\n> **Clarity: I have only seen the term, Tool-Integrated Reasoning, in \\[1] before. Is there a reason for using this term instead of the more common terminology of multi-turn tool use, or LLM agent? I would suggest using the more traditional terminology to make the paper more friendly to readers. I don't see how this is different from multi-turn tool use i.e., agents.**\n\n**Clarification on tool use trajectory formulation and terminology (continued)** \n\nWe adopted the term \"Tool-Integrated Reasoning\" (TIR) following the precedent set by both ToRA and especially ToRL \\[1], which formally defines the term. That said, your observation is accurate: the core concept aligns with multi-turn tool use or agent behavior. We chose \"reasoning\" to emphasize the intermediate thought process preceding each tool invocation, which is central in our formulation. Nonetheless, we recognize the benefit of aligning with more widely-used terminology and will provide clearer explanation and bridge to common terms in the revisions to enhance accessibility.\n\n\n\n***\n\n\n\n> **Clarity: I would avoid referencing an appendix figure and just include that in the main paper (Figure 9)**\n>\n> **Clarity: The experimental tables are too small and make it hard to read. While I appreciate reporting all the numbers, for the main paper I would pick a few of these and allude to the rest in the appendix. What does bold and underline mean?**\n\n**Referencing and table formatting problems** \n\nThank you for the helpful suggestions. We will move key figures such as Figure 9 into the main paper in the next revision. For the experimental tables, we'll restructure the presentation to focus on the most important comparisons in the main text while relocating extended results to the appendix. \n\nTo clarify: **bold** indicates the best-performing method in each block, while underline marks the second-best.\n\n\n\n***\n\n\n\n> **Originality: Since the time of submission in May, there has been numerous works on using RL for multi-turn tool use including frameworks for doing so, so that the claim \"Training LLMs for TIR tasks has predominantly relied on Supervised Fine-Tuning (SFT)\" is no longer applicable.**\n\n**Paradigm of tool use and concurrent works** \n\nWe appreciate the references to concurrent works and fully agree that the landscape of tool learning has evolved rapidly. In our next revision, we will incorporate discussions of these recent developments and refine our phrasing to accurately reflect the ongoing paradigm shift from SFT to RL in this area. Nonetheless, we believe our work remains among the first to systematically explore this shift through the lens of **reward design,** which is orthogonal and complementary to the agent frameworks introduced in these concurrent papers.\n\n\n\n***\n\n\n\n> **Question: Can you provide error bars for the accuracy numbers reported across the numbers?**\n\n**Error bar for results** \n\nWe'd like to clarify that we use temperature 0.0 for all evaluation runs, ensuring that model outputs are deterministic. Moreover, the environment feedback (tool APIs) is rule-based and deterministic, and our prompt templates remain fixed during evaluation. This setup leads to deterministic outputs with no variance across repeated runs.\n\nWe also verified that prior works on API-Bank and BFCL similarly report single-shot deterministic results. Given this, standard deviation or error bars from repeated trials might be meaningless in our setting. However, we have provided all hyperparameters to ensure reproducibility of the reported outcomes.\n\n\n\n***\n\n\n\n> **Question: Do you have any idea how the results might change if you use explicit tool calling mechanism of LLMs instead of the special tokens approach? Similarly if you adopt a ReACT style agent loop**\n\n**Comparison of tool call and special token approach** \n\nThis is a very insightful question. In our preliminary experiments, we compared Qwen-3B-Instruct's performance on BFCL using its native tool calling mechanism versus our special XML-tag-based template. The results were comparable. However, the XML-tag format offers significant benefits: \n\n* It allows us to introduce additional tags like \\<think>, which improves the reasoning capability of the model. \n\n* It provides greater flexibility and customizability for future extensions (e.g., adding structured fields). \n\nThis approach is also common among recent works such as ToRL \\[1], Search-R1 \\[2], and RM-R1 \\[3]. \n\nFurthermore, our framework shares characteristics with ReACT: the \\<think> tag captures the reasoning step, while \\<tool> tag corresponds to action, aligning with the ReACT paradigm in structure and intention.\n\n\n\n***\n\n\n\nWe hope our responses have addressed your concerns clearly and thoroughly. If there are any remaining questions or points you’d like to discuss further, we’d be grateful for the opportunity to continue the conversation. **If you find our clarifications satisfactory, we would sincerely appreciate your consideration in reflecting that in your detailed evaluation score or overall rating.** Thank you very much for your constructive feedback.\n\n\n\n***\n\n\n\n**References** \n\n> \\[1] Li, Xuefeng, Haoyang Zou, and Pengfei Liu. \"ToRL: Scaling tool-integrated RL.\" arXiv:2503.23383 (2025). \n>\n> \\[2] Jin, Bowen, et al. \"Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning.\" arXiv:2503.09516 (2025). \n>\n> \\[3] Chen, X., et al. \"RM-R1: Reward Modeling as Reasoning.\" arXiv:2505.02387 (2025)."}, {"review_id": "l2KSE6jPke", "reviewer": "Reviewer_iJFQ", "summary": "This paper presents a study on reward design for training LLMs to use tools through RL. The authors propose a reward framework that decomposes tool-calling formatting and correctness into multiple components and demonstrate improvements over SFT approaches approaches. The authors also go in-depth about various aspects of the reward - like scale, length, dynamic vs static etc. and perform experiments using GRPO and PPO algorithms on Qwen 2.5 and Llama 3.2 series of small LMs.", "questions": "## Questions\n\n1. Why is R_max set to 3? This seems like an important hyperparameter but no justification or ablation is provided for this choice.\n2. The authors mentioned that they removed KL penalty term in GRPO. In comparison with PPO baselines, do they also have KL term removed?.\n3. Line 157 states that removing KL penalty leads to faster convergence. Do the authors have any plot to support this claim? What are the performance differences between Qwen 3B Cold Start GRPO with and without KL divergence?\n4. How were the 400 datapoints selected? Randomly sampled out of 4K dataset or were the most difficult data points chosen? \n5. Regarding Line 208 \"We hypothesize that SFT initialization leads to memorization and overfitting, which reduces the impact of GRPO's effectiveness in generalization\": Isn't there any loss curve to establish this hypothesis? Otherwise could the authors verify this by seeing if SFT+GRPO plateau faster than cold-start GRPO? or maybe testing with tools that have identical functionality but different parameter names?\n6. What do the error bars represent in Figure 2? How many evaluations were conducted (how many random seeds per baseline)? Also how much smoothing and what type of smoothing has been done in these figures?\n7. If there is significant overlap in error bars, can we also report standard error over multiple evaluation runs in Tables 1, 2 and 3?\n8. Line 215 claims \"PPO tends to be less stable\" - where is stability shown empirically? In Table 1, all I can see is accuracy which shows SFT400+PPO > Cold Start PPO, but nowhere does it establish stability.\n9. Based on the PPO results, does this mean your framework is not widely applicable as you claim in the introduction? I would then want to see results on DPO or at least one more RL method, otherwise the authors should change their claims - especially the claim made in lines 53-56: \"While our reward design is algorithm-agnostic by nature.\"\n10. Line 221 states \"PPO benefits from our reward design\" - I don't think it's correct to say this when in most model sizes and evaluation datasets across Tables 1, 2 and 3, SFT400+PPO > PPO Cold Start. Therefore this line seems misleading.\n11. I have a curious question - do you believe that tool use didn't improve with extended reasoning could be due to smaller LM size? This phenomenon might not exist when using 7B parameters or beyond?\n12. Why no comparison with other recent RL methods like DPO, SimPO, or direct comparison with Search-R1/TORL on overlapping domains?\n\n\n## Suggestions\n\n1. Typos to fix:\n\nLine 105: \"out\" → \"our\"\nLine 109: \"indicates\" → \"indicate\"\nLine 152: \"employs\" → \"employ\"\nLine 153: \"later\" → \"latter\"\n\n2. It would be nice to add numbers to each equation in Section 2.3 for better readability.\n3. Please increase font size and legend size. Not visible when the paper is printed on A4 sheets - especially for Figures 3 and 7.\n4. Lines 258-259 state \"we introduce a length-based reward, motivated by prior findings that longer thinking traces can support deeper reasoning\" - please back this with appropriate citation.\n5. Include error bars and statistical significance tests for main results tables if you have run multiple seeds based on Figures 3 and beyond", "limitations": "The authors have not discuss neither limitations nor societal impact of their work. The authors claim in the checklist that they discussed Limitations in Section 5 but I can only see future works. \n\nThe authors should add a dedicated \"Limitations\" section covering things like limited evaluation on smaller LMs as well as a comprehensive \"Broader Impact\" discussion addressing potential misuse scenarios, safety considerations since this paper is regarding tool-use in LLMs.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is well-motivated and tackles an important problem in the domain. The study of reward design for tool use is novel and timely.\n2. If the claims hold true, this work could be very helpful for many downstream applications involving LLM agents. \n3. I particularly liked the comprehensive analysis presented in Section 4. The systematic exploration of reward design (length, scale, granularity, static vs dynamic reward, etc.) provides valuable insights. The finding that longer reasoning traces can actually degrade tool-use performance is particularly interesting.\n\n## Weaknesses\n\n1. There are several experimental details that are currently unclear (detailed in questions section below), which makes it difficult to properly verify the claims made. Key details about data selection and statistical significance are missing too.\n2. When the authors claim that their approach is agnostic to the RL algorithm used, this claim is not really supported by the results presented. So far it seems their reward design works well only with GRPO, and that too specifically with Qwen 2.5 models. The PPO results are consistently weaker, contradicting their broad applicability claims.\n3. Minor: There are several minor presentation issues, typos, and missing details that affect readability.", "was_revised": true, "final_justification": "Increasing my score to an accept, I had some reservations around the framing of this paper which the authors have now clarified.", "rebuttal": "> **Weakness 2**\n\n**Clarification on Reward Design vs. RL Algorithm**\n\nThank you for the concern. Our claim of RL algorithm agnosticism means the reward design is **orthogonal** to the specific RL algorithm: it can be paired with any optimizer in principle. However, this does not imply all RL algorithms perform equally well under it.\n\nWe adopt GRPO as our main method due to its lightweight nature and empirical advantage over PPO in tool learning tasks, as shown in recent studies [1] and our own experiments. We will further address weakness concerns through responses in the following.\n***\n> **Question 1**\n\n**Correctness Reward Scale**\n\nThanks for the question. We set the correctness reward max (R\\_max) to 3 based on empirical findings that a higher weight than format reward improves RL training. This aligns with prior work like Logic-RL \\[2] and is particularly effective with GRPO.\n\nIn trials on BFCL with Qwen2.5-3B-Instruct:\n|Scale|1|2|3|\n|---|---|---|---|\n|Perf|40.62|51.07|**52.98**|\n\nScale 3 led to the best performance and faster convergence. We'll clarify this design choice in revisions.\n***\n> **Question 2 and 3**  \n\n**KL Divergence Removal**\n\nWe removed the KL penalty in all RL settings, including PPO, to allow more flexible exploration and better adaptation to our custom tool-use format (Line 157). This decision is grounded in empirical findings under the VERL framework:\n- **Faster convergence:** Removing KL led to ~5 steps faster convergence in both format and total reward. Faster format reward gain suggests quicker mastery of executable tool calls.\n- **Comparable or better performance:** On BFCL with Qwen2.5 models, removing KL gave slightly better or similar results (Δ < 0.1):\n|Model|w/oKL|w/KL|\n|---|---|---|\n|Qwen2.5-3B|52.98|**53.05**|\n|Qwen2.5-7B|**58.38**|57.21|\n- **Efficiency:** KL-free training reduced total time by ~1.5×, cutting GPU cost.\n\nDue to rebuttal policy, we cannot link full curves but will clarify this design in revisions.\n***\n\n> **Question 4**\n\n**Justification of 400 SFT Data Choice**\n\nWe selected the 400 SFT samples **randomly** to ensure fairness and consistency across models. Defining the \"most difficult\" examples poses challenges:\n- **Ambiguity:** Difficulty is model-specific: many examples could qualify, or vary by base model.\n- **Fairness:** Choosing different hard samples for each model would break comparability.\n- **Scalability:** Hard-example mining would require re-evaluation per model, adding overhead.\n\nTherefore, we believe random sampling provides a reproducible, scalable baseline.\n***\n> **Question 5**\n\n**SFT’s Effect on Generalization and Reward Dynamics**\n\nThank you for the insightful question. Our hypothesis that SFT leads to memorization and limits RL driven generalization is supported by reward curve patterns (Fig. 2a,b) and consistent with recent findings [3]:\n- **Reward dynamics:** SFT+GRPO shows a **flatter curve** than cold-start GRPO, suggesting early plateau and limited benefit from reward-driven learning.\n- **Generalization gap:** Though SFT+GRPO reaches a **higher training reward**, it performs **worse on all test benchmarks**, where no RL data overlaps. This suggests SFT may lock the model into local optima.\n- **Pure SFT underperforms:** This reinforces the idea that it promotes memorization and hurts adaptation.\n\nWe also reviewed **loss curves** via WandB: cold start GRPO shows more noise but lacks clear plateau signals. We believe reward curves proved more informative for evaluating generalization.\n***\n> **Question 6 and 7**  \n\n**Clarification on Error Bars and Evaluation Variance**\n\nThanks for the questions. The **shaded regions in Figure 2** reflect **variance of the raw training curve (before smoothing)**, not across seeds. We use WandB’s EMA smoothing (factor 0.99). The shaded area captures training instability or noise over time.\n\nOur **evaluation is deterministic**:\n- Generation temperature = 0.0  \n- Tool feedback is rule-based  \n- Prompts are fixed\n\nHence, repeated evaluation yields identical results, and standard error over runs isn't meaningful.\n\nThat said, we ran 3 preliminary seeds for GRPO cold start on BFCL:\n|Model|Seed1|Seed2|Seed3|Avg|StdDev|\n|---|---|---|---|---|---|\n|Qwen2.5-3B|52.98|51.47|53.19|52.55|0.94|\n|Qwen2.5-7B|58.38|57.02|59.21|58.20|1.11|\n\nLow variance and consistent trends still support our findings. Due to compute limits, we didn’t run all configs with multiple seeds but will clarify these settings in the revision.\n***\n> **Question 8**\n\n**PPO Instability Justification**\n\nOur claim about PPO being \"less stable\" refers to two observations:\n- **Ranking inconsistency:** GRPO ranks top-2 across all benchmarks (Tables 1–3), while PPO varies more. For Qwen2.5-3B: PPO ranks **2nd** (BFCL), **4th** (API-Bank), and **7th (last)** on Bamboogle, suggesting weaker generalization.\n- **Training instability:** PPO often shows **sudden drops or collapse** empirically during training (especially early on) under the same reward setup. GRPO does not exhibit this under similar conditions.\n\nWe'll clarify in the revision that \"stability\" refers to both **training robustness** and **cross-task consistency**.\n***\n> **Question 9**\n\n**On RL Algorithm Scope and Claim Clarification**\n\nThank you for the thoughtful point. While our reward design is **algorithm-agnostic in principle**, we focused on GRPO and PPO due to practical and methodological fit.\n- **Why not DPO/SimPO?** These rely on pairwise preferences, which don't suit our **structured, multi-turn rewards** (e.g., tool name and argument correctness). Tool use needs finegrained, interpretable feedback, which cannot easily mapped to preference labels.\n- **Trajectory-level credit assignment:** DPO/SimPO lack native support for intermediate rewards across multi-step tool calls, limiting their applicability in this setting.\n- **Practicality:** Adapting preference-based methods would require major infrastructure changes, with unclear gains.\n\nOur choices align with prior work: Search-R1 uses GRPO/PPO [4], ToRL uses GRPO [5]. Still, we’ll revise Lines 53–56 to clarify that:\n1. The reward design is **theoretically orthogonal** to RL algorithm.\n2. Our empirical validation focuses on **policy-gradient methods** that handle structured, multi-step rewards well.\n\nWe appreciate this suggestion and will update our claims accordingly in the revision.\n\n***\n\n> **Question 10**\n\n**Clarifying PPO’s Benefit from Our Reward Design**\n\nThank you for the observation. You're right that SFT400+PPO often outperforms PPO cold start in Tables 1–3, suggesting PPO struggles to fully leverage our reward design from scratch.\n\nOur original intent was to show that **PPO benefits from our reward signals when built on SFT**: for instance, SFT400+PPO > SFT-only. While PPO cold start is weaker, the structured rewards still provide value when PPO is properly initialized.\n\nWe’ll revise the line to clarify that PPO benefits from the reward design **mainly in the SFT+PPO setup**, and refine how PPO's role is presented throughout the paper.\n***\n> **Question 11**\n\n**Model Size and Extended Reasoning**\n\nVery insightful question! Here’s what we observe regarding model size and extended reasoning:\n\n1. **Larger models benefit slightly**, but gains are limited. On BFCL (Qwen2.5 1.5B, 3B, 7B):\n- Correct answers have slightly longer responses than incorrect ones.\n- This length gap is small and doesn’t scale clearly with model size.\n- In GRPO training, response length grows early but plateaus, suggesting longer reasoning isn’t always needed.\n\n2. **Length reward remains ineffective even at larger scales:**\n- As shown in Sec. 4.1, it underperforms on 1.5B and 3B models.\n- For Qwen2.5-7B, length reward caused a **2.10% drop** on BFCL vs. our default setup in new experiment.\n- We observe many tool learning tasks don’t require long reasoning: length reward can hurt by encouraging unnecessary verbosity.\n\nWhile we haven’t tested beyond 7B, current evidence suggests only **marginal gains from scaling**, and **length reward does not help**, even at larger sizes.\n***\n> **Question 12**\n\n**Comparisons with Overlapping Baselines**\n\nThank you for the question. For DPO/SimPO, please see our response to Question 9.\n\nAs for **Search-R1** and **ToRL**:\n- **ToRL** focuses on math/code tools, which don’t overlap with our benchmarks.\n- **Search-R1** shares Bamboogle as an evaluation task. Below is a direct comparison:\n|Model|Ours(GRPO)|Ours(PPO)|Search-R1(GRPO)|Search-R1(PPO)|\n|---|---|---|---|---|\n|Qwen2.5-3B|**60.00**|40.00|23.20|26.40|\n|Qwen2.5-7B|**72.00**|48.00|40.00|36.80|\n\nOur method, though **not trained on QA or search tools**, outperforms Search-R1, highlighting the **robustness and generalizability** of our reward design.\n***\n> **Other Suggestions and Advice**\n\nThank you for all the helpful suggestions. For error bars, please refer to our responses to Questions 6 and 7. We’ll incorporate your other feedback to improve clarity, writing, figures, and references in the next revision.\n\nWe see our work as a timely contribution to **LLM agentic reasoning and tool use**, offering both a practical training paradigm and **insights into how structured rewards shape agent behavior**.\n***\nWe hope our responses have addressed your concerns clearly and thoroughly. If there are any remaining questions, we’d be grateful for the opportunity to continue the discussion. **If our clarifications were helpful, we would sincerely appreciate your consideration in reflecting that in your overall rating.** Thank you again for your constructive feedback.\n***\n**References**\n> [1] Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\n>\n> [2] Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning.\n>\n> [3] Sft memorizes, rl generalizes: A comparative study of foundation model post-training.\n>\n> [4] Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\n>\n> [5] Torl: Scaling tool-integrated rl."}, {"review_id": "116fGkS1CS", "reviewer": "Reviewer_rARM", "summary": "The authors present a detailed study on reward design for tool selection and application tasks within the RL paradigm. They investigate the impact of four dimensions of reward strategies: reward type, granularity, scale, and temporal dynamics. Based on these analyses, the authors propose a  principled reward design framework for Tool-Integrated Reasoning (TIR) tasks. Specifically, they introduce two types of rewards: format reward and correctness reward. The correctness reward is further decomposed into three components: tool name matching, parameter name matching, and parameter content matching. Using this reward design, the LLMs trained with reinforcement learning algorithms demonstrate notable performance improvements over base models and SFT models across varied benchmarks.", "questions": "1. Given that the dynamic method shown in Figure 5 performs better on more models, why wasn’t it included in the methods section of the paper?\n\n2. Why not conduct a direct comparison with other RL frameworks specifically designed for TIR, such as ToRL or Search-R1?\n\n3. Lines 235–239 mention two examples in Table 4 that illustrate the agent’s ability to proactively refuse inappropriate tool use. Could the authors provide statistical data on how much this reasonable proactive refusal behavior increased after training?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. The paper explores reward design for tool use within the RL paradigm, covering four aspects: reward type, granularity, scale, and temporal dynamics.\n2. Based on the reward exploration, the authors design a reasonable reward strategy framework.\n3. After reinforcement learning training, the model shows significant improvement over both the base model and the SFT model.\n4. The trained model demonstrates good generalization performance, not limited to improving the reward values during training.\n\n**Weaknesses**:\n\n1. The reward design does not take into account the completion of the final task. Although the authors claim that optimizing each step of tool usage aligns with achieving the overall goal, there remains a gap between the two. Introducing final-task rewards (e.g., answer correctness in question-answering tasks) for comparison would benefit the paper.\n2. The fine-grained correctness reward consists of three components (correctness of tool name, parameter name, and parameter content), yet the maximum values of these components differ. The reasonableness of the weight allocation among them lacks experimental validation.\n3. Qwen2.5 and Llama 3.2 offer native support for function/tool calling, but the paper does not utilize their native tool-calling capabilities. Notably, Llama-3.2-3B-Instruct using native tool calling achieves an overall accuracy of 45.86 on the BFCL V3 leaderboard (see https://gorilla.cs.berkeley.edu/leaderboard.html), which is higher than the RL-trained model reported in this paper.", "was_revised": true, "final_justification": "I agree that this work is both interesting and important. I gave it a positive score in my initial rating, and after considering the rebuttal, I have decided to maintain the positive score of 4. The reason I did not assign a 5 is that the work does not present a particularly exciting or novel idea.", "rebuttal": "> **Weakness 1**\n\n**Clarification on training task setting and final-task reward**\n\nThank you for the thoughtful suggestion. In our training setup, tool learning datasets like xLAM provide ground truth tool calls at each step, but not a final answer for the full trajectory. These datasets are designed to train step-by-step tool usage rather than final answer prediction, aligning with typical tool learning settings that emphasize multiturn reasoning and correct tool invocation.\n\nTo test generalization, we evaluate on Bamboogle, a **complementary setting** that lacks intermediate supervision and only evaluates final answers. Our model, trained solely on intermediate signals, performs well here, demonstrating the effectiveness of our reward design.\n\nWe agree that if final answers are available during training, incorporating them (e.g., via an <answer> tag) could further enhance learning. Our framework is flexible and can easily accommodate this, and we will clarify this point in next revision.\n\n\n\n***\n\n\n\n> **Weakness 2**\n\n**Justification of reward scale choice** \n\nWe would like to share an insight regarding the relative scaling of **format** and **correctness** rewards. Specifically, we find that setting the **maximum correctness reward higher than the format reward** leads to more effective RL training for tool use. This design choice is also consistent with prior works such as \\[1, 2], where format correctness typically receives partial reward, while final answer correctness is assigned a higher value.\n\nFor the **format reward**, we follow the setting used in Logic-RL \\[1], fixing its maximum value at 1.0. To determine the appropriate scale for the **correctness reward**, we conducted preliminary experiments on BFCL using Qwen2.5-3B-Instruct trained with GRPO. In these trials, we varied the scale of the correctness reward while keeping the format reward fixed, and observed its impact on performance.\n\n| **Correctness Reward Scale** | 1 (equal to format) | 2 | 3 |\n| --- | --- | --- | --- |\n| **BFCL Performance** | 40.62 | 51.07 | **52.98** |\n\nWhen the correctness reward equals the format reward (scale = 1), we empirically find that the training converges more slowly and the final performance is substantially lower. Based on this evidence, we chose a scale of 3 for correctness reward as it yielded the best performance in our initial trials. We will include this insight and justification more clearly in the next revision of our paper.\n\n\n\n***\n\n\n\n> **Weakness 3**\n\n**Clarification on the use of generation template in experiments**\n\nThank you for raising this thoughtful point. We did consider the native tool-calling capabilities of models like Qwen2.5 and LLaMA 3.2 in our early-stage experiments. Specifically, we evaluated Qwen2.5-3B-Instruct on BFCL using both its native tool-call template and our custom XML-tag-based generation format. The performance across the two formats was **comparable**;or the Qwen model, which motivated our decision to adopt the XML-tag-based generation template for the final experiments. This choice is based on several important advantages:\n\n* **Support for structured reasoning**: Our XML-based template allows the introduction of additional fields, such as \\<think>, which enhances the model's reasoning capabilities, a key focus in our work on agentic behavior.\n\n* **Greater customizability**: Unlike native tool-calling APIs, XML templates offer flexibility to incorporate new tags, fields, or logic, which is critical for extending the framework to diverse tool types or training signals in the future.\n\n* **Wider adoption in recent works**: This approach aligns with recent research such as Search-R1 \\[3], ToRL \\[4], and RM-R1 \\[5], all of which use customized generation formats rather than relying on native tool-calls.\n\nRegarding the LLaMA-3.2-3B-Instruct result on BFCL, we believe several factors may contribute to this outcome:\n\n* LLaMA's **pretraining data** likely includes rich tool-use demonstrations in native formats, giving it an advantage in zero-shot or few-shot settings. In contrast, our XML-based template is model-agnostic and unseen during pretraining, which may explain the initial performance gap.\n\n* Native tool-calling APIs are often optimized for inference-time execution, whereas our focus is on **training-time flexibility and generalization** across unseen tools and tasks.\n\n* Cross-model comparability can be misleading: performance varies across architectures, and native vs. generated tool-call performance is not always directly comparable. Our method prioritizes **training consistency** across models and domains.\n\nAdditionally, our experiments show that RL-based training tends to yield stronger gains on Qwen models than on LLaMA models, a trend also reflected in recent works like \\[3, 4], which focus RL efforts primarily on Qwen. Furthermore, native tool-call APIs offer limited customization, a constraint when applying RL to improve reasoning depth or integrate additional supervision (e.g., dynamic reward signals).\n\nIn summary, while native tool calling may yield strong baseline results in certain models, our XML-based generation approach may offer the flexibility, extensibility, and compatibility needed for RL-based reward optimization, which is the core focus of our paper. We believe this design choice aligns well with the goals of our method and supports broader applicability across future tool-use research.\n\n\n\n***\n\n\n\n> **Question 1**\n\n**Clarification on the exclusion of the dynamic method from the main methodology**\n\nThe dynamic method shown in Figure 5 was effective, but we chose not to include it in the main method section for the following reasons:\n\n- Our reward design is already multi-dimensional (format, correctness, finegrained supervision). Adding dynamics would increase complexity and reduce clarity.\n\n- The dynamic aspects (reward scaling and response length) are orthogonal. Presenting them separately in Sections 4.1 and 4.2 allows for clearer analysis.\n\n- Our main method already achieves strong results. The dynamic variant brings incremental gains, so presenting it as an analytical extension keeps the main narrative clean and focused.\n\n\n***\n\n\n\n> **Question 2**\n\n**Comparisons with overlapping baselines**\n\nThank you for raising this thoughtful point. We would first like to clarify that ToRL and Search-R1 do not propose new RL algorithms or novel reward designs, but rather explore the application of multi-turn RL training on agentic tasks within specific domains, such as math with code tool in ToRL, or QA with search engines in Search-R1. Regarding direct comparisons with these works:\n\n* **ToRL** focuses primarily on math problems and code-based tools, which do not overlap with the domains tested in our benchmarks. \n\n* **Search-R1**;oes include Bamboogle as part of its evaluation. We provide a direct comparison using the same benchmark in the table below:\n\n| Model Name | **Ours: GRPO Cold Start** | **Ours: PPO Cold Start** | **Search-R1 (GRPO)** | **Search-R1 (PPO)** |\n| --- | --- | --- | --- | --- |\n| **Qwen2.5-3B-Instruct** | **60.00** | 40.00 | 23.20 | 26.40 |\n| **Qwen2.5-7B-Instruct** | **72.00** | 48.00 | 40.00 | 36.80 |\n\n\nThese results clearly show that our reward design significantly outperforms Search-R1 across both model sizes, despite the fact that Search-R1 is explicitly trained on QA tasks with search tools, while our models are not explicitly exposed to QA tasks or the search tool during training. We believe this comparison further highlights the **robustness and generalizability** of our reward formulation.\n\n\n\n***\n\n\n\n> **Question 3**\n\n**Detailed statistics of proactive refusal behavior**\n\nThank you for this insightful question. The behavior of proactively refusing inappropriate tool use is best evaluated using a dedicated subset of the BFCL benchmark known as \"Irrelevant\", which includes Irrelevant-Normal and Irrelevant-Live. In each case, the model is presented with a question and a set of tools that are **irrelevant to solving the task**, and is expected to recognize this mismatch and explicitly refrain from making a tool call. Success on this subset directly reflects the model’s ability to exhibit **tool awareness and proactive refusal**, making it an ideal target for the requested analysis.\n\nWe first refer to the results in Figure 3(b), which give a preliminary view of the trained model’s performance on this subset. Below, we provide a more detailed breakdown of proactive refusal accuracy across training conditions for Qwen2.5-3B-Instruct:\n\n| **Setting** | **Irrelevant-Normal** | **Irrelevant-Live** |\n|---|---|---|\n| Raw Model | 0.47 | 0.43 |\n| SFT4K | 0.59 | 0.77 |\n| SFT400 + GRPO | 0.79 | 0.63 |\n| **GRPO Cold Start** | **0.87** | **0.81** |\n\nThese results show a substantial improvement in the model’s ability to proactively avoid irrelevant tool use after training. In particular, the GRPO cold start model achieves the highest scores in both irrelevant subsets, demonstrating strong generalization and precise judgment in unseen tool use scenarios. We will make sure to highlight these findings more explicitly in the next version of the paper.\n\n\n\n***\n\n\n\nWe hope our responses have addressed your concerns clearly and thoroughly. If there are any remaining questions or points you’d like to discuss further, we’d be grateful for the opportunity to continue the conversation. **If you find our clarifications satisfactory, we would sincerely appreciate your consideration in reflecting that in your detailed evaluation score or overall rating.** Thank you very much for your constructive feedback.\n\n\n***\n\n\n**References**\n\n> \\[1] Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning\n>\n> \\[2] Limr: Less is more for rl scaling.\n> \n> \\[3] Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\n> \n> \\[4] Torl: Scaling tool-integrated rl.\n> \n> \\[5] RM-R1: Reward Modeling as Reasoning."}]

Author rebuttal:

Meta-review: **(a) Summary**

The paper proposes ToolRL, a reinforcement learning framework for tool learning in large language models. The key idea is to treat tool use as an RL problem where appropriate reward design drives efficient and robust tool learning without relying heavily on handcrafted supervision. The authors provide extensive empirical evaluations across multiple tool-use tasks, showing that ToolRL can effectively generalize to unseen tools and tasks.

**(b) Strengths**

* The work formulates tool learning in LLMs explicitly as an RL problem, offering a clean and generalizable perspective.

* The proposed approach reduces reliance on task-specific supervision and demonstrates broad applicability.

* Strong experimental results across diverse tool-use benchmarks demonstrate consistent improvements over baselines.

**(c) Weaknesses**
* Despite the title, the approach still requires supervised data and shows dependence on specific RL setups, undermining its claimed generality.

* The training scale is modest (~4K examples, three datasets), and comparisons to stronger baselines (e.g., models with native tool calling) are missing.


**(d) Reasons for acceptance**

The paper makes a timely and relevant contribution to the emerging problem of tool learning with LLMs. Its framing of tool learning as an RL problem is simple, principled, and practically effective, and the empirical results demonstrate consistent benefits. Given the strong motivation, reasonable performance gains, and community interest in this direction, I recommend acceptance.

**(e) Discussion and rebuttal**

Reviewers raised concerns about technical novelty, the over-claimed independence from supervision, limited experimental scale, and clarity of exposition. In the rebuttal, the authors clarified the design motivations, provided further analysis of reward choices, and emphasized the main contributions. While some reservations remained, reviewers acknowledged that these clarifications addressed the key doubts.
Decision: Accept (poster)