Empowering LLM Tool Invocation with Tool-call Reward Model
Authors: Da Ma
Ziyue Yang
Hongshen Xu
Haotian Fang
Kai Yu
Lu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model
Tool invocation
Tool-call reward model
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a Tool-call Reward Model that provides fine-grained signals for tool invocation and adapts classical RL algorithms, significantly enhancing LLMs' tool usage compared to outcome-only reward methods.

Abstract: Large Language Models (LLMs) have recently alleviated limitations in outdated internal knowledge and computational inaccuracies by invoking external tools such as search engines and code generation. While reinforcement learning (RL) has substantially enhanced tool usage in LLMs, most existing agentic RL approaches rely solely on outcome-only reward signals, which assign credit at a coarse granularity and often induce gradient conflict (e.g., correct tool calls may be penalized due to incorrect final answers). To address this, we propose the *Tool-call Reward Model* (TRM), a specialized process reward model meticulously designed to evaluate and reward each tool invocation. Since previous PRM research has predominantly focused on traditional reasoning tasks such as step-wise mathematical reasoning, the introduction of TRM brings two unique challenges: (1) limited understanding of how to construct effective TRMs, including data requirements and model size; and (2) difficulties integrating TRM with classical RL algorithms such as PPO and GRPO, where naive adaptation may lead to reward hacking (minimizing tool calls to avoid penalties). To tackle these challenges, we establish a systematic TRM construction workflow and propose refined credit assignment and turn-level advantage estimation for effective integration with PPO and GRPO. Experiments show that a 3B TRM trained on 10K samples achieves robust performance. On search-based QA and Python code-based math tasks, integrating TRM consistently outperforms outcome-only reward RL methods across models of different sizes.

Published as a conference paper at ICLR 2026
EMPOWERING LLM TOOL INVOCATION WITH TOOL- CALL REWARD MODEL
Da Ma', Ziyue Yang', Hongshen Xu', Haotian Fang', Kai Yu1.3,4 Lu Chen EX-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China Shanghai Innovation Institution, Shanghai, China SJinggu Key Lab of Language Computing, Suzhou, China
"Suzhou Laboratory, Suzhou, China (mada123, @eenlusz]@sjte teu..e.euu.uuccnmmmmmmmmmm cn
ABSTRACT
Large Language Models (LLMs) have recently alleviated limitations in outdated internal knowledge and computational inaccuracies by invoking external tools such as search engines and code generation. While reinforcement learning (RL.) has substantially enhanced tool usage in I.LMs, most existing agentic RI, ap- proaches rely solely outcome-only reward signals, which assign credit at coarse granularity and often induce gradient conflict (c.g., correct tool calls may be penalized due to incorrect final answers). To address this, propose the Tool-call Reward Model (TRM), a specialized process reward model meticulously designed to evaluate and reward each tool invocation. Since previous PRM re- search has predominantly focused on traditional reasoning tasks such as step-wise mathematical reasoning, the introduction of TRM brings two unique challenges: (I) limited understanding of how to construct effective TRMs, including data re- quirements and model size; and (2) difficulties integrating TRM with classical RL algorithms such as PPO and GRPO, where naive adaptation may lead to reward hacking (minimizing tool calls avoid penalties). To tackle these challenges, we establish a systematic TRM construction workflow and propose refined credit as- signment and turn-level advantage estimation for effective integration with PPO and GRPO. Experiments show that 3B TRM trained 10K samples achieves robust performance. On search-based QA and Python code-based math tasks, in- tegrating TRM consistently outperforms outcome-only reward RL methods across models of different sizes T
INTRODUCTION
Large Language Models (LLMs) have demonstrated sophisticated proficiency in addressing com- plex tasks, profoundly impacting broad spectrum of domains (OpenAIl [023] Guo et al. [025] Yang et al. 2025). However, LLMs are fundamentally limited by the static nature of their inter- nal knowledge and their propensity make computational errors (Schick et al., 2023) Qian et al.) 2025a). To overcome these challenges, LLMs increasingly invoke external tools, such as search en- gines for accessing up-to-date information {Jin et al.] 2025 Chen et al. 2025b) and code generation for solving complex mathematical problems (Liao et al. 2024 Feng et al. 2025).
With tool invocation playing an increasingly important role in overcoming L.LM limitations, rein- forcement learning (RL), proven effective in traditional reasoning tasks (Guo et al.) 2025 Team
2025 Team et al. 2025 Wang et al., (2224,, has been widely used to enhance tool usage. In practice,
most RL-based approaches (Jin et al. 2025 Song et al.) 2025 Feng et al., 2025 Li et al. 2025b) for tool invocation rely solely on outcome reward signals, evaluating only the correctness of the final output (e.g., math answer correctness) while overlooking the quality of intermediate tool calls. Consequently, credit for each tool call in a trajectory is assigned solely based on the final outcome,
"Corresponding author
Avaliable on https: //gi thub. cow//penDD//TRMM

---

Published as a conference paper at ICLR 2026
(4) Incorract answer after suitable tool invocation by LLM
(b) Taol call reward complenentation for outcome reward wscinesses
(c) Performance gain from reward
Who Chorles Bretaone Marle be La Trd noille' paternal granffather?
Tool call Reward Madel
Challenges
LLM
Thooght
deteresine Cvaries Brensyse Wone be T-éreilles patereal gr grandtctter may first reed ideniify his father
solved
Teat oot
Search ('uho Chorles Breteone Mare e De Lo Tréncelica futhe")
Obsorettion
Charies Bretsgnc Teérzile charles
Acceet
Bretesne Chorie de Lo Trimolk
Wrong!
Outcome
Reward
Figure 1: Overview of TRM for improving tool invocation in LLMs: (a) limitations of outcome-only reward, (b) benefits of tool call reward, and (c) performance gains from integrating tool call reward.
irrespective of its individual quality or usefulness. With uniform treatment of tool calls, this ap- proach limits the ability of the model to learn effective tool usage, potentially resulting in unstable or suboptimal performance. For example, if the final answer is incorrect, a trajectory with correct intermediate tool usage is still penalized (Figure a and Figure [ This discourages learning of
effective tool invocation strategies and causes gradient conflict (Lightman et al. 2024 Deng et al., 20256), often leading to unstable tool usage and degraded performance.
To this end, we propose the Tool-call Reward Model (TRM), which quantitatively measures the utility of each tool invocation and assigns corresponding rewards. Although TRM can be viewed as a specific type of process reward model (PRM), prior PRM research (Lightman et al. 2024 Zhang al., 22025b) Setlur et al. 2025) has predominantly focused on traditional reasoning tasks, leaving tool invocation underexplored. TRM fills this gap by enabling fine-grained monitoring of tool usage, thereby facilitating more appropriate tool invocation (Figure[Ib)) However, introducing TRM raises two new key challenges (Figure e)): 1) 7RM creation: how to construct an effective TRM, and 2) TRM application: how to integrate it with classical RL algorithms.
For the first challenge, the main difficulty lies in the limited understanding of TRMs, including how to construct training data, the required data volume, and the suitable model size. To address this, we develop workflow to distill training data from frontier LLMs ($ 2.2) and systematically analyze
the impact of data volume and model size on TRM performance (S 33.I Beyond this, integrating TRM with classical RL algorithms such as PPO SPoSSchulman et al. (2017 and GRPO Shao et al. (2024) remains an open challenge, as directly transferring approaches that combine standard PRM and RL. algorithms may not work well for TRM. For instance, in GRPO, our experiments demonstrate that group-level advantage estimation (Shao et al. (202)) of tool call reward can result in reward hacking, where the model prefers fewer tool calls over effective usage (see Appendix E.II. To address these issues, we refine the credit assignment strategy by allocating tool call rewards to the end of each tool invocation, and introduce turn-leve! advantage estimation in GRPO ($ 2.3). Ultimately, our experiments show that the proposed methods yield better overall model performance (Figure [c,, $ $3.2.. Furthermore, we observe that TRM enhances generalization tool invocation, enabling the model to flexibly adapt to unseen tools 3.3).
In summary, this work makes the following three contributions::) We propose the Tool-call Reward Model (TRM) and conduct thorough investigation into its construction. 2) We develop and analyze new algorithms for integrating TRM with classical RL methods, including refined credit assign- ment strategies (PPO) and step-wise advantage estimation (GRPO). 3) We validate our approaches through extensive experiments, demonstrating significant improvements in model performance. We plan to make our data and code publicly available to facilitate future research.
A reasonable way to determine the paternal grandfather of person is to first determine the father of the person, then the father of that father.
2

---

Published as a conference paper at ICLR 2026
2 METHODOLOGY
We introduce a Tool-call Reward Model (TRM) to resolve gradient conflict from outcome-only rewards by supplying fine-grained, per-call utility signals that stabilize the tool invocation. this section, (i) formalize the multi-turn RL framework for tool invocation in LLMs, (ii) detail the construction of TRM, including training data distillation and model optimization. and (iii) integrate TRM with classic RI. algorithms by proposing turn-level credit assignment and enhancing GRPO with turn-level advantage estimation.
2.1 PROBLEM FORMULATION
We formalize multi-turn tool invocation in LLMs sequential decision-making process under the reinforcement learning framework. Following the ReAct paradigm (Yao et al. 2023), the LLM alternates between reasoning steps and tool invocations, enabling dynamic planning and external information gathering for more robust and interpretable task-solving. Formally, consider prompt p and LLM parameterized by 0. Given p. the LLM π engages in multiple rounds of tool invocation, where at each round, the model reasons over the current information and decides on the next tool action. This iterative process continues until the model is ready produce the final answer. Finally, the LLM æ generates trajectory
(p, 11. 01, 01, t,., On. On, +1, 9)
where 1, (1 n, +1) denotes the reasoning thought, ag and o, I i S n2) is the tool invoked and its corresponding output at turn i, n, is the total number of tool invocation rounds, and is the final answer produced by the LLM x. Here, we refer to each triplet (4, 0;, 0;) as a single turn the interaction
Given this formulation, our objective is to optimize the policy my to maximize the likelihood of producing the correct final answer at the end of the trajectory. Formally, the learning objective is to maximize the expected correctness of the final answer y over trajectories generated by the policy ne:
max (1 = y-]], 0
where y" is the ground-truth answer and TI (.) is the indicator function.
2.2 CONSTRUCTION OF TRM
Data Distillation We first describe the process of distilling high-quality training data for TRM from frontier LLMs (Figure a)). This process consists of two main steps: 1) rollout collection and 2) tool call evaluation. In the rollout collection step, the model is provided with set prompts and tool-enabled environment, and generates multi-turn trajectories by autonomously invoking tools to complete the task. For each collected rollout, we further evaluate every tool call a; by re-feeding the whole trajectory into the model assess its utility. Specifically, we assign two binary scores for each tool call a,;
necessity s whether the tool call contributes substantive progress toward task completion
quality s/c whether the tool is invoked with reasonable parameters used correctly
Hence, tool call is assigned score of 1 only when it is both necessary for task progress and executed with high quality; if either criterion not met, the score 0. Formally, for a tool call a; the final score is defined as:
where sic s E {0, 1}. The detailed design of prompts are illustrated in Appendix A.1
The final turn consists of both reasoning and the generation of the final answer, without involving any tool call.
3

---

Published as a conference paper at ICLR 2026
Rollout collection
thought
tool coll
observation
onswer
Rollout
LLM
Tool pool
Prompts
(b) Integration of TRM into RL
Tool call evaluation
LLM
(a) TRM Training
Outcome
Data Workflow
assigement
Turn-level advantage estimation for GRPO
Necessity X
+b
Quality
Training
Tun
Tam
Outcoe
data
Tool call score
TRM
Figure 2: TRM-guided LLM tool invocation. (a) Generation of tool invocation trajectories and turn- level utility labels for TRM training. (b) Turn-level credit assignment and GRPO adaptation via turn-level advantage estimation.
TRM Training The TRM adopts transformer-bast (Vaswani al. 2017 LLM its back- bone. To adapt the model for tool-call utility prediction, we replace the original language modeling head (used for next-token prediction) with a binary classification head consisting of single linear layer. Specifically, for each tool call ag the model produces a probability g E [0, 1] based on the hidden state of the last token of the tool call output o, This score indicates the predicted utility of the tool call. During training, the TRM is optimized using binary cross-entrops loss
LBCE E,
s log 8¹ + (1-15) log (1 (1-)))
(4)
n4
2.3 INTEGRATION OF TRM WITH RL
With TRM in place, we proceed integrate it into established RL algorithms optimize tool invocation in LLMs. Specifically, we focus on two representative policy optimization methods Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO).
Turn-level Credit Assignment To achieve appropriate credit assignment throughout the trajec- tory, we combine TRM scores for intermediate tool invocations with the outcome reward for the final answer (Figure b). In particular, for each turn i (1 < n2) of trajectory т, the reward is given by the TRM score sf, and for the final reasoning step (i = ny + 1), the reward is determined by the correctness of the final answer. Mathematically, the turn-level reward ş is defined as
(3 Ii - 9²).
1 isisn,
(5)
i i= i= +1
Both PPO and GRPO perform policy optimization at the token level, whereas our reward signals are defined at the turn level. To bridge this granularity gap, we also represent each trajectory as (I1, T21,42,.... ................................ Lt)) where zj is the j-th token. For each turn i (1 < i S ny), fei...., Cn, thus marks
sequence of tokens,
we identify e; as the index of the last token of the tool call a,. The set all toll-callennding tokens. We further define mapping I(j) that returns the corresponding turn index for any j ε, and set I(L) ny + for the final answer. To specify which tokens participate in policy optimization, we define M C (1...., L) the set of indices of thought, tool call, and answer tokens that are not masked during RI. training. These notations facilitate our subsequent discussion on the integration TRM with RL.
In practice, a score is also produced at the last token of the answer to indicate its correctness. For clarity, KI. regularization omitted in discussion.
4

---

Published as a conference paper at ICLR 2026
Integration with PPO To enable token-level policy optimization, we map turn-level rewards to the corresponding tokens by assigning the reward for each tool call to the last token the associated action, and the outcome reward to the last token of the answer. Formally, the reward y² of token Jij (1 S < I.) is defined as
a. FTO))
jEC j=L
pIU),
O,
otherwise
where E (0, 1] is hyperparameter controlling the weight of the TRM score. Advantage A is then computed from (c.g.. Generalized Advantage Estimation (Schulman et al. 2016)). With this token-level advantage, the PPO objective is formulated
LPPO Brnno
Σ
min (w) Al, clip (w.(0,11c.1
(7)
JEM
where (0) xo(2)lm<s)) and é is the clipping parameter.
Te (2ilzee)
Integration with GRPO GRPO policy optimization method that compares and normalizes rewards across a batch of trajectories to increase training efficiency. In GRPO, group refers to batch of G trajectories [ T.) For each trajectory Tg (1 S g S G), variables such as nzy Eg. Mg. and other notations follow the same definitions in previous sections, with the addition of the
trajectory index g. Across the group, collect the TRM rewards Rum (for any turn i) and outcome rewards Rout via
R um
(7g 1 S 1
S G, i ),, R.m. -(-) Meg +1 1 , sc
(8)
We then perform turn-level advantage estimation, where rewards for each turn are normalized inde- pendently across trajectories (Figure b). In detail, for each turn i and trajectory Tg. the normalized rewards are computed as
F std
mean
Rim
Fg
mean
Rooa
(1 S i iS nr,))
fg Rogll
Rim
std
Roou
These normalized rewards then assigned the corresponding tokens, and token-level advantages are computed via discounted aggregation:
I(G)
E
E
LØ-1 N +
Fg
nrutt
= Lgg
A
7
m
m-j
(10)
0,
otherwise
mmm
where a is a weighting hyperparameter and y is the discount factor tage, the GRPO objective is formulated as
With this token-level advan-
LORPO = (rrT)
min (wg (0) ,g.,, clip (wg (8), -, c, 1.1 +c A4) JEM
(11)
G
M
EXPERIMENTS
In this section, we focus on two key aspects:
TRM exploration: How can we obtain effective TRM?
7RM exploitation: Does introducing TRM improve the tool-use capabilities of LLMs?
MMaked tokens skipped when computing the discounted of normalized rewards.

---

Published as a conference paper at ICLR 2026
Mode:
Aoticc
0- TRM--00 100-30-124
In- Fn-domain
Samgle
Seeple190201
Mode:
Trodiig Deto
Bethed
Mo sine
Out-of- d--mmmnn
Saeple
(a) Impact of model size with 10K training data
(b) Impact of training data Size for 3B model
(c) Comparison of TRM-3B and Baselines with 10K training data.
Figure 3: TRM performance comparison under different settings
3.1 EXPLORATION OF TRM
Training Model and Data We use the Qwen2.5 (Qwen et al. 2025) series as the backbone archi- tecture for TRM. For training data, we sample 15K prompts each from the HotpotQA (Yang al. 2018) and NQ (Kočisky et al.) 2018) training sets. Rollouts are generated using DeepSeek-R (Guo al. 2025). which interacts with search environment (Jin et al. 2025) produce multi-tur tra- jectories. Each trajectory is annotated with turn-level utility labels based on necessity and quality by DeepSeek-R1. Finally, we randomly sample 10K labeled trajectories for TRM training. More training details are in Appendix B.
Evaluation To evaluate TRM effectiveness, we use a best-of-n strategy (Lightman et al. 2024 Luo et al. 2025): for each prompt, candidate trajectories are generated, and the one with the highest TRM score is selected. The score for a trajectory T is computed as the product of all tool call scores, i.e., S(r) -
Evaluation is conducted in both in-domain (HotpotQA validation prompts) and out-of-domain (2 2WikiMultiHopQA (Ho et al. (2020) validation prompts) settings, All candidate trajectories are generated by the Search-R] (Jin et al. 2025) model, which is PPO-trained based on Qwen2.5-7B. More details are Appendix C.
Results and Analysis
According the results in Figure
observe following key trends:
Key Takeaways for TRM Exploration
Mid-sized TRMs (1.5B/3B) deliver optimal performance with 10K training samples, while larger models (e.g.. 7B) may be prone to overfitting given the same data scale.
10K labeled trajectories sufficient to achieve robust TRM training and stable results.
TRM consistently outperforms the majority vote baseline, though there remains gap to the upper bound established by pass@n.
3.2
EXPLOITATION OF TRM
Setup We conduct experiments in two distinct scenarios: (1) answering questions using search tool. and (2) solving math problems by writing Python code. Following prior works (Jin et al.) 2025 Li et al. 2025b), for the search-based QA task, we evaluate both Qwwn2.5-3B-Inntthe
g indicates the correctness for the final answer as predicted by the TRM.

---

Published as a conference paper at ICLR 2026
Table Performance of Qwen2.5 variants with different methods on various QA tasks. Best results are in bold; second best are underlined.
General QA
Multi-Hop QA
Method
Avg.
NQ
TriviaQA
PopQA
HorpotQA
2wiki
Musique
Bamboogle
Owen2.5-3B-Insttee
Direct Inference
12.08
32.44
13.08
15.98
24.75
2.19
2.40
14.70
IRCoT
26.32
49.47
33.28
24.33
16.19
4.43
19.20
24.75
RAG
37.29
56.05
40.60
26.31
23.08
5.17
6.40
27.84
SFT
27.53
31.37
12.26
20.70
26.28
6.25
11.20
19.37
R1-PPO
19.09
42.16
16.35
19.04
25.91
3,14
8.00
19.10
R1-GRPO
25.10
45,89
18.10
21.55
27.98
5.42
20.00
23.43
Search-RI-PPO
36.93
54.48
35.85
32.65
32.47
12.08
24.80
32.75
Search-R1-PPO-TRA (ours)
39.58
57.78
40.61
34.80
33.22
12.91
25.60
34.93
Search-RI-GRPO
47.01
61.88
45.73
43.34
42.68
18.08
37.60
42.33
Search-RI-GRPO-TA (ours)
47.89
62.57
47.20
44.47
43.48
19.65
39.20
43.49
Qwen2.5-7BB
5-7B--Instrcct
Direct Inference
14.29
43.69
15.10
19.23
25.54
3.68
10.40
18.85
IRCoT
18.23
50.31
30.33
21.61
8.73
4.05
17.60
21.55
RAG
34.88
58.96
39.45
30.16
23.62
5.50
21.60
30.59
SFT
31.97
34.00
12.36
22.23
26.40
9.72
10.40
21.01
23.13
49.60
17.51
22.31
28.15
6.95
30.40
25.29
R1-GRPO
31,61
53.69
21.60
24.96
27.47
8.77
32.00
28.59
Search-RI-PPO
40.86
61.42
40.15
37.84
35.27
14.81
35.20
37.94
Search-R1-PPO-TRM (ours)
43.99
61.18
41.56
39.11
37.76
17.63
38.40
39.95
Search-R1-GRPO
49.97
66.81
47.59
49.06
47.80
22.30
44.80
46.90
Search-RI-GRPO-TRR (ours)
52.11
66.90
48.52
51.32
47.67
24.99
48.80
48.62
and Qwen2.5-7B-Instruuc models; for the code-based math task, we utilize Qwen2.5-Math-11 and Qwen2.5-Math-78 (Yang et al. 2024). The training data for each scenario are also sourced from the corresponding prior works to ensure consistency and comparability. The search tool is allowed up to 5 rounds per query, while the code tool can be invoked up to 3 times per problem. All implementations are based on the Verl (Sheng et al., 2025; Zhang et al. (022)) framework. We set 0.05 for PPO and 0.01 for GRPO, Additional training details provided in Appendix B.2
Evaluation For the search scenario, we evaluate performance on both general QA datasets (NQ (Kočiský al. 2018), TriviaQA (Joshi et al. 2017). PopQA (Mallen et al. (023)) and multi-hop QA datasets (HotpotQA Yang et 2018), 2 2Wiki (Ho et al. 2020), Musique (Trivedi et al., 2022), Bamboogle (Press et al. 2023)). For the code-writing scenario, evaluation
ducted on AIME24, AIME25, MATH500 (Hendyossss et al. 2021), Olympiad (He et al. 2024). and
AMC23. More evaluation details are in Appendix C.2
Baselines For both search and code scenarios, consider: (1) Direct Inference, which an- swers questions without any tool usage; (2) SFT, supervised fine-tuning without tool usage; and (3) R1-PPO/R1 -GRPO, models trained with PPO or GRPO using outcome-only rewards, without tool usage. Additional baselines for the search scenario include: (1) RAG, which retrieves relevant in- formation once before answering: (2) IRCOT, iterative retrieval based previous results; and (3) -iicc-RR--PP//Seacc---R----
trained with PPO or GRPO and allowed to use the search tool. For the code scenario, further include: (1) Instruct, direct inference with the instruct version of Qwen2.5-Math models; (2) Instruct+PAL (Gao et al. (202), generating programs as the inter- mediate reasoning steps; and (3) ToRL -WWLL--POOTOO-----
trained with PPO or GRPO and allowed to use the code tool. More details are shown in Appendix
Results and Analysis Table Table summarize the performance of Qwen2.5 variants across different QA and math tasks. Several key observations emerge:

---

Published as a conference paper at ICLR 2026
Table Performance of Qwen2.5-Math variants with different methods on various math problems.
Best results are in bold; second best are underlined.
Method
AIME24
AIME25
MATH500
Olympiad
AMC23
Avg.
Qwen2.5-Mathe
Direct Inference
7.78
1.11
67.80
28.30
35.00
28.00
Instruct
10.67
7.22
72.60
36.59
57.50
36.92
Instruct+PAL
34.44
0.00
21.80
10.07
17.50
16.76
SFT
0.00
0.00
15.40
7.11
27.50
10.00
R1-PPO
11.00
10.00
74.80
33.48
55.00
36.86
R1-GRPO
14.11
3.67
73.40
31.70
57.50
36.08
ToRL-PPO
19.11
13.89
75.80
43.56
55.00
41.47
TORL-PPO-TRM (ours)
26.00
19.89
75.80
45.78
50.00
43.49
ToRL-GRPO
25.56
19.33
75.80
45.19
50.00
43.18
ToRL-GRPO-TRM (ours)
26.00
27.00
75.80
45.78
52.50
45.42
Owen2.5-Mathe
Direct Inference
12.22
6.67
69.80
30.96
40.00
31.93
Instruct
5.11
8.11
79.60
37.33
52.50
36.53
SFT
0.00
0.00
12.80
5.19
42.50
12.10
R1-PPO
28.11
10.11
77.40
37.93
65.00
43.71
R1-GRPO
21.00
9.78
78.00
37.93
67.50
42.84
ToRL-PPO
32.56
23.11
82.60
53.04
67.50
51.76
TORL-PPO-TRM (ours)
34.33
26.56
83.40
52.44
70.00
53.35
ToRL-GRPO
35.00
21.89
83.80
52.74
67.50
52.19
TRRL-GRPO-TRM (ours)
36.56
23.67
83.20
52.59
72.50
53.70 70
Key Takeaways for TRM Exploitation
TRM consistently enhances model performance in both search and code scenarios, across various model sizes (1.58, 3B, 7B) and training algorithms (PPO, GRPO), indicating that TRM substantially strengthens the ability of LLMS to effectively utilize external tools.
Enabling LLMs to dynamically learn tool use yields notable gains, while reinforcement learning without tool integration leads much lower performance. Importantly, TRM plays critical role by helping models utilize tools more effectively.
GRPO generally outperforms PPO in our experiments; however, integrating TRM reliably boosts performance for both approaches.
3.3 ADDITIONAL ANALYSIS
In this section, we provide further analysis on several key factors related to TRM exploitation and some ablation studies.
Effect of Hyperparameter Figure 5 shows that for PPO, a very small limits the effect of TRM, while very large a overemphasizes tool use. A moderate balances final performance and reasonable tool invocation. Figure 55 shows similar trend for GRPO. We therefore a 0.05 for PPO and a 0.01 for GRPO in our experiments.
Improvement of Tool-Use Generalization by TRM We investigate the generalization ability of LLMs in tool-use scenarios. Specifically, we evaluate models trained in the search scenario on their ability use Python code for solving mathematical problems. As shown in Figure introducing TRM significantly improves generalization in tool invocation across different scenarios.
8

---

Published as a conference paper at ICLR 2026
34.93
1414
46
13,49
TRA
30
35
36.20
25
33.03
Accursoy
Accursry
25
-30
30
25
20
0.01
0.05
0.1
0.3
0.11
0.05 0.1
PPG
GRPO
(a) Impact of for PPO
(b) Impact of for GRPO
(c) TRM for generalization
Figure Summary of key analysis results. Subfigures (a) and (b) present the influence of the hyperparameter a PPO and GRPO in conjunction with TRM. Subfigure (c) demonstrates that TRM improves the generalization capability LLM for tool-use.
turn-level
42.47
Effect Turn-Level Advantage Estima- tion in GRPO Unlike turn-level estimation, which normalizes rewards for each turn indi- vidually, group-level estimation normalizes all tool-call rewards within group together (Shao iet al.] 2024). As shown in Figure turn- level advantage estimation achieves better per- formance than group-level estimation.
group- level
41,18
40.0
40.5
41.0
41.5
42.0
42.5
43.0
Accuracy
Figure 4:
Comparison of group-level and turn- level advantage estimation GRPO
Comparison with other process-supervised tool-use methods We compare method with two representative process-supervised baselines: StepSearch (Wang et al.] (025b), which is tailored for search-based QA and evaluates intermediate search queries for relevance and information gain, and AgentPRM (Choudhury] 2025). a general process-supervised method that labels tool calls based on whether they can eventually lead correct answer. Figure 6-a shows our method consistently outperforms both baselines in the search scenario over Swwe2.55-B-Innstee with PPO, highlighting the advantage of our per-tool-call reward modeling over band-crafted or generic process supervision signals.
Ablation study to disambiguate distillation and TRM To separate the effects of distillation from TRM, we introduce two baselines: ORM, which scores entire trajectories, and TRM used as verifier, which aggregates per-tool-call scores. Figure b shows that trajectory-level ORM under- performs the answer-only baseline in the search scenario over Qwen2.5-3B-Insttnn because scoring entire trajectories introduces additional noise. TRM as verifier improves slightly still lags behind full TRM, suggesting that fine-grained per-tool-call evaluation essential for guiding the model effectively and fully leveraging the distillation data.
with PPO, likely
Ablation study on the necessity and quality of tool calls We evaluate the impact of tool-call ne- cessity and quality on model performance and tool usage. Figureroe shows that using quality-only yields the lowest performance, likely due to excessive tool usage that introduces noise. Necessity- only reduces the number tool calls but may compromise the quality of each call, limiting overall effectiveness. Combining both necessity and quality achieves the best performance while maintain- ing a relatively stable number of tool calls across datasets, suggesting that balancing necessity and quality is important for efficient and effective tool use.
RELATED WORK
Process Reward Model Reward models have been widely adopted in various reasoning tasks supervise output quality, such mathematical problem-solving (Uesato et al. 2023 Shao

---

Published as a conference paper at ICLR 2026
36
$453
$0.56
12.75
31.00
30.00
21-
25:00
50
25
Sear dv-il StepSaorch dh AgentWM (44-4)
OPN
qailty-aty
sacadity-cyy
bonh
Figure 6: Performance comparisons and ablations for tool-supervised methods.
et al. 2024 Zhang al. 2025a). These models are generally divided into outcome reward mod- els (ORMs), which provide holistic evaluations, and process reward models (PRMs), which offer fine-grained, step-level assessments. PRMs have shown strong effectiveness (Lightman al. 2024 Wang et al. 2024 Luo et al. |024| Cheng et al. 2222), especially in mathematical problem-solving solvnn, and have been used both for guiding inference (c.g.. best-of-n selection) and for supervising post- training. By providing more granular feedback, process reward models enable models to learn more interpretable and robust reasoning strategies. However, most existing work on PRMs focuses on tra- ditional reasoning tasks, with limited exploration in tool-use scenarios. In this work, we introduce the Tool-call Reward Model (TRM), specifically designed for tool-invocation of LLMs, and con- duct comprehensive study on both the exploration and exploitation of TRM. Our approach aims to extend process-level supervision to agentic tasks, enabling more effective and flexible tool usage in LLMs.
Agentic RL for LLM Tool Invocation Recent advances in outcome-based RL have enabled LLMs to achieve impressive performance agentic reasoning tasks (Guo et al. 2025 Hu et al, 2023). This paradigm has spurred active research in tool invocation for LLMs, with works such as Search-R1 (Jin et al. 2025), ReSearch (Chen et al. 2025a). R1-Searcher (Song et al.] 2025), DeepResearcher (Zheng et al. 2025). WebRL (Qi al. 2024). WebThinker (Li et al. 2025a). Ze- roSearch (Sun et al.; (025). ToRL (Li et al. 2025b), and TooIRI (Qian et al.] 2025b) extending outcome-superriiedd RI. scenarios where L.LMs autonomously utilize search engines code ecution for complex reasoning and problem-solving While these methods have improved agentic capabilities, the reward signals are typically coarse-grained, focusing only on final outcomes and providing limited guidance for efficient tool-use search strategies. Atom-Searcher (Deng et al., 2025a) and StepSearch (Wang et al., (025a) further consider intermediate tool-use steps by lever- aging existing large models rule-based approaches. In contrast, our work designs and develops dedicated TRM to explicitly monitor and supervise intermediate tool invocations, and validates its effectiveness both search and gedereenerttion scenarios.
CONCLUSION
We present the Tool-call Reward Model (TRM). special process reward model that provides fine- grained supervision for tool invocation large language models. TRM enables more precise credit assignment for each tool call, mitigating issues with outcome-only reward signals such gradient conflict. We systematically study TRM construction and propose effectivv integration strategies with classical RL algorithms, including turn-level credit assignment and advantage estimation. Experi- ments on search- baseh QA and code-based math tasks show that TRM consistently improves tool usage and generalization across various model sizes and RL methods. Our findings demonstrate that robust TRM performance can be achieved with moderate model sizes and limited training data. We believe TRM offers promising direction for advancing agentic capabilities in L.MMs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the China NSFC Projects (62576212, 92370206, U23B2057, 62120106006), and the Shanghai Municipal Science and Technology Project (25X01020286)).
REFERENCES
:, Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bo Li, Bowen Li, Linyi Li, Boyi Jiaheng Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, Tingkai Liu, Yongfei Liu, Rui Long, Jing Mai, Guanghan Ning. Z. Y. Peng, Kai Shen, Jiahao Su, Jing Su. Tao Sun, Yifan Sun, Yunzhe Tao, Guoyin Wang, Siwei Wang, Xuwu Wang, Yite Wang. Zihan Wang. Jinxiang Xia, Liang Xiang. Xia Xiao, Yongsheng Xiao, Chenguang Xi, Shulin Xin, Jingjing Xu, Shikun Xu, Hongxia Yang, Jack Yang, Yingxiang Yang. Jianbo Yuan, Jun Zhang, Yufeng Zhang, Yuyu Zhang, Shen Zheng, He Zhu, and Ming Zhu. Fullstack bench: Evaluating llms as full stack coders, 2025. URL httthhtt/:////tt .wwpiivv.orrg.// http:/bb///// 2412. 00535
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang. Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for Iims via reinforcement learning, 2025a. URL 2503. 19470
Mingyang Chen, Tianpeng Li. Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang. Jeff Z Pan, Wen Zhang. Huajun Chen, Fan Yang, et al. Learning to reason with search for Ilms via reinforce- ment learning. arXiv preprint arXiv:2503. / /9470, 2025b.
Jie Cheng, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Gang Xiong, Yisheng Lv, and Fei- Yue Wang. Stop summation: Min-form credit assignment is all process reward model needs for rea- soning, 2025. URL htthhhtp/://// http:://aaxxi...o//bbbbbs/ wrg..bs///////////o/////////// 15275
Sanjiban Choudhury. Process reward models for IIm agents: Practical framework and directions, 2025. URL httphttsss//axii.or////////////
10325
Yong Deng, Guoqing Wang, Zhenzhe Ying, Xiaofeng Wu, Jinzhen Lin, Wenwen Xiong, Yuqin Dai, Shuo Yang, Zhanwei Zhang, Qiwen Wang, Yang Qin, Yuan Wang, Quanxing Zha, Sunhao Dai, and Changhua Meng. Atom-searcher: Enhancing agentic deep research via fine-grained atomic thought reward, 2025a. /tttht:s:///axiivv wwwp:/iiv.rrg//boaa 25////8.....2..... 12800
Yong Deng, Guoqing Wang, Zhenzhe Ying, Xiaofeng Wu, Jinzhen Lin, Wenwen Xiong, Yuqin Dai, Shuo Yang, Zhanwei Zhang, Qiwen Wang, et al. searhheer Enhancing agentic deep research via fine-grained atomic thought reward. arXiv preprint arXiv:2508. 12800, 2025b.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025. URL https:://aaxiv.or///aa/ /5wpg//bbs/20..... 11536
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1220 2025.
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. StableToolBench: Towards stable large-scale benchmarking tool learning of large language models. In Lun- Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties: ACL 2024, pp. 11143-11156 11156, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/y 1/2024, 1/t.://./n/22....ngggss-ee 664. http:/ttps///alllaaooolooo.or/op feeee..fiiiiigge Findings-acl.66.2

---

Published as a conference paper at ICLR 2026
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi. Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-leves bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- -0280 Bangkok, Thailand, August 2024, Association for Computational Linguistics, doi: https: ttttps::/aalaatthologg.o grgg2022... acl-long. 211/
Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httt:s//pogaaeeevveeeeee///rrffo
Xanh Ho. Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.). Proceedings of the 28th International Conference on Computational Linguistics, pp.
Barcelona, Spain (Online), December 2020. International Com- mittee on Computational Linguistics. doi: 10. 101186611000000.oonggmmm 7/aclanthology. org/ 2020. moling-mainn 580/
URL https:
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference Language Modeling, 2025. URL https: //openreview. wwww://rmfffommmmmmm herriidddwiiiiieeee
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. (0.18653/11/P17-11447 URL /tttps://aclanttee
org/
P17-11477]
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNILP), pp. 6769-6788 6671, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 10.18653/v 1200000mmmmmmmmiii..5... URL httpps://aclaaatoolggy org/ 2020. emnlp-main. 550/
Tomá' Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: 10.1 162/tacl a 00023.
URL httpthss://caaanhllloetooo www.:/rrg/////// 1023/
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability, 2025a. http:/ttttss//axx..ooo////////// 201./
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXIv:2503,23383, 2025b.
Minpeng Liao, Chengxi Li, Wei Luo, Wu Jing, and Kai Fan. MARIO: MAth reasoning with code interpreter output- reproducible pipeline. In Lun- Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 905-924, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653v1vl 4024.fndings--cl.53..
wwwpoogg..rrrgg// wrg. findings-ann maddings-acl.555 acl.537 53/
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth
12

---

Published as a conference paper at ICLR 2026
International Conference Learning Representations, 2024. URL [RRLLtt////ppeeeeeeew.eeo forum?1ddvvLOppNE001
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592 2024.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models with automated process supervision, 2025. URL htt.htttts///peeeiiiiiiiee net/forum?id KwPUQOQIKE
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd- Grabee,,, and Naoaki Okazaki (eds.), Proceedings of the s1 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: URL https: /ttts:://acllnthoologgy. org. 2023. acl-long. 546/
OpenAl. Gpt-4 technical report. arXiv preprint 2rXiv:2333.08774.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10. 10.18633/1/20233 http:///22....iinggeeeemeie findinng---mmlp.378.. Findings-emnlp 378/
URL httpss://cclanthlleee org//023.
Zehan Qi, Xiao Liu, Iat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang. Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang. and Yuxiao Dong. Webrl: Training IIm web agents via self-evolving online curriculum reinforcement learning. CoRR. URL
abs/2411.02331 2024. html#abs-2411 2411-02337
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- Türn Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.139. 2025a.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs, 2025b. https:/ts///arxxi org/ abs/2504. 13958
Qwen,
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:://arxit org. wtw..r////bbb///aoeeee
abs 2412. 15115
Timo Schick, Jane Dwivedi-Yuu Yu, Roberto Dessí, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: language models can teach themselves to use tools. In Proceedings of the 37th International Conference Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:
//arxiv. www.iiv.org/as///e/ hwrgg/bb/11000 02438
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
13

---

Published as a conference paper at ICLR 2026
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URLhtts:: http:t/pp//pppprreeeeeeeeee////e wtt.:etteeoormmm.aaa////laaaaa
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxii. org/abs/2402.03300 2(3):5, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp- 2279-1297, New York, NY. USA, 2025, Association for Computing Machinery. ISBN 0798400711961. //doi. org/ /0. 1045 004553668331111 3696075
doi: 10.
1145/368903 3369665 .3696075.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen. Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592 2025.
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of Ilms without searching. 2025. URL
04588
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. https:
7/qwenlm. gi thub. httpb.io/bboogggq-3221
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multi- hop questions via single-hop question composition. Transactions the Association for Computa- tional Linguisties, 10:539 554, 2022. doi: 10.1 a URL httlhttss///clantthie
org/2022. tacl-1.31/
Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Yamamoto Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-based and basceee-based feedback, 2023. URL https::/openreview.nnr wwt.:iwwwwwe.eoemmiiiiiiioe
MND1kmmNy@0
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. http:tttps://roceeeeeee wwwprrreeeiiinnnnrrirrr.r... cctpipps.ccppppee ccppppr paper hlles/paper/2017/fiiee 3f5ee243547deee1 fbd053c1 caao:3acccca44aaaaaaaaa Paperr..p
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. 2022.
Peiyi Wang, Lei Li, Zhihong Shao, Rumxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human anno- tations. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 9426-9439, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.cuo acl.o......-...511...
URL htttps://cllanthooogyy org.:lllooo.oooo/////e 2224.2244.022 acl-long. 510/
Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting ilms search ability via step-wise proximal policy optimization. arXiv preprint arXiv:2505. 15007 2025a.
14

---

Published as a conference paper at ICLR 2026
Ziliang Wang, Xuhui Zheng. Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting Iims search ability via step- wise proximal policy optimization, 2025b. URL
https://axxiv.org/ wwt. wwwgg/bbs/2300
15107
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao. Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self- eel--mmprvement. arXiv preprint arXiv:2409. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369- 2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10555////DD1...... 1t1.//111119-- 1259. URL httts:://aclaattoolggy D 1259/
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL
WE_vluYUL-X
Chi Zhang, Guangming Sheng, Siyao Liu, Jiahao Li, Ziyuan Feng, Zherui Liu, Xin Liu, Xiaoying Jia, Yanghua Peng, Haibin Lin, et al. A framework for training large language models for code generation via proximal policy optimization. NL2Code Workshop of ACM KDD, 2024.
Yuxin Zhang, Meihao Fan, Ju Fan, Mingyang Yi, Yuyu Luo, Jian Tan, and Guoliang Li. Reward- sql: Boosting text-to-sql via stepwise reasoning and process-supervised rewards, 2025a. URL
https://arxiv.orgg http///wwii..orabb/////aa 04671
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang. Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathe- matical reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: 2C25 pp. 10495- 10516, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979- URL https://aclaathoolgg...
8-89176-256-5. doi: 10.18553/v/20255.. htt.://w///2...nnnnnnnooooo 2025. findings-ac1.5417 5477
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrationss) Bangkok, Thailand, 2024. Association for Computational Linguis- tics. /RLL:ttt://aaxxivorr/////
13372
Yuxiang Zheng. Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui La, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environ- ments, 2025. URL http::ttt///arte orgg abs/ 2504 03160
15

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS
In this work, large language models were used solely for language polishing and improving the clarity of the manuscript. The LLMs did participate in any substantive aspects of the research, including problem definition, research motivation, methodology, experimental design, or analysis. All scientific contributions, conceptual developments, and experimental results were conducted and validated by the authors without the involvement of LLMs in the core research process.
ETHICS STATEMENT
This work does not involve sensitive personal data, or practices that raise privacy or security con- cerns. All datasets used are publicly available and do not contain personally identifiable information. The research does not present potentially harmful methodologies, applications, insights, and does raise issues related to discrimination, bias, or fairness. The authors have adhered to the ICLR Code of Ethics throughout the research and submission process.
REPRODUCIBILITY STATEMENT
All experimental details are provided in the main text 3 and Appendix to ensure reproducibility. Key code components have been submitted with this paper, and the complete codebase will be released publicly at appropriate time.
LIMITATIONS AND IMPACTS
While the Tool-call Reward Model (TRM) demonstrates significant improvements in tool-use super- vision for large language models, several limitations remain. First, our study scoped to tasks with verifiable final outcomes (e.g., factual QA and code generation), the limitations outcome-only reward RL in such settings. Extending TRM to open-ended (Guo al.) 2024) reinforcement learning, where correctness is difficult assess, would require additional mechanisms for outcome evaluation and is left for future work. Second, to keep rollouts manage- moderate length, and our current framework
primary focus is address
able and reduce judge bias, we truncate trajectories does not fully resolve the challenge of providing reliable process supervision for very long tool-use trajectories. Finally, TRM models tool utility via simple binary necessity quullty decomposition, which may be insufficient to capture more nuanced, multi-objective notions of tool usefulness in complex domains.
Despite these areas for improvement, TRM provides fine-grained supervision that enables more interpretable and robust tool usage, advancing the agentic capabilities large language models. This approach can facilitate safer and more reliable deployment of LLMs in real-world tasks requiring external tool invocation, and we hope our work inspires further research in process-level reward modeling.
16

---

Published as a conference paper at ICLR 2026
A PROMPTS
A.1 PROMPTS FOR TRM TRAINING DATA DISTILLATION
Prompt of Tool Call Evaluation for Search Scenario
#tf TASK
You are a professional Tool Call Evaluator for AI agent trajectories. For a given 4 user question and its complete step-by-step trajectory, review every tool
4 call (all are of type 'search') and assess each using the following -evaluation dimensions:
Tool Selection Accuracy
correct (1): It is appropriate to use the 'search' tool for this subtask; <>his call is necessary to make progress.
incorrect (0): Using 'search' is not appropriate here (the information is -already available, the call is unnecessary, or does help answer -th user's question).
Query Quality
perfect (1): The 'query' is clear, directly addresses the user's need, and -uses precise wording.
minor or major error (0):
minor error: There is some ambiguity or slight irrelevance, but the search
-will likely still provide useful results.
mmajor error: The query is unclear or unrelated to the user's actual need.
## INSTRUCTIONS
Evaluate every tool call (all are 'search") on both dimensions.
Briefly justify each score assign.
#* INPUT FORMAT
You will receive:
'user_ querttinesion: The original user question.
'trajectory': The full step-by-step trajectory as list of steps. Each step includes:
'step_id'
'thought': The agent's reasoning or intention before making the search.
'query': The search query issued.
'response': The information returned from the search.
Example:
...
{{
"user queertques:on": "What is the capital of France and the population of Germany <>n 2023?",
"trajectory": {(
sstep_id": 0,
"thought": "I need to find the capital of France.",
"query": "capital of France",
"response": "Paris the capital of France."
}),
{
"step_id":
"thought": "Now should get population figure for Germany in 2023.", "query": "population of Germany 2023", "response": "The population of Germany in 2023 is estimated to be about 84 -millionn"
}}
})
(continued in next page)
17

---

Published as a conference paper at ICLR 2026
Prompt of Tool Call Evaluation for Search Scenario (continued)
## OUTPUT FORMAT
Provide your evaluations as JSON list.
For each step, output object with:
'step_id'
"tool_eelectionacccrray::: or
ttoolielletion_jusifffctiini your brief justification
'query_quality': 0
Sueryycuallt.jissffaattitn'" your brief justification
Example:
[
"step_id": 0,
aeoo__sseecttoo__cccrraaa 1,
"tool_selection attol__seeetion..ssfffiaaaooiii justificatiii"on "The user asked for the capital of France, wwiich is factual information requiring search.",
"query_quality":
"The query is clear directly requests the >needed information."
D.
id"; 1,
pegoo_selectio_acccraae 1,
htw._eeelettoo__ssfffiiaaaaaie "The user needs the population of Germany in +2202,, which requires search.",
qquryy_ualitt"::
"ueryy.uullty.jussffitttoo "The query is specific and unambiguous."
}>
]
## INPUT
(input)
## OUTPUT
18

---

Published as a conference paper at ICLR 2026
Prompt of Tool Call Evaluation for Code Scenario
TASK
You are a professional Tool Call Evaluator for AI agent trajectories. For given + user question and its complete step-by-step trajectory, review every tool call (all of type "program') and assess each using the following =vvaauution dimensions:
Tool Selection Accuracy
correct (1): It is appropriate to use the 'program' tool for this subtask; twriting and executing program is necessary clearly helpful for makiing progress (e.g., for calculation, verification, complex rreasoning)..
incorrect (0): Using 'program' is not appropriate here (the calculation or -reasoning can be done easily by hand, the program is unnecessary, or it -does not help answer the user's question).
Code Quality
perfect (1): The code is complete, correct, and directly serves the intended
Pprrpose (e.g., correct imports, clear logic, no errors, and directly
Canswers the subtask).
minor major error (0):
minor error:
small issues (e.g., missing imports, minor + inefficiency), but will likely still work as intended. major error: The code is incomplete, incorrect, or does not address the +intended purpose.
progrcodeam
**Note**:
this evaluation, is acceptable for the program <or checking results that were derived by hand in previous reasoning steps. The code does not need to independently derive all intermediate long it correctly intended result. This use of code auxiliary verification considered sufficient for a perfect score, provided the
for verifying
parameters replicate the full logical chain,
verifies or computes
+ code is correct and complete.
### INSTRUCTIONS
Briefly justify each score you assign. Evaluate every toal call (all 'program') on both dimensions.
#*# INPUT FORMAT
You will receive:
'user pnser_question' The original question.
'trajectory": The full step-by-step trajectory as list steps.
Each step includes:
'step_id"
'thought': The agent's reasoning intention before programming.
'cade': The code issued (if any; otherwise may be empty).
'output": The output from executing the code (if any; otherwise may +mmpty).
(continued in next page)
19

---

Published as a conference paper at ICLR 2026
Prompt of Tool Call Evaluation for Code Scenario (continued)
Example:
{
":set_question" "What is the of the first 100 positive integers?", "trajectory":
{{
"step_id", id": 0,
"thought": "I the formula for the of the first integers, but lI'll write program to verify the result.", "code": 1e0\nresult "output": "5050"
(n 1) 21nprint(result)",
}),
{{
"ste__id::
"thought": "Now will write a program to sum the integers from 100 ddirectly.""
"code": "priin(suu(rangge(, 101)))", "output": "5050"
}}
}}
#6* OUTPUT FORMAT
Provide your evaluations as a JSON list.
For each step, output an object with:
'step_id'
"tcol_sellcttoon_ccurrra
"too__sellcttion_jsssiffaaaatt your brief justification
'code_quality": or
"code.uallty.uusffiatiin""" your brief justification Example:
((
"step_id": 0,
"tool_selection.ed accuuracycccrrr" 1,
"Using a program verify the formula is reeaooabbe and helps ensure correctness.""
"code_qqulll qoodd___ultty"
httpe_uualty._sssffiiatio.. "The program is correct, complete, and directly computes the required sum."
}) {{
step_id": 1,
wttpsseeeeiio__cccrrreeee 0,
"Writing second program the same licalculation a different way is redundant and not necessary for +solving the user's question.",
"code_que coode_quality":
httee_uuaaltyjsssffiitttto "The program is correct and concise, but does <noo value beyond the previous step."
}}
]
### INPUT
(input)
#*# OUTPUT
20

---

Published as a conference paper at ICLR 2026
Prompt of Tool Call Evaluation for Multi-tool Scenario
##
You are a professional Tool Call Evaluator for AI agent trajectories. For a given user question, list of available tools (with descriptions and -+parameter schemas),
complete step-by-step trajectory, review every tool call and assess each Uusing the following evaluation dimensions:
Tool Selection Accuracy
correct (1):
The chosen
matches intended subtask and is consistent with tith the -oool's description and schema.
The call is necessary or clearly helpful for making progress toward
-answering the user's question fulfilling the user's request.
incorrect (0):
The chosen tool does **not** match subtask (e.g., wrong tool given -intention or user need).
Or the call is redundant unnecessary (e.g., the information is already -vvailabbee from earlier steps, the call does not help answer the 4's question).
Argument Quality
perfect (1):
The arguments to the tool correct, complete, and specific.
They respect the tool's s parameter schema (types, required fields) and align
< the user's s need or the agent's stated intention.
minor or major error (0):
minor error:
Small mismatch, ambiguity, slight irrelevance in arguments that still llikeyy allows tool to work and return useful results.
major error:
- Missing required fields, wrong types, wrong values, or arguments that do -nntt actually reflect the intended subtask or the user request.
The tool would likely fail, error, return irrelevant / unusable
resuuls.
## INSTRUCTIONS
Evaluate every tool call both dimensions. Briefly justify each score you assign.
#* INPUT FORMAT
You will receive:
'user queer_question The original question.
'trajectory': The full step-by-step trajectory as list steps.
Each step includes:
'step_id'
'thought': The agent's reasoning intention before making the search.
'tool_calls' The tools invoked.
'response': The information returned from the tool calls. 'wailaabl_toolss:: list of available tools with their descriptions and -xparameter schemas.
(continued in next page)
21

---

Published as a conference paper at ICLR 2026
Prompt of Tool Call Evaluation for Multi-tool Scenario (continued)
Example:
{
"user_question": "What is the capital of France and the population of Germany <in 2023?",
"available_tools"
{{
"name": "search",
"description": "Use this tool search for factual information from the
wee..."
"parameters": [(
"type": "object", "properties": {{
"query": {{ "type": "string", "description": "The search query string."
}}
)). "required"; ["query"]
}}
}}
].
"trajectory";
{(
"step_id": 0,
"thought": "I need to find the capi tal of France.", "tool_calls" [[C
"name": "search",
"arguments": {{
"query": "capital of France"
}}
}],
"response": "Paris is the capital of France."
)),
{
"step_id":
"thought": "Now should get the population figure for Germany in 2023.".
"tool_calls": [f[
"name": "search",
"arguments": {(
"query": "population of Germany 2023"
}}
}}].
"response": "The population of Germany in 2023 is estimated to be about 84
=millionn"
}}
}}
..oottinuee in next page)
22

---

Published as a conference paper at ICLR 2026
Prompt of Tool Call Evaluation for Multi-tool Scenario (continued)
## OUTPUT FORMAT
Provide your evaluations as a JSON list.
For each step, output an object with:
'step_id'
"too__selection_cccrree.'. or ittoL.eeletioo..usifiiaatiii your brief justification
prrgumentqqallity" 1 or
your brief justification
Example:
[
{
"step_id"; 0,
ieto__seeectionacccrreee 1,
htt.sseelettoo_jssiffictteteee "The user asked the capital of France, wwiich is factual information requiring search.",
"yrgument__uuality"":
http:atttuuuaitttssssiiiitaaaaa "The query is clear and directly requests +he needed information."
}),
[[
"step_id"; 1,
ppool_selection_accraaa)
ccol_eelecionn_usstiiieeeoo "The user needs population Germany -2023, which requires search.",
"argument_quality"::
"rrgumettt_ualllyyuusiifiiioooo "The query is specific and unambiguous " }}
]
## INPUT
...
(input)
## OUTPUT
A.2 SYSTEM PROMPTS FOR TASKS
System Prompt for QA Tasks with Search Tool
Answer the given question. You must conduct reasoning inside <thinke and </think>
+ first every time you get new information. After reasoning, if you find call search engine by <search> query </ +saachh> and it will return the top searched results between <tool_response <> and </soos_rspponse>. You can search as many times as your want. you find no further external knowledge needed, you can directly provide the aanswer inside <answer> and </answer>, without detailed illustrations. For exampple, <answer> Beijing </answer>.
+you lack some knowledge,
23

---

Published as a conference paper at ICLR 2026
System Prompt for QA Tasks without Search Tool
Answer the given question. You should first have a reasoning process in and +heen provides the answer. Show your reasoning in <think> </think> tags and return the final answer in <answer> </answer> tags, for example <answer> BBijjng </answer>.
System Prompt for Mathematical Problems with Code Tool
Solve the following problem step by step. You now have the ability to selectively
write executable Python code to enhance your reasoning process. The Pytton code will be executed by an external sandbox, and the output sttissssrr/ttol_sssssssee
twaapped
returned to aid your reasoning and help you arrive the final answer. The Python code should be complete scripts, including necessary imports. Put your final answer within \bboxed().
System Prompt for Mathematical Problems without Code Tool
Please reason step by step, and put your final answer within Wboxed().
System Prompt for Real-world Problems with Multi Tools
In this environment have access to set of tools you can use to assist with < query. You may perform multiple rounds of function calls. In each round, you can call one or more functions.
Here are available functions in JSONSchema format: ''json (func_schemas)
In your response, need to first think about the reasoning process in the mind and then conduct function calling get the information perform the actions needed. The reasoning process and function calling are enclosed within sthink> </think> and stool_call> </tool_call> tags. The results of the function calls will be given back to you after execution, and you can provide the final answer encclsee within <answer> </answer> tags for the user's question.
continue to call functions until
Tool call example:
<tool_call>
{{"name": <function-name>, "arguments": <args-json-object>))
</tool_call>
Final answer example:
<answer> </answer>
24

---

Published as a conference paper at ICLR 2026
B TRAINING DETAILS
B.1 TRM TRAINING DETAILS
For the search scenario, we use Qwen2.5-3B-Inen examples. For the code scenario, we adopt Swen2.5-Math--1.3 as the backbone and utilize 20K training samples. The training process follows standard supervised fine-tuning procedures, and the key hyperparameters are summarized as follows. We set the number of training epochs 10, with a learning rate le-6. The global batch size 128. The maximum sequence length is 8192, and maximum prompt length is 1024. All experiments conducted using Huggingface implementa- tior
as the backbone model and train with 10K
B.2
RL DETAILS
Tool Execution Environment For the search tool, we follow the setup of Search-RI (Jin al. 2023) and use the 2018 Wikipedia dump (Karpukhin et al. (020) as the knowledge source. The E5 (Wang et al. 2022) retriever is employed to retrieve relevant passages for each query. For the Python code execution environment, we follow the approach in ToRL. (I.i et al. (025b) and utilize the SandboxFusion environment (Bytedance- Seeeee--Seeeeeeee Foundation- Coroonn--Co----aee Team et al., 2025) ) to safely execute code snippets. This setup ensures both the reliability and security of tool interactions during reinforcement learning experiments.
Table 3: Hyperparameters in RL. The notation 3B / 1.5B and 7B / 7B denote the backbone model sizes used for different tasks: the first value corresponds to the search tool for QA, and the second value corresponds to the Python code tool for mathematical problem solving.
PPO
GRPO
Hyperparameter
I.5B
7B 7BB
/ I.5B
7B/7B
trainer. Vooiner.tttal.traae sotin_trainnn
500
300
500
300
algorithm. wtt..adv.estimaa..
gae
gae
grpo
grpo
data. train.batch.size
512
512
512 128
512/ 128
actor_res attorrrolooutreeee ref. actor
www.trppoo.oeeeeeeeee apoomini._mm httpm/iii.bathh.iiii. batch.size
256
256
256 64
256
data. mattamaa._ppp. mex_ppomptt_. progpt_length
8192 / 3072
8192/ 3072
8192 / 3072
8192 3072
data. max_ responess length
512 1024
512/ 1024
512 1024
512/ 1024
max tool resp len
512
512
512
512
actor rollout ret.actor... cecttor.opiim.i... Pritic.optim...
2c-7 / 1c-6
2c-7 / 1e-6
2e-6
2c-6
Se-7 / Se-6
Se-7 Se-6
actor rollout ref.actor entropy coeff actor actor.rollout rollout. temperature octor_rollout.rff rollout.n
eef.actor.entropy
0.001
0.001
0
1.0
1.0
1.0
1.0
1
5/8
5/8
tools. malls.maxtturss.
5 5/3
5/3
5/3
5/3
algorithn. 11lctrl.kl.coef actor act.orrrrrr...aor.........o.....
0.001
0.001
actor. kl_loss_co coef
0.001
0.001
presents the key hyperparameters used in our RL experi- ments. All other training configurations follow standard practices described in the main text.
Training Hyperparameters Table 3
C EVALUATION DETAILS
C.I TRM EVALUATION
For evaluation, we use the checkpoin from Search-R1 (Jin et al. 2025 to collect rollout candidates from the prompts in validation sets HotpotQA and 2wikimultihopQA During rollout generation, we set the sampling temperature to O. Additionally, the agent is allowed to perform up to 3 search steps per query.
3
https://huuginngfcee http://///naaaoooooooo///ttt//// trainer
https/aaa...oeeeeeeenn////errrae https//s/ngggnnneeeee/eeeeeeaasr hotpotqa train-qwen2. 3-7b-ap em- em-ppo-ve ppo-v0.2 v0.2
25

---

Published as a conference paper at ICLR 2026
0.8
0.73
Averoge Tool Call Number
2.5
Call
2.0
0.4
0.3
w/o TRM
TRM (Group Level)
TRM Level)
100
200
300
400
500
Training Step
(a) The trend of TRM scores with respect tool call rounds
(b) The trend of average tool call counts during training
Figure 7: Reward hacking in group-level advantage estimation for GRPO: (a) TRM scores decrease with more tool calls, and (b) turn-level estimation mitigates the penalization of longer tool call sequences.
C.2 LLM EVALUATION
For LLM evaluation, we set the sampling temperature to 0 to encourage deterministic generation. Other parameters, such as the maximum number tool calls, are kept consistent with those used during training. Notably, since AIME24 and AIME25 contain very few problems, report the average results over 30 repeated evaluations for these two datasets to ensure statistical reliability,
D BASELINE DETAILS
Training-fres Methods For IRCoT and RAG, we mainly use the implementation [1]
Re-
Search (Chen et al. 2025a).
Training Methods For SFT, we adopt LLaMA-Factor Zheng et al. 2024). For R1, we simply disable tool invocation in our framework.
E ADDITIONAL EXPERIMENTAL RESULTS
E. REWARD HACKING CAUSED BY GROUP-LEVEL ADVANTAGE ESTIMATION
Group-level advantage estimation in GRPO can lead to reward hacking, where the model prefers shorter tool call sequences. This is because cascading errors make later tool calls less reliable, resulting in lower scores and penalization for longer sequences (Figure aa). In contrast, turn-level advantage estimation alleviates this issue by treating each tool call independently, encouraging more stable tool usage (Figure77bs Tool-call numbers evaluation benchmarks in Table are consistent. Notably, introducing TRM does not significantly increase the number of tool calls compared to outcome-only training methods.
E.2 RESOURCE OVERHEAD INTRODUCED BY TRM
incorporating TRM introduces only an 8.8% overhead per training step, indi- cating minimal additional compute cost. BoN inference experiences a 50% increase in per-sample time with TRM; however, the absolute time remains small (with the full BoN evaluation taking ~ 20 minutes), which is practically negligible.
As shown in Table 5
10 Due to the use of the vL.LM server, same randommess may still be present during evaluation.
whttps:: thub.bb...bbt...ec http.//cwweaaeeeeeeeeeo/rrrrar/e
26

---

Published as a conference paper at ICLR 2026
Table 4: Average tol-call numbers on various QA tasks over Qwen2.5-3B-Innse Instrrcctt with PPO: turn-
level vs. group-level
General QA
Multi-Hop QA
Method NQ
Avg.
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Bamboogle
Group-level 1.99
1.93
2.00
2.42
2.81
3.01
2.37
2.36
Turn-level
2.59
2.44
2.48
2.68
3.01
3.19
2.68
2.72
Table 5: Training and BoN inference speed with vs. without TRM on Qwen2.5-3B-Instruc with 8xA800 GPUs under PPO
Method
Training (s/step)
BoN Inference (s/sample)
w/o TRM
56.9
0.14
w/ TRM
61.9 (+8.8%)
0.21 (+50%)
E.3 TRM TRAINING DATA QUALITY VERIFICATION
Regarding TRM training data quality, randomly sampled 100 trajectories and evaluated them using both human annotators and multi-model judge (DeepSeck-RI, Qwen2.5577BBInnttuu and GPT-4o-mini). As shown in Table 6 the resulting dataset demonstrates relatively high quality.
Table 6: TRM Training Data Quality Evaluation (Accuracy)
Evaluation Method
Necessity (%)
Quality (%)
Human
89
89
Mul mimdeel
86
75
E.4 GENERALITY BEYOND SEARCH AND CODE TOOLS
We follow the ReCall (Chen et al., 2025a) work to evaluate our method in more diverse multi- tool scenarios. These scemarios include tasks such as library management, employee management, and travel planning, which require the use multiple APIs (c.g., searching and reserving books, calculating salaries, updating flight bookings, etc.). These additional experiments (see Table 7 show that introducing TRM continues to provide consistent improvements broader multi-tool settings.
Table 7: Performance in more diverse multi-tool scenarios on Qwen2.5-7BB-nsttuu. with GRPO
Method
Swen2.5-7B-Instee
ReCall
ReCall-TRM
F1
7.37
39.71
43.28
E.5
CASE STUDY
27

---

Published as a conference paper at ICLR 2026
Question:
working
Model w/o TIM
Model
To desermine
the might
Thought general to gather Mency the Working
song
don direct information online, iighh need
recorth music
Tool-call
have morcy working
Toal-call: wnote how menry the working men
Obserratton:: Have Mancy un Working Man Have Mency the
Obsenatton... lord Have Mercy on the Working ng Man "Lord Hove Mercy on the Working Man is song writtan by Rostas and recorded by Amer kaan country music singer Travi Tittt..
Working song Kostas and socordeed by Americaae country
singer
Turn 2:
Sarfurn
Thought: from the search results, found that "Lord hove Marcy on the Working
Thought: From the search results, hound thet "Loed Hove Mency the Working Mon" song pritten dostas and recorded by American erican country musi Trave Trie
Man" a sona written by Sostas and recorded by American councry music ainger
Trauk frim. ineed to cordiom this the cornect Information.
Tool call: who kodtb contout of the song Lord Have Morcy on the Working Mon
Answer Koubas
Obseraaton: Kostas Hatzs Kostas Hatas (born 18, 1936 Greek
musrden Romani
Tara
Travis
Comment::
Both models (with and without TRM) adopt ressonable tool call strategies, but only the TRM-integrated model produces the correct arswer The error made by the model without TRM highlights the motivation for our approach and shows that TRM helps address cascading ervers in tool usige.
Figure 8: Case
Question: ceneinee co
Find remainder
2o
waaliate product ITt 2uf (2) where ulk 13ts mot unity, and then find the nersi noer when
Thought: sabo problem,
[12e, 2n* -r juhore a is 13th unity and than find the remeirder when product divided by 1000
Thought solve given problem.
product
this productis dhiddd by 1000
Rist, let's uncerstane nd the propormes the 15th roots unity. The 13th roots
First, le''s understond the properttes the 13th roots of untyy. The 12th roots of unity are the volumone to the equition yf These mohs can wittten as cirit for 0.122 12).
unity the solutions to the equation a Thise roots can writte of for
espeassion 12, where po
(1)
expression (2 2u"
simplitted product from
sum
unity awro, Xie
then
Let's down problum manageable staps:
Define 18th
Compute the procuct [liia h
python
find the remainder meiideereee when the product videe by 3000.
Import
We will use Python and the sympy librar perform there crloulations.
Took-Cail:
"python
benego crroth.pl
import 26
Compute product
# Define the 13th root ctuiity
omaga sp.e "spppi sp.l/
rangolnkc torm
Compuse the product product
sp.simplify unoble
-...............................
product oomogo ** omago "122
(133)
with high-complexity problem reesonaable time
cart product
imaginary part shoule de negligitee)
product hh 12-22 cmagae umage 1/2
peoduct roundlpend real
Simplify the product
remoinder when product divited
product www.etsssspmaaaaeeeee "ijpmttttrrtte"""""
remairder product, yeal 1000
Find the remsinder when prodruct divided 2000
primrremannee))
remainder Medlproduct, 1000)
printrrmmainder
Obeerraation:
Obeservation: Run result-hhone>
Ture
Thought The rema when the product [Ttla Zee a diviedd [21]]
similar
answert
(
2tt by 1000
answwer
Comment
Without TRM, the nodel gererates everly complex code that fails during execution, resulting a wrong answer With TRM, the model procuces Higher-quall quali--uultyyyyuu code and answers cornectly. This case highlights how TRM improw reproves code generation quality and helps miligete errors, validating our motivetion for integration.
Figure 9: Case 2
28

---

Published as a conference paper at ICLR 2026
Question:
Model w/o TRM:
Model w/ TRM:
G
Turn 2:
Thought: insed find information about Arnald Palmer Regional Airport and Muñoz Maris nnterationll Airport their locations.
Thought: meed find out the locationa of Amald halmar Hogional Aipport and Luia
Mutoe Marin International Airport to determine they are the same country.
Teal-Cell: Arnold Palmer negional Airpor:
Tool-Cal dernnd Palmer Be Regional Airport location
Observatoo:: Aroll Piemet Regiored Airontt Amall hulmer Regional Aipport sublic
Observation Aroold Polmer Regional Airport Amold Poimer Rogional Airport public sirport in the costern Unhted States, located in moreland County. Pe Pennsyvania...
United locoted Westmoretand County.
Pennsyvvania.
G Tum
Turn
information Amold Paimcr Regional found the
Thought: The Arnold Palmer Rleginal Airport located Wendmoellend Courtyy Bennylaanss, the United States. Nuw Inred find lsation Lus Wartes
Marin International Arport
International
Oiservation...
Mar'n Internationll Airport Mutorr Maris Internationall Aipert joint chel-milttry interntioonal nammed Puett flrst democroticaly elected
cid--riltary sirport democraticaly gowemor
suburban Carolina,
Puerto Rice,
Juan..
Regional
Penroshanis, United
Thought: information intemationaal Airporia
Regional Airport found
Puerto
Now States Puerto same country
ifbth airpotss sams country..
Observation: fullowing the Sparikh--Amarion Whe Unitted Staten' axquired Puerto lioo under terma freatty of
The inform motion about Ammid Polmer Regional Airport is found the United Stores, Muñice Warth intemational Airport located Puerto Rico. Now need to determine both airports are some country
Thought:
Overuse of tools
Tool call: United States and Puerto foco same country for arrports Observation: .. transprtation Puerto Rico Transportation Puerto Nico ircludes system of moth, highwweys freeteeys, airport, ports and he herbon...
Jumn
Thought: dionee
anower Teeman Sea
Comment:
The midel without TRM overuses tools and misses key intermal inf information, resultigg in unnecessary teol calls and wrong arower. With TRM, the model makes two well targeted tool calls and answers correctly, showing that TRM helps reduce redundont tool uoage and improves arower accuracy.
Figure 10: Case 3
29

---

Published as a conference paper at ICLR 2026
E.6 PROCESS-LEVEL TRANSPARENCY OF TRM
Table Examples failed tool calls
Question
Tool Calls
Comment
Explanation
What
search(Bill Dvorak tionality)
low-quality
The query asks about Bill Dvorak's tionality which unrelated to the user's need (fifth cousin's nationality). It flects a major misinterpretation of the question.
Bill Dvorak's fifth cousin's
na-
nationality?
Are Arnold Palmer Re-
search(Arnold Palmer Re- gional
unnecessary
Invoke duplicate tools
gional Airport
Air-
and Luis
port)
Muñoz Marín
search International States and Airport lo- cated the
Puerto Rico in the same country) search(United States and Puerto Rico in the same country for airports)
same country?
product $8 .25 2
The
Calculate the product by step
unnecessary
The multiplication of four numbers (8, 0.25, 2, 0. 0.125) is simple and can be eas- ily computed by hand; writing a pro- gram is unnecessary for this straightfor- ward arithmetic.
step
.125 =$
result
0.25
2
0.125
print(result)
Medians SBDS and SCE$ of trian- gle SABCS are perpendicular, $BD=8$, and SCE=12$. The area of trian-
Very long code
low- low-quality
The code contains significant unused symbolic logic (Sympy setup/solving) that's irrelevant to the final arithmetic operation, which inefficient and shows flawed implementation. Though the output is correct, the dead code con- stitutes major structural issue.
gle $ABC$ is
E.7 DETAILED EXPERIMENTAL RESULTS
30

---

Published as a conference paper at ICLR 2026
Table 9: Detailed results of Figure
Method
Model Size
Training Data Size
Avg
2wikimudthopqa (out-of- noutoof-domain (dev split)
Majority Vote
11.73
11.73
10.82
10.76
10.56
11.12
Passên
11.73
15.16
17.40
19.02
20.26
16.71
1K
11.73
13.53
15.33
15.76
14.19
2K
11.73
13.79
15.02
15.80
16.39
14.55
TRM
3B
5K
11.73
14.33
15.94
16.94
17.80
15.35
10K
11.73
14.29
15.92
16.94
17.66
15.31
0.5B
11.73
12.31
12.81
13.10
13.20
12.63
TRM
1.5B
10K
11.73
13.88
15.17
15.97
16.68
14.69
7B
11.73
13.36
14.54
15.20
15.66
14.10
hotpotqa [in iin-domain] (dev split)
Majority Vote Passên
24.52
24.52
25.05
25.08
25.29
24.89
24.52
30.97
33.96
35.83
37.54
32.56
1K
24.52
28.16
29.74
30.40
31.02
28.77
2K
24.52
26.85
27.93
28.66
28.82
27.36
TRM
3B
5K
24.52
28.82
30.70
31.61
32.40
29.61
10K
24.52
29.03
30.90
32.14
32.88
29.89
0.5B
24.52
25.23
25.28
25.21
24.85
25.02
TRM
1 1.5B
10K
24.52
27.86
29.40
30.26
30.71
28.55
7B
24.52
26.91
27.75
28.53
28.94
27.33
Table 10: Detailed results Figure
and Figure b
Method
NQ
TriviaQA
PopOA
HotpotQA
2wiki
Musique
Bamboogle
Avg.
0.01
38.14
55.20
36.17
32,64
31.00
11.21
20.80
32.17
0.05
39.58
57.78
40.61
34.80
33.22
12.91
25.60
PPO
0.1
40.08
55,82
39.11
32.91
32.73
11.12
27.20
34.14
0.3
34.52
50.08
35.98
29.41
26.81
9.10
28.00
30.56
0.01
47.89
62.57
47.20
44.47
43.48
19.65
39.20
43.49
GRPO
0.05
48.09
63.04
46.93
44.66
43.45
19.20
37.60
43.28
0.1
46.68
62.58
45.93
43.47
42.89
16.88
38.40
42.40
Table 11: Detailed results of Figure 5
Search-2 Code
Method
Avg.
MATH500
Olympiad
AMC23
ToRL-PPO
50.40
24.00
25.00
33.13
ToRL-PPO- (ours)
54.20
26.22
27.50
35.97
ToRL-GRPO
52.80
22.81
30.00
35.20
TORL-GRPO-TRM (ours)
56.60
27.70
35.00
39.77
Table 12: Detailed results of Figure
Method
NQ
TriviaQA
PopQA HotporQA
2wiki
Musique
Bamboogle
Avg.
Performance
Search-R1 StepSearch
37.53
55.17
39,20
29.75
27.65
7.74
17.60
30.66
Search-R1 AgentPRM
38.01
54.62
37.07
32.78
31.15
10.22
19.20
31.86
Search-R1 ORM
39.47
56.17
40.93
29.63
26.65
6.83
9.60
29.90
Search-R1 TRM-verifier
35.65
54.10
36.46
33.91
33.90
13.86
27.20
33.58
quality-only
38.95
56.17
38.94
31.06
27.93
8.44
16.00
31.07
necessity-only
37.42
54.06
37.22 47.21
32.46
31.80
11.46
21.60
32.29
Tool-cull Number
quality-only
3.96
3.94
3.93
3.96
3.98
3.99
3.93
3.96
necessity-only
2.58
2.68
2.58
2.83
2.95
3.31
2.84
2.82
2.37
2.39
2.37
2.85
3.35
3.23
2.71
2.75
31


Reviews:
[{"review_id": "U2kwPU1mlP", "reviewer": "Reviewer_TgrD", "summary": "This paper proposes a Tool-call Reward Model (TRM), a process reward model and a training/credit-assignment recipe to integrate TRM with PPO/GRPO. The authors distill per-turn \"necessity\" and \"quality\" labels from tool-enabled rollouts, train a binary classifier head on top of a small LLM (1.5B–7B). Experiments on search-based QA and code-based math claim consistent gains over outcome-only RL baselines.", "questions": "See above", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is straightforward and easy to follow. It shows consistent improvements on search-QA and code-based math.\n\n2. The paper identifies the limitation of outcome-only rewards for agentic tool use and motivates per-call supervision with concrete failure modes.\n\n3. It finds that mid-sized TRMs (1.5B–3B) trained on ~10k labeled trajectories suffice, with larger models overfitting at this data scale", "weaknesses": "1. Insufficient comparison with existing tool-augmented LLMs. It omits direct comparisons to process-supervised tool-use methods that score steps or calls (e.g., rule/PRM-guided selection for search or code).\n\n2. Missing positioning relative to tool-based RM in the literature. The paper does not cite or discuss the line of work referred to as tool-augmented reward modeling [1]. The motivation of proposing Tool-call Reward Model is unclear. The authors did not discuss with previous tool-augmented RM and completely ignores previous literature. \n\n3. While turn-level estimation mitigates \"fewer calls is better\", the paper does not report behavioral metrics, such as average #tool-calls, redundant calls, early-stop rates, or task-time/latency impacts under TRM vs. baselines.\n\n4. TRM inference is performed during RL and optionally at inference (best-of-n). The paper should quantify the compute/memory overheads.\n\n5. Lack sufficient ablations. Ablations to disambiguate distillation vs. TRM:\n   - Train a trajectory-level RM/PRM (no per-call labels) on the same data, and compare to TRM under equal compute.\n   - Train a tool-based RM such as [1] on the same data, to demonstrate the good claims of PRM. The paper states TRM is a specific PRM for tool invocation, yet it does not compare to a generic PRM trained on the same trajectories. At minimum, the authors should implement a standard PRM that scores intermediate steps or turns\n\n6. The training data is completely generated by LLMs, without mentioning any interventions or pipelines. The author did not report the details of distilled data.\n\n7. It is unclear how the authors evaluate Tool-call RM on downstream tasks, such as AIME.\n\n8. The authors evaluate TRM on best-of-n inference. The author should report the comparisons with previous reward models. Additionally, the author may report the RL training performance using proposed TRM.\n\n9. The usage of \"necessity\" and \"quality\" requires a detailed ablation comparison, while the paper only describes the concept/motivation in the method section. Most importantly, and how to do the quality check, how to measure the ground truth score of the intermediate steps, and how does this impact on the final results, remains still unclear.\n\n\n\nReferences:\n\n[1] tool-augmented reward modeling. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MLdGAOcvCx", "reviewer": "Reviewer_8EFK", "summary": "This paper proposes the Tool-call Reward Model (TRM), a specialized process reward model designed to evaluate and reward tool invocations. The authors further enhance TRM's application in PPO and GRPO training through refined credit assignment and turn-level advantage estimation. The effectiveness of the proposed method is demonstrated on search-based QA and Python code-based mathematical tasks.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Current tool-calling agent training predominantly relies on outcome rewards. The proposed TRM addresses a crucial need for process-level evaluation in tool-calling agent training, which is highly valuable for the field.\n2. The introduction in this paper is detailed, with comprehensive explanations of both the background and methodology.\n3. The authors conduct comprehensive experiments, including both scaling assessments of TRM and practical evaluations on search-based QA and coding tasks. The experimental results demonstrate consistent and substantial improvements over baseline methods across different tasks.", "weaknesses": "1. The paper lacks critical details regarding the construction of TRM training data. Specifically, how do the authors ensure the correctness of annotated tool call rewards? Evaluating the correctness and necessity of tool invocations at each turn requires a comprehensive understanding of the overall problem and reasoning process, which poses significant challenges for models, particularly when tools provide information beyond the model's knowledge scope. Have the authors conducted any validation of the annotation quality?\n2. For open-ended tasks such as search-based QA, it is challenging to directly assess the effectiveness of final results. The paper does not adequately address how such evaluations are conducted, which raises concerns about the reliability of the reported improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ffn3cc9nZ", "reviewer": "Reviewer_S77r", "summary": "The paper introduces a Tool Reward Model (TRM) that scores each tool call with a binary “necessity × quality” signal and combines these turn-level scores with an outcome reward for the final answer. TRM is integrated into PPO/GRPO via turn-level credit assignment and is used for RL training.", "questions": "1. How do necessity and quality contribute individually? A comparison among (i) necessity-only, (ii) quality-only, and (iii) the combined  reward (as in the paper) would be helpful.\n\n2. How do you ensure the distillation labels remain reliable for long rollouts, given that feeding entire traces to a judge model may introduce unfaithful judgments or evaluation bias?\n\n3. Beyond search and code, how well does TRM generalize to other tools?\n\n4. How does TRM size affect downstream results? Does a scaling trend emerge for reward-model size?\n\n5. Adding a comparison to existing PRM/agent-PRM baselines such as [1] would further strengthen the evaluation.\n\n[1] Process Reward Models for LLM Agents: Practical Framework and Directions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The writing is clear.\n2. Extensive experiments validate the effectiveness of TRM.", "weaknesses": "1. Necessity/quality labels are auto-judged by the same model family that produced the rollouts, no multi-judge or human verification is reported, which may imprint model bias.\n2. Tool coverage is narrow (search, code), generality to other tool families is unclear.\n3. There is no direct comparison to current PRM/agent-PRM baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DPZNTR0F6r", "reviewer": "Reviewer_R2Sf", "summary": "The paper introduces TRM, a Tool-call Reward Model for LLMs tool learning. TRM is a process reward model that assigns per-tool-call rewards based on necessity and quality, so that it overcomes the limitations of outcome reward signals that provide credit only for the final result. The paper describes methods for constructing effective TRM model and introduces integration strategies with RL algorithms including PPO and GRPO using refined credit assignment and turn-level advantage estimation. Experiments on both search-based QA and code-based math tasks demonstrate robust gains over RL with outcome rewards.", "questions": "- In principle, the reward function designed in the paper encourages the base model to act both effectively (quality) and efficiently (necessity). Most of the evals in the paper are about effectiveness regarding the answer accuracy. Is there any eval regaring the efficiency of tool use by the learned model? For example, the learned model should use less tool calls to get the correct answer compared to baseline methods.\n- How would the proposed TRM construction pipeline adapt to tool types with more complex or less binary notions of utility?\n- Have you analyzed failure cases of TRM or inspected which features/representations contribute most to per-tool-call utility predictions, possibly to increase process-level transparency?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of designing a PRM for tool-integrated reasoning is well motivated and very timely.\n- The reward function designed to evaluate necessity and quality is reasonable, and empirical results showed it worked well.\n- Empirical results showed that the proposed reward functions worked pretty well on search and code tool related datasets.", "weaknesses": "- The eval on search tool and code tool are relatively thorough. Although these are important, the approach is advertised as broadly applicable—yet evidence supporting cross-task generalization is mainly anecdotal, and the set of tools remains narrow. A more ambitious empirical scope (e.g., more diverse tool types, additional real-world external APIs) would strengthen the claim of general utility.\n- TRM relies on a strong model (DeepSeek-R1) to annotate and produce the actual reward values for both necessity and quality. This essentially brings extra knowledge and inductive bias into the training process, which might be a bit unfair. The compared methods (including Search-R1) only use the training dataset (the ground truth answer label).\n- The baseline missed some leading program-aided tool-integration works (e.g., PAL, StableToolBench).\n\nMinor issue:\n1. To be more precise, a reward function instead of a reward model is proposed in this paper to guide more effective/efficient tool calls. There is not really a reward model being trained to guide something that is unverifiable in the answer and needs tool call. As a result, I do not think the \"Reward Model\" in the paper title is appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)