ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
Authors: name: Mayank Agarwal; name: Ibrahim Abdelaziz; name: Kinjal Basu; name: Merve Unuvar; name: Luis A. Lastras; name: Yara Rizk; name: Pavan Kapanipathi
Published: 2025-09-15 14:17:17

Abstract: As large language models (LLMs) increasingly interact with external tools,
reward modeling for tool use has become a critical yet underexplored area.
Existing reward models, trained primarily on natural language outputs, struggle
to evaluate tool-based reasoning and execution. To quantify this gap, we
introduce FC-RewardBench, the first benchmark designed to systematically assess
reward models' performance in tool-calling scenarios. Our analysis shows that
current reward models often miss key signals of effective tool use,
highlighting the need for domain-specific modeling. To address this, we propose
a training framework for outcome-based reward models using data synthesized
from permissively licensed, open-weight LLMs. We train models ranging from 1.7B
to 14B parameters and evaluate them across seven out-of-domain benchmarks.
These models consistently outperform general-purpose baselines, achieving up to
25\% average improvement in downstream task performance and enabling
data-efficient fine-tuning through reward-guided filtering.