Reducing Tool Hallucination via Reliability Alignment
Authors: Hongshen Xu
Zichen Zhu
Lei Pan
Zihan Wang
Su Zhu
Da Ma
Ruisheng Cao
Lu Chen
Kai Yu
Venue: ICML 2025 poster (2025)
Keywords: Tool hallucination; Reliability Alignment; Large Language Model
Primary area: applications->language_speech_and_dialog

Abstract: Large Language Models (LLMs) have expanded their capabilities beyond language generation to interact with external tools, enabling automation and real-world applications. However, tool hallucinations—where models either select inappropriate tools or misuse them—pose significant challenges, leading to erroneous task execution, increased computational costs, and reduced system reliability. To systematically address this issue, we define and categorize tool hallucinations into two main types: tool selection hallucination and tool usage hallucination. To evaluate and mitigate these issues, we introduce RelyToolBench, which integrates specialized test cases and novel metrics to assess hallucination-aware task success and efficiency. Finally, we propose Relign, a reliability alignment framework that expands the tool-use action space to include indecisive actions, allowing LLMs to defer tool use, seek clarification, or adjust tool selection dynamically. Through extensive experiments, we demonstrate that Relign significantly reduces tool hallucinations, improves task reliability, and enhances the efficiency of LLM tool interactions. The code and data will be publicly available.

Reducing Tool Hallucination via Reliability Alignment
Hongshen Xu 123 Zichen Zhu 123 Lei Pan Zihan Wang" 123 Su Zhu
Da Ma 123
Ruisheng Cao 123
Lu Chen 1235 Kai Yu Yu1335
Abstract
external tools (Qin et al., 2024). By integrating diverse tools, LLMs can enhance their ability to tackle complex natural language processing (NLP) tasks (Schick et al., 2023; Hao et al., 2023; Hsieh et al., 2023; Tang et al., 2023). Further- more, recent studies (Qin et al., 2023a; Yao et al., 2022; Cai et al., 2024) have explored how LLMs can interact with real-world tasks through tool use, establishing tool learning critical bridge between LLMs and the physical world.
Large Language Models (LLMs) have expanded their capabilities beyond language generation to interact with external tools, enabling automation and real-world applications. However, tool hallu- cinations- where models either select inappropri- ate tools or misuse them-pose pose significant chal- lenges, leading to erroneous task execution, in- creased computational costs, and reduced system reliability. To systematically address this issue, we define and categorize tool hallucinations into two main types: tool selection ballucination and tool usage hallucination. To evaluate and mitigate these issues, we introduce Rely Behleehh,,,,, which integrates specialized test cases and novel metrics to assess hallucination-awwer
However, a significant challenge arises when LLMs are re- quired to reliably invoke external tools. A major issue is Tool Hallucinations (Patil et al., 2023), where the model ei- ther selects inappropriate tools or misuses them. Addressing tool hallucinations crucial due to their significant impact on both task performance and system reliability. First, hallu- cinations during task execution can directly affect real-world systems, as tools act as the interface between LLMs and the physical world. This is particularly critical in areas such as database operations, robotic control, and scientific ex- perimentation, where incorrect tool use can cause tangible damage and pose greater risks than textual hallucinations (Ji et al., 2023). Second, tool hallucinations produce task hallu- cination that are harder to detect. Since hallucinations occur during task execution, they often result in misleading final outputs, undermining user trust in the system's 's reliability. Finally, from an operational standpoint, hallucinated tool calls increase computational costs, and reduce efficiency by generating redundant or faulty tool invocations. These challenges highlight the need for a systematic approach to understanding and mitigating tool hallucinations.
task success and ef- ficiency. Finally, we propose Relign, reliability alignment framework that expands the tool-use action space to include indecisive actions, allow- ing LLMs to defer tool use, seek clarification, or adjust tool selection dynamically. Through exten- sive experiments, we demonstrate that Relign sig- nificantly reduces tool hallucinations, improves task reliability, and enhances the efficiency of LLM tool interactions. The code and data are publicly available at https: / thub. com/ x LANCE/ Too Hal wallluucnniion
1. Introduction
To address this challenge, we pioneer comprehensive eval- uation framework for Tool Reliability, which consists of three key components. First, we systematically define and categorize tool hallucination, introducing an automated eval- uation framework and corresponding metrics. Tool halluci- nation classified into two main types and four subtypes, as illustrated in Figure 1. Tool selection hallucination occurs when the model incorrectly chooses a tool unrelated to the task or calls it at an inappropriate time. Tool usage halluci- nation involves errors in applying the selected tool, such as providing incorrect parameters or fabricating non-existent ones. To assess these hallucinations, we design a hybrid evaluation approach that combines rule-based methods with LLM evaluators, enabling automated and accurate detection
The primary goal of tool learning is to enable Large Lan- guage Models (LLMs; Gemini Team, 2023; Achiam al.,, 2023; Dubey et al., 2024) to understand and effectively use
'X-LANCE Lab, School of Computer Science, Shanghai Jíao Tong University, Shanghai, China. MMoE Key Lab Artificial Intelligence, Shanghai. China. "Jangsu Key Lab of Language Computing. Suzhou, China. "AlSpeech Co., Ltd., Suzhou, China. Suuzou Laboratory, Suzhou, China. xuhongshen @ sjtu.edu.cr Correspondence to: Lu Chen <chenlusz @ wjtu.edu.en.s. <kai www.uuuuusuuu.n.........
edu. en?> Kai Yu
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reducing Tool Hallucination via Reliability Alignment
of hallucination instances.
Our contributions are follows:
Second, we introduce task-level reliability measurement framework that addresses the cascading effects of tool hallu- cinations on task outcomes. Since tool hallucinations often lead to hallucinated final outputs, we refine the traditional task success rate by incorporating hallucination-aww rections. Specifically, we propose the Reliable Pass Rate (RePR), which discounts task outcomes that are influenced by hallucinated tool calls. Furthermore, capture the trade- offs between task success and resource efficiency, we in- troduce the Benefit-cost Utility metric. This metric not only rewards successful task completion but also penalizes hallucinations and excessive tool-calling costs.
We systematically define and categorize tool hallucina- tions into two major types with four subtypes, propos- ing an automated evaluation framework.
cor-
We introduce the RePR metric for hallucination task evaluation, develop Benefit-cost Utility for cost- sensitive analysis, and curate BelyToooBencc with spe- cialized test cases for rigorous reliability assessment.
We propose Relign, the first tool reliability alignment framework, supported by a data synthesis pipeline and preference optimization techniques to reduce tool hal- lucinations and improve overall reliability.
Third, we develop Rely Bohlech,,,,,, a specialized bench- mark derived from MableToollBence (Guo et al., 2024), to rigorously evaluate tool hallucinations. RelyToolBench in- cludes two additional types of subsets: one addressing tool selection hallucinations through tool-task mismatches, and another focusing on tool usage hallucinations through pa- rameter deficiencies. These subsets enable precise assess- ment of model behavior in tool invocation tasks, providing granular insights into tool hallucination and task reliability.
2. Tool Hallucination and Reliability Evaluation
2.1. Tool Hallucination
Task Execution
Tool Selection
Tool Using
User:
Type Hallucination
Format Hallucination
To mitigate tool hallucinations and enhance task reliability, we propose Relign Reliability alignment framework for tool calling. Tool invocation can be regarded as a sequential decision-making process in which LLMs execute specific actions. Unlike conventional reinforcement learning tasks, where all actions within the action space are generally valid, tool invocation involves the risk of issuing invalid or hallu- cinated tool calls. Prior works (Qin et al., 2023b; Yu al.,
I'm football anthunistt more Lionel Mesal's you provide me with infor- about Masiis clubs, managers, teammattes and reforeest I'm also curious about wny notxale trandfers he has made.
Assistat:: call get_iffffrr_ footpppaes
Assistant: trander_
X tool name
Fabricated
X Miisin paras
*..
Assistant: transter search (ssser""Mess"))
Assistan::
google_sranslltee
Inrelevant tool X
x Unexpected paras.
Tool J Tool
Name:
Parameters::
srander nearch
Timing Hallucination
Content Hallucination
2024) primarily focus improving the model's ability to make better tool selections from predefined set of decisive actions (i.e., directly invoking a tool). However, when the preconditions for tool usage are not met-such as when the user fails to provide sufficient parameters or when the selected tool is unsuitable for the given task-any action chosen from the decisive action space inevitably leads to an invalid tool call and results in hallucination. To address this issue, Relign introduces the concept of an indecisive action space within the tool-use environment, allowing the model to proactively learn when to defer tool invocation, engage in clarification dialogues with the user, or switch to an alternative tool. This expansion of the action space enables LLMs to make more informed decisions, reducing hallucinations while improving overall system robustness.
"narne {
"ypp''stting.
Assistant:: (name="Mees")
transfor_jearch
Assistant:
'recuired" True)
name Crisiano Ronaido")
Duplicated calling
X
Halludiated
parameter content
Calling
Callingi
Teol Calling
Teat
trarafer_eearch ramee"
Callng!
Tool Calling TC
Figure I. Different types of tool hallucination.
With the emergence of intelligent agent frameworks like AutoGPT and MetaGPT (Hong et al., 2024), LLMs now demonstrate enhanced capabilities in world interactions through tool use. However, when LLMs interface with external environments via these tools, hallucinations during tool execution can directly affect real-world systems.
Additionally, Relign incorporates optimization techniques like Supervised Fine-tuning (SFT: Zhang et al., 2023) and Direct Preference Optimization (DPO; Rafailov et al., 2024), along with a specialized data synthesis algorithm to gener- ate reliability-focused training data. Experiments on Rely- ToolBench demonstrate that Relign effectively reduces tool hallucinations and significantly improves model reliability.
We view tool calling as a eeeiiiion-making action. When the conditions for making decision are not met when the decision fails, it can result in unexpected hallucinations. We categorize the tool hallucination into two major types:
Tool selection hallucination occurs when the model either selects an inappropriate tool or calls a tool at the wrong time. This category can be further divided into two subtypes:
2

---

Reducing Tool Hallucination via Reliability Alignment
Tool-task relevance assessment. The evaluator infers a tool's intended function from its description and pa- rameters, verifying alignment with the user query. It also detects errors in tool invocation timing, such as redundant or premature calls.
Tool
No
Tool type hallucination
TC₁ valld?
No
Tool format hallucination
Yes
Parameter authenticity verification. The evaluator checks whether tool parameters (e.g., user IDs, city names) are derived from the user's input or fabricated. If parameter lacks a clear basis in the query, it is classified as hallucination.
Ti relevant task?
No
Tool type hallucination
Valid joon format?
Required perto present?
No unsxpected paras?
Paras types match?
Is TC; duplicated?
Tool timing hallucination
content
Tool content
ja true?
hallucination
We design structured prompts (Appendix B, C) to guide these evaluations. Human assessment shows that LLM- based evaluation achieves 92.7% accuracy (Appendix A), closely aligning with expert judgments.
T. avallable tool sot
Yes
Rule based checking
Tj callin of
Tool T with specitic Calling TC₁
Valid Calling
LLM based chocking
Figure 2. Evaluation process of tool hallucination.
To quantify hallucinations, we introduce the tool halluci- nation rate, measuring the proportion of hallucinated tool calls in a given task. For a task with Nooaal tool calls and Nhalluciaaton hallucinated calls, we define:
Tool type hallucination indicates that the model either invokes a tool that is unrelated to the task or fabricates a tool that does not exist in the available tool set.
Nusiiiittion Notal
if Nivtal > 0, otherwise.
H sample
0
Tool timing hallucination involves errors related the sequence of tool calling. Specifically, this form of hal- lucination occurs when the model calls the same tool repeatedly with identical inputs and outputs, indicating that the tool should not have been called again at that point in the task process.
The task-level hallucination rate is computed as the average sample-level hallucination rate across all task instances, providing an overall measure of tool-use reliability.
2.2. Reliability Evaluation
In this section, we assess the reliability of tool use LLMs by introducing two key metrics: Reliable Pass Rate (RePR) and Benefit-Cost Utility. Both metrics offer distinct in- sights into the model's ability to perform tasks efficiently and accurately while managing tool-related challenges such as hallucinations. The Reliable Pass Rate emphasizes the accuracy of task completion, focusing on the reduction of hallucinations, while the Utility metric incorporates both the success of the task and the associated costs, penalizing excessive tool usage and hallucinations.
Tool usage hallucination pertains to errors in how the model uses tool after selecting it. This category includes:
Tool format hallucination refers to errors in the struc- ture or format of the tool's usage, such as providing an invalid JSON format, using incorrect parameter names, omitting required parameters, or specifying parameters with values of the wrong data type.
Tool content hallucination represents cases where the specific content of the tool's 's parameters is fabricated by the model, meaning that the input provided to the tool was not based on the user's query but was instead invented by the model.
1.2.1. RELIABLE PASS RATE
The Reliable Pass Rate (RePR) measures the proportion of tasks successfully completed without hallucinated responses. RePR subtracts the hallucinated outcomes from the standard pass rate, offering a clearer view of a model's 's reliability in executing tasks without introducing errors. The formula for calculating RePR
We then determine tool hallucinations and their types fol- lowing the process in Figure 2. To ensure comprehensive evaluation, integrate
methods with an LLM evaluator (GPT-4o). Rule-based methods detect structural errors such as invalid ISON formats and missing parame- ters. However, assessing tool-task relevance and parameter authenticity requires deeper semantic understanding, which the L.M evaluator provides:
RePR = Pass Rate Task Hallucination Rate.
Here, Pass Rate refers to the proportion of tasks that are successfully completed (for detailed evaluation methods, please refer the ToolBench paper (Qin et al., 2023b)), and
3

---

Reducing Tool Hallucination via Reliability Alignment
Task Hallucination Rate quantifies the proportion of tasks affected by tool hallucinations. Importantly, tasks with original pass rate of O are not considered as having result hallucinations to avoid repeated punishment. Once tool hallucinations are identified, task hallucination is further assessed by using an LLM to verify whether the known hallucinated tool calls correlate with the final task result (for the prompt, refer to Appendix D). We also present comparison between the original pass rate and our RePR metric in Figure 3. We observe that, across different datasets, there are certain result hallucinations that are not directly identifiable by various evaluation models. As a result, our RePR metric is significantly lower than the pass rate.
ToolBench (Guo al., 2024). Rely TelyoooBBench extends the original test set by synthesizing two additional categories of subsets designed to simulate challenging conditions for the model. Both of the two categories are critical for evaluating the tool reliability of LLMS in real-world applications. The dataset statistics are provided in the Table Specifically, we generate two additional subsets:
Missing Parameter Subset: This modification ob- scures certain parameters related to tool callings within the task. Specifically, this data is generated using LLMs to hide parameters within the tasks. This aims simulate scenarios where crucial tool-related infor- mation is hidden, allowing to evaluate the model's ability to function effectively under incomplete infor- mation. The prompts can be found in the Appendix E.
100.0
100,0
100
91.8
40
86.4
82.6
1alk rallcinaton Rare (N)
67.3
55,3
69.8
We replace the tools spec- ified in the task with irrelevant mismatched tools. This modification tests the model's ability to handle hal- lucinations arising from errors in tool selection, where the model may incorrectly identify or call the wrong tool for a given task.
Unmatched Tools Subset:
PR
Relablle FR
Task Holludatiinn Race
GPT-4-Turbo 4P1-4omm
CPT-40
Cat-O
CPC-WP
C.c.UT
Figure 3. Metric comparison between reliable and original pass rate. O, MP, and UT represent the original, missing parameter, and unmatched tools subsets, respectively.
Table I. Data Statistics for Rely RelyToolBench.
Subset
Category
Solvable
Total
II-Inst.
12-Cat.
13-Inst.
2.2.2. BENEFIT-COS UTILITY
Original
163
124
61
348
The Benefit- Utility takes a more holistic approach by evaluating both the task outcome and tool usage efficiency. The utility score is designed reflect the overall quality of task execution, considering not only task success but also the penalties incurred from hallucinations and excessive tool calls. The formula for calculating the utility score is:
Miss Parameter
X
99
103
60
262
Unmatched Tools
X
163
124
61
348
Total
425
351
182
958
3. Reliability Alignment
Utility Reask
Poool
Pballucination
In this section, we introduce the aligoment framework for tool reliability, Relign. As illustrated in Figure 1, Relign consists of three components: the tool-level alignment ob- jectives (3.1), the process for synthesizing tool preference data(3.2), and the overall training pipeline (3.2). We iden- tify the primary cause of tool hallucinations the model's lack of proper modeling of the decision conditions when making tool-related decisions. Specifically, if an LLM calls a tool under conditions where the decision criteria are not met-such as when the tool's parameters are un- known or when the tool is mismatched with the task-the the occurrence of tool hallucinations significantly increases. To address this, introduce the concept of indecisive action space, where alternative actions such as Change 1s and TalkToUser when tool invocation conditions are un- met. Furthermore, we enhance the model's s awareness of boundary conditions in tool calling by training it with syn- thesized alignment data.
where Rusk is the reward based on the task outcome (20 for success, 0 for failure), Proot is the penalty for exces- sive tool usage (calculated as min # (total tool calls) # (necessary tool calls), 10)), and Phallucination is the penalty for hallucinations in the task (-10 for hallucination). Unlike RePR, which focuses solely on the task completion rate after excluding hallucinations, the Utility metric incor- porates both the quality of the result and the efficiency of tool usage, making it a more comprehensive measure of tool reliability. It explicitly penalizes hallucinations and excessive tool calls, thus balancing the success rate with the cost of tool inefficiency and errors.
2.3. Construction of Rely ToohBench
To better evaluate tool hallucination and task reliability, we introduce Rely RelyTooBBench, which builds upon Stable-
4

---

Reducing Tool Hallucination via Reliability Alignment
Correct Selection
Correct
Correct Parameters
I'm a footboll enchusiast and I want know more about Lionel Mesols career. Can you proside me with information about Mesil's cubb. managers, refereest
Tool Call
trandder search/namee '
Wrong Tool Selection or
Tool Calling History
rinish
Hallucinated
B
translactsonnngg- Yoorbal!
Fabricated Parameters
canderrarkk ser
(aad Mest for sn has D4 1981 Soart Argrora Exposhsord
SFT
ranshh
Tool Call
bor and teral boon la
Unmatched
Unckaar Task?
wrademsuuk dead
Toarfo cookittte Alts Loiled eetrycnngettc Wieer Hdeetfeed SUL Tord reef
Tool?
thellapo
Indecisive
Change
Ask
Sampling Tool Call
Preference Pair www.ersAERRRR....
Action Space
Finish
Tool
Tool
Hallucination
Tool Call
Finish
DPO
Detector
Helpfulness
Finish
B
Tall
Call
B
Finissh
Reliability
Finish
>>
B
Tall
Finish
Finish
B
Alignment Objective
Preference Data Generation
Training Pipeline
Figure 4. The system illustration of Relign.
3.1. Alignment Goal
the number of required tool callings is typically fixed. Mod- els that invoke tools excessively tend to introduce more hallucinations. Thus, minimizing tool hallucination not only improves task reliability but also reduces the overall number of tool callings, leading to more efficient tool usage.
The alignment goal for reliable tool calling in LLMS focuses on ensuring accurate and effective interactions with external tools throughout the task completion process. From task perspective, the primary objective is to maximize successful tool calling while minimizing hallucinations. This can be formalized as a preference hierarchy: formally, let T rep- resent a tool calling trajectory, and (T) the hallucination rate within that trajectory. For a given task, we define the alignment preference as:
3.2. Alignment Methods
The overall data synthesis and alignment training process is illustrated in the middle and right parts of Figure 4. First, we construct reliable supervision data for Supervised Fine- Tuning (SFT) to train the model on the appropriate use of indecisive actions. After the initial SFT training, we sample and synthesize reliable preference data based on the model, which is then used to further train the model through the Direct Preference Optimization (DPO) approach. This two- step process ensures that the model not only learns to handle indecisive actions effectively but also aligns with the desired task-level preferences in the long run.
T success
T Fraluure
ballucination:
where T successs represents a successful trajectory, Tiature rep- resents a trajectory resulting in task failure or abandonment, and Thallucination represents trajectory that generates a hal- lucinated result.
In our experiments, we observe that when model exhibits a high rate of tool hallucinations, it tends to favor producing hallucinated outputs rather than acknowledging failure. This behavior poses a significant risk, as hallucinated results may mislead users into believing the task was successfully completed, thus undermining trust in the system.
1.2.1. RELIABLE SFT
We observe that among the four types of hallucinations, tool type and content hallucinations occur more frequently. Therefore, the main objective of our Supervised Fine- Fine-Tuning (SFT) is to train the model to address these hallucinations by learning to invoke "ChangeToow Tools" for tool type halluci- nations and "Taak for tool content hallucinations. Similar to the data synthesis in RelyToolBench, we select specific samples from ToolBench's 's training data, modifying them by replacing tool sets or removing tool parameters to simulate these hallucinations. The model is then trained to output the appropriate actions based on these modifications. To ensure the generalizability of the experiment and results, the training data we selected involves only a single tool.
Therefore, building upon the task-level alignment prefer- ence, we define the alignment preference at step level for tool interaction tasks. Specifically, a non-hallucinated tool call is preferred over indecisive actions and hallucinated tool calls. This preference hierarchy can be formalized as:
Agonct Aisdecisive
Analicuinated:
where Acoreats Annecissiee Anallcuinated represent correct tool calling actions, indecisive actions and hallucinated tool call- ing actions, respectively. We also find that for many tasks,
5

---

Reducing Tool Hallucination via Reliability Alignment
3.2.2. RELIABLE DPO
tool-calling tasks, we fine-tune them using 5,000 instances randomly sampled from ToolBench as the baseline. For all the experiments, we set the training batch size to 32, and the max sequence length to 8192. We utilize the DEEPSPEED- CHAT framework for efficient model training. In all meth- ods, the learning rate is set to le-5 for SFT and le-5 for DPO to ensure consistency, with all training conducted over two epochs. We did not perform a grid search but used stan- dard hyperparameters for all experiments to ensure stable fine-tuning and consistent results.
After obtaining model capable of outputting indecisive actions through SFT, we further optimize it using Direct Preference Optimization (DPO). Direct Preference Opti- mization (DPO) focuses on constructing preference data to fine-tune the model's behavior. The goal of DPO is to guide the model in selecting optimal tool calling traces based on task success and hallucination types. Specifically, for each tool calling trajectory, at any given step, we sample multiple responses and use hallucination evaluator categorize these responses into three types: hallucinated tool calling action Apallucinated: non- oonhhallueinatd tool calling action A Acoreect, and indecisive actions Aindecisive. Based on these classifications, we generate three types of preference pairs: (Acorvect > Asndecisiue), (Amdeessive > Ahallucinated), (Acorrect > Aballucinated). For each type of preference pair, we randomly select one from the sampled responses to construct the DPO training data.
To construct the SFT training data, we randomly split 10,000 samples into three subsets: 4,000 remain unchanged, 3,000 are modified by replacing tool choices, and 3,000 are used to construct missing parameter cases. The model is directly fine-tuned on these curated tool-call traces. For DPO data construction, we start with the same 10,000 tasks and al- low multi-round interactions with the environment. In each round, we sample ten tool-use trajectories at tempera- ture of 0.7 and use GPT-40 as hallucination evaluator to construct preference pairs. To ensure data diversity, we retain at most one trajectory per step for each preference combination. While the SFT dataset size remains fixed, DPO training steps vary depending on the number of valid preference pairs. Not all interaction rounds yield usable DPO samples, resulting in total of 10,000 to 20,000 pref- erence pairs in practice. For instance, Relign training yields 17,000, 19,000, and 14,000 pairs for Toolllama, Llama3. I, and Qwen2.5, respectively. We train with a batch size of 32 for two epochs, with the final number of training steps determined by the number of collected DPO pairs.
The DPO framework ensures that the model learns to rank trajectories according to these preferences. By optimizing over such data, the model becomes capable of choosing the most reliable tool calling sequence or recognizing when to abandon a task or seek further clarification from the user.
4. Experiments
4.1. Experimental Setup
Datasets. In our experiments, we primarily utilized three datasets: ToolBench, StableToolBench and RelyTool- Bench. ToolBench (Qin et al., 2023b) was constructed by scraping various APIs from RapidAPI and generating corresponding tasks and executions, comprising a total of 120,000 data samples. StableToolBench (Guo et al., 2024) is selected subset of solvable samples from ToolBench, and it also proposed stable environment for evaluation. We randomly select 10,000 samples from Toolbench for constructing reliability alignment data. We further construct RelyToolBench based on StableTooBBenchh for evalution.
Additionally, evaluations regarding tool hallucinations and task success were performed using the GPT-40 model, and other evaluation details follow the setup in StableTool-- Bench (Qin et al., 2023b). The prompt for evaluating Sta- bleToolBench across all models is provided in Appendix F. For computing both benefit-cost utility, the necessary tool calling number is set to for solvable tasks (original sub- sets) and 0 for unsolvable tasks ( missing parameter and unmatched tools subsets). We conduct all experiments us- ing Nvidia A800 GPUs
Baselines. (1) RLHF & StepTool. The two baselines are proposed in Steptool (Yu et al., 2024), which augments tool learning by introducing step-level rewards. RLHF is simplified reinforcement learning version of Steptool with only trajectory-level reward. (2) SFT & DPO. We also implemented a joint training approach that combines SFT and DPO losses on the same base model as baseline. The combination ratio is set to 0.5.
4.2. Main Results
As shown in Table 2, we assessed the performance of our model on three subsets of StableToolbench. Our primary fo- cus was on measuring the tool hallucination rate and the task pass rate. Our results indicate that the implementation of the Reliability Alignment framework, which includes both SFT and DPO, significantly reduces the tool hallucination rate of the baseline model. Moreover, observed a decrease in the average number of tool callings required for each task. Although previous tool-learning methods, such as StepTool, effectively improve task success rates, they provide no re-
Training and inference details. We use LLAMA-3.1 I- and TOOLL- LAMA 7B (Qin et al., 2023b) as our experimental models. Since L.LMMA and Qwen models struggle with complex
SB-INSTRUCT, SWEN-2.5-7B-INSTER
6

---

Reducing Tool Hallucination via Reliability Alignment
Table Performance comparison across different methods. RePR 1: reliable pass rate. Tool Hallu L: tool hallucination rate. Tool Num J: average tool calling number. Utlliyy Benifit-cost utility..
Solvable Unsolvable
Instraction
Calegory Solvable Unsolvable
Solvable Uasobvable
Instruction
Overall
Method
RePR
Tool Hallo
Tool Num
Udilyy
RePRI
Tool Halla
Tool Num
Utlity
RePR
Took Hallo*
Tool Num
Uuiäty
RcPR
Tool Hallu*
Tool Num
Utility
Clused-avuree LLMs
3, 5-turbo
66.9
$7.0
8.0
57.6
64.4
7.3
4,9
50.7
68.9
11.0
2.6
58.4
63.4
5.2
GFT 4o-mini
12.2
GPT 4o
80.1
5.9
1.2
14.8
75.1
9.6
1.3
13.3
69.1
9.5
1.8
11.9
74.8
8.3
1.4
13.4
Tooliloma (7B) (Qin et al., 2023b)
Raseline
64.2
55.1
8.8
64.4
56.4
58.9
$7.9
4.1
6.9
62.5
56.5
8.0
RLIPP
65.1
54.2
3.2
9.0
64.6
55.7
3.5
8,7
58.5
57.5
4.1
6.7
62.7
55.8
3.6
8.1
Steptool
53.7
59.0
SFT
67.8
24.8
2.7
10.8
67.6
52.7
2.6
10.3
65.7
29.6
2.5
9.99
67.0
29.0
2.4
10.3
DP0
66.1
41.5
1.2
11.2
66.9
35.7
1.2
11.4
63.5
46.0
1.3
10.3
65.5
41.1
11.0
8PT&DPO
25.8
11.8
69.0
10.9
66.6
32.1
2.5
9.9
69.0
10.9
Belign
68.2
17.1
0.9
12.2
68.1
21.6
0.9
12.2
71.0
16.3
1.0
13.0
69.1
18.3
0.9
12.4
Lluma3.1 (88) (Dubey et al., 2024)
Baseline
68.5
51.2
10.5
66.1
2.3
49.6
2.4
8.3
65.3
9.4
RLIP
68.5
51.7
2.4
10.2
66.3
50.8
2.6
9.4
61.0
49.1
2.7
7.9
65.3
50.5
2.6
9.2
Steptool
50.8
11.2
69.7
2.3
10.5
55.8
2.5
65.7
52.9
9.5
SFT
71.6
17.3
1,4
12.1
71.3
24.2
1.7
11.8
66.0
28.6
2.1
9.7
69.6
23.4
1.7
11.2
DPO
69.6
43.3
2.0
11.8
66.2
44.9
2.2
11.1
62.2
50.2
2.7
9.0
66.3
46.
2.3
10.6
82TsDe0
14.0
13.5
72.5
11.5
20.1
73.7
12.0
3elign
80.5
8.7
1.3
14.5
75.8
22.1
1.6
12,5
75.3
13.1
1.8
12.4
77.2
14.6
1.5
13.2
@wen2.5 (7B) (Yang al., 2024)
Daseline
46.1
66.2
2.4
$1.9
65.1
SLRP
67.8
48.7
2.3
10.7
65.4
47.3
2.4
9.8
62,8
49.4
2.6
9.6
65.3
48.5
2.5
10.0
Steptool
46.0
S2T
69.9
18.7
1.5
11.6
69.9
25.2
1.7
11.4
62.7
33.1
2.0
8.8
67.5
25.7
1.7
10.6
D20
69.9
45.0
2,3
10.6
68.7
45.5
2.4
10.0
65.2
45.9
2.9
8.7
67.9
45.4
2.5
9.8
62T&De0
10.2
Belign
77.0
12.0
1.3
13.3
69.0
22.6
1.7
10.6
72,9
18.3
2.0
11.6
73,0
17.6
1.7
11.9
90.8
$3.4
67.6
Avg Tool Calling Num
2.5
14,8
Reliable Pass Rate
78.
Tool Hallucination
56.8
12.9
2.0
60
$4.3
15
1.4
Utiliyy
8.7
1.1
40
34-1
1.0
20
5
4.1
8.6
0.5
LLaMAAS1
(1aMAS 1+
MP
UT
MP
UT
MP
UT
MP
UT
Figure 5. Comparison of performance metries between the baseline and Relign across three subsets: Original (O), Missing Parameter (MP), and Unmatched Tools (UT).
duction tool hallucination rates and not decrease the number tool calls. contrast, our approach not only en- hances task success rates but also significantly reduces hal- lucinations. Furthermore, our Relign-trainet trained model based on I.LMMA 3.1 outperforms many closed-soures models of similar scale, such as GPT-3.5-Tune S-Tubbo and GPT-4o Mini, approaching the performance of GPT-4o.
4.3. Tool Hallucination Analysis
Figure 6 provides detailed distribution different types of tool hallucinations. With Relign framework, halluci- nation types show significant reductions, with the greatest improvement observed in the Content Hallucination cate- gory, particularly in the Unmatched Tools subset. However, Timing Hallucination demonstrates a relatively smaller im- provement, likely due to the inherent difficulty in learning the sequential dependencies required to avoid repetitive or redundant tool calls. This underscores the robustness of our method while identifying areas for further optimization.
Figure further demonstrates the effectiveness of our Relign framework in reducing tool hallucination and improving sys- tem efficiency across three categories of subsets: Original (O), Missing Parameter (MP), and Unmatched Tools (UT). Relign achieves remarkable and consistent improvement in all the metrics over all subsets.

---

Reducing Tool Hallucination via Reliability Alignment
Type
Timing
Contenc
Tedtf leteraao
Aug Toul Cilling Man
Orgiiall Subsets
gpe-3
gpt 4o-iin
LLaWA3.11
Mdlulle Pass Rat
65
Mum
Tool Haluddattooo Racs
50
700
17.5
Rodign
60
1.0
15.0
Missing Parameter Subsets
12.5
gre33-eurbo
46
100
1LANAS1
Rod
3k
Unmatched Tools Subsers
Data Size
Dora Sic
gpe-60
min
Figure 7. Impact of Training Data Size Performance and Hallucination Metrics. Experiments are conducted on LLaMA3.1.
LLaMAS1
Rdign
Rddibll
Figure 6. Distribution of tool hallucination types. The results highlight the effectiveness of Relign in reducing hallucination across all subsets.
70
15
90
24
Tool Halluciaaion Base
25
Bebable Plos Ptace
AvE Tod Calling
Task Hall Holludddooo
70
16
50
4.4. Comparison with Existing Metrics
Our RePR metric a refined version of Pass Rate, as we found that some tasks in the original pass rate metric exhib- ited result hallucinations. Therefore, we believe the pass rate has certain inaccuracies and did not include its results in our paper. As shown in the Table 3, we have supple- mented some of the original Pass Rate results. The results indicate that RePR is lower than Pass Rate, and the gap is more pronounced in base models without Relign alignment. This suggests that unaligned models tend to produce more tool hallucinations, which in turn mislead the final results. Moreover, whether using Pass Rate or RePR, our Relign framework consistently improves task success rates.
30
1.56
38
76
145
1.58
38
75
145
Model Size
Model Size
Figure 8. Impact of Model Size on Performance and Halluci- nation Metrics. Experiments are conducted Qwen2.5 with 5k data.
that reducing tool hallucinations helps models learn better tool-use strategies (AST represents tool-calling accuracy).
HugsingFace
TorchHub
Tensorhub
Method
Hallu.(L) AST(1) Hallu ASI(f) Hallu.()) AST(1)
LLaMA3.1 + BM25
9.7
14.3
11.7
47.8
10.5
40.3
Instruction
12 Category
Instruction
Overall
Relign BM25
5.5
Method
LLaMA3.1 Oracle
9.3
10.4
86.1
7.8
89.6
PR
RePR
PR
RePR
PR
RePR
PR
RePR
Relign Oracle
6.5
89.5
7.7
88.9
3,2
91.3
ToMA
76.8
64.2
77.0
64.4
72.3
58.9
75.4
62.5
-Relign
77.1
68.2
77.3
68.3
76.2
71.0
76.9
69.1
Table 4. OOD generalization experiments on APIBench.
LLaMA3.1
83.0
68.5
84.8
66.1
80.1
61.3
82.6
65.3
Relign
87.0
80.5
89.2
75.8
86.9
75.4
87.7
77.2
Qwen2.5
86.3
71.5
84.3
66.2
80.6
57.7
83.7
65.1
4.6. Discussion
Relign 87.6
77.0
87.5
69.0
85.5
72.9
86.9
73.0
Does More Data Reduce Hallucination? Increasing the amount of training data from Ik to 5k consistently improves model performance, as reflected by the rising Reliable Pass Rate in Figure 7. However, this increase in data size does not always lead to a reduction in hallucination rates. Both Tool and Task Hallucination rates remain stable or even slightly increase with more data. This could be attributed to overfitting on ToolBench data, which consists of well- structured tool interaction trajectories and lacks samples that handle edge cases (e.g., failure scenarios where a task cannot be completed). Thus, the model learns correct tool calls but struggles with failure scenarios, defaulting to ex- cessive hallucinated tool calls instead of invoking indecisive actions actions-preecseey the issue Relign aims to address.
Table 3. Comparison of models on original Pass Rate and RePR.
4.5. duttof-distribuutin Generalization to APIBench
As shown in the table below, we also evaluate our method on the out-of-domain test set APIBench. We follow the evaluation setup described in (Qin et al., 2023b) (for more details, please refer to Qin et al., 2023b). Rather than training APIBench, we treat each API in the prompt as a function call to assess how well our trained model generalizes to OOD datasets. Experimental results with two types of tool retrieval indicate that Relign improves model performance on other tool-use tasks as well, suggesting
8

---

Reducing Tool Hallucination via Reliability Alignment
Does a Larger Model Reduce Hallucination? Scaling up model size from 1.5B to 14B, while keeping the training data fixed at 5k, leads to significant improvements in both performance and hallucination reduction, as seen in Figure 8. Larger models exhibit a better ability to capture complex task dependencies and context, resulting in a noticeable decline in both tool hallucinations and excessive tool calls, leading to more reliable and efficient tool use. Moreover, the Reliable Pass Rate increases with model size, highlighting the enhanced robustness and generalization capabilities of larger models. These results suggest that scaling model parameters is key strategy for improving the reliability and accuracy of tool usage in complex environments.
nations and employs LLMs evaluators for hallucination. Furthermore, we integrate the concept of reliability and pro- pose the reliability alignment framework to mitigate the adverse effects of tool ballucinations.
6. Conclusion
In this work, we systematically investigate the challenge of tool hallucinations in LLMs and propose a comprehen- sive framework to evaluate and enhance tool reliability. We categorize tool hallucinations into selection and usage er- rors, design an automated evaluation pipeline, and introduce new reliability-awar aware metrics. To mitigate hallucinations and improve reliability, we propose Relign, a reliability- focused framework that expands the model's decision space beyond direct tool invocation. By introducing indecisive actions- such switching tools engaging in user clari- Reltio allows the model to make more informed decisions and avoid hallucinations. This work highlights the importance of reliability LLMs and provides a foundation for future research on en- hancing decision-making in real- weal-world tool-use scenarios.
5. Related Work
5.1. Tool Learning
Recent advancements in tool learning have enabled LLMs to effectively integrate external tools, enhancing real- time knowledge retrieval, multimodal functionalities, and domain-specific expertise (Yang et al., 2023; Gupta & Kem- bhavi, 2023; Jin et al., 2024). Methods range from lever- aging in-context learning for tool descriptions and demon- strations (Hsich et al., 2023) to explicit training on tool- enriched datasets (Patil et al., 2023; Tang et al., 2023; Qin et al., 2023b). Evaluation of tool-augmented LLMs pri- marily focuses on metrics like tool call accuracy and task success (Zhuang et al., 2023; Guo et al., 2024), often em- phasizing execution over other factors. However, limited
training in nol-auummnted
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
research has addressed the critical issue of tool hallucina- tions (Patil et al., 2023; Chen et al., 2023b) and tool effi ciency (Xu et al., 2025), which can undermine the reliability and trustworthiness of tool usage in real-world applications.
A Acknowledgments
This work is funded by the China NSFC Projects 1010006 92370206, U23B2057) and Shanghai Municipal Science and Technology Major Project (202ISHZDZX0102)
5.2. Mitigating Halucinations
LLMs are prone generating hallucinations errors where content conflicts with user inputs, prior generated informa- tion, or world knowledge (Guerreiro et al., 2023; Mündler et al., 2023; Min et al., 2023). Strategies to mitigate these issues include improving pre-training data quality (Penedo et al., 2023; Touvron et al., 2023), fine-tuning with curated datasets (Chen et al., 2023a; Zhou et al., 2023), using exter- nal knowledge sources ground responses (Gao et al., 2023; Jain et al., 2024), designing better decoding strategies (Shi et al., 2023) and estimating uncertainty (Azaria & Mitchell, 2023; Xiong et al., 2023; Zhao et al., 2023; Varshney et al., 2023). Xu et al., 2024; Zheng et al., 2025 introduce the con- cept of reliability, which emphasizes maximizing helpful- ness while simultaneously minimizing hallucinations. Some studies (Patil et al., 2023; Chen et al., 2023b; Zhang et al., 2024) introduce the notion of tool hallucination with limited types and assessments of tool hallucination. In contrast, our work systematically defines different types of tool halluci-
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint urXiv:2303.08774 2023.
Azaria, A. and Mitchell, T. The internal state of 11m knows when its lying. arXiv preprint arX1v.2304.1333, 2023.
Cai, T., Wang, X., Ma, T., Chen, X., and Zhou, D. Large Language Models as Tool Makers. In Proc. The Twelfth International Conference on Learning Representations (ICLR 2024), 2024. URL https:/ net/ pet//oorumidd=gV333B htt.t:://o//eeeiiiiiii /pppr/vien....e.eee
Chen, I.,, Li, S., Yan, J., Wang. H., Gunaratna, K., Yaday,

---

Reducing Tool Hallucination via Reliability Alignment
V., Tang, Z.,, Srinivasan, V., Zhou, T., Huang, H., al. Alpagasus: Training better alpaca with fewer data, arXiv preprint ar arXiv:2307.08701, 2023a.
Jain, N., Kwiatkowski, R., Ray, B., Ramanathan, M. K., and Kumar, V. On mitigating code IIm hallucinations with api documentation, arXiv preprint DXXiv:2407009722 2024.
Chen, Z., Du, W., Zhang, W., Liu, K., Liu, J.. Zheng, M., Zhuo, J., Zhang, S., Lin, D., Chen, K., et al. T-Eval: Evaluating the Tool Utilization Capability Step by Step. ArXiv preprint, abs/2312. 1400333, 2023b. URL https: //arxxia //axiiv... wrg.o/////.o.. / 2212. 14033.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of halluci- nation in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Jin, Q. Yang. Y., Chen, Q. and Lu, Z. GeneGPT: aug- menting large language models with domain tools for improved access biomedical information. Bioinfor- matics, 40(2):btae075, 2024. ISSN 1367-4811, doi: 10. 10.109//ooinffrmmtticccbooo//// URL https: //ttts:::/oi.. cs cs/btae075.
Dubey, A., Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.22 20l.:2407.2/783, 2024.
Gao, Y., Xiong, Y., Gao, Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmentte generation for large language models: A survey. arXiv preprint arXiv:2312. 109997 2023.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the Seventeenth International Conference on Machine Learning (ICMI. 2000), Stanford University, Stanford, CA, USA, June 29 July 2, 2000, pp. 1207-1216. Morgan Kaufmann, 2000.
Gemini Team. Gemini: A Family of Highly Capable Multi- modal Models, 2023.
Guerreiro, N. M., Alves, D. M., Waldendorf, J., Haddow, B., Birch, A., Colombo, P., and Martins, A. F. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics, 11:1500- 1517, 2023.
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-L, Koh, W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. Factscore: Fine- tine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305. 14251, 2023.
Guo, Z., Cheng, S., Wang, H.. Liang, S., Qin, Y., Li, P., Liu, 7.,, Sun, M., and Liu, Y. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.
Mündler, N., He, J., Jenko, S., and Vechev M. Self- contradictory hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint arXiv:2305. 15852, 2023.
Gupta, T. and Kembhavi, A. Visual programming: Compo- sitional visual reasoning without training. In In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), pp. 14953-14960 2023.
Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large Language Model Connected with Massive APIs. ArXiv preprint, aabb/2305.15334 2023. URL https: / //arxiv. www.ii..rg//bb/////e 2tt.///5555..133....
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cap- pelli, A., Alobeidli, H., Pannier, B., Almazrouei, E.,, and Launay, J. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. ArXiv preprint, abs/2306.01 011.10000000111...000111111 2023. URL https: /tt////avvv//////////////////ava www.riii.....aaaeeee 2w0..bb///33..111
Hao, S., Liu, T., Wang, Z., and Hu, Z. ToolkenGPT: Aug- menting Frozen Language Models with Massive Tools via Tool Embeddings. ArXiv preprint, abs/2305.11554 2023. URL https: /ttp:p:://rxxiee htt//rxvvvv// org/ abs 20........551
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y.,
Wang, J., Zhang. C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, I., Ran, C., Xiao, L.,, Wu, C., and Schmidhuber, MetaGPT: Meta programming for multi-agent collabo- rative framework. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: !/openreviews www.nrevieew... net. forum? http:/mmmmmdmtmmmmtteemmm
Qin, Y., Cai, Z., Jin, D., Yan, L., Liang. S., Zhu, K., Lin, Y., Han, X., Ding, N., Wang. H., et al. WebCPM: Interactive Web Search for Chinese Long-form Question Answering. ArXiv preprint, abs/2305, abs/22305.0684, 2023a. URL https: / farxiv.orge org. abs 220522355000 230./
Hsieh, C.-Y,, Chen, S.-A., Li, C.-L.,, Fujii, Y., Ratner, A., Lee, C.-Y., Krishna, R., and Plister, T. Tool documen- tation enables zero-shot tool-usage with large language models. ArXiv preprint, abs/2308.0065 2023. URL https: /ttas///axvvv..// www.rviiwgggaaaaaeeeeoe 2albs///30.0000
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, I.., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao. S., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., and Sun, M. TooILLM:
Facilitating Large Language Models to Master APIs, 2023b.
16000+
10

---

Reducing Tool Hallucination via Reliability Alignment
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., Fung, Y. R., Su, Y., Wang, H., Qian, C., Tian, R., Zhu, K., Liang, S., Shen, X., Xu, B., Zhang, Z., Ye. Y., Li, B., Tang, Z., Yi, J., Zhu, Y., Dai, 7.,, Yan, I.., Cong, X., Lu, Y., Zhao, W., Huang, Y., Yan, J., Han, X., Sun, X., Li, D., Phang, J., Yang, C., Wu, T., Ji, H., Liu, Z., and Sun, M. Tool learning with foundation models, 2024. URL https: //arxiv. org/ abs/ / 23344 08354.
Yang, L.,, Chen, H., Li, Z., Ding, X., and Wu, X. ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling. ArXiv preprint, abs/2306. 1489, 2023. URL https: / //arxiv. wrgpiiiorrg/aaae abs 2306. 11489.
Yao, S., Chen, H., Yang. J., and Narasimhan, K. Web- Shop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Koyejo, S., Mo- hamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), In Proceedings of the Advances in Neural Infor- mation Processing Systems (NeurIPS 2022), volume 35, pp.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D.D Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Curran Associates, Inc., 2022.
Yu, Y. Wang, Z., Ma, W., Guo, Z., Zhan, J., Wang, S., Wu, C., Guo, Z., and Zhang, M. Steptool: A uep-grained reinforcement learning framework for tool learning in llms. arXiv preprint arXiv:2410.07745, 2024.
Schick, T., Dwivedi-Yuu J., Dessi, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Tool- former: Language Models Can Teach Themselves to Use Tools. ArXiv preprint, abs/2302.04761, 2023. URL https:/ / arxiv..oo orgg wwwp//s/////oooooooooooooo....o/ 2302 04761.
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308. 2023.
Shi, W., Han, X., Lewis, M. Tsvetkov, Y., Zettlemoyer, L., and Yih, S. W.-t. Trusting your evidence: Halluci- nate less with context-aware decoding. arXiv preprint arXiv:2305. 14739, 2023.
Zhang, Y., Chen, J. Wang. J., Liu, Y., Yang, C., Shi, C.. Zhu, X., Lin, Z., Wan, H., Yang, Y., et al. Toolbehon- est: A multi-level hallucination diagnostic benchmark for tool-augmented large language models. arXiv preprint arXiv:2406.2003 2024.
Tang, Q. Deng, Z., Lin, H., Han, X., Liang, Q., and Sun, L. ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases, 2023.
Zhao, R., Li, X., Joty, S., Qin, C., and Bing, L. Verify- enhanced chain-of- of-th-of-thought frame- work. arXiv preprint arXiv:2305. 03268, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.0922. 2023.
and-edit: A
Zheng, H., Xu, H., Liu, Y., Chen, L., Fung, P., and Yu, K. Enhancing IIm reliability via explicit knowledge bound- ary modeling. arXiv preprint arXiv:2503.0223:: 2025.
Varshney, N., Yao, W., Zhang, H., Chen, J., and Yu, D. A stitch in time saves nine: Detecting and mitigating hallu- cinations of lims by validating cowfoidieeece generation. arXiv preprint ar
Zhou, C.,, Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.1 /206, 2023.
2023.
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. Can Iims express their uncertainty? an empirical evaluation of confidence elicitation in llss. arXiv preprint arXiv:2306. 1 13063, 2023.
Zhuang, Y., Yu, Y., Wang, K., Sun, H., and Zhang, C. ToolQA: A Dataset for LLM Question Answering with External Tools. ArXiv preprint, 0tt/2300613300 2023. URL https::///// arxiv. www.i.irgiaabl.... /ttp:/bbb///3..33333333
Xu, H., Zhu, Z., Ma, D., Zhang, S., Fan, S., Chen, L.., and Yu, K. Rejection improves reliability: Training Ilms to refuse unknown questions using rl from knowledge feedback. arXiv preprint arXiv: 2403 2024.
Xu, H., Wang, Z., Zhu, Z., Pan, L., Chen, X., Chen, L., and Yu, K. Alignment for efficient tool calling of large lan- guage models. arXiv preprini arXiv:2503.06700 2025.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1 15, 2024.

---

Reducing Tool Hallucination via Reliability Alignment
A. Human Evaluation for Hallucination Detection
To validate the reliability our LLM-based evaluation process, we conducted human evaluation focused on hallucination detection. Specifically, we evaluated outputs generated by GPT-4, GPT-3.5, and Too3LLaMA3.1 using the criteria defined in Figure 2. The evaluation targeted three categories: no hallucination, parameter value hallucination, and tool relevance hallucination. For each category, annotators assessed the correctness of the ballucination classification in these cases, with the results indicating 46 correct annotations out of 50 for no hallucination, 45 out of 50 for parameter value hallucination, and 48 out of 50 for tool relevance hallucination. This resulted in an overall accuracy of 92.7%.
randomly sampled 50 cases, resulting in total of 150 evaluation cases. Human
These results highlight the high reliability of our evaluation framework distinguishing hallucinated tool calls and determining their types. The findings further support the alignment between our LLM evaluator and human judgment in this intricate domain.
B. Prompt for Checking Tool Relevance
The prompt used to check tool relevance hallucination is shown in Table 5.
Table 5. Prompt used to check tool relevance hallucination.
Check Tool Relevance Prompt
Query: {query]
Tool Description:
(tool_description)
Tool Parameter:
)arameerr))
Given the above query, along with the description and parameter information of a certain tool, you need to infer the tool's purpose and determine whether might be relevant to completing a specific task within the query. You need to provide tool_ree relevance according to the following rules:
If the tool's purpose completely irrelevant to the query, return Irrelevant.
2. If the tool's purpose be used solve the user's query, return Relevant.
3. If the tool's purpose might be relevant to the query, or if the tool's description does not contain enough information to determine its use, return Unsure.
Now give your reasoning in content and tool.relevance in JSON format to check.tool.suitabbiitt
C. Prompt for Checking Content Hallucination
The prompt used check tool content hallucination is shown in Table
D. Prompt for Checking Hallucinated Answer
The prompt used to check the hallucinated answer is shown in Table 7.
E. Prompt for Data Generation
The prompt used to generate the missing parameter dataset is shown in Table
12

---

Reducing Tool Hallucination via Reliability Alignment
Table 6. Prompt used check tool content hallucination.
Check Content Hallucination Prompt
User History: {user_history
Tool Parameter:
{tral_parameter)
Specific Tool Calling: (tooL_calling)
Given the interaction history with the user and the introduction tool parameters, you need to determine whether the value of each parameter in a specific tool call hallucinated. If there are hallucinated parameters, then the entire tool call is deemed untruthful. You need to provide calling truthfulnesss based on following rules:
1. If the tool parameters require specific values from the user (such user product IDs, specific flight numbers, etc.), and the parameter value in the tool call does not appear in the user's interaction history, then return untruthful
2. If the tool parameters can be values inferred from the interaction history (such as query keywords, pages), and the current parameter value can be inferred from the user's interaction history, then return tru
If the tool parameters are explicitly mentioned the user's s interaction history, then return truthful.
Now give your reason content and calling trufulness JSON to evaluate calling truthfulness.
Table 7. Prompt used to check the relevance between provided answer and hallucinated tool calls.
Check Answer Hallucinated Prompt
Given the results of several tool calls and final answer, need to determine the relevance between the final answer and the tool call results based on the following rules, and provide the answer www.weerreeevaaceee
If the final answer or a part of the final answer is essentially the same as the result of any tool call, return Relevant. If the final answer or a part of the final answer can be inferred observed from any tool call result, return Relevant. If you cannot determine whether the final answer is related the tool call results, return Unsure.
4. If there is clear relevance between the final answer and the tool call results, return I Trrel evant..
Tool calls:
[tool_calls}
Final Answer:
(answer)
Now you are requested to give reason in content and of to cheok
F. Prompt for evaluating ToolBench
The prompt used evaluate StableTooll BoobBechoB tasks is shown in Table 9.
13

---

Reducing Tool Hallucination via Reliability Alignment
Table 8, Prompt used to generate the missing parameter dataset.
Data Generation Prompt
You are data annotator, and you need to help me rewrite the user's query according to specific requirements. I will provide you with a list of tools, and you need to hide any parts of the query that might correspond to tool invocation parameters. You can do this by either removing those parts or modifying their expression to obscure them, while ensuring that the query remains fluent and free of special characters, and keeping the other parts as unchanged as possible. If there are no tool invocation parameters to hide in the query, keep the query unchanged.
Tool List:
[tool_list}
Query: {query]
Table 9. Prompt used evaluate Stable ToolBench tasks and call tools.
Data Generation Prompt
You are AutoGPT, you can use many (funsciusc to do the following task. First will give you the task description, and your task start. At each step, you need to give your thought to analyze the status now and what to do next, with function call actually excute your step.
Your output should follow this format:
Thought:
Action:
Action Input:
After the call, you will get the call result, and you now in state.
Then you will analyze your status now, then decide what do next...
After many (Thought-call) pairs, you finally perform the task, then you can give your finial answer. Remember:
1.the state change is irreversible, you can't go back to one of the former state, if you want to restart the task, say "I give up and restart".
2.All the thought is short, at most sentence. You can do more then one trys, so your plan is continusly try some conditions, you can do one of the conditions per try.
Let's Begin!
Task description: You should use functions to help handle the real time user querys. Remember:
1. The tools provided to you are limited. You need to carefully discern the purpose of each tool. If you believe that the purpose wessh-givve.up.anne www.dddchaggetooolooo.
of tools cannot satisfy the user's query, please directly call the function
2.Some tool invocation parameters might not be present in the user's query. You need to determine if the tool invocation parameters are already included in the query. If they are not, you need use the function Finish->give wwwpgggvveeeeaeeeeeeeeee wwtpeeeuppppaaaaeeeeeeeeeee and_aanddtaae.. o_user to ask the user your questions.
3.ALWAYSS call Finish function at the end of the task. And the final answer should contain enough information to show to the user, If you can't 't handle the task, or you find that function calls always fail(the function is not valid now), use function finishh-i seiishh-iivvee >eiieeiviie........ weiiveeup_aade redstart
4.Do not use origin tool names, use only subfunctions' names.
You have access of the following tools: Jescriptoo))
Specifically, you have access to the following APIs:
(api }
14

---

Reducing Tool Hallucination via Reliability Alignment
G. Case Study
The case study of the baseline (Llama3.1) method and Relign shown Figure 9.
A Case of Correcting Hallucination Tool Use
Baseline
Relign
User Query: need download the translated docu- ment for the job in the project. Can you help with that?
User Query: noed to download the translated docu- ment for the job in the project. Can you help me with that?
Step 1: Tool:
Step 1: Tool:
get project for textking translation
list. projects for textking translation
Args: ("projectid": project(33")
Args: [
Tool Response: The project with project 123" has been successfully retrieved, Details include project sta- tus, assigned translators...
Tool Response: ("description": "Missing access to- ken.")
Step 2: Finish
Step 2: Tool:
Chose to give and talk to the user.
get_job_for for_for_uee textking.tansslation
Args: ("projectid": "project123". "jobid": "job123") Tool Response: The 'Get job' API call retrieves de- tailed information project. the job specific translation job within 'job123' in project 123) the response would typically include de- tails such as the joh's current status...
Step 3: Finish
The project with ID "prujeci123' and the job with ID [job123| have been successfully retrieved, For down- loading the translated document, please use the project job's UUID designated translation management interface initiate the download process directly with the provided details.
Task With Hallucinated Answer.
Task Passed.
Figure 9. case study the missing parameter subset comparing the Baseline method and Relign: Relign successfully corrects the parameter hallucination.
15


Reviews:
[{"review_id": "SPmAX3Cama", "reviewer": "Reviewer_AGyu", "summary": "The paper focuses on the problem of tool hallucinations. The authors categorize tool hallucinations into selection and usage hallucinations, providing a framework for targeted improvements. A new benchmark is introduced to capture these categories (RelyToolBench) and model fine-tuning methods (Relign) are explored to fix these hallucinations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and valuable feedback. We provide our responses below:\n\n# Response about RePR\nWe apologize for any confusion caused by the description of the metric. In fact, **the model is not penalized multiple times**. RePR represents the proportion of tasks that were originally considered as successfully passed but were later verified to be free of result hallucinations through our hallucination detection. The deducted portion consists of tasks that were initially deemed successful but were later identified as containing result hallucinations. Importantly, tasks with a pass rate of 0 are not considered as having result hallucinations.\n\n# Response about RAG and Q1\n### (1) Relevance of Our Work to RAG\nIn our tool-based tasks, we provide the model with relevant information about each tool through the api description field in the prompts, as shown in Table 7 in the appendix. This includes descriptions of tool functionalities, tool parameters, and example values of parameters. **This setup can be considered a special form of RAG tailored to tool documentation.**\n\nSince the APIs in our dataset do not appear in the model’s training data, feeding tool API documentation into the model is essential; otherwise, the model would not be able to complete the tasks.\n\n### (2) Why RAG is Not Suitable for Tool Hallucination Scenarios\nAs mentioned above, we already provide the model with tool API usage instructions in the prompt, yet the model still exhibits different types of tool hallucinations during task execution. This could be due to the relatively **limited knowledge** contained in RAG-based knowledge retrieval.\n\nAnother possible way to construct a RAG knowledge base is to generate and retrieve real usage examples for specific tool APIs, similar to in-context learning. However, in our inference setting, many of the APIs we encounter are not present in the training database. This makes it difficult to pre-prepare corresponding demonstration examples. A potential alternative would be to synthetically generate a large number of tool invocation examples and then filter out high-quality examples. However, this approach does not generalize well to large-scale real-world tool-use scenarios and is beyond the scope of this paper.\n\n### (3) Discussion on RAG and Post-Training for Addressing Hallucination\n\n**a. Challenges in Constructing RAG Knowledge Bases**\n\nRAG relies on external knowledge bases, which can be difficult to construct. If the knowledge base from the training data does not sufficiently cover the inference-time data, applying RAG becomes challenging.\n\n**b. Fundamental Differences Between RAG and Post-Training**\n\nRAG is designed to mitigate hallucinations **by introducing external knowledge at inference time**. Unlike post-training methods, RAG does not update the model’s parameters, making it relatively lightweight. However, RAG typically retrieves specific pieces of information, which at most allows the model to mimic certain patterns, similar to in-context learning.\n\nIn contrast, post-training methods can deeply **alter the model’s cognition and capabilities through training**. When hallucinations stem from the model’s inherent weaknesses in understanding and interacting with the tool environment, RAG combined with prompting may be insufficient to mitigate these hallucinations. This is particularly relevant for the tool hallucinations observed in our experiments. In such cases, post-training is necessary to enable the model to better learn and model the tool interaction environment.\n\n# Response to Q2\nFirst, the RelyToolBench dataset includes both the original subset from StableToolBench and additional subsets specifically designed to evaluate particular hallucinations, namely the Unmatched Tools and Miss Parameter subsets. It is important to emphasize that the **original subset closely resembles real-world applications**. However, it also contains four different types of hallucinations as shown in Figure 6 in our paer, and both its tasks and APIs are relatively realistic. This demonstrates that **similar hallucinations occur in real-world scenarios**, not just in our two synthetically constructed subsets.\n\nAdditionally, as described in our response to Reviewer fZsh’s Weakness#3, we conducted **evaluations on an out-of-distribution test set APIBench**. We observed that models trained with our Relign method did not show a decline in tool-calling capabilities; in fact, they even exhibited some improvements. This provides further evidence **supporting the generalization ability** of our approach.\n\n# Response to Scientific Literature\n\nThank you for providing the references. We will cite them in future versions of our paper and supplement our discussion accordingly.\n\nWe greatly appreciate your thoughtful feedback and look forward to your response.", "claims_and_evidence": "The paper dives deeper into source of hallucinations in language model outputs when using tools. They find that models can not only generate invalid tool invocations but also not realize how many or which tools to use. The authors validate these on RelyToolBench and have discussed the construction of RelyToolBench. A concern here is that RelyToolBench is artificially constructed by purposely introducing errors, such as partial tool information in the prompt, and performs evaluations on a distorted distribution. The real-world generalization of the findings are unclear.\n\nThe new training method proposed “Relign” is the typical way of performing post-training of large language models [1, 2], i.e., SFT followed by DPO. They report a baseline (SFT&DPO) which is not as widely adopted a strategy. So the proposed method is not really new as such, and the Relign method has not been discussed as such.\n\n[1] Grattafiori, Aaron, et al. \"The llama 3 herd of models.\" *arXiv preprint arXiv:2407.21783* (2024).\n\n[2] Mehta, Sachin, et al. \"Openelm: An efficient language model family with open training and inference framework.\" *Workshop on Efficient Systems for Foundation Models II@ ICML2024*. 2024.", "methods_and_evaluation_criteria": "1. RePR = Pass rate - task hallucination rate. If task hallucination occurs, I assume that pass rate will be also be zero in that scenario. So, the model is penalized multiple times for failures here. I’m not sure if it makes sense to combine these into a single metric. Why not just view these separately, as they are?\n2. SFT&DPO is not a required baseline. Relign should be identified as SFT+DPO with the relevant references identified.", "theoretical_claims": "N.A", "experimental_designs_or_analyses": "1. The paper identified hallucination issues by creating a new benchmark. This benchmark is synthetically generated, which is largely OK, but some concerns raised above need to be addressed.\n2. See comments on RAG under various headings below.", "relation_to_broader_scientific_literature": "The paper is related to the body of work on tool usage with LLMs. It is also related to post-training strategies to reduce hallucinations. The paper dives deep into the mode of hallucinations [1, 2] for tool [3, 4] usage using a new benchmark and applies various post-training strategies [5, 6] to reduce hallucinations.\n\n[1] Varshney, Neeraj, et al. \"A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation.\" arXiv preprint arXiv:2307.03987 (2023).\n\n[2] Jain, Nihal, et al. \"On Mitigating Code LLM Hallucinations with API Documentation.\" arXiv preprint arXiv:2407.09726 (2024).\n\n[3] Patil, Shishir G., et al. \"Gorilla: Large language model connected with massive apis.\" Advances in Neural Information Processing Systems 37 (2024): 126544-126565.\n\n[4] Schick, Timo, et al. \"Toolformer: Language models can teach themselves to use tools.\" *Advances in Neural Information Processing Systems* 36 (2023): 68539-68551.\n\n[5] Grattafiori, Aaron, et al. \"The llama 3 herd of models.\" *arXiv preprint arXiv:2407.21783* (2024).\n\n[6] Mehta, Sachin, et al. \"Openelm: An efficient language model family with open training and inference framework.\" *Workshop on Efficient Systems for Foundation Models II@ ICML2024*. 2024.", "essential_references_not_discussed": "There is a large body of work concerning hallucinations and RAG which has not been adequately discussed in this paper. I think RAG serves as a simple baseline to reduce tool hallucinations and I did not follow the arguments to perform the RAG experiment for this paper. If the RAG experiment is not possible, at least the paper should recommend the reader about when to perform RAG and when to use their method.", "other_strengths_and_weaknesses": "The main novelty in the paper arises from the way the data is constructed and analyses around sources of hallucinations. These would serve as meaningful insights to the community.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How does RAG compare with the methods discussed to improve the hallucination rates? I understand a paper cannot have all experiments but I think RAG is simple and popular enough to concretely justify the need for post-training over RAG to reduce hallucinations. There are pros/cons of each and I think the paper should highlight these for the application(s) being discussed. When should someone follow the research of this work and not RAG methods for their work? Questions like these remain unanswered in the paper.\n2. RelyToolBench and the preference datasets were synthetically created. How do you justify the use of the training data and the evaluations to reflect real-world scenarios? Can you comment on this distribution shift?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P6oMhbbm3c", "reviewer": "Reviewer_ysVs", "summary": "This paper addresses tool hallucination in LLMs, where models incorrectly select or misuse tools. It introduces RelyToolBench for evaluation and Relign, a reliability alignment framework that enables LLMs to defer, clarify, or adjust tool use. Using SFT and DPO, Relign reduces hallucinations, improves task reliability, and enhances efficiency. Experiments show Relign outperforms baselines and achieves competitive performance with GPT-4o while reducing unnecessary tool calls.\n\n## update after rebuttal\nI’m convinced by the authors' opinion for including LLMs in the core framework. However, I don’t agree that Tool Num can serve as a sufficient proxy for efficiency. I also agree with the rest of the authors’ responses and will raise my score by one point.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and valuable feedback. We provide our responses below:\n\n# Re. to Evaluation Criteria & Q4\n\n### (1) Why did we introduce LLM-based evaluation?\n\nOur evaluation process for tool hallucinations consists of both rule-based evaluation and LLM-based evaluation. Rule-based evaluation can determine whether the LLM’s tool calls follow correct syntax. However, a key question arises: **Is a syntactically valid invocation actually non-hallucinatory?** This is why we introduced LLM-based evaluation—only an LLM can assess aspects beyond syntactic correctness and evaluate the content itself. LLM-based evaluation has also been widely used in prior research on hallucination assessment [1-2].\n\n### (2)How do we ensure evaluation accuracy?\n\nIn our LLM-based evaluation process, we decompose the assessment into specific evaluation tasks. On the one hand, **these evaluation tasks are significantly simpler than the full tool-calling task**, as making a critical judgment is easier than generating a valid action. On the other hand, we design **dedicated evaluation prompts** tailored to each evaluation task, guiding the LLM to assess various aspects in a structured manner. Furthermore, we explicitly **instruct the LLM to output unsure when it is uncertain**, rather than forcing it to provide a definitive evaluation result (as shown in Table 3 in the appendix). \n\nIn summary, we believe that using LLMs for automated evaluation is reasonable, and our human evaluation results further validate its accuracy.\n\n[1] FACTSCORE: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.\n\n[2] SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models.\n\n# Re. to Computational Efficiency\nWhile we did not conduct direct tests on the actual computational efficiency or time reduction, we actually use **Tool Num as an indicator for computational efficiency**. Since the model calls a tool once per interaction round, the average number of tool calls effectively corresponds to inference time and output length. Compared to dynamic and less stable metrics, the number of tool calls remains relatively fixed. Therefore, we primarily use the Tool Num metric to reflect the efficiency gains achieved by our method. As shown in Table 2, our model significantly reduces the average number of tool calls per task (ToolLlama reduces from 3.7 to 0.9, Llama3.1 from 2.2 to 1.5, and Qwen2.5 from 2.4 to 1.7). \n\n\n# Re. to Existing Metrics\n\nOur RePR metric is a refined version of Pass Rate, as we found that some tasks in the original pass rate metric exhibited result hallucinations. Therefore, we believe the pass rate has certain inaccuracies and did not include its results in our paper. As shown in the table below, we have supplemented some of the original Pass Rate results. The results indicate that RePR is lower than Pass Rate, and the gap is more pronounced in base models without Relign alignment. This suggests that unaligned models tend to produce more tool hallucinations, which in turn mislead the final results. Moreover, whether using Pass Rate or RePR, our Relign framework consistently improves task success rates. We will include the full results table in future versions of our paper for reference.\n\n|Method|I1  PR/RePR|I2 PR/RePR|I3 PR/RePR| Overall PR/RePR| \n|-|-|-|-|-|\n|Toolllama |76.8/64.2|77.0/64.4|72.3/58.9|75.4/62.5|\n| + Relign |77.1/68.2|77.3/68.3|76.2/71.0|76.9/69.1|\n|Llama3.1 |83.0/68.5|84.8/66.1|80.1/61.3|82.6/65.3|\n| + Relign |87.0/80.5|89.2/75.8|86.9/75.4|87.7/77.2|\n|Qwen2.5 |86.3/71.5|84.3/66.2|80.6/57.7|83.7/65.1|\n| + Relign |87.6/77.0|87.5/69.0|85.5/72.9|86.9/73.0|\n\n\n# Re. to Q1\nIn some special cases, repeated tool calls may be beneficial (e.g., when a tool fails due to high latency). However, our evaluation environment is based on StableToolBench, which **ensures that the tool environment remains stable**. In this scenario, repeated tool calls are typically an **abnormal behavior** of the model. Besides, in our specific evaluation of repeated tool calls, a tool call is only considered a hallucination if both the tool call itself and its environmental feedback are identical. This approach can partially address your concern.\n\n# Re. to Q2 and Q3\nIn the Indecisive Action Space, ChangeTool and TalkToUser are designed as **two special termination functions**. When the model invokes either of these functions, the tool invocation process is immediately terminated, as the model has determined that the task cannot be completed in the current environment. As a result, we did not design specific pipelines for tool switching (which would require a tool retriever ) or for user interaction (which would require a user simulator). Instead, we directly exit the process upon encountering these exceptional actions. We will explore how to better design task execution workflows following exceptional actions in future work.", "claims_and_evidence": "It is an obvious fact that LLMs can use tools, and the associated issues are well known. This paper categorizes these issues in detail, proposes evaluation metrics, and suggests improvements using SFT and DPO.", "methods_and_evaluation_criteria": "A significant portion of the evaluation process relies on the judgment of the LLM itself, which makes it somewhat contradictory to assess LLM hallucinations using another LLM.", "theoretical_claims": "There are no theoretically advanced descriptions, and no major issues seem to be present.", "experimental_designs_or_analyses": "All evaluation metrics (RePR, Tool Hallu, Tool Num, Utility) were newly proposed, which raises the question of whether none of the existing metrics were useful.", "relation_to_broader_scientific_literature": "There would be various tools that can be used by LLMs in the neat future (space travel, airplane, military, etc.).", "essential_references_not_discussed": "If there had been related works, I would have referred to them to explore recent trends, but unfortunately, there were none.", "other_strengths_and_weaknesses": "I like papers that define new problems and develop benchmarks or evaluation metrics for them. However, the problem itself seems to have already been widely discussed. Additionally, the introduction of the Indecisive Action Space is likely to reduce time or computational efficiency, but there is no analysis of this aspect.", "other_comments_or_suggestions": "The figures and experimental results were well-organized, making them easy to understand.", "questions_for_authors": "1. The repeated use of a tool was portrayed negatively, but aren’t there fields where repeated use is necessary to improve accuracy?\n2. In the Indecisive Action Space, how is it verified that the newly selected tool is actually correct when the tool is switched?\n3. Asking the user for input is not essentially an automated methodology, is it?\n4. As mentioned earlier, isn’t there an excessive reliance on LLMs to prevent incorrect generations by LLMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CDvjWB4PYM", "reviewer": "Reviewer_fZsh", "summary": "LLMs can solve diverse tasks by using external tools. Investigating tool hallucination is crucial because hallucination can cause problems that are hardly recognized. This paper introduces RelyToolBench to evaluate tool hallucinations, including four types of hallucinations. Authors also propose Relign to reduce tool hallucination by SFT and DPO. As shown in the experiments section of the paper, Relign generally outperforms other baselines. In addition, the large data size increases both tool and task hallucinations.\n\n## update after rebuttal\nI would like to acknowledge the contributions of this work in terms of both the benchmark and the proposed method. I have read other reviews and rebuttal comments. However, the scope and positioning of the paper are somewhat ambiguous. Thus, I keep my original score; please see the reasons below.\n\nThe benchmark proposed in this paper is not the first to tackle tool hallucinations, which may raise questions in terms of originality. The paper needs a detailed discussion of the pros and cons of the previous work. I am not convinced by the rebuttal comment, such as the lack of taxonomy. The previous work has proposed a well-organized benchmark in terms of in-depth and in-breadth. Also, the paper had to include the discussion before submission because the previous work is easily found by topic alone and published before ICML submission. The authors highlight the alignment and training as the one of key differences, but, more analysis and investigation are needed. Based on the conducted experiments and used hyperparameters, it is difficult to understand why the proposed method is better than others.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and valuable feedback. We provide our responses below:\n\n# Re. to Experimental Designs or Analyses\n\n- (1) All experiments share the same hyperparameters unless stated otherwise. In all methods, the learning rate is set to 1e-5 for SFT and 1e-5 for DPO to ensure consistency.\n\n- (2) The SFT dataset contains 10,000 fixed samples, so the number of training steps remains the same. However, DPO training steps vary across methods and models. We select 10,000 tasks and allow multi-round interactions with the environment. In each round, we sample ten outputs at a temperature of 0.7 to construct the DPO dataset. This results in a theoretical maximum of 10,000 tasks × number of interaction rounds, but not all rounds yield valid DPO samples. In practice, the number of DPO preference pairs ranges from 10,000 to 20,000. For example, in Relign training, Toolllama, Llama3.1, and Qwen2.5 yield 17,000, 19,000, and 14,000 pairs, respectively. The final training steps depend on the number of DPO pairs, using a batch size of 32 for two epochs.\n\n\n- (3) We did not perform a grid search but used standard values (e.g., 1e-5 for SFT, 1e-6 for DPO) to ensure stable fine-tuning, focusing on training data synthesis rather than hyperparameter optimization.\n\n- (4) Relign follows a two-stage training: SFT first, then DPO on the SFT model. The key distinction between Relign and other methods lies in the base model used for DPO data sampling. In Relign, the DPO training data is sampled from a model that has undergone SFT training, whereas in other methods, it is not. We believe SFT introduces an indecisive action space, which is then refined during DPO. Without SFT, the model generates fewer indecisive actions, leading to suboptimal DPO training. This explains Relign’s superior performance.\n\n- (5) We apologize for any confusion caused. Larger models generally reduce tool hallucinations due to improved general capabilities. However, increasing training data slightly raises hallucinations, likely due to overfitting on ToolBench data, which consists of well-structured tool interaction trajectories and lacks samples that handle edge cases (e.g., failure scenarios where a task cannot be completed). Thus the model learns correct tool calls but struggles with failure scenarios, defaulting to excessive hallucinated tool calls instead of invoking indecisive actions—precisely the issue Relign aims to address.\n\n- (6) We apologize for the typo. Our intended meaning was that larger models significantly reduce both tool hallucinations and excessive tool calls, leading to more reliable and efficient tool use.\n\n# Re. to Weakness #2\nThank you for the references. We highlight the key differences below:\n\n### (1) Systematic Definition and Real-World Evaluation\nThe cited work lacks a **comprehensive taxonomy** of tool hallucinations and **omits key types**, such as excessive tool calls (tool timing hallucinations) and fabricated tool parameters (tool content hallucinations). Additionally, its evaluation relies on sub-tasks like classification and description, which **do not reflect real-world tool use**. A model’s competence in these tasks does not necessarily prevent hallucinations during actual tool interactions. In contrast, our evaluation occurs within real-world tool-use scenarios, ensuring direct assessment of hallucinations in execution. We also introduce an **LLM-based automated evaluation** to systematically assess these hallucination types.\n\n### (2) Beyond Evaluation: Alignment and Training\nWhile the cited work focuses on dataset construction, **it does not address model training or alignment**. Our work goes further by proposing a framework and data synthesis approach that effectively reduces tool hallucinations and improves tool-use efficiency and reliability.\n\n# Re. to Weakness #3\nAs shown in the table below, we also evaluate our method on the out-of-domain test set APIBench. We follow the evaluation setup described in [1] (for more details, please refer to [1]). Rather than training on APIBench, we treat each API in the prompt as a function call to assess how well our trained model generalizes to OOD datasets. Experimental results with two types of tool retrieval indicate that Relign improves model performance on other tool-use tasks as well, suggesting that reducing tool hallucinations helps models learn better tool-use strategies (AST represents tool-calling accuracy).\n\n|Method| HuggingFace Hallu. | HuggingFace AST | TorchHub Hallu. | TorchHub AST | Tensorhub Hallu. | Tensorhub AST| \n|-|-|-|-|-|-|-|\n|Llama3.1 + BM25 |9.7|14.3|11.7|47.8|10.5|40.3|\n| + Relign + BM25 |**6.4**|**15.7**|**7.9**|**50.1**|**5.5**|**42.5**|\n|Llama3.1 + Oracle |9.3|87.8|10.4|86.1|7.8|89.6|\n| + Relign + Oracle |**6.5**|**89.5**|**7.7**|**88.9**|**3.2**|**91.3**| \n\n[1] Toolllm: Facilitating Large Language Models To Master 16000+ Real-World Apis.\n\n# Re. to Q1\nPlease see the response to Reviewer fZsh’s Existing Metrics for more details.", "claims_and_evidence": "Please refer to the weaknesses.", "methods_and_evaluation_criteria": "- The proposed method, Relign, is reasonable to mitigate the tool hallucination. SFT and DPO are widely used algorithms. \n- The evaluation criteria used in the paper are tool hallucination rate, task hallucination rate, RePR, and Utility. The used criteria are clear.\n\t- Tool hallucination rate considers the number of hallucination tool calls.\n\t- Task hallucination rate considers the sample-level hallucination.\n\t- RePR is the pass rate without hallucination.\n\t- Utility measures the quality in terms of success, the number of calls, and hallucinations.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "- The hyper-parameters used in other baselines, such as SFT, DPO, and SFT & DPO, are ambiguous. For example, does Line 325, \"the learning rate was set to 1e-5\", mean the learning rate is used in SFT & DPO or the proposed method? The experimental details should be clear.\n- I wonder if the training steps of SFT, DPO, SFT & DPO, and Relign are the same or not.\n- If authors use the grid search for finding hyper-parameters, the details should be included.\n- Relign is the sequential training of SFT and DPO. It is unclear why the proposed method outperforms single training of SFT and DPO and the joint training of SFT & DPO. \n- In Lines 381-382, the authors said that \"This could be attributed to overfitting\". However, considering (1) the number of tool calling number is reduced as the data size increases and (2) the hallucination is reduced as the model size increases, the overfitting might not cause the degradation of performance. Including the additional analysis can make the claim clear.\n- The authors discussed \"a noticeable decline in both Tool and Task Hallucination rates\" in the discussion of \"Does a Larger Model Reduce Hallucination?\". However, task hallucination increases as the model size increases.", "relation_to_broader_scientific_literature": "The paper is related to large language models with tools. The key contribution is the investigation of tool hallucination. It is related to the finding of prior work that LLMs have hallucinations.", "essential_references_not_discussed": "Prior work [R1] has proposed ToolBH to evaluate the hallucination in LLMs with tools. Since one of the main contributions is the evaluation of tool hallucination, discussion and comparison are needed.\n\n[R1] Toolbehonest: A multi-level hallucination diagnostic benchmark for tool-augmented large language models, EMNLP'24", "other_strengths_and_weaknesses": "**Strength** \n- Hallucination is an important research topic to improve the responsibility.\n- The paper categorizes the types of tool hallucinations and defines the metrics to evaluate the hallucination.\n- Sequential training with SFT and DPO shows the higher performance than others.\n\n**Weakness**\n- The major concerns regarding experiments are mentioned in **Experimental Designs Or Analyses**, such as experiment details and analysis. Please see **Experimental Designs Or Analyses**.\n- I respectfully disagree with \"pioneer a comprehensive evaluation framework for Tool Reliability\" because prior work [R1] has proposed ToolBH to evaluate the hallucination in LLMs with tools. Since one of the main contributions is the evaluation of tool hallucination, discussion and comparison are needed, as mentioned in **Essential References Not Discussed**.\n- Does Relign also outperform others on the prior benchmarks? Reducing hallucination is important, but the performance on the existing benchmarks is questionable.\n\n[R1] Toolbehonest: A multi-level hallucination diagnostic benchmark for tool-augmented large language models, EMNLP'24", "other_comments_or_suggestions": "Please see weaknesses.", "questions_for_authors": "- Why does the paper not report a naive pass rate?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L0Pt4v1YB3", "reviewer": "Reviewer_D96Q", "summary": "This paper discusses how LLMs suffer tool hallucinations, which can cause task failures and higher costs. It defines these errors as two main types: picking the wrong tool or using a tool incorrectly. The paper introduces RelyToolBench, a set of specialized tests and new metrics to measure and reduce such issues. The paper introduces Relign, a framework that helps LLMs make better tool-related decisions by allowing them to pause, ask for clarification, or switch tools when unsure. The experiments show that Relign effectively decreases tool hallucinations, making LLMs more reliable and efficient in using tools.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and valuable feedback. We provide our responses below:\n\n# Response about Weaknesses\nThank you for recognizing the novelty of our framework. We clarify our key contributions as follows:\n\n### (1) A systematic evaluation framework for tool hallucinations.\n\nOur framework integrates rule-based and LLM-based evaluations. While rule-based methods ensure syntactic correctness in tool invocation, they overlook whether **a valid invocation is actually non-hallucinatory**. This issue has been largely ignored in prior work, which often neglects tool content hallucination. Additionally, we introduce the **Reliable Pass Rate (RePR)** metric to address limitations in traditional pass rate evaluations by explicitly considering tool hallucinations.\n\n### (2) Joint alignment of helpfulness and reliability.\n\nUnlike previous work that focuses solely on task completion, our Relign framework also emphasizes **the reliability of tool usage**. This reduces both hallucinated and ineffective tool calls while maintaining high task success rates. Our approach jointly optimizes these factors using a comprehensive **utility** metric to ensure balanced performance.\n\n### (3) Differences from conventional SFT+DPO training.\n\nWhile our training builds on preference optimization, we redefine the optimization objectives by **introducing an indecisive action space**, teaching the model when to execute indecisive actions instead of hallucinating. Unlike general preference alignment methods that optimize multiple factors, we explicitly focus on helpfulness and reliability, leveraging **structured preference data** to refine tool invocation decisions. Rather than modifying the core algorithm, our contribution lies in refining **training objectives and data synthesis**.\n\n\n# Response about Q1\n\nIn our work, indecisive actions are implemented as special termination functions, such as direct termination, tool switching, and TalkToUser. When the model invokes these functions, the tool invocation process ends immediately, as the model determines the task to be infeasible. The model learns to use these functions through SFT and DPO training.\n\n- **SFT stage**: We introduce the indecisive action space by modifying tasks to be unsolvable (e.g., changing toolsets) and training the model to invoke appropriate termination functions instead of hallucinating.\n\n- **DPO stage**: The model refines its decision-making through preference learning. At each tool invocation step, multiple outputs are sampled.\n\n   - If the task is feasible, the preference order is: correct tool invocation > hallucinated tool invocation > indecisive action.\n\n   - If the task is infeasible, the order is: indecisive action > hallucinated tool invocation (as correct tool use is impossible).\n\nThis data-driven approach enables the model to assess the task's feasibility and determine whether to continue tool invocation or execute an indecisive action, effectively reducing hallucinations.\n\n\n# Response about Q2\n\n\n### (1) Is there additional human evaluation for the metrics?\n\nOur evaluation metrics are improvements upon the **Pass Rate** introduced in ToolBench, which has been validated through human evaluation in its original work. While our study introduces several new evaluation metrics, they primarily rely on tool hallucination evaluation. Thus, we conducted targeted human evaluation on the LLM-based components to ensure accuracy. Given the alignment with prior metrics, we believe this evaluation is sufficient. However, if specific aspects require further assessment, **we are open to conducting additional evaluations**.\n\n\n### (2) Is LLM-based automatic evaluation feasible and reasonable? How do we ensure evaluation accuracy?\n\nOur evaluation is divided into two specific tasks: assessing tool-task relevance and checking parameter-user input consistency. These tasks are **simpler than full tool invocation**, as critique-based judgments are easier than generative actions. We further enhance reliability by **using specialized prompts to guide assessments** and instructing the LLM to **return “unsure” when uncertain** (see Table 3 in the appendix). This approach minimizes errors and ensures accurate evaluations. Our human validation further confirms its effectiveness.\n\n\n# Response about Q3\n\nApologies, but we have not conducted additional exploration into model architectures. To the best of our knowledge, existing research on hallucinations has not discussed the impact of different model architectures (e.g., MoE, Mamba) on hallucinations. Our work primarily focuses on proposing an evaluation framework for tool hallucinations and improving tool hallucination mitigation based on this framework. We believe investigating the relationship between model architectures and tool hallucinations is beyond the scope of this paper.\n\nWe greatly appreciate your valuable feedback and look forward to your response.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "The main result uses specific training data sizes and model scales, which might not generalize to all LLMs or tool-use scenarios. Please show detailed results in multi scales.", "relation_to_broader_scientific_literature": "The contributions of this paper build on and extend existing works by providing detailed evaluation metrics, a specialized benchmark, and a new alignment framework. These advancements aim to enhance the reliability LLMs in their interactions with external tools.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper presents a novel framework for addressing tool hallucinations in LLMs. The categorization of tool hallucinations into selection and usage errors, with further subtypes, is a good approach to systematically addressing this issue. \n\nWeaknesses:\n\nWhile the framework is novel, it builds on existing ideas from reinforcement learning and preference optimization. The paper could benefit from a more detailed comparison with related methods/benchmarks to highlight the specific advancements it makes.", "other_comments_or_suggestions": "See Questions", "questions_for_authors": "1. The concept of \"indecisive actions\" in the Relign framework seems crucial for reducing tool hallucinations. Could you clarify the specific mechanism or threshold the model uses to decide between executing a tool call, deferring action, or seeking clarification? \n\n2. The evaluation metrics are central to demonstrating the effectiveness of Relign. Beyond the limited human evaluation mentioned in Appendix A, have these metrics been validated against more extensive human judgments of task success and system reliability?\n\n3. The paper shows that larger models generally perform better and hallucinate less. Have you explored how different model architectures (beyond just scale or model type(company)) might affect hallucination rates? Any insightful analysis??", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of LLM's using the wrong tool or hallucinating tool parameters by introducing RelyToolBench, a new benchmark that extends StableToolBench with two new subsets and new metrics. Further, the authors propose Realign, a finetuning approach using synthetic data, along with SFT and DPO, to enable an LLM to better select tools or seek clarification from the user.

The paper received 4 reviews with split scores: weak accept, accept, weak reject, weak reject. All reviewers acknowledged the authors rebuttal. 

The most substantive concerns raised by reviewers include:
1. Concerns about the benchmark being synthetically generated, and may not generalize. In response, the authors provided an evaluation on the out-of-distribution test set from APIBench, showing improved performance over baseline Llama3.1 setups. 
2. Lack of novelty in 'Realign'. The AC agrees that effectively, Realign is SFT + DPO, and the main contribution to the method is including the examples of seeking clarification in the training data.
3. Missing connections to previous work on hallucinations, and in particular the paper Toolbehonest: A multi-level hallucination diagnostic benchmark for tool-augmented large language models, EMNLP'24 (published several months before the ICML deadline). The AC's opinion is that missing ToolBH is a significant oversight. In rebuttal, the authors argued to diminish the contributions of ToolBH.
4. Metrics are new and unproven. Here, the AC agrees with the authors that the new metrics are simple and intuitive enough that verification (for example, via correlation with human judgements) would not be required.

Overall, the main new idea in the paper is training the LLM to seek clarification rather than hallucinate a tool call. The AC read ToolBH and confirmed that this contribution is novel. Therefore, the paper warrants acceptance provided the authors can update the related work to include the missing references (with a gracious assessment of the value of previous work).
Decision: Accept (poster)