ICML 2025

Reducing Tool Hallucination via Reliability Alignment
Hongshen Xu 1 2 3 Zichen Zhu 1 2 3 Lei Pan 4 Zihan Wang 1 2 3 Su Zhu 4
Da Ma 1 2 3 Ruisheng Cao 1 2 3 Lu Chen 1 2 3 5 Kai Yu 1 2 3 5
Abstract
Large Language Models (LLMs) have expanded
their capabilities beyond language generation to
interact with external tools, enabling automation
and real-world applications. However, tool hallu-
cinations—where models either select inappropri-
ate tools or misuse them—pose significant chal-
lenges, leading to erroneous task execution, in-
creased computational costs, and reduced system
reliability. To systematically address this issue,
we define and categorize tool hallucinations into
two main types: tool selection hallucination and
tool usage hallucination. To evaluate and mitigate
these issues, we introduce RelyToolBench, which
integrates specialized test cases and novel metrics
to assess hallucination-aware task success and ef-
ficiency. Finally, we propose Relign, a reliability
alignment framework that expands the tool-use
action space to include indecisive actions, allow-
ing LLMs to defer tool use, seek clarification, or
adjust tool selection dynamically. Through exten-
sive experiments, we demonstrate that Relign sig-
nificantly reduces tool hallucinations, improves
task reliability, and enhances the efficiency of
LLM tool interactions. The code and data are
publicly available at https://github.com/
X-LANCE/ToolHallucination.
1. Introduction
The primary goal of tool learning is to enable Large Lan-
guage Models (LLMs; Gemini Team, 2023; Achiam et al.,
2023; Dubey et al., 2024) to understand and effectively use
1X-LANCE Lab, School of Computer Science, Shanghai Jiao
Tong University, Shanghai, China. 2MoE Key Lab of Artificial
Intelligence, Shanghai, China.
3Jiangsu Key Lab of Language
Computing, Suzhou, China. 4AISpeech Co., Ltd., Suzhou, China.
5Suzhou Laboratory, Suzhou, China. xuhongshen@sjtu.edu.cn.
Correspondence to: Lu Chen <chenlusz@sjtu.edu.cn>, Kai Yu
<kai.yu@sj...