Title: ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
Benchmark: ToolBH
Summary: This paper presents ToolBH, a benchmark designed to diagnose hallucinations in tool-augmented large language models (LLMs). Hallucinations are evaluated from two dimensions: depth, using a multi-level evaluation framework, and breadth, encompassing three distinct scenarios that are likely to induce hallucinations. The authors developed seven tasks and curated 700 evaluation samples through multiple rounds of manual annotation.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Hallucination diagnosis for tool-augmented LLMs.
Phenomenon defined by authors: No
Phenomenon definition: Hallucination occurs when the output is inconsistent with the input, contradicts established knowledge, or cannot be verified against factual data. In this paper, hallucination for tool-augmented LLMs occur when LLMs attempt to address tool-use requests that they believe are solvable but are inherently unsolvable. How do tool-usage LLMs behave on unsolvable tasks?
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Take a user query and a set of available tools (multi-level diagnosis), and: decide if the query is solvable using only those tools (yes or no), if solvable, provide the steps needed to solve it using the tools (a solution plan), and if any parts aren't solvable, describe what kind of tools would be needed (specify missing functionality). In a different level (hallucination-inducing), take a user query and an incomplete or misleading toolset, and check whether the model correctly identifies tool limitations or hallucinates missing tools.
Task item definition: It consists of a user query (task), a list of tools and sub goals
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 700
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Across each subtask 
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Author validity detail: The authors argue that the data curation pipeline is designed to reflect real-world tool usage scenarios based on user queries, and this is manually validated to ensure quality. The Level 2 (in-breadth) analysis focuses on hallucination-inducing settings, where the toolset is deliberately altered to introduce incomplete or misleading information, testing whether LLMs recognize these limitations or hallucinate missing tools or capabilities.
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use