ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models
  in Multi-Hop Tool Use
Authors: name: Junjie Ye; name: Zhengyin Du; name: Xuesong Yao; name: Weijian Lin; name: Yufei Xu; name: Zehui Chen; name: Zaiyuan Wang; name: Sining Zhu; name: Zhiheng Xi; name: Siyu Yuan; name: Tao Gui; name: Qi Zhang; name: Xuanjing Huang; name: Jiechao Chen
Published: 2025-01-05 11:06:55

Abstract: Effective evaluation of multi-hop tool use is critical for analyzing the
understanding, reasoning, and function-calling capabilities of large language
models (LLMs). However, progress has been hindered by a lack of reliable
evaluation datasets. To address this, we present ToolHop, a dataset comprising
995 user queries and 3,912 associated tools, specifically designed for rigorous
evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful
interdependencies, locally executable tools, detailed feedback, and verifiable
answers through a novel query-driven data construction approach that includes
tool creation, document refinement, and code generation. We evaluate 14 LLMs
across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and
GPT), uncovering significant challenges in handling multi-hop tool-use
scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%,
underscoring substantial room for improvement. Further analysis reveals
variations in tool-use strategies for various families, offering actionable
insights to guide the development of more effective approaches. Code and data
can be found in https://huggingface.co/bytedance-research/ToolHop.