NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of
  Large Language Models
Authors: name: Han Han; name: Tong Zhu; name: Xiang Zhang; name: Mengsong Wu; name: Hao Xiong; name: Wenliang Chen
Published: 2024-10-15 17:33:43

Abstract: Large language models (LLMs) combined with tool learning have gained
impressive results in real-world applications. During tool learning, LLMs may
call multiple tools in nested orders, where the latter tool call may take the
former response as its input parameters. However, current research on the
nested tool learning capabilities is still under-explored, since the existing
benchmarks lack of relevant data instances. To address this problem, we
introduce NesTools to bridge the current gap in comprehensive nested tool
learning evaluations. NesTools comprises a novel automatic data generation
method to construct large-scale nested tool calls with different nesting
structures. With manual review and refinement, the dataset is in high quality
and closely aligned with real-world scenarios. Therefore, NesTools can serve as
a new benchmark to evaluate the nested tool learning abilities of LLMs. We
conduct extensive experiments on 22 LLMs, and provide in-depth analyses with
NesTools, which shows that current LLMs still suffer from the complex nested
tool learning task.