ToolSandbox: A Stateful, Conversational, Interactive Evaluation
  Benchmark for LLM Tool Use Capabilities
Authors: name: Jiarui Lu; name: Thomas Holleis; name: Yizhe Zhang; name: Bernhard Aumayer; name: Feng Nan; name: Felix Bai; name: Shuang Ma; name: Shen Ma; name: Mengyu Li; name: Guoli Yin; name: Zirui Wang; name: Ruoming Pang
Published: 2024-08-08 05:45:42

Abstract: Recent large language models (LLMs) advancements sparked a growing research
interest in tool assisted LLMs solving real-world challenges, which calls for
comprehensive evaluation of tool-use capabilities. While previous works focused
on either evaluating over stateless web services (RESTful API), based on a
single turn user prompt, or an off-policy dialog trajectory, ToolSandbox
includes stateful tool execution, implicit state dependencies between tools, a
built-in user simulator supporting on-policy conversational evaluation and a
dynamic evaluation strategy for intermediate and final milestones over an
arbitrary trajectory. We show that open source and proprietary models have a
significant performance gap, and complex tasks like State Dependency,
Canonicalization and Insufficient Information defined in ToolSandbox are
challenging even the most capable SOTA LLMs, providing brand-new insights into
tool-use LLM capabilities. ToolSandbox evaluation framework is released at
https://github.com/apple/ToolSandbox