GTA: A Benchmark for General Tool Agents
Authors: name: Jize Wang; name: Zerun Ma; name: Yining Li; name: Songyang Zhang; name: Cailian Chen; name: Kai Chen; name: Xinyi Le
Published: 2024-07-11 17:50:09

Abstract: Significant focus has been placed on integrating large language models (LLMs)
with various tools in developing general-purpose agents. This poses a challenge
to LLMs' tool-use capabilities. However, there are evident gaps between
existing tool-use evaluations and real-world scenarios. Current evaluations
often use AI-generated queries, single-step tasks, dummy tools, and text-only
interactions, failing to reveal the agents' real-world problem-solving
abilities effectively. To address this, we propose GTA, a benchmark for General
Tool Agents, featuring three main aspects: (i) Real user queries: human-written
queries with simple real-world objectives but implicit tool-use, requiring the
LLM to reason the suitable tools and plan the solution steps. (ii) Real
deployed tools: an evaluation platform equipped with tools across perception,
operation, logic, and creativity categories to evaluate the agents' actual task
execution performance. (iii) Real multimodal inputs: authentic image files,
such as spatial scenes, web page screenshots, tables, code snippets, and
printed/handwritten materials, used as the query contexts to align with
real-world scenarios closely. We design 229 real-world tasks and executable
tool chains to evaluate mainstream LLMs. Our findings show that real-world user
queries are challenging for existing LLMs, with GPT-4 completing less than 50%
of the tasks and most LLMs achieving below 25%. This evaluation reveals the
bottlenecks in the tool-use capabilities of current LLMs in real-world
scenarios, which provides future direction for advancing general-purpose tool
agents. The code and dataset are available at
https://github.com/open-compass/GTA.