Benchmark: GTA: A Benchmark for General Tool Agents
Summary: GTA is a benchmark designed to evaluate LLM-based tool agents in realistic settings using multimodal written-by-human queries and executable tools. It features 229 tasks requiring reasoning, planning, and real-world tool use, spanning perception, operation, logic, and creativity.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: tool-use capabilities, reasoning, planning, natural language understanding, creativity
Phenomenon defined by authors: Yes
Phenomenon definition: Tool-use capability refers to the ability of a model to reason about and execute real-world tasks by using appropriate tools in the right sequence, based on user queries and multimodal inputs.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Each task requires the model to interpret the query, reason about which tools are needed, plan a sequence of tool invocations, and execute them to get to a final answer. There are four categories - perception, operation, logic, and creativity, and all are executable. Models are evaluated both step-by-step (tool prediction at each step) and end-to-end (solving the task through actual tool use).
Task item definition: A single item in the dataset is composed of five parts: a set of one or two image files (F), a query based on the files (Q), the set of tools involved (T), a reference tool chain with steps (C), and a final answer (A).
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Tasks are constructed to closely mimic real-world scenarios using executable tools and realistic multimodal inputs. However, all queries and tool chains are human-designed for evaluation, not sourced from real user interactions. Therefore, GTA is ecologically realistic but at the end rather synthetic.
Dataset size: 229
Dataset size detail: Tasks use 252 image files and span 14 tools, with 1–4 tools used per task.
Dataset metadata available: Yes
Dataset metadata detail: Query type (objective, subjective, image generation), tool usage, number of steps, tool categories
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring), Distribution (perplexity, calibration, correlation), Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric statistics: Simple mean/sum, correlation (Pearson's r)
Metric subscores: Yes
Metric subscores detail: InstAcc, ToolAcc, ArgAcc, SummAcc, AnsAcc, AnsAcc w/ ImgGen, F1 score for tool selection in Perception, Operation, Logic, and Creativity categories.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors discuss construct validity via analysis of tool execution, error types, and correlations between intermediate and final metrics.
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use