Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
Authors: Zhi Gao
Bofei Zhang
Pengxiang Li
Xiaojian Ma
Tao Yuan
Yue Fan
Yuwei Wu
Yunde Jia
Song-Chun Zhu
Qing Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: Multimodal Agents
Vision-language Model
Tool usage
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The advancement of large language models (LLMs) prompts the development of multi-modal agents, which are used as a controller to call external tools, providing a feasible way to solve practical tasks. In this paper, we propose a multi-modal agent tuning method that automatically generates multi-modal tool-usage data and tunes a vision-language model (VLM) as the controller for powerful tool-usage reasoning. To preserve the data quality, we prompt the GPT-4o mini model to generate queries, files, and trajectories, followed by query-file and trajectory verifiers. Based on the data synthesis pipeline, we collect the MM-Traj dataset that contains 20K tasks with trajectories of tool usage. Then, we develop the T3-Agent via Trajectory Tuning on VLMs for Tool usage using MM-Traj. Evaluations on the GTA and GAIA benchmarks show that the T3-Agent consistently achieves improvements on two popular VLMs: MiniCPM-V-8.5B and Qwen2-VL-7B, which outperforms untrained VLMs by 20%, showing the effectiveness of the proposed data synthesis pipeline, leading to high-quality data for tool-usage capabilities.

Published as a conference paper at ICLR 2025
MULTI-MODAL AGENT TUNING: BUILDING A VLM- DRIVEN AGENT FOR EFFICIENT TOOL USAGE
Zhi Gao 1.2+ Bofei Zhang?* Pengxiang Li Xiaojian Ma², Tao Yuan", Yue Fan
Yuwei 3 Yunde Jia' Song-Chun Zhu 1.2.5 Qing Li
School of Intelligence Science and Technology. Peking University
2 State Key Laboratory of General Artificial Intelligence, BIGAI
"Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology,
Beijing Institute of Technology
"Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University DDpartment of Automation, Tsinghua University ma mat-agent.... g thub. i
Query: want to buy a PS5 for each child in the photo. How
User dollars will need to spend in total?
Thought: the facedetection
detect
First analyze the Image of identifiable children. Then determine the price of from image2.
Thought:
number
human faces in the two images.
Code:
faces, imagel fecedetection! "image, 1.jpg"|
GPT-4
Code:
T3 Agent
faces image2 facedetection( 'image_2.jjp 2/ppg']
drivenl
faces_ image - facedetecton""image j.jgg| price =image_qa ("what is price? ", "Image_? ___-pe"))
Observation:
Agent
Faces in Image 1:4 bounding boxes
Observation:
Faces image I]
Image 1: bounding baxes
Thought: There are children in total. The price
Price PSS: $479.99
a PSS is approximately $500, so the cost 4" 500.
Thought: Using the price of $479.99 for each console.
Code:
Code:
Final _answer - 500.
Final_answer 4*499.99
Query:
picture want to buy one NVIDIA GeForce According to the price in January, how many dollars will they need spend total?
SUPER
User
Thought: Use the "acedetection" tool to detect the number of human faces in the image
Thought: Use the Tacedetection" tool to detect the number of human faces image
Code:
Code:
face facedetection!" imagee 1.jpg")
GPT-4
face facedetecttion! "miag___jjp"" 1-jpg")
Observation:
driven!
Observation:
Agent
Faces 3 bounding boxes
Agent
Faces :3 bounding boxes
Thought: Use the 'image_ tool for the price
Thought: the 'eb "search'' for the price of NVIDIA GeForce RTX 4D70 4070 SUPER.
NVIDIA GeForce RTX 4070 SUPER.
Code:
Code:
image qa what the price GeForce
price web_wwssee search (QUIry-" GeForce RTX
RTX 4070 SUPER for three men", "image_ 1.jpg))
4070 SUPER price")
Observation:
Observation:
This image does not provide any price. cannot olatiin useful information.
On lanuary 2024, Nvidia released the RTX
Thought:
4070 SUPER at $599,
price about $1800 three men.
Thought: price $599. are three men.
Code:
Final, answer 1800
Code:
Final, _an__ansserr 3 599
Figure 1:
The comparison of the LLM (GPT-4))driven agent and our T3-Agent. Our agent chooses more precise tools based on the given files and intermediate observations.
ABSTRACT
The advancement of large language models (LLMs) prompts the development of multi-mode agents, which are used as controller to call external tools, providing a feasible way to solve practical tasks. In this paper, we propose a multi-modal agent tuning method that automatically generates multi-modal tool-usage data and tunes vision-languaa model (VLM) as the controller for powerful tool-usage reasoning. To preserve the data quality, we prompt the GPT-4o mini model to generate queries, files, and trajectories, followed by query-fil file and trajectory verifiers. Based the data synthesis pipeline, we collect the MM-Traj dataset that contains 20K tasks with trajectories of tool usage. Then, we develop the T3-Agent via Trajectory Tuning on VLMs for Tool usage using MM-Traj. Evaluations on the GTA and GAIA benchmarks show that the T3-Agent consistently achieves improvements on two popular VLMs: MiniCPM-Ve V-8.5 SB and Qwen2-VL. VL-7B, which outperforms untrained VLMs by 20%. showing the effectiveness of the proposed data synthesis pipeline, leading to high-quality data for tool-usage capabilities.
"Equal contribution.
X Corresponding author.

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
Integrating external tools to solve diverse multi-modal tasks is a promising research direction towards multi-modal agents (Surís et al., 2023; Gupta & Kembhavi, 2023; Gao et al., 2024; Yuan et al., 2024; Zhong et al., 2023). Existing agents usually use a large language model (LILM) as the controller that generates plans via prompt engineering to call tools, achieving impressive performance in multiple domains, such as image editing (Wu et al., 2023), robotic manipulation (ichter et al., 2023), question answering (Shen et al., 2024), video understanding (Fan et al., 2024), and desktop APPs (Trivedi et al., 2024). Despite their success, prompt engineering faces limited reasoning abilities for tool usage in tackling practical tasks, shown in Fig. 1. (1) The in-context examples in prompts only involve textual information, degrading the efficiency of tool usage in the multi-modal world. For the query 'How many dollars will I need to spend buy PS5 for each child?', the agent may select improper tools if does not know what the two images depict. (2) The pre-defined in-context examples are fixed and cannot tackle all tasks in the real world. For the task that requires searching for information from the web, the agent cannot use the proper tools, if in-context examples tend to use the 'image_qa tool. This motivates us to enhance the controller's ss reasoning ability for efficient tool usage.
In this paper, we propose multi- agent tuning method that automatically generates a large num- ber of multi-modal tasks with usale trajectories and tumes a vision- llsion-language model (VLM) (Liu et al., 2024b; Chen et al., 2024d; Yao et al., 2024) as the controller for powerful tool-usage reasoning. Compared with L.L--driven agents, VI.M-driven agents can utilize multi-modal information (such as required knowledge domains in the multi-modal data) instead of only using the query for reasoning, benefiting efficient tool usage (Liu et al., 2024c; Wang et al., 2024a; Sun et al., 2024). Many efforts are made to enhance specific capabilities of VLMs via finetuning, such as the chain-of bhain-of-thought ability (Hu et al., 2024), grounding ability (Peng et al., 2023), and feedback beebback-refining ability (Li et al., 2024). This inspires to construct large number of multi-modal tool-usage data for VLM-driven agents, which improves the reasoning ability when using tools for real-world tasks.
In doing so, we need overcome two challenges. (1) Collecting multi-modal tasks is challenging. Tasks in the real world usually involve multiple tools for multiple files (images, textual files, videos, audio, and etc). There are few off-the-shelf datasets for such tasks, and prompting models to generate natural and diverse queries with matched files is non-trivial. (2) Generating trajectories is challenging. Due to the complexity of trajectories, existing methods usually manually define templates and fill in key information for trajectory generation. This will limit the diversity of synthesis data and cause weak generalization for real-world tasks.
To overcome the above challenges, we introduce a novel tool-usage data synthesis pipeline that auto- matically generates a large number of multi-modal tool-usage data via three steps: query generation, file generation, and trajectory generation. Concretely, we first prompt GPT-4o mini (OpenAI, 2024) to generate queries and analyze what files are needed to solve the queries. Then, we produce files via two strategies. If needed files are images, we search for them from existing image datasets; otherwise, we prompt GPT-4o mini to produce codes to generate the needed files. Finally, we prompt a zero-shot agent to solve the generated tasks (i.e., queries and files) and collect trajectories, including the thoughts and codes in task solving. To preserve the data quality, the generated tasks and trajectories are passed through two verifiers to discard tow-quality data. After that, we these data tune VLM for efficient tool usage, through which one agent driven by the trained VLM could generate precise thoughts and codes for real-world tasks.
With the data generation pipeline, we construct MM-Traj. a dataset that contains 20K multi-modal tasks with tool-usage trajectories. Based on MM-Traj, we introduce the T3-Agent, a VLM-driven agent in the ReAct framework (Yao et al., 2023). The VLM controller of the T3-Agent is developed via Trajectory Tuning for Tool usage using MM-Trai We conduct comprehensive evaluations of the T3-Agent on the GTA (Wang et al., 2024b) and GATA benchmarks (Mialon et al., 2023), where two popular VLMs are used as the controller, that MiniCPM-V-8.5K (Yao et al., 2024) and Qwen-VI.- 7B (Wang et al., 2024c). The T3-Agent consistently achieves improvements outperforms the untrained VLMs by 20%. This indicates that our multi-modal agent tuning method enables agents powerful tool-usage capability for complex and diverse trajectories.
the two VLMs and
In summary, our contributions are three-fold. (1) We propose a multi-moda agent tuning method that automatically generates a large number of multi-modal tasks with trajectories and tunes VLMs using the generated data for powerful tool usage. (2) We introduce MM-Traj, a modal tool-usage dataset that contains 20K tasks across diverse knowledge domains with 15K files and high-quality
2

---

Published as a conference paper at ICLR 2025
trajectories. (3) We develop the T3-Agent, multi-modal tool-usage agent that significantly improves the tool usage performance on two popular benchmarks: GTA and GAIA.
2 RELATED WORK
2.1 MULTI-MODAL AGENT
Using external tools to address complex tasks is important ability for multi-modal agents. Ac- cording to different controllers, existing agents can be categorized into LLM-driven agents and VLM-driven agents. LLM-driven agents utilize powerful LLMs as the controller and produce pseudo code (Gupta & Kembhavi, 2023; Gao et al., 2024). python code (Surís et al., 2023; Yuan et al., 2024), or JSON format (Shen et al., 2024) call tools via one-step reasoning. Considering the complexity of practical tasks, some methods (Yang et al., 2023; Fan et al., 2024; Yang et al., 2024) empower the agent with step-by-step reasoning. which allocates tools based on observations of previous steps. Compared with L. agents, VLM-driven agents are more efficient in task solving, since the VLM controller can utilize information from visual data in tool usage, showing superior performance in visual design (Sasazawa & Sogawa, 2024), web search (Zheng et al., 2024a), image editing (Wang et al., 2024e). embodied scenario (Zheng et al., 2024b), robotic manipulation (Sun et al., 2024), and etc. However, VLM (LM-driven agents have weaker reasoning ability, compared with LLM-driven agents. Thus, several works synthesize training data to tune open-source VLMs for general tool usage (Wang et al., 2024a; Liu et al., 2023a; 2024c). Due the challenges of trajectory generation, existing methods mainly focus on simple tasks requiring one or two tools, and only synthesize a small amount of data (e.g., 1K in (Liu et al., 2024c)). Different from them, our T3-Agent is tuned using scaled-up multi-modal data with complex tool-usage trajectories, through which our agent could solve more practical tasks with strong tool- uosle capability.
2.2 TOOL-USAGE DATASET
Several tool-usage datasets have been established for agents, such APIBank (Li al., 2023), Toolal- paca (Tang et al., 2023), ToolBench (Qin et al., 2023), AnyTooo (Du et al., 2024), agentohana (Zhang al., 2024a), APIGen (Liu al., 2024d), and AgentInstruct (Zeng et al., 2023). The above datasets contain little multi-modal data (e.g., images, videos, and audio) that are commonly encountered in the real world. Thus, to evaluate the performance of agents in solving multi-modal tasks, some multi- modal agent benchmarks have been built, such as the GUI benchmarks: OSWorld (Xie et al., 2024) and MMInA (Zhang et al., 2024b), multi-modal question answering benchmarks: GAIA (Mialon et al., 2023), GTA (Wang et al., 2024b), and m&m' (Ma et al., 2024), and comprehensive benchmarks: AgentBench (Liu et al., 2023b) and AgentGym (Xi et al., 2024). In addition, some efforts are paid to synthesize trajectory data using LLMs to improve the tool-usage ability of multi-modal agents. DEDER (Choi et al., 2024) resorts to in-context learning to generate trajectories, through which the ofain-of-thought reasoning ability is distilled from LLMs to small model. Lumos (Yin et al., 2024) converts ground-trul truth reasoning steps in existing benchmarks into the expected format of tool-usage trajectories. TASKBENCH (Shen al., 2023) samples trajectories from a predefined graph, and then generate queries. MILM-Tool (Wang et al., 2024a) and VL-Vlss (Liu et al., 2023a) collect trajectories based on image and tool descriptions. VisualAgentBench (Liu al., 2023b) manually designs trajectory templates to collect data. Different from the above methods that usually focus on simple and predefined trajectories, or rely on queries in off-the-shelff datasets, our data collection pipeline generates diverse tasks and complex trajectories without any constraints, improving the volume, complexities, naturalness, and diversity of the tool-usage dataset.
3 DATA COLLECTION
3.1 FORMULATION
Data Synthesis Pipeline. The proposed data synthesis pipeline is shown in Fig. 2, including three steps: query generation, file generation, and trajectory generation. To preserve the quality of data, we design query-file verifier and trajectory verifier discard low-quality data.
Data format. We format the multi-modal tool-usage data as {Fopa. Q, Q.T.C.COO. A... where Fogt denotes the multi-moda files, Q means the query, T mean the generated thought (i.e., plans to call tools), and C' means the generated code, O means observation (outputs of using tools), and A means the ground truth answer ope means that the files are optional, iee., some queries do not involve files.
3

---

Published as a conference paper at ICLR 2025
Prompt
Query Generation
tasked with generating
queries that will promot an agent to call various tools.
Query
toolse: <TDOL_SET>
Circle habitat of this Lnnial the map.
will now provide user query examples: SEED>
Query Seed
Please output the Queries a json format
File Generation
Image database
retrieval
Image
Fill 1: jig file. Two images contain callithrix flaviceps
Query- file check
and the world map. File .pff file. One file
File content generation
various animals, including the Brazil habitat for callithrix flaviceps.
Other files
PDF
Code for file generation
Trajectory
Trajectory Generation
Thought: Use the image_ga tool to identify animals,
Code:
Zero-Shot Agent
image, qaí what animal?, 'mmage1jjg'' print The animal is (a1].')
Observation:
The animals is callithri flaviceps.
Trajectory check
O
Thought: Use the file, inspector tool identify the habitat.
Code:
11 file_ ine_pecooto ('habitat of callithrix flaviceps', 'habitat,pef") print (f habitat is (12)))
MM-Traj
Observation: habitat Brazil.
Thought: Use the image, edit toal to circle Brazil,
Code: final answer image, edit ('drow circle on Brazil', 'inage2.pdf"
Figure The pipeline for data generation.
The query could be divided into two categories, question answering and image generation, where the answer A would be some descriptive texts for the former and images for the latter. In our setting, F includes 11 types of files, such as JPG, PDF and PPTX files (details in Appendix). Considering that solving one real-world task may require multiple steps involving multiple tools, T, C, and O can represented by the integration
thought, code, and observation multiple steps, and the data format is reformulated as (Fooe Q, (4. ('..... (n [ci,... Cn {or,... ,On) ) A}. where 4, G,, and o indicate the thought, code, and observation in the i-step respectively, and there are n steps in total. The thought, code, and observation composed of a trajectory {t1, C1, 01, ,ta Cre 0n of n steps to solve the query.
Image source. Visual information plays important role in multi-modal tasks. To enhance the diversity and comprehensiveness of our tool-usage data, we compile about 93K mmage-captioning pairs from 8 source datasets, including ChartQA (Masry et al., 2022), COCO (Lin et al., 2014), LLaVA (Wang et al., 2023a). SAM (Kirillov et al., 2023), TextVQA (Singh et al., 2019), Web- Celebrity (Liu al., 2015), Web-Lanndmal (Weyand al., 2020), and WikiArt (Saleh & Elgammal, 2015). These datasets cover multiple multi-modal tasks: visual question answering, chart/table/doc- ument analysis, science question answering. and etc. We use the ShareGPT4V model (Chen et al., 2024b) to produce captions for each collected image.
3.2 QUERY GENERATION
Our goal is to generate a large number of diverse, practical, and feasible queries. We manually write some seed queries by brainstorming and double-checking them to ensure their practicality. In each step of query generation, we feed several randomly sampled seed queries, tools with descriptions, and a designed prompt to the GPT-4o mini model that generates multiple queries. Adding tool descriptions to the prompt makes GPT-4o mini better understand the desirable queries, improving the feasibility generated queries. We tune the hyperparameters (such as temperature) GPT-4o mini to improve the diversity of generation.
4

---

Published as a conference paper at ICLR 2025
3.3 MULTI-MODAL FILE GENERATION
Different from existing multi-modal data synthesis methods that first sample multi-modal files (images in most cases) from off-the-shelf datasets and then feed the files to language models (e.g., ChatGPT) for query generation, we opt to first generate queries without files and then produce relevant files for the queries. The reasons are two aspects. (1) Practical tasks usually involve not only images but also other multi-moda files, such as DOCX, PPTX. XLSX, and PDF files. It is challenging to construct off-the-shelff datasets that contain enough files for real-world tasks. (2) Tasks usually based on multiple files instead of only one, while randomly sampled files may have weak relevance, and feeding them language models usually produces non-natural queries. is non-trivial to design a standard to automatically sample relevant files for query generation. (3) Using existing files to generate queries may limit the knowledge domain, decreasing the diversity of tasks. In contrast, generating files based on generated queries may lead to more diversity.
Concretely, for each generated query, we prompt GPT-4o mini to output the needed file type and file content. The files are divided into two categories: images and others. For needed images, we use the BGE model (Chen al., 2024a) to extract textual embeddings of the file content and compare its similarities with collected source images from off-the-shelf datasets. The top similar images are collected for the query. For other needed files, we prompt GPT-4o mini to extend the file content and generate Python code to produce the files.
3.4 ZERO-SHOP AGENT INFERENCE
Trajectories are collected by prompting zero-shot agent (without training) to solve our generated tasks (i.e., queries and files). We utilize the framework of ReAct agents (Yao et al., 2023), where GPT-4o mini is employed as the controller. It solves the query into multiple steps and generates thought and code for each step based on the observations of previous steps. We collect trajectories whose code can be executed, including the thoughts, codes, and observations of all steps. The details of the agent can be found in Section Appendix and Appendix D.
3.5 DATA VERIFICATION
To preserve the quality of generated tool-usage data, we design query-fill verifier and trajectory verifier to discard low-quality data. Using LLMs to verify the quality of LLM outputs has shown effectiveness in multiple methods, such verifying the generated instructions (Wang et al., 2023b) and verifying the generated plans (Liu et al., 2024d). Inspired by them, we argue that using LLMs can also verify the synthetic tasks and trajectories.
Query-file verifier. Considering that the generated queries may be infeasible to solve and the produced files may not match the queries, the query-file verifier filters out low-quality query-file pairs. We prompt GPT-4o mini as the verifier based on the following factors: (1) whether the query and files are relevant; (2) whether the files contain enough information to solve the query; and (3) whether the query can be solved based on the given tools.
Trajectory verifier. Similarly, we prompt GPT-4o mini as the trajectory verifier based on the following factors: (1) the trajectory should use the provided tools as much as possible; (2) the trajectory should be reasonable, that is, the trajectory should align with the object and context of the query; (3) the tool usage in the trajectory should consistent with the query and files; (4) the input arguments for tools in the trajectory should be correct; (5) the answer should be correctly summarized from observations of tool-usage; and (6) the final answer should be relevant to the query.
3.6 MM-TRAJ DATASET
Data that passes through the two verifiers is considered high-quality and collected in an MM-Traj dataset. summary, we collect 5K data points from query generation and file generation. After passing through the two verifiers, 20K data points are left with 15K files.
Scalability, Note that our method can extend to additional modalities by incorporating more tools and leveraging advanced multi-modal models. For example, to extend our method to the video modality (MP4, MOV), we can integrate a video search model into the data synthesis pipeline (like the image modality), and apply video-language model to the agent controller with powerful video processing models as tools. This approach ensures seamless adaptation to new modalities while maintaining efficiency and coherence.
5

---

Published as a conference paper at ICLR 2025
Geography
Enterkainment
ipg
Technology
Noture
25. 25.9%
Others
Sports
2.1%
4.2
Social
Travel
4.8%
Science
2.6%
Transporaaion
35.59 png
10.1% Food
Anance 16.4
uh other
mp3
History
2.% mp4
Ciy
Pptr
dock
Ervironment
14. 5%
Culture
(a) File type
(b) Domain Knowledge
Visual
Image Editing Segmentation mome moge Generation
File Inspector
5..6%
8.3%
3.7%
6.0%
60
Object Locaization
24.5%
40
3.6%
Face Delection
Web Search
20-
24.5%
Image
670
Answering
Step Number
(c) Tool Statistic
(d) Step Number
Figure 3: Data statistics on the MM-Traj dataset.
3.6.1 DATASET ANALYSIS
We provide four key statistics: file type, knowledge domain, step number, and used tools in the collected MM-Traj Dataset. File type. We show the distribution of files Fig. 3()), which reflects the diversity of our dataset. MM-Traj covers more than kinds of files, all of which commonly encountered the real world. Thus, the T3-Agent trained on MM-Traj can handle practical tasks since the multi-modal knowledge is not limited to images in our lives. Knowledge domaín. We show the involved knowledge of the generated tasks in Fig. 3(b), which can be divided into 16 non-overlap categories, spanning across finance, environment, culture. health, history, food, and etc. Training in such data provides rich knowledge to use tools to solve diverse practical tasks. Tools. In Fig. 3(c), we show the distributions of used tools in generated trajectories. The web search tool is the most commonly used tool, which is consistent with practical tasks requiring specific knowledge. Moreover, other tools are also widely used in our dataset. Step number. We show the distributions of step numbers of generated trajectories in Fig. 3(d). Trajectories in the MM-Traj dataset have diverse step numbers. Most tasks require 2-6 steps to solve, and some tasks require 7-8 steps, showing the complexity and diversity of our dataset.
4
T3-AGENT
4.1
WORKFLOW
To handle practical tasks that require complex trajectories, we opt for the framework of the ReAct agent that performs step-by-step reasoning for tool usage based on observations of previous steps. In each step. the agent generates thought and corresponding code to execute tools. Compared with other formats (e.g., ISON format), code is more flexible to handle different inputs and output types for various tools. Concretely, given a query Q and files F, the i-step of the agent is formulated as
t, c arg P(t., eq Fop T,Q,h.)
(1)
where t and of are thought and code for the i-th step, and h; = (t,,c,,0, O1, -4-1,.--1110 denotes the history (thought, code, and observation of previous steps).

---

Published as a conference paper at ICLR 2025
Table 1: Used tools in the T3-Agent
Tool name
Tool description
Web search
Perform complicated web browsing to answer a question Answer questions for queries based on attached images Answer questions for queries based on given files
Image question answering
File inspector
Visual segmentation
the given image Localize objects in given images and output the bounding boxes
Do instance segmentation
Object localization
Image generation
Create image according a textual prompt Edit image based on the textual prompt Detect human faces in given images and output the bounding boxes Various packages such as 'matplotlib' and 'openev'
Image editing
Face detection
Python package
4.2 TOOLS
We deploy real-executable tools for the agent instead of only providing tool names. Our tools are across multiple categories: web search, visual perception, image generation/edtting file understand- ing, multi-modal understanding, and multiple Python packages, shown in Tab.
4.3 TRAINING
Given a data point (Fapt: Q, ft..... Sntifer.... ,n,, fol. ont, A}, we train the VLM con- troller using the cross-entropy loss,
min (. - - P(t., cj Fup T.Q.
(2)
where D is the collected MM-Traj dataset and we sum the loss values of the n steps in the trajectory. Note that, in training VLMs, we do not use the final answer A, as we encourage the controller to leverage tools in solving given tasks, instead of directly producing an answer based on its internal knowledge. After training, we obtain the T3-Agent.
Model. We use the same model architectures as MiniCPM-V V-8.5B (Yao al., 2024) and Qwen2-VL. 7B (Wang et al., 2024c) as our VLM controllers, including their visual encoders, resamplers, and LLMs. We initialize the model using their released versions.
EXPERIMENTS
5.1 SETTING
To evaluate the effectiveness of the proposed multi-modal agent tuning method, we evaluate the T3-Agent on the GTA (Wang et al., 2024b) and GAIA (Mialon et al., 2023) benchmarks and compare with agents that use closed-source models (GPT-4, GPT-4o, and Claude3) and open-source models (LLaMA-3-77) 3-770-iintrus (Dubey al., 2024), Owenl.5-72B-ol 8B (Liu et al., 2024a), Intern VL.2-BB (Chen et al., 2024c), Qwen2-VL-7B (Wang et al., 2024c), and MiniCPM-V-8.5B (Yao al., 2024)) as the controllers. Concretely, we compare the T3-Agent with Lego Agent (AgentLego Contributors, 2023), Sibyl Agent (Wang et al., 2024d), and the Warm-up Act Agent (Mialon al., 2023). The huggingface agent (HF Agent) (HuggingFace Contributors, 2024) is the baseline agent, using the same tools as the T3-Agent. We conduct ablation experiments to evaluate our data synthesis pipeline and visualize the solving process of our T3-Agent.
(Bai et al., 2023), LLaVA-NeXT.
Training. To preserve the visual perception and reasoning capabilities of MiniCPM-V and Qwen2- VL, we combine the training data in MM-Traj with the data Cauldron (Lindström Abraham, 2022) and open-LI -aa-NeeX (Chen, 2024) datasets. We train 5 epoch over all data. In the training process of our VLM controller, we freeze the vision encoder and visual token compressor, and fine-tune the language model using LoRA (Hu al., 2022). We set the rank as 64 and apply LoRA on query, key, and value projection matrices in all self-attention layers. We use the Adam W optimizer with cosine annealing scheduler. The learning rate is 1e 6 and the batch size is 2. We set the max context window to 10240 to support the long trajectory our agent.
Benchmark. GTA and GALA benchmarks comprehensive evaluation benchmarks for multi-modal agents. GTA contains 229 tasks with 252 images, and the steps required to solve tasks range from 2 to 8, with most questions requiring 2 to 4 steps. It requires multi-modal agents to build powerful
7

---

Published as a conference paper at ICLR 2025
Table 2: Results the GTA benchmark
Method
Controller
AnsAce
ToolAce
CodeExec
Lego Agent
GPT-4
46.39
Lego Agent
GPT-4o
41.52
Lego Agent
GPT-3.5-turbo
23.62
Lego Agent
Claudc3-opus
23.44
Lego Agent
(wenl.55722Bcch
13.32
Lego Agent
LLaMA3-70B-inse -instruct
8.32
HF Agent
GPT-4o
$7.05
63,41
95.12
Agent
GPT-4o mini
57.69
56.10
100.00
HF Agent
LLaVA-NeXT:
14.10
14.97
25.08
HF Agent
InternVL2-88B
32.05
36.75
52.18
HF Agent
MmmICPM-V-8.5
33,97
36.59
56.10
HF Agent
Owen2-VI.-7B
42.31
44.85
65.19
T3-Agent
Tuned MiniCPM-V-885B
52.56
65.85
80,49
T3-Agent
Tuned Qwen2- VL-7B
53.85
64.63
84.32
Table 3: Results the validation set of the GAIA benchmark
Method
Controller
AnsAce
Level
Level
Level 3
Sibyl Agent
GPT-4-turbo
29.70
43.40
27.90
7.70
Warm-up Act
GPT-4-turbo
17.60
30.20
15.10
0.00
Agent
GPT-4o
33.40
47.17
31.40
11.54
HF Agent
GPT-4o mini
26.06
33.96
27.91
3.84
HF Agent
LLaVA-NeXT-8B
3.64
9.43
1.16
0.00
HF Agent
IntemVL2-8B
4.85
7.55
4.65
0.00
Agent
MiniCPM- V-8.5B
7.27
13.21
5.81
0.00
HF Agent
Qwen2-VL-7B VL-7B
9.70
16.98
8.14
0.00
T3-Agent
Tuned M-V..M
15.15
26.42
11.63
3.84
T3-Agent
Tuned Qwen2-VL-7B
16.97
26.42
15.12
3.84
perception, operation, logic, and creativity abilities on visual data. In addition to visual data, diverse files (such PPTX, PDF, XLSX files, etc) also commonly encountered in practical multi-modal tasks. To evaluate agents on such files, we use the GAIA benchmark that contains 446 tasks with 109 files. The tasks in GATA are divided into three levels, the steps of which range from arbitrarily long sequences, evaluating the capabilities of document understanding. web surfing, logic reasoning, and answer summarization.
Metric. In the GTA benchmark, we measure three metrics for agents, including AnsAcc, ToolAcc, and CodeExec. AnsAce measures the correctness of predicted answers. ToolAce means the accuracy of tool selection and answer summary. CodeExec quantifies the percentage of generated codes that could be executed without errors. In the GAIA benchmark, we measure AnsAce of its three levels.
5.2 GTA RESULTS
The performance of agents on the GTA benchmark is shown in Tab. 2, where AnsAcc, ToolAcc, and CodeExec are reported. Our agent achieves better results than the Lego agent that uses closed-source models (e.g., GPT-4 and GPT-4oo) and HF agent using open-source models (e.g., Intern VL2-8B), showing its effectiveness in solving complex tasks. The comparison agents using the tuned and untuned VLMs shows the effectiveness of our multi-modal agent tuning method. For example, tuning MiniCPM-V-S -- leads to about 18%, 29%, and 24% improvements in the answer accuracy, tool correctness, and code executability, respectively. In addition, compared to the HF agent using GPT-4o and GPT-4o mini, our agent has higher ToolAce while lower CodeExec, showing that our tuned VLM has more powerful reasoning capability for tool usage, while the weak programming capability results in worse AnsAcc. This inspires us to develop VL.Ms for writing codes.
5.3 GATA RESULTS
In Tab. 3, we report the performance of T3-Agent on the validation set of GAIA. T3-Agent performs better than agents driven by open-source models. For example, Qwen2-VL-7B achieves the best performance among all open-source models, while our agent is still 10% higher than it. The performance improvements across multiple VLMs validate the effectiveness of our dataset. Compared with agents driven by closed-source models (e.g., GPT-4), T3-Agent achieves worse performance.
8

---

Published as a conference paper at ICLR 2025
(s)
[b)
ACCU
DAY POST
Veed De gorhor
60
64
70
66
69
70
74
42
52"
52
neod spond toca??
Thought: meet firet eégecticcaton tool skentely the oumber of dagp. the to the
temprnhhe? And whet the wart diffreence between the mudimmm temperater and the
WWhich day this week low the wided
Code:
prist" Number degs" number dags)
Nowl mush the Image?.jR)
Thought:
calculate sead will between muamum maneum
To determine which day has the whtest mage tempennure
pprrice per_.oy)
weather foreaat image. then, 1ll icuntily the days with the greatest
Number disp:
Obsenostion:
ou
Code:
tempertture ringe
image path- "datujjta dattaerthmagelmma a00.ae"
Thought: need calculene the total cast for purchaaing dog boy for wach of the shown the first image. To do this cilculoiion on, IIll Gimplr multily the number shige by price per toy.
for
dops
eash day the 7-day forecant image puth-loricast image path)) prem)ammenttee sttrr
"Surday"" " tigh tar, low 421, "Morddy tigh Low 52 Tuanday tagg no: Low 52's "ngadweddy" to tigh law sa ""salurday"
rumber, of digg-4
price poi Joy
number TesalTotaloot loia cool
priceperr pee_ppr__ty
Thought how the videet range
entracted the temperatores for each day, will day identily
temperature
Code:
caksdnne tamperatin cemeltemmpaattuee return wat.mmeeettettteettm
(c)
mellttoneratuues
Query If wanted onder the two lowest priced
Manu
need dishes from menu and calcuaate their total
Beer
Product
Price
SS1
tuenday page coenrate ferr perpttue range) 13]
mondoy, jarge calcuaos tomgeratur wttpoootorrrooggglleeeee pangol154 520)
Hamburge
S1:
thusdex jange e calcu litee Jemperstre, Findo 5X0
The at S10, Milk So. 'will snrrect thile prices
555,
tompersturs rongel/c
saturdky ringe calou comperature (inget)) 541I
and shnn them
max range masjunday rangs mondoy Jange. tussdit_ pangt.
Iiday, saturdayy
pars =I ("Frddur" "Teef" "Price: ('Poddutt: "aanbugee'
sundda, range max range:
"cace": 101 ('Pndurrs "Mtic "Pree"-di
das with wessst range "Sundas"
minil rie sumipprce for bein in lnawe prices for price in femm sikeelll)
day with wadek, rarge Mord
rings
peetl Torslomtt tota roc)
Iuuaaday range max, rang::
Obsenation: Code
traed due to tollowing enon umupported
widsst rargu- "uaodds'
operand hypely for bet and for liw lets remyy sace oie not to repeor
twithesdyy rage max range
completely diferee
wadk
approash."
I Ithirddy, range in may jange.
Thought it snems the hose with makng diferene types data colouistio
with wadest, rergs- Thurdday" triic
Code:
day wnth widest rargs- "Idday"
menu hama ("roddutt "Meef" "Troe's 351, ('Frodust" "tamburger" "Pria":
Tsstundy
"MBl.. "ilee"
with, wides, /arge- Saturday"
proeu- [Bem] Vn ce Yor nem memertttrrs
arknt Day th wirest to peeatture rasge: day with wadgest (onge)
somedjprkell 21
total sundloeet, criss)
Obsencation
penti Totel tota coan)
wides temperature ronge Satunday
Temperature
Figure 4: Case study of the T3-Agent in the GTA benchmark.
The reason is that closed-soures source models use larger model sizes and more training data. These factors may primarily contribute to the performance differences.
5.4 DATA QUALITY
Table 4: Average scores from humans.
To evaluate the data quality of generated data in MM-Traj, we conduct a user study. Concretely, randomly sample 600 data points from the
MM-Traj
Filtered out Data
Task Trajectory 8.32
Task Trajectory
MM-Traj datasets and filtered out data. We ask 30 persons (with rich programming experience) to provide scores (1-10) for the task quality and trajectory quality, and they do not know whether the data is from MM-Traj or filtered out data. We ask the persons to provide scores for the tasks (the queries and files) and trajectories. The score is in the range (1-10), where higher scores mean better quality. Results are shown in Tab. 4. The quality of MM-Traj is higher than the filtered-out data, demonstrating the used verifiers can discard lower-quality data.
8.67
6.36
6.38
5.5 ABLATION
Table 5: Ablation
We conduct ablation experiments to evaluate the ef- fectiveness of our two verifiers, shown in Tab. 5.
Method
GTA
GAIA
We observe that on both the two benchmarks, the data using the two verifiers leads to better per- formance (e.g., 2.56% improvements on the GTA benchmark), showing the effectiveness of the two verifiers.
w/o two verifier
50.00
13.33
Ours (two verifiers)
52.56
15.15

---

Published as a conference paper at ICLR 2025
5.6 VISUALIZATION
In Fig. 4 and Fig. 5, we visualize cases solved by our T3-Agent in the GTA and GAIA benchmarks. We have the following conclusions. (1) Our agent could handle multiple-imaae reasoning tasks. By utilizing the visual information from given images, the agent could apply the correct tools and write correct arguments for given images. (2) Our agent could solve complex tasks requiring long code. (3) Our agent could revise code errors based on observations.
(4) T3-Agent can solve tasks with multi-hop questions. For example, in the first case of Fig. 5, our agent searches for information from the web, based obtained information in the first step. (5) T3-Agent could handle multi-modal files, such as audio and PDF files in Fig. 5.
obeny: During the firs wask of August 2015, one the hask Astronamy Pictuns of
Day shows the lights of dry on the horbon The romesskk this city alo has
trying figure out quantiinn
(b)
spring
was aragram.
Friday,
attached audoo reconding of cuestion thet asked, so you coule Prose limit your resporie to the omagra tee that could be generites from the origral line fubtib professor's regant, without any other
that dedg this landmars bulckig? Give the firs name appesing in the rame the fimm ex June 2012.
fird name the archinecuraal fimm dedigned landmark bullding
Chicao namodi sherthe diy ohe horioo, feedd to gather information about both the
please include
scorch rngarding the kisk datronomyy Picture the Day August 2015 and idontify the city
attccmment: ag) 2r1////A//20255 haal clation oldiio/ cbdielite-ccb 4356-4725--
mreniione balon wis its fandwarr bui in Oricago.
Thought: will the 'rspect, Sie seet tool to analyas the auaio recording and estroct the from This way, (on decemmiie the asagram Codl:
city info wibh Fead the city shown in the MASA Astronomyy Picture of the Dwy durigg tile wosk August 2015.)
raquest
commentary
Obscovvtion: Scarch outcome (short ve slont: The city shown the NASA Actronorry
path "data//auu//20a valdattion madie wat----------------------------- 421---4----------3 714---719-/22mmee Dacllooaloommp
2015 Mtrpuette Mitapen Now reed find out sbout landmank teaiding in Chirago denify dichtectural
Thought: The derbiiid from NiSA scaronory Pictree of the during August
Day
week
Michigans
of the tard mmme bincer trne. you thane it, anyone hewe's in the aragram. in one of the Sart's best thought of tragedin, our el fram queres Thought: hore sucrendedll extracted the see
boran axsgrmmett the following quctation adtually
precileses cestert)
Disy,
litlle cowll
designddlt.
Code:
web Find landmart bukling Orcago namod after
Mbrquette, Markeann
the anr I'm designed
atour haw austo notter pi prolidol is related to lane from clakesccre's play Harrie Now, will aragrom
gentllandiukk, jolo)
the
bestmork derigned by the sediitetund I'mm liceterd & forte, completed in 1895.
Though: hooe frunt mat the andmork hilding in therago nomed atter Marruettee,
Cook:
orginell
asys play.
Code:
Are groum
the tardh bent thought of trapedhes, our inshitent herm,
artiieetunll
ongannl Fime wet porrheguary "itton then lovrler say It'n oee of the traged our incisient horo, Hamiic, suo fronts how tum rostan??)
arthhecttun /iim split))1
fest
fnol jrower/fis (ist camre)
pi rnter gaiaal line)
(short
provided
Qvery: The sttached POF lats sccommodat one the resort community of Sealoore
not direct ciuatoon from Hamlest it most licly allcks to csstential themes exema fied
solliogay, ls.. Thought: The unagann am rafers to Hamree's femous sollicaa liluuuy "To be, not to be:
Attachment 22522/ kdatian 16 1578 Ho Soal 43/5 9044 os2/s//bu/Baa /aa pul"
Thought: dssermin dn tyoe accommodiioon on a higher average rating idand
read
After
information into condiie resporse that fultili
extracting the rakevant information on regeding aecommodaiionn and their ratings can
profissio' requaat..
Code:
type lighte swerage.
Codk:
bal))
pff pontent Inspcct fil, as boetifie wwwtoteeeeatttoooeeeoeo hwtpppotthhhhaaaaooe dew. /6703678
con--gys-autt--eel- rejoot content]
Obscrvatiove Namny Houess Nepoune's Polacc ad Sturgson Cur The Loughtng
5.l Leser Towers Motels
Revlow Yes
No
No Yes Yes D'rss 4 No 4 Yes No 4 hes a No No 3 Yes Yes You 5 Vis
Yes His
PDF
Thought: hare extractdd the ramrs of recormoddatonn and their co corresponding
Yes
rating from the how. pcccss
cakubee average rating accommadation Dotth, motoss rontal houses, damagroo roancca
Code:
"Hora"" C. "Mosct". "Rental Housest: "Camppround""
ratnng - 2.
5,4
5,5,4
miing wegpccommmbiiiiee kwysl), meting?:
Fraiig not hond:
chomml apperd)penng
retings [
name miting murroomodstttoos bemal
intingirrme suntrating (entrraing)))
print Vharage Ratings: awernge cstinl
Auerage Rotingi Theeess Motel Rental Houses?
rurdds
Thought: h clear that the hotels
higherr average rating 00 compared to average rating lest. eeeeforee will return the concuuton my firal anower.
Code: frol anl.aasonn""" hove the highest everage otting (Sano
Figure 5: Case study of the T3-Agent in the GAIA benchmark.
6 CONCLUSION
In this paper, we have presented multi-modal agent tuning method that improves the tool-usage capability of agents by generating a large number of tool-usage data and tuning VLM using these data. Given proper prompts and sufficient source images, our data synthesis pipeline can produce high-quantity multi-modal tasks with trajectories. We collect these generated data into an MM-Traj dataset to tune a MiniCPM-V V model, and the T3-Agent with the tuned model has achieved significant improvements on two mulli-modal benchmarks, demonstrating the effectiveness of the data synthesis pipeline and the collected MM-Traj dataset.
Limitation. In the current T3-Agent, we only consider the multi-modal information in queries. Practical tasks usually involve multi-modal data in trajectories of agents, such as the intermediate results in image editing tasks. We will study how to utilize multi-modal information in agent trajectories, which benefits in performing more powerful by-peby reasoning for tool usage.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. This work supported by the China Postdoctoral Science Foundation (No. 2024M760114) 14), National Science and Technology Major Project (2022ZD01 149000)) China Post- doctoral Science Foundation (No. 2024M7601 14), the Opening Project of the State Key Laboratory of General Artificial Intelligence (SKLAGI20240P0L SKLAGI20200P14) the Natural Science Foundation of China (NSFC) under Grants No. 62172041 and No. 62406009, the Key Program of the National Natural Science Foundation of Shenzhen under Grant No. 2024 120230003, and the Key Research Platforms and Projects of the Guangdong Provincial Department of Education under Grant 10.20232DZX0034,
REFERENCES
AgentLego Contributors. AgentLeeoo Open-source tool API library to extend and enhance LLM
agents, December 2023. URL wwtpthtts///// /gi higgtbtthb..ggggggggggggg..g.h wttpbb/.ooon/ttrteeeeee agent lego. 7
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18392-18402, 2023. 20
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, funciiotnnni multi-granulant text embeddings through geff-knowledge distillation. 2024a. 5
Lin Chen. Open-llava-next-miin Im dataset, 2024. URL https://hugging:: co/ datasets / httpee/sssss/tccce//e/eeaaaa///a NeXT- mi.
Lin Chen, Jisong Li, Xiaoyi Dong. Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision (ECCV), 2024b. 4
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi gpt-4v? closing the gap commercial
Hu, Jiapeng Luo, Zheng Ma, al. How far multimodal models with open-source suites, arXiv preprint arXiv:2404. 16821, 2024c. 7
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic disual-linguiste tasks. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024d.
Wonje Choi, Woo Kyung Kim, Minjong Yoo, and Honguk Woo. Embodied cot distillation from 11m to tffftthe-shelf agents. In International Conference on Machine Learning (ICML). pp. 8702-8721, 2024. 3
Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reftective, hierarchical agents for large- scale api calls. In International Conference Machine Learning (ICML), pp. 11812-11829 2024. 3
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024. 7
Yue Fan, Xiaojian Ma. Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: memory- augmented multimodal agent for video understanding. In European Conference on Computer Vision (ECCV), 2024. 2,
Zhi Gao, Yuntao Du. Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Clova: A closed-loop visual assistant with tool usage and update. In The IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 33258-13268 2024. 2,3
Chun Zhu, and Qing Li.
11

---

Published as a conference paper at ICLR 2025
Tammay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14953-14962, 2023. 3
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference
on Learning Representations (ICLR), 2022.
Yushi Hu, Otilia Stretcu, Chun-TT Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9590-9601, 2024. 2
HuggingFace Contributors. Agents and tools, 2024. URL hww.hhssss/hgggggggaaeeeeee/ co/ docs/
brian ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang. Sally Jesmonth, Nikhil Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as i can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL), pp. 287-318, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Interna- fional Conference on Computer Vision (ICCV), pp. 4015-4024 2023. 4
Jian Li, Yabiao Wang, Changan Wang, Ying Tai, Jianjun Qian, Jian Yang. Chengjie Wang, Jilin Li, and Feiyue Huang. Dsfd: dual shot face detector. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5060-5069, 2019. 20
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: comprehensive benchmark for tool-augmented 1lms. Annual Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3102-3116. 2023. 3
Pengxiang Li, Zhi Gao, Bofei Zhang, Tao Yuan, Yuwei Wu, Mehrtash Harandi, Yunde Jia, Song-Chun Zhu, and Qing Li. Fire: A dataset for feedback integration and refinement evaluation of multimodal models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05:3 05358, 2022. 7
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. //llava-vl gittiutuhh htt//blog///2/
URL https:
01 01-0000000000 30- allvaaaaett/////
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36. 2024b.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Ilava-plus: Learning to use tools for creating multimodal agents. arXiv preprint ar Mrxx:v3311.0A.
2023a. 3
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Imms as agents. In International Conference Learning Representations (ICLR), 2023b. 3
12

---

Published as a conference paper at ICLR 2025
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long. Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06332, 2024c. 3,
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference Computer Vision (ICCV), 2015. 4
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406. 18518, 2024d. 3,
Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. m&m's: A benchmark
to evaluate tool-use for multi-step multi-modal tasks. In CVPR Workshop, 2024. 3
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: benchmark for question answering about charts with visual and logical reasoning. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 2279, 2022.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. arXiv preprint arXiv:2311.1298. 2023. 3,7
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amna, Mostafa Dehghani, Zhuoran Shen, al. Simple open-vocabulary object detection. In European Conference on Computer Vision (ECCV), pp.
728-755. Springer, 2022. 20
OpenAl. Gpt-4o system card. gpt 40 syst em card. pdf.
2024.
URL https://cdn.oppnall com/
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306. /4824, 2023. 2
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toollim: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations (ICLR), 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2022. 20
Babak Salch and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint 3xXiv:5055.00 2015.
Yuichi Sasazawa and Yasuhiro Sogawa. Layout generation agents with large language models. arXiv preprint ar arXiv:2405.08007, 2024. 3
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. arXiv preprint arXiv:2311. 18760, 2023. 3
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems (NeurIPS), 2024. 2,3
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang. Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In The IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8317-8326, 2019. 4
Xiaowen Sun, Xufeng Zhao, Jae Hee Lee, Wenhao Lu, Matthias Kerzel, and Stefan Wermter. Details make a difference: Object state-sensitive neurorobotic task planning. arXiv preprint arXiv:2406.0990 099888 2024. 3,3
13

---

Published as a conference paper at ICLR 2025
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In International Conference on Computer Vision (ICCV), pp. 11888- 11888811898, 2023. 2,
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases, arXiv preprint N.hv::2306.05301, 2023. 3
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Anmual Meeting of the Association for Computational Linguistics (ACL), pp. 16022-16076. 2024. 2
Chenyu Wang. Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, XM Xuan, Zhengxin Li, Lin Ma, and Shenghua Gao. Mllm-tool: multimodal large language model for tool agent learning. arXiv preprini arXiv:2401.10727, 4, 2024a. 3
Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: A benchmark for general tool agents. In Advances in Neural Information Processing Systems (NeurIPS), 2024b. 2, 3,
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:231..07574 2023a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the 2024c. 2,
world at any resolution. arXiv preprint
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Annual Meeting of the Association for Computational Linguistics (ACL), 2023b. 5
Yulong Wang, Tianhao Shen, Lifeng Liu, and Jian Xie. Sibyl: Simple yet effective agent framework for complex real-world reasoning. 2024d. URL https::/arxii.rrrg
10718.
Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal IIm as an agent for unified image generation and editing, arXiv preprint arXiv:2407.05600 2024e. 3
T. Weyand, A. Araujo, B. Cao,, and J. Sim. Google Landmarks Dataset v2 A Large-Scale Benchmark for Instance-Level Recognition and Retrieval. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 4
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 2023. 2
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprint arXiv:2406.0 04115, 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972 2024. 3
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 138/, 2023. 3
Zongxin Yang. Guikun Chen, Xiaodi Li. Wenguan Wang, and Yi Yang. Doraemongpt: Toward understanding dynamic scenes with large language models (exemplified as a video agent). In International Conference Machine Learning (ICMI.,, pp. 55976-55997, 2024.
14

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. 2,5
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone, arXiv preprint
arXiv:2408. 0100, 2024. 7
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai- Rei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents, In International Conference on Machine Learning (ICML), pp. 12380-12403, 2024.
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R Fung, Hao Peng, and Heng Ji. Craft: Customizing Ilms by creating and retrieving from specialized toolsets. In International Conference Learning Representations (ICLR), 2024. 2, 3
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning:
Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.1283 2023. 3
Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint arXivv2402. 15506, 2024a.
Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal internet agents. arXiv preprint arXiv:2404.00999 09992,, 2024b.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. In International Conference on Machine Learning (ICML). pp. 61349-61384 2024a.
Sipeng Zheng, Yicheng Feng, Zongqing Lu, et al. Steve-eye: Equipping Ilm-based embodied agents with visual perception in open worlds. In International Conference Learning Representations (ICLR), 2024b. 3
Yaoyao Zhong, Mengshi Qi, Rui Wang, Yuhan Qiu, Yang Zhang, and Huadong Ma. Viotgpt: Learning to schedule vision tools towards intelligent video internet of things. arXiv preprint arXiv:2312.00401, 2023. 2
15

---

Published as a conference paper at ICLR 2025
A HUMAN VERIFICATION OF MM-TRAJ
A.1 DATA SYSTHESIS PIPELINE
We recruited persons with rich programming and Al experience to evaluate the tasks and trajectories generated by our method. Each evaluator is tasked with assessing 20 samples, which are randomly selected and mixed from both MM-Traj and filtered-out data.
The evaluation was conducted using 5-level rating scale: Poor, Fair, Good, Very Good, and Excellent, corresponding to numerical scores of 2, 4, 6, 8, and 10, respectively, with maximum score of 10. The results in Tab. 6 show that verified cases consistently outperform filtered-out cases in both task and trajectory evaluations. Verified tasks scored an average of 7.96, while filtered-out tasks averaged 6.30, indicating that verified tasks are more natural, coherent, and complex. For trajectories, verified cases scored 8.64 versus 6.24 for filtered-out cases, demonstrating better reasoning, code coherence, and feedback effectiveness. These results confirm that the verification process effectively filters out lower-quality data, ensuring the reliability of our data synthesis pipeline and the MM-Traj dataset.
MM-Tra Traj
Filtered out Data
Task Trajectory
Task
Trajectory
8.32
8.67
6.36
6.38
Table 6: Average score of the human verification.
The evaluation criteria for the generated tasks and trajectories are as follows. Task Evaluation Criteria
Naturalness: The degree to which the task appears natural and realistic.
Coherence: The logical consistency and smooth flow of the task.
The extent to which the task exhibits sufficient complexity, requiring the use of multiple tools for effective resolution.
Complexity:
Trajectory Evaluation Criteria
Reasoning:
The logical soundness and clarity of the agent's thought process.
The clarity, consistency, and structure of the code produced by the agent. Feedback Effectiveness: The agent's ability to effectively respond to and incorporate results from the tool executions.
Code Coherence:
The interface of the user study for data quality is shown in Fig. 6.
A.2 AGENT OUTPUT
We conducted a user study on agent outputs on the GTA benchmark. We recruited 20 participants, each evaluating 20 tasks with agent outputs, where the agent is with without fine-tuning. The agent outputs (w/ or w/o tuning) were shuffled for each task, and the participants were not informed about the source, ensuring preference of the two agent outputs, based on the accuracy, helpfulness, and relevance. We measured the percentages of results, shown in Tab. 7. Outputs from the tuned agent have significantly high preference, indicating its better performance in solving practical tasks. The interface of the user study for agent output is shown in Fig. 7.
unbiased assessment. The participants were asked to provide the
Agent w/o tuning better
Tie
Agent tuning is better 66%
21%
13%
Table 7: User study for agent outputs on the GTA benchmark.
16

---

Published as a conference paper at ICLR 2025
Images
15:91
Serrr Shonssate
@ informacce
Conversation
Assiatant:
<image- average displayed image? Attachment:
hewaallongggee///eeeeeeee wwwpgegeeeeeceteeeeeeeeeeeeeeeee png? png:
Poor
Good
Good
Excellent
Ascletant:
Thought To find the average price the Itema displayed in chart, need to extmet the prices from the image. The prices appear be labeled on top of each bar. will use the visualizer tool extract the relevant information from the image provided. Code: ""ir "py piico, _datt vieualizerqqueationn What are the prices disp this bar chart?", col.1666 ong") primtiorice data)
path- data/tonga baata/tongggent/chie httpaaappennccaaaaaaariiiireeeer
Rate:
Poor
Fair
Good
Very Good
Excellent
[OUTPUT STEP Obsarvation displayed chart follows: $29.19 2010: 0200:$20.00 $29.93 2011: $31.73 2014::
$34.03
Assistant:
Thoughht Now that the prices 2009: $29.99 2010 $29 93 1: S31 23- 2012: $30.3 $30.59 2013: S31 .0- 2014: $31 :88 2015: $34.0311 will sum these prices and then divide by the number of years (?) find the average, Code: "py Prices from the abservation prices 129.10, 0. 20 03,98 31 77. 30.59, 31, 31,38, 24.00 average. price sum(prices) len(prices) final.anseereee answerisverrge, price)
chart, caloulate average price over years, prices follows:
Rate:
Poor
Fair
13 Good
Very Good
Excellent
NEXT
Figure The interface of the user study for data quality.
17

---

Published as a conference paper at ICLR 2025
evaluate which model you hased trajectory response.
Question of 10
Menu
Product
Hamouugee
R20
Pince
236
Taok:
would katal?
Apeat Trujectery:
Agent B Trajectory:
sel's
toas ron for deeiity citaiio thus cnkuee codee noractas citi
meeett and
bas antosat Lesttar
soter delow antine
su'l tsant rile, to cnp ditennee op repe
Kakers priie mase and stoilie rec snst cad sead uoi any
to
caestl, our interis
epentin ciee treegeer Lome
freeeen ooer semenin fitter tallune aner ielare sart in's
thaggh raa oe cremm rigees sires
Naateen aetisss satts dittered
spptteds camec er tne cemmitao diseettt
den stse for de mes leee senee
dovering
wwwn svenall sown, by rep attimns op
New
tand) time caen late celited stant pnitors und
Brown naer fntal teloney ae apsirte telere prticct at cunnt tortiie aot rake neen priits
"ppesen "Jeest to Sur sil enee ancee ees ris sespcien exsatie tou
lighest dalkes
rpproech Fad the tw lighes priced daaes and cakolate Doxk tild cost Cide: by /f Menu data estasted Sues isage nets Jees (Prodkot. Beef. Thise: Paza, Hashagee Piiee (Phodact
$56.
9),
weinl masu wanzkoresn lern, key-harlki afPrid]] sousse-Thee J Os the lop two mod ioganiin horo bo lvo. Jemns (Prskat for dee wrisa 2ill
heme
Taak: Thak: wosted ordertde hightd priced dabes, bus mony dillms would con toteT
Groundtraak: [0]]
Bette
Figure The interface of the user study for agent outputs on the GTA benchmark.
18

---

Published as a conference paper at ICLR 2025
B MORE EXPERIMENTS
B. ABLATION
To improve the interpretability of our method, we add ablation experiments to show the contributions of different modalities in decision-making. Ablation results on the GTA dataset are shown in Tab. where removing the image modality reduces the performance by 40%, highlighting the importance of input images.
Method
AnsAce
ToolAce
CodeExec
T3 Agent w/o image
10.67
25.32 65.85
20.09
T3 Agent w/ image
52.56
80.49
Table 8: Average score of the human verification.
B.2 DATA NUMBER
We show the agent's 's performance the GTA benchmark the dataset size increases, in Tab. 9. With the increase data number, the agent achieves better performance, the memory consumption is constant, and the time consumption linear increases. Compared with the accuracy improvements, think that the consumption of memory and time is acceptable.
Data Number
6K
12K
20K
Accuracy
43.59%
48.08%
52.56%
Memory
214 GB
214 GB
214 GB
Training Time
276 mins
532 mins
946 mins
Table 9: Performance on the GTA benchmarks.
C TOOLS
We will show details of used tools in T3-Agent.
C.I
WEB SEARCH
The web search tool is actually another agent. It has three sub-tools: Searchinformation, Visit, and Webqa.
searchinformation. Given query to this tool, it performs Google search and outputs the title, abstract, and URL of multiple entries.
Visit. The input is the URL of an HTML page, and the output is the textual content of the HTML. page.
Webga. Given a question and search textual content, the tool outputs the answer.
C.2 IMAGE QUESTION ANSWERING
We the GPT-4o-mini model the image question answering tool. The input is an image and question, and the output is the answer.
C.3 FILE INSPECTOR
The input question and one multi-modal file. We use the Python package 'MarkdownConvertter that converts the given files into the markdown texts. Then, we feed the question and texts to the GPT-4o-mini model for the answer.
19

---

Published as a conference paper at ICLR 2025
C.4 OBJECT LOCALIZATION
We the OWL--VT model (Minderer al., 2022) for object localization. The input includes one image and a query, and the output is Python list of bounding boxes of the query.
C.5 IMAGE GENERATION
We the stable diffusion model for image generation (Rombach al., 2022). Given textual query, the tool produces an image that matches the query.
C.6 IMAGE EDITING
We use the InstructPix2Pix model for image editing (Brooks al., 2023). The inputs are an instruction and an image, and the output is an edited image to match the instruction.
C.7 FACE DETECTION
We use the DSFD model for face detection (Li et al., 2019). The input is an image, and the output is Python list of all bounding boxes of faces.
C.8 PYTHON PACKAGE
We allow the agent to use the following Python package in code writing: "requests", "'iifilee, "os", "pandas", "numpy", "sympy", "json", "bs4", "pubchempy". "xml", "yahoo_fiiie financci""" "Bio", "sklearn", "scipy". "pydub", "io", "PIL", "chess", "PyPDF2", "pptx", "torch", "datetime", "esv", "fractions", "matplotlib", "pickle", "cv2", through which the agent is more flexible in writing code.
D PROMPT
D.1 PROMPT FOR QUERY GENERATION
The prompt for query generation is shown in Fig. 8.
You tasked with generating queries that will prompt agent call various tools (only the tool listed our toolset), including internet search capabilities, to solve real-world, practical problems. The problems should natural, varied, and challenging, requiring the agent reason across different domains. Ensure that the problems span range of practical scenarios.
Our toolset: TOOL_SET
now provide examples, along with the tools. Examples of user queries: IN PLCONTEXT EXAMPLES
Please output the Queries in json format. Make sure that the queries share similar style of the in-context examples. The output template is:
"query": "What is the weather today?". <The user query to the agent.> "tools": ["tooll", "tool2"....J. <A list consisting of the tool names related to the query.>
Figure 8: Prompt for query generation.
20

---

Published as a conference paper at ICLR 2025
D.2 PROMPT FOR FILE GENERATION
The prompt for file content generation is shown in Fig. 9 and Fig. 10, and the prompt for file code generation is shown in Fig. and Fig. 12.
You are reasoner that can restore a query_ scene between human and an agent. Human gives a complex query and several images to the agent, and then the agent answers the query by searching the Internet and applying tools to the images with step-by-step reasoning. Now, you will be given the query with suggested tools. suggest you analyze the needed information to solve the query, and divide information into two groups: searching from Internet extracting from images using tools. Based on the information from the images, you need to further infer the content of these images, through which the agent could correctly solve query.
Our toolset: TOOL_SET Output MUST use the following json template.
"information": <Needed information to solve the query. For the query including creating/generating images, the information should NOT be the description of the describe image.> "information from the Internet": <Information from the Internet inferences based on the given query and suggested tools. Determine which information suitable to obtained from the Internet. Or say no information is required from the Internet.>
"information from images": <IIformation extracted from given images based on the suggested tools to solve the query. It should be several sentences, including information extracted from the images using tools. Determine which information suitable to he obtained from the images, and using which tools. Do not generate image_ cone__cttentttt for the query including generating/creating image. Or say no information is required from the images.>
"file":
gamage_numbert <set an int number, the number is depended on needed information from images>, "image_content":
<The image content should he natural language, descrihe the content of the first image relevant to query. The content should concrete, such as concrete number, concrete name. The content should match the query and the above images.> <if you think the query needs more than image, please output image content 'image_2's
"image_1":
Figure 9: System prompt for the file content generation.
Now given the query: QUERY, firstly analyze the needed information to solve the query and divide the information into two groups: scarching from the Internet or extracting from images using tools. Then for information from images, imagine possible answers of each information (it should be concrcte answers instead descriptions). Finally, output the json the inferenced information and content of images.
Figure 10: User prompt for the file content generation.
D.3 PROMPT FOR QUERY-FILE FILTER
The prompt for the query-file filter is shown in Fig. 13 and Fig. 14.
21

---

Published as a conference paper at ICLR 2025
You helpful assistant and can generate <file type placeholder> file by writing Python code. You will be given a description of the content of the file. You need to firstly largely extend the content, and then write Python code generate <tile type placeholder> file. GUARANTEE that the provided content is the file. The output Python code MUST the following template.
## extention start
Extened content: <here is the extented content>
## extention end
## code start
<here is the Python code to generate a <file type placeholder> file>
## code end
Figure 11: User prompt for the file content generation.
Now, given the following content: <file content>, first largely extend content, and output code to generate <file type placcholder> file, where the file name is <file name> and the file will be saved in <save path>.
Figure 12: User prompt for the file content generation.
D.4 PROMPT FOR TRAJECTORY FILTER
The prompt for the trajectory filter is shown in Fig. 15 and Fig. 16.
D.5 PROMPT FOR AGENTS
The system prompt for the T3-Agent shown in Fig. 17.
E MORE VISUALIZATION
provide more visualization of our T3-Agent on the GTA and GAIA benchmarks, as shown in Figs. 18 to 24.
22

---

"updated :uuuruuuer_rrrrr <lf you judge the correctness as 'no", please rewrite the query to make more Published as a conference paper at ICLR 2025
You are a helpful assistant that is given a query and several images. You need to check whether the images relevant the query. The query and images used evaluate the perception ability, reasoning ability, and information search ability of an AT agent. The agent solves the query by searching information from the Web and extracting information from the images. some cases, based on the given images, the agent could not solve the query, even it searches for information from the Web (e.g., some specific knowledge). You need to pick up these bad cases.
The agent can call the following tools to solve the query. TOOT_SET.
Thus, images should follow these requirements.
1. Relevance: The depicted scenarios objects in images should be relevant to the query. The images should contain scenarios or objects that are mentioned in the images.
Usefulness: The image should contain necessary information address the query, such as specific details that cannot be obtained from the Web.
Some queries require the agent scarch for knowledge from the Web and combine the information in the image to solve the queries. Thus, in some cases, the images do not contain all the information to solve query, but missed information could searched on the Web. These cases should regarded correct cases.
The output MUST use the following json template to check the images. "information for_ query": <Required information to solve the query.>,
in in_image": <Useful information that can be extracted from images to solve
the query>.
"missed information in images": <Missed information that is necessary to solve the query but does not exist in the images.>,
"missed information web search": <You need justify whether the missed information could be searched from the Weh, using your rich experience in surfing the Internet.>,
"missed information_ofee soririoddd_taidddd <You need to justify whether the missed information could he obtained via computing or reasoning based on information extracted from the images or searched from the Web.>,
"thought": <Now, you need determine whether the images can solve the query. If the missed information could be searched from the Web or obtained based on existing information, the images solve the query. If not, the images cannot solve the query.>,
"correct": <According the above reasoning, you consider the images reasonable for the query to be solved by the tools, set the value to 'yes', otherwise set the value to 'no" .>
relevant to the given images. If you judge the correctness 'yes', please output "no revision is needed."
Figure 13: System prompt for the query-file verification.
Following images, the query: <query>, inference whether the images solve the query based the perception ability, reasoning ability, and information search ability of an Al agent.
Figure 14: User prompt for the query-file verification.
23

---

Published as a conference paper at ICLR 2025
As a data quality evaluator that need to determine whether query-solving trajectory between a human and agent is correct. The human gives images and query, and the agent calls tools solve the query. The trajectory of query-solving contains a task query, thoughts and codes generated by the agent tools (Python functions), tool-response of each step, and final answer. You must assess the alignment between the task query, corresponding tool usage (generated thoughts and codes from agent), the execution results (tool-response). Your goal to ensure used tools, arguments to the tools, and summarized answers in the trajectory accurately reflect the human's ' intentions. Our toolset: TOOL__ETT
The query-solving trajectory is incorrect if: 1. The tool usage does not align with the query's 's objective and the context, or there is useless or unreasonable tool usage. In addition, the agent does not use tools and solve the query by itself. 2. The input arguments the tools appear incorrect unreasonable. The final answers or intermediate results summarized from the observation appear incorrect or unreasonable. 4. The final answer is relevant the task query the final answer incorrect. The trajectory (such as tool-usage and observation) conflicts or is not consistent with the image content.
Figure 15: System prompt for the trajectory verification.
Now, given used images and corresponding information, determine whether the trajectory is correct or
User Query: QUERY
2. Image Content: IMAGE CONTENT
Trajectory, including generated thought and code from the agent, intermediate results of using
tools: TRAJ
Execution Results: RESULT
Output MUST use the following json template to determine whether the query-solving trajectory is correct or not.
"thought": "Concisely describe your reasoning here", "correct": "yes" or "no"
Figure 16: User prompt for the trajectory verification.
24

---

Published as a conference paper at ICLR 2025
You expert assistant who solve any task using code blobs. You will be given task solve as best you can. To do so, you have been given access to list of tools: these tools are basically Python functions which you series of steps, in a cycle of "Thought:, 'Cade:', and 'Observation:' sequences.
call with code. To solve the task, you must plan forward proceed in
At each step, in the "Thought:" sequence, you should first explain your reasoning towards solving the task and the tools that you want to use. Then in the 'Code:" sequence, you should write the code simple Python.
code sequence must end with the "<end action> sequence. During each intermediate step, you can use 'print()' save whatever important information you will then need. NOT generate code which does not call 'print('' because you will lose this information. You can assume all tools must have return that can be printed. These print outputs will then appear in the 'Ohservation:" field, which will be available as input for the next step. You will save intermediate file outputs to folder by the relative path 'cacch". In the end, you have to return final answer using the "final answer' tool.
Here few examples using notional tools: IN CONTEXT EXAMPLES
The above example was using notional tools that might not exist for you. You only have access those tools: TOOI_SET You also can perform computations in the Python code that you generate.
Here are the rules you should always follow to solve your task:
1. Always provide a "Thought:" sequence, and 'Code: py" sequence ending with '<end accioon> sequence, else you will fail.
2. Use only variables that you have defined!
Always use the right arguments for the tools. DO NOT pass arguments as a dict as in 'answer ask search agent( 'uery": "What the place where James Bond lives?"), but the arguments directly as in 'answer = ask search agent(query= chain many sequential tool calls in the same code block, especially when the output format is unpredictable. For instance, a call to search has an unpredictable return format, so do have another tool call that depends its output in the same block: rather output results with print() to use them in the next block.
is the place where James Bond lives?")'.
4. Take
Call tool only when needed, and never re-do tool call that you previously did with the exact same parameters.
Don't name any new variable with the same name as tool: for instance don't name a variable
Never create any notional variables our code, as having these your logs might derail you from the true variables.
You can use imports in your code, but only from the following list of modules: sauthorized imports
The state persists between code executions: if in one step you've 'u've created variables imported modules, these will all persist.
10. Don't give up! You're in charge of solving the task, not providing directions solve
Now Begin! you solve the task correctly, you will receive reward of $1,000,000.
Figure 17: System prompt for the agent.
25

---

Published as a conference paper at ICLR 2025
RANK
COMPANY
APPLE
2
TESLA
ditszorr
4
Alphabet
Who is :uo
the top-ranked company?
Thought: To find the CEO of the top-ranked company, need to analyze the provided image. task involves identifying which company ranked and then determining its CEO. will use the image_qp toal to extract textual information from the image.
Code:
mmage_path- "data/gta, latpa/gta_daasset/iin bataset/image/lmaage _399__pp_pp
is the name the top-ranked company?", image_ path-image _paah)
company rank image qulqqestiion=
print("Tpp-ranked company.", company_mapn
Observation: Top-ranked company: The Top-ranked company is Apple.
Thought: Now
know top ranked company Apple, need to who its CEO is. Since this information may not be directly visible in the image,
will 'ask _ock__eerch_____ search_agent" tool search for current CEO Apple.
Code:
ceo of apple =aak__saa search agent(query= "Who the CEO of Apple?")
print("CEO of Apple?", CEO of__o__Aaa of_lp
Observation: Search outcome (short version): Tim Cook the current CEO of Apple Inc., having assumed the
August 2011. Under leadership. Apple has seen significant increase in revenue and profit, and a strategic shift towards
inclusive company culture.
Thought:
have gathered information about Tim Cook being the current CEO of Apple. Now, will summarize this information to provide concise final answer.
Code:
final "aaanssswe""" "im Cook the current CEO Apple.")
Figure 18: Case study of the T3-Agent.
26

---

Published as a conference paper at ICLR 2025
Query have 10 lights in my home total, how many lights are there in the other rooms?
'image qa' tool analyze image and information about the lights present. This should provide clear count of how many lights visible the picture.
Thought:
Code:
lights__cuunt count image calquestion="How many lights are visible in this image?", image__aan _upppath"'h patppaadtta/gtt "data/gta dataset/image/lmage 392.jpg""
print(lights, count)
Observation: There are two lights visible in the image.
determined that to find out how many lights are in the other rooms, will subtract this count it's given that there lights
Thought:
lights visible in the image. Now,
from the total number lights home.
Code:
total lights 10
lights in othertheerrro rooms total lights answer(lights rooms))
Figure 19: Case study of the T3-Agent.
27

---

Published as a conference paper at ICLR 2025
NVIDIA
Query: Who is the founder of this company
Thought: To find who the founder the company depicted the image, will use the 'image qa tool to analyze the image and extract relevant information about the company. This should help identify key details such its name and any associated individuals. Code:
company Info image qalquestion="What "What is the name the company shown this image?", image pimaghhaaaah________ 3m0. print(company Info)
Observation: The company shown in the image is NVIDIA.
Thought: Now that know the company NVIDIA,
the 'web search'
tool find out who founded This should provide me with name of the founder.
founder www_wweeweeeeeeeeeeeeeee search(query- the founder of NVIDIA?") print(founder)
Observation: Search outcome (short version]: NVIDIA was founded on April 5, 1993, by Jensen Huang, Chris Malacchowsky, and Curtis Priem.
Thought:
have gathered information about the founders of NVIDIA. The company founded by Jensen Huang, Chris Malachowsky, and Curtis Priem. Now, will compile this information into final answer.
Code:
founders ["Jensen Huang", "Chris Malachowsky". "Curtis Priem"]
final, answer((" Founders NVIDIA f. Join(founders))"
Figure 20: Case study of the T3-Agent.
28

---

Published as a conference paper at ICLR 2025
Congeryy
Inn
Roal Taki
Chisa
Forme triees
Pand Daa
23
Aulrda
lele
Shows
Sanvon
don't flats, Each should be different in style and color. What specific items can buy? Please with short Thought: need to identify pair pants and shoes that are different style and color within budget. Since have $70, I'll look for options under the given price range. and extract relevant information about clothing items, focusing on styles and colors.
Query only $70 buy pants
phrase.
image_ qai tool analyze image provided
Code:
moth_items image splquestion="Please identiff specific pants and shoes from the image based image_ "aa___ath""mma__22""" print[cloth_ jitems)
on
their styles
and colors.",
Observation:
**Pants:** Jeans ($30)-Chinna) Chinas ($35)- Formal Trousers ($40)
""Shoes:** Sneakers ($40)- Ballet Flats (530)- Ankle Boots (560)
Thought: Isee that table lists specific clothing items
their prices.
However, it doesn't provide visual information about styles or colors of pants
and shoes directly from the image provided. Since I don't have clear visual specific items,
reference
the options available
on the listed prices:- ** Jeans ($30), Chinos ($35), Formal Trousers ($40), Ballet ($30), Ankle (560)Given
($40)- *Shhoes***
different items-one of pants and one pair of withhut buying flats. I'lll propose combination where buy one type jeans and sneakers, ensuring they distinct in style and color.
my budget $70,
Code:
item1
price1 =0
item2 "Sneakers"
price2
total cost price1 price2
brinti"Tota cost:", total_cost cost)
Observation: Total cost: 70
Thought:
confirmed the of buying jeans sneakers
$70, which matches budget. Now, will finalize this purchase Anlyll the specific
with short phrase indicate what buy within the given constraints
Code:
final_ Inal_answer("M JeJans and Sneakers")
Figure 21: Case study of the T3-Agent.
29

---

Published as a conference paper at ICLR 2025
Query gender
Bulgarian census
who have completed tertiary education. Subtract the smaller number from the larger number, then return the difference in thousands of women. So if were 30.1 thousand more men, you'd give 30.1.
Thought: will first need to gather the data regarding the gender split from those
completed tertiary education. Once I have these numbers, can subtract the smaller number from the larger the wwbb search" tool to find the specific data related to the gender split from the 2011 census those with tertiary education.
Bulgarian
number and return result thousands of in'n'l
Code:
gender data web search(" split from the 2011 Bulgarian census wee__sercc(""eee
about those who have completed tertiary education") print[gender data)
Observation: Search outcome: the 2011 Bulgarian Census, 22.3% of women 16.7% men completed tertiary education.
Thought: have retrieved the relevant data from the 2011 Bulgarian census. According the information, there
thousand females
(representing 22.3% of women] 16.7% of men) who had completed tertiary education. Now, will calculate
thousand males (representing
the difference between these two numbers in thousands of women subtracting the smaller number from the larger number: Larger number
(females); 791.8)n- Smaller number (males): 556.9. The difference calculated follows: text(Difference) 791.8 556.9 perform
calculation using Python code.
Code:
larger_number
smaller smaller_number number = $56.9
difference larger_ nwwberumbeeeeee smaller number
print[ "Difference thousands womee:", difference)
Observation: Difference in thousands women: 234.9
Thought: difference between numbers of women men completed tertlary education the 2011 Bulgarian census approximately thousand. Now, 'final answer' tool.
final
using the
Code: final, answer("TT difference in thousands of from those tertiary education the 2011 Bulgarian census is approximately 234.9.)
Figure 22: Case study of the T3-Agent.
30

---

Published as a conference paper at ICLR 2025
The longest-lived vertebrate is named after an island. According to Wikipedia as January 1, 2021, what that island, to the nearest thousand?
Query
population
Thought: To solve this the longest-live sisstt-iiedd vertebrate named. Then, will search the estimated population of that island referenced the question, I'll use the 'web_see search" tool to gather this information.
find the
the island after which
January 2020. Since Wikipedia
longest_ livellied__ Island = web seabbb_earchqurr="TTTaaa The name of the island after which the longest-lived vertebrate named")
"intl"I name:", longest lived island]
population
web search(query-f""he estimated population
(longest lived Island) as January 2020.")
print EEstmated population (to nearest thousand);", population_ _uerrr)_)
Observation: Search outcome: The longest-lived vertebrate the Greenland shark, named after the island of Greenland.
Thought: From the search results, have found that the longest- sigest-liveed vertebrate is Greenland shark. estimated population Greenland as January 1, 2020, was not explicitly mentioned in the provided Information. However, since it's related to specific island, can assume this refers the population of Greenland itself. As there no direct data the population Greenland from 2020, will need to use general knowledge or perform
another search for an estimate. will ask the 'web search' again for approximate population of Greenland
the
thousand.
Code:
population aereoonngeeeeedneeeee web searclquery= "What estimated population of Greenland as of January 1, 2020?")
printl ''Estimated population
Greenland Ito nearest thousand);",
population_ greenland)
Observation: Search outcome: Greenland's estimated population
January 1, 2020, was 56,081.
Thought: The estimated population Greenland January 2020, reported approximately 56,081.
use the 'iial, answer' tool to provide concise answer regarding both the longest-lived
(Greenland shark)
associated island's
population.
Code:
population info "The longest-lived vertebrate is the Greenland shark, named after the island of Greenland, with an estimated population of approximately 56,000 2020." final_ answer(population_iiig
Figure 23: Case study of the T3-Agent.
31

---

Published as a conference paper at ICLR 2025
Query: : The Metropolitan Museum of Art has portrait in its collection with an accession number 29, 19.100.5. consecrators co conseeccaatrss
portrait's subject as bishop, what is the name of the one who never became pope?
Thought: To find the consecrators and co-consecrators of the portrait's subject
bishop, specifically looking for those who became pope, will need search for information about this particular portrait the Metropolitan Museum of Art collection. This may involve checking online databases museum
(n'ni'll 'web search tool to provide detalls on the portrait with accession number
asking
29.100.5 the Metropolitan Museum of Art.
Code:
portrait Info web search("Detalls about the portrait with accession number 29. 29.100.5 the Metropolitan Museum of Art") print[portrait__ _inftttiooo____
Observation: Search outcome:- Title: Cardinal Fernando Niño de
Greco (Domenikos Theotckopoulos) Date: ca. 1600 Medium: Oil on canvas - Location:
Artist:
Metropolitan Museum of Art, Accession Number: 29.100.5 search results indicate
Thought:
portrait
Cardinal Fernando
Niño Guevara, painted by El Greco around 1600. now need to out who were the consecrators and co-consecrators of this cardinal subject bishop, specifically identifying those who became pope. will ask the 'web searchearc tool again for details on the consecrators and co- consecrators of Cardinal Fernando Niño de Guevara, focusing particularly those who didd not become pope.
Code:
consecrators info web Dearch("Consecraten and co-consecrators
Cardinal Fernando Niño Guevara from the Metropolitan Museum Art collection, specifically those who never became pope.") orint(consecrators info)
Observation: Search outcome (short version):- Cardinal Fernando Niño de was consecrated by Pope Clement Viii, with
Camillo Borghese (later Pope Paul V) Alfonso Visconti.- Alfonso Visconti was not No direct connection the Metropolitan Museum of Art was found for Alfonsa Visconti.
Thought: Based on the search results, have identified that Cardinal Fernando consecrated Pope Clement VIIl Camillo Borghese and Alfonso Visconti. Among these individuals, Alfonso having
Niño de
Visconti noted
pope. will compile information into a final answer format to present clearly.
Code:
final, answer("TTi T"The co-consecrator Cardinal Fernando Niño de Guevara who never became pope is Alfonso Visconti." ")
Figure 24: Case study of the T3-Agent.
32


Reviews:
[{"review_id": "nEv2GUxa5U", "reviewer": "Reviewer_4Mmx", "summary": "This paper proposes a method for multi-modal agent tuning for tool usage and presents a dataset designed to train an agent for tool usage. The authors claim that their T3-Agent achieves significant improvements.", "questions": "Please address the concerns raised in the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The idea of using an LLM to separately generate queries, files, and trajectories, followed by a query-file verifier and trajectory verifier is neat.\n2) The paper addresses the problem of using the appropriate array of tools corresponding to the information relevant to a query well. \n3) The experiments are thorough.", "weaknesses": "1) Verifying the output of an LLM by the LLM itself does not seem accurate. I am skeptical about the quality of the generated MM-Traj dataset. \n2) You need quantitative verification for the dataset. (It is not clear whether a user study involving a few people on 100 data points out of 15K would provide sufficient confidence.)\n3) Experimental results on the GTA benchmark are more promising than those on the GAIA benchmark. However, the overall performance of T3-agent is not superior to that of the other agents in comparison. Specifically, on the GAIA benchmark, the HF agent with GPT-4o performs twice as well as the T3-Agent.\n4) If the T3-Agent’s performance were clearly superior, I would be more optimistic about the generated dataset. However, the results seem to support my doubts about the dataset.\n\nMinor comment: In Tables 2 and 3, the best results should be highlighted in bold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbzGOiaBLB", "reviewer": "Reviewer_uszN", "summary": "This paper presents a novel approach for multi-modal agent tuning, aimed at enhancing agent performance in complex environments through better utilization of multiple data modalities. The authors propose a tuning framework designed to leverage cross-modal data (e.g., visual and text info) to improve agent task performance, with specific emphasis on tool usage within the agent's capabilities. Their T3 agent is multi-modal agent that can efficiently use tools to solve practical tasks by tuning a VLM as the controller. Evaluations over the various dataset show the significant improvement using their agent with closed as well as open source model. Additionally, they curated dataset using multimodal info having trajectory of various length for broader study. In this work focus is on the correct tool selection and code is given more importance than the widely used JSON schema. In summary, they generate data, tune the VLM and create dataset followed by leverage of tool agent to make use of tool in a better way.", "questions": "Q: Could you please elaborate on how not using the final answer A aligns with your goal? Specifically, how does this choice benefit your approach to enhancing tool-usage capability?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Innovation in Multi-Modal Interaction: The approach shows potential in pushing the boundaries of how agents interact with cross-modal data sources. By focusing on practical applications of tool usage, this work could offer useful insights into building agents that understand and respond to complex queries across various media.\n2. Comprehensive Methodology: The paper describes a well-structured experimental setup and provides a clear description of the multi-modal tuning process. This includes thoughtful considerations on data processing, model architecture, and task-specific tuning steps, making it easy to follow.\n3. Evaluation Metrics: The authors employ a diverse set of metrics to evaluate agent performance. This choice not only validates the model’s accuracy in the tasks but also emphasizes the practical utility of the proposed framework in real-world applications.", "weaknesses": "1. Interpretability: While the approach demonstrates performance gains, the interpretability of results remains limited. Additional analyses, such as ablation studies or attention maps, would be beneficial to understand how each modality contributes to the decision-making process. For example paper first generate queries first without files before relevant queries, what is the impact if we don't do that, or this is based on some past work/observations?\n\n2. Scalability: The paper does not thoroughly address the scalability of the proposed method, particularly as the number of modalities or the dataset size increases. It would be beneficial to test how the method's performance and computational requirements scale with additional modalities or larger datasets. For example, experiments that measure latency, memory usage, and accuracy as more data is introduced could illustrate the framework's robustness and its viability in resource-constrained or high-throughput environments.\n\n3. User Study: To evaluate the practical usability of the framework, a small user study or qualitative feedback from users would provide valuable insights into the query handling experience. Specifically, gathering feedback on aspects like ease of use, perceived accuracy, responsiveness to complex queries, and the intuitiveness of the tool-usage process could highlight areas for refinement in real-world settings.\n\nMinor hints:\n1. sec5.6 typo.. \"wen based\"--> web based.\n2. sec 3.4- Author(s) mention details can be found in but missed cross-referencing it.\n3. cross reference missing at end of sec 3.4", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ScGCrs01oH", "reviewer": "Reviewer_zCvz", "summary": "This paper introduces a new approach for improving tool usage in multi-modal agents by fine-tuning a vision-language model (VLM) controller with synthesized tool-usage data. To overcome the limitations of traditional LLM-driven agents—such as reliance on prompt engineering and limited reasoning for tool usage across modalities—the authors create a three-step data synthesis pipeline. First, *Query Generation* uses GPT-4o mini to generate diverse, tool-aware prompts. Next, *File Generation* retrieves images from similar datasets and creates other files programmatically. Finally, *Trajectory Generation* employs a zero-shot agent using GPT-4o mini to solve tasks, capturing thought processes, code, and observations for each step. Quality is controlled through query-file and trajectory verifiers, also based on GPT-4o mini, producing a dataset called MM-Traj. The resulting agent, T3-Agent, uses the ReAct framework and the MiniCPM-V model trained on MM-Traj, enabling versatile tool usage across categories like web search, visual perception, image editing, and file handling. Benchmarks on GTA and GAIA demonstrate the T3-Agent’s significant improvements in tool usage over both untrained VLMs and other state-of-the-art LLM-driven agents.", "questions": "I lean towards acceptance because the paper introduces a promising approach to enhancing the reasoning capabilities of multi-modal agents. The proposed data synthesis pipeline and the resulting MM-Traj dataset are significant contributions to the field, potentially advancing the state of multi-modal learning. However, the paper lacks a discussion on the T3-Agent's robust programming capabilities and does not explore how these might be improved. I would like the authors to comment on this aspect.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The data synthesis pipeline introduces a scalable, automated approach to generating diverse, complex multi-modal data for tool usage scenarios.\n- Verification mechanisms embedded within the pipeline enhance data quality, resulting in a robust, comprehensive dataset.\n- With training on the MM-Traj dataset, the T3-Agent demonstrates significant performance gains, surpassing agents built on closed-source models like GPT-4 in certain benchmarks.\n- Ablation studies underscore the critical role of data verification in achieving top performance.\n- The paper includes detailed visualizations of the T3-Agent’s reasoning process.", "weaknesses": "- The T3-Agent exhibits a gap in programming capabilities, which leads to lower accuracy in predicted answers compared to GPT-4o. \n- While the paper acknowledges the T3-Agent’s limited programming capabilities, it does not suggest potential improvements or outline future directions to strengthen this aspect.\n- The reliance on GPT-4o mini throughout the pipeline raises questions about biases and limitations from this closed-source model. Exploring alternative methods or open-source models could enhance transparency and address these limitations.\n- What is $p_i$ in Equation 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lKbzXFBqf9", "reviewer": "Reviewer_snR1", "summary": "This work introduces a multi-modal tool-usage data generation pipeline designed to finetune vision-language models (VLMs) for tasks requiring tool-usage reasoning. The pipeline consists of three primary steps. First, a large language model (LLM) is prompted to generate query tasks. Next, relevant images or files are retrieved or generated based on the specified query tasks. Finally, ReAct agents are employed to generate trajectories that address the query task problem, followed by an additional LLM prompt to verify the generated data.\n\nThis study also introduces the MM-Traj dataset generated through the proposed scheme and uses it to finetune the MiniCPM-V model to create the T3-agent. The T3-agent's effectiveness is subsequently assessed on the GTA and GAIA benchmarks, showcasing a 20% improvement in performance over untrained VLMs and achieving results comparable to other baseline agents, such as the Lego Agent, Warm-up Act Agent, and HF Agent.", "questions": "1. In Figure 3, the sum of trajectories—214 + 14,273 + 8,740 + 2,520 + 1,242 + 697 + 199 + 202—totals 28,087, which exceeds the stated 20k tasks in the abstract. In addition, the paper mentions that only 15k files remain after passing the query-file and trajectory verifiers. What is the final size of the generated dataset?.\n2. Why does GPT-4o mini outperform GPT-4o in Table 2, specifically in the row with HF Agents on AnsACC and CodeExec, given that GPT-4o is expected to be more powerful than GPT-4o mini?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed data generation pipeline first generates queries independently of any specific files, followed by producing relevant files to align with these queries. This approach allows the pipeline to create more diverse and expressive tasks, unrestricted by file format or quantity limitations.\n2. This work introduces the novel MM-Traj dataset, containing 20k diverse tool-usage data points, supported by a human study to demonstrate dataset quality. \n3. This work also performs an in-depth statistical analysis of the dataset and shows that MM-Traj offers broad coverage across various data modalities and knowledge requirements, as shown in Figure 3.", "weaknesses": "1. In Section 3.3, this work states, “for other needed files, we prompt GPT-4o mini to extend the file content and generate Python code to produce the files.” However, the methodology for file generation remains unclear. For example, if a file is required to be an MP4 video of Taylor Swift’s latest album, it’s uncertain how this content could be generated through Python code alone. Furthermore, if GPT-4o mini generates the Python code to produce such files, it raises concerns about data quality and how the model ensures that the generated content is not hallucinated.\n2. While including a human study to assess dataset quality is commendable, having only five experienced raters for a subset of the data may be too limited, potentially introducing biases based on individual preferences. Gathering feedback from a **larger pool of participants**, even with fewer data points per person, could strengthen claims about the dataset's effectiveness. Additionally, comparing MM-Traj to filtered-out data may not yield meaningful insight. Instead, comparisons with other established tool-usage datasets would likely provide more meaningful insights.\n3. The evaluation results in Table 3 reveal mixed outcomes, with the T3-agent performing significantly worse than other methods, such as HF Agent and Sibyl Agent, on the GAIA benchmark. What could lead to this performance discrepancy on the GAIA benchmark?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)