Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
Authors: Pengxiang Li
Zhi Gao
Bofei Zhang
Yapeng Mi
Xiaojian Ma
Chenrui Shi
Tao Yuan
Yuwei Wu
Yunde Jia
Song-Chun Zhu
Qing Li
Venue: NeurIPS 2025 poster ()
Keywords: Tool usage
Multimodal agents
Primary area: applications

Abstract: Multimodal agents, which integrate a controller (e.g., a vision language model) with external tools, have demonstrated remarkable capabilities in tackling complex multimodal tasks.
Existing approaches for training these agents, both supervised fine-tuning and reinforcement learning, depend on extensive human-annotated task-answer pairs and tool trajectories.
However, for complex multimodal tasks, such annotations are prohibitively expensive or impractical to obtain.
In this paper, we propose an iterative tool usage exploration method for multimodal agents without any pre-collected data, namely SPORT, via step-wise preference optimization to refine the trajectories of tool usage. Our method enables multimodal agents to autonomously discover effective tool usage strategies through self-exploration and optimization, eliminating the bottleneck of human annotation.
SPORT has four iterative components: task synthesis, step sampling, step verification, and preference tuning.
We first synthesize multimodal tasks using language models. 
Then, we introduce a novel trajectory exploration scheme, where step sampling and step verification are executed alternately to solve synthesized tasks.
In step sampling, the agent tries different tools and obtains corresponding results. 
In step verification, we employ a verifier to provide AI feedback to construct step-wise preference data. 
The data is subsequently used to update the controller for tool usage through preference tuning, producing a SPORT agent.
By interacting with real environments, the SPORT agent gradually evolves into a more refined and capable system.
Evaluation in the GTA and GAIA benchmarks shows that the SPORT agent achieves 6.41% and  3.64% improvements, underscoring the generalization and effectiveness introduced by our method.

Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
Pengxiang Li Zhi Gao 1.2,3,+ Bofei Zhang² Yapeng Mi Xiaojian Ma²
Chenrui Shi Tao Yuan² Yuwei W Yunde Ji Song-Chun Zhu Qing Li
"Beijing Key Laboratory of Intelligent Information Technology.
School of Computer Science & Technology. Beijing Institute of Technology
State Key Laboratory of General Artificial Intelligence, BIGAI
"State Key Laboratory of General Artificial Intelligence, Peking University "Harbin Institute of Technology
GGanggong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University Department of Automation, Tsinghua University
httpss//sport-agentsss.. github..o
Abstract
Multimodal agents, which integrate controller (e.g., vision language model) with external tools, have demonstrated remarkable capabilities in tackling complex multimodal tasks. Existing approaches for training these agents, both supervised fine-tuning and reinforcement learning, depend on extensive human- auman--nnotated task- answer pairs and tool trajectories. However, for complex multimodal tasks, such annotations are prohibitively expensive or impractical obtain. In this paper, we propose iterative tool usage exploration method for multimodal agents without any pre-collected data, namely SPORT, via step-wise preference optimization to refine the trajectories of tool usage. Our method enables multimodal agents autonomously discover effective tool usage strategies through self-exploration and optimization, eliminating the bottleneck of human annotation. SPORT has four iterative components: task synthesis, step sampling, step verification, and preference tuning. We first synthesize multimodal tasks using language models. Then, we introduce a novel trajectory exploration scheme, where step sampling and step verification are executed alternately to solve synthesized tasks. In step sampling, the agent tries different tools and obtains corresponding results. In step verification, we employ a verifier to provide Al feedback to construct step- wise preference data. The data is subsequently used to update the controller for tool usage through preference tuning, producing a SPORT agent. By interacting with real environments, the SPORT agent gradually evolves into more refined and capable system. Evaluation the GTA and GAIA benchmarks shows that the SPORT agent achieves 6.41% and 3.64% improvements, underscoring the generalization and effectiveness introduced by our method.
Introduction
Leveraging large language models (LLMs) or vision-language models (VLMs) as controllers to call external tools (e.g., web search, visual reasoning, file understanding. and object localization) has become promising direction in building multimodal agents [48] 19 14, 49 30, achieving impressive
performance for complex tasks [17 37 [3]. To enhance the planning and reasoning abilities of agents, existing studies focus on collecting tool usage trajectories to fine-tune the controller of an agent [21 18), using human annotation or distillation from closed-source APIs. However, collecting high-quality tool usage data is inttrnintensive and high-cost, and such pre-collected data may lead to biased distributions inconsistent with the target environments (such as task distributions and available tools), causing inferior generalization.
"Equal contribution.
E Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generation
Step Sampling
Step Verification
Proferance Tuning
Thaugg
Code
moogle
Task Seed
Model
Controller
Tool
Contrela
Image
Thought Coode
RovllS
lhought Codes
Figure 1: Pipeline of the proposed SPORT method, including four iterative components: task generation, step sampling, step verification, and preference tuning.
In this paper, we study whether multimodal agents can improve their tool usage capability via self- exploration without any pre-collection. We are inspired by existing research in LLMs and VLMs, which has shown impressive performance self-instruction [53 34], self-verification [40] [9], and self-learning [T0 25.. Based on the above observation, expect that agents automatically generates tasks, searches for useful tools in solving synthetic tasks, evaluates the exploration by itself, and updates controllers using the exploration process. In this case, agents will improve the generalization of tool usage by interacting with environments.
To achieve this goal, we must address two key challenges in such tool usage exploration for complex multimodal tasks. (1) Lack of off-the-shelf tasks and annotation. There is no off aff-tff-shelf dataset with ground truth annotations (answers and trajectories) for solving multimodal tasks. These tasks are usually domain-specific [41 making it non-trivial to verify whether the outcome is correct. Solving these tasks requires long trajectory to call diverse tools, and thus, it is challenging to produce an exactly correct trajectory and compare the quality among different trajectories. (2) Low sampling efficiency and high cost. The exploration process requires executing the sampled tools (e.g., large language models, web search, and image generation), which results in both high monetary and computational costs, making it challenging scale up.
To solve the above challenges, we propose SPORT, an iterative tool usage exploration method via step-wise preference optimization to refine trajectories of multimodal agents, shown in Figure [ SPORT operates through four iterative components: task synthesis, step sampling, step verification, and preference tuning. Firstly, we generate queries and multimodal files for task synthesis based on provided task seeds. Secondly, we introduce a new search scheme that samples step-level candidate actions (including the thoughts and codes) call tools. Thirdly, we employ multimodal verifier that, given the task context, step actions with exccution results, provides Al-generated feedback to estimate step-level preferences. Finally, we perform step-wise preference tuning to refine the controller and obtain the SPORT agent, which is then used to guide tool sampling in the next itcration.
SPORT enables agents to autonomously generate tasks and explore tool usage trajectories, removing reliance on pre-collected datasets. Step-level verification is easier than trajectory- evaluation for pre-trained LLMs, circumventing annotation difficulties. Furthermore, SPORT improves data utilization by extracting useful step-level preferences even from failed trajectories, enabling more effective learning with the same number of samples. These capabilities enable stable and scalable self-exploration, achieving better generalization for complex multimodal tool usage tasks.
We conduct experiments on the two multimodal reasoning benchmarks: GTA and GAIA, and results show that our SPORT agent outperforms the SFT agent by 6.41% % and 3.64%, respectively. This indicates that our SPORT agent method leads to a more powerful reasoning and planning capability for tool usage by interacting with the environment.
In summary, our contributions are threefold. (1) We propose SPORT, a tool usage exploration framework for multimodal agents, providing possible way for multimodal tool learning, leading to generalization without any annotation. (2) The obtained SPORT Agent achieves significant performance improvements compared with SFT agents two popular benchmarks: GTA and GAIA. (3) We collect the explored preference tuning data into a dataset composed of 16K data, which is conducive the subsequent research on tool usage and multimodal agents.
Related Work
2.1 Agent Tuning
Due to the disparity between the L.LMs and the requirements of agents, agent tuning is necessary to adapt to practical tasks. Research for agent tuning could be divided into two categories: supervised

---

fine-tuning (SFT) and reinforcement learminng (RL). SFT methods collect trajectory data via distillation from closed-source API (e.g., GPT-40) [8 38 60 or human annotation 36 9.. Then they use these collected data to tune the controller via SFT. However, the SFT methods suffer from huge costs and inferior generalization [4). To solve this issue, researchers have paid attention to RI. agent tuning methods that allow agents to interact with the environment and learn from the feedback. Some methods utilize the policy gradient technique [65] to update the controller with a reward model that is designed as fine-tuned model [44] 62], environment feedback 61). human-designed rules 42 66). or tree search results [T. Especially, some methods resort to the policy gradient method for tool learning [23 [1]. where they use the prediction correctness the reward. To simplify this procedure, the direct preference optimization (DPO) methods are applied to agent tuning [56], which construct step-level 4 M trajectory-lever 664 47 preference data based on whole correct trajectories. Nevertheless, most existing agent tuning methods rely on answer or trajectory annotations that are difficult obtain in multimodal tool usage tasks. In contrast, our tool usage exploration framework does not rely on any annotation via step-wise preference tuning.
2.2 Step-wise Preference Tuming
Preference tuning methods rely on paired data, which is not readily available for complex tasks with multi-step reasoning, making it non-trivial to determine which trajectory is better. Furthermore, for long trajectories, only an overall preference verification can not capture the relationships among steps and ignores the fine-grained preference between different steps. To overcome this problem, step-wise preference has been studied. STEP-DPO (26] and SCDPO [39] collect step-wise preference data by
localizing error steps or disturbing the correct path. OREO 55] and SVPO [50] train value models for step-wise verification and inference guidance. SDPO [24] combines step-level, turn-level, and session-level preference data for full-grained optimization. The above methods mainly focus on code generation and math reasoning tasks that are easy to obtain correct trajectories to construct step-wise preference data. In contrast, this method focuses on multimodal agents for tool usage, where obtaining correct trajectories is challenging. Thus, our agent explores the tool usage by itself via an iterative manner, which uses designed Al feedback construct step-wise preference data without any annotation.
2.3 Learning from AI Feedback
Using models to generate AI feedback for performance improvement has emerged as a critical paradigm (4). Existing methods can be broadly divided into three categories. The first category adds AI feedback into prompts for in-context learning 40). LLaVA-Cirtic (55) trains a model to provide
multimodal AI feedback. VLM-F [32, VolCaNo 28 and Clarify 29 use AI feedback address
visual hallucinations. CLOVA 17 and CompAgent [54] refine prompts using Al feedback. The
second category uses AI feedback to filter data for supervised fine-tuning, such as M-STAR (35) for visual mathematical reasoning and APIGen (38), MAT [18), and visualagentbench (36] for agent trajectories. The third category uses AI feedback for reinforcement learning, producing rewards for
policy gradient optimization [27 16 45 or preference data for DPO [59]. Different from them, our
Al feedback is well-designed for evaluating tool usage complex scenarios of multimodal agents.
3 Method
3.1 Formulation
We opt for the framework of the ReAct agent (57] that performs step-by-step reasoning for tool usage. In each step, based on the input n,, the agent outputs an action og for tool calling.
of = arg max To(a, (a,
where Ta is the controller (an VLM in our method) of agents with 0 being the parameters, I, is composed of the task (including a query Q in natural language and multimodal files F) and the history h; of previous steps, i.e., I; = [Q, F. h.). The action consists of the thought t, and code e; for tool calling, a; = (t,, ). . T denotes available tools, and we follow the work [8] using the same tooliit. In this case, we further rewrite Eq. ①1 as
t cf = arg max (4., Cal,, F.h.
(2)
3

---

Task: What the total percent of commission agents marine issues?
Task: Hew many of sugar will take if drink two drinks of this 12 oz can day like that picture for week?
User
SPORT Agont
Papalall
Aetten
Attion
SPORT Agent
ask ssunch qpaation
Znage cat queastion" for rach clas? Taage wt///egg..... J00
camsart
Action22
Ieepe iquaatiie commooottt agents oztegaries Voag "./leag-11""
C <"_2tngsooppp"
33
Obx
of clea7
(cagar (ie)) desc
Oew
heral_rager, ferraon
Obr
50 Prrm
12 or
Movarna Vorl
Marsonal 33.0%
Nosehake
Funix
Acten22
Altiona
Plasl,
Action
Hawwr sarine-tang
Plsal Jeeeer Worten-
jeger 33 33
lasar
das = Eaga
contoons)
Fack
411 ager auger deo
2.3
Nkgora
272
Data Collection
Data Collection
Tack
Preferedd
Task
Aettinn 1.2
Ai
Ariioo
Prefored
Prefoned
Teak
Zetion
Arbioe
Prefored
Tack
1-1
Os Prefenee
Tack
Aletion
Di- retterdddedd
Tack
Ds Preferl
Task
Aetion 2.2
Action
De Proterteddd
Text
Arnae
1-3
Ou Prefened
Arcionn
Dio Pretemeed
Tark
Aroisn
2-2
Ds Uufferrre
Action
Der Proteredd
Figure 2: Demonstrations of the search scheme used in the SPORT method. Given task, the agent samples potential actions for each step and verifies their qualities in an online manner. Then, we construct the step-wise preference data based on such self-exploration.
where 13 and of are thought and code for the i-th step, and the history by = {t1, C1 01,-.. 14-11 0 is composed of thought (t1.....-1) code {c1,....-1],, and observa- tion {o of previous steps.
Agent tuning aims to update A to increase the tool usage capabilities of agents. This paper proposes an iterative tool usage exploration method, SPORT, to update via step-wise preference optimization in refining trajectories, as shown in Figure [ Concretely, SPORT has iterative components: task synthesis, step sampling, step verification, and preference tuning. In one iteration, SPORT first generates some multimodal tasks. For each generated task, SPORT performs step sampling and step verification alternately to construct step-wise preference data, Finally, SPORT uses the step-wise preference data to tune the controller.
3.2
Task Synthesis
Since multimodal task is composed of language query and multimodal files, the task synthesis component is divided into query generation and multimodal file generation. We first generate queries and then generate files, rather than the reverse order, since the multimodal files are diverse (such as DOCX, PPTX, XLSX, and PDF), and it is challenging to construct a diverse file dataset in advance. In addition, tasks are usually based on multiple files instead of only one. First obtaining files and then generating queries may cause weak relevance of files and unnatural queries.
To produce diverse and practical queries, we collect seed from the existing method MAT [8 and employ an LLM (e.g., Qwen2.5-7B) to generate queries. We feed randomly sampled seed queries, used tools, and a designed prompt to the LLM that generates multiple queries at once. For each generated query, we prompt the LLM to output the needed files. If images are needed, we search for source images from off-the-shelf datasets based on similarities. For other files, we prompt the LLM to generate Python code to produce files.
3.3 Data Construction
The two components: step sampling and step verification, are performed to construct high-quality preference data, playing key roles in method. To avoid potential bias issues in constructed data, we introduce an online search scheme, where the step sampling and step verification are executed alternately in each generated task, as shown in Figure
The step-wise preference data is formulated as a triplet (xi, afee a.fi), where x; = =QQ, F, h:) pre denotes the input including the query Q, files F, and the history h, of previous steps, aj is the preferred action in the current step, and afee is the dispreferred action.

---

Concretely, given a task with the query Q and files F, the agent expands the search space for the first step by sampling n actions {aj ai.-.. ... aff; including the thought and code (t..c).... .t cf from the controller, and execute them to obtain n observations (o) of Then, we feed the query Q, actions, and n observations to LLM,, and ask an LLM to select the best action (4i,e]) with its corresponding observation oj.
Along ) and of. we expand the search space for the second step. Regarding {ti, of) the history l12, the controller samples actions {42 -2....... 2.2 2 from the controller, and executes them to obtain n observations fol. : o ). We feed the query Q. history h2, actions in this step, and n corresponding observations to the LLM, and ask the LLM to select the best action {tz c with corresponding observation 0².
this case, the agent gradually expands the search space and selects the best action for the next step, until the agent believes that the task is over. Here, we provide observations {o): .... ož } and prompt the verifier check which tools lead to desirable observations, instead of verifying which observation is correct.
Assume there are m steps in solving one task. In this case, we could collect m(n - 1) preference data pairs. In the i-th step, the selected best action { c is the preferred output, and the rest n 1 actions are the dispreferred outputs, collected in set Di DDiiis = n 1. The preference data in one task is denoted as {(T; af adit)) }}} where a." = h ag (4, 2. E Di and i [1, m).
3.4 Preference Tuning
Objective. In one iteration, we may generate multiple tasks and construct preference data for them. After that, we denote the obtained preference data set as D {(1;, af"e adfi))) We choose the flexible DPO algorithm,
To Ta(af" (2:)) (a (x5) Tref f (a (x:) }}
To To (x;))
[()) = E (a "00""""" logo log
log
(3)
Tref (a
where is the controller to be updated, Kref is the controller for reference (the model after SFT in practice), 33 is the weighting parameter that controls the deviation from the reference controller, and (.)) is the logistic function.
Training Scheme.
The proposed tool usage exploration is performed after an SFT stage for controllers, since the effectiveness of self-exploration requires the controller to have the ability to generate accurate actions. The SFT stage is the same MAT [18] where 20K trajectories are used to align the agent controller (Qwen2VL--7B in practice) with desirable outputs. In the self-exploration stage, we use preference tuning to update Qwen2-VL. The preference tuning process is summarized in Algorithm
Algorithm 1:
Training process SPORT
Require: Seed of tasks, initial agent controller after SFT, and nreef me. Preference data 2> 0.
Ensure: Updated agent controller me-
1: while Not converged do
Set
3: Randomly sample task seeds, and send them to an LLM to generate tasks.
4: for Each generated task do
5: for the i-step in solving the task do
Sample actions [1) .t.......... ..,, based on the history hs, execute them to obtain results (o...... ).).
7:
Select the best action c Construct Add t of, into ha.
8:
preference pairs, and add them into D.
10:
end for
11:
end for
12:
Use D to update vy via the preference tuning algorithm in Fq.
13:
end while
5

---

Table I: Results on two benchmarks: GTA and GAIA. The bold results represent the best performance compared to the open-source models.
GTA CodeExec
GAIA
Method
Controller
ToolAce
AnsAce
Level
Level 2
Level 3
AnzAce
Closed-source Conroller
Lego Agent
GPT-4
46.59
Lego Agent
GPT-4o
41.52
Warm-up Agent
GPT-4-turho
30.20
15.10
0.00
17.60
HF Agent
GPT-4o
63.41
95.12
$7.05
47.17
31.40
11.54
33.40
HF Agent
GPT-4o-mini
56.10
100.00
57.69
33,96
27.91
3.84
26.06
Open- Open-Source Controller
HF Agent
LLAV--NeXT-BB
14.97
25.08
9.43
1.16
0.00
3.64
HF Agent
Intem
36.75
52.18
32.05
7.55
4.65
0.00
4.85
HF Agent
MiniCPM V-8.5B
36.59
56.10
33.97
13.21
5.81
0.00
7.27
HF Agent
Qwen2-VL-7D
44.85
65.19
42.31
16.98
8.14
9.70
T3-Agent
MAT-MiniCPM: V-8.5B
65.85
80.49
52.56
26.42
11,63
3.84
15.15
T3-Agent
MAT-Qwen2-VI--7B
64.63
84.32
53.85
26.42
15.12
3.84
16.97
Owr
SPORT Agent
Tuned QwenZ VI.-BB
72.41
91.87
60.26
35.85
16.28
3.84
20.61
4 Experiments
4.1
Setting
The performance of the proposed SPORT approach is assessed the GTA 51 and GAIA 41
benchmarks, Results are compared against agents powered by both closed-source models (e.g., GPT-4, GPT-4o, Claude3) and open-source models, including LLaMA-3-70B-inn-te
[12], Qwenl.5-72B-
[33. Intern (8. Qwen2- [52), and MiniCPM- V-8.5B (58] Specifically, we perform direct comparisons with leading agents, such as Lego Agent 1 and Warm- up Act Agent [41]. As a baseline, we use the Huggingface Agent (HF Agent) [22], which operates with the same toolset the SPORT Agent. We first evaluate these agents on benchmarks, then assess the quality of the produced preference data, and finally show several visualization examples to demonstrate the effectiveness of our method.
chat [] LLaVA-NeXTas
We employ the Qwen-2-V VL model as the controller. In the training process of our VLM controller, freeze the vision encoder and visual token compressor, and fine-tune the language model using LoRA 20) We set the rank as 32 and apply LoRA on query, key, and value projection matrices in all self-attention layers. We use the AdamW optimizer with cosine annealing scheduler. The learning rate is 1.0c and the batch size is 2 per device. We set the max context window as 10240 to support complex trajectories of our agent. All training is conducted on node equipped with 8xA100 GPUs. The training time is positively correlated with the number of iterations and iteration step size d. For all the evaluations, we disable the sampling and verification during inference for fair comparison.
Benchmark. The GTA and GALA benchmarks serve as robust evaluation frameworks for assessing multimodal agents. The GTA benchmark includes 229 tasks paired with 252 images. where task completion requires to steps, with most tasks involving 2 to 4 steps. This benchmark challenges multimodal agents to exhibit advanced perception, operational skills, logical reasoning, and creative thinking based on visual data. In real-world multimodal scenarios, agents often need to handle diverse file formats such as PPTX, PDF, and XLSX. To evaluate agent performance benchmark is used, comprising 446 tasks across 109 files. GAIA's tasks are organized into three levels, with task complexity varying from 2 steps sequences of indefinite length. It evaluates document comprehension, web navigation, logical reasoning, and summarization abilities.
such files, the GAIA
agent performance using three key metries: AnsAcc, ToolAcc, and ExdeExec for the GTA benchmark. AnsAce gauges the accuracy of predicted answers. ToolAce evaluates the correctness of tool usage and the quality of answer summarics. CodeExec measures the percentage of generated code that executes without errors. In the GAIA benchmark, we focus measuring AnsAce its three levels.
Metric. Following existing methods [8),
4.2 GTA Results
The results on the GTA benchmark are shown in Table D where key metries including AnsAce, ToolAcc, and CodeExec reported. Our agent surpasses the Lego agent that utilizes closed-source

---

models (e.g. GPT-4 and GPT-4o), as well as the HF agent that uses closed-source models and open-source models (e.g.. VL2--VL2-8BB) showcasing the ability of our SPORT Agent tackle complex tasks with greater efficiency. A comparison between agents through SFT (ie., T3-Agent) and our SPORT Agent demonstrates the effectiveness of our online self-exploration framework and the advantages of our Step-wise optimization approach. Our method has about 7% improvements on the final accuracy, since it calls more suitable tools (8% improvements) and reduces code error (7% improvements). Compared with the HF agent using GPT-4o and GPT-4o mini, our agent achieves higher ToolAce and comparable CodeExec. This indicates that the proposed SPORT method improves the planning and reasoning capabilities of agents again.
4.3
GAIA Results
In [ we report the performance of SPORT Agent on the GAIA validation set. SPORT Agent achieves best results among agents that use open-source models, surpassing the best-performing open-source model, Qwen2-VL-7B, by about 11% on AnsAcc. The consistent improvements across different levels underscore the efficacy of our online self-exploration framework. Furthermore, SPORT Agent demonstrates significant gains over the SFT-tuned controller, with an improvement of about 4% over MAT-Qwen2-VL-7B. However, when compared to agents leveraging flosed-source models such as GPT-4, SPORT Agent exhibits a slight performance gap. We attribute this discrepancy to the larger model sizes and more extensive training data available to closed-source models.
4.4 Ablation
Effectiveness of Iteration Step Size We conduct an ablation study on the GTA benchmark to investigate the impact of the iteration step size d that denotes the number of used trajectories in each iteration. We set d E {200, 500, 1000], adjusting the number of iterations to (5, 2, 1), respectively, to ensure total of 1000 trajectories are processed in each setting. As shown in Table 2, setting d = 500 yields the best overall performance across metrics. When
1000, the model sees all tasks in single pass, which limits adaptability and leads to slight drop in answer accuracy and execution success. In contrast, using a smaller step size (d = 200) results in less diverse updates per iteration, which may reduce robustness. These results suggest that moderate value of d offers a good balance between update frequency and data diversity, leading to more stable and effective training.
Comparison with Static Preference Data We conduct ablation experiments to compare our method (online exploration) with DPO on static tasks. doing so, we directly apply DPO to the MM-Traj dataset (18). Specifically, we treat the MM-Traj data (GPT-4o generated) as "preferred" samples and synthetically generate an equal number of "dispreferred" examples via the MAT-SFT model, matching the total volume of our SPORT preference data. We then fine-tune MAT-SFT using DPO under this constructed preference dataset.
Table 3 reports results on the GTA benchmark. Compared to vanilla MAT-SFT, MAT-SFT-DPO yields only modest improvements (AnsAcc +1.28, ToolAcc +2.67, CodeExec +1.58), indicating that naïvely applying DPO to MAT provides limited gains. In contrast, SPORT substantially outperforms MAT-SFT-DPO (AnsAce +5.13, ToolAcc +5.11, CodeExec +5.97), demonstrating the effectiveness of our framework in leveraging diverse, multimodal preference data.
Training with Different Base Models To evaluate the generalizability of SPORT, we apply it to different base models. We compare four configurations the GTA benchmark: Qwen2-VL-s VL-BB (base model), MAT-Qwen2-VL-777 (base model with MAT applied), SPORT-QQwen2VI--7B (SPORT applied directly to base model), and SPORT-MAT-QWen2--WV (SPORT applied to tun-t model).
Table 2: Ablation on iteration step d in the GTA benchmark.
Table 3: Ablation on preference data: MAT-SFT v... MAT-SFT-DPO vs. SPORT on the GTA benchmark.
AnsAce (%)
ToolAce (%)
CodeExec (%)
Method
AusAce (5)
ToolAce (%)
CodeExee (%)
200
56.41
68.58
88.46
MAT-SFT
53.85
64.63
84.32
500
57.69
69.87
89.74
MAT-SFT-DPO
55.13
67.30
85.90
1000
57.05
69.87
88.46
SPORT (Ours)
60.26
72.41
91.87
7

---

Table 4: Answer accuracies (%) on GTA benchmark with different base models.
Table Impact of task diversity on the GTA benchmark.
Model
AnsAce (%)
Method
AnsAce (%)
Owen2-VL-7B
42.31
MAT-Qwen2-VL-77B
53.85
MAT-Qwen2-VL SPORT-Qwent VL-7B
53.85
SPORT w/ 5 from 100 seeds
58.33
55.13
SPORT-MAT- -I-RT--MAT-QQen2-VV--78
60.26
SPORT w/ 20 from 425 seeds
60.26
Table presents the answer accuracies. Applying SPORT directly to Qwen2-VLL VL-7B improves accuracy from 42.31 % to 53.13% (+12.82%), demonstrating SPORT's effectiveness as a standalone method. When applied to MAT-tuned baseline, SPORT achieves the highest accuracy of 60.26% (+5.13% over MAT alone), indicating that SPORT can effectively complement existing tuning methods. These results confirm that SPORT is flexible approach that enhances agent performance both independently and in combination with other preference optimization methods.
Sensitivity to Task Quality and Diversity To assess SPORT's robustness to the quality and diversity of synthetic tasks in early-stage self-exploration, we conduct an ablation study where the in-context examples are reduced from 20 5 and the task seed pool is narrowed from 425 to 100. This setup mimics low-diversity scenario during early-stage exploration.
As shown in Table []] this reduction leads to moderate drop in performance (60.26 + 58.33). However, the result still significantly outperforms the SFT baseline (58.33 vs. 53.85), demonstrating that SPORT maintains strong performance even under constrained task diversity. These findings indicate that while task quality and diversity do impact SPORT's effectiveness, the method exhibits robustness to variations in the synthetic task generation process.
4.5 Statistic
We aggregate step-wise preference data from all iterations, each with 500 and a sampling size of 5, resulting in total of 16K samples. We analyze the differences between the chosen and rejected step-wise data in terms of code error rate, tool selection, and content variations.
Tool Distribution We analyze the distribution of tools the chosen and rejected steps by ex- amining their frequency of occurrence, as shown
Tool Frequency Distribution (Chosen Rejected)
2000
Chosen
Rejected
in Figure 3 The results highlight differences in tool usage between the two groups.
Frequency
1000
In the chosen steps, tools such as vi visulizer (2101 occurrences) and objectloc (1051 oc- currences) are used more frequently. In con- trast, in the rejected steps, objectloc (1442 oc- currences) and visualizes izer (1524 occurrences) are more prevalent. Additionally, tools like ocr and seg appear more often in the rejected steps. To quantify the overall discrepancy in tool us- age, we computed tool distribution difference of 45.62%, indicating substantial variation in the tool selection between the chosen and rejected steps.
500
VOA
Web search
OCR
Face datect
SEG
image gene
image edit
Object
Toat
Figure 3: Tool distribution for the chosen and re-
jected steps.
Code Error Rate We compared the code execution status of chosen and rejected steps by measuring the proportion of execution results (Observations) that contained code errors. The rejected steps exhibit significantly higher етror rate (81.9) (81.94% compared to the chosen steps (18.35%). This indicates that our step-wise preference data favors code that executes successfully. Consequently, this preference also leads to the improvement in the code accuracy of the SPORT Agent, as shown in Table I (Code Exec 84.32% + 91.87%).
Content Difference We compared the BLEU scores of steps selected by our verifier and those selected randomly. BLEU scores measure the similarity berween different sequences, with lower scores indicating higher discrimination. As shown in Table 6 our verifier consistently achieved

---

Table 6: Comparison of BLEU scores (lower
Table 7: Average scores from humans data quality.
scores indicate greater discrimination) between
verifier and random selection.
Random Select
Verifier
(T(
0.53
B2(L)
0.41
(3)
0.36
B4 (L)
0.34
Reszonalle
Task
Natura
Tool 8.78
Trajectory
Couteez
Code
Preference
Ours 0.30
0.18
0.14
0.11
8.16
8.48
9.08
8.14
82%
lower BLEU scores across all n-grams compared to random selection. Specifically, for BLEU-1, BLEU-2, BLEU-3, and BLEU-4, our verifier's scores were 0.30, 0.18, 0.14, and 0.11, respectively, while random selection yielded higher scores of 0.53, 0.41, 0.36, and 0.34. The results demonstrate that our verifier selects more distinct steps, enhancing the quality of the chosen steps.
4.6 Data Quality
To evaluate the effectiveness of the constructed preference data, we conducted user study involving 20 Al researchers with coding and development experience from various universities and research institutes. These participants were not provided with any background information about method- ology; instead, they were only briefed on the purpose and functionality of the agent. They were required to justify whether the preferred and dispreferred pairs were proper.
The evaluation was performed for tasks and trajectories, based on five criteria: (1) 'Reasonableness' to evaluate whether the generated tasks are infeasible; (2) 'Naturalness' to evaluate whether the generated tasks are natural; (3) 'Code' to evaluate the accuracy code in action; (4) 'Tool' to assess the appropriateness of tool selection; and (5) 'Parameter' to assess the correctness of parameter passing. Participants provided scores ranging from to 10, with higher scores indicating better performance. As shown in Table 7 the average scores for our task and framework exceeded 8, demonstrating the validity of the preference data collected by our approach.
To further assess the effectiveness of the verifier in generating preference data, we conducted an additional study with a separate group of researchers. Each participant was asked to evaluate 50 steps sampled in parallel by making preference selections. The preferences were determined using the same three criteria: code accuracy, tool selection appropriateness, and parameter passing correctness. We measured the agreements between the verifier's 's preferences and those of the human participants, denoted as 'Preference' in Table 7 The results revealed an 82% overlap. This high level of agreement validates the reliability of the verifier in capturing human-like preferences.
Query: Find out the latest reviews for the smartphone featured this image. Include comparisons with its competitors. Attachment: "image.jpg"
Attached Image:
Thought
Thought
reviero for the Siroethe
use ar vodee von eeight contum thephann's
ovamphonne diploand in
Then,
spacifi
Fmage canta in tee "Seefoo Nox Plc"")
Jeace aqent to wrterr with coneaation.
from iegort Age
whot owu and ipecfication about the amariphonns hiop
nockl to pertara naling
Cods:
iegx mulgn maimess hytecaiie Gitac euge To
CCR
in the image" image, puttee imags
renovo Zentore Mhe Phis ondiorppe compperiee
seach qpary
d Search revins of ontoucted modde
conoxttinn aalk auarch apertpperpenn Ivge auth- Vanage
Compastors
rewerd soarch searth agvet) bne the lates (oatlssnatt
Samdie
Sample
Sample
Varification hemsen
onorphhese conermiin fore the CCR coneot obean thbbaand ard the talk aol.serr bot sannoi proces Imag...
The conconsecly adeettly
brond
Paral
Sampling
ili snnees le lie a this and feotre Nowe Pio
Aedive Vie Rut ved range swa rptone to better edlve you, ccmmpar s cange rphine pattooo
rovill and look ferncw rocnt reciiieeips renipua or nover ana raanaess the
Step2
Max
Code::
Cede:
prinspumntt peeina
price brucken performanc andiess the??
the uni bracke")
3ample
Sample
Semple'
Vutticatioe Beason:
The stice wavc #ilrrrionn.oo wwver mani-ongg coepetionn vhile the angriie wilusann wos.i.c... compor Jwrdian Mia Rucheth rpeciil cnonphoeee or dons avettion Jealirs Mai Pla smartption..
Figure 4: Example of constructed preference data.

---

Qpery ronyy iiince firtne rosel del borme?? Aaatmernc "ale www.etttee///maaa....mmoo 278,96
Atteced Irage:
what ViaM product imagel Atschment Yea bendhnalygeggge jns
Attcceed Image:
Nough:
Apr
Hough
SPORT Agen Jusi)
5ET Agett
Houghs
Ganel crmmn couelegs, "Ite tre pocden''
tnced ciard via seco sook
Colec surtee your frito sncci
1o infuu mlsu fice socet
vealior lad
ottgery
raser jusid apin
if hest
cage lead Vittal camuar
sunkrat toee
Somee woth
Lock
Ceevntin
BcD The tiwe bul mocver Taaiy pur ausaiiasa saspet
FSs 400
ou the calie antireele linge
admia cllce and vesse.
Foac fooll fortt her poee formddoo Heveee inca
cal
de bod' Ha butf san thoee
thoe sibler' par
be Da
vibo, upi praghtiliieoooo
Liaee Phrr oiercs Sooer new lorowed
sager you path Singee
1188
(a) Different tool selection.
(b) Hallucination vs. Tool calling.
Figure 5: Comparisons between SFT Agent and SPORT Agent.
4.7 Visualization
We visualize the preference data generated by the online self-exploration framework, as shown in Figure 4
The verifier can successfully choose actions that lead to correct intermediate results. In step 1, the action that produces the correct brand and basic information about the smartphone is selected as the preferred data. In step 2, the action that searches for content most relevant to the task selected as the preferred data. compares the smartphone with mid-range competitors, while the rest compares the smartphone with a special one.
We visualized the task-solving procedure of the SPORT Agent compared with the SFT Agent (T3 agent), as shown in Figure The SPORT Agent after the step-wise preference tuning can well solve the issues of code hallucination and tool error. For example, case (a), the SPORT Agent correctly selects the web search tool while the SFT Agent uses the wrong tool with an incorrect image path. In case (b), the SPORT Agent utilizes tools solve the task, while the SFT Agent produces an answer via hallucination.
5 Conclusion
In this paper, we have presented an online self-exploration framework for multimodal agents, through which the agents can learn via automatic interaction with new environments without accessing any annotations. Based on this framework, we have presented a step-wise optimization for refining trajec- tories (SPORT), which can produce in-distribution preference data in complex environments. Given proper prompts, the proposed SPORT method can generate diverse multimodal tasks and provide good verification of agent actions aligned with humans. Experiments on two challenging benchmarks, GTA and GAIA, show that the proposed SPORT method achieves significant improvements multimodal agents, demonstrating its effectiveness.
Limitations
The verifier plays important role in the current SPORT method. However, it heavily relies on human-designed rules and prompts, causing inferior generalization for some outliers. In the future, we will explore the self-expioration techniques for the verifier, that is, learning to verify, through which the verifier can adapt to new environments with the controller together. Furthermore, will explore the theoretical guarantee for the verifier, allowing to seale to open settings.
10

---

Acknowledgements This work was supported by the Natural Science Foundation of China (NSFC) under Grants No. 62406009, No. 62172041 and No. 62176021. Shenzhen Science and Technology Program under Grant No. JCYJ20241202 under Grant No. TYX1200230807142203006
20114100222110803 30548062, and Natural Science Foundation of Shenzhen
References
[1] AgentLego Contributors. AgentLeeg:: Open-source tool API library to extend and enhance LLM agents, December 2023.
[2] Hao Bai, Yifei Zhou. Jiayi Pan, Mert Cemri. Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement lcarning, Advances Neural Information Processing Systems (NeurIPS), 37: 37:12461-12499 2024.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv: 1200073, 2022.
[5] Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathematical reasoning. In Annual Conference on Empirical Methods in Natural Language Processing 2024.
(EMNLP), pages
[6] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision (ECCV), pages 370-387. Springer, 2024.
[7] Sijia Chen, Yibo Wang. Yi-Feng Wu, Qingguo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. Advancing tugl-nngeet large language models: Integrating insights from errors in inference trees. Advances in Neural Information Processing Systems (NeurlPS). 37:106555- 106581, 2024.
[8] Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 22..244444.2... 2024.
[9] Chunyuan Deng, Xiangru Tang. Yilun Zhao, Hamming Wang. Haoran Wang, Wangchunshu Zhou, Aman Cohan, and Mark Gerstein. Mimir: A streamlined platform for personalized agent tuning in domain expertise. arXiv preprint 2xXv:22444.04285 2024.
[10] Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training image comprehension. In Advances Neural Information Processing Systems (NeurlPS), 2024.
[11] Zhirui Deng. Zhicheng Dou. Yutao Zhu, Ji-Rong Wen, Ruibin Xiong. Mang Wang. and Weipeng Chen. From novice expert: I.lm agent policy optimization via step-wise reinforcement learning. arXiv preprint arXiv:2411.038// 2024.
[12] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The laaa 3 herd models. arXiv preprint arXiv:2407.1178. 2024.
[13] Yue Fan, Xiaojian Ma. Rongpeng Su, Jun Guo, Rujic Wu, Xi Chen, and Qing Li. Embodied videoagent: Persistent memory from egocentric vidcos and embodied sensors enables dynamic scene understanding. International Conference on Computer Vision (ICCV), 2025.
[14] Yue Fan, Xiaojian Ma. Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao. and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. European Conference on Computer Vision (ECCV), 2024.
[15] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Banquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint wrXiv:2504.11566 2025.
[16] Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet. Furl: visual-language models as fuzzy rewards for reinforeement learning. In International Conference on Machine Learning (ICML), pages 14256 14274, 2024.

---

[17] Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li. Clova: A closed-loop visual assistant with tool usage and update. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 33258-13268, 2024.
[18] Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, and Qing Li. Multi-modal agent tuning: Building a International Conference on Learning Representations (ICLR), 2025.
agent for efficient tool usage. In
[19] Tanmay Gupta and Aniruddha Kembhavi. Visual programmingg Compositional visual reasoning without training. In The IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 14953-14962, 2023.
[20] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations (ICLR), 2022.
[21] Yushi Hu, Oulia Stretcu, Chun-Ta Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2024.
[22]
HuggingFace Contributors. Agents and tools, 2024.
[23] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503,00 09516, 2025.
[24] Aoho Kong, Wentao Ma, Shiwan Zhao, Yonghin Li, Yuchuan Wu, Ke Wang, Xiaoqian Liu, Qicheng Li, Yong Qin, and Fei Huang. Sdpo: Segment-level direct preference optimization for social agents. arXiv preprint 2111:2500.01121, 2025.
[25] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop. Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. 12917, 2024.
[26] Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang. Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406. 18629, 2024.
[27] Harrison Lcc, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellic Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement lcarning from human feedback with ai feedback. In International Conference Machine Learning (ICML), pages 26874-26901. PMLR, 2024.
[28] Scongyun Lee, Sue Park, Yongrae Jo, and Minjoon Seo. Volcano: Mitigating multimodal hallucination through self self-feedhack guided revision. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (MAACL-LTT) pages 391-404, 2024.
[29] Yoonho Lee, Michelle S Lam, Helena Vasconcelos, Michael S Bernstein, and Chelsea Finn. Clarify: Improving model robustness with natural language corrections. In ACM Symposiun on User Interface Software and Technology (UIST), pages 1-19, 2024.
[30] Pengxiang Li, Zechen Hu, Zirui Shang, Jingrong Wu, Yang Liu, Hui Liu, Zhi Gao, Chenrui Shi, Bofei Zhang, Zihao Zhang, et al. Efficient multi-turn rl for gui agents via decoupled training and adaptive data curation. arXiv preprint arXiv:2509.2366
2025.
[31] Shilong Li, Xingyuan Bu, Wenjie Wang, Jiaheng Liu, Jun Dong, Haoyang Hc, Hao Lu, Haozhe Zhang Chenchen Jing, Zhen Li, et al. Mm-browsecomp: A comprehensive benchmark for multimodal browsing agents. arXiv preprint arXiv:2508.1 13186, 2025.
[32] Yuan-Hong Liao, Rañdd Mahmood, Sanja Fidler, and David Acuna. Can feedhack enhance semantic grounding in large vision-language models? arXiv preprint 10Xiv:2404.06510, 2024.
[33] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning, oct, and world knowledge, January 2024.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 36:34892 34916, 2023.
12

---

[35] Wei Liu. Junlong Li. Xiwen Zhang, Fan Zhou, Yu Cheng, and Junxian He. Diving into self-evolving training for multimodal reasoning. arXiv preprint 2rXiv:2412.17451, 2024.
[36] Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song. Shudan Zhang, Hanyu Lai, Xinyi Liu. Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327. 2024.
[37] Ziyu Liu, Yuhang Zang, Yushan Zou, Zijian Liang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual agentic reinforcement fine-tuning. arXiv preprint arXiv:2505. 2025.
[38] Zuxin Liu, Thai Hoang. Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems (NeurIPS), 37:54463 1154463-54482,, 2024.
[39] Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Step-controlled dpo: Leveraging stepwise error for enhanced matherntical reasoning. arXiv preprint arXiv:2407.00782. 2024.
[40] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems (NeurIPS), 36:46534-46535 2023.
[41] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. International Conference on Learning Representations (ICLR), 2023.
[42] Feng Peiyuan, Yichen He, Guanhua Huang. Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. Agile: A novel reinforcement learning framework of Ilm agents. Advances in Neural Information Processing Systems (NeurIPS). 1145244-52844 2024.
[43] Pranav Putta, Edmund Mills, Naman Garg. Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07111 2024.
[44] Zehan Qi. Xiao Liu, lat Long long. Hanyu Lai, Xueqiao Sun, Xinyue Yang. Jiadai Sun, Yu Yang, Shuntian Yao, Tianjie Zhang, et al. Webrl: Training 1lm web agents via sellvevolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337 2024.
[45] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310.1299 2023,
[46] Wentao Shi, Menggi Yuan, Junkang Wu, Qifan Wang. and Fuli Feng. Direct multi-turn preference optimization for language agents. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 232222244 2024.
[47] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration- based trajectory optimization of I1m agents. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 7584-7600, 2024.
[48] Dídac Surís. Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In International Conference on Computer Vision (ICCV), pages 11888 11898, 2023.
[49] Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Zhengxin Li, Lin Ma. Shenghua Gao, al. Tool-Imm: A large multi-modal model for tool agent learning. arXiv preprint arXiv:2401.10727, 2024.
[50] Huaijie Wang, Shibo Hao. Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline reinforcement learning for I1m multi-step reasoning. arXiv preprint 2xXiv2412..61145 2024.
[51] Jize Wang. Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: A benchmark for general tool agents. In Proceedings of the Neurad Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2024.
[52] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. et al. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. aix preprint arXiv:2409. 12191 2024.
[53] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Senf-instruct: Aligning language models with self-generated instructions. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 13484-13508, 2023.
13

---

[54] Zhenyu Wang, Enze Xie, Aoxue Li. Zhongdao Wang. Xihui Liu, and Zhenguo Li. Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation. arXiv preprint arXiv:2401. 2024.
[55] Tianyi Xiong, Xiyao Wang, Dong Guo. Qinghao Ye. Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Llava-critic: Learning evaluate multimodal models. arXiv preprint 12i::2410002212. 2024.
[56] Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang. Ke Wang. Cheng Li, Wei Peng. and Sujian Li. Watch every step! Ilm agent learning via iterative step-level process refinement. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1556-155 1572, 2024.
[57] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[58] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li. Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm your phone. arXiv preprint WxXiv22408.01800 2024.
[59] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua. et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 17220, 2024.
[60] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 3053-3077, 2024.
[61] Simon Zhai, Hao Bai. Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Ma, al. Fine-tuning large vision-language models as decision-makinn making agents via reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 37:1 10935-1 11935-11097 10971, 2024.
[62] Yuanzhao Zhai, Tingkai Yang. Kele Xu, Dawei Feng, Cheng Yang. Bo Ding, and Huaimin Wang. Enhancing decision-making for Ilm agents via step-level q-value models. In AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 271 27161-27161 169, 2025.
[63] Bofci Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song- Chun Zhu, and Qing Li. Tongui: Building generalized gui agents by leamning from multimodal web tutorials. arXiv preprint arXiv:2504.1 102504..1669 12679, 2025.
[64] Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang. YiL. Chanqi Wang. Mingyu Ding, Dieter Fox, and Huaxiu Yao. Grape: Generalizing robot policy via preference alignment. arXiv preprint 2411v:241..19300 2024.
[65] Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xions Wang. Sergey Levine, and Erran Proposer-agent-evaluuar (pae): Autonomous skill discovery for foundation model internet agents. arXiv preprint arXiv:2412.13144 2024.
[66] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. In International Conference on Machine Learning (ICML), pages 12178-62009 PMLR, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:(Yes
Justification: We believe the main claims made in the abstract and introduction accurately reflect the paper's contributions in multimodal tool usage agents.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
The limitation has been discussed in Section S (Conclusion).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
No thcoretical assumption is needed.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The code and data will all be open-sourced once the final decision of this paper is given.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the codes (including the data generation pipeline) and the prefer- ence data in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //niss.c/public/guuut http://///giiieee/eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are described in Section (Experiment) in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, the appendix, or supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the error bar for the main results the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The computing resources are discussed in Section 4 (Experiment).
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: stttps://neuripss http://wwurrsss.ccccpu/cc/tttcct
Answer: [Yes]
Justification: We confirm that our research fully complies with the NeurIPS Code of Ethics. All experiments were conducted responsibly, and ethical considerations were carefully addressed throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts are discussed in the Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
19

---

Answer: [Yes]
While our work leverages pretrained large language models, they are used solely enhance the agent's ability interact with tools in controlled setting. Therefore. we believe the risk is minimal and no additional safeguards are necessary beyond standard responsible use practices.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets used in this paper are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: The data generation pipeline, data format and examples are well discussed in Section3iin detail.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
20

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We report the number of participants and the detailed human study setting in Section 4.6 and Appendix D.I
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer:[Yes:
Justification: We invite humans to evaluate the quality of the generated data. All human evaluations were conducted in accordance with the NeurIPS ethical guidelines and approved by an appropriate ethics review process. Participants were informed of the study's 's purpose, any potential risks, and their rights, and provided informed consent before participation.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We VLMs for data generation and agent tool calling. We fine-tune VLMs as the agent controller.
Guidelines:
The answer NA means that the core method development in this research does not involve L.LMs as any important, original, or non-standard components.
Please refer to our LLM policy http:::///eerips.s///oeeeeeeeeee for what should or should not be described.
21

---

A Broader Impacts
SPORT's ability to autonomously explore tool usage and refine behavior via preference feedback promises to lower the barrier to creating versatile multimodal agents, enabling researchers and practitioners even those with limited resources to build domain-adapted systems for tasks such as document analysis, scientific data interpretation, and accessible educational or healthcare interfaces. By reducing reliance on costly human annotation and manual rule-crafting, SPORT can accelerate innovation across diverse fields, fostering more inclusive and scalable Al solutions.
At the same time, increased agent autonomy raises risks of unintended bias, error propagation, and resource inefficiency. SPORT's verifier, grounded in heuristic rules and model judgments, may inadvertently reinforce spurious behaviors or hallucinations, particularly in high-stakes domains like legal or medical analysis. We therefore advocate for transparent auditing, oversight, and careful management of exploration budgets to ensure responsible deployment.
-he---
B Comparison with Existing Sampling Frameworks
Our method constructs step-level preference data for complex multimodal tasks. To contextualize its contributions, we compare against representative searremeee--e-amnng-----e sampling frameworks along three dimensions: (1) Task Domain, (2) Collection Cranularity, and (3) Annotation Format. Table summmaries this comparison.
Table 8: Comparison with existing sampling schemes
Method
Task Domain
Collection Gramularity
Ansotation Format
WebRL
4E]
GUI control
Trajoctory-Levs
Finetune reward model
PAE 655
control
Trajectory-Leve
a pre- trained model
ETO 4x
GUI contral & Embodied Al
Trajectory- evel
Expert labels for comparisons
DMPO 46
GUI control Embodied Al control
Trajectry-Level
Finctune a reward model
DigiRL
Step-Level
Finetune a reward model
TP-L.LAMA
H
API calling
Step-Level
Use expert data
IPR [6
GUI control & Embodicd Al
Step-Level
Use expert data
MeepAgent-Ineers
control Embodied
Step-Level
Use expert
Ours
Multimodal Reasoning
Step-Level
Use pre-trained model
The multimodal reasoning domain poses unique challenges that prior sampling frameworks do not adequately address:
1. Data scarcity. There is a shortage of collected tasks and expert trajectorics in complex multimodal settings.
2. Inadequate reward modeling. Approaches effective in GUI or API domains-suce as pre- trained classifiers or rule-based
to capture multimodal task complexity. 3. Low sampling efficiency. Generating high-quality trajectories for multimodal tasks requires expensive tool usage (e.g. LLM calls, web searches), making naive sampling impractical.
To overcome these issues, we leverage pre-trained LLMs to generate step-wise preference data automatically, providing a scalable and practical solution for training agents on complex multimodal reasoning tasks.
C Computational Efficiency Analysis
We provide comprehensive comparison of compute time and GPU usage between the baseline method (MAT) and our SPORT framework. Table Table(Isume summarizes the estimated compute time and GPU cost for both methods, measured in GPU hours (GPUh) using single A100 80GB GPU.
Data Generation Efficiency. The computational cost of Task Synthesis, Step Sampling, and Tool Calling is closely tied to the data scale. MAT synthesizes and samples steps for 20K tasks and 20K trajectories, resulting in high overhead across all three components. In contrast, SPORT performs Task Synthesis on only 2K tasks and constructs 16K preference pairs for tuning, leading to substantially reduced compute time these stages. Specifically, SPORT reduces Task Synthesis time from 29.13h to 3.6 1,6h, Step Sampling from 69.92h to 4.53h, and Tool Calling from 58, 88,93h to 9.86h.
22

---

Table Estimated compute time and GPU cost comparison between MAT and SPORT. GPU hours (GPUh) are measured as u of a single A100 80GB GPU for one hour. The estimates for MAT are derived from reported tool invocation frequency, combined with empirical measurements of the GPT--4ommiii API and tool execution latency our hardware.
Component
MAT Time (h) SPORT Time (h)
MAT Cost (GPUh)
SPORT Cost (GPUh)
Data Generation
Synthesis
29.13
(GPT-4o mini API ~5500)
3.61 (Qwen2 (0we2--VL-7B) VL-7B)
Step Sampling
09.92
4,53
(GPPT-4o-mini API -S1500)
4,53 (Tunes-Queen-VV----
Step Verification
0
3,36
0
3.36 (Qwca2.5-7B)
Calling
58.93
Trining
Model Training 4x A100 80G GPL
15.77
15.20
63.08
60
Total
173.75
36.56
122.01
82.16
Step Verification. SPORT introduces an additional Step Verification process to obtain step-level preferences, which incurs 3.36 GPU hours. While this stage adds computational cost, it remains acceptable relative to the total runtime due to the smaller data volume (16K preference pairs vs. 20K trajectories in MAT).
Training Efficiency, Model training with 4x A100 80GB GPUs shows comparable efficiency between both methods, with SPORT requiring 15.20h (60.80 GPUh) compared to MAT's 15.77h (63.08 GPUh). This marginal difference demonstrates that SPORT's efficiency gains primarily stem from the data generation stage rather than the training phase.
Overall Cost Reduction. As shown in Table 9 SPORT achieves total compute time of 36.56h compared to MAT's 173.75h, representing a 4.75x speedup. In terms of GPU cost, SPORT requires 82.16 GPUh versus MAT's 122.01 GPUh, resulting in a 32.7% cost reduction. These substantial improvements in computational efficiency, combined with SPORT's superior performance, validate the effectiveness of our preference-basee learning approach.
D Task Generation
Following MAT 18), first use an LLM to generate queries, and then generate both the file content and file type based on the query. Depending on the file rype, we adopt different strategies for file generation:
For image files, we retrieve relevant images from large image dataset generated file content.
6 based on the
For non-image files, PDF, XLSX, DOCX, or MP3, etc, we employ the LLM to write Python code that calls relevant libraries to convert the file content into the corresponding file format.
After generating the files, adopt two-step query-fil file verification process Revision and Filter- ing to ensure the quality of each task (i.e., query- file pair).
In the Revision step, both the query and the corresponding file are fed into vision-language model (VI.M), which is instructed to revise the query better align with the file if necessary. For image data, we directly input the image into the VIMM; for non-image data, we input the file content instead.
In the Filtering step, the VLM is no longer allowed modify the query. Instead, it asked assess whether the task meets a predefined quality threshold. Only tasks that pass this quality check are retained, while all others are discarded.
D.1 Model Comparison on Task Generation
We conducted a comparative analysis of task quality between open- and closed-son -ouuce models. For this evaluation, we employed Qwen-2VL-7B (open-source) and GPT-4o-mini (closed-source) to generate 200 tasks each under identical system prompts. The resulting 400 tasks were subsequently randomized and anonymized to eliminate source bias. We recruited 20 human evaluators, with each participant assessing 20 tasks according to two key metrics: naturalness and reasonableness, rated on 10- poin scale (higher scores indicating superior quality). As demonstrated in Table 10 the tasks
23

---

produced by both models achieved comparable quality ratings, providing compelling evidence that open-source models possess sufficient capability for high-quality task generation in this domain,
Table 10: User study for open-source vs. close-source models generated tasks. Scores are scaled from 1 to 10 and higher score denotes better quality.
Model GPT-4o-mini QWen-2VI.-7B
Task Naturalness
Task Reasonableness
8.71
8.37
8.75
8.35
E Error bar for the main results
We conduct each experiment five times and report the performance variance, as shown in Table The results demonstrate that our improvements are statistically significant compared to the observed variances,
Table 11: Performance with variance the GTA benchmark.
Method
Controller
AnsAce
ToolAce
CodeExec
T3-Agent
MAT Tuned Qwen2-VI.-7B
53.85
64.63
84.32
SPORT Agent (Ours)
Tuned Qwen2-V1.-78
60.26 + 1.51
72.41 1.11
91.87 # 1.41
F
System Prompts
We referenced the task generation prompts from MAT for our implementation.
F1 Prompt for Query Generation
The prompt for query generation is shown in fig.
You tasked with generating queries that will prompt agent call various tools (only the tool listed our toolset), including internet search capabilities, to solve real-world, practical problems. The problems should be natural, varied, and challenging, requiring the agent domains. Ensure that the problems span a range of practical scenarios.
different
Our toolset: TOOL_SET
I will now provide examples, along with the tools. Examples of user queries: IN CONTEXT EXAMPLES
Please output the Queries in json format. Make sure that the queries share similar style to the in-context examples. The output template is:
"query": "What is weather today?", <The user query to the agent.>
"tools": ["tool1", "tool2"....] <A list consisting of the tool names related to the query.>
Figure 6: Prompt for query generation.
F.2 Prompt for File Generation
The prompt for file content generation shown in fig. and fig. and the prompt for file code generation is shown in fig. g and fig. I0
24

---

You are smart reasoner that can restore a query_solvingg scene between human and an agent. Human gives a complex query several images the agent, and then the agent answers the query by searching the Internet and applying tools to the images with step-by-step reasoning. Now, you will be given the query with suggested tools, suggest you analyze the needed information to solve the query, and divide the information into groups: searching from the Internet and extracting from the images using tools. Based on the information from the images, you need to further infer the content of these images, through which the agent could correctly solve the query.
Our toolset: TOOL_SET Output MUST use the following json template.
"information": <Needed information solve the query. For the query including creating/generating images, the information should NOT be the description of the described image.> "information from the Internet": <Information from the Internet inferences based on the given query and suggested tools. Determine which information is suitable to be obtained from the Internet. Or say no information is required from the Internet.>
"information from images": <Information extracted from given images based on the suggested query. should several sentences, including information extracted from images using tools. Determine which information is suitable to be obtained from the images, and using which tools. Do not generate image_ say information is required from the images.>
tools to
mmage_content content the query including generating/creating an image.
"file":
"image_ hanage_numbers": <set int number, the number is depended on needed information from images>,
"image_ contentt:
"image_ 1": <The image content should be natural language, describing the content of
the
first image relevant to the query. The content should be concrete, such concrete number, concrete name. The content should match the query and the above images.> <if you think the query needs more than image, please output image content like 'image_2"s
Figure 7: System prompt for the file content generation.
Now given the query: QUERY, firstly analyze the needed information solve query and divide information into groups: scarching from the Internet extracting from images using tools. Then for information from images, imagine possible answers for each information (it should be concrete answers instead of descriptions). Finally, output the json for the inferenced information and the content of images.
Figure 8: User prompt for the file content generation.
25

---

You are helpful assistant and can generate <file type placeholder> file by writing Python code. You will be given a description the content of file. You need to first largely extend content, and then write Python code to generate a <file type placeholder> file. GUARANTEE that the provided content the file.
The output Python code MUST use the following template.
nen
## extention start
Extened content: <here is the extented content>
## extention end
## code start
<here is the Python code to generate a <file type placeholder> file> ## code end
nen
Figure 9: User prompt for the non-image file generation.
Now, given the following content: <file content>, first largely extend content, and output code to generate <file type placeholder> file, where the file name is <file name> and the file will be saved in <save path>.
Figure 10: User prompt for the non-image file content generation.
F.3 Prompt for Query-file Filter
The prompt for the query-file filter is shown in fig. 11 and fig. 12
E4
Prompt for Step Verifier
To evaluate the quality of intermediate steps taken by an agent during task execution, we design step verifier consisting of two key components: a system prompt and user prompt. The system prompt (Figure 13) provides the verifier model with detailed instructions for evaluating multiple candidate steps (CURRRENT STEP") based on their coherence, logical progression, and effectiveness in advancing the task. guides the model consider contextual alignment with the prior step, tool usage, hallucination, and content relevance. The verifier is required to select the best step and justify its decision in a structured json format.
The user prompt (Figure [4) supplies the concrete input to the verifier, including the task description and list of candidate step sets. Each step set contains the result from the previous step, the current step (thought and code), and the result produced by executing that step. Together, these prompts simulate human-like evaluation process, encouraging the model perform judgment aligned with human preferences in multi-step step reasoning scenarios.
26

---

You are helpful assistant that is given query and several images. You need to check whether the images are relevant the query. query and images are used to evaluate perception ability, reasoning ability, and information search ability of Al agent. The agent solves the query by searching for information on the Web and extracting information from the images. In some cases, based on the given images, the agent could not solve the query, even though it searched for information from the Web (e.g., some specific knowledge). You need to pick up these bad cases.
The agent can call the following tools to solve the query. TOOL_SET.
Thus, the images should follow these requirements.
1. Relevance: The depicted scenarios objects in images should be relevant to the query. The images should contain scenarios or objects that are mentioned in the images.
Usefulness: The image should contain necessary information address the query, such as some specific details that cannot be obtained from the Web.
Some queries require the agent search for knowledge from the Web and combine the information in the image to solve the queries. Thus, in some cases, the images do not contain all the information to solve query, but missed information could searched on the Web. These cases should regarded correct cases.
The output MUST use the following json template to check the images. "information lor__uurrr query": <Required information to solve the query.>,
"useful information in image": <Useful information that can be extracted from images to solve the query>,
"missed_information int.iiiaatiooiinniiieeee in_images" <Missed information that is necessary to solve the query but does not exist in the images.>,
"missed information web_search"' <You need justify whether the missed information could be searched from the Web, using your rich experience in surting the Internet.>,
"missed information_ee "otimaatiooobbtieed"..." <You need to justify whether the missed information could be obtained via computing or reasoning based on information extracted from the images or searched from the Web.>,
"thought": <Now, you need to determine whether the images can solve the query. If the missed information could be searched from the Web obtained based existing information, the images can solve the query. If not, the images cannot solve the query.>,
"correct": <According the above reasoning, you consider the images reasonable for the query to he solved by the tools, set the value to yess, otherwise set the value to 'no" .>,
"updated_ query": :: <If you judge correctness as 'no", please rewrite the query to make more
relevant to the given images. If you judge the correctness as 'yes', please output "no revision is needed."
Figure 11: System prompt for the query-file verification.
Following images, the query: <query>, inference whether the images solve the query based the perception ability, reasoning ability, and information search ability of an Al agent.
Figure 12: User prompt for the query-file verification.
27

---

You are an evaluation assistant responsible for analyzing and evaluating agent trajectories. Your goal is rank <N> "CURRENT_ "TTN" entries based on their coherence, logical progression, and effectiveness in addressing the TASK. as observed in the 'CURENNT RESULT', and their alignment with the PREVIOUS STEP'.
Input Description:
You will receive <N> sets of the following:
"PREVIOUS RESULT": The prior results obtained by agent.
"CURRENT STEP: The agent's output, containing 'thought' and "code" intended to complete the task based on the observation.
"CURRENT_RESULT The result state produced exccuting the "CURRENT_STED
Your Task:
1. Evaluate each 'CURRENT STEP":
-Aseess how well the proposed 'CURRENT STEP aligns with the context established by the
"PREVIOUS_STE STEP" and the observation reflected in the CURRRENT_RESUTT
CCheck for coherence, logical progression, and contextual relevance.
Prioritize outputs that effectively build upon or adapt to the 'PREVIOUS SSEP' while addressing the "CURRENT RESULT"
Sclect the BEST of the 'CURRENT_ STEP "TEPP entries:
Pick the best "CURRENT_STI "TEE" according to the following guidelines.
3. Provide concise explanation for choice: Highlight key factors that influenced your decision, such logical flow, contextual relevance, effectiveness, and uniqueness of the result.
Evaluation Guidelines:
Hallucination: Penalize the directly hallucinated content the code instcad of being produced
from tools.
Tool selection: Pay attention to whether the controller selects the proper tool.
Best content pass into the tool: For the two step that uses the same tool, pay attention to the query
that the controller sends to the tools, such as the "question" in visualizer() and ask search agent().
Task Relevance: Ensure the CURRENT STEP contributes meaningfully solving the task.
Maintain objectivity and avoid assumptions beyond the provided inputs.
Output Format:
Return your evaluation in the following json structure:
"reason": "<concise_ "<comcise_oxplaaatioo"
_wtp_ccaappp_ppceccc _f rankings"
"best_id": <An int that indicates the id for the best Since there are five CURRENT CURRENT_RESULLTS, the id should only be of 1,2,3,4, and 5>,
Figure 13: System prompt for the step verifier.
28

---

The following are the given task, results of previous steps. and result of the current step.
TASK: <task>
Step Sets: <step_sets <et>
Format of set:
#
"PREVIOUS RESULT::
<The prior results obtained by the agent.>
# SERREENT_TTEP'. <The agent's output, containing "thought" and "code" intended to complete the task based on the observation.>
"CURRENT RESULT" <The result state produced by executing the CCURENTT STEP
#
Now, you need determine the best of the current steps based the above information.
Figure 14: User prompt for the step verifier.
G User Study Interface
G.1 Preference Alignment Study
Figure 15a presents the web interface used to evaluate how well automated verifier's preferences align with those of human judges. each trial, participants were shown a single task case along with collection of candidate next-step actions (each consisting of brief "Thought" description and an optional code snippet or tool invocation). These options were the same ones ranked by our verifier, but presented in random order prevent positional bias.
Participants were instructed to review each candidate step and select the one they considered most appropriate for progressing the task. No additional scoring rubric was provided: judges were simply asked to choose the option they "would use" if they were guiding the model. Once selection was made, participants clicked "Submit" to lock in their preference and proceed to the next case.
By comparing the human- human-selected selected option against the top choice of the verifier, we compute an agreement rate for each model and task type. High agreement indicates that the verifier captures human judgments of step quality; lower agreement reveals areas where the verifier's ranking diverges from human intuition. These results were aggregated over 50 cases per participant, enabling both per-case analysis and overall statistics on human-verifier alignment.
G.2 Data quality
Figure 15b illustrates the web interface employed in our user study. For each case, participants proceeded through two consecutive scoring phases:
Task Evaluation.
Reasonableness (1-10): Does the prompt and the displayed interaction trajectory form logical, feasible, and well-defined user request?
1-3: Highly unreasonable or ill-posed.
6: Somewhat reasonable but with noticeable flaws.
7-9: Mostly reasonable, with only minor issues.
10: Fully logical and indistinguishable from genuine queries.
Naturalness (1-10): Is the phrasing realistic, user-like, and fluent?
1-3: Artificial or awkward.
4-6: Acceptable but with unnatural turns of phrase.
- 7-9: Mostly natural, with only minor awkwardness.
10: Fully natural and conversational.

---

Trajectory Evaluation. After reading both the "Chosen" and "Rejected" responses, participants
rated the model's 's proposals on three dimensions, each on 1-10 scale (1 lowest. 10 highest):
Code Accuracy: Correctness and completeness of any code snippets.
Tool Effectiveness: Appropriateness and utility of the suggested APIs or functions.
Content Accuracy: Relevance and factual correctness of the text or image descriptions.
In both phases, brief written examples anchored scores at the low, mid, and high ends promote consistency. Upon completing all six ratings for case, participants clicked "Next" to submit their responses and proceed the following item.
(a) Interface for the user study on verifier performance. Users select their preferred steps.
(b) Interface for the user study on task execution trajectory. Users evaluate the quality of tasks and tra- jectories.
Figure 15: User interface.
30

---

H More Visualization
To further demonstrate the effectiveness and interpretability of our data generation and selection strategy, we present additional case studies in Figure 17 Figure T8] and Figure T6 These visualizations provide qualitative insights into how our method synthesizes diverse and informative training instances, and how the selection mechanism prioritizes samples that contribute meaningfully to model learning.
Query: Can edit the image of my backyard to make look like tropical paradise? Attachment: "image jpg"
Attached Image:
thought
Thought need to ad the imagg to locc alea put eiteru sand,
smage ceiting comminan wis
bli a purecio, the toet rodis irage fand on wsa's reg uset Tuil Inptiel
wases
alve
create tis pronpn that decoribec desined changga to
the lge day claa bus calar.
Image will uar creatry sin prompt Liing image
Fowr laok inportt nereeeeeeee novtthe image
Code:
Enpar iruge prescaus
Code:
Eimeraae howt anowart image wiiking
Code:
aditedd LLA sod
inage purs fhmoan
petrwes"
musil jeagy logeneoo, www.eieernenn.ooommoooooooooo prnnpe))
and feoce tuandion in into tropiie partent,
The
lor"
bachaard ae gras,
resnett's trpaii para or love siani muge
trom other plants Creats
wLWAkarrdeee Jasee, lodhoet outzut.uut
evcrron
eren Expouricv
Fou vtline Faint da kook somre mayia poranios
diservations
PULED atppedd Iispor
be lovingg erren inpat mmap permntor not
ddoveiral Yeat? rmportt
tind yoataiey
lightngg 'aivihe nuke wore look picplaal para se with the
oxcumon saili ustion line "hor Bloa
tarch 2uead INF Set 2un pepodddiaar nii tandom, hall ihhed budted fradone. 1o 191)
% Obsurvadie:
impert Lussorlenetve beseaee of the follow aenon
No Stuarvatio bound from the coda eacctionn fournould sue "ptit furchon heed parr anfemmatin
(sa) pisde, Wal fodeorn (axt Nodior Jonn) to mpsl
Sumoaee
Sansle
Sample
S Verification Reaseess
Suneaies heid cispertt sTrp beuses costing desalied
dicari dessrioos changes
The auppppaan souch anouns that taici wel foathned and rign winh the sui of tranato ring the backyard into troiical peraciis Sampli and Samplid both abwrgpt diacce edi the imuge but opeert
The cock socuution di roi produce any outpus, and cating that thare farwand
Thougic: lhwe enated promot.
newiio ate imgi mraefren lnnagn
Red, will use the image eciting tool to upply than pordiio
rmpiiil paratior Hoseewny conner wdit imagy directty. reed brart creating prompt
Coee:
reloeecoo
with
maan, custon + imuge depicti bechyard with groo. Pooeet, and iditin
Cenerata hor prampt image aeggtadiinng
lenca To
follow chtangee lits prophcal per other
oepical aeril or imngg weitnd iagge path
editiee roste
inagy: rustion Ii imuge deyienc hai
govee
Iy Orencie pool wi oettia and moairo lys Paint the sky tor lu kenso with wooden purpiaa Lignt
tmup, aditarnnttiin Louztin,
Fooero, und
traraaane imae iis trcaical
Jumey day in an. tmplical tnpiicalighinng
mepicil awril linag avaltnee
'elns Add palt treas and ooher tropical planta los Chasts pod
to make the bescqurd fo look los trop tul paradiss with the cime ents lited above
jucked, patt)
with vestrriah and fountaira, you
stytt kooi aucnme
greenseddioe sciidd imap. puet mospe www.eeoooootteeeeeo mege cxption.
witting rool
gi the crean bright roppical lightine Vilathe veltiin tool ahould al mase backyund sok thk cropiccl peradios aineens
mmaee sothe mage.pptt pathi
% Obsavartie:
4 Obsercatio::
seneoorr forne from ceeves ownc mine clonall use jerent Auncion read some information from excution.
21 2430000 1901, Jchwodio
G3 Obserwative
obuorr found from the uode enscation in You
Easrica
Nembrattin flenen:
Samp ab has the cahewwr and logital progension directy usss procioualy greassed sus proiptt to call the image aditing tool which the tragghrbrward relevart approcch
Figure 16: Case study for data generation and selection.
31

---

Query: wanted buy two cheapest visible picture, much would cost? Attachment: "image.jpg"
Attached Image:
thorgght
Thought
Thought:
Thes will the andddldd the
palcuira they told cos, fros, fwll Voualbur logi tecits
thet first lar" sdentifydee price of each unat on then seleet the choppta orau. Alae
mhoggnd uarti miaaly caluuee of
ronur the wares encoffte oucie image
Code:
Code:
peitit oviig "Wosi ecracr the prloes
brack diss deeal berty certon "ahoe orices ohe pretimee patal
paturn", pustrr pe'lt privtarad prorc-il
(Eoetprouu at 100) ingrices)
Step1
ooted, conee suoood
poice dranpeden
fool ansanneol yaat
ccal Loricsa
Obseratberc
two, shoupet paint conslpoiesap
1270, unaco provud sround #238, 1i58.
canc alon falldd fellowa 1g smoor could nox
The price of snacos the image range between ViLL
moot prices around
Semple
Socape
BB Vertication Reasenn
Karpiir" un (iiito StRP reart oofeeeet and aligend with the uock my inmineeees Picaseellyy you lalu red iatort cork paiiee withoe streviri te nover rumreei orings fealt sauadd emorr Trajecionss 1 and Athougs did camplits the proviing anawrr solld lourcetios danahinng the correct inadl pross, raliing the bust option aronn
Though::
Though:
inade with lovest culate cost.
d'eapar cracks congrehan
the minmuur prices let's was a to thit
out the price and thin aot plem to find the teo lovant value.
Gode:
Stepi
(piccs ii pprecastie 19) 100 for pricri inac denae wall
cheapeat atou
and
tnchwopare, pricett
prices 124,, 222, 332, 338, 140, 148, ADO, 156, 000 2700
Choppen (a
Code:
Find two cheppar prices
prices
Code use 2o let tas the rollowing emon inval Wiore
Diserator::
Obsenater:
ritl
Sample
Ventrration Rlason:
Sumeate2 the best OU STEP beciuse conreclly es and sants phopries twe chorpest uniks withest any emoes. Sumplet fass durto reall lama erre, and Sempletrereeett sorts the prices revers order waulingin in the two har prices being silecte.
Thaggh
Theught
choieer mmap Now, well the Wieatser agrint aprin
Now theaest
Heree silr cheeaett
acd thase two price sigether
these prices to caliclaee the tztal coes Let's sasume two sama price
inadl vouallee pouutoeee West yen goat ttragg ind')
Haraped wade maaoortrrak do Juine))
Tex dhoppet price
Obeeratbes
Diseretton
The anuiaa that coe 1&B yar in the image the two puckage lcaned the suction of the Hoplay They
No doservsion found from the code
succtioo. You shoul pnrtt hanasanit
Obsescation:
closed pricn be 128 F'yes soding for spectic anuck at 222 yeu you need to chack another loaat on difiirent rorvs
340
Sample
Sampl
Sumeiie proviccs cohanvet progres on eppitting Resmectly stentifies two cheapett aries caruuatls
uneecesery
Suredizz
identifius the snacis connigorcingg th priees which and 2olk.
lotal sus chappart unuch
saail cast dreaeen stacia picure
The toral now chrapest anude
Code:
Code
EX00
frot sowe The necal over for dheo starte
Stepe
Code ranc dinn halld dan followinng wingiiigggnnnn arrr convenly
Disenwttrr
Cod encurton futee proezeed the salil levala chasutee
upde
Q Oberrvaoon:
[ca menoers ie
Sample
Sampu
vacification Rasson
Soeuling srbllec bevoe thought presnss and legically eruataa ed code that comnally usic "Saal Inowen" function to pansect the satal and for the dhaapee sadde. it aawido caues preuent Sumpli Semplii, aich incomndd conutteraion attnn and meilddhh securs makting it mest offectiw und ligadd with
Figure 17: Case study for data generation and selection.
32

---

Query: In the attached photo, you locate and mark the fruits present the image? Attachment: "image.jpg"
Attached Image:
Thoughtt
Thought Volutiocution
Thought isedito the
derth nubs arowntt image.
persue the snage
Code:
Step
fruts repertiraa anpp peietiiadd
Codtr
"rrapepg"))
prirch
rut, Joan odjed to Irai
Code:
parteh
image
image sperim au, dita boines boundinng bosn for hulls
import smpo su Lowd the tmage
b.a.nndd Joms eppciiicaioon peopece imapet perietrae hoarddngg
"map by raperes repenih gettinaps
Obvevatters
3 Clsernation
123.11 306-201
Pooi ravortin baledd the frilow spreme melll imageaae an Rurertoe applied
the snage foom 1ulll fur incernityy dieainsettee with iiviings balaon. Thowliie, Scepler Luthe Fediastie
Thought
Thought
Thought Yow
borol froi.
Thaught the beunding bocar
the mmep powernt loal.
narted. Yiaa, Jaroeer' lost prow be aiput
Impert metpldlls rystrcin
(inggo "inage
Code
Code
[1906, 340
sonve ireps puit iinge adiepenene "rook thie.
arowet
Ihe en pi sholets un misheeeee spl
pettermege mareemeeseeee prentrmsted, Imagg,
un paist Lartanglllloo Stodtt booo 2too thoost foloo, sdgroalor Inwwith--g
pit www.ghiii.
Observaioo::
Oisesatio::
0O0T0R77 ruso ang
Eication
Somple iins beet
Figure 18: Case study for data generation and selection.
33


Reviews:
[{"review_id": "h0sUYRpVOg", "reviewer": "Reviewer_VNFD", "summary": "This paper introduces SPORT, an iterative self-exploration framework for training multimodal agents in tool usage without any annotated data. It enables the agent to autonomously generate tasks, explore tool actions, and collect step-wise preference feedback via an AI verifier. The agent is then optimized using DPO based on this feedback. Experimental results on GTA and GAIA benchmarks show that SPORT significantly outperforms strong supervised and DPO baselines, and achieves competitive results with some closed-source models. The framework demonstrates that agents can improve tool usage capability purely through self-exploration and AI feedback, providing a scalable solution for complex multimodal tasks.", "questions": "1. While the SPORT framework uses DPO for preference tuning, is there a reason why fully online RL-based methods such as GRPO or PPO-style optimization were not adopted?\n2. If I directly apply the SPORT framework or preference tuning on MAT-Qwen2-VL-7B, rather than just the base Qwen2-VL-7B, would the agent performance improve further? Has this variant been tried?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Annotation free: SPORT does not rely on any human-annotated data or pre-collected trajectories, which addresses a major bottleneck in current multimodal agent research and greatly reduces data collection costs.\n2. Novel online self-exploration: The method leverages an iterative, online self-exploration and preference optimization process, allowing the agent to adaptively discover effective strategies and improve performance through interaction with the environment.\n\nWeaknesses:\n1. The paper does not always specify the exact models used for each component (e.g., task synthesis, file generation, step verification, filtering). More explicit clarification on which LLM or VLM is used for each step, including possible ablation on model choices, would improve reproducibility and help understand the framework's sensitivity to model selection. Could the authors provide more detailed information on the models, prompts, and configurations for all stages of the pipeline?\n2. No Comparison with True Online RL: Although they ablate DPO vs. static DPO, they never compare against an online RL baseline (e.g., PPO or GRPO) on the same framework. It is unclear whether SPORT is merely an intermediate between offline DPO and true online RL, or if incorporating conventional RL techniques could yield further gains. A direct comparison to PPO/GRPO baselines would clarify SPORT’s position on the spectrum from static preference tuning to fully online reinforcement learning.", "was_revised": true, "final_justification": "Most of my concerns are addressed. I will maintain my score.", "rebuttal": "Thank you for the detailed and thoughtful review. We appreciate your recognition of SPORT's ability to train multimodal agents entirely without human-annotated data, as well as the novelty of our self-exploration and preference optimization framework. To simplify the discussion, we use **W** for each weakness, **Q** for each question, and **A** for our response. For clarity, all mentions of Qwen2-VL-7B and Qwen2.5-7B refer to their Instruct variants. We address your comments one by one below.\n\n> **W1.** The paper does not always specify the exact models used for each component (e.g., task synthesis, file generation, step verification, filtering). More explicit clarification on which LLM or VLM is used for each step, including possible ablation on model choices, would improve reproducibility and help understand the framework's sensitivity to model selection. Could the authors provide more detailed information on the models, prompts, and configurations for all stages of the pipeline?\n\n**A.** We will add the following details about model choices into the main manuscript.\n\n- *Task Synthesis* (including query generation, file generation and task revision&filtering)：Qwen2-VL-7B\nWe chose Qwen2-VL-7B for task synthesis. To assess sensitivity to this choice, we conducted a human study (detailed in Appendix C.1), comparing task quality generated by Qwen2-VL-7B versus GPT-4o-mini on \"Naturalness\" and \"Reasonableness\". As shown in Table 7, both models produce tasks of comparable quality, demonstrating that Qwen2-VL-7B is robust and capable of high-quality task generation.\n\n**Table 7.** User study for open-source vs. closed-source models generated tasks. Scores are scaled from 1 to 10, and a higher score denotes better quality.\n\n| Model| Task Naturalness | Task Reasonableness |\n|-|-|-|\n| GPT-4o-mini|8.71|8.37|\n| Qwen2-VL-7B|8.75|8.35|\n\n- *Controller*: Tuned-Qwen2-VL-7B\n- *Step Verification*: Qwen2.5-7B\n\nWe prioritized open-source models to ensure reproducibility and facilitate future research.  We selected Qwen2.5-7B as the verifier because qualitative analysis showed it performed better in evaluating code accuracy and tool selection appropriateness compared to Qwen2-VL-7B. We attribute this to Qwen2-VL-7B's image inputs occupying significant context window space, limiting available context for verifier instructions. Quantitative experiments confirm this: re-generating preference annotations for using Qwen2-VL-7B as verifier yielded 55.13% accuracy on the GTA benchmark, while Qwen2.5-7B achieved 60.26% (Table A), validating our choice. Additionally, our human evaluation (lines 324–330 and Table 5) shows that Qwen2.5-7B achieves an 82% alignment rate in step verification with human participants, confirming the effectiveness of our design choice.\n\n\n**Table A.** Ablation about AI verifier, accuracies (%) on GTA benchmark.\n| Baseline | Qwen2-VL-7B as Verifier | Qwen2.5-7B as Verifier |\n|-|-|-|\n| 53.85| 55.13 |60.26|\n\n\n\n\n- *Prompts and Configurations*: All system prompts are provided in Appendix E (Figures 6-14). Training configurations are detailed in Section 4.1. Code and data will be open-sourced for replication.\n\n\n> **W2.** No Comparison with True Online RL: Although they ablate DPO vs. static DPO, they never compare against an online RL baseline (e.g., PPO or GRPO) on the same framework. It is unclear whether SPORT is merely an intermediate between offline DPO and true online RL, or if incorporating conventional RL techniques could yield further gains. A direct comparison to PPO/GRPO baselines would clarify SPORT’s position on the spectrum from static preference tuning to fully online reinforcement learning.\n\n> **Q1.** While the SPORT framework uses DPO for preference tuning, is there a reason why fully online RL-based methods such as GRPO or PPO-style optimization were not adopted?\n\n**A.** Thank you for this excellent question. We have not adopted a fully online RL like PPO/GRPO for the following reasons.\n\n1. Absence of reliable rewards. SPORT targets multimodal reasoning tasks *without ground-truth answers or labels*. On-policy methods like PPO/GRPO are sensitive to reward quality, and constructing reliable reward signals in this setting is non-trivial. Inaccurate rewards often lead to unstable or ineffective training.\n\n2. Low trajectory success rates. In complex multimodal scenarios, most sampled trajectories fail, making it difficult to identify successful rollouts for policy updates in PPO/GRPO. SPORT's step-wise DPO approach better utilizes the correct steps within failed trajectories, improving data utilization.\n\n3. High sampling cost and low efficiency. Each task requires multi-step interactions with real-world tools (e.g., web search, OCR, code execution), which are both computationally expensive and time-intensive. This results in very low sampling throughput, significantly slowing down the collection of trajectories needed for on-policy updates. As a result, applying PPO/GRPO becomes infeasible in our setting.\n\nGiven these challenges around reward quality, task difficulty, and sampling efficiency, SPORT uses Step-wise DPO for practical and stable training. In future work, we plan to explore ways to address these sampling environment challenges, such as designing systems for more realistic, fast, and accurate rewards.\n\n\n>**Q2.** If I directly apply the SPORT framework or preference tuning on MAT-Qwen2-VL-7B, rather than just the base Qwen2-VL-7B, would the agent performance improve further? Has this variant been tried?\n\n\n\n**A.** Thank you for your suggestion. SPORT method already incorporates SFT on MAT data, as described in *lines 193–196* of the manuscript. To provide a more comprehensive analysis, we additionally applied SPORT directly to the base Qwen2-VL-7B without MAT.\n\nThe results in Table B show that SPORT improves performance in both cases. Applying SPORT to the MAT baseline yields the best accuracy (60.26%), while applying it directly to the base Qwen2-VL-7B also results in a substantial gain (55.13% vs. 42.31%). The 5.13% additional improvement over MAT alone demonstrates that SPORT can effectively complement existing tuning methods, and can serve as a general enhancement strategy for already-trained agents. These results will be included in the revised manuscript.\n\n**Table B.** Answer accuracies(%) on the GTA benchmark.\n|Qwen2-VL-7B | MAT-Qwen2-VL-7B|SPORT-Qwen2-VL-7B|SPORT-MAT-Qwen2-VL-7B|\n|-|-|-|-|\n|42.31|53.85|55.13|60.26|"}, {"review_id": "4kI4ODZMeG", "reviewer": "Reviewer_rUdq", "summary": "This paper presents SPORT (step-wise preference optimization to refine the trajectories of tool usage), a framework for training multimodal agents to use tools effectively without requiring pre-collected human annotations. Current multimodal agent training relies heavily on expensive human-annotated trajectories and tool usage data. This creates bottlenecks and may lead to poor generalization due to distribution mismatch with target environments. On the contrary, SPORT operates without human annotations through four iterative components: 1) Task Synthesis: Uses LLMs to generate diverse multimodal tasks (queries + files). 2) Step Sampling: Agent samples multiple candidate actions for each step. 3) Step Verification: AI verifier evaluates and ranks step-level actions based on quality. 4) Preference Tuning: Uses Direct Preference Optimization (DPO) to fine-tune the agent controller.\n\nResults on GTA and GAIA benchmarks show that SPORT achieves 6.41% and 3.64% improvements over supervised fine-tuning baselines, demonstrating effectiveness in multimodal reasoning tasks.", "questions": "1. Computational cost: Seems like the proposed method requires expensive tool execution (LLM calls, web searches) during exploration. The reviewer hope the author could provide more analysis on computational cost estimation and comparison.\n\n2. Selection of the LLM model in task synthesis and AI verifier. The reviewer is wondering how the LLM model and AI verifier selection would influence the final results. It seems that in the first three steps, all the knowledge comes from these two model (and maybe the unlabeled data). However, there is no ablation study on what influences these two models bring in the whole framework. It would be valuable to provide some experiment results on this.\n\n3. Influence of unlabeled data used in the training. It would be better to provide experiment results on using different portion of unlabeled data for training the framework.", "limitations": "1. Computational cost: Seems like the proposed method requires expensive tool execution (LLM calls, web searches) during exploration. The reviewer hope the author could provide more analysis on computational cost estimation and comparison.\n\n2. Selection of the LLM model in task synthesis and AI verifier. The reviewer is wondering how the LLM model and AI verifier selection would influence the final results. It seems that in the first three steps, all the knowledge comes from these two model (and maybe the unlabeled data). However, there is no ablation study on what influences these two models bring in the whole framework. It would be valuable to provide some experiment results on this.\n\n3. Influence of unlabeled data used in the training. It would be better to provide experiment results on using different portion of unlabeled data for training the framework.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. Self-Improving Framework: The iterative nature allows agents to continuously improve through interaction with real environments\n2. Evaluation: Tests on two established benchmarks (GTA, GAIA).\n3. Practical Design: The proposed method works with open-source models (Qwen2-VL-7B), generates diverse synthetic tasks covering multiple file types and achieves competitive performance against closed-source models.", "was_revised": true, "final_justification": "The reviewer's concerns are resolved by the response. The reviewer is willing to maintain the positive score.", "rebuttal": "Thank you for your insightful and constructive feedback. We’re glad you found SPORT’s annotation-free, self-improving framework practical and effective, particularly its ability to scale using only open-source models and synthetic tasks. To make our replies easier to follow, we label each Question as **Q**, and our corresponding Answer as **A**. All references to Qwen2-VL-7B and Qwen2.5-7B denote their Instruct versions. We address your comments one by one below.\n\n\n> **Q1.** Computational cost: Seems like the proposed method requires expensive tool execution (LLM calls, web searches) during exploration. The reviewer hope the author could provide more analysis on computational cost estimation and comparison.\n\n\n**A.** Thanks for your suggestion. We present a detailed analysis of the computational cost and time required for sampling and training on 500 tasks within the SPORT framework, summarized in Table C.\n\nTable C encompasses time for task synthesis, tool calling, model inference, and model updating. Notably, tool calling, particularly web searches via the Google Search API, accounts for a significant portion of the time due to real-time external interactions. We will incorporate this detailed computational cost analysis into the revised manuscript.\n\n**Table C.** Time and computation cost estimation for each stage in the SPORT framework for 500 tasks.\n|Stage|Model|Time(mins)|Time Proportion|Config|\n|-|-|-|-|-|\n|Task Synthesis (Inference)|Qwen2-VL-7B|54.18|12.47%|1$\\times$A100 80G GPU|\n|Step Sampling (Inference)|Tuned-Qwen2-VL-7B|68.02|15.65%|1$\\times$A100 80G GPU|\n|Step Verification (Inference)|Qwen2.5-7B|50.45|11.61%|1$\\times$A100 80G GPU|\n|Tool calling|Tool Models + Google Search API|147.87|34.04%| 1$\\times$A100 80G GPU|\n|DPO Training|Tuned-Qwen2-VL-7B|114.00|26.23%|8$\\times$A100 80G GPU|\n|Total|N/A|434.53|100.00% | ~20 A100 GPU Hours Total |\n\n> **Q2.** Selection of the LLM model in task synthesis and AI verifier. The reviewer is wondering how the LLM model and AI verifier selection would influence the final results. It seems that in the first three steps, all the knowledge comes from these two model (and maybe the unlabeled data). However, there is no ablation study on what influences these two models bring in the whole framework. It would be valuable to provide some experiment results on this.\n\n**A.** We thank the reviewer for this valuable question. Here we provide more experimental results and analysis about this.\n\n**Task Synthesis:** We chose Qwen2-VL-7B for task synthesis. To assess sensitivity to this choice, we conducted a human study (detailed in Appendix C.1), comparing task quality generated by Qwen2-VL-7B versus GPT-4o-mini across Naturalness and Reasonableness dimensions. As shown in Table 7, both models produced tasks of comparable quality, demonstrating that Qwen2-VL-7B is robust and capable of high-quality task generation.\n\n**Table 7.** User study for open-source vs. closed-source models generated tasks. Scores are scaled from 1 to 10, and a higher score denotes better quality.\n\n| Model| Task Naturalness | Task Reasonableness |\n|-|-|-|\n| GPT-4o-mini|8.71|8.37|\n| QWen2-VL-7B|8.75|8.35|\n\n**AI Verifier.** We prioritized open-source models to ensure reproducibility and facilitate future research. We selected Qwen2.5-7B as the verifier because qualitative analysis showed it performed better in evaluating code accuracy and tool selection appropriateness compared to Qwen2-VL-7B. We attribute this to Qwen2-VL-7B's image inputs occupying significant context window space, limiting available context for verifier instructions. Quantitative experiments confirmed this: re-generating preference annotations for using Qwen2-VL-7B as verifier yielded 55.13% accuracy on the GTA benchmark, while Qwen2.5-7B achieved 60.26% (Table A), validating our choice.\n\n**Table A.** Ablation about AI verifier, accuracies (%) on GTA benchmark.\n| Baseline | Qwen2-VL-7B as Verifier | Qwen2.5-7B as Verifier |\n|-|-|-|\n| 53.85| 55.13 |60.26|\n\nAdditionally, our human evaluation (lines 324–330 and Table 5) shows that Qwen2.5-7B achieves an 82% alignment rate in step verification with human participants, confirming the effectiveness of our design choice.\n\n> **Q3.** Influence of unlabeled data used in the training. It would be better to provide experiment results on using different portions of unlabeled data for training the framework.\n\n**A.** Thank you for your valuable suggestion. We have added ablation experiments using different portions of unlabeled data for training. The experimental results are shown in Table D. The results show that SPORT's performance improves consistently with more unlabeled data, achieving gains of 1.92% (4K→8K) and 2.57% (8K→16K). This indicates that our method can effectively utilize self-generated preference data to improve tool usage capabilities. \n\n**Table D.** Answer accuracies (AnsAcc) on GTA benchmark with varying amounts of unlabeled data used in DPO training.\n|Unlabeled data num|AnsAcc (%)|\n|-|-|\n|4K (25%)|55.77|\n|8K (50%)|57.69|\n|16K (100%)|60.26|"}, {"review_id": "m565Zuf3au", "reviewer": "Reviewer_avvW", "summary": "This paper presents SPORT, a framework for training multimodal tool-use agents using step-wise preference optimization. SPORT builds its own preference data via self-exploration: at each decision step, it samples multiple tool-usage actions, executes them, and uses an LLM-based verifier to select the most desirable one. Crucially, the verifier compares outcomes in terms of usefulness rather than correctness, providing more flexible supervision. The authors evaluate SPORT on two multimodal benchmarks: GTA (image-based reasoning) and GAIA (document-based tasks).", "questions": "- How sensitive is SPORT to the quality and diversity of the synthetic tasks generated in the early stages of self-exploration?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- SPORT demonstrates clear improvements over SFT baselines and performs competitively with agents built on top of strong models like GPT-4o.\n- The full training pipeline, including task generation, tool sampling, and preference labeling, is entirely self-supervised, requiring no human-annotated trajectories.\n\n### Weaknesses\n- The main components of SPORT, including self-instruct task generation, LLM-based comparison, and DPO, are all established techniques. While the paper positions itself as \"explores the tool usage by itself via an iterative manner\", it does not introduce any novel mechanisms for agent-level self-improvement, such as strategic planning, error reflection, or hierarchical reasoning. Instead, it looks more like a specific application of existing methods to tool-usage domain with tool-based actions as action space. Due to the limited novelty, I put a low confidence score. \n- While very relevant related works such as Step-DPO are discussed, it is not included in experimental comparisons. A fair adaptation of such methods to the tool-use setting would strengthen the empirical claims. For example an adapted version of Step-DPO that does error localization and correction.", "was_revised": true, "final_justification": "The authors provide additional experiments, which help address some of my concerns. My only remaining concern is still W1, regarding the novelty of the work. However, I'm not an expert in this specific domain, so I maintain a low confidence score.", "rebuttal": "Thank you for your insightful and constructive feedback. We appreciate your recognition of SPORT's fully self-supervised training pipeline and its ability to perform competitively with stronger models like GPT-4o, all without any human-annotated trajectories. For simplicity, we mark each Weakness as **W**, each Question as **Q**, and our Answer as **A**. Throughout this response, Qwen2-VL-7B and Qwen2.5-7B refer to the Instruct versions. We address your comments one by one below.\n\n> **W1.** The main components of SPORT, including self-instruct task generation, LLM-based comparison, and DPO, are all established techniques. While the paper positions itself as \"explores the tool usage by itself via an iterative manner\", it does not introduce any novel mechanisms for agent-level self-improvement, such as strategic planning, error reflection, or hierarchical reasoning. Instead, it looks more like a specific application of existing methods to tool-usage domain with tool-based actions as action space. Due to the limited novelty, I put a low confidence score.\n\n**A.** We appreciate the reviewer’s feedback. SPORT addresses a different but important challenge compared with agent mechanisms: *how can agents autonomously discover effective tool usage strategies in multimodal scenarios through self-exploration when no ground truth exists?*\n\nOur key contribution is **a tool usage exploration framework** for multimodal agents, which integrates task synthesis, parallel step sampling, step verification, and DPO updates. In this case, the agent could learn tool usage without any pre-collected data. While based on existing components, SPORT repurposes them into a new training paradigm that supports scalable self-improvement without manual supervision. SPORT deliberately focuses on learning tool usage primitives by itself, which is **orthogonal to agent abilities like strategic planning, error reflection, or hierarchical reasoning**. These abilities are not in conflict with our goal and can be naturally incorporated into our framework to further enhance agent capabilities.\n\n> **W2.** While very relevant related works such as Step-DPO are discussed, it is not included in experimental comparisons. A fair adaptation of such methods to the tool-use setting would strengthen the empirical claims. For example an adapted version of Step-DPO that does error localization and correction.\n\n**A.** Thank you for the suggestion. We have conducted experiments comparing our method with an adapted variant of Step-DPO[a] for the multimodal agent setting. Specifically, we use GPT-4o-mini to sample positive step-level responses for the same tasks used in SPORT, while responses from MAT-Qwen2-VL-7B serve as negative examples. This allows us to construct step-level preference pairs for DPO training. The resulting model, MAT-Qwen2-VL-7B-DPO, shows modest improvements over the SFT baseline (e.g., +1.28% AnsAcc, +2.67% ToolAcc), as shown in Table 3. In contrast, our SPORT method achieves larger gains across all metrics, demonstrating the effectiveness of our adaptive self-exploration framework under unlabeled multimodal settings.\n\n**Table 3.** Ablation for MAT-SFT vs. MAT-SFT-DPO vs. SPORT on the GTA benchmark.\n|Method|AnsAcc (%)|ToolAcc (%)|CodeExec (%)|\n|-|-|-|-|\n|MAT-Qwen2-VL-7B |53.85|64.63|84.32|\n|MAT-Qwen2-VL-7B-DPO|55.13|67.30|85.90|\n|SPORT (Ours) |60.26|72.41|91.87|\n\n[a] Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs.\n\n> **Q1.** How sensitive is SPORT to the quality and diversity of the synthetic tasks generated in the early stages of self-exploration?\n\n\n**A.** Task quality and diversity in early stages do have an impact on SPORT’s performance. We conducted an ablation for task synthesis, where the in-context examples were reduced from 20 to 5 and the task seed pool was narrowed from 425 to 100. This setup was designed to mimic a low-diversity scenario in early-stage exploration. As shown in Table E, this led to a moderate drop in performance (60.26 → 58.33). However, the result still significantly outperforms the SFT baseline (58.33 vs. 53.85).\n\n\n**Table E.** Impact of task diversity on the GTA benchmark.\n|Method|AnsAcc (%)|\n|-|-|\n|MAT-Qwen2-VL-7B |53.85|\n|SPORT w/ 5 from 100 seeds| 58.33 |\n|SPORT  w/ 20 from 425 seeds|60.26|"}, {"review_id": "2ERSugxdAW", "reviewer": "Reviewer_Zye7", "summary": "This paper introduces SPORT (Step-wise Preference Optimization for Refining Trajectories), a novel method for training multimodal agents without requiring pre-collected human annotations. Current methods for training multimodal agents rely heavily on expensive human-annotated preference pair data or distilled tool usage trajectories data from closed-sourced APIs, which creates a bottleneck for developing agents that can handle complex multimodal tasks. SPORT enables agents to automatically improve their tool usage capability through self exploration and iterative refinement. This method consists of four iterative components: \n\n1. **Task generation**: synthesizes diverse multimodal tasks (queries + files) using LLMs. \n2. **Step sampling**: the agents explores multiple possible actions at each step by sampling different tool calls.\n3. **Step verification**: a verifier evaluates the sampled actions to identify the preferred option for each step.\n4. **Preference tuning**: uses the collected step-wise preference data to update the agent controller via DPO.\n\n**Main contributions**:\n\n* A self-exploration framework that eliminates the need for pre-collected annotations.\n* Step-wise preference optimization enables improvements over SFT baselines on GTA and GAIA benchmarks.\n* A dataset of 16K preference data samples is created through the exploration process.\n\nThrough multiple ablation experiments and user study, this paper demonstrates that agents can effectively learn to use tools through autonomous exploration, which provides an alternative to traditional supervised approaches.", "questions": "1. **Task seed collection and diversity**: \nThe paper says \"we collect seed from the existing method MAT\" but doesn't clarify the specific source or scale. Could you please clarify the following: Did you sample seed tasks directly from the 20K MM-Traj dataset, or did you use the MAT framework to generate new seed tasks? How many seed tasks were used in total for task synthesis? And lastly, how did you ensure diversity across task categories? (e.g., did you sample uniformly across MAT's 16 non-overlapping categories mentioned in their paper?) \n2. **Verifier details**: \n What model(s) do you use for the verifier? How did you decide on the selected verifier or evaluate the verifier? Your appendix has a section for the prompt for step verifier, but it would be helpful to include more technical details and some analysis in the main section (e.g. Section 3.3).\n3. **Experimental baselines and comparisons**: \nIt’s great that you list the GTA and GAIA results for many combinations of methods and controllers. Do you also have the GTA and GAIA results for the SPORT Agent with the initial agent controller (the post-SFT model, i.e., the MAT-Qwen2-VL-7B)? This might create a more direct comparison showing SPORT's improvement over the same base model. Also, did you conduct experiments with your agent using closed-source controllers? I'm curious if with the same closed-source model, your agent would perform better than other agents (e.g. HFAgent). \n4. **Efficiency and cost analysis**: \nI think it might be helpful to have some discussion about the compute time and computational costs. For example, a high-level comparison between SPORT and conventional methods. If the gains are much larger than the overhead, this would strengthen the contribution.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Quality\n**Strengths**: \n* Technical merit: The paper presents a well-design iterative framework with clear technical components. Its step-wise preference optimization approach is theoretically grounded for DPO. \n* Experimental validation: The method is experimented on two popular benchmarks (GTA and GAIA) with extensive ablation studies. The authors also conduct detailed statistic analysis on the generated preference data in three aspects (code error rate, tool selection, and content variations). \n* Human validation: The work has thorough validation including user studies to validate both data quality and verifier alignment with human preferences (82% agreement rate).\n\n**Weaknesses**:\n* To my understanding, the verifier component seems to be an important element in this system. The paper would benefit from more details and analysis of its implementation and impact on performance. \n\n## Clarity\n**Strengths**: \n* Well-structured presentation: The paper follows a logical flow from problem motivation through method description to experimental validation.\n* Clear visualizations: The figures and tables in the paper effectively illustrate the pipeline (Figure 1), search scheme (Figure 2), concrete examples of preference data construction (Figure 4 and 5), and training process through algorithm. \n* Clear communication and replication support: The authors presents technical concepts with clear and accessible language while providing comprehensive implementation details about model, training, and prompt templates.\n\n**Weaknesses**:\n* Verifier design: While prompt templates are included in the appendix, the main paper sections could better elaborate on the verifier's design choices and implementation details. A more thorough discussion of the verifier's architecture, design rationale, and key parameters would enhance the paper's technical depth.\n\n## Significance\n**Strengths**: \n* SPORT removes the dependency on expensive human annotations for multimodal agent training, which solves a key practical challenge.\n* This method works with existing open-source models (Qwen2-VL-7B) and standard toolkits, making it accessible to the broader community.\n* The 16K preference dataset generated will be valuable for future experiments and research.\n\n**Weaknesses**:\n* Future performance potential: While the experiments show promising results with open-source models, it has not yet matched closed-source model based agents' performance. This gap suggests potential for future improvements.\n\n## Originality\n**Strengths**: \n* The idea of constructing preference data at the step level rather than trajectory level is innovative and allows learning from partial successes.\n* The system effectively combines task synthesis, online exploration, and preference learning in a novel way for multimodal settings.\n\n**Weaknesses**:\n* The work seems effectively build upon and enhances the MAT framework and MM-Traj dataset. Although effective, more original architectural contributions could further strengthen the work's novelty.", "was_revised": true, "final_justification": "After reviewing the author's rebuttal and our discussion, I maintain my recommendation to Accept this paper.\n\nResolved Issues:\n- Verifier details: Authors clearly explained using Qwen2.5-7B as verifier, with ablation results showing 60.26% vs 55.13% accuracy on GTA compared to using Qwen2-VL-7B. This choice makes sense.\n- Computational costs: The efficiency analysis shows SPORT actually uses less compute than the MAT baseline (82 vs 122 GPU hours), which strengthens the contribution.\n- Task diversity: Clarified use of 425 human-written seeds with random sampling for diversity—this is reasonable.\n\nRemaining Limitations (Acceptable):\n- Performance gap with closed-source models: While SPORT underperforms on complex benchmarks like GAIA, this is due to model scale limitations rather than the method itself. Given the focus on open-source accessibility, this is understandable.\n- Building on existing work: Though not architecturally novel, the step-wise preference optimization genuinely addresses an important problem.\n\nKey Strengths: The paper tackles a real bottleneck (expensive human annotations for agent training) with a practical solution. The experimental validation is thorough (including 82% human-verifier alignment), and the method works with accessible open-source models. The generated 16K preference dataset will benefit future research.\n\nBottom Line: This is a technically solid paper with clear practical value. The step-wise preference optimization is well-executed and addresses a genuine need in the multimodal agent community. The authors have satisfactorily addressed my concerns, and the remaining limitations don't diminish the core contribution.", "rebuttal": "Thank you for your comprehensive review. We’re especially grateful for your recognition of the technical design, experimental depth, and practical impact of our step-wise preference optimization framework. To streamline our response, we denote each Weakness as **W**, each Question as **Q**, and our Answer as **A**. For brevity, Qwen2-VL-7B and Qwen2.5-7B refer to their Instruct versions. We address your concerns one by one below.\n\n> **W1** To my understanding, the verifier component seems to be an important element in this system. The paper would benefit from more details and analysis of its implementation and impact on performance.\n\n> **W2.** Verifier design: While prompt templates are included in the appendix, the main paper sections could better elaborate on the verifier's design choices and implementation details. A more thorough discussion of the verifier's architecture, design rationale, and key parameters would enhance the paper's technical depth.\n\n> **Q2.** Verifier details: What model(s) do you use for the verifier? How did you decide on the selected verifier or evaluate the verifier? Your appendix has a section for the prompt for step verifier, but it would be helpful to include more technical details and some analysis in the main section (e.g. Section 3.3).\n\n**A.** Thanks for your suggestion. We prioritized open-source models to ensure reproducibility and facilitate future research.   We selected Qwen2.5-7B as the verifier because qualitative analysis showed it performed better in evaluating code accuracy and tool selection appropriateness compared to Qwen2-VL-7B. We attribute this to Qwen2-VL-7B's image inputs occupying significant context window space, limiting the ability for verification. Quantitative experiments confirmed this: re-generating preference annotations for using Qwen2-VL-7B as verifier yielded 55.13% accuracy on GTA benchmark, while Qwen2.5-7B achieved 60.26% (Table A), validating our choice.\n\n**Table A.** Ablation about AI verifier, accuracies (%) on GTA benchmark.\n| Baseline | Qwen2-VL-7B as Verifier | Qwen2.5-7B as Verifier |\n|-|-|-|\n| 53.85| 55.13 |60.26|\n\nAdditionally, our human evaluation (lines 324–330 and Table 5) shows that Qwen2.5-7B achieves an 82% alignment rate in step verification with human participants, confirming the effectiveness of our design choice.\n\n> **W3.** Future performance potential: While the experiments show promising results with open-source models, it has not yet matched closed-source model based agents' performance. This gap suggests potential for future improvements.\n\n**A.** Thank you for this thoughtful suggestion. We acknowledge there is substantial room for future improvement. While the SPORT Agent demonstrates strong performance on GTA with open-source models, it underperforms closed-source based agents on the more complex GAIA benchmark. This underperformance stems primarily from the limited parameter scale of open-source base models, which constrains their reasoning capabilities. In contrast, closed-source models like GPT-4o achieve superior performance through their larger model parameters, more extensive training data, and longer context windows, enabling enhanced reasoning abilities. We plan to explore stronger open-source backbones and improved verifier generalization in future work.\n\n> **W4.** The work seems effectively build upon and enhances the MAT framework and MM-Traj dataset. Although effective, more original architectural contributions could further strengthen the work's novelty.\n\n**A.** Thank you for the valuable feedback. Our contribution primarily lies in the design of an iterative tool usage exploration framework for multimodal agents. Specifically, we address an orthogonal yet important challenge to architectural design: *enabling agents to autonomously discover effective tool usage strategies through self-exploration, especially in the absence of ground-truth supervision*. We agree that architectural innovations are also vital for advancing agent capabilities, and we look forward to exploring this direction in future work to further enhance the capabilities of agents.\n\n> **Q1.** Task seed collection and diversity: The paper says \"we collect seed from the existing method MAT\" but doesn't clarify the specific source or scale. Could you please clarify the following: Did you sample seed tasks directly from the 20K MM-Traj dataset, or did you use the MAT framework to generate new seed tasks? How many seed tasks were used in total for task synthesis? And lastly, how did you ensure diversity across task categories? (e.g., did you sample uniformly across MAT's 16 non-overlapping categories mentioned in their paper?)\n\n**A.** We directly adopted the 425 human-written task seeds from the task generation process in MAT [b], as these seeds offer high quality via their manual curation. To ensure diversity in the generated tasks, we randomly sample and shuffle 20 seeds as in-context examples for each query generation. Additionally, when generating tasks, we provide the complete toolset descriptions to the model to facilitate more reasonable and diverse task outputs.\n\n\n[b] Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage.\n\n> **Q3.** Experimental baselines and comparisons: It’s great that you list the GTA and GAIA results for many combinations of methods and controllers. Do you also have the GTA and GAIA results for the SPORT Agent with the initial agent controller (the post-SFT model, i.e., the MAT-Qwen2-VL-7B)? This might create a more direct comparison showing SPORT's improvement over the same base model. Also, did you conduct experiments with your agent using closed-source controllers? I'm curious if with the same closed-source model, your agent would perform better than other agents (e.g. HFAgent).\n\n**A.** To ensure a fair comparison, the SPORT Agent uses the same toolset (lines 212-213 in the manuscript) and agent architecture as the T3-Agent and HF Agent. So the results in Table 1 already provide a direct comparison among MAT-Qwen2-VL-7B (using T3-Agent), original Qwen2-VL-7B (using HF Agent), closed-source models (using HF Agent), and our method, where only the controllers are different. As shown in Table 1, our controller already has comparable, and even better performance than closed-source models in the GTA benchmark, but worse performance in the GAIA benchmark, mainly due to the complexity of GAIA. We plan to explore stronger open-source backbones in future work.\n\n> **Q4.** Efficiency and cost analysis: I think it might be helpful to have some discussion about the compute time and computational costs. For example, a high-level comparison between SPORT and conventional methods. If the gains are much larger than the overhead, this would strengthen the contribution.\n\n\n**A.** Thank you for the insightful suggestion. We add a high-level comparison of compute time and GPU usage between the baseline method (MAT) and our SPORT framework. Table F summarizes the estimated compute time and GPU cost for both methods. \n\nThe computational cost of *Task Synthesis*, *Step Sampling*, and *Tool Calling* is closely tied to the data scale used in the data generation stage. MAT synthesizes and samples steps for 20K tasks and 20K trajectories, resulting in high overhead across all three components. SPORT performs Task Synthesis on 2K tasks and constructs 16K preference pairs for tuning, leading to reduced compute time in these stages.\n\nIn addition, SPORT introduces an extra Step Verification process to obtain step-level preferences. While this stage incurs some additional costs, it remains acceptable relative to the total runtime due to the smaller data volume.\n\n\n**Table F.** Estimated compute time and GPU cost comparison between MAT and SPORT. GPU hours (GPUh) are measured as usage of a single A100 80GB GPU for one hour. The estimates for MAT are derived from its reported tool invocation frequency, combined with empirical measurements of the GPT-4o-mini API and tool execution latency on our hardware.\n\n\n| Component| MAT Time (h) | SPORT Time (h) | MAT Cost (GPUh) | SPORT Cost (GPUh) |\n|-|-|-|-|-|\n|***Data Generation***|20K tasks, 20K trajectories |2K tasks, 16K perference data |||\n|Task Synthesis | 29.13| 3.61 | 0 (GPT-4o-mini API ~500\\$)| 3.61 (Qwen2-VL-7B)|\n|Step Sampling| 69.92 | 4.53| 0 (GPT-4o-mini API  ~1500\\$)  | 4.53 (Tuned-Qwen2-VL-7B)|\n|Step Verification| 0 | 3.36| 0 | 3.36 (Qwen2.5-7B)|\n| Tool Calling | 58.93 | 9.86 | 58.93|9.86 |\n| ***Training*** | |  |  | |\n| Model Training w/ 4$\\times$ A100 80G GPU|15.77|15.20| 63.08| 60.80 |\n| **Total**|173.75 |36.56|122.01|82.16|\n\n\n> **Paper Formatting Concerns** The paper is well-formatted overall. A few minor editorial suggestions:\n> 1. In Section 1 Introduction (line 37): “Based on the above observation, we expect that the agent automatically generates tasks” - should be \"expect that agents automatically\" (plural consistency).\n> 2. In Section 5 Conclusion (line 354): \"causing inferior generalization for some outlines\" - \"outlines\" seems like a typo, possibly meant \"outliers\".\n> 3. In Appendix C Task Generation (line 559): “we first use an LLM to generate queres” - \"queres\" seems like a typo, possibly meant \"queries\".\n\n**A.** We sincerely thank the reviewer for the careful reading and for pointing out these typos. We will correct all of them in the revised version and perform a thorough proofreading pass to ensure overall consistency and clarity in the manuscript."}]

Author rebuttal:

Meta-review: This paper introduces SPORT, an annotation-free, iterative self-exploration framework for training multimodal tool-use agents. The method mitigates the reliance on costly human annotations by generating tasks, sampling step-level actions, and leveraging an AI verifier with step-wise preference optimization (DPO). Experiments on GTA and GAIA benchmarks show consistent improvements over strong supervised and DPO baselines, with competitive results compared to closed-source agents. In addition, the authors also contributed a 16K preference data for the research community.

This submission has several strengths including: 1) Annotation-free training pipeline which does not rely on human annotations but a self-evolvement system to gradually acquire the knowledge to interact with the environment. 2) Novel training paradigm where step-level preference optimization from self-generated tasks and verifier feedback enables learning from partial successes and 3) Strong experimental results showcasing the compared with SFT baselines and other related works. Reviewers also posed a few concerns about the method including the technical novelty of the proposed method and the analysis on the effect of chosen models as verifier and controller. Delightfully, the authors spent tremendous efforts to address the concerns raised by the reviewers with more experimental results and analysis. After the rebuttal, the reviewers converged on positive assessments: one accept (5) and three borderline accepts (4), all maintaining their scores after rebuttal. 

The AC finds that the paper presents a technically solid, practically valuable contribution, with sufficient novelty at the level of training paradigm. While some limitations remain (performance gap to large closed-source models, limited architectural innovations), the strengths outweigh the concerns. As such, the ACs recommend a clear acceptance.
Decision: Accept (poster)