Title: Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Tool usage

Multimodal agents
Abstract: Multimodal agents, which integrate a controller (e.g., a vision language model) with external tools, have demonstrated remarkable capabilities in tackling complex multimodal tasks.
Existing approaches for training these agents, both supervised fine-tuning and reinforcement learning, depend on extensive human-annotated task-answer pairs and tool trajectories.
However, for complex multimodal tasks, such annotations are prohibitively expensive or impractical to obtain.
In this paper, we propose an iterative tool usage exploration method for multimodal agents without any pre-collected data, namely SPORT, via step-wise preference optimization to refine the trajectories of tool usage. Our method enables multimodal agents to autonomously discover effective tool usage strategies through self-exploration and optimization, eliminating the bottleneck of human annotation.
SPORT has four iterative components: task synthesis, step sampling, step verification, and preference tuning.
We first synthesize multimodal tasks using language models. 
Then, we introduce a novel trajectory exploration scheme, where step sampling and step verification are executed alternately to solve synthesized tasks.
In step sampling, the agent tries different tools and obtains corresponding results. 
In step verification, we employ a verifier to provide AI feedback to construct step-wise preference data. 
The data is subsequently used to update the controller for tool usage through preference tuning, producing a SPORT agent.
By interacting with real environments, the SPORT agent gradually evolves into a more refined and capable system.
Evaluation in the GTA and GAIA benchmarks shows that the SPORT agent achieves 6.41% and  3.64% improvements, underscoring the generalization and effectiveness introduced by our method.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SPORT (Step-wise Preference Optimization for Refining Trajectories), a novel method for training multimodal agents without requiring pre-collected human annotations. Current methods for training multimodal agents rely heavily on expensive human-annotated preference pair data or distilled tool usage trajectories data from closed-sourced APIs, which creates a bottleneck for developing agents that can handle complex multimodal tasks. SPORT enables agents to automatically improve their tool usage capability through self exploration and iterative refinement. This method consists of four iterative components: \n\n1. **Task generation**: synthesizes diverse multimodal tasks (queries + files) using LLMs. \n2. **Step sampling**: the agents explores multiple possible actions at each step by sampling different tool calls.\n3. **Step verification**: a verifier evaluates the sampled actions to identify the preferred option for each step.\n4. **Preference tuning**: uses the collected step-wise preference data to update the agent controller via DPO.\n\n**Main contributions**:\n\n* A self-exploration framework that eliminates the need for pre-collected annotations.\n* Step-wise preference optimization enables improvements over SFT baselines on GTA and GAIA benchmarks.\n* A dataset of 16K preference data samples is created through the exploration process.\n\nThrough multiple ablation experiments and user study, this paper demonstrates that agents can effectively learn to use tools through autonomous exploration, which provides an alternative to traditional supervised approaches.\n\nSTRENGTHS AND WEAKNESSES:\n## Quality\n**Strengths**: \n* Technical merit: The paper presents a well-design iterative framework with clear technical components. Its step-wise preference optimization approach is theoretically grounded for DPO. \n* Experimental validation: The method is experimented on two popular benchmarks (GTA and GAIA) with extensive ablation studies. The authors also conduct detailed statistic analysis on the generated preference data in three aspects (code error rate, tool selection, and content variations). \n* Human validation: The work has thorough validation including user studies to validate both data quality and verifier alignment with human preferences (82% agreement rate).\n\n**Weaknesses**:\n* To my understanding, the verifier component seems to be an important element in this system. The paper would benefit from more details and analysis of its implementation and impact on performance. \n\n## Clarity\n**Strengths**: \n* Well-structured presentation: The paper follows a logical flow from problem motivation through method description to experimental validation.\n* Clear visualizations: The figures and tables in the paper effectively illustrate the pipeline (Figure 1), search scheme (Figure 2), concrete examples of preference data construction (Figure 4 and 5), and training process through algorithm. \n* Clear communication and replication support: The authors presents technical concepts with clear and accessible language while providing comprehensive implementation details about model, training, and prompt templates.\n\n**Weaknesses**:\n* Verifier design: While prompt templates are included in the appendix, the main paper sections could better elaborate on the verifier's design choices and implementation details. A more thorough discussion of the verifier's architecture, design rationale, and key parameters would enhance the paper's technical depth.\n\n## Significance\n**Strengths**: \n* SPORT removes the dependency on expensive human annotations for multimodal agent training, which solves a key practical challenge.\n* This method works with existing open-source models (Qwen2-VL-7B) and standard toolkits, making it accessible to the broader community.\n* The 16K preference dataset generated will be valuable for future experiments and research.\n\n**Weaknesses**:\n* Future performance potential: While the experiments show promising results with open-source models, it has not yet matched closed-source model based agents' performance. This gap suggests potential for future improvements.\n\n## Originality\n**Strengths**: \n* The idea of constructing preference data at the step level rather than trajectory level is innovative and allows learning from partial successes.\n* The system effectively combines task synthesis, online exploration, and preference learning in a novel way for multimodal settings.\n\n**Weaknesses**:\n* The work seems effectively build upon and enhances the MAT framework and MM-Traj dataset. Although effective, more original architectural contributions could further strengthen the work's novelty.\n\nQUESTIONS:\n1. **Task seed collection and diversity**: \nThe paper says \"we collect seed from the existing method MAT\" but doesn't clarify the specific source or scale. Could you please clarify the following: Did you sample seed tasks directly from the 20K MM-Traj dataset, or did you use the MAT framework to generate new seed tasks? How many seed tasks were used in total for task synthesis? And lastly, how did you ensure diversity across task categories? (e.g., did you sample uniformly across MAT's 16 non-overlapping categories mentioned in their paper?) \n2. **Verifier details**: \n What model(s) do you use for the verifier? How did you decide on the selected verifier or evaluate the verifier? Your appendix has a section for the prompt for step verifier, but it would be helpful to include more technical details and some analysis in the main section (e.g. Section 3.3).\n3. **Experimental baselines and comparisons**: \nIt’s great that you list the GTA and GAIA results for many combinations of methods and controllers. Do you also have the GTA and GAIA results for the SPORT Agent with the initial agent controller (the post-SFT model, i.e., the MAT-Qwen2-VL-7B)? This might create a more direct comparison showing SPORT's improvement over the same base model. Also, did you conduct experiments with your agent using closed-source controllers? I'm curious if with the same closed-source model, your agent would perform better than other agents (e.g. HFAgent). \n4. **Efficiency and cost analysis**: \nI think it might be helpful to have some discussion about the compute time and computational costs. For example, a high-level comparison between SPORT and conventional methods. If the gains are much larger than the overhead, this would strengthen the contribution.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SPORT, a framework for training multimodal tool-use agents using step-wise preference optimization. SPORT builds its own preference data via self-exploration: at each decision step, it samples multiple tool-usage actions, executes them, and uses an LLM-based verifier to select the most desirable one. Crucially, the verifier compares outcomes in terms of usefulness rather than correctness, providing more flexible supervision. The authors evaluate SPORT on two multimodal benchmarks: GTA (image-based reasoning) and GAIA (document-based tasks).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- SPORT demonstrates clear improvements over SFT baselines and performs competitively with agents built on top of strong models like GPT-4o.\n- The full training pipeline, including task generation, tool sampling, and preference labeling, is entirely self-supervised, requiring no human-annotated trajectories.\n\n### Weaknesses\n- The main components of SPORT, including self-instruct task generation, LLM-based comparison, and DPO, are all established techniques. While the paper positions itself as \"explores the tool usage by itself via an iterative manner\", it does not introduce any novel mechanisms for agent-level self-improvement, such as strategic planning, error reflection, or hierarchical reasoning. Instead, it looks more like a specific application of existing methods to tool-usage domain with tool-based actions as action space. Due to the limited novelty, I put a low confidence score. \n- While very relevant related works such as Step-DPO are discussed, it is not included in experimental comparisons. A fair adaptation of such methods to the tool-use setting would strengthen the empirical claims. For example an adapted version of Step-DPO that does error localization and correction.\n\nQUESTIONS:\n- How sensitive is SPORT to the quality and diversity of the synthetic tasks generated in the early stages of self-exploration?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SPORT (step-wise preference optimization to refine the trajectories of tool usage), a framework for training multimodal agents to use tools effectively without requiring pre-collected human annotations. Current multimodal agent training relies heavily on expensive human-annotated trajectories and tool usage data. This creates bottlenecks and may lead to poor generalization due to distribution mismatch with target environments. On the contrary, SPORT operates without human annotations through four iterative components: 1) Task Synthesis: Uses LLMs to generate diverse multimodal tasks (queries + files). 2) Step Sampling: Agent samples multiple candidate actions for each step. 3) Step Verification: AI verifier evaluates and ranks step-level actions based on quality. 4) Preference Tuning: Uses Direct Preference Optimization (DPO) to fine-tune the agent controller.\n\nResults on GTA and GAIA benchmarks show that SPORT achieves 6.41% and 3.64% improvements over supervised fine-tuning baselines, demonstrating effectiveness in multimodal reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n1. Self-Improving Framework: The iterative nature allows agents to continuously improve through interaction with real environments\n2. Evaluation: Tests on two established benchmarks (GTA, GAIA).\n3. Practical Design: The proposed method works with open-source models (Qwen2-VL-7B), generates diverse synthetic tasks covering multiple file types and achieves competitive performance against closed-source models.\n\nQUESTIONS:\n1. Computational cost: Seems like the proposed method requires expensive tool execution (LLM calls, web searches) during exploration. The reviewer hope the author could provide more analysis on computational cost estimation and comparison.\n\n2. Selection of the LLM model in task synthesis and AI verifier. The reviewer is wondering how the LLM model and AI verifier selection would influence the final results. It seems that in the first three steps, all the knowledge comes from these two model (and maybe the unlabeled data). However, there is no ablation study on what influences these two models bring in the whole framework. It would be valuable to provide some experiment results on this.\n\n3. Influence of unlabeled data used in the training. It would be better to provide experiment results on using different portion of unlabeled data for training the framework.\n\nLIMITATIONS:\n1. Computational cost: Seems like the proposed method requires expensive tool execution (LLM calls, web searches) during exploration. The reviewer hope the author could provide more analysis on computational cost estimation and comparison.\n\n2. Selection of the LLM model in task synthesis and AI verifier. The reviewer is wondering how the LLM model and AI verifier selection would influence the final results. It seems that in the first three steps, all the knowledge comes from these two model (and maybe the unlabeled data). However, there is no ablation study on what influences these two models bring in the whole framework. It would be valuable to provide some experiment results on this.\n\n3. Influence of unlabeled data used in the training. It would be better to provide experiment results on using different portion of unlabeled data for training the framework.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SPORT, an iterative self-exploration framework for training multimodal agents in tool usage without any annotated data. It enables the agent to autonomously generate tasks, explore tool actions, and collect step-wise preference feedback via an AI verifier. The agent is then optimized using DPO based on this feedback. Experimental results on GTA and GAIA benchmarks show that SPORT significantly outperforms strong supervised and DPO baselines, and achieves competitive results with some closed-source models. The framework demonstrates that agents can improve tool usage capability purely through self-exploration and AI feedback, providing a scalable solution for complex multimodal tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Annotation free: SPORT does not rely on any human-annotated data or pre-collected trajectories, which addresses a major bottleneck in current multimodal agent research and greatly reduces data collection costs.\n2. Novel online self-exploration: The method leverages an iterative, online self-exploration and preference optimization process, allowing the agent to adaptively discover effective strategies and improve performance through interaction with the environment.\n\nWeaknesses:\n1. The paper does not always specify the exact models used for each component (e.g., task synthesis, file generation, step verification, filtering). More explicit clarification on which LLM or VLM is used for each step, including possible ablation on model choices, would improve reproducibility and help understand the framework's sensitivity to model selection. Could the authors provide more detailed information on the models, prompts, and configurations for all stages of the pipeline?\n2. No Comparison with True Online RL: Although they ablate DPO vs. static DPO, they never compare against an online RL baseline (e.g., PPO or GRPO) on the same framework. It is unclear whether SPORT is merely an intermediate between offline DPO and true online RL, or if incorporating conventional RL techniques could yield further gains. A direct comparison to PPO/GRPO baselines would clarify SPORT’s position on the spectrum from static preference tuning to fully online reinforcement learning.\n\nQUESTIONS:\n1. While the SPORT framework uses DPO for preference tuning, is there a reason why fully online RL-based methods such as GRPO or PPO-style optimization were not adopted?\n2. If I directly apply the SPORT framework or preference tuning on MAT-Qwen2-VL-7B, rather than just the base Qwen2-VL-7B, would the agent performance improve further? Has this variant been tried?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}