VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents

Authors: Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Song XiXuan, Yifan Xu, Shudan Zhang, Hanyu Lai, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, Jie Tang
Primary area: datasets and benchmarks
Keywords: Large Multimodal Models, Agents, Evaluation

Abstract:
Large Multimodal Models (LMMs) have ushered in a new era in artificial intelligence, merging capabilities in both language and vision to form highly capable \textbf{Visual Foundation Agents} that are postulated to excel across a myriad of tasks. However, existing benchmarks fail to sufficiently challenge or showcase the full potential of LMMs as visual foundation agents in complex, real-world environments. To address this gap, we introduce VisualAgentBench (VAB), a comprehensive and unified benchmark specifically designed to train and evaluate LMMs as visual foundation agents across diverse scenarios in one standard setting, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across 9 proprietary LMM APIs and 9 open models (18 in total), we demonstrate the considerable yet still developing visual agent capabilities of these models. Additionally, VAB explores the synthesizing of visual agent trajectory data through hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations, offering insights into obstacles, solutions, and trade-offs one may meet in developing open LMM agents. Our work not only aims to benchmark existing models but also provides an instrumental playground for future development into visual foundation agents. Code, train, and test data are available at \url{https://github.com/THUDM/VisualAgentBench}.

Decision: accept (poster)
Reviewer scores: 6, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1729536359364

thread_type:
official_review

<thread_object>
creator: reviewer_cjyi

<content>
summary:
The authors propose a benchmark for evaluating LMMs across a wide range of task: including proposing (1) standardized interfaces, prompting, and data formats, (2) a strategy for creating valid test sets, (3) multitask multi-environment trajectory train sets, (4) benchmarking of 18 open-sourced and closed-sourced LMMs, (4) analyses of LMMs' abilities for grounding and planning.

strengths:
I appreciate the diversity of tasks explored, from embodied agents to visual design. I believe that general-purpose LMMs that can perform well on a wide range of tasks are essential, and such benchmarks are necessary. Interaction through text/vision-based environment feedback is an important aspect of the paper. I also appreciate the scale of evaluation in the paper, and the finetuning of open LMMs.

weaknesses:
1. Can you elaborate on how the task description, action spaces, few-shot demonstrations, and notices for each environment are formatted as prompts? How are visual outputs of the environment passed in as text in the interaction rounds for embodied and GUI agents? Could you elaborate on how error recovering works in planning? Interested in seeing experiments on error recovery behavior across all models, or in general, some evaluation on the interaction rounds instead of final success rate only (e.g., average number of steps to recover from an error). 
2. I’m also interested in seeing more detailed analyses on the tasks that these models fail on. For example, which “prototypes” lead to failure? Does each model fail in a way that is consistent with one another (e.g., is there correlation between the accuracy on each prototype/subtask?) Does finetuning an open LMM help on the grounding aspect more, or the planning aspect more?
3. On a high-level, it would be great to have a quantitative metric established in this benchmark that systematically measures performance on grounding vs reasoning, instead of as an analysis only. Also, related to the first point, quantitative evaluation on interaction behavior (perhaps some proxy metric on progress to the goal).

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730437554918

thread_type:
official_review

<thread_object>
creator: reviewer_onsv

<content>
summary:
This paper introduces a multimodal benchmark VisualAgentBench to evaluate LMMs as agents performing interactive tasks. The benchmark includes three scenarios based on five datasets/environments: Embodied (VAB-OmniGibson, VAB-Minecraft), Graphical User Interface (VAB-AndroidLab, VAB-WebArena-lite), and Visual Design (VAB-CSS). Besides benchmark data, the authors also provide additional task trajectories for training. All the task instances in the training and testing data are constructed by prototyping and instantiation. This paper applies a mix of three strategies to collect training trajectories according to the characteristics of different datasets. Experiment results show that the proposed benchmark is challenging for current LMMs and further SFT could improve the performance. The paper also conducts an analysis of visual grounding and planning to provide insights for the future development of LMM agents.

strengths:
1. The proposed benchmark is a good complement to current multimodal and agent benchmarks to evaluate LMMs in challenging interactive scenarios.
2. The proposed benchmark has standardized environments with good consistency and reproducibility.
3. The paper also provides training trajectories for SFT.
4. The experiments are comprehensive, revealing problems of current models, and the analysis provides good insights.

weaknesses:
1. The number of training data is still very limited. The paper does not show whether it is possible to scale up training data in the proposed environments in an efficient way.
2. There is no analysis and experiments to verify whether the proposed environments could be effectively used for RL training.
3. It would be helpful to train some non-LLM specialist models in each environment using RL/IL and report their performance as a reference.
4. After fine-tuning LMMs with the collected training data, authors should also evaluate their general multimodal abilities on other multimodal benchmarks. Also, the authors should explore whether it is possible to maintain the original abilities while improving the performance on the proposed benchmark after SFT.
5. The authors should provide some trajectories of models tested for better illustration.

questions:
1. When training open LMMs, the paper says that they only use the vision input of the latest turn. Does this mean each turn in the training trajectory is treated as an independent sample (with previous turns provided in context) instead of training as a multi-turn conversation sample?
2. For evaluating LMM APIs, what do you mean by "Vision Input image won’t be kept in the dialog"? Wouldn't the images that appear in the previous turns in one conversation automatically be kept?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698982959

thread_type:
official_review

<thread_object>
creator: reviewer_cywd

<content>
summary:
The authors propose to apply large multimodal models to visual agent tasks, which is mored grounded than existing VQA tasks.
The authors collected 5 datasets including 1 for robotics in simulation, 1 for game playing, 2 for GUI manipulation and 1 for web page design.
The authors design a function calling-based action space for each task.
The agent tasks are created by first generating a bunch of templates with placeholders and then instantiating these templates.
Action trajectories for each task are collected by 1) human-written scripts(for web GUI tasks) 2) prompting existing LMMs like GPT-4o 3) human demonstration.
The authors collected 746 test cases and 4482 training trajectories across 5 tasks and benchmarked 18 proprietary and open-source LMMs.

strengths:
1. The proposed benchmark provides a unified “function-call” action space as well as diverse task domains for benchmarking the agent ability of LMMs
2. Experiments and ablation are solid. A wide range of both commercial and open-source LMMs are evaluated on the proposed benchmark. Ablations of input image prompting(labels, SoM), reflection(injecting error) and planner(ReAct w/ & w/o Thought) are conducted.

weaknesses:
1. No clear advantage over existing benchmarks. There are plenty of existing benchmarks for both high-level planning for robot manipulation in simulation like RoboVQA as well as for GUI agent tasks like Webshop-V and VisualWebArena. The proposed visual agent benchmark is surely more grounded than VQA benchmarks like MMMU, but I don’t see what’s the real contribution is if compared with domain-specific benchmarks. 
2. Low quality training trajectories. Take the GUI agent for instance, the proposed VAB-WebArene-Lite uses code script-based trajectory collection, which is well-known for its limited diversity compared with real-world human web browsing action trajectories. 
3. The function calling action space biases toward LMMs with a strong coding ability so that some LMMs unfairly got low scores(like Qwen-VL-Max and Gemini-1.0-Pro, which both do a good job for pure VQA benchmarks.

questions:
1. Does the physical dimension of robot arm given in Gibson? As shown in Figure 1 Round 3, I don't think the grasp banana is feasible given its lying on the far end of the countertop
</content>
</thread_object>

</thread>
<thread>
creation date:
1731074996955

thread_type:
official_review

<thread_object>
creator: reviewer_4jcv

<content>
summary:
The paper introduces VisualAgentBench (VAB), a benchmark designed to evaluate and train LMMs as visual agents in diverse, realistic scenarios, including embodied, GUI, and visual design tasks. VAB provides a unified, standardized framework for assessing LMMs across multiple domains, synthesizes high-quality multimodal data using a mix of programmatic solvers, LMM bootstrapping, and human demonstrations, and benchmarks 18 LMMs, uncovering both strengths and limitations in real-world task performance. Key insights include challenges in visual grounding, planning, and error recovery, offering a valuable testbed to push LMMs toward more adaptable and practical visual agents.

strengths:
- Comprehensiveness: The main strength of this paper is its comprehensiveness in benchmarking Large Multimodal Models (LMMs) as visual agents. The authors introduce VisualAgentBench (VAB), a benchmark that covers a wide range of real-world application scenarios by including five major task categories: embodied agents, GUI-based agents, web agents, gaming agents, and visual design agents. This breadth makes VAB a thorough evaluation tool, enabling a more holistic assessment of LMMs' capabilities across different domains rather than focusing on a single application area.

- Extensive Experiments: The paper demonstrates substantial experimental rigor by benchmarking 18 different LMMs, encompassing both proprietary and open-source models. This extensive testing provides a solid foundation for the insights presented, which shed light on various LMM challenges, such as visual grounding and error recovery. These experiments allow for more reliable comparisons between models, offering valuable insights into how different LMMs perform in complex, interactive tasks. The conclusion on ReAct framework is also interesting.

- Insightful Analysis: Through the VAB benchmark, the authors provide some useful observations on the current state of LMMs as visual agents. They highlight specific limitations in visual grounding, action planning, and error handling across various environments, which helps to pinpoint areas for future improvement in LMM design. While these insights are not groundbreaking, they add value by identifying practical challenges that developers and researchers may encounter when deploying LMMs in real-world applications.

weaknesses:
- Insufficient Explanation for VL Model Performance: Some vision-language models perform poorly without adequate explanation. For instance, the paper doesn’t explore why certain models achieved low scores, leaving questions about the benchmark’s application across models.
- Unclear Role of Visual Information in Certain Tasks: The paper lacks clarity on how specific tasks, such as those in Minecraft, leverage visual information effectively and whether VLM is genuinely necessary for all actions. For instance, Minecraft actions like "Teleport" don't inherently require visual information since they can execute without reference to the visual state, raising doubts about the added value of VL models in such contexts. Clarifying how the benchmark ensures each action necessitates visual input, as opposed to pure language model decision-making, would help demonstrate the benchmark’s relevance and justify the use of VL models over text-only approaches in specific environments.
- Ambiguities in Figure Interpretation and Process Flow: Figures like Figure 2 could benefit from clearer annotations or explanations. The figure includes multiple input and output connections but lacks a clear process flow or indication of sequential dependencies, making it challenging to follow the intended agent behavior across rounds.

questions:
1. Could you clarify the role of bounding boxes and object tags in Figure 7? Does this mean that objects and tags must be visible in the input images so that the simulator can recognize and interact with these objects by their tag names? In Section 5.1, the authors discuss the use of object labels in embodied environments. How exactly does the agent operate when no object label or tag is provided?

2. To ensure ease of use, what practice does VAB provide? unified API access or modular code structure across different task environments? More details on engineering side for easy usage could be beneficial.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733312620185

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We would like to express our sincere gratitude to all the reviewers for their thoughtful and constructive feedback on our work. The insights and suggestions provided have been invaluable in helping us refine and strengthen our submission.

Multiple reviewers appreciated our benchmark for setting up diverse LMM agent application scenarios (Reviewers 4jCV, cYwD, CJyi), and extensively evaluating a wide range of 18 proprietary and open-source LMMs (4jCV, cYwD). Reviewers also highlighted the provided training trajectories for SFT open-source LMMs (oNsV, CJyi). The comprehensive and insightful analysis on planning and visual grounding are especially noted (4jCV, cYwD, oNsV).

In response to Reviewer 4jCV and CJyi's suggestion for a more detailed analysis of agent error modes, we have included statistics on the major types of errors for gpt-4o and internvl-2 in the rebuttal discussion, along with examples of agent trajectories in Appendix G. Additionally, we have provided an elaborated statistic on error recovery behaviors and proxy metrics of progress in VAB-OmniGibson, VAB-Minecraft, and VAB-CSS, as advised by Reviewer CJyi.

We are sincerely grateful that Reviewer 4jCV, oNsV and CJyi support our work and are generally satisfied with our response. Our primary point of disagreement with Reviewer cYwD pertains to the action space within VAB-OmniGibson. While Reviewer cYwD advocates for a physically low-level interaction with the household environment to ensure closed-loop evaluation, we emphasize that VisualAgentBench is focused on high-level planning and reasoning. Our designed action configuration is sufficient for this purpose, and VAB-OmniGibson supports closed-loop evaluation in a high-level manner, as the agent must understand environmental feedback and recover from errors (demonstrated in Figure 2).

Once again, we thank all the reviewers for their constructive reviews and suggestions, which have significantly contributed to the improvement of our work.

Best regards,

The authors of VisualAgentBench
</content>
</thread_object>

</thread>
<thread>
creation date:
1738768173277

thread_type:
public_comment

<thread_object>
creator: ~jian_yao4

<content>
title:
Thank you for releasing the benchmark

comment:
Thank you for your work on this project. I am interested in knowing when the training dataset will be available to the public.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a unified benchmark that supports investigation of LLMs/VLMs/LMMs for decision-making, including for Embodied AI, GUI control, and Visual Design. A number of baselines/models (both open and closed) and an analysis of synthetic data considerations is conducted. Reviewers appreciated the comprehensiveness of the LMM benchmark, extensive experimentation, insightful analysis, and importantly inclusion of SFT trajectories. Concerns included lack of rigor in justifying/selecting the particular benchmarks chosen, insufficient explanation of the model performance (especially failures/error modes), quality of the SFT trajectories, and lack of proxy metrics for progress to the goal that can aid understanding of interactive behavior. In response, the authors provided additional error-mode and error-recovery behavior analysis as well as proxy metrics for progress, among other things. After rebuttal, reviewers 4jCV, oNsV, and CJyi had positive scores, while reviewer cYwD still had concerns that the selection of datasets was not rigorous and the simplification of OmniGibson as a robotics benchmark. 

  After considering all of the materials, I overall agree with the three positive reviewers. Specifically, unifying high-level decision-making problems, and not focusing on difficult low-level problems such as robot grasping and manipulation, is still a valuable contribution given that even such high-level policies do not fair that well even with state-of-art LMMs. The contribution to the community made through the effort to including SFT trajectories and an overall unified benchmark outweighs some of the remaining limitations mentioned. I highly encourage the authors to incorporate elements of the discussion and especially analysis (e.g. failure modes), and perhaps even strengthen the benchmark further thorugh a reasoned-out inclusion of additional benchmarks that can test new axes of LMM capabilities and provide more insightful analysis.