VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
Authors: Xiao Liu
Tianjie Zhang
Yu Gu
Iat Long Iong
Song XiXuan
Yifan Xu
Shudan Zhang
Hanyu Lai
Jiadai Sun
Xinyue Yang
Yu Yang
Zehan Qi
Shuntian Yao
Xueqiao Sun
Siyi Cheng
Qinkai Zheng
Hao Yu
Hanchen Zhang
Wenyi Hong
Ming Ding
Lihang Pan
Xiaotao Gu
Aohan Zeng
Zhengxiao Du
Chan Hee Song
Yu Su
Yuxiao Dong
Jie Tang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Multimodal Models
Agents
Evaluation
Primary area: datasets and benchmarks

Abstract: Large Multimodal Models (LMMs) have ushered in a new era in artificial intelligence, merging capabilities in both language and vision to form highly capable \textbf{Visual Foundation Agents} that are postulated to excel across a myriad of tasks. However, existing benchmarks fail to sufficiently challenge or showcase the full potential of LMMs as visual foundation agents in complex, real-world environments. To address this gap, we introduce VisualAgentBench (VAB), a comprehensive and unified benchmark specifically designed to train and evaluate LMMs as visual foundation agents across diverse scenarios in one standard setting, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across 9 proprietary LMM APIs and 9 open models (18 in total), we demonstrate the considerable yet still developing visual agent capabilities of these models. Additionally, VAB explores the synthesizing of visual agent trajectory data through hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations, offering insights into obstacles, solutions, and trade-offs one may meet in developing open LMM agents. Our work not only aims to benchmark existing models but also provides an instrumental playground for future development into visual foundation agents. Code, train, and test data are available at \url{https://github.com/THUDM/VisualAgentBench}.

Published as a conference paper at ICLR 2025
VISUALAGENTBENCH: TOWARDS LARGE MULTI- MODAL MODELS AS VISUAL FOUNDATION AGENTS
Xiao Liu Tianjie Zhang Yu Gu Iat Long long', Yifan Xu', Xixuan Song', Shudan Zhang', Hanyu Lai', Xinyi Liul.t, Hanlin Zhao!-1, Jiadai Sun Xinyue Yangh., Yu Yang!. Zehan Qi', Shuntian Yao -1, Xueqiao Sun', Siyi Cheng*i, Qinkai Zheng', Hao Yu', Hanchen Zhang', Wenyi Hong' Ming Ding', Lihang Pan', Xiaotao Gu', Aohan Zeng', Zhengxiao Du', Chan Hee Song', Yu Su², Yuxiao Dong', Jíe Tang'
TTinghuu University, The Ohio State University, 'Zhejiang University, *Peking University
ABSTRACT
Large Multimodal Models (LMMs) have ushered in a new era in artificial intelli- gence, merging capabilities in both language and vision to form highly capable Visual Foundation Agents that are postulated to excel across myriad of tasks. However, existing benchmarks fail sufficiently challenge or showcase the full po- tential of LMMs as visual foundation agents in complex, real-world environments. To address this gap, we introduce Aietalhg (VAB), a comprehensive and unified benchmark specifically designed to train and evaluate LMMs as vi- sual foundation agents across diverse scenarios in one standard setting, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across 9 proprietary LMM APIs and 9 open models (18 in total), we demonstrate the considerable yet still developing visual agent capabilities of these models. Additionally, VAB explores the synthesizing of visual agent trajectory data through hybrid methods including Program-based Solvers, LMM Agent Boot- strapping, and Human Demonstrations, offering insights into obstacles, solutions, and trade-offs one may meet in developing open LMM agents. Our work not only aims to benchmark existing models but also provides an instrumental playground for future development into visual foundation agents, Code, train, and test data are available at ht tps: : //ttthub..b//./// com/ THHDM//VsuallAeene
VAB-OmniGibson
gpt-1o
gpt-4-vislon-preview
gpt-4-turbe
29.9
daude-335.somme
Proprietary
VAB-CSS
WAB- Minecraft
gemini-1.5-pro
claude 3: opus
gpt-4e-mini
19.8
20.5
21.9
(Prompted)
LMMs
gemini-1.0-pro
qwen vi max
IntervLL (88)
16.0
Owen2.VL (78)
GLM-4V (138)
LLVA NeXT (8B)
105
Open LMMs
CogAgen: (188)
CogVLM2 (198)
8,9
10.3
(Fine-tuned)
WebArenaiLitt
VAB
VAd-AndroidL.ab
LLaVA-1.5 (138)
CogyLM (176)
Qwen-VL (98)
10.
2v
gemini 3.5
LLaVA
0
10
20
30
40
50
dhude-3.5-sonnee
GLM-RV
VisualAgentBench Avg.
(a) Typical LMMs' VAB performance (relative) (b) Average VAB Success Rates of tested LMMs across 5 against the best in each environment.
environments. Dashed lines for two LMM types' average.
Figure 1:
Overview of Proprietary and Open LMMs on VISUALAGENTBENCCH After Behavior Cloning (BC) on trajectories, Open LMMs demonstrate potential to serve as visual foundation agents.
"Equal contribution. Email: {shawl 1 mi steeee66384 9045338444449 8mgaa 1 com, gu. 8260osu.edd Work done when these authors visited Tsinghua University. Corresponding author: JT.

---

Published as a conference paper at ICLR 2025
Real-Worke Challenging Tasks
Interactive Environments
T Benans and appll
oiomole tosk
the countetip them
LMM-as- -aaaa-----iisaaa Visual-Agent
Embodled
VHS
Gotd acscia fance gote your ovantory in Minecoift
Popelrary
VeB MiB--Wrecrah
Check (he neantot and
MEAA
United Prompts
LMM4 Devv is Ea Toe
Dekamit Shoning
Prempeingg
Wilteesa-Lite
Ooen latest updited iskue the i aboow
Action Spaccs
Visual
WAB CSS
bet heme twadeer son
Design
horinorelly alinedt
expectid Ploase
Figure 2: VISUALAGENT is the first systematic benchmark for training and evaluating LMM- Agent with both proprietary and open LMMs across a diverse set of practical challenges. Based on created tasks, we unify the benchmarking of both proprietary LMM APIs via prompting and open LMMs via behavior cloning training in interactive environments.
as-
INTRODUCTION
Recent advancements in Foundation Models, particularly Large Language Models (LLMM)) (Brown
et al. 2020 Chowdhery et al. 2022, Touvron et al. 2023 Zeng al., (022) and Large Multimodal Models (LMMs) (Liu et al. 2024c| OpenAl 2023 2024a Anthropic 2024). have showcased their profound capabilities in understanding and processing vast amounts of world knowledge, factual information, and common sense reasoning. Notably, these models have demonstrated potential as intelligent agents (Searle) 1970 Maes] 1994; Wooldridge & Jennings 1995). addressing broad spectrum of real-world challenges (Liu et al. (023a)). LMMs, in particular, enhance the capabilities of these agents by integrating visual inputs, thereby expanding the scope of intelligent agent applications.
This progress has given rise to the concept of Foundation Agents-generaliss agents adept at mastering a plethora of skills across various virtual and embodied environments, mirroring human versatility. These agents, especially those powered by LMMs, are envisioned to excel in multitask enviromments without the need for task-specific fine-tuning. paradigm already set by LLM- --beddd language agents. The burgeoning field of visual foundation agents offers promising pathways toward achieving AGI, with the potential significantly elevate human productivity and creativity.
However, the setup for LMM-as- LiMM-a--Visual-iie VisuullFoundation-- Aggett remains underdeveloped. Most existing evaluations on LMMs focus on traditional vision tasks (Singh et al., 2019 Lu et al.. 2022 Liu et al.
2023b Kazemzadeh et al. 2014), on performance in standardized human exams (uue et al.) 2023
Lu et al. 2023). They rarely measure the models' higher-level planning abilities or specific strengths as visual agents. In contrast, the role of LLMs as agents in text environments has been extensively
explored and validated as a reliable measure of their capabilities (Yoo et al. 2023 Liu et al. 2023a).
Recent benchmarks for multimodal agents, while valuable, inadequately address the comprehensive multitask evaluation required for LMM-as- coauallFoundation.. Agent. Their focuses are often limited
to single environments such Household (Shridhar et al. 2020a| b), Gaming (Fan et al., 2022 Wu
et al. 2023). Web (Deng et al.) 2024 Zhou et al. [022]] Koh et al. 2024)). or Desktop (ie et al.
2024 Kapoor al. 2024)). The narrow scope and varied settings prevents holistic assessment of
LMMs' multitask agent capabilities. Furthermore, the prevalent prompting-only evaluation in existing
benchmarks does not suffice for open LMMs (Liu al., 2024c [ai et al., (023), which typically
show limited instruction-follont
capabilities so far, thus hindering a comprehensive evaluation.
To bridge this gap. we introduce BEUNECC (VAB) the first systematic benchmark to multitask train and evaluate visual foundation agents across a diverse array of realistic vision- centric tasks. We present three representative scenarios and develop five distinet datasets for this study: Embodied (VAB-On MAB-OmmiGibson, VA VAB-Minecraftt, Graphical User Interface (GUI) (VAB- AndroidLab (Xu et al. 2024). VAB- Webe-rrrrrrrlttt (Zhou et al., 2023)), and Visual Design (VAB- CSS), enabling comprehensive testing and development of agents that can navigate complex spaces, interact with digital interfaces (Liu et al. 2024d), and understand aesthetic and functional aspects of visual design. This diversity not only challenges the agents' capabilities across different settings but also enhances their adaptability and utility in practical applications, paving the way for more robust and versatile visual foundation agents.
2

---

Published as a conference paper at ICLR 2025
Contributions, Our main contributions in VAB are as follow: Standardized Environments, LMM- oriented Test Data Creation, Train Data Synthesis, Extensive Benchmarking, and Analytical Insights.
Standardized Environments. VAB standardizes the interfaces, prompting, and data formats to facilitate a consistent evaluation of LMM agents across environments. The efforts include adapting previously LMM-irrelevant environment and creating the new. Each task is assessed through interactive evaluation, where LMMs engage directly with the environment, and their performance is measured by specific judge functions. The feature substantially distinguishes
VAB from many other benchmarks (Deng et al. 2024 Rawles et al. 2024 Kapoor et al. 2024)
based on offline auman-annotated trajectories.
LMM-oriented Test Data Creation. Test set queries and judge functions are newly created for all environments except for WebArena. To acquire massive tasks, we develop a "Prototype- Instantiation" method to evolve valid task instructions and judge functions grounded on environ- ments. We refer to task categories and prototypes in some previous study (Li et al. 2023 Zhu et al. 2023) which unsatisfies the practical use for evaluating LMM agents or has no public data.
Train Data Synthesis. VAB strives to offer the first multitask multi-environment trajectory train set for developing LMM agents, containing 4, 4,482 high- quulliyaitty training trajectories spanning environments. It explores and investigate how to synthesize multimodal agent task instructions and trajectory data via hybrid strategies of Program- and Human Demonstrations. Thereby, it also for the first time enables holistic agent evaluation that includes many open LMMs with valid scores for meaningful comparison.
Solvers, LMM Agent Bootstrapping,
Extensive Benchmarking & Analysis. Our extensive testing over 18 LMMs, including 9 proprietary LMM APIS and open LMMs, demonstrates the impressive progress of LMM- Top proprietary LMMs, such as gpt-4o, are solving 36.2% of challenging problems with mere prompting. Behavior cloning (BC) on the VAB train set remarkedly enhances the capabilities of open LMMs as visual foundation agents, with most surpassing the performance of weaker proprietary I.MSs and approaching close towards the strong gemi
as- S--iisual-FFundation-Aaa-
Analytical Insights. VAB provides deep insights into the general status quo and detailed dimensions of grounding and planning for LMM agents. For grounding, we quantitatively analyze the use of object labels, set-of-marks, and visual difference ability. For planning, we study the actual impact of Chain-of-Thought and error recovering ability for LMM agents.
2 VISUALAGENTBENCH: TASKS AND ENVIRONMENTS
In this section, we will first introduce the problem definition of and then the detailed description of each environment and dataset.
An agentic problem could be generally formulated as a Partially Observable Markov Decision Process (POMDP) problem, where S denotes the state space, A denotes the action space. T denotes the transition function, R refers to the reward function, refers to the instruction space, and ( refers the observation space. Compared to L.MM-as-Agent (Liu et al. 2023a),, the observation space O must incorporate visual inputs (e.g.. images or videos) in LMM-as- Visual-Foundation-Agmm significantly extending the application scope but also casting substantial challenge for LMMs to reconcile their multimodal understanding and high-level reasoning.
Overview of VA B. In VAB, we carefully select the most representative and promising tasks that could be enabled by LMM--baed agents. These tasks generally fall into three categories: embodied agents, including household and game environments; GUI agents, covering mobile and web apps: and visual design agents, focusing on frontend CSS debugging (Figure 2 They span diverse domains and feature unique challenges, providing an ideal testbed for comprehensive evaluation of L.MM-based agents. When constructing VA B, we strictly follow the principles outlined in Appendix A. Our efforts focus on addressing gaps in evaluating L.M--based agents while leveraging existing resources to avoid redundancy, ensuring all our work is meaningful and avoids reinventing the wheel. For 4 out of 5 tasks, we collect new data from scratch. For web agents, we adapt and clean WebArena (Zhou ett al. 2023) as our test set, it is already suitable for LMM-based evaluation. For household agents, we use the OmniGibson environment from Behavior- 1k (Li al. 2023) and create new tasks based on high-level actions we defined, which are crucial for evaluating LMM- agents and absent in
3

---

Published as a conference paper at ICLR 2025
Table Comparison between VAB and related benchmarks. VAB the first comprehensive multi- domain agent benchmark offering interactive environments, supporting multimodal agent evaluation, and providing large and diverse set of training trajectories for visual agent tuning. "#Test Ins." refers to the number of test instances; "#Train Traj." refers to the number of training trajectories SFT, "RL" means no training trajectory available and only a reinforcement learning setup provided; "#Avg. Turns" refers to the average number of turns per training trajectory.
Categury
WEnv. #Test
MTrain Traj. #Avg. Turns Multimodal Interactive Env.
AL.FWorld (Shridhar ot al.
2020h
Household
134
6,374
7.54 X
Alfied (Shridhar
Houschold
1,529
6,374
7.51
Bchavior-IK 1K Li al.
2024
Houschold
1,000
RL
MineDojo Fan at al.
2022
Game
3,141
RI.
SmartPlay Wa el al.
202.3
Game
20
X
Mind2Web Deng el al. 2024
1,341
1,009
7.71
X
WebArena Zhou ci
Web
812
VisualWehAArea [Koh ct al.
2024
Web
910
META-GUI (un cr 2022
Mobile
483
3,692
7.64
X
OSWorld Xie Z124
Desktop
369
OmniACT Kappor al. 2024
Desktop
9,802
AgentBench al.
Multi-domain
1,091
X
Multi-domain
4,482
existing datasets. We similarly construct our tasks in Minecraft using the MineRL environment with our self-defined high-level actions. Finally, for our mobile app and CSS debugging tasks, we create new interactive environments due the lack of suitable resources in the literature and collect data based on these environments. An overview of VAB is shown in Table
2.1 EMBODIED AGENT
Embodied agents have been a central topic in AI, naturally involving multimodal sensory data, including language and vision signals. The multimodal capabilities of LMMs could enable new possibilities for embodied agents.
VA YAB-OmniGibson. One of the most actively researched environments in embodied Al is the household environment due to its complexity and range of everyday tasks (Huang et al., 2022 Song al. 2023: Shridhar al. (220aa). We build the household environment for embodied agents using Omniiibson a high-fidelity simulator based on Nvidia Omniverse that features diverse scenes and realistic physical effects I An example activity in VAB-OnniGibson would be "Put all 8 plates from the countertops into the cabinet in the kirchen", where agents should accomplish the tasks using provided high-level actions (e.g.. "grasp", "pputinnsinn We adopt the task success rate as the evaluation metric. (Cf. Appendix B
VA B-Minerrfft. Minecraft has become a popular open-world environment for developing generalist embodied agents due to its diverse tasks (e.g., survival, harvesting, crafting, combat, and creative tasks), varied environments, and interactive mobs, necessitating generalized agent abilities (Fan et al.) 2022 Lifshitz et al., 2024). In VAB-Minecrae the agent is expected to accomplish wide range of tasks, including item collection and killing hostile mobs. An example task in VAB-Miiecraft would be "Get a fishing rod in your inventory". and the LMM agent need to interact with the game environment using provided scripts (e.g.." "tgg.""raatt, "smelt") or calling a low-level controller Steve- 1 (Lifshitz et al. 2024) with prompt. We adopt the task success rate metric. (Cf. Appendix
2.2
GUI AGENT
GUI is another typical scenario where LMM agents may excel. Compared to embodied environments, GUI environments are more information inttnnsivee and require good understanding of UI elements and layouts. We provide two interactive and reproducible GUI environments, Mobile (i.e., Android) and WebArena, evaluate LMM GUI agents in a practical manner.
https::/www.nt com/ hom/en-wa /ee//en-////// omn verse/
4

---

Published as a conference paper at ICLR 2025
VAB-AndrvidLab Xu et al., 2024). Automated agents on Android GUI are instrumental. Although pioneer works like (Burns et al. 2022 Rawles et al. 2024) have explored training and evaluating these agents, they typically use Step Success Rate evaluated offline. Recent works (Yang al.. 2023b;
Wang et al. 2024a) leverage LMMs as Android GUI agents but lack reproducible executive evaluation frameworks. We address this by creating tasks for LMM agents to perform human-like actions (e.g..
Tap, Swipe) on smartphones using Android Virtual Device (AVD). For example, "Find a hotpot restaurant nearby and make a reservation for me tonight." Agents must understand the Android GUI and make decisions based on screen observations (Cf. Appendix D). The visual SoM-based prototype developed in VAB for benchmark was later enriched with XML-based text setting, both text and multimodal training, and further detailed metrics and analysis on more models in (Xu et al. 2024).
VAB-WebArena-Lite (Zhou et al., 2023). Web browsing is an ideal testbed for evaluating LMMs as GUI agents. Previous works (Shi et al. 2017, Liu et al.) 2018 Deng et al. 2024 Yao et al. 2022) mainly focus on offline evaluation. We adopt WebArena (Zhou al. 2023), a benchmark for text-based web GUI agents with 812 tasks across 5 websites. LMMs perform tasks based user instructions, such as finding and summarizing customer reviews on One opShop. We use HTML SoM (Koh et al. 2024) to annotate operable HTML elements, enabling LMMs to generate actions via aaaayriight. WebArena- evaluation, removing cross-website tasks and fixing implausible conditions. (Cf. Appendix E
a subset of 165 tasks, refined and adapted for multimodal
2.3 VISUAL DESIGN AGENT
Visual design tasks demand a nuanced understanding of visual signals, which text-only LLMs cannot handle with any easy augmentation, unlike embodied or GUI agent tasks that can rely on externl object detectors (Song et al. 2023) or textual representations like accessibility trees (Xie et al.) 2024).
VAB-CSS. We create a new task to evaluate LMMs on web frontend design, focusing on CSS style adjustments. Fixing CSS styles a labor-intensive task that often requires engineers to iteratively adjust element through trial and error. Such task inherently entails fine-grained visual grounding and reasoning across a series of rendering outcomes resulting from iterative CSS edits. VAB-CSS, the agent iteratively edits the CSS style using provided tools until it thinks the rendering matches a given target design. We adopt success rate (SR) as the metric, which evaluates whether the final rendering matches the target design. (Cf. Appendix (F)
METHODOLOGY FOR VAB DATA COLLECTION
For agent tasks, it is known to be very challenging to design practical and verifiable task instances; let alone creating high-quality training trajectories on top of them later. In constructing VAB, we not only aim to deliver a high-quality agent benchmark but also endeavor to develop a systematic methodology for the problem of LMM-as- VisuualFouudattin collection, we follow two-stage paradigm (prototyping and instantiation) for each new task instance to ensure data quality and executability. Additionally, we harness a suite hybrid strategies to collect training trajectories that can be used to tune open LMMs into better visual foundation agents. Our rigorous data collection process in VAB is crucial for presenting a high-quality resource for LMM-based agents (Figure 3). The statistics of different tasks in VAB are shown in Table 2
data curation. For task instance
3.1 TASK INSTANCE COLLECTION: PROTOTYPING AND INSTANTIATION
Curating meaningful and testable task instances for LMM agent tasks can be difficult. On one hand, they should be diverse and useful to cover real-world applications. On the other hand, they should be grounded to environments carefully to ensure feasibility and practicality. As a result, we collect all our task instances in a two-stage paradigm:
Prototyping: We gather many task prototypes representing high-level goals based on the func- tionality provided by the environment. Related items are temporarily set to placeholders.
Instantiation: Task prototypes are grounded concrete items and conditions collected from the environment. Judging functions are thereby set up by instantiated tasks. Instructions are then rephrased by L.MMs to enhance expression diversity.
5

---

Published as a conference paper at ICLR 2025
OmniGibson Tasks
Interactive Evoluation
Sirwtegy Program based Solvers
Teak Prototypex
Behavior Cloning Trajectories
Help as tidy the <lnor in
Judge
<po0ED by putting all
Strology LMM Agent Bootstrapping
<object> <oontainer>
Environmental Groundings
Environments
Stratogy 3: Human Demonstrations
Open LMM Agents
[object Candlo Sseaee
VAB
container (ze X (sint (shat)) (tabie)
Teat
Inipection
Figure 3: Data collection process in VAB. We follow principled two-stage paradigm to collect task instances and then adopt various methods to further collect training trajectories for them.
Table 2: Statistics of all datasets in VAB.
VAB OAB-OonnGi VAB-Minecraft -nB-Anddoidaaa
VAB- WebArena-Lite
VAB-CSS
#Action Space
20
6
7
12
4
#Test Instance
181
116
119
165
165
#Train Trajectory
872
382
1,213
1,186
829
#Train Step
20,153
5,197
10,175
9,522
5,250
#Max Round Limit
100
100
25
20
10
Following the mentioned principles, we develop diverse task sets for all VAB environments. For source 89 prototypes of general household activities, instantiating them in various scenes to create 992 instances, with 181 reserved for testing. In VAB-Minecraft, we focus on object-collecting tasks, generating 628 instances from our defined prototypes, allocating 1 116 for testing. For VA B-Androiddaa,,, we select 8 Android applications and create 119 test instructions, while developing approximately 70 task prototypes from 18 common apps for training. In WebArena- Lite, we refine 165 test samples and generate 1186 training instances from 40 task prototypes across various web applications. For VAB-CSS, we design tasks around CSS rule corruptions, creating 1210 instances with 165 for testing, each annotated with natural language deseriptions of visual changes. This comprehensive approach ensures rich and varied task environment across our selected domains. For more details, please refer to Appendix A.3]
3.2 TRAINING TRAJECTORY COLLECTION: 3-LEVELED STRATEGIES
Recently, there has been a rise in benchmarks for interactively evaluating LLM LMM agents (Liu et al. 2023a| Zhou et al. 2023 Xie et al. 2024). Despite showcasing the substantial potential of L.M and L.M agents, they usually only provide the test and thus fail to facilitate the improving of open LLMs and LMMs on agent tasks. In light of the challenge, in VAB we are devoted to offering a preliminary behavior cloning (BC) (Nakano et al. 2021 Zeng al. 2023) setup for training open LMM agents. In VAB, we summarize our trajectory collecting into 3-leveled strategies:
Program-based Solvers:
Trajectories are collected by prototype-specific programs written by human experts (c.g., Playwright scripts for automating web browsing tasks).
2. LMM Agent Bootstrapping: Trajectories
collected by prompted LMM agents (e.g.. with optional memory augmentations (Wang et al. 2023c). For instance, in Minecraft we allow agent to access memories for solving easier sub-goals (e.g., how to collect stick) when constructing trajectories for more complex goals (e.g., how to collect a hammer).
3. Human Demonstrations: Trajectories are annotated by human experts. It is necessary for scenarios where humans are indispensable (e.g., mobile apps require logged-in human accounts).
These strategies are quite different from each other and present their own unique advantages in certain environments. We summarize their recommendation levels on 4 dimensions (Cf. Table 33)
The most important dimension. Program-based solvers are most cost-effective, followed by human demonstrations. LMM bootstrapping is currently the most expensive due to proprietary API costs, but this may decrease as open LMMs improve.
Average Cost:
Adaptability: It indicates how easy
implement strategy to an environment. LMM bootstrapping is highly adaptable with good prompts. Program-bass based solvers require moderate implementation time. Human demonstrations need training and may face accessibility issues.

---

Published as a conference paper at ICLR 2025
Table Recommendation levels for 3 strategies used in curating VAB's agent-tuning trajectory data on different dimensions. (Cf. Section 3.2 for detailed explanation on each dimension)
Strutegy
Avg. Cost Adaptability Versatility Flexibility
Adoption
Program- serraan-bbeed Solvers
VAB-Omniiiisonn VAB WebArena-Lite VAB MAi-MMncrraff,, VAB-AndroidLab, VAB-CSS VAB-AndroidLab
LMM Agent Bootstrapping
Human Demonstrations
Versatility: It refers how versatile tasks strategy could deal with. Human annotators can handle the widest range of tasks, followed by LMM agents. Program- based solvers are limited to predefined prototypes.
Flexibility:: It denotes the trial and error process in the solution trajectories. LMM bootstrapping naturally incorporates trial-and-error processes. Program-basse based solvers struggle with this, while human annotators are often discouraged from for quality control reasons.
Considering all mentioned dimensions and their trade-offs, adopt hybrid set of strategies for each of the environments in VAB as shown in Table 3 We employ diverse strategies tailored to each domain's 's unique characteristics. For VA AAB-OmniGisson to high hardware requirements and need for cost-effective, adaptable solutions. In BAB-Minecraff, we opt for LMM agent bootstrapping to handle inherent randomness and exploration requirements. For VAB -A--AddroddLa,, we primarily rely on human demonstrations, supplemented by LMM agent bootstrapping for offline apps, addressing XML legibility and app-specific login challenges. In VAB- WebArena-Lite, we choose program- based solvers, leveraging mature llaywright automation tool and addressing difficulties faced by human annotators with unfamiliar interfaces. For VAB-CSS, we implement LMM agent bootstrapping to accommodate iterative CSS debugging, using gpt 40 for initial trajectories with hints to improve success rates. For more details, please refer to Appendix A.4
utilize program-basee -oaeed solvers due
4 BASELINE EXPERIMENT
4.1 SETUP
Baselines. We evaluate on both proprietary LMM APIs and selected open LMMs. For proprietary
LMMs, we include models from OpenAl GPT (OpenAl 2024a 2023 2024b). Anthropic Claude (An-
thropic 2024), Google Gemini (Reid et al. 2024 Team al. 2023) and Qwen-VI--Max (Bai et al.
2023). For open LMMs, we select nine offte-ofthhe-ar models representative fine-tuning baselines
in VAB: Intern (Chen et al. 2024), Qwen2-VL (Wang et al. 2024b), GLM-4V (GLM et al.
2024), CogVLM2 (Wang et al. (023b)) CogAgent (Hong al. (023). CogVLM (Wang et al. al. 2023b).
LLa VA-NeXT (Liu et al.) 2024b). LL. (Liu et al. 2024a). Qwen- VL (Bai et al. 2023).
Prompting & Open LMM Training. We format agent interactions user-assistant conversations
with system prompts containing task descriptions and action spaces. For most environments, only the latest visual input is provided to reduce costs, except for CSS agents which require comparing multiple screenshots. During fine-tuning of open LMMs, we remove few-shot demonstrations, concatenate history with role tokens, and vertically combine multiple images for CSS agents when needed. All models undergo full-parameter fine-tuning for 5k steps with batch size 64, with CSS data duplicated improve adaptation the screenshot format. See details in Appendix A.5
4.2
MAIN RESULTS
Table
shows the main results VAB, including both prompting proprietary LMMs and fine- tuned open L.Mss. We have several important observations on the status quo of LMM-as- --M-as-Visual- Foundation-Agent.
VAB is challenging for existing LMMs. We observe that existing LMMs face significant challenges when evaluated on VAB. The majority of proprietary LMMs, with mere prompting, achieve overall success rate above 20%, demonstrating their multimodal understanding and reasoning abilities. The most capable LMM, got-4o. 4o, achieves overall success rate of 36.2%. However, these performances are still far from satisfactory and not yet qualified for direct deployment. Notably, despite its superiority existing benchmarks, cl aldde--. 3.dde-3... 5-sonnet still falls significantly
7

---

Published as a conference paper at ICLR 2025
Table Main results on VISUALAGENTBENCCH The metric reported is success rate (SR), which measures the proportion of successful tasks in all evaluated tasks. Open LMMs are evaluated using multitask fin-ttnnin rather than direct prompting, they were unable to effectively follow system prompts from VAB in our preliminary trials. For Params of open LMMs, we report the sizes of their language and vision part respectively.
Embodied
GUI
Visual Design
Type
Model
#Params
OmniGibson
Andraidl
qpt 2004--0-22-----11
N/A
36.2
41.4
55.2
31.9
18.2
34.5
4pt-4-viaion preview
N/A
31.7
36.5
17.4
26.9
18.8
29.1
gpt 4 turbo 0409 turboo
N/A
29.9
26.5
50.0
26.9
18.2
27.9
Proprietary
claude-3.5-semne
N/A
26.9
24.3
56.0
31.1
7:2
15.8
LMMs
claude-3-onnd
N/A
21.9
14.9
51.7
15.1
7.9
20.0
(Prompting)
qpt-4o- 4o-miin-2022-- 2024-077-8
N/A
20.5
12.2
30.2
22.7
20.6
17.0
gemini 1.5 pro pro
N/A
19.8
22.1
41.4
16.8
7.9
10.9
gemini 1.0 pro
N/A
6.3
4.1
11.2
11.8
4.2
0.0
swen-vi11 max
N/A
2.7
0.0
6.0
2,5
3.0
1.8
IntersVI-2 [Chen al.
2024
7B + 0.3B
16.0
16.0
28.4
3.4
7.9
24.2
Qwen2-VL Wang 20246
7B 0.33
15.6
13.8
24.1
5.9
6.7
27.3
GLM 1V GLM ct al 2024
9B 4B
12.0
8.8
19.8
2.5
5.5
23.6
LLaVA- Nex.
[Liu 2024b
8B 0.3B
10.5
3.3
23.3
3.4
4.2
18.2
Open LMMs
(Fine-tuning)
CogVLM2
Hong
X024
8B 12B
10.3
3.3
25.9
3.0
17.6
CogAgent
Hong
2023
7B 11B
8.9
6.6
20.7
2.5
13.9
CogVLM
Wang el
20236
7B 10B
8.4
3.3
19.8
4.2
10.3
MLaVa-1
(Liu
2024a
13B 1B
1.7
25.9
7.9
Qwen-V1.
Bai cl.
2102
7B 2B
1.7
18.1
4.8
behind gpt-4o. Additionally, we present the first systematic evaluation of 2ost-4o-mine on agent tasks, which reveals that its performance is considerably inferior to gpt-4o oo but comparable to claude-3- and gemini 5-pro.
Trajectory SFT can improve LMM agents. For open LMMs, we find they can rarely follow the sys- tem prompt's instruction without fine-tuning in preliminary trials, resulting in 0% success rates. After training on VAB, open LMMs present significant improvements. The strongest one, InternVL-2, outperforms gemini mimiii--1 0-pro all evaluated environments and c claude-3-opu CSS agent task. These results suggest that learning from trajectories would be a promising direction for to build visual foundation agents.
Gaps between top proprietary and open LMMs are huge but likely to be narrowed. Despite training improvements, the gap between proprictary and tested open LMMs is wider than expected. While many claim performance comparable to sppt--vaa svvvion swwssssoopieeeee wwwoooo-ppveeee preview on traditional vision benchmarks like image captioning and VQA, their fundamental ability to function as practical visual foundation agents remains far from comparable, even after fine-tuning on VAB datasets. This demonstrates that VAB could serve as an ideal testbed for benchmarking practical LMM performance. With larger backbone LLMs (insufficiently tested here due to computing resource limitations) and more high-quality trajectory data, open LMMs may eventually match or surpass proprietary ones.
ANALYSIS
Multimodal agent tasks encompass two critical challenges: visual grounding and planning. We conduct fine-grained analyses to gain deeper insights into performance in these two aspects and offer valuable perspectives for the future development of visual foundation agents based on LMMs.
5.1 VISUAL GROUNDING ANALYSIS
Visual grounding refers to the ability to associate language concepts with content in visual percep- tion (Fukui al. 2016 Zheng al. 2024), which is crucial for LMM-as- Aoauall-oundation-A We look into typical design choices in VAB related to visual grounding to show its current status and challenges.
The use of object labels in embodied environment. Despite the strong image caption and object recognizing ability of LMMs, they seem not play well in the context of an embodied agent task.
8

---

Published as a conference paper at ICLR 2025
gpe turbo 0409
gemini-1.5-pro pro
gemini-1.0
VAB-AndroidLab
WebArena-Lite
26.5
30
20
22.1
17.4
4,4
10
8.7
10
4.2
1.7
Figure 5: Compare SoM and REC in GUI agent tasks, trained VLMV VAB-AndroidLabb here is an earlier ver- sion different from the one in Table
Object Label
Object Label
Figure 4: Compare w/ and w/o Object Labels,
Table 5: LMM performances drop drastically on VAB-CSS when the language description is removed.
gpt ppt-4o--e 9pt-44--002-05-13
9pt-4-turbo-CACS
gpt-4 vi vis s oon-oooviiii
NL w/o NL
34.5
27.9
29.1 2.4 126.7%
24.2 $10.3%
1.9 126.1%
In AAB-OnniGibson, we compare the LMM-as- LMM-as-Visual- without object labels annotated in the vision input. The result in Figure Souree/ssows shows that LMM agents significantly underperform without object labels. It indicates that notwithstanding LMMs' strong performance on downstream benchmarks, they can still struggle in the same task in agent context.
performance with and
The use of Set-of-Marks (SoM) in GUI environment. For GUI tasks, we augment image input with SoM by default as eliciting accurate bounding box coordinates from LMMs difficult, representing referring expression comprehension (RFC) task (Qiao et al. (2020). Using our training trajectories, we evaluate whether L.MMs can perform visual grounding by directly outputting bounding boxes without SoMs. We fine-tuned CogVLM2 with and without SoM, and results in Figure 5 show that it struggles to directly output bounding boxes, highlighting SoM's instrumental role in visual grounding.
Visual difference grounding. Our new frontend design task examines visual difference grounding, which unlike traditional single-scene visual grounding, requires the LMM to ground "layout differ- ences" (Cf. Appendix FF55) between two images through comparison. Current VAB-CSS results in Table use a lenient setting where we provide explicit language descriptions adjustments needed to match the images, rather than requiring direct perception of differences (Cf. Table and Appendix F2 for examples).
5.2 PERFORMANCE ON PLANNING
The role of thought in ReAct. ReAct Yao et al., 2023) is one of the most commonly used frameworks for language agents. The central concept emphasizes the importance of integrating the agent's 's reasoning and actions by intertwining the output with both thought and action components. However, in our study, we find that the thought step may not be essential. When using gpt-4o and claude-3 3. 1.33..5000000 -onnett as the backbone of the agents, directly outputting an action field can yield comparable or even superior performance compared to using the ReAct framework (see Table 6b.
Recovering from errors during planning. In real-world applications, agents require the error recovery ability to dynamically adjust their actions and plans based environmental feedback. To understand it in LMMs, we analyze two representative models: gpt-4o, the most powerful model currently available, and gl gim-4v, prominent open LMM. Their performance, illustrated in Figure reveals that 4o exhibits robust error recovery across most tasks, with GUI tasks being an exception due their often irreversible nature. Importantly, recovery scenarios in training data significantly enhances the performance of fine-tuned open LMMs, observed VAB-OmniGibson and VAB-CSS (Cf. Appendix B. and Appendix
find that incorporating error
RELATED WORK
LMM-as- Visual-Agent. In Lre-LMM era, most visual agents are built with task specific train-
ing (Shridhar et al. 2020a) and reinforcement learning (Kempka et al. 2016). With the rapid
development of LMMs (OpenAl 2024a Reid et al. 2024 OpenAl 2023 Bai et al. 2023 Anthropic

---

Published as a conference paper at ICLR 2025
Table 6: ReAct (w/ CоT) has varied (potentially positive or negative) impacts on visual agent tasks.
Model
Prompting
VAB Mineeraft
VAB-Androidl.at ab
VAB-CSS
w/ TToough
55.2
30.4
34.5
gp
w/o Thought
48.3 10.0%
31.9 11.0%
38.2 13.7%
w/ Thought
56.0
29.0
15.8
claude- 3. S-sonnet
w/o Thought
55.2 Lo.85%
31.1 12.1%
17.6 11.85
0LM-dr
W9 Wa-0m1 Goian
12.
26
Wilroracte
VA8-555
24.5
Rasc
50
20
Figure 6: Comparison of overall success rates and success rates when incorrect actions are present in trajectories for various tasks.
2024 Team et al. 2023 GLM et al. 2024), the study of LMM-based visual agents begins to thrive.
Leveraging the general capabilities of LMMs, these visual agents have the potential to perform
plex tasks in various scenarios, including embodied and game tasks (Brohan et al. 2022 Yang et al.
2023a; Driess al.) 2023; Tan et al. 2024), GUT interaction (Zheng et al., 2024] Zhou al. 2023
Koh et al. ||02]] Xie et al. 2024| Kapoor et al. 2024] Yang et al. 2023b), and visual design tasks (Si
al. 2024: Laurençon et al.J 2024). However, these complex scenarios pose several challenges
for LMM-based visual agents: basic visual understanding and grounding (Zheng et al.) 2024 Yue
al. 2023), vision-text information comprehension (Kil et al.) 2024), instruction following, and
long-term planning ability (Wu et al. [2033] Liu et al. 2023a). Most general-purposs LMMs still lack strong zero-shot capabilities, leading to different application paradigms when deploying LMMs as visual agents. While prompting methods offer great convenience, they may not achieve satisfactory performance in many areas (Zhou et al. 2023] Drouin et al., 2024). Consequently, task-specific training and alignment remain common practices in these applications (Lai et al. 2024). In response, VAB aims to establish a comprehensive benchmark for LMM- Mbseed visual agents, covering a wide range of typical applications. In the meantime, VAB seeks provide in-depth evaluation of both prompting and training approaches, ultimately fostering the development of LMM visual agents.
Benchmarking LMMM-baedd visual agents. With the rapid development of LMM agents and their impressive performance in various scenarios (Xie et al., 2024 Kapoor et al. 2024] Yang et al. 2023b|ai a Si et al. 2024 Mu al., 2024), has made the evaluation of LMM agent an urgent problem.
In the GUI interaction domain, recent works have proposed static datasets (Deng et al. 2024 Rawles
al. 2024. Sun al., 2022) and interactive environments (Zhou al., 2023 Koh et al. [2024] Xie
et al. 2024) to evaluate LMM agents in different applications, including web (Zhou et al. 2023
Koh et al.] 2024 Deng et al. 2024). mobile phone (Rawles et al., 2024; [Sun et al.) (022), p, and desktop (Xic al. [2224). In the embodied domain, previous works have proposed various game environments (Guss et al. 2019) Fan et al., 2022) and household environments (Li et al.) 2023).
but few works have explored benchmarking LMM agents these environments. Most existing benchmarks are designed for relatively narrow domains and lack a comprehensive evaluation across different applications of LMM agents. Additionally, many benchmarks focus solely the prompting evaluation of LMM agents. VAB aims to provide a training set for open-source foundation LMMs, offering a new perspective benchmarking these models and advancing their diverse applicabilition.
7 CONCLUSION
We present VisualAAgetBench (VAB), a comprehensive benchmark for evaluating Large Multimodal Models as visual foundation agents across diverse scenarios. Our testing of 18 LMM models reveals their developing capabilities in this domain. VAB also explores methods for synthesizing visual agent trajectory data, providing insights for future advancements.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We would like to thank the anonymous reviewers for their suggestions in refining this work. This work is supported by NSFC 624B2084, 62495063, and 62425601, Daimler Greater China Ltd. and Tsinghua University Joint Institute for Sustainable Mobility, and New Comnestonee Science Foundation through the XPLORER PRIZE.
REFERENCES
Anthropic. Introducing the next generation of claude, 2024. URL http:tt:://w.wattrrrttt
com/news news/ / c aude- family
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:2212.0688 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerrbertVVoss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Prucessing Systems, NIPS' 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 978171382987
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. A dataset for interactive vision languaea navigation with unknown command feasibility. In European Conference Computer Vision, pp. 312-328. Springer, 2022.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 268.:22404.6821,,, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.089 02311, 2022.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, pp. 8469-8488. 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint 4rXiv:2403.07718, 2024.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Conference Empirical Methods in Natural Language Processing, pp. 457-468. ACI., 2016.
11

---

Published as a conference paper at ICLR 2025
Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel LK Yamins, James J DiCarlo, Josh McDermott, Antonio Torralba, et al. The threedworld transport challenge: A visually guided uusk-and-ootion planning benchmark towards physically realistic embodied ai. In 2022 International conference on robotics and automation ([CRA), pp. 8847-8854. IEEE, 2022.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang. Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406. (2Xv:2400.11793,, 2024.
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang. Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations, 2019.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914 2023.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint urXiv:2408. 16500, 2024.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 91 18-9147. PMLR, 2022.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2023.
Raghav Kapoor, Yash Parag Butala, Mclisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv: 17553,
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pp. 787-798. 2014.
Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Viz-
doom: A doom-based ai research platform for visual reinforcement learning. 2016 IEEE Conference Computational Intelligence and Games (CIG), pp. 1-8, 2016. URL https 7/api. www.etttiacccoaaaooooooooooe
Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei-Lun Chao. Dual-view visual contextualization for web navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14445-14454, June 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649 2024.
Hanyu Lai, Xiao Liu, Iat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: Bootstrap and reinforce large language model-based web navigating agent, 2024.
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint arXiv:2403.09022 2024.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-Ik: 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference Robot Learning, pp. 80-93. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila Mcllraith. Steve-1: A generative model for text-to-beiie sot-tt-beehavir in minecraft. Advances in Neural Information Processing Systems, 36, 2024.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupar, Tianlin Shi, and Perey Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations, 2018.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, January 2024b. URL https
sillava-vi.githaae http://hhhh..oo//oo///2222100010 http://oooo2222-0---------a----- 30-11ava--exxt.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. arXiv preprint arXiv:2308. 03688, 2023a.
Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, lat Long long, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint ar arXiv:2411.0088 2024d.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, al. On the hidden mystery of ocr in large multimodal models. arXiv preprint urXiv:2305.0789) 2023b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-We Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprini arXiv:2005.0766 2020.
Pattic Maes. Agents that reduce work and information overload. Commun. ACM, 37:30-40, 1994.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webgpt: Browser-assistee question-answering with human feedback. arXiv preprint 2rXiv:2112.0933:) 2021.
OpenAl. at
New 2023.
models
and
developer
products announced
devday,
URL
new-models and httpa//awo.....ooooooooooooooao-
OpenAl. Hello got-4o, 2024a. URL out.h/ooer/pppaa...c//////////l/
OpenAl. Gpt-4o mini: advancing cost-efficient intelligence, 2024b. 2024b. URL /RR/atpp:///peaal
com/
Yanyuan Qiao, Chaorui Deng, and Qi Wu. Referring expression comprehension: survey of methods and datasets. IEEE Transactions on Multimedia, 23:4426 4440, 2020.
13

---

Published as a conference paper at ICLR 2025
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055530 2024.
John R. Searle. Speech acts: An essay in the philosophy of language. Language, 46:217, 1970.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hermandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. International Conference on Machine Learning, pp. 3135-3144. PMLR, 2017.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020a.
Mohit Shridhar, Xingdi Yuan, Alea-Alexandre Cote. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2020b.
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are from automating front-end engineering? arXiv preprint arXiv:2403.03163. 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326. 2019.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm--panner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-3009, 2023.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on robot learning, pp. 477-490. PMLR, 2022.
Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: Towards multi-modal conversational agents mobile gui. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 6699-6712 12, 2022.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Em- powering foundation agents towards general computer control. arXiv preprint arXiv:2403.33186 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09286 2023.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint ar arXiv:2305.1622 2023a.
14

---

Published as a conference paper at ICLR 2025
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. aobile-agent: Autonomous multi modal mobile device agent with visual perception. arXiv preprint ar arXiv:2401.16158, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the 2et.:24499121,1,, 2024b.
world at any resolution. arXiv preprint
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0035333 1.00079, 2023b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Zihao Wang. Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang. et al. Jarvis-1: Open-world multi-task agents with memory- augmented multimodal language models. arXiv preprint arXiv:2311 1.05997, 2023c
Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice. The knowledge engineering review, 10(2):11--555) 1995.
Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. Smartplay: A benchmark for Ilms intelligent agents. In The Twelfth International Conference on Learning Representations, 2023.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:240.00997.. 2024.
Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng. Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents. arXiv preprint arXiv:2410.24024 2024.
Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang. Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. arXiv preprint arXiv:2310.08588, 2023a.
Zhao Yang. Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang. Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-disciplin multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16500 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. 2022.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for 1lms. arXiv preprint 1rXiv:2310.12823. 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024. URL htt.pp/anreereeee.eeeeooooriiaoa
httts:://ppeneet net htt///ooommmddiiceeeee
15

---

Published as a conference paper at ICLR 2025
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tranyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference Learning Representations, 2023.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144, 2023.
16

---

Published as a conference paper at ICLR 2025
Part I
Appendix
Table of Contents
Overview
18
A.1
Design Features of VAB
18
A.2
Details on Use of Visual Information in Each Environment
18
A.3
Details on Task Instance Collection
19
A.4
Details Training Trajectory Collection
A.S
Details Prompting and Training Open LMMs in Experiments
20
A.6
Analysis Agent Error Modes
20
A.7
Proxy Metrics for Evaluating Progress of Task Completion
21
A.8
Analysis on Error Recovery Behaviors
22
VA
OmniGibson
22
Detailed Description
22
B 2
Actions
................................
23
B33
Program-based Solver for Training Trajectory Collection
23
B.4
Prompt Example
23
VAB B-nnerarraaft
27
Detailed Description
................................
28
Actions
28
Prompt Example
28
VA
An-drrrod
33
D.1
Detailed Description
33
Actions
33
D33 Metrics
33
D.4 Prompt Example
34
E
WebArena-LLit Lite
35
E.
Detailed Description
35
Actions
................................
36
E.3
Metrics
36
E.4
Task Amendment
37
F.5
Prompt Example
39
F
VAB-CSS
40
F
Detailed Description
40
F2
Data Collection
40
F3
Actionsl.
41
F4
Metrics
42
F5
Prompt Example
42
Case Studies
43
MAB-OmniGibson Cases
43
VAB-Mineeris Cases
47
G.3
VAB LadroidLab Cases
51
G.4
WebArena-Liee Cases
53
G.5
VAB-CSS Cases
55
G.6
Error Recovery Cases
57
17

---

Published as a conference paper at ICLR 2025
A OVERVIEW
A.1 DESIGN FEATURES OF VAB
Given that LMMs are still evolving rapidly, we adhere to several principles in our design of VAB to accommodate the current capabilities and limitations of LMMs.
Vision-Centric: VAB agent tasks are designed to primarily rely on visual inputs to solve problems. While additional text imputs could be beneficial, VAB aims to evaluate how LMMs perform when perceiving the environment as humans do in agent tasks. For example, while HTML is shown useful for Web GUI Agent (Zhou et al. 2023 Deng et al. 2024), humans typically browse the internet from screens without reading HTMLs.
High-Level Decision Making: VAB focuses evaluating LMMs' high-level decision-makien abilities. Compared to prior smaller visual-language models that specifically target low-level policies (Lynch & Sermanet 2020 Brohan et al. 2022; Lifshitz et al. 2024). L.Mss excel high- level planning and interacting (Driess et al. 2023) in text response thanks to their commonsense, knowledge, and flexible instruction following with mere prompting. Therefore, in VAB, we simplify the low-level control by providing convenient action interfaces, and ask tested LMMs to concentrate delivering high-level decision sequences in text.
Interactive Evaluation: Evaluating LLMs or LMMs on real-world agent tasks challenging, as task goals can be achieved by various means. As result, it becomes a mainstream practice to
evaluate in an interactive manner (Liu et al. [203aa Zhou et al. 2023 Jimenez et al. 2023 Xic
et al. 2024). VAB also adheres to this principle.
Trajectories for Behavior Cloning: Many previous inecution-based agent benchmarks for LLMs and LMMs, despite being realistic and challenging, often fail to provide effective training sets for the community to use for improvement. L.LMs and LMMs need behavior cloning training
on trajectories for better performance (Nakano et al.) 2021 Zeng et al. [023] Lai et al.) 2024). However, creating such datasets consisting of valid instructions, trajectories, and reward functions is costly and requires annotators' good understanding of the environment. In response to the challenge, for each VAB environment we endeavor to deliver instructions created with hybrid set of strategies (Cf. Section 3.2). Experiments show that our constructed training sets can effectively improve the performance of open LMMs on VAB.
Note that as the field advances, some of the above principles may become obsolete and irrelevant. We will continuously update VAB to accommodate the progress of LMMs.
A.2 DETAILS ON USE OF VISUAL INFORMATION IN EACH ENVIRONMENT
Vision centric design is crucial feature for VAB with regard to planning and grounding. Here will give a brief look at the specific use of vision from both perspectives.
VA B-OnniGiisoon: The agent perceives the simulated environment through egocentric vision. It must interpret visual information to identify the affordances of objects within its view and understand their spatial relationships (c.g., whether an object reachable).
VAB-Minerraftt Similar to VAB- AAn-OnniGiison, the agent must interpret the current scene the game environment to determine its next action. In addition, VAn-Mine includes moving elements such as animals and monsters, which places greater demands on the agent's visual understanding to complete tasks.
VAB VAB-AndroidLab: The agent operates on the graphical interface of an Android system to complete tasks without access to system APIs and relies solely on visual inputs. At each step, analyzes the current screenshot to predict an ietiion-identifyiing interactive elements such as app icons or buttons based solely on the screenshot. Executing the action generates new screenshot for the next iteration.
WebAreea-Litt Similar to Androidddd,... WebArena-Lii is also a GUI-based environment. In the original paper of WebArena (Zhou et al. 2023). they use HTML/Accessbilltyy Tree the input, whereas in VAB, we focus on visual inputs and mainly focus on screenshots to predict actions each step.
18

---

Published as a conference paper at ICLR 2025
The agent must carefully adjust the CSS style file until the rendering matches the given screenshot. It needs to perceive fine- greiiraind visual differences between two screenshots, which can often be minimal. This makes VAB-CSS highly vision-intensive task.
VAB-CSS:
A.3 DETAILS ON TASK INSTANCE COLLECTION
For VAB-OmniGibson. a prototype is a general household activity, such as recycling office papers. We source these prototypes either by sampling from Behavior- 1K or by annotating them ourselves. Instantiating prototype involves grounding it in specific scene (e.g., specific rooms with office papers and recycling bins) generated OmniGibson. To increase task diversity, we instantiate each prototype with multiple random scenes and various initializations of object positions in the room. In total, we collect 992 instances using 89 prototypes. We sample 181 out of them as our test set.
For VAB-Minecraas instantiate them with game configurations using different world seeds or spawn points. We manually check to ensure that each high-level goal is achievable within its configuration. In total, we collect 628 task instances using high-quality prototypes defined by us, with 116 instances designated as the test set. Additionally, we sample 132 task prototypes from JARVIS-1, resulting in 596 task instances that could be leveraged to collect our training trajectories later.
target high-level task prototypes related to object collecting and then
For VAB-Androidl.ab we first select typical Android applications, from system services third- party applications (e.g.. Maps, Music, etc.) that could be evaluated offline. We come up with 119 test instructions for them and prepare valid groundings in the AVD snapshot (e.g., an MP3 file to play in the Music APP). For the training task construction, we filter 18 commonly used APPs and summarize their major functions to around 70 task prototypes.
For WebArena-Lite, we filtered and cleaned 165 test samples from the original WebArena dataset and collected new task instances for web applications to use in training trajectory collection. Specifically, we summarize each website's basic functions and valid items for syntheric queries, created 40 task prototypes, and fill them with valid and invalid items (e.g., product categories es, pr ces) to generate specific instructions, resulting in 1,186 training task instances.
For VAB-CSS, task prototype simply corresponds to one possible corruption of CSS rule such as adding altering a CSS property. To instantiate task for specific website, we randomly select corruption that results in noticeable visual changes, determined by an SSIM (Wang et al.) 20044) score below 0.8 In addition, we manually annotate each instance with natural language description of the difference between the two images as an additional clue to the agent. In total, we collect 1,210 instances and use 165 form the test set.
A.4 DETAILS ON TRAINING TRAJECTORY COLLECTION
For VAB- OaB--mniGiiboon, we adopt the program program-based based solvers focusing on the cost and adaptability. OmniGibson has no friendly interface for humans to operate on, and requires high-end laptops with GPUs supporting ray tracing and large main memory (> 10 GB) to run. Thus it is unlikely for us to find a large number of qualified annotators to label for OmniGibson. LMM agent bootstrapping is fine but uneconomical, as the task usually takes more steps than others (i.e., up to 100). Program-based solvers, instead, are suitable for collecting massive high-quality trajectories in OmniGibson.
For VAB-Minecraft, we adopt LMM agent bootstrapping considering adaptability. Minecraft requires some flexible explorations (as environments are generated randomly), which is beyond the scope of program- based solvers. Humans need to be well-trained for some time on playing Minecraft before becoming qualified annotators. Since previous work has explored the usage of memory augmentation (Wang et al., 2023c) for improving LMM agents in Minecraft, it becomes practical to leverage the bootstrapping strategy by LMM APIs such as gpt-48 4o for creating training trajectories.
For VAB-AndroidL we primarily adopt human demonstrations, accompanied with some L.M Agent Bootstrapping considering the versatility and flexibility. As android XMLS are less legible and operable than HTMLs on web with existing automation tools, program-based solvers are not applicable. Additionally, for many apps require login and internet connection, human demonstration
TThis is empirical choice hased our own experience.
19

---

Published as a conference paper at ICLR 2025
is the best solution. LMM agent bootstrapping is employed in some offline APPs such as system settings to enhance trajectory flexibility.
For JAB-WebArena-Lite we adopt program prgrdmm-baedd solvers due to cost and adaptability. On the one hand, there have been a mature web automation tool Playwright that supports Python. On the other hand, although WebArena (Zhou et al. 2023) is adopting some mirror websites for their real-world counterparts, their interfaces could be vastly different (e.g., OpenStreetMap in WebArena vs. Google Maps in real-world), Consequently, human annotators struggle to label demonstrations on these websites efficiently in our preliminary trials. For LMM agents, they tend to perform too poorly under mere prompting on WebArena (with success rate less than 20%) for efficient trajectory construction.
For VAB-CSS, we adopt LMM agent bootstrapping, mostly owing to concerns on flexibility. A critical challenge for the agent in debugging CSS styles is to iteratively adjust the CSS rules through a trial and error process, which can be flexibly achieved using the LMM agent bootstrapping scheme. In particular, we first use gpt 4o to collect trajectories that finally resolve the CSS issue. However, gpt- 4o can only achieve a success rate lower than 40%. To collect additional trajectories, we hint the agent with the target CSS rule to edit, after 5 steps of trials, on tasks where the agent initially fails.
A.5 DETAILS ON PROMPTING AND TRAINING OPEN LMMS IN EXPERIMENTS
Prompting. We format L.MM-s--isuull-oumee Foundalloundtion-AAeee Agent as two roles (i.e., user and assi istant) interacting in multiple rounds. The task description, action spaces, few-shot demon- strations, and important notices for each environment are formatted as the system prompt at the beginning of the conversation. Task instruction is given in the first user round. Environmental observations and feedback are passed via user in later rounds. Considering current LMM APIS' poorer support of multi-image image and outrageous cost when interaction rounds soar up, in Embodied and GUI agents we only offer the vision input of the latest user round (following (Koh et al., 2024)) while reserving history text contents. An exception is the CSS agent in Visual Design. In this case, comparing differences in visual inputs is essential, and the interaction rounds are typically fewer than 10. Therefore, retain all image inputs in the conversation history for this task.
Training for Open LMMs. We generally follow the prompting format of proprietary LMM APIs in each environment to organize our training trajectories, and make several minor modifications. In the system prompt we remove the few-shot demonstrations as we would fine-tune models. In addition, during fine-tuning, since open LMMs perform poorly on multi-image input (especially for CogVLM and CogAgent, whose expert architecture disallows simple implementation of multi-image input), we only use the vision input of the latest user turn, and concatenate histories together using role tokens (i.e.. "< juser | >>") and linebreaks. For CSS agent where multi-image input is necessary, we concatenate history images vertically into one as the input. To benchmark the potential of LMMs to serve visual foundation agents, we conduct multitask fine-tuning over the dataset aggregation of all environments. To optimize performance, all LMMs undergo full-parameter fine-tuning, with configured using the default
batch size of 64 and 5k training steps. Other hyperparameters ones provided by the model's 's original repository or the third- hird-partyyss 's integrated training framework. For data composition, we uniformly combine all training samples except for VAB-CSS, which we duplicate an additional 2 times as the preliminary experiments show that the task requires more extensive training for open LMMs adapt to the screenshot concatenation format.
A.6 ANALYSIS ON AGENT ERROR MODES
Here we provide some statistics about major types of errors observe, by sampling around 20 error traces in each environment for go and internvl-2 (in Table 7 and Table 8)
Visual Grounding Error: Wrong detection or recognition of objects/elements in the visual observation.
Invalid Action: Outputting wrong formats of actions.
Loop: Agent repeats generating the same actions without quitting.
Task Limit Exceed:
The agent does not accomplish the goal within reasonable maximum steps.
20

---

Published as a conference paper at ICLR 2025
Hallucinated Task Completion: Agent makes wrong judgment whether it has accomplished the task.
Table Statistics on gpt 40 agent error modes.
Tosks
Visual Grounding Error
Invalid Action
Loop
Task Limit Exceed
Hallucinated Task Completion
AAB-OmniCitoon
0.30
0.04
0.17
0.17
0.30
VAB-Minecratt
N/A
0.00
0.24
0.76
0.00
WebArena-Lite
0.15
0.10
0.40
0.05
0.30
YAB-AndroidLab
0.10
0.00
0.65
0.15
0.10
VAB-CSS
N/A
0.00
0.05
0.55
0.40
Table 8: Statistics on internvl-2 agent error modes.
Tasks
Visual Grounding Error
Invalid Action
Loop
Task Limit Exceed
Hallucinated Task Completion
AAB-OmniCihson
0.00
0.25
0.50
0.25
VAB-Minecrall
N/A
0.00
0.76
0.24
0.00
WebArena-Lite
0.05
0.00
0.40
0.10
0.45
VAB-AndroidLab
0.05
0.05
0.60
0.05
0.25
VAB-CSS
N/A
0.00
0.45
0.30
0.25
A.7 PROXY METRICS FOR EVALUATING PROGRESS OF TASK COMPLETION
For embodied and visual design problems in VAB (including 3 environments: VAB- MAB-OmniGibson, VA -MMinecraft,, and VAB-CSS), we are setting proxy metrics for evaluating progress of task completion (in Table 9).
VAB-OmniGibson: To complete task, the LMM agent must achieve multiple subgoals (c.g., opening specific door). Upon task termination, we compute the percentage of successfully achieved subgoals to provide an intermediate score.
VAB-Minecraft: To acquire the goal item, the LMM agent must gather series of items as ingredients. Consequently, we allocate intermediate scores to the agent as they collect these ingredients.
VAB-CSS: To fix the CSS style to match the target screenshot, we can use screenshot similarity proxy metric for measuring progress of completion.
Table 9: Proxy Metrics for Evaluating Progress
Models
NAB-OmniGibson VAB-Minecraft
VAB-CSS
gpt gpt-4o-2024-05-13
62.6
61.9
46.7
apt-4-vision-prevveew
58.4
55.9
38.2
gpt-4-turbo-0409
50.6
59.7
37.6
2laude-3.5-scmet-220000
59.7
63.6
20.6
claude-3-opus
33.8
60.1
24.8
gpt-4o-mmini-2024--7-11
41.5
37.3
23.6
gemini-1.5-pre
46.3
49.4
13.9
gemini-1.0-pro
12.7
15.3
0.0
qwen-vl-max
1.3
9.0
3.0
InternVL-2
41.6
35.5
32.7
Owen2-VL
24.1
29.8
34.5
GLM-4V
32.2
29.4
29.7
LLaVA-NeXT
17.3
30.5
23.6
CogVLM2
26.9
32.6
20.6
CogAgent
33.0
32.2
17.0
CogVLM
17.1
32.3
11.5
LLaVA-1.5
14.9
33.5
9.1
Owen-VL
18.8
26.5
4.8
21

---

Published as a conference paper at ICLR 2025
A.8 ANALYSIS ON ERROR RECOVERY BEHAVIORS
analyze the average steps needed for agents to recover from emror the correct directions (as below in Table IO] based on proprictary gpt-4o and opened glm- 4v, only for those finally successful tasks). We find that GUT tasks usually require more steps recover, the action spaces for them very large (e.g., any clickable elements on the web pages). And fine-tuned glm-4v has shorter mean error recovery steps compared to gpt 4o, probably because it can only recover from simpler errors.
Table 10: Statistics average steps recover from emror
Models
VAB OmniGibson VAB-Minecraft Webarena-Lite VAB-AndroidLab
VAB-CSS
190-44-2024405-11
2.5
3.3
6.0
8.3
2.6
glm-4v
2.3
2.3
4.0
N/A
2.2
B VAB-OMNIGIBSON
In this section, we provide additional details about VAB-OmniGitson that are not covered in the main paper due to space limitations.
B.1 DETAILED DESCRIPTION
Current household datasets or benchmarks are not originally designed for LMMs, making them less suitable for evaluating today's LMMs. Behavior- (Li et al.) 2023) offers action space focused on low-level physical control over the robot joint angles), while Alfred (Shridhar et al.) 2020a requires actions output masks on images, which may be practical for most LMMs. The ThreeDWorld Transport Challenge (Gan et al. 2022) provides high-level action APIs, but the simulator environment less realistic and the tasks may not fully challenge LMMs. The recent work Octopus (Yang et al., (223a) sets up household tasks for LMMs in the OmniGibson simulator. However, this setting, vision input less critical as the observed objects are also listed in text input for LMMs.
In order set up realistic and challenging benchmark for testing LMMs' embodied planning ability, we select the recent household simulator OmniGibson (Li et al., 2023) as the interactive environment, and build a pipeline for L.MM to serve as a high-level planner on everyday household activities. An example of the task is shown in Fig. 7 The ego-centric image with annotated bounding boxes, high-level activity instruction and environment feedback are fed into the LMM, and it is tasked with reasoning over the current progress to decide the next low-level action. It must interact with objects using the corresponding tags attached to the bounding boxes.
Test Set. We select 45 activity instances from Behavior- IK, and manually adapt some of them to ensure these activities are solvable within our provided action space and suitable for evaluating current L.Mss embodied planning ability. We instantiate each activity in several scenes, resulting in a total of 181 test task instances. All the activity instructions are manually annotated by us.
Training Set. We provide set of successful trajectories using both program-baset solving and LMM bootstrapping. We newly design 47 activities, each instantiated in several different scenes with various initializations of object positions, resulting in a total of 901 task instances. To solve these tasks, we develop program-basse based solver that decomposes the angghorizon activities into subtasks and solves them sequentially. Running the program-basee based solver on the 901 training task instances yields 785 successful trajectories. Then we manually add type of error recovery process (agent fails to place an object into a closed container, and then opens the container) into these trajectories, aiming to enhance LMMs capability rectify errors. Additionally, select 464 training instances and utilize gpt- - -vvision wwwiiiiipreviieeeineeeeeeeeeeeee preview to bootstrap 87 successful trajectories, resulting in a total of 872 training trajectories.
Metrics. We adopt task success rate as the metric of MAB-OmniGibson. In Behavior- KK, each activity is defined in the form of BEHAVIOR Domain Definition Language (BDDL) (Srivastava et al. 2022). which describes the concrete initial and goal conditions of a specific activity. Only when all the goal conditions are met within the limit of 100 turs,, the task is judged successfully completed.
22

---

Published as a conference paper at ICLR 2025
B.2 ACTIONS
In VAB-OmniGibson, we provide the LMM agent with 20 low-level actions to interact with objects and navigate the household environment. The actions marked with asterisk (*) adapted from OmniGibson, while the others are newly defined and implemented by us. With these provided actions, the LMM agent possible to solve all the testing instances.
grasp: Grasp a specific object into the robot's hand.
move: Move towards specific object.
move to room: Move to specific room in the house.
left: Tum the robot left 90 degrees.
turn turn.right: Turn the robot right 90 degrees.
raise camera: Raise the camera of the robot to see higher objects.
lower camera: Lower the camera of the robot to see lower objects.
put inside:: Place the object from the robot's hand inside another object.
put_on_top: Place the object from the robot's hand on top of another object.
put under: Place the object from the robot's hand under another object.
put next to: Place the object from the robot's s hand next to another object.
get ree.fridgeevieww Obtain the view inside nearby fridge.
cook Cook specific object.
burn#: Burn specific object.
freeze": Freeze a specific object.
heat#: Heat a specific object.
open*: Open a specific object.
close#: Close specific object.
toggle on Turn on specific object.
loggle_off's Turn off specific object.
B.3 PROGRAM-BASED SOLVER TRAINING TRAJECTORY COLLECTION
BDDL task goals. Among activities of VAB-OnniGibson, each of the BDDL task goal can be decomposed into sequence of subgoals (e.g.. a specific door should be open, or specific bottle should be a specific countertop). All subgoals can be categorized into 2 types: identifying the state specific object, or the positional relationship between two objects.
Method of program-based solver. To achieve the BDDL task goal of a VAB-OmniGibson activity, the program-base based solver need to sequentially fulfill all the subgoals. For the first type of sub- goal, the program-based solver can navigate (move to to.rom,, move, turn turn_right to find the specific object; and then
raise seise-camera, 1ower_camera, get.fridge.vit move towards it (move) and change its state (cook, burn, freeze, heat, open, close, toggle_on le_on, toggle_off).. For positional relationships, the solver should find and approach an ob- ject, grasp it (grasp), move the other object, and finally complete the subgoal with put. in de, put_on_top, put_under or put_neet__to.
B.4 PROMPT EXAMPLE
The system message that describes the detailed task information to the agent shown as follows:
Setup
You are an intelligent agent exceling at solving household tasks. You are in a household environment given a task to finish.
You can interact with the envirenment by performing actions using python- style pseudo code. For each turn, please call exactly predefined action.
Valid Actions
Predefined Action List:
23

---

Published as a conference paper at ICLR 2025
1.bottie
3
bottle
8.countertop
botile
Figure 7: This an example task of AAB-OnniGibson. The task asks the agent to bring all 4 bottles from the countertop into the fridge. The agent should grasp one bottle, navigate to find the fridge, open the fridge, put the grasped bottle into the fridge, and then repeat this process for the remaining bottles.
def (cbj)
Grasp the object in your hand.
Args:
:param obj: the digital identifier of the object to grasp.
Returns:
A string message the environment feedback.
def move (obj):
'Move yourself towards the object.
Args:
:param obj: the digital identifier of the object to move towards.
Returns:
A string message of the environment feedback.
...
def move_to rro_roommroos
'MMv yourself to a random position in the room.
Args:
:param the name of the room to move
Returns:
A string message the environment feedback.
def turn_left()
"'TTurn the robot left 90 degrees.
Returns:
A string message of the environment feedback.
def (urn_right():
'rrTurn the robot right 90 degrees.
Returns:
A string message of the environment feedback.
def [iise_camera(:
'aise the camera to see objects that are higher.
Returns:
A string message of the environment feedback.
def 1ower_camert ()
24

---

Published as a conference paper at ICLR 2025
Lower the to objects that lower. Returns:
A string message of the environment feedback.
def put (obj_n, obj2):
"ruu cbjl within your hand inside obj2. If obj2 is openable, make sure it open before putting objl inside.
Args:
:param objl: the digital identifier of the object to put inside. :param obj2: the digital identifier of the object to put inside of.
Returns:
A string message of the environment feedback.
def put on topiobil, obj2):
PPu cbjl within your hand te the top objj.
Args:
:param objl: the digital identifier of the object to put on top. :param obj2: the digital identifier of the object to put top of.
Returns:
A string massage of the envi ronment feedback.
def put under (objl, obj2):
'Pu ebjl within your hand the bottom of obj2.
Args:
:param objl: the digital identifier of the object in your hand. :param obj2: the digital identifier of the object to put obji under.
Returns:
A string message of the environment feedback.
def to (objl, obj2):
'Putut ebji within your hand next to obj2.
Args:
:param objl: the digital identifier of the object in your hand. :param obj2: the digital identifier of the object to put obji next to.
Returns:
A string message of the environment feedback.
def get (ie
"'rGet the image captured by camera in the fridge. This function only valid when you are near a fridge and the fridge is open.
Returns:
A string message of the environment feedback.
""
def cook(cb))
cook the object.
Args:
:param obj: the digital Identifier of the object to cook.
keturns:
A string message of the environment feedback.
def burn (cbi):
"rrn the object.
Args:
:param obj: the digital identifier of the object to burn.
Returns:
A string message of the environment feedback.
def (rbe)e((bj)*
"''Freeze the object
Args:
:param obj: the digital identifier of the object to freeze.
25

---

Published as a conference paper at ICLR 2025
Returns:
A string message the environment feedback. ..
def heat (cbi);
Heat the object.
Args:
:param obj: the digital identifier of the object to heat.
Returns:
A string message of the environment feedback.
...
def open (obj):
Open the object.
Args:
:param obji the digital identifier of the object to open.
Returns:
A string message the environment feedback.
def close (cbj):
close the object.
Args:
:param abj: the digital identifier of the object to close.
Returns:
A string message of the environment feedback.
...
def toggle on (ebi):
Toggle on the object.
Args:
:param obj: the digital identifier of the object to toggle
Returns:
A string message the environment feedback.
def togglc_off (obj):
Toggle off the object.
Args:
:param obj: the digital identifier of the object to toggle off.
Returns:
A string message of the envi ronment feedback.
''
def done():
'''Call this function if you think the task is completed. Note that you have no chance to take any actions after calling this function.
Returns:
None. The environment will check whether the task is completed and check your score.
## Reminger
1. You can only hold one object at a time.
2. When moving to new positicn, you can always turn Left, turn right, raise carera or lower camera to see around before making decision.
3. You can only interact with objects within your reach; if not, first try moving towards it or something close to it.
4. You can only interact with objects that are visible to you (annotated with bounding box in the image); if it's not visible, try to move other
inside the open refrigerators or other enclosures
and look around find it. You
see inside them.
5. You can interact with objects that are very close to you, such as those you' ve just moved towards, even if you don't see them currently
6. When you are out of the room and nothing useful, try moving room.
7. You can always move to something in the same room with you, if you have seen it before, even though you cannot see it now, So when you in room, try to around and around to record
26

---

Published as a conference paper at ICLR 2025
objects in your observation that you can move them flexibly afterwards.
8. Don't repeat the failed action in the next round. Try to understand what went wrong and make a different decision.
9. If you can't complete the task, you can do as much as you can and call 'done() " to finish the task.
Input
For each dialog, will De given the following information at the
beginning.
Task Geal: The task shni only when these conditions are met.
2. Reachable Rooms: Rooms you can move to. Please refer to them with their nares provided here.
For each turn, you will be given the following information.
1. Action Feedback: Environment feedback the last action.
2. At Hand Object: The object you are currently holding.
3. Current Room: The room you are currently in.
4. Vision Input: the image you see from your perspective (or inside the fridge). All task-related objects appear in your vlew will be annotated with bounding boxes and unique identifiers. Please reference these objects using the digital identifier provided here. Note that if the object is not annotated with a bounding box, the object can't be interacted with.
Output
Now, given these information, you need to think and call the action needed to proceed with the task. Your response should include parts in the following format in each turn:
OBSERVATION: <What you observe in the image> Note that the Vision Input image won't be kept the dialog, so make sure you capture all the key information (eg, the identifier of the object you see) here for future use.
ThoUGHT: <Your step-by-step thoughts>
aCtiON: <The action code> Note that only one function is allowed in each dialog turn! Only one line of code allowed in each dialog turn! If your output contains multiple actions or multiple turns of actions, only the first one will 1 be executed!
Here is concrete example of the task input shown in Fig. where the image is enclosed within "(0)"
Your task is: There 4 beer bottles on countertop in the kitchen. Please put all then into the fridge.
The reachable rooms during the task are: corridor_0, dining_room_O, kitchen_0, 1oving_roon_a, pantry_room__, storage_room_O. Astion Feedback; None actions before.
At Hand Object: None.
Current Room: kitchen_0 0. Vision Input: I [IImage]]
C VAB-MINECRAFT
In this section, we provide additional details about VAB-Minecraft that are not covered in the main paper due to space limitations.
The game Minecraft has become popular open-world environment for developing generalist embod- ied agents (Fan et al. 2022 Lifshitz et al. 2024) due to its diverse tasks (e.g., survival, harvest, craft, combat, and creative tasks), varied environments, and interactive mobs, necessitating generalized agent abilities. Recent pioneering works (Zhu et al. 2023 Wang et al., 2023a c) have integrated LLMs into embodied agents to tackle Minecraft tasks. However, these efforts did not focus standardized pipeline for evaluating LMMss planning abilities. So we adapt the JARVIS-1 (Wang et al. 2023c) pipeline assess LMMs' high-level action planning abilities in item-obtaining tasks.
27

---

Published as a conference paper at ICLR 2025
C.1 DETAILED DESCRIPTION
In VAB-Minecraft, we adapt the action space of JARVIS-I Wang et al. 2023c) to develop a pipeline for LMM, enabling to function as a high-level embodied planner. Additionally. we also use item-obtaining tasks to benchmark LMMs' high-level embodied planning abilities. These tasks are comprehensive, requiring task analysis and decomposition, as well ingredient collection. Each aspect respectively challenges LMM agent's planning skills and its ability to interact with the environment.
Test Set. We manually annotate 1 116 different tasks, each with a specific target item and a cor- responding initial configuration to ensure the task solvable. For example, Fig. g illustrates the VAB-Minecraft task of obtaining cake, where we have set up the initial configuration of necessary surrounding resources and inventory items. These 116 test tasks span the Minecraft tech tree, covering items across material levels (wood, stone, iron, gold, diamond and netherite) and involving a diverse range of raw ingredients from various resources: types of plants, types of animals, and 6 types of hostile mobs. This diversity greatly challenges the agent's ability to interact with the environment.
Training Set. Training trajectories are collected using bootstrapping from two sources: pure gpt-4-turbo bootstrapping on newly designed tasks, and gpt-4o 4o bootstrapping with JARVIS-1 memory on tasks from JARVIS-1. For the first type, we design 40 new tasks instantiated in different world seeds or spawn points, resulting in 512 task instances, and gpt uutuoo bootstraps 176 successful trajectories. For the second type, we use 132 tasks from JARVIS-I, set in 596 task instances, and run with memory using gpt 4o, resulting in 206 successful trajectories. In total, gain 382 successful trajectories.
Metrics. We adopt success rate as the evaluation metric VAn-Minerraft. For specific item- obtaining task, if the agent can obtain the specific item within the limitation of 100 rounds, the task is regarded as successfully completed.
C.2 ACTIONS
In VAB-Minecraft, we provide types of actions for LMM agent. 4 actions, marked with an asterisk (*), are adapted from the JARVIS-1 pipeline (Wang et al. 2023c), while the remaining 2 are newly implemented by us to enhance the LMM agent's capability to solve a wider range of tasks.
craft*: Utilize the inventory or crafting table to craft specific item.
smelt*: Utilize furnace smelt specific item.
equip=: Equip a specific item the agent's hand.
celeport.to_spave Teleport the agent back to the spawn point. As we will prepare necessary ingredients around the agent's ' spawn point, this action enables the agent conveniently collect these ingredients. This function is also helpful the agent stuck somewhere (c.g., underground),
look_un up:: Look up the crafting/smelting information about a specific item. This reference guides the agent to make plan on how to accomplish the task.
execute:: Use natural language prompt to instruct a low-level minecraft planner, Steve-1 Lifsshitz jt al. 2024). With proper prompting, it can solve most basic tasks, like mining common blocks, collecting plants, interacting with animals and hostile mobs, and navigating between different biomes.
C.3 PROMPT EXAMPLE
The system message that describes the detailed task information to the agent is shown as follows:
Setup
You are a skilled Minecraft player. You are born in the survival mode and asked to obtain specific item. You can interact with the game environment by outputing actions using pythen-style pseudo code. For each turr, please call exactly one predefined function.
Valié Actions
Predefined Function list::
28

---

Published as a conference paper at ICLR 2025
Hove with a, 5 and June with Space
Figure 8: This an example of VAB-Minecraft task. This task asks the agent obtain a cake in the inventory. Initially, we provide 3 buckets and 64 logs in the inventory. Additionally, we grow mature wheat and sugar cane front of the agent and spawn few chickens and cows around it, ensuring that the agent can conveniently find the necessary ingredients.
...
def craft (item: str, num: int 1):
''Crratt specified number of itens. Please ensure that get enough ingredients and craft_table in your inventory.
Args:
obj: the name the item to craft.
num: the number of items to craft. Default is 1.
Returns:
A string message about whether the crafting is successful.
Examples:
>>> craft ("wooden_ag cecooden_pickee pickae"")
Successfully crafted 1 vccden_pickaxe.
>>> craft ("bookshelf", 2)
Not enough materials for bookshelf.
You don't have 12
planks and 6 books in your invertory.
def smelt (item: "r'smelt specified number of itens. Please ensure that you get enough fuels, ingredients, a furnace and r=xooden_pickawt in your inventory.
int 1):
Args:
the item to smelt. num: the number of items smelt. Defauit is 1.
obji the
Returns:
A string message about whether the smelting is successful.
Examples:
>>> smelt ("iron_ingot",
Successfully smelted 2 iren_ingot. >>> smelt ("glass")
Net enough fuels. $ You don't have enough coals, logs planks fuel.
...
def cqti: stri:
rrrselect item from your inventory to your hand. Note that if you want some tem, you must equip it first!
29

---

Published as a conference paper at ICLR 2025
Args:
item: the name of the item to equip.
Returns:
A string message about whether the equipping is successful.
Examples:
>>> equip aataamoodd_worrd
Successfully equipped diamend_sword
>>> equip ("diamond_axes axx")
Can not find diamond in inventory. # You must have the iten in your inventory betore equipping it.
...
def te teleport_to_spann
telepport yourself to the spawn position.
Args:
None.
Returns:
A string message about whether the teleportation is successful.
Examples:
>>> celeport_ttospamn
Successfully teleported.
def Book_uplie (item: stri:
Look up the information about crafting the item.
Args:
item: the name of the item/tag to look up.
Returns:
A string message about the information of the iter. Note that if the argument is tag, information about all possible items will be returned.
Examples:
>>> look_up "'iroo_piccaxe""
iron_pickaxe: Crafting iren_pickaxe needs 2 stick, 3 iron_ingot. Every time you craft iron_pickaxe with the ingredients above, you will get 1 iron_pickaxe.
>>>
stone_toal_material is tag. Fallowing items beleng ong to this tag cobblestone,, blackstone.
cobblestone: is raw iter. you can mine from the environeent. blackstone: It is raw item you can mine from the environment.
def execute (prompt: str, goal_item: Optional [str] None, num: Optional [
int] None)
Instnsutct a lover-level executor model to perform sore simple tasks
like mine something, collect something, move to some place.
Args:
prompt:
the prompt to instruct the lower-level executor model. should be a simple #xverb-object phrase**.
iten (optional): the name of the item to obtain during the execution. If the item is obtained, the executor model will stop.
num (optional): the number of items to obtain.
Returns:
A string message about the execution.
Negative Examples: examples that may Your Inventory: Now your inventory has 1 stone_pickexe, stick. Equipped Item: Now you hold the stone_pickake in your hand.
failure
>>> exccute ("break iron_ere blocks", "iron_ore", 64)
The executor has reached the maximum number of steps for this without completing your subgoal. Each time, 64 iron_ore is too much for one turn.
limited
Your Inventory: Now inventory has 1 weoden _xe, 12 logs, 4 stick, seed, iron pickaxe.
30

---

Published as a conference paper at ICLR 2025
Equipped Item: Now you held the wooden in your hand. >>> execute ("find and mine diamond", "diamond_ore"" 1) The executor has reached the maximum number of steps for this turn without completing your subgoal. # You are not holding the right tool for mining diamonds. You should equip the iron_pickaxe first.
Your Inventory: Now your inventory has 64 dirt. Equipped Item: Now you hold nothing in your hand. >>> execute ("elimo
tree")
The executor has attempted to execute the action according
your prompt. You should check whether your intention has been fulfilled. The executor can't plan for complex tasks; you have to break down complex tasks into simple ones. For example, break down the task of 'alimb tree', 'use dirt blocks to elevate', and 'jump on the tree
tree" into 'find
Your Inventory: Now your Inventory has nothing. Equipped Item: Now you held nothing your hand.
>>> execute ("dig a hole and jump in")
Error: No complex sentences alloved. Keep the prompt simple * verb-object phrases**. # Your prompt contains multiple tasks that may be confusing to the executor.
Your Inventory: Now your inventory has 4 logs.
Equipped Item: Now you held nothing in your hand. >>> execute ("craft wocden_axe", "wooden_axe", 1) Error: You cannot
'execute' to craft items. Use 'craft" instead. # The executor cannot craft or smelt items, call craft for 'smelt' function instead.
Your Inventory: Now your Inventory has 4 logs, 1 crafting_table. Equipped Item: Now you held nothing in your hand.
>>> execute execute("place crafting_table")
Error: You cannot use 'execute' to craft items or place the crafting_table Directly use 'craft inatead. No need to place the crafting_table
# The 'craft function will automatically place the taatingg_table during crafting.
inventory has nothing. Equipped Item: Now you held nothing in your hand. >>> execute ("hold down left button to punch the tree to collect wood", "logs", 1)
Your Inventory: Now
The executor has reached the maximum number of steps for this turn without completing your subgoal. # The description of the task is too complex, it should be exverb-object phrase **.
Positive Examples: good examples for reference
Your inventory: Now your inventory has stone_pickaxe, stick. Equipped Item: Now you held the stone_pickake in your hand. >>> execute execute("bres) ("break iron_cre blocks", "iron_ore", 2)
Your subgoal has been successfully completed by the executor. $ You have seen the iron ore and you are using the correct tool Note that if you haven't use 'break stone, obtain iron ore" as your prompt.
the iron_ ore, you'd better
Your Inventory: Now your inventory has nothing.
Equipped Item: Now you held nothing in your hand. >>> execute ("collect. weed", "logs", Your subgoal has been successfully completed by the executor. $ The executor can only understand the instructions of simple *vverb-object phrases**.
31

---

Published as a conference paper at ICLR 2025
Your Inventory: Now your inventory has nothing. Equipped Item: Now you held nothing your hand.
>>> execute a hole", "dirt", 4)
Your subgoal has been successfully completed by the executor. $ Your instructions are simple and easy to understand.
Your Inventory: Now your inventory has 1 wooden_axes _xx,, stick. Equipped Item: Now you held the wooden_axe in your hand. >>> execute execute("find
river")
The executor has attempted to execute the action according to your prompt. You should check whether your intention has been fulfilled. The executor has the ability to find the environment you are looking for, despite the possibility of failure.
Prompt Examples: # some simple prompts for reference "chop down the tree", "brcak leaves", "collect seeds", "break a flewer", "dig down", "break stone, obtain iron ore", "break gold blocks", "mine diamond ore", "kill sheep", "milk cow", combat spider", "find river", "break stonea", "break sand blocks", "move out the cave".
## Reminger
1. You can only call one functicn each turn.
2. If you have no idea on how to sclve the task or are unfamiliar with
some iters, please call the 'look_up' function to check the item.
3. For some itens that you can not mine obtain with your bare hand, try equip pickaxe (xcoden_pickaxe stone_pickaxs ...) before mining it.
4. Some necessary resources (e.g., mobs, plants) might be prepared for you near the spawn point. If you' re struggling to find certain ingredients
find yourself stuck somewhere, you use the celeport_to_spawe function to return there.
5. When calling the executor, keep the positive examples es and negative examples in mind! If the executor cannot complete your subgoal, check whether you have the right item in your hand, and try to break your prompt into smaller steps and adjust your subgoal, modify the prompt, or carefully repeat the prompt.
6. Do not repeat the failed acticn in the next round. Try to understand what went wrong and make different decision.
Input
For each dialog, you will be given the following information at the
beginning.
Task Goal: The item you should cbtain in inventery.
For each turn, you will be given the following information.
1. Feeaback en the Action: The feedback on the action you output in the
last turn.
2. Your Inventory: The items in
inventory.
3. equipped Item: The item you are currently holding in your hand.
and 4. location and Orientation including x, Y, 2, Pitch and is the height. Pitch measures the tilt
are horizontal coordinates; of the player's s view: 0, positive values and negative values mean the player is looking horizontally, downward, and upward, respectively. Yaw measures the rotation around the player's vertical axis: 360 degrees north, degrees cast, 180 degrecs south, and 270 degrecs west.
5. Vision Input: What you see from your perspective.
Output
Now, given these information you need to think and call the action needed to proceed with the task. Your response should include parts in the following format in each turn:
32

---

Published as a conference paper at ICLR 2025
OBSERVATION: <What you
in the image> Note that the Vision Input image won't t kept the dialog, so make sure you capture all the key information (eg, the biome or items you see) here for future use. THOUGHT: <Your stop-by-step thoughts> ACTION:
<The action code> Note that enly one function is allowed in each dialog turn! Only one line code allowed in each dialog turn! If your output contains multiple functions or multiple turns of functions, only the first one will be executed!
Here is concrete example of the task input shown in Fig. where the image enclosed within "(O)"
Your task to get a cake in your inventory.
Feedback on the Action: No action betore.
Your Inventory: Now your inventery has 64 oak_1og, 3 bucket.
equipped Item: Now you hold the cak_log in your hand.
location and Orientation: Now you locate 50, Pitch: D. 0.00, Yaw: 0.00.
431.50, 65. 8:
Vision Input: [[mmage)
D VAB-ANDROILLAB
this section, we provide additional details regarding VAB-AndroidLab that are not covered main text due to space limitations.
D.I DETAILED DESCRIPTION
To introduce the Android Eval benchmark, we developed framework including an operational environment and benchmark tailored for agents interacting with Android.
Android Eval benchmark comprises 119 tasks across 8 different apps, offering evaluation suites considering the device's ' and screen's 's state. It implements evaluation frameworks for both the ReAct (Yao al. 2023) and SeeAct (Zheng al. 2024) methods. For reproducibility, the Android virtual device provides standard evaluation virtual machines preloaded with various apps' operation histories and offline data, ensuring that network or temporal factors do not affect evaluations. To simulate real-world tasks, we offer Android virtual machine images with randomized operations, ensuring evaluations do not have start from an initial usage state and enabling more complex task completion recognition based on the machine and current page state.
D.2 ACTIONS
In VAB-AndroidLahh agents required accomplish diverse user tasks through predefined actions.
tap:
Tap element with specific id.
type: Type the message into the input box and press enter if needed.
long press: Tap element with specific id for long duration.
swipe: Swipe with distance and direction.
finish: Finish the task with optional message.
press back: Press back button.
press home: Press home button.
D.3 METRICS
The metric we designed is directly oriented towards task completion. We can directly assess the task's 's success rate by checking whether the operation sequence includes necessary screens or device states that indicate task completion. For example, in setting alarm time, sequentially check if the task sequence includes the correctly set alarm time and if the alarm turned on. Specifically, the metrics designed follows:
33

---

Published as a conference paper at ICLR 2025
Success Rate: We measure the success rate by device state and screen state for the operation task. We measure the success rate for the query task by comparing the model answer with the ground truth.
D.4 PROMPT EXAMPLE
Here is the system prompt
You are an agent that trained complete certain tasks on smartpaone. You will be given a screenshot of a smartphone app. The interactive UI elements on the screenshot are labeled with numeric tags sstarting from 1.
You can call the following functicns to interact with those labeled elements to control the smartphone:
1.tap (index: int)
Taps the UI element labeled with the given number.
Example: (5)
2.text (input_str: str)
Inserts the given text into an input field. Example: text ("Hello, world!")
Since ADB keyboard, if ADB keyboard ON is displayed the bottom the screen, you can use this functior.
If you think that the keyboard is displayed after your previous operation you can try to use this function to input text.
_log__presss (index: int)
Long presses the UI element labeled with the given number.
Example: (cng_press(5)
4. swipe (index: int, direction: str, dist: str)
swipes the UI element in the specified direction and distance. "direction is string that
represents one of the four directicns: up, down, left, right. "dist" determines the distance of the swipe and can be one of the three options: short, medium, long.
Example: swipc(21) "up", "medium")
5. b)ck
Simulates back button press on the smartphone.
6. home
Simulates a home button press on the smartphone.
7. wait (interval: int)
Pauses the execution for the given number of seconds. Default is second
8. finish (message: str)
Ends the task and provides the final output. You can return the final output of the task as a string. example: Minish("Tasi completed")
34

---

Published as a conference paper at ICLR 2025
Now, given the following labeled screenshot, you need to think and call the function needed proceed with the task. Your output should include only action part in the given format:
Action: <The function call with the correct parameters to proceed with the task. If you believe the task is completed there is nothing to be done, you should use finish function. You cannot output anything else except a function call in this field.>
Whenever you think the task is finished, you should use finish function to avoid extra operations.
If you found yourself in loop or the task is not proceeding as expected you might consider changing your operation and try other methods. If you operate same action times, the program will automatically stop.
If tap operation is not working, you can try long press operation.
You can only take action time, please directly call the function.
E WEBARENA-LITE
In this section, we provide additional details regarding WebArena-Lite that are not covered in the main text due to space limitations.
E.I DETAILED DESCRIPTION
WebArena (Zhou al. 2023 is designed evaluate the ability of agents to perform complex user tasks described in high- level natural language in realistic, interactive web environment. To achieve this goal, WebArena presented highly simulated and interactive web environment, which consists of five common websites, including Gitlab, map, forum, online shopping, and content management platform. also equipped with external tools such as sketch pad and calculator, which enhance the ability of the agents to perform user tasks. In contrast to other benchmarks where the agents constrained to act as website users, WebArena proposed innovative ways to simulate different user roles. For instance, they constructed content management platform (CMS) and granted the agent full administrative privileges. This assesses the agent's 's capacity to assume various roles in complex scenarios.
Task Description: As web GUI agents, LMMs are asked to accomplish user instructions on certain websites. For example, OneSt.opShop wehsite, an instruction would be "What do customers say about brush from sephora", and LMM agents should search for the product, enter the review section, and summarize the customer reviews (or turn finding no review). To enable the action of LMM agents with visual input, we implement HTML SoM (Koh) lt al. 2024) annotate operable HTML elements with ids on the screenshot, we also provide list of textual information for all clickable elements. LMM agents generate actions and the id of elements being operated by alaywrighht
Test Set: We build WebArena-Lite, a subset of 165 representative tasks by selection, refinement, and adaptation multimodality evaluation (i.e., sereenshot). Our refinement focuses resolving implausible judge conditions, where 30 tasks are being manually fixed (Cf. Appendix [.4). The implausibility may involve wrong answers, too-strict criteria (e.g., exact .match), impossible tasks due to environment bugs, etc. Additionally, we remove cross-website tasks for simplicity of testing.
Training Set: Creating environmentt-eppenden task instructions and trajectories for training on web could be challenging. In VAB, for each website we first summarize the basic functions and valid items for synthetic queries to condition on. Based on summarized functions, we come up with array of task prototypes (with item placeholders) and manually write playwright scripts as rule-based solvers for each task prototype. We fill task prototypes with both valid and invalid items to yield detailed instructions (later being rephrased by L.LMs for expression
35

---

Published as a conference paper at ICLR 2025
Glue
Projects
Byea
Diliine
Tepis
name
personal
Scell tipsytalltliattr
Acwerble and davt synto lrghtinng twene
Skg o.t
lptcloocasbl pracitinets
Stai Inaeshetoo @ duwm))
Npt0tee gewragg
The A11F meneiilort mike ecora
13
X
litild Demeetiioee commice martaa pattamu
86
2.0 contenllet
Upokied Adt indees
Updated ytorago
Figure 9: This is an example WebArena- Lite task where we use the SoM approach highlight actionable elements. This task requires the agent to modify the user's ss status information. To accomplish this, the agent initially clicks on the user's avatar, which directs them the status shown in the figure. At this point, the agent should select the option labeled (21) Edit Status" in order to access the modification page and complete the task.
diversity), and run corresponding solvers on the website to collect groundtruth trajectories with screenshots and operations. authors create around 40 task prototypes with corresponding solvers, and generating 1,186 valid training samples (i.e., instruction, trajectory, and reward function) for WebArena-Lite.
E.2 ACTIONS
In WebArena-Lite, agents are required to accomplish diverse tasks through series of predefined actions. However, real-world webpages are often complex, and thus, we provide these actions in order ensure simplicity and practicality.
click: Click element with specific id.
hover: Hover element with specific id.
type: Type the message into the input box with specific id and press enter if needed.
press: Emulates keyboard key combination.
scroll: Scrolls the page up or down.
new teb: Opens a new tab in the current browser.
tab focus: Switches the tab with specific index.
close tab: Closes the current tab.
goto: Go to specific URL.
go. back: Go back to the previous page.
go forward: Go to the next page exists.
stop: Terminates the operation, returns the response, and exits.
E.3 METRICS
In real-world web browsing scenarios, there can be multiple ways for an agent to accomplish task. Consequently, WebArena- only considers whether the task has been completed or not, without considering the execution trajectory of the agent, therefore, the metric used in WebArena-Lite is Success Rate (SR). We maintain the evaluation method described by WebArena (Zhou et al. 2023 which can be categorized into three categories based task type.
36

---

Published as a conference paper at ICLR 2025
Question Answering: Agent needs to give answer and the score depends the string- matching result.
Agent must navigate to a specific web page. The completion of the task is dependent on the URL of the page on which the agent terminated.
Webpage Navigation:
Content modification: Agent needs to interact with the environment to modify the configuration of the webpage, and the evaluation will extract the content of the page and match it to check whether the content meets the expectations.
In light of the aforementioned considerations, distinct categories:
patterns can be classified into three
exact_matel The response of the agent is scored when it exactly matches the token sequence
corresponding to the answer.
must include: Answers that contain a specific token sequence are considered a match. fuzzy_match: Utilizes LLMSs such as GPT-4 to assist in determining whether answer is correct.
The selection of appropriate evaluation metrics for distinct types of tasks enables the construction of comprehensive and relatively accurate test set.
E.4 TASK AMENDMENT
Some tasks in WebArena have typos, incorrect answers, and inaccurate scoring criteria. Therefore, we selected 165 tasks from WebArena with different templates and then corrected 39 of them, as shown in Table Considering that the model uses natural language to answer the questions, we change the tasks that require exact match must include or fuzzy match, and also correct the answers.
Table 11: Task instructions fixed in WebArena-Lite
ID
Website
Task
Before
After
Map
Tell the full address of all international airports that exact_mated are within driving distance 50 km Carnegie Mellon University.
fuzzy match
33 Map
I will arrive Pittsburgh Airport soon. Provide the name must must-include of a Hilton hotel in the vicinity, if available. Then, tell the the shortest walking distance supermarket from the hotel.
fuzzy match
Map
Check the police station pittsburgh can be reached must_include in one hour by car from gates building at CMU,
fuzzy_match
43
CMS
List the top 3 search terms in my store.
hollister, Joust Bag, Antonia
hollister, Joust Bag, nike
Racer Tank
65
CMS
Which customer has completed the fifth most number Jane Doc orders the entire history?
Matt Baker
71
Map
What is the zip code of Chatham University?
exact match
must include
82
Map
What is the duration required to first walk from Massachusetts Institute Technology to Harvard University, and then drive Boston Logan International Airport?
63 min
64 min
97 Map
Tell me the distance to drive from Carnegie Mellon University to the top computer science school in massachusetts.
must include
fuzzy -mtch
Map
Where is the nearest tea cafe to University of Pittshurgh, must include and what is the walking distance to it?
fuzzy match
103 Gitlab
Display the list of issues in the kkroening/tfmpegg python repository that have labels related to questions.
URL: sort by
created_date, _dtte state is opened
37

---

Published as a conference paper at ICLR 2025
Table 11: Task instructions fixed WebArena-Lite
ID
Website
Task
Before
After
109 CMS
Presents the monthly count of successful orders ([period)} } MM:COUNT format.
January: 11
01:11, 02:16
orders, February:
16 orders
127 CMS
What brands appear most frequently among the top search terms?
Hollister, Joust, Antonia
Hollister
135 Gitlab
How many commits did Eric and Kilian make yproject on 1/3/2023?
0
167 OSS
What are main criticisms of this product? Please
must include
fuzzy match
extract the relevant sentences.
215 CMS
What are the key aspects that the customers don't like about Circe ice fleece.
fuzzy-match maach ("Material quality....."
exact_math
("NIA")
225 OSS
What do customers say about brush from sephora.
N/A
No reviews
available
235 OSS
Get the order number of my most recent under delivery fuzzy_match order.
must_include
236 Map
Where is the nearest pharmacy from Camegie Mellon I iustuie can walk within 20mins.
fuzzy match
240 OSS
am doing market survey for one stop market, show me the most expensive product from competative swimwear category.
URI.
254 Map
What is the phone number of Western Pennsylvania Hospital.
exact match
fuzzy .match
259 Gitlab
Get me my RSS feed token.
exact mathh
must include
287 Map
How much time does it take from Pittsburgh to Philadelphia by car?
5h 47m
8h 33m
288 CMS
Tell the name of the customer who has the most cancellations in the history.
exact_match
must incluue
296 Gitlab
Show me the command clone the best GAN python implementation with SSH.
metis.lti. cs.cmu.edu
localhost
311 Gitlab
Tell me who has made the most contributions, in terms of number of commits, to the Pytorch GAN project.
exact_match
must inst_include
348 CMS
How many reviews shop received in May 2023?
inst_include
fuzzy match must_include
349 Gitlab
Who else have access my repo gimmiethat.space show me their usernames.
exact_match
367 Map
Measure distance between Carnegie Mellon University exact_match and CVS (closet onc) by walking.
Juzzy_match
376 OSS
Summarize customer reviews for Amazon Echo Dot 3rd N/A generation.
Dizzy_macch ("Serena: it..."
392 Gitlab
Post \ \"Good idea)," for the merge request related color ulitity in al lyproject.com project".
URL
528 OSS
Draft a refund message via their \ \"contact us)." form must include for the phone screen protector bought March 2023. It broke after three days of use. The shop requires the order id, the reason and the amount refund the message. Don't 't submit yet.
fuzzy match
538 CMS
Modify the address of order #299 to 456 Oak Avenue, New York, NY, 10001.
455 Oak Avenue, Apartment SB, New York..
...4 Oak Avenue, New York...
641 Reddit
Post a notice on a virtual meetup for Harry Potter enthusiasts on July 8th in the books subreddit.
iustuinclude
fuzzy_match
645 Reddit
Post in books subreddit about what could machine learning help the correpong field.
must include
fuzzy match
38

---

Published as a conference paper at ICLR 2025
Table 11: Task instructions fixed WebArena-Lite
ID
Website
Task
Before
After
657 OSS
Fill the ("contact us\" form in the site for refund the speaker bought, stating that broke after just three days of use, Also, ensure include the order number #148 and the product SKU. Don't submit yct, will check.
instude
fuzzy_min match
668 Gitlab
Submit merge request for al ypproject.com//edeeiign branch to merged into master branch, assign Roshan Jossy as the reviewer.
Justin Armstrong
Roshan Jossy
693 OSS
Draft an email to the shop owner via their contact us function for coupon my refund is suppoed to be replaced by coupon.
"rogram_mathh url_ _a__macch
798 OSS
Change the delivery address for my recent order fuzzy_match 77 Massachusetts Ave, Cambridge, MA.
must_include
E.5
PROMPT EXAMPLE
Here is the system prompt we use, you can find more prompt examples in VisualWebArena (Koh
et al. 2024).
You are autonemous intelligent agent tasked with navigating web browser. You will be given web-based tasks. These tasks will be accomplished through the use of specific actions you can issue.
Here's s the information you'll 11 have:
The user's s objective: This is the task you' re trying to complete.
The current web page's accessibility tree: This is simplified
representation of the webpage, providing key information.
The current web page's s JRL: This is the page you' re currently navigating.
The open tabs: These are the tabs you have open.
The previcus action: This is the action you just performed. may heipful track your progress.
The actions you can perform fall into several categories:
Page Operation Actions:
"click [id]' This action clicks on an element with specific on the webpage.
.ttyyp [id] [content]""' Use this to type the content into the field
with id. By default, the "Enter" key is pressed after typing unless press_cnter_affee is set to 0, 1.e., tytpp [id] [content] [ hhover [id] Hover over an element with id.
press [key_comb]"">
Simulates the pressing of a key combination on the keyboard (e.g., Ctri+v).
sccolll [down]
scroll [upl""s Scroll the page up down.
Tab Management Actions:
"'new_tab">> Open a new, empty browser tab.
".ttab focus [tab_index]"* " Switch the browser's focus specific fic tab using its index.
"close_tab"" '*, close the currently active tab.
URL Navigation Actions: [url]''; "*;
goto [url]
Navigate to specific URL. go_back" . Navigate to the previously viewed page.
go_ forward **; Navigate to the next page (if a previous 'go_ action was performed).
Completion Action:
39

---

Published as a conference paper at ICLR 2025
stop [answer] '': Issue this action when you believe the task is complete. If the objective to find text-based answer, provide the answer in the bracket.
Homepage:
If you want to visit other websites, check the homepage http:// com. It has a list of websites you can visit.
homepage, com. attp::/homeppe http//hommppeeeeeeee wwtp///aww...d......o html lists all the account name and password for the websites. You can use them to log in to the websites.
very important to follow the following rules: 1. You should only issue an act that is valid gi given the current
successful,
observation
2. You should only issue one action at time.
3. You should follow the examples reason step by step and then issue the next action.
4. Gencrate the action in the correct format. Start with a "In summary, the next action will perform is" phrase, followed by action inside ....., For example, "In summary, the next action will perform is
cllcic [1234] "
5. Issue stop action when you think you have achieved the objective. Don'
t generate anything after stop.
F VAB-CSS
In this section, provide additional details regarding VAB-CSS that are not covered in the main text due to space limitations.
F.1 DETAILED DESCRIPTION
Existing datasets for frontend design have two major shortcomings: 1) They focus mainly on single- round interactions, and They do not provide definitive success metrics for individual tasks. Instead, these benchmarks assess using continuous metrics like CLIP score (Si et al. 2024) or qualitative analysis only (Laurençon et 2024). The reason is that they expect the model to output an entire HTML file replicating the target web design, which is too challenging and unrealistic for current LMMs. Therefore, employing definitive success rate as the metric is meaningless for them. Consequently, they may fail to adequately assess LMMs' potential in serving as adaptive agents that can make new decisions based on varying observations. Also, binary success rate often more decisive and crucial to determine whether agents can faithfully execute human instructions, which essential for practical use. To address these issues, we introduce a VAB-CSS, which better tailored for evaluating multimodal agents. In VAB-CS an agent is expected to strictly take sequence of actions using our provided toolkit to accomplish a task (Section. F33). Specifically. needs to iteratively refine the CSS definition based the rendering outcomes it receives. The more constrained action space based on our toolkit, compared to outputting an entire HTMML file, along with more practical goal for current LMMs (i.e., CSS bug- bug-fixing), makes it possible evaluate definitive success rate for a given task. Additionally, VAB-CSS makes minimal assumptions in terms of simplifying the task environment, such embedding all CSS definitions within single HTML page or replacing images with placeholders in existing datasets. Instead, the agent directly operates over the entire web frontend project fix the CSS style. See a comprehensive checklist in Table
F.2 DATA COLLECTION
Random CSS Corruption. To ensure the task manageable for LMMs, each task instance involves comupting single categorical property of a random CSS rule by either altering its value removing entirely. Note that, even fixing a single corruption is already highly challenging for current LMMSs, and tiny corruption the task's 's complexity in the future by involving multiple corruptions once the single-corruption task has been mastered.
often lead to drastic change in visual effect (see Figure 10). We can increase
40

---

Published as a conference paper at ICLR 2025
Table 12: line-grained comparison of VAB-CSS with existing datasets on web frontend develop- ment. VAB-CSS provides both training and test data. Additionally, its multi-round nature, definitive success rate metric, and multi-file environment make it well-suited as a practical multimodal agent task.
Train
Test
Multi Round
Definitive Eva.
Multi-File Env.
WebSight (Laurençon al. 2024) Design2Code (Si et al. 2024)
x
X
X
X
x
X
X
VAB-CSS
/
/
J
Target Design
Corrupted Web Page
eyaniclols
Cyanictols
Templlata
Terplai Libraa
Scster Manu
Lonens brem Delor St Avret
A
Leve' poar Dr Arril
Search
Niss
Liver Loun
to Push
Loven
Figure 10: This is an example of our annotation task. Authors are shown the target design and corrupted web page side by side to prompt them describe necessary adjustments in natural language. In this example, the instruction is: "Correct the background color of the footer and main section, and adjust the positioning of elements, including centering the website logo in the header by moving it downward." The two screenshots, along with the HTML code and annotated instruction, will collectively serve the initial task input for the agent.
Human Annotations. Existing LMMs struggle to identify the difference between the current rendering and the target design, so we manually annotate each instance with natural language description the difference between the two images. Such natural language descriptions could serve additional clues for the agent to perceive the visual difference (see concrete example of annotation in Figure 10P.
Training Trajectories. To collect training trajectories, primarily sample from the predictions of got-4o on our training instances, retaining the successful trajectories for training. Given the success rate of 4o is around 35%, also sample its trajectories in more lenient setting where the target CSS rule to edit is provided as input. For task instances where gpt-44 succeeds in the lenient setting, we combine its successful trajectory with its failure trajectory in the standard setting to create a more realistic trial-and-error trajectory.
F.3 ACTIONS
In VAB-CSS, the agent is expected to interact with practical frontend project, potentially with numerous CSS files, to fix its style issues. Inputting the impractical and inefficient. Instead, the agent has access only screenshots and the current HTML.
entire project directly into the agent is
41

---

Published as a conference paper at ICLR 2025
code. To facilitate effective navigation and editing within the project, we provide the agent with toolkit. This toolkit allows the agent to locate and edit incorrect CSS definitions seamlessly, without needing to know the specifie file containing the CSS rule.
get selectors by html element: This function allows the agent to locate list of CSS selectors, potentially from various files, associated with an HTML element whose style appears to be incorrect.
select rule: This function allows the agent to check the definition of a CSS rule by specifying CSS selector.
This function enables the agent to update the property value of a CSS rule for specified CSS selector.
revert last edit: During the trial and error, the agent can revert an edit later determines be incorrect.
F.4 METRICS
As discussed earlier, a critical feature VAB-CSS, compared with existing benchmarks, is its definitive success rate evaluation. The most straightforward way to determine whether a task is successfully handled is to check whether the SSIM similarity between the target design and the final rendering is 1.0. However, than 0.9 indicates minimal differences that are hard for humans to perceive Therefore, we define task successful if the final similarity is greater than 0.9. Finally, adopt two metrics entire test set.
have observed that this can be too strict. Typically, SSIM greater
Success Rate (SR): This the primary metric indicating the ratio of tasks in the test that have been successfully fixed based on our definition.
Improve Rate (IR): This metric evaluates the ratio of tasks where the final rendering is more similar to the target design than the initial rendering. serves as a complementary soft metric to the success rate.
F.5 PROMPT EXAMPLE
The system message that describes the detailed task information to the agent shown as follows:
You are a CSS agent. You will be given a target screenshot and an htmi file. Your job is to correct perceive the layout difference between the current rendering and the target screenshot, then accordingly fix the ess rules used in the atml file to match the target screenshot facilitate the process, you can use the following tools provided by the system:
1.
Sometimes, the exact selecter of the rule you want to edit is not clear. This tcol takes the html element specification that could be directly passed soup. find all as input and returns the matched selectors. byyollattorr_bby__mmlloolooo
("'a', ('data-custom'': custom-value' ), string=' haha', class_=' (xxas_="xxx'") xxx"", The argument should be the string representation of valid arguments of the find_all method in feautifulSoup, which means can directly ((argument))) Please strictly stick to the usage of ReautifulSoup. Make sure the arguments are valid (e.g., the tag name must be wrapped with quotes, attributes should be dictionary). You can use this tool to first find the selector of the rule of specific html element whose style you want to fix.
For example,
eval (f"soup. find_a
2. select_rule
This takes the css rule's selecterText as input and returns the rule. You can use this tool to view the properties of rule, which may help you decide which rule to edit. Usually, it's recomnended to first use this tool to view the rule before deciding which rule to edit.
3. edit_rule
*This threshold of 0.9 empirical choice based observations.
42

---

Published as a conference paper at ICLR 2025
This takes the css rule's selectorText, the property nane, and the value the property as input. You can use this tool to change the value of a property of a rule or insert a new property to the rule, if you believe this change would make the rule closer to the target screenshot. Note that, most of the layort issues are related to the categorical properties, such as border, float, diaplay, overflow, position, etc.
4. reeert_last_eeee
This tool reverts the last single edit you made. You can use this tool to undo the last edit, it you believe it mistake. This action takes no arguments.
Make sure the selectorText is valid based on the html file, 1.e., it's from the class or id of the haml elements. In addition, please focus the major layout issue! Ignore the fort size, font family, and color of the text, even you believe they are not perfect.
You can only take ONE action at a time!! For each step, may first state your thought, then take action following the format of Thought:
Action: (do not add any linebreak after the colon).
For example, you may output
"Thought: I think I should adjust the alignment preperty of the rule, because the target screenshot shows the text should be centered. Action: edit_rule! rule (" templatemo menu li', 'text-align', "center')"
After editing rule or inserting rule, you will see the updated screenshot of the html file. You should decide your next action (e.g . revert the last edit keep adjusting the css) based the updated screenshot. please say exactly "I have fixed the css style".
you think you have already fixed the css style
glease strictly follow the format specified above, and please don't repeat the same action in multiple rounds. Also note that, you don't t need to worry about how these tools are executed, your job is just to correctly predict the tool invocation.
Here is concrete example of the task input shown in Fig. 10 where variables enclosed within "(0)"
Here is screenshot of the target design:
([Image 11}
Here is the screenshot of the current web page:
[[Image 2 Here is the HTML code of the current web page:
((HTML file]]
Correct the background color the footer and main section, and adjust the positioning of elements, including centering the website logo in the header by moving it dowaward.
G CASE STUDIES
G.1 VAB-OMNIGIBSON CASES
In this section, show successful and failed examples of gpt-pp4- and InternVL-2 in Figure
Figure 12
Figure 13 and Figure 14
43

---

Published as a conference paper at ICLR 2025
Touad
& Mounston
euu to pilt CPTA
Bound EEe
heed ionterrnaaee corercotee you Ale
fecked
Tok col noonnd serorss
shech codee poopett
weate udbia Freee Duskes
actic
ACTIC
Figure 11: This is an example of go successfully completing VAB-OmniGibson task. Task instruction: "Place the cartons and the money the checkout counter in the grocery store."
44

---

Published as a conference paper at ICLR 2025
Frosl Bedt
Rows Belaie Todbad To cenee
Romed &
ACTES playys
#CTCs rumell
JOTIS perrlt
beberr
rounds omitted here,
Pood
the
ACTER flauid
Fares budrgg
Kown
Mowel
alet and Tiee
Sawe Baood wee rooing
ACTION snauto
This is an example of gpt-4o failing to complete VAB-OmniGibson task. Task instruction: "There are 7 books in the living room. Please put books into the carton, and 3 books on top of the carton." go incorrectly identified the presence of "10.book" in the image, and the erroneous actions are highlighted in red.
Figure 12:
Paed DESP
ofrae Gured ROTO
Artuat Rad
Figure 13: This is example of InternVL-2 successfully completing a VAB-OnniGibson task. Task instruction: "Pick up a document from the breakfast table or the floor, and then throw it into the recycling bin in the private office."
45

---

Published as a conference paper at ICLR 2025
dwert Mapie
ESB DegarOOOLLO Saand Coader toeeo
Coee rechuete actoss yourl uueds
fo Devec
bued
moved pare
ZCTic perof seriid
rounds omitted here.
Puad
Suadal Cuidee Boadds Kulees
Phocad bosee CrraeeDeed
TeL Smonng de temeee
#1 Ded
adder
Srreed Clees
Ouredy lund spnerr ACTER mivihAnndi
Londd Scards To Traless
Figure 14: This is example of InternvI.-2 failing complete VAB-OmniGibson task. Task instruction: "There are 4 candles, 4 cheeses and 4 baskets in the living room. Please place 5.candle and 13.cheese into basket, 6.candle and 14.cheese into b.bkkk,, 7.candee and 15.cheese into basket, c.caddle and 16.cheese into basket.. InternVL-2 didn't adn't put "5.candle" into the basket, but terminated the task with action "done()".
46

---

Published as a conference paper at ICLR 2025
oifacs
Larae comphoa
fu de
This
Figure 15: This is an example of gpt-4p 4o successfully completing a VAB-Minccrafi task. Task instruction: "Your task is to get white bed in your inventory."
G.2 VAB-MINECRAFT CASES
In this section, show successful and failed examples of gpt-4o 4o and InternVL-2 in Figure 15 Figure 16 Figure 17 and Figure 18
47

---

Published as a conference paper at ICLR 2025
R
Loani Pad anon
Bend COR
ACTES
cale the tade
cull'
Loder Ner prop couR
adl 1EES 2u
The ligh oadare
doae shitk
vestioe worh fos nan
Tound
vable
Mavel
Dagin
and doles
uee Bann you
plats
codden
resiuved
Wander for another 25 rounds without finding spider...
and
ank
Moaed co The
th boddeg
comdes irsudd deve
cae
dame and rugaent
Figure 16: This is an example of gpt-pptg 4o failing to complete a VAB-Minecraft task. Task instruction: "Your task is to get rod in your inventory." 4o didn't 't notice the spiders near the spawn point and couldn't 't find any other spiders elsewhere. Instead of wandering around (actions marked as red), it should "teleport to spawn" to find spiders.
48

---

Published as a conference paper at ICLR 2025
Maatl: Ponsiie un
our lack
on ane high
sariy
cymree den thi reeah
dloan the
Nour sout comnent MELAT
Hignn morerrtlyy by
the
Figure 17: This is an example of InternVI-2 successfully completing a VAB-Minecrafi task. Task instruction: "Your task is to get a oak oak fence fence_gate in your inventory."
49

---

Published as a conference paper at ICLR 2025
Soadd Fooded laber andee
Muee und poices de
Daand cna buttin
lags
Erpnc' Eaate on shop
Bour forteen
Anal Sed hoeber
Annad (leg) Ohe
Hignd
Morel
Repeating "exaculaa"" chop down the tree") "stick", for rounds...
Feae Bod Famiu de Tour
Hawrd CENTAAATI OOR
ceeple Sat meee qhappos
Figure 18: This is an example of InternVL-2 failing complete VAB-Minecraft task. Task instruction: "Your task is to get wooden shovel in your inventory." InternVI-2 made wrong plan for it (marked as red in Round 2), and persistently tried to obtain sticks by chopping down trees, which is infeasible (actions marked as red from Round 7 to 16).
50

---

Published as a conference paper at ICLR 2025
G.3
VAB-A AABDRNDDO CASES
In this section, we show successful and failed examples of -ppt-4p 4o in Figure 19 and Figure 20
Check my default browser and change it to firefox
Settings
Apps
Default apps
Default browser app
Default browser app
fo Search settinpe
Perfix
Delauk app
Calle 5
saworig unice
Ares
Aup Lang
Bullory
Dlangg AB
Tap(5)
Tap(3)
Tap(2)
Tap(3)
Finish("Task ciniseeTTssccomlleeee)))
Figure 19: This is an example of gpt-4o successfully completing a VAB-AndroidLab task. Task instruction: "Check my default browser and change firefox"
51

---

Published as a conference paper at ICLR 2025
Check the walking distance and time between Bus Stop of Stanford Campus Oval and Bus Stop of Oxford Street & University Avenue
Search
Search
The Racrooo Morry
Tap(13) 3)
Tap(6)
Tap(8)
Tap(8)
Search
Search
Search
haroo
Repeat Typing "Bus Stop of Stanford Campus Oval" for steps...
shrie
strie.
Tap(8)
Type("Bus Stop
Type("Bus Stop
Stanford Campus Oval")
of Stanford
Campus Oval")
This is an example of gpt-4o failing to complete a AAB--nddoidLaab task. Task instruction: "Check the walking distance and time between Bus Stop Stanford Campus Oval and Bus Stop of Oxford Street & University Avenue" got-4o repeated doing typing without noticing the success of previous typing.
Figure 20:
52

---

Published as a conference paper at ICLR 2025
G.4 WEBARENA-LITE CASES
In this section, we show successful and failed examples of gpt- 4o -o in Figure and Figure 22
is the phone number of Western Pennsylvania Hospital
V
E
M
Let's think step by step. The objective is to find the phone Pennsylvania Hospital. This requires a search action to locate the hospital map. type Pennsylvania Hospital" into the search input box whose ID is (15) press search.
Let's think step hy-step. The objective is to find the phone number Pennsylvania Hospital. Currently, the search resultts show the some
Let's think step by step. The current objective find the phone number Western Pennsykania Hospital.
Let's think step by-styys The objective is to find the phone number Western Pennsylvania Hospital, From the observations ons, there an labeled "phone" followed by the phone number of the hospital. The number 412 578
hospital's additional links. find more information, including the number. should follow the link to more results.
Currently, indicate two different hospitall related entries. Since the goal find Information specific Western Pennsylvania Hospital, need to investigate link that corresponds this hospital.
perform
5020.
summary, the action pertorm "type [15] [Western Pennsylvania Hospital] [1]
summary. the next action will perform is ""click [21)"".
believe have achieved objective. issue the stop action with the answer.
summary, the next action perform to corresponcing to the Western Pennsylvania Hospital to more details.
link
summary, the next action will perform is "ssoop [+ 412 578 5000)""
summary, the next action perform ""click (20)
Figure 21: This an example of gpt-4o 40 successfully completing WebArena-Lite task. Task instruction: "What is the phone number of Western Pennsylvania Hospital"
53

---

Published as a conference paper at ICLR 2025
Promote autht/onguuar storage subreddit technology with the description from the repo Itself.
step.
Let's think step by step. The objective is to promate "suthbtorg gular-ttorage to the subreddit "echnology' with description from
Let's think step by step. The objective is to promote
Let's step
Let's think step-ity-lep. To promote the repositrry to the subraddi: technology, the steps are:
promote authörangulas storage' library ogy' first
"authElomgular objective storage""" to
burhhlangg/ar-strragee to the
subreddi 'technology with descriprion from
button in
reposhory
repacitory
repository isair.
Provide the tite for the post.
necessary datalls. summary,
Enter authorang gular age
ado body to submission.
2. Provide the description the body section.
the action 15)
Inced
repostooy.
will descrission body. Lastly, repesitory the designated forum (subreddit technology'
Provicee URL the
Select approprtane subreddit ( ttechnology case).
post. Enter description
field.
repository.
2. Select the subreddit "echnologg"
Submit post.
technology'
summery. "tpe (10) Angular Storage]"
Submit start by entering
3. Add the description from the repository issalf the body section.
ensering be "Authi Angular Storage" type this idemified (10).
repasttory URL fleld. The UsL
httpps//ggtubb.co www.bb.ooommm.ttrrrr..ooo.o the
Rirst, let' input the URL to the repesitory. The field for the URL has the ID (9)
action will perform
"type [9]
will perform "tyee (10]
(iitps//gthubb httt///ggiuc..cmuttiiiinni
In summarry the nest action will perform is "type (9) http:///ghhu..cmmuutttttt ular storagtrage]"
(Auth0 Angular
ular-storagel"
(Start Repeat Doing.....
Figure 22: This is an example of gpt-4o 4o failing to complete a WebArena- Livena-Litee Lite task. Task instruction: "Promote authO/angular stooage to subreddit technology with the description from the repo itself." got-4o repeated doing typing without noticing the success of previous typing.
54

---

Published as a conference paper at ICLR 2025
G.5 VAB- BBSS CASES
We show successful and failed cases of both gpt 4o and InternVL-2 - in Figure 23 Figure 24
Figure 25 and Figure 26
Darget Uffect
Teads
Sood
pric be lep reprrrirng plorisier sndoonad
wasth und lecuk addu coth
sloc torio
(dled) legit
Figure 23: This an example of gpt- 4o successfully completing a VAB-CSI task.
Tageet Etfect
801
Figure 24: This is an example of gpt-4o failing complete a VAB-CSS task.
Target Effect
Thi cn anteretlly tethe detly
in resic sorten tall whe
aeportos Wen the ba fom
dudta pranttya
ulinia Eo deliie
th velire ACTOO veleren fire dloeent/t))
reneend vall tuiegg thr
Figure 25: This is an example of InternVL-2 successfully completing VAB-CSS task.
55

---

Published as a conference paper at ICLR 2025
Tarpet lfffect
Rower' tf deree aau seta Michoed
fewedy: prgues of ni the aekeoo Woleoe
mapers wit vivess
bod))
pap ASTOKk Rihhond
ACTO0 and Miprree
Figure 26: This is example of InternVL-2 2 failing complete VAB-CSS task.
56

---

Published as a conference paper at ICLR 2025
G.6 ERROR RECOVERY CASES
In this section, we show examples of error recovery in each environment, as illustrated in Figure 27
Figure 28 Figure 29
Figure 30 Figure 31
(Sed Fondiack grapp dooument bedbeck
Roand a Bird "Caret adionia jus, chesed"
(Ra] Acdic Faccbatc Spann
feccted
Dueee Reve piaar 2ii.. 2 disenenn Alain 8/ dise banriow shoud ACTIONE
met Ruc presityatinn offiins
Currci Rooer crtch.. vfiin
Tee, savuueed side
doee
saneed androve
ACTION:
ACTION jnsbeppdscaaeer..
Figure 27: VAB-OmniGibson, the agent encounters a failure when attempting to place the document into closed bin. Upon receiving feedback from the environment, the agent first opens the bin and successfully deposits "2.document" into it.
Roee (Red Fookeck te Autore Your oer
Round Bhod herson on to Acter alles
Foxnd ffeed norbark Aotayy bodl adlen
Plxad 1ildd nochook on Autore Your
Your sdtiial sacinaal dy
snough how crnting saste
bound wosden plank.
cattec
Fn sire Now nosl cal tie uoue
soctinn and Chinaraee New you ch 2i
recaion thee tie you mude ai
hool
doue 2 -00-0 Pedis 25.6, naw d2400.
(lgeed) desorTvA onc
rand
(lgend DeSERVATIO to lawe
tard
roo0 183 a1, Pact 24-14 Van -3000
lp colecctd seiee
ducti need corves "daaee""
wesdion
cachec cas hokding
wer then menooiiin nottii anndeeiarrais
bahi (et- thet
nooden
actichu undlt wesddee, ullnes))
ACTICOe wortt soottypeeury pleeaey)
Teco dhell Kowe out town was slanee, oor cult
ACTIONe cortfrruting sabar))
lanis act moccinn picusse.
Figure 28: In VAB-Minecraft, the agent initially attempts to craft a crafting table without the required ingredients. Upon receiving feedback, the agent first attempts to craft "wooden planks". which is an incorrect ingredient name. Subsequently, it corrects its approach by crafting "planks" and is then able successfully craft the crafting.table
57

---

Published as a conference paper at ICLR 2025
Alm
Alem
w:30
7:30
7:30
9:00
9:00
8:30
8:30
8:30
9:00
4:00
9:00
4 4:00PM
9:00
4:00 PM
4:00
6
M
CoB
Tap(18)
Tap(8)
Tap(188
Tap(28)
Finish(
Figure 29: In AAB-AndroidLab, the agent initially clicks on the correct alarm to be deleted. However, popup unexpected appears and trigger wrong clicking of the agent. After the wrong action, it realizes the error and again perform the correct action.
o urddt romy ptpte
Le's
stpsore lusel up now
dapc no suoee upu -ews odgect we si rew
then
LetS ea s0pp by stop. Thee
Lot3 ch stop by stap.
"yeostie mid Jin reading g Alter
"nnooon, in reading: Tu
uenoeme, Iin neach nnt /hor
sevescee im reeding To do
emoty repoatory with name
omply repoabory namred
laboold ba tiian paye
bock for an option that allows
opthon cheit reav
crsee new project of
oprinnt
this, reed to create new
page to enere a blanc
cenals such extuiname name.
will diikk on the "Frejcns" button
repostiiry
buthoo with (3g
dkkk "Gnoee tonkppogent projec" decals.
enter project
To proceed ineed to thddthe projet same and loulion
option New
the creation a new project
there a link "Vew projecss
in summany the neon action will
propectropoo ttry"
summanng the nest acticn will
lead
1P53 clickng
Prst clicion "Creato blonk projett
rntrrrnn into
summary septictiion
perforn "likk4f"
credte rew project.
summmny, the actoni
Insurmerry roott sotken "clck
Insummary thened adtan hwll (44 jessumme_ "rurediingg
Figure 30: In WebArena-Litte the agent initially searches for the target repository that need it to create. After realizing the problem, it struggles to return to the repository creating page to finally perform the correct action.
Targei Diflct
Oc aeg th or gf
toterrouel vay bebore
and ale
Soad tac
Figure 31: In VAB-CSS, the agent initially edits a wrong CSS rule. After rendering the new webpage, the agent realizes that this is not the correct change to make so it decides to revert the last edit and makes new edit that finally fixes the layout.
58


Reviews:
[{"review_id": "yDNztpwmG6", "reviewer": "Reviewer_4jCV", "summary": "The paper introduces VisualAgentBench (VAB), a benchmark designed to evaluate and train LMMs as visual agents in diverse, realistic scenarios, including embodied, GUI, and visual design tasks. VAB provides a unified, standardized framework for assessing LMMs across multiple domains, synthesizes high-quality multimodal data using a mix of programmatic solvers, LMM bootstrapping, and human demonstrations, and benchmarks 18 LMMs, uncovering both strengths and limitations in real-world task performance. Key insights include challenges in visual grounding, planning, and error recovery, offering a valuable testbed to push LMMs toward more adaptable and practical visual agents.", "questions": "1. Could you clarify the role of bounding boxes and object tags in Figure 7? Does this mean that objects and tags must be visible in the input images so that the simulator can recognize and interact with these objects by their tag names? In Section 5.1, the authors discuss the use of object labels in embodied environments. How exactly does the agent operate when no object label or tag is provided?\n\n2. To ensure ease of use, what practice does VAB provide? unified API access or modular code structure across different task environments? More details on engineering side for easy usage could be beneficial.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Comprehensiveness: The main strength of this paper is its comprehensiveness in benchmarking Large Multimodal Models (LMMs) as visual agents. The authors introduce VisualAgentBench (VAB), a benchmark that covers a wide range of real-world application scenarios by including five major task categories: embodied agents, GUI-based agents, web agents, gaming agents, and visual design agents. This breadth makes VAB a thorough evaluation tool, enabling a more holistic assessment of LMMs' capabilities across different domains rather than focusing on a single application area.\n\n- Extensive Experiments: The paper demonstrates substantial experimental rigor by benchmarking 18 different LMMs, encompassing both proprietary and open-source models. This extensive testing provides a solid foundation for the insights presented, which shed light on various LMM challenges, such as visual grounding and error recovery. These experiments allow for more reliable comparisons between models, offering valuable insights into how different LMMs perform in complex, interactive tasks. The conclusion on ReAct framework is also interesting.\n\n- Insightful Analysis: Through the VAB benchmark, the authors provide some useful observations on the current state of LMMs as visual agents. They highlight specific limitations in visual grounding, action planning, and error handling across various environments, which helps to pinpoint areas for future improvement in LMM design. While these insights are not groundbreaking, they add value by identifying practical challenges that developers and researchers may encounter when deploying LMMs in real-world applications.", "weaknesses": "- Insufficient Explanation for VL Model Performance: Some vision-language models perform poorly without adequate explanation. For instance, the paper doesn’t explore why certain models achieved low scores, leaving questions about the benchmark’s application across models.\n- Unclear Role of Visual Information in Certain Tasks: The paper lacks clarity on how specific tasks, such as those in Minecraft, leverage visual information effectively and whether VLM is genuinely necessary for all actions. For instance, Minecraft actions like \"Teleport\" don't inherently require visual information since they can execute without reference to the visual state, raising doubts about the added value of VL models in such contexts. Clarifying how the benchmark ensures each action necessitates visual input, as opposed to pure language model decision-making, would help demonstrate the benchmark’s relevance and justify the use of VL models over text-only approaches in specific environments.\n- Ambiguities in Figure Interpretation and Process Flow: Figures like Figure 2 could benefit from clearer annotations or explanations. The figure includes multiple input and output connections but lacks a clear process flow or indication of sequential dependencies, making it challenging to follow the intended agent behavior across rounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gytgcvlu6p", "reviewer": "Reviewer_cYwD", "summary": "The authors propose to apply large multimodal models to visual agent tasks, which is mored grounded than existing VQA tasks.\nThe authors collected 5 datasets including 1 for robotics in simulation, 1 for game playing, 2 for GUI manipulation and 1 for web page design.\nThe authors design a function calling-based action space for each task.\nThe agent tasks are created by first generating a bunch of templates with placeholders and then instantiating these templates.\nAction trajectories for each task are collected by 1) human-written scripts(for web GUI tasks) 2) prompting existing LMMs like GPT-4o 3) human demonstration.\nThe authors collected 746 test cases and 4482 training trajectories across 5 tasks and benchmarked 18 proprietary and open-source LMMs.", "questions": "1. Does the physical dimension of robot arm given in Gibson? As shown in Figure 1 Round 3, I don't think the grasp banana is feasible given its lying on the far end of the countertop", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed benchmark provides a unified “function-call” action space as well as diverse task domains for benchmarking the agent ability of LMMs\n2. Experiments and ablation are solid. A wide range of both commercial and open-source LMMs are evaluated on the proposed benchmark. Ablations of input image prompting(labels, SoM), reflection(injecting error) and planner(ReAct w/ & w/o Thought) are conducted.", "weaknesses": "1. No clear advantage over existing benchmarks. There are plenty of existing benchmarks for both high-level planning for robot manipulation in simulation like RoboVQA as well as for GUI agent tasks like Webshop-V and VisualWebArena. The proposed visual agent benchmark is surely more grounded than VQA benchmarks like MMMU, but I don’t see what’s the real contribution is if compared with domain-specific benchmarks. \n2. Low quality training trajectories. Take the GUI agent for instance, the proposed VAB-WebArene-Lite uses code script-based trajectory collection, which is well-known for its limited diversity compared with real-world human web browsing action trajectories. \n3. The function calling action space biases toward LMMs with a strong coding ability so that some LMMs unfairly got low scores(like Qwen-VL-Max and Gemini-1.0-Pro, which both do a good job for pure VQA benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFc5KZXnxf", "reviewer": "Reviewer_oNsV", "summary": "This paper introduces a multimodal benchmark VisualAgentBench to evaluate LMMs as agents performing interactive tasks. The benchmark includes three scenarios based on five datasets/environments: Embodied (VAB-OmniGibson, VAB-Minecraft), Graphical User Interface (VAB-AndroidLab, VAB-WebArena-lite), and Visual Design (VAB-CSS). Besides benchmark data, the authors also provide additional task trajectories for training. All the task instances in the training and testing data are constructed by prototyping and instantiation. This paper applies a mix of three strategies to collect training trajectories according to the characteristics of different datasets. Experiment results show that the proposed benchmark is challenging for current LMMs and further SFT could improve the performance. The paper also conducts an analysis of visual grounding and planning to provide insights for the future development of LMM agents.", "questions": "1. When training open LMMs, the paper says that they only use the vision input of the latest turn. Does this mean each turn in the training trajectory is treated as an independent sample (with previous turns provided in context) instead of training as a multi-turn conversation sample?\n2. For evaluating LMM APIs, what do you mean by \"Vision Input image won’t be kept in the dialog\"? Wouldn't the images that appear in the previous turns in one conversation automatically be kept?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed benchmark is a good complement to current multimodal and agent benchmarks to evaluate LMMs in challenging interactive scenarios.\n2. The proposed benchmark has standardized environments with good consistency and reproducibility.\n3. The paper also provides training trajectories for SFT.\n4. The experiments are comprehensive, revealing problems of current models, and the analysis provides good insights.", "weaknesses": "1. The number of training data is still very limited. The paper does not show whether it is possible to scale up training data in the proposed environments in an efficient way.\n2. There is no analysis and experiments to verify whether the proposed environments could be effectively used for RL training.\n3. It would be helpful to train some non-LLM specialist models in each environment using RL/IL and report their performance as a reference.\n4. After fine-tuning LMMs with the collected training data, authors should also evaluate their general multimodal abilities on other multimodal benchmarks. Also, the authors should explore whether it is possible to maintain the original abilities while improving the performance on the proposed benchmark after SFT.\n5. The authors should provide some trajectories of models tested for better illustration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5mnt6ceyv", "reviewer": "Reviewer_CJyi", "summary": "The authors propose a benchmark for evaluating LMMs across a wide range of task: including proposing (1) standardized interfaces, prompting, and data formats, (2) a strategy for creating valid test sets, (3) multitask multi-environment trajectory train sets, (4) benchmarking of 18 open-sourced and closed-sourced LMMs, (4) analyses of LMMs' abilities for grounding and planning.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I appreciate the diversity of tasks explored, from embodied agents to visual design. I believe that general-purpose LMMs that can perform well on a wide range of tasks are essential, and such benchmarks are necessary. Interaction through text/vision-based environment feedback is an important aspect of the paper. I also appreciate the scale of evaluation in the paper, and the finetuning of open LMMs.", "weaknesses": "1. Can you elaborate on how the task description, action spaces, few-shot demonstrations, and notices for each environment are formatted as prompts? How are visual outputs of the environment passed in as text in the interaction rounds for embodied and GUI agents? Could you elaborate on how error recovering works in planning? Interested in seeing experiments on error recovery behavior across all models, or in general, some evaluation on the interaction rounds instead of final success rate only (e.g., average number of steps to recover from an error). \n2. I’m also interested in seeing more detailed analyses on the tasks that these models fail on. For example, which “prototypes” lead to failure? Does each model fail in a way that is consistent with one another (e.g., is there correlation between the accuracy on each prototype/subtask?) Does finetuning an open LMM help on the grounding aspect more, or the planning aspect more?\n3. On a high-level, it would be great to have a quantitative metric established in this benchmark that systematically measures performance on grounding vs reasoning, instead of as an analysis only. Also, related to the first point, quantitative evaluation on interaction behavior (perhaps some proxy metric on progress to the goal).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)