AgentStudio: A Toolkit for Building General Virtual Agents
Authors: Longtao Zheng
Zhiyuan Huang
Zhenghai Xue
Xinrun Wang
Bo An
Shuicheng YAN
Venue: ICLR 2025 Poster (2025)
Keywords: Environment
Benchmark
Agent
Digital Automation
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We introduce AgentStudio, a trinity of environments, tools, and benchmarks for general virtual agents.

Abstract: General virtual agents need to handle multimodal observations, master complex action spaces, and self-improve in dynamic, open-domain environments. However, existing environments are often domain-specific and require complex setups, which limits agent development and evaluation in real-world settings. As a result, current evaluations lack in-depth analyses that decompose fundamental agent capabilities. We introduce AgentStudio, a trinity of environments, tools, and benchmarks to address these issues. AgentStudio provides a lightweight, interactive environment with highly generic observation and action spaces, e.g., video observations and GUI/API actions. It integrates tools for creating online benchmark tasks, annotating GUI elements, and labeling actions in videos. Based on our environment and tools, we curate an online task suite that benchmarks both GUI interactions and function calling with efficient auto-evaluation. We also reorganize existing datasets and collect new ones using our tools to establish three datasets: GroundUI, IDMBench, and CriticBench. These datasets evaluate fundamental agent abilities, including GUI grounding, learning from videos, and success detection, pointing to the desiderata for robust, general, and open-ended virtual agents.

Published as a conference paper at ICLR 2025
AGENTSTUDIO: A TOOLKIT FOR BUILDING GENERAL VIRTUAL AGENTS
Longtao Zheng! Zhiyuan Huang" Bo An Shuicheng Yan !Nayyang Technological University, Singapore Skywork AI "National University of Singapore Siinaporre Management University
Zhenghai Xue
Xinrun Wang
EETH Zurich
github. ii/iii/aaaa hhtp//gent-stttee studio
ABSTRACT
General virtual agents need to handle multimodal observations, master complex action spaces, and self-improve in dynamic, open-domain environments. However, existing environments are often domain-specific and require complex setups, which limits agent development and evaluation in real-world settings. As a result, current evaluations lack in-depth analyses that decompose fundamental agent capabilities. We introduce AgentStudio, a trinity of environments, tools, and benchmarks to address these issues. AgentStudio provides a lightweight, interactive environment with highly generie observation and action spaces, e.g., video observations and GUI/API actions. It integrates tools for creating online benchmark tasks, annotating GUI elements, and labeling actions in videos. Based on our enviromment and tools, curate online task suite that benchmarks both GUI interactions and function calling with efficient nuto-evaluation. We also reorganize existing datasets and collect new ones using our tools establish three datasets: GroundU1, IDMBench, and CriticBench. These datasets evaluate fundamental agent abilities, including GUI grounding, learning from videos, and success detection, pointing to the desiderata for robust, general, and open- virtual agents.
INTRODUCTION
Building general virtual agents that utilize every software enhance human productivity is long- standing research challenge in artificial intelligence (Shi et al., 2017). Such agents perceive computer interfaces (c.g.. GUI, keyboard, and mouse) or function calling (e.g.. APIs) in response to natural language instructions. Fueled by advancements in large language models (LLMs) (Brown al., 2020; Chowdhery et al., 2022; OpenAl, 2023), there has been impressive progress in virtual agents (Zheng et al., 2023; Zhang et al., 2024; Tan et al., 2024). However, deploying these agents real- settings requires satisfying three key desiderata: (1) achieving general grounding by interacting with both GUIs and APIs and handling complex observations like videos, (2) possessing extensive digital world knowledge complete tasks across diverse applications, and (3) self- celf-improving and generalizing to unseen situations through trial and error in open-ended environments. For example, an agent exporting charts from spreadsheet and adding them to video needs interact precisely with GUI elements in both the spreadsheet and the video editor. Using API actions can speed up the process, and video observations can help monitor and make adjustments. The agent needs knowledge of these applications, e.g., understanding their workflows, to effectively perform the task. If the video editor's s layout is out of distribution, the agent should learn how to use it by interacting with the environment.
states as observations (c.g., screenshots) and take actions through
Therefore, the environments of virtual agents must (1) model various observation and action spaces to support all possible tasks, (2) provide online interactions for learning through trial and error, and (3) systematically benchmark agents' abilities to satisfy the three desiderata. However, many existing virtual agent benchmarks are static datasets (Deng et al., 2023; Rawles et al., 2023; Kapoor et al., 2024) or only consider domain-specifie actions (e.g., web operations) (Zhou et al., 2023; Koh et al., 2024), which do not encompass the full range of tasks humans can perform. Recent environments
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Desiderata for General Virtual Agents
AgentStudio Environments
AgentStudio Tools
AgentStudio Benchmarks
General grounding
Video observations GUI & API actions
GUI element annotation
GroundUI
Online benchmark tasks
Digital world knowledge
Video action annotation
Real-world devices
IDMBench
Open-ended
Interactive environments
Task creation & validation
CriticBench
learning
Figure 1: AgentStudio focuses on three desiderata for general virtual agents. For general grounding in any software, it provides universal observation and action spaces, a GUI annotation tool, and the GroundUI dataset for enhancing and evaluating GUI grounding. To leverage digital knowledge, our tools enable recording human trajectories and annotating videos on real-world devices, which can be used to create datasets like IDMBench to evaluate agents' abilities to learn from videos. To support open-ended learning, AgentStudio offers interactive environments with language feedback, tools for task creation, and CriticBench to benchmark the ability for success detection. 'hhe overall agent performance is measured through an online task suite developed using our task creation tool.
have advanced toward more realistic settings (Xie al., 2024; Rawles et al., 2024), but they still do not include video observations or allow taking actions through both GUIs and APIs. Moreover, they typically use success rates as the only metric, which may result in overoptimizing agent frameworks for specific tasks rather than real-world performance. Additionally, setting up these environments or customizing tasks within them is often non-trivial. As a result, current environments and benchmarks struggle to reliably evaluate agent performance in real- world scenarios. They also difficult to expand to cover a broader range of tasks and provide detailed analyses of fundamental agent abilities.
AgentStudio provides interactive, realistic, and lightweight environment with generic observation and action spaces, enabling agents to interact with arbitrary software (Figure 1). The observation space incorporates multiple modalities, ranging from screen recordings (videos) and screenshots (images) to code execution results (text), Agents can act through human-computer interfaces (e.g.. keyboard-mout operations) control third-party applications, and perform function calling to interact with APIs. These features expand the task space to massively open-domain and real-world tasks typically performed by humans. The interactive nature of online environments allows agents to learn through trial and error, which is enhanced by the language feedback on failure reasons provided by our environment. We also integrate various tools for customizing and validating benchmark tasks, annotating GUT elements with bounding boxes, and labeling actions for actionless videos. Our implementation is designed to be lightweight, ensuring ease of installation, usage, and reproducibility.
Using AgentStudio environment and tools, we introduce online task-completion benchmark and three datasets to evaluate fundamental agent abilities in real-world settings. The benchmark suite consists of 205 real-world tasks across various applications such as VS Code, Google Workspace, and Office suites. Solving these tasks requires agents to process complex observations, interact with GUTs, and perform function calling. We curate the benchmark through a three-stage process: (1) creating basic tasks and their auto-evaluators for single-application scenarios, (2) compositionally expanding tasks to cross-application workflows, and (3) manually validating task configurations and auto-evaluators. To improve evaluation efficiency, we implement over 100 functions for automatic environment resetting and evaluation. The three datasets, GroundUI, IDMBench, and CriticBench, focus on UI grounding, labeling actions in videos, and success detection, respectively. GroundUI includes 18K single-step incorrect instructions recaptioned. Agents must generate precise UI element coordinates to complete these instructions. The other two datasets are based on multi-step agent trajectories. In IDMBench, models need to label actions in a video clip, reflecting their ability to leverage knowledge from vidco demonstrations. CriticBench evaluates models' ability to determine a task has been successfully completed based the trajectory, which is crucial for efficient evaluation and open-ended learning. Overall, these benchmarks are designed to be accessible and extensible, demonstrating the potential of AgentStudio for agent evaluation and in-depth analysis in real-world scenarios.
pairs across various devices, with ambiguous or
Experimental results indicate that even state-of oftth-r languae models (VLMs) like GPT-4o experience significant performance declines in GUT and compositional tasks. For tasks that could be
2

---

Published as a conference paper at ICLR 2025
completed using APIs, providing screenshot observations can lead poorer performance compared to text-only observations, as models might be misled into using GUI actions instead of APIs. Notably, most existing models struggle with professional applications such image editing software, whereas humans can solve 72.2% of those tasks. Evaluation results from the three datasets reveal that current ceneral-purpose VL.MS struggle accurately predict the exact coordinates of GUT elements in screenshots. This inaccuracy accumulates over time, significantly reducing the success rate of multi- step tasks. We also find that as screen resolution increases, the accuracy decreases, leading to higher accuracy in mobile environments compared to web and desktop scenarios. Additionally, our findings suggest that current models are not yet effective labeling actions in videos and struggle to reliably evaluate whether trajectory is successful. Therefore, future work could enhance these abilities and refine agent frameworks to better utilize them.
AGENTSTUDIO ENVIRONMENT
Each computer task can be modeled as a partially observable Markov decision process (POMDP) A, O, T, R, F2,, ,2). with state space S, action space A, observation space O transition function T, reward function R. feedback space F, and instruction space 4. AgentStudio offers an interactive and realistic environment with comprehensive observation space O and action space A. The transition function T reflects the real-time dynamics of computer enviromments. The environment supports both scalar rewards from R and language feedback from F to facilitate open-ended LLM agents to self-correct (Shinn et al., 2023; Wang et al., 2023). Additionally, it comes with tools for creating benchmark tasks (S, A. O, and 24) validating functional euto-evaluators (R and F), annotating GUI elements, and labeling actions in videos. These features are designed according to the three desiderata for general virtual agents (Figure 1), aiming to promote research on agents that can handle real-time video observations, utilize both APIs and GUIs for improved efficiency, learn from video demonstrations, and adapt through environment interactions.
Universal observation and action spaces. AgentStudio features comprehensive observation space O and action space A that mirror the complexity of real-world human-computer interactions. Specifically, the observation space is defined as union of text, image, and video modalities, ie., O OTxx U 1magge U Video- This allows the agent to receive observations that include screen recording videos, screenshots, and code execution results. Agents can also leverage tools to obtain additional observations that are not directly visible to human users, such as HTML code and accessibility trees. The video observation facilitates research on agents' ability to process complex. multimodal observations and solve tasks requiring real-time vidco understanding. The action space is similarly expansive, defined as union of GUI interactions and API calls, i.e., = AGuI U AAPI- This encompasses both high-level function calling (AAPI) and low-level operations like keyboard and mouse controls (Agur). Agents can execute code via a Python kernel to call APIs, import tools, and manipulate human-computer interfaces. This universal action space enables agents to control arbitrary software: they can call APIs for applications with accessible internals (e.g.. terminal commands and Google Workspace) to enhance efficiency and accuracy, or automate operations via
interfaces when API access is unavailable (c.g., third-party applications). In contrast, most environments use domain-specifie observation and action spaces, such screenshot (Ormage) or HTML snippet (Orea) for each web operation (AArI), as shown in Figure 2.
Interactive, realistic, and lightweight environments with language feedback. AgentStudio adopts fully online interactive environments on real-world devices, where both screen recording and action execution occur in real time. The transition function T : S A S x O reflects these real-time changes, capturing the complexity of real-world scenarios. For example, when the agent clicks an icon, the observation may not display an immediate change, despite the application being in the process of launching. In contrast, existing environments sidestep this challenge by adding a delay when getting the observation after each action, which can miss dynamic interactions or delays in actual scenarios. Our interactive environment allows agents to explore, learn from environment interactions, and accumulate new skills over time. To enhance agent learning, the environment provides both a binary reward signal R S + {0, 1} and language feedback F S + {failure reason, success) ) that informs agents why a task may have failed. This promotes research on self-correction and open-ended learning for LLM agents. Online environments offer more effective benchmarks than static datasets, which can be exploited, rely on domain-speciff action spaces, or cannot accept multiple valid solutions. The real-world feature also enables agent evaluation and data collection any task that
3

---

Published as a conference paper at ICLR 2025
Web Browser Environments
AgentStudio Environments
Web-Based Observations HTML/DOM (Text)
Web-Based Actions
Universal Observations
Universal Actions
Element Operations
Screen Recording (Video)
Keyboard
Accessibility Tree (Text)
Tab Operations
Screenshots (Image)
Mouse
Screenshots (Image)
Operations
Execution Outputs (Text)
Figure 2: Comparison of observation and action spaces between web browser environments and AgentStudio. AgentStudio provides the most generic observation and action spaces, which signifi- cantly expands the task space, allowing for developing and evaluating agents in real-world settings.
human can perform. AgentStudio can operate in both local and remote modes. remote mode, launches Virtual Network Computing (VNC) remote desktop, supporting various VNC-compatible operating systems (e.g.. Windows, Linux, macOS) and devices (e.g., Docker containers, virtual machines, physical machines), making it lightweight and flexible set up and use.
AGENTSTUDIO TOOLS
To facilitate the development and evaluation of agents within the AgentStudio environment, we provide three tools for creating in-the-wild benchmark tasks and developing high-quality datasets for agent training and evaluation. These tools, combined with the realistic environment of AgentStudio, contribute the generation of rich, structured data for training and evaluating agents.
Benehmark task creation and validation. Users can create tasks by specifying S, A, O, and 44 and equip these tasks with functional auto-evaluators representing R and F. These tools enable manual validation of task performance and evaluator correctness, i.e., ensuring the accuracy of R and F, as well as the measurement of human success rates and operation times.
Step-level GUI element annotation. This pipeline allows users to capture screenshots o E O1mage= annotate UI elements by drawing bounding boxes, and add step-level instructions E U. This facilitates the creation of diverse datasets for training and evaluating single-step UI grounding models, which map instructions and screenshots precise mouse click coordinates.
Trajectory-levee video-action recording and refinement. This is a two-step process including action recording and video refining. During action recording, users can record the rajectory-leeee instruction E (4, the screen recording 01:7 E V and all keyboard and mouse operations a, AcuI, where is timestamp within time horizon of length T. In the video refining phase, an editor interface allows users to replay and trim the recorded video 01:7 and process the action sequence a1.7-1- Specifically, users can remove noisy mouse movements and aggregate atomic actions (e.g., individual key presses) into higher-level actions (e.g., typing string or executing shortcut). This crucial for training IDMs that can infer action sequences from videos without explicit action labels, enhancing the agent's 's ability of learning from video demonstrations.
AGENTSTUDIO OVERALL BENCHMARK TASKS
To illustrate how AgentStudio facilitates agent evaluation in complex and real-world scenarios, we introduce benchmark suite consisting of 205 tasks. These tasks span API usages such terminal and Gmail and GUI software like VS Code in the AgentStudio environment. Solving these tasks requires various fundamental agent abilities, including general grounding through complex action space (AAPI U Agui) and compositional generalization over diverse instruction space 4. The tasks designed to be easy evaluate, circumventing common issues in current agent evaluations such as unrealistic environments, complex setups, and vulnerability to hacks. Despite their conceptual simplicity, these tasks present significant challenges state-of-the-art VLMss.
4

---

Published as a conference paper at ICLR 2025
4.1 BENCHMARK CONSTRUCTION
Each task in our benchmark can be formulated tuple (u, language task instruction, 8p E the task-specific initial state, R. automatically evaluates the final state and returns a binary reward, and F provides the language feedback if the task fails. For example, for the instruction u "Create one-hour event Team Meeting in Google Calendar at 5pm today", So is obtained by removing any existing event with the same name, and R. evaluates success by checking via API calls whether the event was correctly created after the agent's actions. If the task fails, FF provides feedback which success conditions instructions from the Internet and previous work (Guo et al., 2023; Li et al., 2024; Xie et al., 2024) and construct the benchmark suite compositionally through the following three stages.
80, 80, F), where U is the natural
not satisfied for the final state. We source
Stage I: Crafting the minimal evaluators and action spaces. We begin by organizing basic instructions into 50 single-API tasks and 95 single-GUI tasks, each focus- ing API or GUI operations within a single applica- tion. We implement over 100 auto-evaluators for R and F. Single-APi tasks consist of tasks that can be accom- plished through direct function calling, with the observa- tion space = OTea- Single-GU tasks involve common daily applications where agents additionally provided with screenshots, i.c., = Oext Omage. In both cases, provide the agent with minimal while generic action space A = U.Gu-- including basic keyboard, mouse, and terminal operations. We leave higher-level actions like skill libraries to future research on agent frameworks.
vs
3.4%
5.4%
9,3%
6.%
05 9.3%
Docs 7.9%
Single-AP
Single-GUI
Soread: 7.3%
Compositiona
Slides 7.3%
Ed
5.4%
Stage I: Compositional task expansion. By compo- sitionally extending the instructions and auto-evaluators from Stage I, we generate 60 more challenging tasks benchmark compositional generalization. This category covers cross-application tasks requiring interactions with multiple GUTs, multiple APIs, or both. For example, agents may need navigate the operating system and multiple applications, like VS Code and document, to complete a task.
Figure 3: Task composition.
Stage III: Manual validation and difficulty measure. To verify the correctness of the auto- evaluators (R and F), we use AgentStudio to collect human trajectories, evaluator outcomes, and task completion times. Therefore, we can identify and correct potential issues and mitigate false evaluations. The task completion times serve empirical measures of task difficulty.
4.2 EVALUATION
Single-API
Single-GUI
Model
Compositional
Total
OS
Google
OS
Office
VS Code
GIMP
Claude Sonnet
94.7
74.2
94.7
5.0
0.0
25,0
36.6
GPT-4o
100.0
54.8
94.7
4.4
15.0
0.0
23.3
35.6
Gemini 1.5 Pro
68.4
16.1
63.2
0.0
5.0
0.0
5.0
16.6
Gemini 1.5 Flash
52.6
12.9
47.4
0.0
0.0
0.0
6.7
13.2
Humans
73.7
83.9
73.7
31.1
85.0
54.6
80.0
72.2
Table I: Success rates (%) AgentStudio overall benchmark tasks. Existing models can perform well in simple command-line tasks but struggle with complex API calls, GUI operations, and cross- application tasks that require both GUI operations and API calls.
Table 1 compares model performances on our benchmark tasks. While state-of-the-an models like GPT-4o perform well on simpler operating system tasks (Single-API OS) that only involve OTest- they struggle to reliably solve tasks that require complex API calls (e.g., Google Workspace). To evaluate the impact of the observation space, we apply identical instructions and action spaces in Single-API OS and Single-GUI OS, but Single-GUT OS tasks include additional screenshot observations, ie.., Otx U Image- Even though the tasks remain identical, the performance on the Single-GUI OS task declines for some models, they are misled into using keyboard/mouse operations instead of
5

---

Published as a conference paper at ICLR 2025
efficient API calls. Specifically, GPT-4o and Claude 3.5 Sonnet maintain comparable success rates across both setups, as they consistently favor API calls even when presented with screenshots. In contrast, Gemini models exhibit performance drop, they prone to relying GUI interactions. This emphasizes the limitations of current models and the need for agents to determine when to use APIs instead of GUT actions. Also, most existing models struggle with professional software applications, such as Office suites, code editor (VS Code), and image editor (GIMP). Furthermore, compositional tasks that span multiple applications or require a combination of GUI operations and API calls remain challenging for current models, despite being moderately difficult for humans. We manually review the trajectories and categorize five failure modes: (1) False Finish: The model incorrectly marked the task as complete but failed the auto-evaluation. (2) Parse Error: The model
generated unparseable actions. (3) Step Limit: The model exceeded the allowed steps. (4) Time Limit: The model surpassed the time limit. (5) Repetition Limit: The model repeated the same action many times. Table 2 presents the distribution of failure modes for each model, and Table 3 shows the proportion of tasks actively marked completed by each model. Gemini 1.5 Flash and Claude 3.5 Sonnet exhibit relatively low trigger rates for Step. Time, and Repetition Limits, indicating their capability actively mark termination. In contrast, GPT-4o's active finish rate is significantly lower than the other models, suggesting that it tends to continue task execution until a certain limit or error is triggered. For most GUI tasks, existing models struggle accurately click on GUI elements, as further demonstrated by our fine the-grained evaluation in Section 5, When executing actions through code, existing models fail to handle dynamic real-time environments with appropriate sleep delays, and they struggle to determine when new observations are needed, leading to difficulties in completing complex tasks. Further results and detailed analysis are provided Appendix F.
False Finish
Parse Error
Step Limit
Time Limit
Repetition Limit
Active
Model
Model
Finish
Gemini 1.5 Pro
70.8
2.9
11.7
9.4
5.3
Gemini 1.5 Pro
74.1
Gemini 1.5 Flash
62.9
35.4
0.0
0.0
1.7
Gemini 1.5 Flash
67.8
Claude 3.5 Sonnet
96.2
2.3
0.8
0.0
Claude 3.5 Sonnet
97.6
GPT-4o
31.8
48.5
4.5
8.3
6.8
GPT-4o
28.8
Table 2: Distribution of failure modes for each model (%).
Table 3: Active finish rate (%).
5 AGENTSTUDIC
EVALUATION
To gain deeper insights into agent capabilities beyond the overall performance measured by online benchmark tasks, we develop three datasets using AgentStudio: GroundUI, IDMBench, and Crit- icBench. These datasets target the desiderata for general virtual agents, i.e., general UI grounding, learning from videos, and success detection. Unlike benchmark tasks where success rates are often zero, our datasets provide dense signals to guide research and mitigate overoptimizing specific tasks.
5.1 GROUNDUI: EVALUATING UI GROUNDING AND LOCALIZATION
Using AgentStudio, we confirm that accurately grounding UI elements with precise coordinates still a main chal- lenge for virtual agents (Li et al., 2020). As illustrated in Figure 4, we collect sample task with over 200 screenshots from various desktop operating systems using AgentStudio. In this task, models are required to write code snippets that include click types (such left click or double click) and the corresponding click locations. We find that current VLMs consistently perform poorly in matching locations accurately. Previous studies have also confirmed that while current models can generate high-level plans in textual form, they struggle to ground these plans into the correct UI elements (Zheng et al., 2024). This grounding capability is necessary because not all elements existing benchmarks have varying formats and do not directly evaluate this ability across diverse applications and platforms. More importantly, many instructions in these datasets either incorrect
Location Match
Click Type Match
100
80
60
20
Claude-3 Sonnett
Gomini-Pro
GPT-4V
Owan-VL
Figure 4:
Accuracy of location/type match in the sample task.
readily accessible in text form (Koh et al., 2024). However,

---

Published as a conference paper at ICLR 2025
ambiguous (see examples in Appendix C.2), and some instructions task-level rather than step- level. These issues render existing datasets less reliable for benchmarking UI grounding. To address these issues, we systematically reorganize existing datasets and incorporated AgenSStudio-colletted data to create a more comprehensive and reliable evaluation.
Dataset curation. We define UI grounding task tuple (u, o, bbox). where u E U represents a single- step instruction, bbox = (I1, MI, 22, y2) denotes the bounding box of the targeted UI element. Given u and o, the model is expected to output an action = (a, y), the normal- ized coordinates click within the range (0, 1). The action is considered correct if the predicted point lies within the bounding box. For example, if provided with screenshot of music player interface and the instruc- tion "Click the Play button", the predicted action should correspond to the bounding box of the Play button. We collect screenshots from the test sets of existing grounding datasets across web, desktop, and mobile platforms, including MoTIF (Burns et al., 2022), ScreenSpot (Cheng et al., 2024), OmniACT (Kapoor et al., 2024), along with additional screenshots collected in our sample task. After carefully examining the datasets, we exclude data lacking anno- tated bounding boxes. To recaption problematic instructions, overlay the ground truth actions onto each screenshot and use GPTT-4o to generate detailed descriptions for the plotted GUI elements, which proved to reasonably reliable. This process results in GroundUI- an 18K UI grounding dataset for benchmarking. For efficient evaluation, we conduct experiments on subset, GroundUl-1K, which consists 400, 300, and 300 samples web, desktop, and mobile devices, respectively.
MindgNee (ast
MindtWWbb ltest 3.8%
7.7%
8.0%
Otmage is the screenshot, and
MindaWee 19.1%
ScreenSoort
ORNJACT 24.3%
Desktop
SorpenBoot
Mobile
3.3%
AgoniStudio 2.4%
MoTiF 26.55 55.5%
Figure
Source of GroundUI-IK 1K data.
Model
Web
Desktop
Mobile
Total
Raw Instructione
SeeClick
64.3
44.3
73.7
61.1
Recaptioned
CogAgent
25.3
15.7
35.7
25.5
40-
CogVL.M2-Llamaa-eee 19B
2.7
3.4
MiniCPM-Llamats 2.5
0.0
0.3
2.7
0.9
Owen-VL-Chan
0.0
0.0
0.0
0.0
20
Gemini 1.5 Pro
31.2
24.3
51.3
35.2
Claude 3.5 Sonnet
13.0
17.3
GPT-4o
7.5
8.3
26.3
13.4
SeeCiick
GPT-40
CogVLMe
Gemini 1.5 Flash
0.5
4.3
26.3
9.4
Figure 6: Accuracy w/ and w/o instruction recaptioning.
Table 4: Single-step GroundUI-1K accuracy of open (top) and proprietary (bottom) VLMs.
Results. As presented in Table SeeClick, specialized UI grounding model, consistently achieves the highest overall grounding accuracy across different platforms. However, due to multi-step error accumulation, its performance still fails to meet the robustness required for practical deployment. SeeClick is trained with large-scale grounding data, highlighting the importance of scaling up UI grounding datasets. Among proprietary models, Gemini 1.5 Pro outperforms others like GPT-4o and Claude 3.5 Sonnet. However, these general-purposs models still lag behind specialized models such as SeeClick. The accuracy on mobile platforms is generally higher than that on web and desktop platforms, which can be attributed to the comparatively lower screen resolution of mobile screenshots. We also conduct ablation studies to assess the impact of instruction recaptioning on three models. As illustrated in Figure 6 in Appendix A, the differences in performance suggest that the recaptioned dataset provides more reliable evaluation of UI grounding capabilities. Additional results, data samples, and prompts are available in Appendix
5.2 IDMBENCH: EVALUATING LABELING ACTIONS FROM VIDEOS
Internet-scale data have significantly advanced language and vision models. However, virtual agents have not similarly benefited due to the lack of large-scale action datasets, because annotating computer operations is both time-consuming and expensive. While numerous tutorial videos exist.
7

---

Published as a conference paper at ICLR 2025
on the Internet, they lack the action labels necessary for training. Leveraging IDMs (Baker et al., 2022), which estimate PINx(Q11T11(1111) allows us to label pseudo actions in videos, unlocking the knowledge embedded in Internet-scale videos. Unfortunately, no benchmark currently exists to evaluate this ability. Therefore, we evaluate VLMS as IDMs to promote research on enabling virtual agents to learn from videos and generalize across diverse scenarios.
Dataset curation. Our benchmark introduces two settings: IDM-Single and IDM-Multiple. IDM- Single focuses on single-step action labeling, where the task is to identify an action o, AGu at timestep i given the observations before and after it, denoted as or and 0-+11 Although useful for benchmarking, selecting appropriate pairs of video frames in real-world scenarios is challenging because the effect of an action may not immediately appear after execution. Therefore, we propose IDM-Multiple to evaluate multi-step action labeling a more realistic scenario. This setting requires the model to predict PIDM PIDM(41:T (41:7-1 (01:r), meaning it must label all actions performed within a trajectory of T observations or for E [1, without being informed of the number of actions. We collect trajectories from Mind2Web (M2W) (Deng et al., 2023), Android in the Wild (AITW) (Rawles et al., 2023), VisualWebAremi (VWA) (Koh et al., 2024), and AgentStudio (AS). Notably, excluded ggentStudio-collette trajectories that involved code in actions due the difficulty of evaluating their correctness, as well as trajectories with fewer than two steps. As a result, we created dataset of 345 trajectories, with 100 from existing environments and 45 from AgentStudio. The IDM-Single dataset is derived from these trajectories by extracting individual transitions (o, a, o'). For simplicity, we formulate each task as a multi-choice question and incorporate the action space A into the prompt as the answer choices. For example, in Mind2Web, the action space includes actions like clicking, typing, and scrolling. More details can be found in Appendix D.
IDM-Single
IDM-Multiple
Model
M2W
AITW
VWA
AS
Total
M2W
AITW
VWA
Total
Claude 3.5 Sonnet
73.0
56.0
50.0
72.0
61.4
18,0
8.0
7.0
22.2
12.5
GPT-4o
70.0
56.0
45.0
78.0
60.0
13.0
8.0
2.0
20.0
9.3
Gemini 1.5 Pro
62.0
51.0
46.0
48.0
52.3
0.0
0.0
1.0
2.2
0.6
Gemini 1.5 Flash
65.0
34.0
31.0
60.0
45.7
0.0
0.0
0.0
0.0
0.0
Owen-VI--Chee
37.0
20.0
5.0
20.0
20.6
0.0
0.0
0.0
0.0
0.0
Table Accuracy on IDMBench. Model performance drops significantly in IDM-Multiple.
Results. Table 5 presents the accuracy on IDMBench for various VLMs across four environments
with different action spaces. Among the tested models, Claude 3.5 Sonnet and GPT-4o consistently outperform the others in both settings. Tasks involving more complex action spaces present greater challenges. For example, the action space of M2W is less complex than that of AITW, leading to higher accuracy. In IDM-Multiple, accuracy is calculated based on the exact match of the action sequences, which explains the notable decline in model performance, as any deviation from the expected sequence can lead failure. Since models in IDM-Multiple can generate action sequences of variable lengths, Table 9 in Appendix A further evaluates the effort required to convert the predicted action sequence to the ground truth, using edit distance as a metric. Notably, two Gemini models sometimes fail to stop correctly and generate extremely long action sequences in IDM-Muutiple. To mitigate this, we used a different prompt for IMM-Multiple experiments with these two models, instructing them to generate fewer than ten actions. However, Gemini 1.5 Flash still produced long sequences, resulting in the highest edit distance in IDM- Multipple Overall, current models are still unable accurately label actions from videos and are therefore not yet capable of learning from real-world tutorial videos. The discrepancy between performance in IDM-Single and IDM-Multiple also indicates that more effort is needed to improve action labeling in realistic settings.
5.3 CRITICBENCH: EVALUATING SUCCESS DETECTION
Although we have implemented auto-evaluators to assess functional correctness in our online bench- mark, evaluating the success all tasks in this manner is neither scalable nor feasible due to the massive diversity and open-ended nature of computer tasks. Therefore, general critic model would significantly reduce the burden of creating online benchmark tasks. Such a model also enables agents to self-correct and engage in open-ended learning to master software when human demonstrations are limited. We argue that training a general critic is easier than training general agents. Unfortunately, no existing benchmarks measure this success detection ability in current VL.Ms for computer tasks.
8

---

Published as a conference paper at ICLR 2025
Dataset curation. We represent each data point triplet (u, y), where U denotes the trajectory-level task instruction, T is the corresponding trajectory consisting of multiple transitions (o, a. d'), and y E {0, 1} is the binary outcome indicating the success failure of r. Given the instruction u and the trajectory т, the model required to predict Similar to IDMBench, we collect trajectories different devices from M2W (Deng al., 2023), AITW (Rawles et al., 2023), VWA (Koh et al., 2024), and AS. Since most human trajectories are successful, we balanced the dataset by generating failure cases from partial trajectories. For example, we can create failure trajectory by extracting the first few steps of a successful one. We curate subset of 350 trajectories, with 300 trajectories from existing environments and 50 from AgentStudio.
Observation-Action Pairs
Observations Only
Model
Web
Desktop
Mobile
Total
Web
Desktop
Mobile
Total
Gemini 1.5 Pro
72.5
84.2
52.5
69.7
61.1
88.1
48.1
62.9
Gemini 1.5 Flash
65.6
85.2
57.8
67.3
64.1
75.0
52.5
63.1
GPT-4o
68.0
91.5
41.7
85.7
39.4
60.8
Claude 3.5 Sonnet
68.4
92.9
36.6
65.6
64.6
87.3
47.5
64.2
Qwen-VLL VL-Chat
55.6
55.1
64.4
58.5
63.2
73.4
67.6
66,2
Table 6: F1 score on CriticBench. More metrics can be found in Table 10 in Appendix A.
Results. Table presents the scores for predicting the outcome y using current VLMSs that accept multi-image inputs. Additional metrics, including accuracy, precision, and recall, are provided in Table 10. These results indicate that existing models perform similarly on this task, with no significant discrepancy between open and proprietary models. Furthermore, unlike the benchmark results in GroundUI, where all models perform better on mobile environments, success detection on mobile environments does not appear to be easier than other platforms. We also observe that incorporating action information offers some benefits, but the effects are limited, as illustrated in the right part of Table 6. In summary, while current VLMS show promise, their performance is still far from effective for eef--imprvemment and open-ended learning. This benchmark provides a standard for measuring the success detection capabilities of current VLMs, facilitating further research in their prediction accuracy. Additional examples and prompts are provided in Appendix E.
6 RELATED WORK
Inter-
Data & Task Language
Light- Decompose weight
Environment
Domain
Obs. Space
Action Space
active
Tools
Feedback
Abilities
World of Bits
Local Web
T/VV
K/M
X
WebShop
Local Web
TM
Web Ops
X
Mind2Web
Web Dataset
TA
Web Ops
X
WehArena
Web
Weh Ops
X
VesualWebArena
Local Web
TM
Web Ops
X
AndroidEny
Mobile Emulator
TAV
Touchscreen
X
AITW
Mobile Dataset
T/
Touchscreen
X
X
X
World
Emulator
T/VV
Touchscreen
X
OmniACT
Desktop Datasct
M
K/M
OSWorld
Desktop
T/I
K/M
X
AgentStudio
Desktop
1VVV
K/M/Code
Table AgentStudio supports online interactions with generic observation (text (T), image (I), and video (V)) and action spaces (keyboard (K). mouse (M), and code) among the listed environments. offers benefits such as user-friendly GUT/video annotation and task customization, lightweight installation, and fine-nrained datasets that decompose fundamental agent abilities.
operating systems using the most
Environments for virtual agents. Table presents comparison of our environment with other existing ones. Numerous efforts have been made to develop simulators for creating virtual agents that can automate computer tasks. For example, World of Bits (Shi et al., 2017) offered minimalist web simulator, while AndroidEnv (Toyama et al., 2021) provided an emulator wrapper for Android devices. Early attempts in these environments primarily used deep reinforcement learning (Liu et al., 2018; Gur et al., 2018; 8; Jia et al., 2018; Gur et al., 2021; Humphreys et al., 2022), which struggled generalize beyond the environments. With novel capabilities like code policies (Liang et al.,

---

Published as a conference paper at ICLR 2025
2023) and tool use (Schick et al., 2024; Qin et al., 2023) of LLM-based virtual agents (Gur et al., 2023;; Kim et al., 2023; Zheng et al., 2023; Gur et al., 2023a), these simplified environments have been gradually solved. To build more realistic and complex settings, WebShop (Yao et al., 2022) introduced a web shopping environment. However, its tasks and ohservation/action spaces were browser-specifie i.e., O Oee U immae and A AAPI- Similar limitations apply to recent web environments like WebArena (Zhou et al., 2023) and VisualWebArena (Koh et al., 2024), where the action spaces are limited web operations such as element clicking and URL navigation. Meanwhile, AndroidWorke (Rawles et al., 2024) focused specifically on Android devices. OS World (Xie et al., 2024) is a recent extension of WebArena to desktop environments focusing on GUI interactions, with O = OTex U Ommage and A = Acun. In contrast, our environment supports video observations and both API and GUI actions, i.e., O = Orex U Ormuge U OVitea and A AGu U AAPI- This setting expands the task space and reveals new challenges for virtual agents, such as deciding when to use API or GUI actions to maximize efficiency and handling real-time dynamics. Additionally, provide more lightweight implementation. along with tools for benchmark task customization, open-domain GUI annotation, and video action annotation in real-world environments.
Benchmarks for virtual agents. Most environments above introduced accompanying benchmark tasks with functional evaluation as the reward function R. However, their task space is limited by the observation and action spaces of the environments. Also, most of them focused only on overall success rates, failing to promote research efforts towards fundamental agent capabilities. In contrast, our generic environment allows for any task humans can perform. While we source part of instructions from existing benchmarks, we add new tasks targeting the unique challenges posed by our environment, c.g., dealing with complex observation and action spaces. We also conduct rigorous checks on our auto- nnto--vvaluators R, fix errors in tasks within existing environments, and offer a simplified task structure and evaluation process. Additionally, we provide language feedback JF on task failures for LLM agents. In addition to benchmark tasks in online environments, there are also static datasets designed to evaluate overall agent performance. AITW (Rawles et al., 2023) is a large-scale dataset focused on Android operations. Mind2Web (Deng et al., 2023) and WebLINX (Lu et al., 2024) evaluate web agents. Some benchmarks specifically measure agent performance in interacting with APIs (Xu et al., 2023; Liu et al., 2023; Mialon al., 2023; Yao al., 2024). More recent static benchmarks have also been developed for desktop environments (Cheng et al., 2024; Niu et al., 2024; Kapoor al., 2024; Zhang et al., 2024; Wu et al., 2024). Unlike functional auto-evaluators in online environments, these datasets only accept a single successful trajectory, which limits their effectiveness in evaluating overall task completion, as done in existing benchmarks. We believe that static datasets are more appropriate for benchmarking specific agent abilities. While diverse datasets exist, a systematic approach lacking to target fundamental agent abilities beyond measuring success rates. Although several GUI grounding-specific datasets similar to our GroundUT exist (Li et al., 2020; Burns et al., 2022), a unified format can significantly facilitate benchmarking across diverse applications and devices. As for IDMBench and CriticBench, no similar benchmarks currently exist. More detailed rationale of our datasets can be found in Appendix B.
DISCUSSION
Conclusion. AgentStudio provides a comprehensive solution for general virtual agents by offering interactive real-world environments, holistic tools, online benchmark tasks, and datasets for specific agent capabilities. Our design choices follow the desiderata for such agents: general grounding to interact with any software, digital world knowledge from Internet-scale video tutorials, and open- ended learning for self-improvement and generalization to new situations. We hope this work offers practical insights and promotes the evaluation and development of next-generation virtual agents.
Limitations and future work. We acknowledge certain limitations in our work. First, despite rigorous examinations, there might still be cases where auto-evaluators make incorrect judgments due to real-world complexity, a common issue of existing real-world benchmarks. Developing a general critic performing well on our CriticBench could alleviate this issue. Second, the GroundUI datasets might still have problematic instructions due to the automatic recaptioning process using GPT-4o. Third, our datasets are designed for easy evaluation but might be vulnerable to hacks. Continually updating them can mitigate issues like data leakage. Finally. although our implementation is compatible with real- world devices, our results for online benchmark tasks are obtained within an Ubuntu Docker environment, tradeoff between reproducibility and real-world complexity.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
All resources, such as code and datasets, are publicly available our project page. The benchmark tasks and datasets are also hosted on Hugging Face.
ETHIC STATEMENT
Autonomous agents that can control real-world digital devices have inherent risks, such as deleting important files and sending spam emails. To mitigate these risks, our framework allows users to examine and confirm each action before execution. Users should take extra caution when using these agents to interact with real-world environments.
ACKNOWLEDGMENTS
This research supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISGAward No: Start-up Grant A-0010106-00-00, and Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier grant (No. MSS24C005).
NUS
REFERENCES
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. 2022.
Advances in Neural Information Processing Systems,
Tom Brown, Benjamin Mann, Nick Ryder, Melamie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset interactive vision language navigation with unknown command feasibility. In European Conference on Computer Vision (ECCV), 2022.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SceClick: Harnessing GUI grounding for advanced visual GUI agents. arXiv preprint arXiv:2401. 10935, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.00 023111 2022.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijic Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards generalist agent for the web. In Thirty-eeentth Conference on Neural Information Processing Systems, 2023.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang,
An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building open-ended embodied agents with Internet-scalc knowledge. Advances in Neural Information Processing Systems, 35:18343-1888 2022.
Yiduo Guo, Zekai Zhang, Yaobo Liang, Dongyan Zhao, and Duan Nan. PPTC benchmark: Evaluating large language models for PowerPoint task completion. arXiv preprint arXiv:2311.00700 2023.
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani- Tur. Learning to navigate the web. In International Conference on Learning Representations, 2018
Izzeddin Gur, Natasha Jaques, Yingjic Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning. In Advances Neural Information Processing Systems, volume 34, 2021.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world WebAgent with planning, long context understanding. and program synthesis. In The Twelfth International Conference on Learning Representations, 2023a.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin V Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding HTML with large language models. In ICLR 2023 Workshop Mathematical and Empirical Understanding of Foundation Models, 2023b.

---

Published as a conference paper at ICLR 2025
Peter C Humphreys, David Raposo. Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In International Conference on Machine Learning, pp. (166-9482. PMLR, 2022.
Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba. DOM-Q-NET: Grounded RL structured language. In International Conference on Learning Representations, 2018.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. OmniACT: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402. /7553, 2024.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. In Thirty--svennti Conference on Neural Information Processing Systems, 2023.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv: 02.4401116445 13649, 2024.
Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Miao-Xiang Zhang. SheetCopilot: Bringing software productivity to the next level through large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8198 8210, 2020.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations, 2018.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations, 2023.
Xing Han LU, Zdenök Kasner, and Siva Reddy, WebLINX: Real-world website navigation with multi-tum dialogue. arXiv preprint arXiv:2402.0590 2024.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: benchmark for general Al assistants. In The Twelfth International Conference on Learning Representations, 2023.
Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xucyuan Leng, He Kong, Yi Chang, and Qi Wang. ScreenAgent: A vision language model-driven computer control agent. arXiv preprint arXiv:2402.0794 2024.
OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. TooILLM: Facilitating large language models to master 16000+ real- wolldwrrdd APIs. arXiv preprint arXiv:2307.1673 2023.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the Wild: A large-scale dataset for Android device control. In Thirth-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajan et al. AndroidWorld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.i 240.532405.1453, 2024.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances Neural Information Processing Systems, 36, 2024.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hermandez, and Percy Liang. World of Bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pp. 3135-3144 144, PMLR, 2017.
12

---

Published as a conference paper at ICLR 2025
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang. Longtao Zheng, Xinrun Xu, et al. Towards general computer control: multimodal agent for Red Dead Redemption as a case study. arXiv preprint 24:3v:2003.033 2024.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. AndroidEnv: A reinforcement learning platform for Android. arXiv preprint arXiv:2 13231, 2021.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint 1xX1v:2355.1621, 2023.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. OS-Copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07455 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongehan Shin, Fangyu Lei, et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXi::2404.07972, 2024.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305. 16504, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WehShop: Towards scalable real-world weh interaction with grounded language agents. In Advances in Neural Information Processing Systems, 2022.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-use user interaction in real-world domains. arXiv preprint arXiv:2406. /2045, 2024.
Chaoyun Zhang, Liqun Li, Shilin Hc, Xu Zhang, Bo Qiao, Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. UFO: A Ul-focused agent for Windows OS interaction. arXiv preprint arXiv:2402.07939, 2024.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) a generalist weh agent, 2024.
grounded. arXiv preprint
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory exeemlla prompting with memory for computer control. In The Twelfth International Conference Learning Representations, 2023.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Danicl Fried, Uri Alon, et al. WebArena; A realistic web environment for building autonomous agents. In The Twelfth International Conference Leaminng Representations, 2023. 2023.
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS
We provide complete results of GroundUI-1K (Table 8), the edit distance of IDM-Multiple (Table 9), more metrics in CriticBench (Table 10), and application-wise results of the grounding sample task collected with AgentStudio (Table 11). Figure analyzes the relationship between element size and accuracy. suggests that larger element size may lead to higher grounding accuracy and explains why current models tend to perform better in mobile settings compared to web and desktop scenarios. In all experiments, we utilize the 001 version of the Gemini models and the 20240620 version of Claude 3.5 Sonnet. For online benchmark tasks, use GPT--4o (0513). The results presented in the paper were obtained using greedy decoding, with the temperature set to O. For non-GUI tasks, we limit the maximum number of execution steps 1. For GUI tasks, we limit the maximum number of execution steps to 30, For non-GUI tasks, we limit the maximum execution time to 30 seconds. For GUI tasks, limit the maximum execution time 60 seconds. If the model performs repeated actions, i.e., executes the same action consecutively three times, the task is considered failure.
GPT-44o (0806), while for other results,
Model
Web Open Models
Desktop
Mobile
Total
SeeClick
64.3
44.3
73.7
61.1
CogAgent
25.3
15.7
35.7
25.5
CogVL.M2-LLama3-chn
19B
2.5
2.7
5.3
3.4
MiniCPM-LJama3-V
2.5
0.0
0.3
2.7
0.9
Qwen-VL-Chat
0.0
0.0
0.0
0.0
PaliGemma-3B-mix-co
0.0
0.0
0.0
0.0
PaliGemma-33B-
0.0
Proprietary Models
Gemini 1.5 Pro
31.2
24.3
51.3
35.2
Claude 3.5 Sonnet (0620)
13.0
14.0
26.3
17.3
GPT-4o (0513)
7.5
8.3
26.3
13.4
GPT-4 Turbo (0409)
5.3
11.0
23.0
12.3
Gemini 1.5 Flash
0.5
4.3
26.3
9.4
Gemini 1.0 Pro
0.5
0.3
5.0
1.8
Table 8: Complete accuracy on GroundU1-1K.
Model
Mind2Web
ATTW
VWA
AgentStudio
Total
Claude 3.5 Sonnet
2.0
2.1
2.9
1.6
2.3
GPT-4o (0513)
3.5
2.0
2.5
Gemini 1.5 Pro
6.0
4.4
7.0
3.8
5.5
Qwen-VL--Cha VL--Cht
5.1
15.4
5.8
6.3
8.4
Gemini 1.5 Flash
294.5
7.2
7.2
7.8
90.6
Table 9: Edit Distance on IDM-Multiple (J). The edit distance is the number of operations (insertion, deletion, and replacement) needed convert the predicted action sequences the ground truth.
60
Success
Frequency
40
20-
10"
10a
10a
10"
1d"
Element Area (pixels)
Figure 7: The distribution of element areas between successful and failed clicks in AgentStudio sample task. Tasks with smaller elements tend to fail.
14

---

Published as a conference paper at ICLR 2025
Observation-Action Pairs
Only Observations Desktop
Model
Web
Desktop
Mobile
Total
Web
Mobile
Total
Accuracy
Gemini 1.5 Pro
73.5
82.0
62.0
71.4
65.0
86.0
59.0
66.3
Gemini 1.5 Flash
68.5
82.0
65.0
67.0
72.0
62.0
66.3
Claude 3.5 Sonnet
70.0
92.0
55.0
68.9
66.0
86.0
58.0
66.6
GPT-4o (0513)
68.5
90.0
58.0
68.6
63.5
84.0
57.0
64.6
-L--Cat
52.0
38.0
48.0
48.9
54.5
58.0
52.0
54.3
F1 Score
Gemini 1,5 Pro
72.5
84,2
52.5
69.7
61.1
88.1
48.1
62.9
Gemini 1,5 Flash
65.6
85.2
57.8
67.3
64.1
75.0
52.5
63.1
GPT-4o (0513)
68.0
91.5
41.7
66.5
61.4
85.7
39.4
60.8
Claude 3.5 Sonnet
68.4
92.9
36.6
87.3
64.2
Qwen- VL--Cha
55.6
55.1
64.4
58.5
63.2
73.4
67.6
66.2
Precision
Gemini 1.5
75.3
88.9
70.0
76.7
68.8
89.7
65.5
72.5
Claude 3.5 Sonnet
72.2
100.0
61.9
75.9
67.4
96.0
63.3
71.4
Gemini 1.5 Flash
72.3
83.9
72.7
74.8
70.2
80.8
70.0
72.1
GPT-4o (0513)
69.1
93.1
68.2
73.6
65.2
92.3
66.7
70.6
Qwen-VL--Chat VL.-Chat
51.7
48.7
49.0
50.2
53.1
59.2
$1.0
53.4
Recall
Qwen VL-Chat
60.0
63.3
94.0
70.0
78.0
96.7
100.0
87.2
Gemini 1.5 Pro
70.0
80.0
42.0
63.9
55.0
86.7
38.0
55.6
Gemini 1.5 Flash
60.0
86.7
48.0
59.0
70.0
42.0
56.1
GPT-4o (0513)
67.0
90.0
30.0
60.6
80.0
53.3
Claude 3.5 Sonnet
65.0
86.7
26.0
57.8
62.0
80.0
38.0
58.3
Table 10: Accuracy, score, precision, and recall on CriticBench.
Windows
Linux Browser
MacOS Music
Model
OS
Games
PowerPoint
Word
OS
Calculator
iMovie
SceClick
42.3
47.6
24.0
9.1
60.7
9.1
69.0
53.8
35.7
Claude-3 Sonnet
3.8
4.8
24.0
9.1
7.1
4.5
0.0
30.8
0.0
GPT-4V (1106)
11.5
23.8
20.0
9.1
14.3
4.5
3.4
11.5
3.6
Gemini- Pro
7.7
8.0
4.5
10.7
Qwen-VL--Chal VL-Chat
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Table 11: Application results on AgentStudio-collected GroundUI sample task.
B DESIDERATA FOR GENERAL VIRTUAL AGENTS
In this section, we elaborate on the three essential pillars for building next-generation virtual agents that can generalize and self- relf-improve. The rationale behind AgentStudio is to provide benchmarks cov- cring the desiderata for such agents, as well as useful environments and tools to pave the way for the emergence of general virtual agents. Specifically, tion/action spaces and real-world interactions (Section 2), tools that facilitate agent benchmarking and data annotation (Section 3), and benchmarks that both measure overall agent performance (Section and decompose these desiderata (Section 5).
need online environments with generic observa-
General UI grounding in the wild. Though UI grounding is long-standing research problem and considered a relatively simple task, it is still one of the major issues of current virtual agents. has been validated that an oracle grounding can drastically improve agent performance (Zheng et al., 2024). However, most existing benchmarks focused on the grounding tasks of mobile phones (Burns al., 2022; Rawles al., 2023). Though there are some recent datasets of diverse trajectories in web and desktop environments (Deng et al., 2023; Li et al., 2024; Kapoor et al., 2024), they have different formats and accessibility, which are not readily applicable for evaluating the grounding abilities of new models. For example, recording in these datasets often consists of high-level instruction and the corresponding human trajectory. However, for the grounding action at each step, the UI element may not directly relate the instruction. Also, note that there or incorrect instructions within those grounding-spert -ppeiifcc datasets, which can result inaccurate evaluation. Therefore, facilitate research on UI grounding in the wild, it is necessary to curate
lot of ambiguous
15

---

Published as a conference paper at ICLR 2025
diverse and cross-platton coss-platform benchmark with clear instructions, as well as scalable data pipeline for both finetuning and benchmarking purposes. Additionally, many current online environments often overlook incorporating videos in the agent observation modality, where agents can fail to capture real-world dynamic changes and solve tasks that require monitoring the screen.
Learning from Internet-scale actionless videos. Internet-scale data have contributed to remarkable success in language and vision. The development virtual agents can also benefit from the wealth of instruction videos available on the web, similar MineCLIP (Fan et al., 2022) and VPT (Baker et al., 2022). Compared to recording human trajectories, collecting videos is also cheaper and more scalable. However, one of the main challenges for current agents to benefit from video data is the lack of action labels in videos. Unfortunately, there is currently no relevant computer agent benchmark and infrastructure to guide research in learning to act from videos (i.e., inverse dynamics models). Such models can add pseudo action labels to video recordings to unlock the knowledge within Internet-scale videos, enabling virtual agents to generalize.
Open-ended learning with universal critic. It is not scalable and feasible to examine whether task successful through task-specific, hand-written rules, because computer tasks are massively diverse and open-domain. Though it is challenging for current models to master these tasks, it may be easier for models to do success detection first (i.c., discriminating trajectory outcomes as a critic model). A general critic model allows agents bootstrap via reinforcement learning tree search. It can also facilitate agent self-correction if the model can provide natural language feedback. Compared to binary scalar rewards widely used in previous web environments and reinforcement learning environments, natural language feedback is considered more helpful for language agents to fix their own mistakes (Shinn al., 2023). This ability is particularly significant for software where human demonstrations are limited, but there exist no benchmarks measuring it.
C DETAILS OF GROUNDUI
C.I DATA SAMPLE
Figure is an example of the data in GoundUI datasets. We deliberately curated a host of existing datasets for virtual agents across different platforms from diverse datasets. ScreenSpot (Cheng et al., 2024) is a dataset with 610 screenshots. Based on these screenshots, 1,272 instructions are collected, with 502/334/436 for mobile/desktop/web environments, respectively. In addition ScreenSpot, collect 9,268 single-step data from Mind2Web test sets, 3804 from OmniACT test sets (Kapoor et al., 2024), 3,455 from MoTIF test sets (Burns et al., 2022), 1,272 from ScreenSpot benchmark, and 227 annotated with AgentStudio tools. There are 18,026 data entries in total, comprising 13,522 unique screenshots. After deliberately examining existing datasets, the data without UI elements annotated in bounding boxes are filtered out.
C.2 RECAPTIONING AMBIGUOUS AND INCORRECT INSTRUCTIONS
Many existing UI grounding datasets consist of language instructions automatically inferred from non- readable internal meta information and did not go through rigorous cleaning. Some of the instructions are ambiguous, incorrect, or N/A, and not human--nddestanddable This makes the evaluation results of agent grounding these datasets inaccurate. Therefore, recaption the instructions with the help of GPT-4o. Figure 9, Figure 10, Figure 11, and Figure 12 demonstrate the examples of recaptioning in mobile, desktop, and web environments. The recaptioning process GroundU1-1K costs 1M input tokens and 9K output tokens in total.
16

---

Published as a conference paper at ICLR 2025
Feel better about finding healthcare
Clocto addide
Screenshot
AAA Aer
Instruction
Click on "Search",
Click on "C1ick to add title".
Click on the blue plus (+) button.
Bounding Box
(487, 217,31777711
1972,203,,125..22)
Resolution
[1280,720)
[1929,
[1170,
Source
mind2web_ tetoitttttt___iiiii
agent_studão
Platform
web
desktop
nobile
Figure 8: Data samples in GroundUI-IK/I8%
Top Lists
SHEIN.
Shere & Soportt
VEMES
OCARTY
HOME FC
FROS EC SAN
communiyy
Copy
Valentine's Day
Tap Charts Costa Sica
SALE
ES0
Screenshot
19%
20% OFF
Raw Instruction
the icon at the bottom left corner
the iten the top
the icon the top
Recaptioned Instruction
Click on the magnifying glass icon to search.
Click on 'Share' icon.
Click on "Category"
Figure 9: Examples of ambiguous instructions on mobile platforms sampled from MoTIF (Burns al., 2022). The recaptioned instructions on GroundUI-1k IK are shown under the raw instructions from the original dataset.
17

---

Published as a conference paper at ICLR 2025
Seatember 2025 [Septemberr 2023)
E
1404
200
22
Pk
26.
Screenshot
Cral
11
10
36
Ast ranedo
Raw Instruction
09/30
Recaptioned Instruction
Click
23" to view events for Saturday, September 23, 2023.
Figure 10: Example of incorrect instructions on desktop platforms sampled from OmniACT (Kapoor et al., 2024). The recaptioned instructions on GroundUI- IK are shown under the raw instructions from the original dataset.
Radio
Today's Hitss
Taligating
Screenshot
Raw Instruction
remind _usddd________
Recaptioned Instruction
Click on "Nusic Hits" to explore songs you know and love.
Figure 11: Example of ambiguous instructions on desktop platforms sampled from OmniACT (Kapoor et al., 2024). The recaptioned instructions on GroundUl-IK are shown under the raw instructions from the original dataset.
18

---

Published as a conference paper at ICLR 2025
Walmart
Pepattuns
Sosden
Cdliue
Screenshot
$12.95
00
Raw Instruction
NA
Recaptioned Instxuction
Click on 'Add to cart'.
Figure 12: Example of incorrect instructions on web sampled from OmniACT (Deng al., 2023). The recaptioned instructions on GroundUl-15 1K are shown under the raw instructions from the original dataset.
19

---

Published as a conference paper at ICLR 2025
C.3 PROMPTS FOR GROUNDUI
The prompts for GroundUl experiments are shown below. We match the model output with a regular expression. We use the response from Claude 3.5 Sonnet for demonstration.
User
<datasets/ httpt////iiggrooiiiigogee images/ wwwpwmee_eeeeeeeeeeeeeeeeeeeeeee www.aaeesseeeeeeeeeeeeeeeeeeeeee png?
Please output coordinate for next action based on instruction screenshot. Your answer should be of the following format: '(X, Y)' (without quotes) where X, Y is the coordinates ranging from O to I.
Instruction: Click "Discounts"..
Answer:
Assistant
Based the screenshot and the instruction click on "Discounts", the approximate coordinates would
be:
(0.85, 0.07)
This location corresponds to where the "Discounts" link appears in the top menu bar of the AMC website, slightly right of center near the top of the image.
Evaluation
AgentStudio will parse the response into (0.85, 0.07). This coordinate will be multiplied by the screenshot resolution to (1224.0, 63.0), which is outside the ground truth bounding box (1064, 99, 1148, 140) (left, top, right, bottom). Therefore, this result is incorrect.
20

---

Published as a conference paper at ICLR 2025
D DETAILS OF IDMBENCH
D.1 DATA SAMPLE
Figure 13, Figure and Figure 15 three examples IDM- MoD-liggle dataset on different platforms.
Figure 16 and Figure 17 are two examples for IDM-Multppe dataset.
Search cettings
To Thaeeday Bass 10
Montity music you
Nerwork & internet
rt modili, datu usugs, and hotspet
Apps & notincationss
Observations
Ricer oppi, cetluli -apu--ipp
Battery
Dieplay
Wulls skeea, laccstoe
Shorae
66% sted--t 7006 fhee
Losttio
Observation
Next Observation
Operation
press home
Bounding Box
raw": "rajjcttraw"";:: null, "from" : null, "repr": press home", "text": null, "to": null, "value": null3
{"action_oke cicloon_ooectt..e
Metadata
Instruction
Search for "bose quietcomfort 35" on walmart com, select the first entry, and add it to the cart.
Source
aitw_webshoppine
Platform
mobile
Action Space
["scroll up", "status task complete", "click",
*press enter", "scroll right", "scroll left". "type", "scroll down", "press back", "press home"]
Figure 13: Data sample of pressing home button on mobile platform in IDM-Single dataset.
21

---

Published as a conference paper at ICLR 2025
phok
Paue
Derte
Update faster, better YouTube
uoditets and sdde
and sabaa
Observations
ABS
Like
Mayi
Soow
URGVTE
Observation
Next Observation
Operation
click
Bounding
t"x": 0.903, "y": 0.9153
Box
Metadata
cerrriionoojeett..m null, "from": null, "repr": "click PLAY", "text": null, "to": null, "value": null3
Instruction
Open Youtube and go to "Your channel"
Source
aitw_googleapps
Platform
mobile
Action
["scroll up", "status task complete", "click",
"press enter", "scroll right", "scroll left", "type", *scroll down", "press back", "press home"]
Space
Figure 14: Data sample touching (clicking) on mobile platform in IDM-Single dataset.
22

---

Published as a conference paper at ICLR 2025
Wichorns
Weloame:
Observations
Observation
Next Observation
Operation
TYPE
Bounding Box
("height": 54, "width": 599, "x": 340.5, "y": 278)
Metadata
""action_abjort cbtpe_bbjjertarreeeeeeeee raw":: null, "fron': null, "repz": "[textbox] City, ZIP, School, Addzess, Neighhordoodd +> TYPE: Yorkville, TL", "text": null,, *to": null, "value": "Yorkvillo, TL*)
Instruction
Search for single family homes in Yorkvil le, TL with 2 bedroors bathroon budget $109,000 $200,000
Source
Platform
web
Action
("CLICK", "SELECT', 'TYPE")
Space
Figure 15: Data sample on web in IDM-Single dataset.
Instruction
GaneSpot
click 6.8]
click [9.
type ganaspot zeviews]
0.054]
RM
Trajectory
Source
aitw_genezal
Platform
nobile
Action Space
up", "status task completa", "click", "press anter",
right",
*scroll left", "type", "scroll down", "press back", "press hone")
Success
False
Figure 16: Data sample on mobile platform in IDM-Multiple dataset.
23

---

Published as a conference paper at ICLR 2025
apply for delivery driving job at Grand Rapids, MI, USA, Name: selson Freeman, Enail; Nelsonfree2greil.co con, phone: 2634556656.
Instruction
\"yes\"
options
aay starting most
Higher starting 613thour for most roles
[0,0]
CLICK [32,198]
Trajectozy
ay Starting at 519thour for most roles
(64, 668)
Source
mindZweb data__tett test_donainn
Platform
meb
Action Space
["CLICK", "SELECT", TYPE""]
Success
False
Figure 17: Data sample on web in IDM-Multiph dataset.
24

---

Published as a conference paper at ICLR 2025
D.2
PROMPTS FOR IDM-SINGLE
The prompt for IDM-Single:
User
Ratraetts/rrseecoooooe/////////r htsp://///////////aaaaa///////// eapps/ 1116088822354479941____..
<catasets/bzajjecttor http://wyyetttiiteeeeeeeeeee /ait www.g//gggggggaaappsss... 11660883229547098811 2.pgg
Analyze the two sequential images provided. These images represent observations before and after action taken. Your task is to identify which type of action was executed between these two observations.
Available actions:
A. scroll up
B. status task complete
C. click
press enter
E. scroll right
F. scroll left
G. type
H. scroll down
press back
J. press home
After your analysis, conclude your response with the answer in this format:
Answer:
Where X the letter corresponding the action you believe was taken, chosen from options provided above.
For example, if the options were:
A: Type
B: Click
Scroll
And you determined the action was "Click", your response would end with:
Answer: B
Assistant
Analyzing the two images, we can observe the following changes:
1. In the first image, we see partial list of settings including "Bluetooth", "Apps & notifications", "Battery", and "Display".
2. In the second image, the list has moved upwards, revealing additional settings below "Display", namely "Sound" "Storage".
3. The top items from the first image ("Bluctooth" and "Apps & notifications") are no longer visible the second image.
4. The battery icon and percentage (100%) remain constant between the two images.
5. The time (2:12) and status bar icons the top of the screen remain the same.
These observations indicate that the sereen content has moved upwards, revealing more options at the bottom while hiding options that were previously at the top.
Given these changes, the action that most likely taken between these two observations is scrolling down the settings Answer:
Evaluation
AgentStudio will parse the response into H, which is the same as the ground truth action. Therefore, the score is 1.0.
25

---

Published as a conference paper at ICLR 2025
D.3 PROMPTS FOR
For IDM-Multiple, we prompt with up to six consecutive observations from the sampled trajectories.
User
cctpraacessaaicoooo...//nnnnnaie 80355897590:32204__.ppppp
catreett//iiiicccoo.../////ggagt 8335822992232044..ppppp
hctpoc/c/hteeeeooo/l///////hs/tt 8035882690:3222422..pppgg
cctpsssssttaacccoo..///////saats 605555226922322044_...ppp.
Gatootasttaacccc/////////eee//an 803559229023220444..pppgg
<lacasets/trajjettory__tt/////e http:/iaaaaaaaaaa http:///ww...oooo/oo//oaaaaaaa/a 80555925002388888......m
Analyze the sequence of images provided. These images are snapshots taken from a video screen recording.
Your task is to identify the types of actions executed in order throughout this recording.
Available actions:
scroll up
B. status task complete
C. click
D. press enter
E. scroll right
F. scroll left
G. type
H. scroll down
press back
press home
After your analysis, conclude your response with the answer in this format:
Answer:
X
Y
Where X. Y. Z. etc. are letters corresponding to the actions you believe were taken, chosen from the options provided above. For example, if the options were:
Type
B: Click
C: Scroll
And you determined the actions "Type", then "Scroll", your response would end with:
Answer:
A
C
26

---

Published as a conference paper at ICLR 2025
Assistant
Based on analyzing the sequence of images, here's 's my interpretation the actions taken:
1. The first image shows the home screen.
2. The second image shows search or app drawer opened.
3. The third image displays the Google Play Store page for the Contacts app.
The fourth image shows search results page for "contacts" within app store or system.
5. The image displays search results for apps and games.
6. The last image shows search results for YoouTube Kids".
Given this sequence, here's my assessment of the actions taken:
1. The clicked open the app drawer search (C)
2. They typed "contacts" search (G)
3. They scrolled down to view more results (H))
4. They clicked on the Contacts
result (C)
5. They pressed back return to the search (I)
6. They typed to modify search query, likely changing to "YouTube Kids" (G)
Answer:
C
G
H
G
Evaluation
AgentStudio will parse the response into (C, G, H. C, I, G). The correct action sequence is (C. C, C. C. G). Each letter represents available action in the action space. Therefore, the model generates incorrect action sequence. The predicted action sequence requires at least three times of modification (delete, insert, or replace) to become the correct one. Therefore, the score is 0.0, and the edit distance is
27

---

Published as a conference paper at ICLR 2025
E DETAILS OF CRITICBENCH
Figure 18 and Figure 19 are two examples from CriticBench.
Instxuction
Find symptons slaap apoes
TO [1238.19]
TYPE [slaep apнea symptons]
Trajectory
How to Loos 300 Pourds or More
[779.75]
Source
nincheeh data, test destt dona in
Platform
Action
["CLICK", "SELECT", "TYPE")
Space
Success
Figure 18: Data sample on web in CriticBench dataset.
Instruction
What's Sur
Trajectory
pcess
click (8.355,8.060) 6.090]
type
click [8.2,0.130]
Suzinare]
Source
aits, goneral
Platform
mobile
Action
['scroll up", "status task corplete", "click", "press enter". "scroll right". "scroll left", "rype", "Scroll" down".
Space
"press back". "press home"
Success
Tzue
Figure 19: Data sample on mobile platform in CriticBench dataset.
28

---

Published as a conference paper at ICLR 2025
E.1
PROMPTS FOR CRITICBENCH WITH OBSERVATION PAIRS
User
2566800339922055222___..22
press home
1556700339200155522__...22
click NAV BAR CIRCLE
<datasets/traaecceteet 116680339227555222__222222
type the news in Suriname
clatasets/ttrajectory wwwprryyy_ieeeeeeeeeeee http://aaccctootiiitem////////ae http:////e..aaaaaaaaaaaaa//////r images/generall 156670333922555544__..22
click MAGNIPYING GLASS
Analyze sequence images provided. These images are snapshots taken from a video screen recording. Your task to evaluate whether the given instruction has been successfully completed based the provided trajectory. This trajectory is depicted through series of actions and the corresponding observations before and after each action.
Instruction: What's the in Suriname?
Action space: scroll up, status task complete, click, press enter, scroll right, scroll left, type, scroll down, press back, press home
Carefully examine the sequence of actions and their results. Determine if the final state successfully fulfills the given instruction.
Conclude your analysis with one of these two responses:
If the trajectory successfully completes the instruction:
Answer: True
If the trajectory fails complete the instruction:
Answer: False
Assistant
Let's analyze the sequence of images and actions:
1. The first image shows the Google Drive app interface.
2. The second image displays the Android home screen.
3. The third image shows the Google scarch bar with recent searches and trending topics.
4. The fourth image shows the Google search bar with "the news in Suriname" typed in, along with search suggestions related to Suriname news.
Based this sequence,
that:
1. The user navigated from Google Drive the home screen.
2. They then accessed the Google search bar.
3. They typed "the news in Suriname" into the search bar. However, the sequence ends before actually searching for displaying any from Suriname. The
has only typed the query, but hasn't pressed enter clicked on any search results to view the actual news. Therefore, while the user has taken steps towards finding news Suriname, they haven't 't actually accessed or viewed any news content yet.
Conclusion:
Answer: False
The trajectory does not fully complete instruction of finding out what the news Suriname is. only reaches the point of typing the search query, without actually retrieving or displaying any news content.
Evaluation
AgentStudio will parse the response to get the answer "False". However, the given trajectory successfully completes the instruction. Therefore, the score 0.0.
29

---

Published as a conference paper at ICLR 2025
E.2
PROMPTS FOR CRITICBENCH WITH OBSERVATIONS ONLY
User
catuece//aaaccoog/l///////s/seie 60580031124453332666....o0
ketpes/s/saaiccccra/r/////a/aaat 6055033122453566___ppngggg
haapett//heaaaccooo//////////a/r 505500.312221333366610.....gg
"attaetss/taeeettooo_ttteeeegeee htt.gg////atttmmmg////eeeeallll 3058033122583463.11..
<latasets/trajecttory_t haatasttt/aaeerrrrr_ttt/////ess htt.://etrrryttttttttteeeeeeeeee /aitw 50558031222488322112222
Analyze the sequence of images provided. These images snapshots taken from video screen recording. Your task evaluate whether the given instruction has been successfully completed based the provided images.
Instruction: Check the settings for the Amazon Alexa app Carefully examine the sequence of images. Determine if the final state successfully fulfills the given instruction.
Conclude your analysis one of these two responses: If the trajectory successfully completes the instruction:
Answer: True
If the trajectory fails complete the instruction:
Answer: False
Assistant
After carefully examining the sequence of images provided, can conclude:
Answer: False
Reasoning:
1. The images show a smartphone home screen with a pink gradient background.
2. The date. time, and weather are displayed at the top.
3. There several app icons visible at the bottom of the screen, including Google Play, Messages, Clock, Chrome, and Gmail.
4. A Google scarch bar present at the bottom.
5. In some images, there's 's "My apps" menu partially visible.
However, throughout the sequence, there is no indication that the Amazon Alexa app is opened or that its settings being checked. The instruction images do not show any interaction not been successfully completed based on the provided images.
"Check the settings for the Amazon Alexa app." but the Alexa app or its settings. Therefore, given instruction has
Evaluation
AgentStudio will parse the response to get the answer "False". However, the given trajectory successfully completes the instruction. Therefore, the score 0.0.
30

---

Published as a conference paper at ICLR 2025
F DETAILS OF ATUNIO BENCHMARK TASKS
F.1 SYSTEM PROMPT
System
You are world-class programmer who can complete any instruction by executing Python code. Now you operating real computer-based environment, and you may be given screenshot of the current computer screen. The only way interact with environment is to write Python code.
You are given a task instruction in the form of string and you need to write Python code to complete You are using Jupyter Notebook to execute the code and shell commands, so generate code/shell commands step by step with multiple blocks. You
Jupyter internal operators "% and "!". The generated 'python'y n" and "nn x... *" Your response should include and only include one code block. You will get the execution result You can interact with the Notebook multiple rounds. Thus, you are not sure about the code, you can submit the code to see the result and modify the code if needed. When you think the instruction is completed and the code is correct, end the code block with 'exit ()".
code/shell commands should be wrapped between
code.
For simplicity, you can use following code snippets:
You are probably given a screemshot of the current computer screen. You can only use the Jupyter Notebook to interact with the environment. We have provided the initial code to access the mouse and keyboard:
agent_studis envs. www...desssee.ee desttoppeeepppe import Mouse, Keyboard mouse Mouse () keyboard Keyboard
the 'mouse' and 'keyboard' objects interact with the environment. "mouse. cl ick (x: int, y y:int, button:s con:str, cl cllickssint interval float) can used to click the "button" at the specified position "click" times with a specific interval. You can choose "button" from "leff", "right", and middle. 'keyboard, type (text:str, interval: can be used type the specified text with specific interval. "keyboard.hotke" (keys: List can be used press hotkeys.
You
If your task needs to access the Google service, you can use the "credentials file the "/a- gent sent_studiocconig directory. Also, there are six token files, 'docs 'docs_token, json". "drive serive.token.jpe. joken.json' gomail_tokee.jjaee token.json', 'sheets token.json', 'slides. token,json', 'calendar token.json' and 'forms token.json' in the congent_studooconnit directory, and you service. E.g. you can use following code to access the Google Drive API:
any of them access the corresponding Google
import json
from google . cauth2 import credentials from woggleeapiciient.issoveee import build
json"
with open (token_path, "r") f:
token lo Crwdentials.....o.o......ii
creds irddmmieess.creeettttalll.. from fron user info_ tcken, /httpss:/ww..gogllapiss..oo
service build
*v3" credentials-creda
service , filcs()) let.geettiiaaeeeeee (filcId=xxxxxe))
Also, you should assume the timezone is UTC+0 if there's no further specification.
F.2
PROMPTS FOR TASKS WITH TEXT OBSERVATIONS ONLY
System
<SYSTEM PROMPT in Appendix F.1>
User
The task instruction: <TASK SPECIFIC INSTRUCTION-
31

---

Published as a conference paper at ICLR 2025
Assistant
Action: <PYTHON CODE
User
Observation: <PYTHON EXECUTION RESULT 1>
Assistant
Action: <PYTHON CODR
User
Observation: <PYTHON EXECUTION RESULT
F.3
PROMPTS FOR TASKS WITH TEXT AND VISUAL OBSERVATIONS
System
<SYSTEM PROMPT in Appendix F.1>
User
The task instruction: <TASK SPECIFIC INSTRUCTION
Assistant
Action: <PYTHON CODE
1>
User
Observation: <PYTHON EXECUTION RESULT 1>
Assistant
Action: <PYTHON CODE
User
Observation: (<SCREENSHOT>>
<PYTHON EXECUTION RESULT 2>)
32

---

Published as a conference paper at ICLR 2025
F.4 EXAMPLE TASK CONFIGURATION
Key
Value
Task ID
Instruction
Give the slide 2 right aligned title, "Note". True
Visual
Max Steps
30
Max Time
60.0
Evaluation Procedure
Compare between ref. and target xptx
Reset Procedure
1. Create folder structure, 2. Copy file, 3. Open PPTX file
Cleanup Procedure
1. Delete folder structure, Kili LibreOffice process
Table 12: Task configuration with simplified evaluatioonrresetllllamp procedures.
F.5 EXAMPLE FAILED TRAJECTORY
Task ID: 0a2e43bf- 463-433--b266-4611-222
Instruction: Work out the monthly total sales in new row called "Total" and then create a line chart to show the results (x-axis Months).
Model: GPT-4o (0806)
User
The task instruction: Work the monthly total sales a new row called "Total" and then create a line chart to show the results (x-axis be Months).
Assistant
complete this task, we need perform the following steps:
1. Calculate the total sales for each month and add new row called "Total".
2. Create a line chart to display the monthly total sales.
Let's proceed with the code to achieve this.
'python
import pandas as pd
import matplet ib. pyplot as plt
# hoad the Fxce1 file file pith path = "SalesRep.xssx xlsx'
df = read excel (file path, sheet name=/ Sheetl')
calculate the total sales for each month df. loc['ootal]] df. sum (numeric_cnly-trre)
# Plot the line chart plt. fit.figure(sn (figsize= (ft.figgere (10, 6))
plt.plot collccmmns [1:], df 1fcl Total'] [1:], marker='o' of) pit.title (1 Monthyy Total Sales')
plt.xlabel ('Months')
pit.ylabell aotaal Sales')
pit.gria (True)
plt. p ()
33

---

Published as a conference paper at ICLR 2025
User Ohservation: Screenshot
E
RoT a
New E
Jupyter output
"error"; : Fe:leeooFFonndBror: [Errno 2] such file or directory: "aleesep. xxsx'
34

---

Published as a conference paper at ICLR 2025
Assistant
calculate monthly total sales create line chart, follow steps:
1. Close the "Tip the Day" dialog.
2. Add new row for the total sales.
3. Use formula
each month's 's sales.
Create line chart.
Let's proceed with the steps using Python code to automate this process.
" "python from agent_studio envs envs. desktop_eer import Mouse, Keybcard import time
mouse Mouse()
keyboard Keyboard
# Close the "Tip of the Day" dialog www.cclcckkcx00000 y=400, outton=' left', clicks=1, 1ntervai=0.1) time. (ime.sleep(1)
Click on the cell to enter the total formula for January nouse.click. (x=200, y=300, button='left: clicks=l, interval=0.11 keyboard. (eyyboad.tyyp('aaee (BB2 keyboard. hotkey ]nter']) time. (ime.slocp
interval=0.1)
# Repeat for the other months columns = ['C', 'D', 'E', 'F', 'G', [H'] for i, in enumerate (columns, start=3)
mouse.click
x=200 + (1-2)
y=300,
citton="left',
clicks-1,
interval-0.1
keyboard. =E' ((col 12: (cel)ll)', interval=0.11
keyboard, hotkey (1 Enter'] ])
time.sleep(1)
# Create line chart # Click the Insert
mouse.clict (x=50, y-50, button='left', clicks-1, interval=0.1 1) slme.))eep(1)
Click Chart mouse.click (x=100, (x=10xx0000 y=200, button=' Left', clicks=1, interval=0.1) sime.sleep
I Follow the chart wizard steps te create a line chart
# This part may require manual adjustments depending the interface
exit ()
35

---

Published as a conference paper at ICLR 2025
User Ohservation: Screenshot
E
RoT
New 2
18042218
Jupyter output:
("error": "Jupyter timeout: "}
Exit hecause of: Agent outputs "exit()"
Evaluator
0.0
feedback: "rule sheet data failed, sheet data matching failed"
36

---

Published as a conference paper at ICLR 2025
F.6 EXAMPLE SUCCESSFUL TRAJECTORY
Task ID: 34ccodl1-77554422251994422224423
Instruction:
Open VSCode with the file
Model: GPT-4o (0806)
User
The task instruction:
Open VSCode with the file "/home/buuntw/geet studio/vscode workspace/RAADME..n.
Screenshot:
Assistant
open VSCode with the specified file, execute command to launch VSCode the terminal.
'python
!code /home/ubunne httpt/////gentts_tteteeee stunttssssdiiee wwwpvvvvoddeeeeeeeeooo space RRADMR md v
37

---

Published as a conference paper at ICLR 2025
User Screenshot:
Get Started with VS
indiing
Sun
Dan Geiaa
kpliingh Gendas
Selreiishnneg LA
Jupyter output No output
Assistant
It looks like VSCode open with the README.ni md file. you need further assistance to perform another action, let me know!
Exit because of: No valid action
Evaluator score: 1,0
38

---

Published as a conference paper at ICLR 2025
F.7 ADDITIONAL EXAMPLES
Task ID
Observation
Text
Text & Screenshots
Instruction
Delcte the file remporary.doex, in the http:bbuwwwttgaget.ssuuuttt///// rectory.
Delcte the file 'temporary.doxx in the
di-
di- Define the file path f file.path / pogone//uuuut//gggeee studio// tmp/docs/tentt http//br///emoorarr..... doopeaarrddooxxx docx' Check if the file exists it path.exists (file.path): * Delete the file os remove (file-path)
rectory.
Response
To delete the file 'temporary.doxx in the specified directory, I'll execute the neces- sary shell command. !rm
import os
/home
docs/ teccsatepprray..oo docx
print (f"File 'file path' has been deleted.")
else:
print (f"File 'file 'file.path' path' does not exist.")
Action 1
!rm /ttp/uuuuutttttttttttttto///////
import
Define the file path filc.path hrtomm//uuuutuu/geetssuudd.
doca/tempora.y docx
hmpp/dooss/temprrayy/ docx' Check if the file exists os os. path.exists (file path): # Delete the file os remove (file path) print (I"File 'file path' has been deleted.")
else:
print (f"File 'file-path' does not exist.
Response
seems you have provided a screenshot of desktop environment. If you need assis- tance with task related to this environment, please let know how can help!
The file 'temporaryddcex has been suc- cessfully deleted from the 'hhom/uubuntu/a gent stntsttudiomm////s/// directory. If you have any further tasks need additional assistance, feel free ask! None
Action 2
None
Table 13: Additional comparisons between GUI and non-GUI online benchmark tasks.
F.8 SOFTWARE SELECTION
Our software selection is based on systematic criteria. (1) Open-source: We select open-source software to avoid copyright issues and encourage community contributions. (2) Popularity: We include widely used tools, such as Google Workspace, VS Code, and LibreOffice, which are among the most prevalent applications in their respective categories. GIMP (GNU Image Manipulation Program) is one of the most popular open-source image editors. (3) Diversity: To represent broad range real-world use cases, we select software across different domains, including OS components, office suites, code editors, and image editors. Although our environment can support proprietary software like Photoshop, we prioritize using ones that do not require licenses for running benchmarks. Meanwhile, our benchmark tasks for GIMP are designed to be easily applicable to Photoshop well, since most image-related tasks are evaluated by checking the output images.
F.9 MORE TECHNICAL DETAILS
Our implementation uses Docker for a lightweight environment compared to virtual machines used by OSWorld (Xie et al., 2024). Key technical aspects include: (1) Performance latency: The runtime
39

---

Published as a conference paper at ICLR 2025
mainly depends the time for task reset, action generation & execution, auto-eval, and cleanup. AgentStudio designed to mimic real-world computers operating in real time. The latency of reset, auto-eval, and cleanup depends allocating additional resources to Docker. At present, the main bottleneck is the response time of LLM action generation, well the time required execute L.LM-generated code, rather than the latency in AgentStudio. (2) Scalability: Our architecture supports multiple instances running concurrently. The separation the front end and back end ensures that LLM operations within Docker containers do not interfere with each other. Google API operations can also be isolated by setting up different accounts. (3) Reliability: Our Docker setup safeguards local files and systems from potential vulnerabilities. For sensitive operations, AgentStudio supports enabling human verification to enhance reliability and safery.
the Docker resource configuration, which can be improved by
Instruction
Update content 'agenda.txt' in 'home/ubuntu/ageet studio/mp/mmettngss" to 'Updated Meeting Agenda'. from agent_studic envs. desktop_eny import Mouse, keyboard mouse Mouse () keyboard Keyboard()
Non-visual
with open
' http:///wwnnus/seesettt///////te txt', 'w'
as f: f.write (' ('Updated Mecting Agenda') exit
Visual
from agent.studic envs. desttooooo Import Mouse, Keyboard mouse Mouse
keyboard = Keyboard
beyyoard.hotker ([' super'])
keyboará.type
gedit /home//bbuntu/agent studi stadio///m/meeetingg// agenca.txt!
keyboard. hot.eh.ooooeyyyyoo.. [I'enter'] 1)
mouse. mouse.click(99 (900, 500, 'left', 1, 0.1) hetkke ctrl', 'a'1)
Elyboard.type (' Updated Meeting Agenda') mouse. (20, 20, 'left', 1, 0.1)
couse.click(1. (15, 20, 'eet'' 1, 0.1)
exit()
Table 14: Actions of Gemini 1.5 Pro in visual and non-v vosuvi tasks. When providing visual informa- tion (screenshot),
Gemini model more likely to be distracted by the visual information. will try to use keyboard and mouse operations to complete the task, no matter how easy the problem is solved with the API.
40

---

Published as a conference paper at ICLR 2025
G DETAILS OF AGENTSTUDIO TOOLS
G.1 GUI ANNOTATION
To use the tool for GUI element annotation, first, click the "Capture" button to take screenshot, which will appear the left side of the interface. After the screenshot is captured, input the annotation instruction in the text box labeled "Enter instruction here." t Then, draw a bounding box around the relevant area in the screenshot match the instruction. Once the instruction and bounding box are set, click "Save" to store the annotation. If users need to start over, click "Reset" to clear the screenshot and instructions, then repeat the process needed for more annotations.
(a)
(b)
Figure 20: (a) Step 1: Capture screenshot. (b) Step 2: Input instruction and draw the bounding box.
G.2 ACTION RECORDING
The trajectory recorder is Command Line Interface (CLI) tool designed to capture mouse actions, keyboard inputs, and screen recordings. As shown in Table 15, to use the tool, begin by selecting path where the recording will be saved and then input instruction deseribing the task users intend to complete. The recorder will imitiate a 5-second countdown, after which users will hear chime signaling the start of the recording. Once have completed the task, stop the recording by pressing the ctrl+shift+o key combination.
$ as-traj-recorder
Please enter the folder path where you want to save the record (Press Rnter to use the default path: "/home/ubuntu):
Please input the instruction:
Please help open the autosave feature VS Code and delay AutoSave operations for 500 milliseconds in the Vs Code setting.
recording will start in 5 seconds You will hear sound when recording starts
3
2
press etri+shift+o to stop recording
Table 15: Illustration of the trajectory recorder in use.
41

---

Published as a conference paper at ICLR 2025
G.3 VIDEO REFINEMENT
Figure 21 illustrates the user interface of our trajectory editor. Users can open recording saved with the tool in Appendix G.2 and view the keyboard mouse operations in the right list view. Here, users can leverage shorteuts to instantly modify the recorded operations, e.g. deleting an operation, aggregating several operations, and changing operation information.
cance
LaHo (1S1ON))
(1D (00/204)
20an lycas (SCCP))
Figure 21: The interface for video refinement.
G.4 ONLINE BENCIIMARK VISUALIZATION & HUMAN VALIDATION
Figure 22 shows the GUI of our online benchmark, where users select and view task configurations, execute the task, view the action execution results, and get feedback. Users can leverage this interface evaluate task and calculate human completion time.
page Eecciiiinne wee inrt toartom FiA scctvvi
off Sengert
Cragp fani suvirnn Jaberron
come
1it sostii dur cuiies cla tnd
Figure 22: Online benchmark in GUI mode.
42


Reviews:
[{"review_id": "llx0CNzOfs", "reviewer": "Reviewer_w3vg", "summary": "The paper introduces AgentStudio, a toolkit designed for building general virtual agents capable of handling multimodal observations and complex action spaces in dynamic, open-domain environments. It includes a set of environments, tools, and benchmarks that address the limitations of current domain-specific virtual agent evaluations. AgentStudio provides a lightweight, interactive environment with generic observation and action spaces, integrating tools for task creation, GUI element annotation, and video action labeling. The paper also presents three datasets—GroundUI, IDMBench, and CriticBench—that evaluate fundamental agent abilities such as GUI grounding, learning from videos, and success detection, aiming to advance the development of robust, general, and open-ended virtual agents.", "questions": "How does AgentStudio handle user interactions, and what mechanisms are in place for agents to learn from and adapt to user feedback? Is it feasible to conduct a human evaluation of user experience when using the agent?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  This paper provides a lightweight, interactive environment with highly generic observation and action spaces, such as video observations and GUI/API actions, which expand the task space to a massively open domain and real-world tasks. AgentStudio comes with tools for creating and validating benchmark tasks, annotating GUI elements, and labeling actions in videos, which are essential for customizing and validating tasks in real-world settings.\n-  The toolkit enables online interactions for learning through trial and error, providing language feedback on failure reasons, which is crucial for open-ended learning and self-improvement of LLM-based agents.\n- The paper introduces three datasets—GroundUI, IDMBench, and CriticBench—that target UI grounding, action labeling from videos, and success detection, respectively, providing a structured approach to evaluating and improving fundamental agent capabilities.", "weaknesses": "-  Although the authors have made the code available in the supplementary materials, it would be beneficial to offer a more detailed guide to assist users in understanding and implementing the benchmark effectively.\n- The paper's claims are somewhat overstated. While AgentStudio's tasks primarily focus on interactions within 2D graphical user interfaces (GUIs), the capabilities of a general virtual agent extend beyond these to include interactions with 3D virtual environments, such as those found in the metaverse. To align the paper's title with its scope or to enhance its benchmark, it would be essential to incorporate additional 3D world scenarios, like 3D video games, which would provide a more comprehensive assessment of a virtual agent's capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "59DV6o1SLY", "reviewer": "Reviewer_v3Q2", "summary": "In this paper, AgentStudio, a trinity of environments, tools, and benchmarks for building general virtual agents is proposed. The benchmark contains GroundUI that evaluates UI grounding capability, IDMBench that evaluates action labelling capability and CriticBench that evaluates success detection module. Overall experimental results show that the existing VLM is still far away from being able to fully solve these benchmarks. Compared w/ previous works, AgentStudio provides the better observation and action spaces, and therefore can help develop and evaluate agents in real-world settings.", "questions": "1. Will AgentStudio be open sourced?\n2. I did not quite get \"For example, we can create a failure trajectory by extracting the first few steps of a successful one\" How exactly is failure example being created?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written, easy to follow. \n2. The dataset curation process makes sense to me.\n3. The experiments are adequate. Compared to previous works, AgentStudio has many advantages including interactivity, supporting data/tasks/tools, supporting language feedback, etc.\n4. AgentStudio shows the short coming of existing models. For example, existing models can do pretty well on single API tasks, but very poorly on compositional tasks.\n5. Also the benchmark shows that specialized models can do better than general model. For example, SeeClick does better than Gemini, Claude, etc for GroundUI.", "weaknesses": "1. Not sure # of tasks is enough. I would love to see the # of tasks can continue to grow.\n2. Currently, the software seems to be randomly selected. One potential improvement is that maybe the author can get some statistics of most used software and include the top ones into the Benchmark. For example, I would imagine Photoshop could be an interesting case to add to the evaluation suite.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7kktkGbBXY", "reviewer": "Reviewer_uDrN", "summary": "This paper presents AgentStudio, a toolkit for building and evaluating general virtual agents that can interact with software through GUI and API interfaces. The main contributions are: (1) An interactive environment supporting multimodal observations and actions, (2) Tools for creating benchmark tasks and annotating data, (3) A benchmark suite of 205 tasks and three datasets (GroundUI, IDMBench, CriticBench) for evaluating fundamental agent capabilities.", "questions": "Can AgentStudio enable large scale data collection, and agent evaluation? Will it support multi-threading that can be integrated to online RL training of agents? \n\nWhat are the key technical differences between AgentStudio and existing environments like VisualWebArena?\n\nWhy are the evaluation datasets so small? Are there plans to scale them up?\n\nIn IDM-Multiple, accuracy is calculated based on the exact match of the action sequences. Is this too restrictive? Would there be multiple optimal action sequences? Even suboptimal, but successful sequences demonstrate agent capabilities. \n\nThe paper presents an interesting system but falls short in terms of technical novelty and scale of evaluation. The main contribution appears to be integration of existing components rather than fundamental technical advances. The small scale of the evaluation datasets also limits the impact. While the direction is promising, the current work would benefit from more technical depth and larger-scale evaluation.\n\nPost rebuttal:\nThere are still further questions on how useful AgentStudio is as many components are very basic, e.g. annotation tool as characterized \"minimalism\". Furthermore, it is not clear college students are typical users and whether there are any quality control processes, the 51% calls into questions to the labeling process.\n\nI still feel the paper is borderline. Nonetheless, I am happy to update my score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The interactive environment design allowing both GUI and API interactions is valuable\n\n2. It introduces an online task-completion benchmark and three datasets to evaluate fundamental agent abilities in real-world settings. The benchmark suite consists of 205 real-world tasks across various applications such as VS Code, Google Workspace,\nand Office suites\n\n3. Evaluating current LLM-based agents (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Qwen-VL-Chat) on real-world software interaction tasks, it provides good analysis of failure modes and limitations of existing models.", "weaknesses": "1. Limited technical novelty - the environment appears to be largely an integration of existing components without significant new technical contributions.\n\n2. The three datasets created for fine-grained evaluation are quite small in scale: IDMBench, criticBench has only 345, 350 trajectories respectively.\n\n3. Insufficient technical details about the implementation: \nOnly cursory mention of using VNC and Docker\nNo discussion of performance, latency, scalability, reliability considerations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3yHMqn7Sm1", "reviewer": "Reviewer_iPnp", "summary": "This paper presents a comprehensive toolkit that includes an environment, tools, benchmarks, and datasets designed to evaluate virtual agents’ capabilities across diverse software tasks. The authors focus on enabling realistic evaluations of agents in real-world settings, providing tools to easily construct these environments within a POMDP framework. They conduct extensive experiments on their benchmark, covering various GUI and API-related tasks, to assess the overall performance of current agents and their effectiveness in three specific tasks. Additionally, they introduce three datasets that address fundamental challenges, highlighting limitations in current models.", "questions": "See above Weakness", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. AgentStudio provides a holistic toolkit, including environments, tools, and benchmarks, addressing the need for flexible agent training across varied virtual scenarios​. It also supports diverse input modalities (text, images, videos) and action types (GUI, APIs), making it versatile for training agents to handle real-world tasks and improving generalizability.\n2. The creation of GroundUI, IDMBench, and CriticBench highlights an effort to evaluate core agent skills such as GUI grounding and success detection, advancing the field with nuanced benchmarks​.\n3. AgentStudio's lightweight design and compatibility with various operating systems (including Docker environments) improve accessibility and ease of use for a wider research audience​.", "weaknesses": "1. Although AgentStudio emphasizes versatility and real-world applicability, the benchmarks and tasks are still simulated in controlled environments, which may not fully capture the complexities of real-world application. The challenges faced by agents in uncontrolled, dynamic environments are difficult to replicate, which could limit the generalizability of the findings to actual user settings.\n2. Success rates and accuracy are used as primary metrics, which may not fully capture an agent's capabilities, particularly in scenarios requiring complex decision-making or nuanced interaction. For example, success rates don’t account for partial progress in multi-step tasks, which may lead to an incomplete assessment of an agent’s performance and fail to identify areas where improvement is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)