What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Authors: Wendong Bu
Yang Wu
Qifan Yu
Minghe Gao
Bingchen Miao
Zhenkui Zhang
Kaihang Pan
liyunfei
Mengze Li
Wei Ji
Juncheng Li
Siliang Tang
Yueting Zhuang
Venue: ICML 2025 oral (2025)
Keywords: Virtual Agent; Digital Agent; Multidimensional Benchmark
Primary area: applications->computer_vision

Abstract: As multimodal large language models (MLLMs) advance, MLLM-based virtual agents have demonstrated remarkable performance. However, existing benchmarks face significant limitations, including uncontrollable task complexity, extensive manual annotation, and a lack of multidimensional evaluation. In response to these challenges, we introduce OmniBench, a self-generating, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity through subtask composition. To evaluate the diverse capabilities of virtual agents on the graph, we further present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities. Our synthesized dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate. Training on our graph-structured data shows that it improves generalization across environments. We conduct multidimensional evaluations for virtual agents, revealing their performance across various capabilities and paving the way for future advancements. Our project is available at https://omni-bench.github.io.

What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Wendong Bu 12* Yang 2u 2d Qifan Yu 1" Minghe Gao Bingchen Miao
Zhenkui Zhang' Yueting Zhuang
Kaihang Pan
Yunfei Li Mengze Li
Wei Ji
Juncheng Li 1 Siliang Tang
Abstract
2024b; Gao et al., 2024a; Miao et al., 2025) have demon- strated promising performance web navigation (Shen et al., 2024), mobile device control (Ge et al., 2024), and computer interaction (Hu et al., 2024). To explore real-world values of visual agents, current research mainly evaluates their performance based on offline trajectory similarity with human demonstrations (Rawles et al., 2024; Lu et al., 2024; Deng et al., 2024) or by using expert-crafted functions in interactive online environments (Xie et al., 2024; Zhou et al., 2023; Yao et al., 2022).
As multimodal large language models (MLLMs) advance, MLLM-based virtual agents have demonstrated remarkable performance. How- ever, existing benchmarks face significant limi- tations, including uncontrollable task complex- ity, extensive manual annotation with limited see- narios, and a lack of multidimensional evalua- tion. In response to these challenges, we in- troduce OmniBench, a gelfrrenn,,,,ng, cross- platform, graph-based benchmark with auto- mated pipeline for synthesizing tasks of control- lable complexity through subtask composition. To evaluate the diverse capabilities of virtual agents on the graph, we further present OvniEval, a multidimensional evaluation framework that in- cludes bubtask-level evaluation, graph-based met- rics, and comprehensive tests across 10 capa- bilities. Our synthesized dataset contains 36k graph-structured tasks across 20 scenarios, achiev- ing a 91% % human acceptance rate. Training on our graph-structured data shows that it can more efficiently guide agents compared to manually annotated data. We conduct multidimensional evaluations for various open-source and closed- source models, revealing their performance across various capabilities and paving the way for fu- ture advancements, Our project is available at https:/ /omni bbrnhh tttb.ib..///
However, these two types of benchmarks mentioned above still have notable limitations: 1) Uncontrollable and fixed task complexity. Existing benchmarks typically propose tasks entirely rather than progressively with fine-grained guidance, which results in uncontrollable and fixed task complexity. Uncontrollable task complexity makes it hard to design fine-grained test data for various capabilities, while fixed complexity makes it challenging for benchmarks to keep up with agents' growing capabilities. 2) Extensive manual labor and limited task seenarios. The exist- ing benchmarks rely on manual annotations to synthesize demonstration trajectories or evaluation functions, making the cost of designing benchmarks unaffordable and hinder- ing the expansion of scale. Moreover, the annotated data with a limited amount is influenced by human prior expe- rience, making it difficult cover comprehensive scenar- ios. 3) Absence of multidimensional evaluation: Existing benchmarks commonly evaluate agents based on the final state of tasks, lacking an evaluation of the intermediate steps in task execution. Additionally, the various capabilities re- quired by virtual agents to complete tasks (e.g., planning, instruction understanding, etc.) cannot be quantified by coarse task success rates, failing to provide sufficient feed- back for potential future improvements. In summary, an ideal benchmark should include not only diverse task scenar- ios with controllable complexity, but also comprehensive evaluation across multiple dimensions.
1. Introduction
With the development of MLLMs (Fei et al., 2024c; Wu et al., 2024a), recent MLLM-based based virtual agents (Xu et al.,
'Equal contribution 'Zhejiang University, Hangzhou, China 2 AAt Group, Hangzhou, China "The Hong Kong University of Science and Technology, Hong Kong SAR. China "Najjing University, Nanjing, China. Correspondence to: Juncheng Li <junchengli@ @ zju.edu.en>>
cost-effectively construct diverse task scenarios with complexity at multiple granularities for comprehensive agent evaluation, we propose novel self- seff-generating, graph-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Scalable Multi- Dimensional Benchmark for Essential Virtual Agent Capabilities
OmniBench
Systematic Effective
Scalable Multidimensional
Controllable Task Complexity Across Five Dimensions
High-Quality Graph-Based Data
Dependieccy
Branch
Instruction Knowledgo Hierarchy
Task Complexity
Task Graph
Origedt propetigae Sill
Opes lccalti sunnbarr Inipecoryy
Exnt
Level
oo
Witth: 2
Prodesss
(Priged, imspossipioo Engeectrry
Bus dile FounFricd (rrgeei mogpost qpi"
U
(Dopttcc
Appect
Ghee lead Ba racge masl geetrelty
Lil Pawchh for Jhrien colusro
Medium Level
Task Instruction:
Effectiveness Graph- Structured Date
Croate a recer flneer ou Eddorurr ed anoun prpcospiis sund the lnno the prrsenter Thei copy content from the
d oo
was
Appuz
Douch
Creas Convent
Depoodeeccy deyondercy
exnd the Posseroiir the haccis Finll..
Donnoio Setritss
Task Inteitt:
Ansonisiie
Cheste a pernanal Intrdd untinn FowerFoltecte for Sinky
Level
Generalist
Task Trajectory:
ooooo
Knowledge
underntenden
3d
(utrgpecory Elngeetayy E. tagettory pocagn coneeeeerr Extrppttoyy
KDam 23 Mi11
Long Range Planning Hard Dependency Hard Hlerarchy
Parallel Planning 1 Hard Dependency & Hard Branch
Planning
Farialo
Planzing
Lovg Aawg
47
Plaming
Long Contevt
[mmniBench]
Decivon:
Moking
ceecsoo den Maniig
Seperotial
Deehaon Makking
Cress Domain
Automated Benchmark
Construction
Real-World Task Scenarios Office (ave
Cross-Domain Decision-Making dp Hard Knowledge & Hard Branch
23
54
Subtask Synthesis:
Long fequence
Instruction
Subtrsk Decogery
Domain-Specificc Knowledge
Rupponing
Tollowingg
Subtzuk
CDK
LRP
Traaetory
Screen Recovding
Cheativv Editing
Duee soare LM+
DSK
COOK
Covering Mainstream:
FAil Fune
Platforms
2. 2.Task Synthesis
CUl Ap (l
SDK
000
Trikk Compontion
Tusk Frach
Team Callaberation
Thei Inoruction
Deshtop
Mobile
Web
Graphe
Averago Coveraage Nate en Omniterch
Comprehensive Evaluation of 10 Agent Capabilities
Scalable and Cross-Platform Task Scenarios
Figure 1. Overview of OmniBench, a systematic benchmark with five-dimensional task complexity and bottom-up automatic task synthesis for generating structured task graphs. It evaluates ten virtual agent capabilities using high-qualit, graph-based based data, ensuring scalable and realistic task evaluation.
based benchmark, OmniBench. It dynamically synthe- sizes tasks with controllable complexity based on a bottom- up pipeline. OmniBench spans five fundamental types of task complexity to construct 10 evaluation dimensions (see Figure 1). Test tasks across these dimensions are catego- rized based on combinations of complexity types. For ex- ample, a long-range planning test task typically exhibits higher dependency complexity and hierarchy complexity. OmniBench consists of 36k high-quality graph-structured tasks across 20 distinct scenarios (e.g. image editing, video editing) derived from its self self-generating framework, with the task scale being 40x larger than most settirommnn--bassee benchmarks, as shown in Table 1. This automated process opens up the possibility of scaling up virtual agent evalua- tion in a low-resource manner. Therefore, OmniBench facil- itates the easy construction of agent benchmarks on desk- top, mobile, and web platforms, as shown in Table 1. For multidimensionall task synthesis, we take motivations from the DAG topology (Xu et al., 2024a) to design bottom- up pipeline. Specifically, we define five fundamental task complexities and synthesize tasks with controllable com- plexity by constraining the DAG composition process. Notably, we extracted task intents to guide this process to
avoid composing meaningless tasks (e.g., opening a food delivery app and immediately closing it). We further in- corporate quality control modules to optimize subtasks and ensure semantic alignment. In this way, we cost-effectively synthesize high-quality graph-structured tasks, significantly breadening task scenarios without requiring human an- notations.
Moreover, due to the inherent complexity of conducting comprehensive and fine-grained evaluations of agents, we propose graph-based multidimensional evaluation frame- work, Oal.Eval. In contrast to previous coarse- sarsee-rained evaluation methods, we introduce graph-based evaluator that leverages subtask-level evaluation functions synthe- sized in OmniBench. Specifically, we design two novel fine-grained metrics to evaluate agents' performance on graph-structured tasks and their alignment with human logic. Based on OmniBench, we comprehensively evaluate 12 virtual agents, including both open-source and proprictary models, across all 10 capability dimensions as shown in Figure 1, fully revealing the capability boundaries and providing concrete directions for future improvement.
The performance comparison between various models pro-
2

---

A Scalable Nulti-Dimensiona Benchmark for Essential Virtual Agent Capabilities
Table 1. Comparison of virtual agent benchmarks across environment, task, and evaluation dimensions. Unlike previous benchmarks, OmniBench features automatic task composition, dieennnnnio task complexity, and a 10-capability evaluation framework.
Rastruamet
Bralaaiin E
Internctive
Real Woeld
Mation
Instunce
Campl.
Dyna
Batent
Secuaris
Dam
Tra).
Construction
Instraction
Cap.
Erai
Evohsation Strategy
AnW (Rtviio at 2004))
30776
Morrcd Ammecaion
Tigh &e Low
Tixk
Vnjjavey hoort
MisdEWid elles
2350
Maruud
Vinjstooy-
McTIP (Burss ctol. 2002)
2002)
756
Masaal Aunctaaion
Higk Low
1
Tisk
Tujjstoyy hased
OmniACT et 2015)
9862
Marazd Asretazto
Low
link
Virpateey hased
OUI Odyseyy (Lu d., 2024)
7735
Mirzzá Ausctazion
Low
Tujjstory- beaed
Wedotrmm (Zho 2003)
512
Masual Aunitaiion
Low
Tisk
Resth
21.1)
05%ull (Xe al, 2024)
300
Massal Aunccaaiion
Low
Tisk
134
Redi bused
Spidet-- (Coo atal. 2001)
Mirrodl Ansicaation
High Low
OKa 20246)
100
Marsa Compoiiion
Low
Suhtikk
Gruph-based
OzzniBeack (Duzx)
O 36075
20
Auourati Composiion
Verfication
High &
10 Subtikk
Gruph-bascd
vides valuable findings for future virtual agent application. Specifically: 1) Existing agents struggle to handle graph- structured tasks. Compared to tasks with linear struc- tures, the agents fall significantly short when facing graph- structured tasks, with even GPT-44 achieving only 20.5% performance, while humans can reach 80.1%. 2) Task in- tents are crucial for task planning. Incorporating task intents into the prompt offers a plug-and-play improvement to planning performance, with an average increase from 23.4% to 28.9%. Similarly, using task intents in fine-tuning data improves planning performance from 30.5% to 31.9%. 3) Mainstream agents are sensitive to expression order in task instructions. We observe significant performance fluctuations in existing agents when altering the expression order of task instructions. In contrast, agents fine-tuned with graph-structue trajectories exhibit more stable per- formance.
for virtual agents are generally categorized into two types: trajectory- -based and result-based. Trajectory- bench- marks (Rawles et al., 2024; Cheng et al., 2024; Deng et al., 2024) compare agent trajectories to human demonstrations but can be inaccurate due to the existence of multiple valid trajectories. Result-based benchmarks (Xie et al., 2024; Zhou et al., 2023; Cao et al., 2024) focus on the final state of the environment, overlooking the fine-grained evalua- tion of intermediate processes. More recently, some studies (Shen et al., 2023; Xu et al., 2024a) have introduced graph- based evaluations, which support both multiple feasible trajectories and the evaluation of intermediate processes. TASKBENCH (Shen et al., 2023) evaluates agents for task automation, but its simplistic metrics fail to fully utilize the potential of graph structures. CRAB (Xu et al., 2024a) evaluates agents using handcrafted graphs, but it lacks sys- tematic task analysis, limiting fine-grained capability assess- ment. Notably, to the best of our knowledge, OmniBench is the only scalable benchmark for virtual agents that de- fines composable task complexity using graphs to evaluate multiple essential capabilities.
Furthermore, we fine-tune two open-source agents with distinct architectures on synthesized graph-structured task trajectories. As shown in Figure 1, both agents exhibit performance improvements on AndroidControl and Omni- Act. Compared to agents trained on manually annotated datasets, our agents achieve better performance across di- verse benchmarks, benefiting from reasoning-rich trajecto- ries that demonstrate their broad applicability and strong potential.
3. OmniBench
OmniBench consists of 36k high-quality graph-structured tasks across 10 evaluation dimensions to simulate the way humans perceive the digital world, including planning, decision-making, etc. In this section, we first introduce task graphs of Om- niBench and systematically define corresponding task complex- ities (Section 3. Then, we present the bottom-up data collec- tion pipeline for controllably synthesizing tasks (Section 3. 3.2). Additionally, we explain how to control the quality of the syn- thesized data (Section 3.3). Finally, we showcase the statistics of OmniBench (Section 3.4).
2. Related Work
Virtual Digital Agents. With the development of MLLMs (Pan et al., 2025b; 2024b; Li et al., 2023a; Fei et al., 2024a), virtual agents have greatly improved task automa- tion across platforms. CogAgent (Hong et al., 2024) intro- duced an 18B visual language model for GUI understanding, achieving offthe---rrretttttt performance. SeeClick (Cheng et al., 2024) developed vision-only model that interacts with GUIs via screenshots, eliminating the need for struc- tured data. UGround (Gou et al., 2024) proposed a universal grounding model, accurately mapping GUI elements across platforms. Iris (Ge et al., 2024) enhances GUI automa- tion by tackling challenges in complex digital environments. Evaluating these visual agents is crucial for real-world ap- plications.
3.1. Task Complexity on Task Graph
In this section, we define task complexity on graphs, which is later constrained in Section 4.2 to construct test tasks for multidi- mensional capability. We propose a new complexity definition because existing benchmarks (Wang et al., 2024a; Cao et al., 2024; Koh et al., 2024) typically define task complexity based the number of steps in human demonstrations. However, this approach has two limitations: 1) The inherent subjectivity of human demonstrations makes this definition unreliable; 2) It is one-dimensional and fails to capture the multifaceted complex-
Benchmarks for Virtual Agents. Mainstream benchmarks
3

---

A Scalable Multi- Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Subtask Discovery
Iterative Synthesis
Composition
Task Validation
Novel
Graph-based Task
Trajicurr
Compust
>>> D i JNEKERO
>>>>>>>>>>> pnrmection
36K
Task Instructions
295K
Demonstrations
Task Nice!
recevstert
AN
Evelazic
Sabtouk
Pool
Roa liteet Beoutiyy a'wa lpsper
Caroirnee
Goctimeny
Verified Taoks
Five
Complexities
10
Copabilities
Stage 1: Subtask Synthesis
Stage 2: Task Synthesis Figure 2. Bottom-up task synthesis pipeline
OmniBench
Table Complexity Dimensions and Their Corresponding Levels
Table 3. Ablation study evaluating the impact of each quality con-
Complexity Dimension
Calculation
Fasy
Medium
Hard
trol module on acceptability the synthesized tasks.
Dependency Complexity
Number Fdges
Cross Verilication
Intent Extraction
Consistency Validator
Human Acceptance
Instruction Complexity
Number of Nodes
2
3 3md
6
41.2%
Knowledge Complexity
Number of Application Categories
1
2-3 3
24
61. 61.2%
Hierachy Complexity
Depth
52
3 3-4
82.7%
Branch Complexity
Widh
22
5
86.5%
ity of real-world tasks. Inspired by previous works (Xu et al.,
90,7%
2024a; Shen et al., 2023) that represent tasks as graph structures, we introduced the concept of the task graph and systematically defined five fundamental task complexities on the task graph.
decompose tasks into task graphs. However, this process is typically based on uncontrollable MILMS or expensive manual efforts. Therefore, to effectively synthesize tasks with con- trollable complexity, we designed a bottom-up automated task synthesis pipeline, as shown in Figure 2.
Specifically, we define a subtask as a smaller, independent task that contributes completing a more complex task. Each subtask has input and output resources to constrain the depen- dencies between them. To formalize this, we assume each subtask as s and define task graph as g = {S, where S = {s1, (5.,,,,...... } is the collection of subtasks, and R is a set of relations {(... {b)} indicating that subtask Sb depends on subtask 8, when the output resources of sa match the input resources of sb.
Overview. The task synthesis pipeline we propose consists of four processes. First, we synthesize a series of simple subtask instructions from the explorable environment. Then, we itera- tively synthesize subtask trajectories and evaluation functions. Next, the subtasks are combined into a task bottom-up. Finally, we validate the semantics of the tasks.
Subtask Exploration. We designed an environment contain- ing 49 diverse applications, inspired by OSWorld (Xie et al., 2024), allowing advanced MILMS to thoroughly explore each application to propose diverse and achievable subtasks. During exploration, documennatiin and example subtasks for each appli- cation are provided to help synthesis. To accurately synthesize the dependencies between subtasks, we provide predefined resource list, which MLLMS use determine the input and output resources for each subtask. The implementation details can be found in Appendix B.1.
After representing the task graph, natural idea is to de- fine task complexity based on the topology of the task graph. We systematically analyzed five characteristics of the graph and designed a corresponding five-dimensional task complexity. Specifically: 1) Dependency Complexity. Since each edge in the task graph represents dependency between subtasks, we define dependency complexity based on the number of edges. 2) Instruction Complexity. The semantics of task instruc- tion are composed of all subtasks, with more subtasks leading to more complex instruction semantics. Therefore, we define instruction complexity based on the number of nodes. 3) Knowl- edge Complexity. We categorize all 49 applications and define knowledge complexity based on the number of applications from different categories (e.g.. multimedia playback. productiv- ity) in the task graph. The detailed categorization is provided in Appendix A.4. 4) Hierarchy Complexity. The depth of the task graph represents the number of hierarchical levels in the task structure. Thus, we define hierarchy complexity based on the depth. 5) Branch Complexity, A wider task graph indicates more branches that can be executed concurrently. Therefore, we define branch complexity based on the width. The classification criteria for three complexity levels are shown in Table 2.
Iterative Synthesis. We leverage advanced MILMS to syn- thesize trajectories and evaluation functions of subtasks. For trajectories, our crafted prompts guide the MILM to describe screenshots and output thoughts, improving inference on trajec- tories. For evaluation functions, we predefine 11 basic APIs to retrieve information such as clicked text, keyboard inputs, and file directory existence. Subsequently. we use Claude-3.5. Sonnet, which excels in the code domain, to compose these ba- sic APIs into evaluation functions for subtasks. To improve the quality of synthetic data, we propose a novel cross-verificttion algorithm that iteratively refines the synthesis process. The implementation details can be found in Appendix B.2.
Task Composition. For high-quality subtask samples that pass crossvverffication, we add them to the subtask pool. Directly bottom-up composing these subtasks into a task graph using in- put and output resources may result in tasks that lack a coherent core goal, such as "opening food delivery app and immediately
3.2. Controllable Task Synthesis
Although we define five fundamental task complexities on the task graph, converting task instructions into a graph remains challenging. A straightforward idea is to directly top-down

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
closing it". To avoid synthesizing such low-quality tasks, we extract task intents for the composition scenarios from the sub- task pool, such as "create personal introduction PowerPoint for Emily," as shown in Figure 1. Each task intent involves group of subtasks, which are then combined into a task graph using input and oulput resources. Since the bottom-up composition process is rule-based, The synthesis of the task graph control- lable, ensuring that tasks with controllable complexity can be synthesized. Implementation details are in Appendix B.3.
Statisties
Value
Total Tasks
36076 (100%)
dreeork-dependent Real-world Tasks
16614 (46.05%)
betwork-independent Local Tasks
19462 (53.95%)
Avg. Number of Used App Per Task
2.21
Task Instruction
Avg. Words of High-level Instruction
51.7
Avg. Words of I Low-level Instruction
237.9
Total Tusk Scenarios
20
Total Subtasks
255
Task Validation. For the task graphs constructed through com- position, we employ GPT-4o to summarize the task instruc- tion based on the subtask instructions and the graph structure. However, such synthesized instructions may deviate from the original semantics of the task graph, such as losing the graph's nonlinear semantics and degenerating into simple sequential task. To ensure the synthesis of high-quality graph stpp--ttuctured tasks, we designed consistency validator to verify the seman- tic alignment between the task graph and the summarized task instruction. Specifically, GPT-4o determine the dependency for subtasks solely bused on the task instruction. If the inferred de- pendencies match those in the task graph, the instruction passes validation; otherwise, the instruction needs to be re-summarized. Implementation details are in Appendix B.4.
Easy
Medium
Hard
Hierarchical
31.9%
47.3%
20.8%
Knowledge
31.4%
52.4%
16.3%
Instruction
21.0%
41.0%
38.0%
Branching
18.5%
46.7%
34.8%
Defendency
31.8%
32.2%
36.0%
0%
20%
40%
60%
80%
100%
Figure 3. Statistics of OmniBench
mensional evaluation framework. In this section, we first intro- duce graph-based evaluator with two novel metrics for fine- grained and diverse evaluation (Section 4.1). Then, we describe the construction of test tasks designed to evaluate 10 distinct capabilities by constraining task complexity (Section 4.2).
3.3. Quality Control
4.1. Graph-based Evaluator
Since the quality of graph-structured tasks is critical to the ac- curate evaluation of the virtual agents, we further introduce three designs to enhance the quality of synthesized data: a cross-verification mechanism, intent extraction module, and consistency validator. The cross-verificatio mechanism iter- atively optimizes the demonstration trajectories and evaluation functions of subtasks, the intent extraction module ensures that the tasks have coherent goals, and the consistency validator aligns the semantics the task graph and task instructions. We perform an ablation analysis of these three quality control meth- ods, validating their effectiveness, as shown in Table 3. For each ablation shown in the table, we sampled 400 task graphs and calculated the average acceptance by three specially trained annotators. The experiment shows that removing any quality control module decreases human acceptance, with the removal of the cross- vorssiverii algorithm resulting in the largest drop to 61.2%.
Currently, most benchmarks still evaluate agents in coarse- grained and unreasonable paradigm. Specifically, result-based evaluations (Xie et al., 2024; Zhou et al., 2023) consider whether the final environment state aligns with expectations, lacking fine- grained intermediate evaluation. As shown in Figure 4, although both trajectories ultimately fail to complete the task, the progress of trajectory is superior to trajectory 1, and they should not simply be categorized as failures. While trajectory-based eval- uations (Rawles et al., 2024; Lu et al., 2024) compare the agents' predicted actions to human demonstrations at each step, they overlook multiple feasible trajectories. As shown in Figure 4, both trajectories can accomplish the task, but trajectory is deemed a failure because it does not align with the demonstra- tion trajectory, which is unreasonable.
Considering the limitations of these two evaluation paradigms, we introduce a graph-based multi-metric evaluator inspired by previous research (Xu et al., 2024a), as shown in Figure 4. Specifically, we define three evaluation states for each node on the task graph: Completed, Evaluating, and Waiting. Initially, when the evaluator is set up, nodes with an in-degree of 0 are marked as Evaluating, while the remaining nodes are marked as Waiting, After the agent executes each action, checks whether all Evaluating nodes have been completed. Once node is completed, it is marked as Completed, and new Evaluating nodes are added according to the topological order. We set a maximum number of steps N, and if the agent does not complete any subtasks within N steps, the entire task is considered a failure.
3.4. OmniBench Statistics
OmniBench comprises a total of 36,076 task instances, spanning across 20 common interactive scenarios and involving 49 diverse applications, as illustrated in Appendix A.4. In Section 3.1, we introduced the five-dimensional complexity metrics for each task, with the distribution of different complexity levels for each dimension shown in Figure 3.
4. OmniEval
To comprehensively analyze the limited capabilities of existing agents, we further propose OmniEval, graph-based multidi-
5

---

A Scalable Multi- Multt-Dimensional Benchmark for Essential Virtual Agent Capabilities
Task Instruction: Using ChatGPT search online for how to cook turkey.
similarity between agents and humans. This metric is motivated by the observation that humans prefer to complete all possi- ble subtasks within an application before switching to another, unless necessary. It is computed as the ratio of the agent's Coherency Score (CS) to the maximum possible CS:
Result-based Evaluation
Expert-designed Evaluation Function
check, upt weaccliteguun Luf, meosgee mebetatal:
''becbend-an' in requaed Lirt:
message "How should cook turkey? and "searct" in metadata
return true
return halee
Coagent
Trajectory 1
2iss CPV "Glaee Brewsarr
B- Furctinttion
Chock Rosli
0.0
T.C = CSmaz
2te d
J
where CS quantifies the coherence of the subtask sequence. For each pair of adjacent subtasks (s;, (,+1) in sequence, CS increases by if both subtasks belong to the same application. CSagent is the coherence score calculated from the executing subtask sequence, and C'Smaa the maximum possible coher- ence score calculated among all topological sequences.
Trajectory
Olit "Sarch Mork
baley??
Theck Beart
0.0
0.7
Trajectory-based Evaluation
Sour hoold
Sogu
Human Domonstration
Trajectory
Search Mode
Trajectory
Loodi torkey
shoul!
Hop
Clii SSanch Maute
How Cive
Score
0.3
hut
10
Trajectory
Sto chl "Seerch Model"
Bew Chet Sore
10
1.0
4.2. Evaluation Strategy
Graph-based Evaluation
Currently, there is limited discussion on the categorization of capabilities in the virtual agent field. The previous mainstream
toput turbey?
Graph Evaliuaton Completed Evalusting Waiting
Subtosk mesaag ChatGPT
classifications (Lin et al., 2024; Gou et al., 2024) typically di- vided the capabilities into two simple categories: grounding and planning. However, this simple classification is quite coarse and does not take into account other essential and fine- tne-ggriied capabilities for agents, such as decision- decision-making and instruction understanding. To address this, we propose 10 fine-grained capabilities, derived from five categories that we consider essen- tial, with each category contributing two capabilities for agents. Specific test tasks are constructed for each capability based on the combination of five complexity dimensions, as shown in Figure 1.
Subtosk 2: Enahie the Search
footure.
Score 0.7
Trajectory
Ully Search
0.7
Trajectory
2
Jtep3
Figure 4. Comparison of mainstream virtual agent evaluation strategics with the evaluation strategy propose.
Additionally, to fully leverage the potential of the graph-based evaluator, we have designed two novel graph-based metrics. Traditional metrics fail to evaluate intermediate processes and alignment with human operational logic. For example, common metrics such as Success Rate (SR) (Xie et al., 2024) focus on task outcomes rather than the process, while Action Match Score (AMS) (Li et al., 2020b) treats action sequences strings and compares the similarity between human demonstrations and agent predictions, rather than their logical similarity. To comprehensively quantify agent performance on the task graph, we propose two novel metrics inspired by its topology. The Coverage Rate (CR) assesses agent progress on the task graph, while the Logical Consistency (L.) reflects the similarity in operational logic between agents and humans.
Taking long-range planning capability as an example, we cate- gorize tasks with higher dependency complexity and hierarchy complexity as test tasks for this capability. This is because higher dependency complexity means the task involves more de- pendencies, requiring stronger planning capability. Meanwhile, higher hierarchy complexity indicates the task has deeper lev- els, which places higher demands on long-sequence processing capability. Therefore, we select tasks with dependency complex- ity and hierarchy complexity at the hard level as test tasks for long-sequence reasoning capability. For the test tasks of these 10 capabilities, we engaged professionally trained annotators to filter and construct high-quality test data. The specific defini- tions and corresponding explanations for the other 9 capabilities can be found in Appendix D.1.
Coverage Rate (CR). It evaluates an agent's progress on a task graph by weighting subtasks based on their depth, where deeper subtasks are assigned higher weights due to their increased num- ber of prerequisite subtasks. Referring to the relevant definitions in Section 3.1, let d(s;)) denote the depth of subtask st. The weight w(x:) is:
5. Experiments
In this section, we first introduce the experimental setup (Sec- tion 5.1). Then, we comprehensively compare the differences in capabilities across various models on OmniBench, along with several key findings (Section 5.2). Finally, we provide an in- depth analysis of the reasons behind the poor performance and methods for performance improvement (Section 5.3).
d(si)
w(s:)=
Σラ= d(s,))
The Coverage Rate is then:
CR = Σx w(ss)) I((s))
E w(xs)
where T(s;) = 1 if subtask is completed, and O otherwise. This metric emphasizes deeper, more complex subtasks, provid- ing a refined measure of agent performance.
5.1. Experimental Setup
Settings. We evaluate various models including MLLMs and Virtual Agents on OmniBench. For all virtual agents, we use the default prompt provided by each agent for inference, if available.
Logical Consistency (L.C). quantifies the operational logic
6

---

A Scalable Multi-Dimension Benchmark for Essential Virtual Agent Capabilities
Table 4. Performance of models on OmniBench. For each capability, we use the CR metric on test tasks for quantification. Abbreviations adopted: PP for Parallel Planning: LRP for Long Range Planning: CDDK for Cross-Domain Decision-Makinge SDK for Sequential Crrision-Maaing. SI for Subtask Identfiicatin;; DI for Dependency Identification; LSR for Long Sequence Reasoning; LIF for Long Instruction Following: DSK for Domain-Specific Knowledge; CDK for Pross-Domain Knowledge. An asterisk (*) indicates that the agent uses GPT-4o as the planner.
Instruction Understanding
Generalist Knowledge
Overall
Planning
Decision-making
Long Context
CR
LC
PP
LRP
CDDK
SDK
SI
DI
LSR
LIF
DSK
CDK
Human
80.1
92.8
80.1
76.9
91.9
93.0
69.1
72.1
79.5
66.1
89.4
71.5
Open-source Multimodal Large Language Models (AI/Y+ Screenshho))
Qwen2-VI. VL-7B (Wang et al., 2024b)
14.8
9.0
15.5
13.5
16.5
17.8
14.1
13.8
14.7
12.4
15.8
13.8
IntemVL2-88 (Chen ct al., 2024)
14.2
13.0
15.0
13.5
16.2
16.9
12.1
12.9
15.8
11.7
15.8
12.4
InternVL2.5-8P (Chen et al., 2024)
17.4
18.8
18.2
16.7
19.6
21.5
16.4
15,3
15.8
15.4
19.0
16.3
Closed-soure
Multimodal Large Language Models (AI]Y+Screenshen))
Vree-VV-Maxx (Bai al., 2023)
18.4
23.3
18.7
19.4
19.6
23.3
15.0
16.7
18.3
16.1
19.6
17.3
Gemini-2.0-Flast
25.9
38.0
24.8
24.6
31.5
33.2
22.5
22.5
25.7
21.9
27.8
24.8
Claude-3.5-Somne GPT-4o (Hurst et al., 2024)
27.6
35.0
30.5
24.7
32.0
31.3
24.5
25.0
26.6
23.5
33.4
24.5
38.7
49.0
38.4
37,8
43.2
49.4
30.6
35,5
42.7
32.2
43,2
34.2
Visual
Digital Agents (Screenshor)
Aguvis-7B (Xu et al., 2024b)
22.9
27.1
21.2
23.5
25.5
28.1
20.2
20.0
22.8
20.1
26.3
21.6
OS-Atlas-Pro-1B (Wu et al., 2024b)
19.1
23.9
20.6
17.6
22.9
23.6
15.0
17.7
18.7
15.9
22.0
16.8
ShowU1-2B' (Lin et al., 2024)
23.2
24.6
23.2
23.1
26.3
26.6
21.5
20.3
24.7
20.4
24.8
20.7
OS-Atlas AS-AAtas-Base-4B BBs (Wu et al., 2024b)
22.2
23.8
23.2
21.9
26.2
25.6
19.4
19.5
23.5
20.0
23.4
19.3
UGround-VI-TE (Gou et al., 2024)
25.0
26.3
25.7
25.1
30.6
31.4
21.5
21.3
24.8
21.3
27.2
21.5
Supervised Tiniin Agents (Screenshot)
Omni-OS-Atlas- AtS--ell--Base-BB (Ours)
29.7
30.1
24.2
33.0
34.9
35.3
28.7
24.2
27.9
26.5
33.8
28.2
Omni-UGround UG-ound-VV--7 (Ours)
34.4
37.4
33.2
31.3
43.1
42.4
21.9
35.0
40.3
31.7
36.7
27.6
If models do not provide prompts for agent tasks, we use a unified prompt designed by us. We also report results trained on OmniBench data for some selected models. All experiments are conducted with NVIDIA A100 80G GPUs.
Capability Boundaries of Mainstream Agents. As detailed
in Table 4, although advanced agents such as GPT-4o and our supervised fine-tuning models (e.g..
demon- strate strong performance in overall metrics and in capabilities like planning and decision decision-making, clear limitations remain in Subtask Identification (SI) and Long Instruction Following (LIF). Specifically, even the strongest models only achieve 30,6 (GPT-40) and 21.9 (Omni-UGround) in SI, and 32.2 and 31.7 respectively in LIF, which are significantly lower than the hu- man baselines of 69.1 and 66.1. as shown in Figure These results highlight persistent difficulty in decomposing complex instructions and maintaining coherence over extended task flows. Compared to their performance in other capabilities, the results suggest that instruction understanding in long and semantically complex contexts remains a key bottleneck for current agents. Future improvements in agent performance will likely depend on more robust handling of multi-step semantics and long-context alignment.
Baselines. We conduct comprehensive evaluation of the four types of models as shown in Table 4. The specific details about the baselines can be found in Appendix D.2.2.
5.2. Main Results
Alignment Between OmniFyal and Human Evaluation. Be- fore delving deeper into the concrete evaluation results, we first compare the alignment between OmniEval and human evalu- ation for agent tasks. Specifically, we randomly sampled 50 trajectories from all models to calculate the correlation between OnnilEall and human evaluation. Each trajectory was scored by two specially trained annotators, who referenced the task in- structions to assign task completion scores and logical alignment scores from (0%, 10%, ..., 90%, 100%). }. The final human evalu- ation score was determined by averaging the scores given by the two annotators. In Figure 5, we present the Pearson correlation between OvaiEval and human evaluation. The results indicate
70.0
19uman Danaline
00.0
30.0
Coverage Rato and Human Evaluation
Logical Consistoncy and Hum on Evaluation
Long Etentricaiion
222
SL.
Covergg fun
8
Loycul Conciziona
20.0
24.53.5
22.21.2
21.32u4
8
-0
20.0
26415
10
10
0.
Gemani 2
ceeen Mar
20
10
Dlede
Figure 5. Correlation between Coverage Rate and Logical Consis-
tency with Human Evaluation.
Figure 6. Comparison of model performance on Subtask Identifi-
cation (SI) and Long Instruction Following (LIF).
7

---

A Scalable multi-Dimensiona Benchmark for Essential Virtual Agent Capabilities
Table 5. We compare the performance of different virtual agents on tasks with varying complexity levels. Medium and Hard levels show performance drops compared to the previous level, with downward arrows indicating the magnitude of decline.
Dependency Comp.
Branch Comp. I
Instruction Comp.
Knowledge Comp.
Hierarchy Comp. T
Agents
Easy
Medium
Hard
Easy
Medium
Hard
Easy
Mediam
Hard
Easy
Medium
Hard
Easy
Medium
Hard
Aguvis-7B (Xu et al., 2024b)
32.8
27.6 152 24.3 133
41.2
3688 ua 30.6 ua
49.5
369 1126 25.3 116
38.4
32.5 159 27.6 Lto
37.9
33.6 L43
29.7 159
OS-Atlas-too-78 (Wa at. 2024b)
32.3
26.8 155 23.7
39.1
31.0
25,4 i56
44.3
34.8 1s5 21.8
u30
33.9
28.4
24.3
34.5
28.1
164
25.6
ShowUU-22B' (Lin et al., 2024>
34.0
157
25.6
13
41.3
32.7
P6 28.2 u1
45.9
36.6 19 25.4
378
32.6
37.6
32.0
(56
(S-AAtas-Base-44 (Wu al., 2024b)
32.7
29.1
13 24.9
142
35.2
324
1a
27.6
482
37.5
26.7
108
39.1
34.2
p40
28.9
153
43.1
38.4
33.2 isa
UGround-7B (Gou et al., 2024)
34.1
30.0 p1 27.1
(29
44.6
38.3
32.4
uss
53.0
39.4
27.2
42.3
36.
159
32.6
(18
35.7
28.8
25.5
Challenges in Handling Graph-structured Tasks, We com- pared the performance differences of each model on chain- structured tasks and influence of other factors, we utilized OmniBench's 's controllable task synthesis mechanism construct a set of strin-structured tasks, each with the same number of nodes and edges. All tasks belong to the same knowledge domain and share the same level of knowledge complexity. As shown in Figure 7, GPT-4o (with AIlY), the most advanced agent, achieves only 20.5% accuracy on graph-structured tasks, far below the human performance 80. 10.1% This phenomenon can be attributed to the fact that most existing agents are predomi- nantly fine-tuned on chain-structured tasks, which may result in their tendency to interpret graph-structured tasks as linear. Such misinterpretation can significantly impair the agents' capabil- ity to accurately identify the dependency relationships between subtasks, ultimately leading to task execution failures.
grouped by complexity levels: Easy, Medium, and Hard. Un- surprisingly, all agents exhibit significant performance drops as task complexity increases, with an average decrease of 6.19 points. This trend is consistent across all five dimensions. Such systematic degradation in harder cases confirms OmniBench's ss effectiveness in scaling task difficulty. Furthermore, perfor- mance on hard tasks may serve as a more accurate indicator of an agent's expert capabilities than average scores, revealing the upper bounds of its potential.
tasks. To eliminate the
and
Effectiveness of Graph-Struccured Task Trajectories. We follow the training details provided in the OS-Atlas paper and adopt the same experimental setup to train our backbone mod- els: OS- 4t and UGround-77 -..V.. As shown in Table 6 and Table 7, while agents such as OS-Atlas and UGround, which are pretrained on GUI grounding tasks, exhibit strong GUI understanding capabilities, their limited planning capability hinders their performance in complex action reasoning. In con- trast, the high-quality multi-step navigation dataset synthesized by OmniBench significantly enhances the model's capability to make decisions regarding action types, thereby improving the success rate in GUI navigation. Specifically, Omni-OS- Atlas-4B achieves an average success rate improvement of 0.46 points on AndroidControl and 0.73 points on OmniAct, while achieves improvements of 0.4 points on AndroidControl and 0.3 points on OmniAct.
41.
40
35.2
25.1
30.1
31.1
284
286
26.7
20
19.1
19.5
20.5
86
10.6
115
16.
13
Gwnnd An testeett
NetemVL2- 58
lsternVLZ 4-00
denutii D-Rass)
Claudin Stmmet
GPT 440
Agunts-7
OS- Pro-78
Showll 28
05-8ttes Maie 48
Uorannd-V 17
Sensitivity to Expression Order in Task Instructions. We define the impact of textual order on the model as its instruction sensitivity, conducting experiments with standard deviation as the metric. We construct 10 specially designed test tasks, cach associated with three task instructions that are semantically iden- tical (based on the same task graph) but differ in textual order. As shown in Table 8, the original MIJ tend to be less sensi- tive to instruction variations, but perform poorly overall. Though fine-tuning them on navigation tasks enhances the performance, it also compromises the models' robustness to instructions. OS- Atlas-Pro and Aguvis exhibit significantly higher sensitivity, with an average increase of 8.21 points. Moreover, after incor- porating graph-structures task samples from OmniBench into fine-tuning, the models' performance is further improved while largely preserving their robustness. Omni-OS-Atlaa and Omni- Aguvis exhibit reduced sensitivity, with an average reduction of 7.91 points. This indicates that the diverse and structured task trajectories from OmniBench can help models better rec- 85.53
Chain-structured
Graph-structured Tasks
Figure 7. Performance comparison of various models on chain-
structured tasks and graph-structured tasks.
Table 6. Evaluation results on AndroidControl benchmark. Bold
values indicate the best performance across all baselines.
Models
Type
Androdt miimi-1oor
Grouding
SR
Type
Andraid High
Grounding
ImemVL-2-48 (Chen et al. 2024)
90.94
$4.05
20.10
$4.09
72:75
66.72
Qwen2-V1--1 (Wing.et et al, 2024h)
91.94
56.50
82.56
$3.63
77.68
69.72
Saellick (Ching 2004)
93.00
73.42
75.00
$2.94
62.57
59.11
OS-Alla-4B (Wu al. 2024b)
53.76
30.64
73.79
67.54
Ulmand-79. (o-und----V1-
83.00
78.55
Ommi-OS-Allas AR (Qus))
92.49
$3.51
81.38
$4.86
73.11
67.71
Omai--GGaandd7B. 7B-V (Duss
92.37
57.24
83.57
$4,89
78.97
7033
Table Evaluation results on OmniAct benchmark. Bold values
indicate the best performance actoss all baselines.
Mudels
Type
OmniAAttWWib
Grounding
SR
Pype
ChmniAct-Desktop
Grounding
SR
ImemVL-7-D (Chen al. 2014
47.51
$1.34
19
61.00)
44.47
29:0
Qwenl-V1-57 oW al. 2024h)
55:94
38.58
94.52
SocClick (Cheng 2024)
86.98
75.48
68.59
96.79
70.22
72.59
QS-Atha-4B (Wa et al. 2024b)
83.56
$2.00
75.91
96.51
$5.53
86.78
(Kmmmd-7B-VI (Gin et all, 2004)
90.16
$6.08
10.85
97.13
94.99
91.89
Ommi-OS-Alia--8 (Qurs)
89.96
$2.74
74.62
97.64
86.37
Omni- Cronnd- 7B-VI (Ours)
91.24
$7.35
80.24
97.93
95.21
92.10
ognize complex dependencies embedded in task instructions, improving their overall stability and performance.
5.3. In-Depth Analysis
Performance Differences Across Complexity Levels. As
The Effect of Task Intent Planning, We design two exper-
shown in Table 5, we analyze agent performance across tasks
8

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Table 8. Average sensitivity across different models.
save file operation requested in the image editing instruction. 2) Lack of Knowledge. We find that 21% of the failures are caused by the agent's lack of knowledge about the target application, such as being unfamiliar with how to create a reference list in Zotero. 3) Environmental Error. We observe that 3% of the failures result from environmental interference, such as network delays. 4) Grounding Error. We find that 17% of the failures are due to the model's lack of grounding ability,, meaning the agent knows the target to click next but locates it in the wrong position. 5) Hallucina- tory Success. Finally, 36% of the failures occur when the agent incorrectly assumes the task is complete, which may stem from its weak contextual memory capabilities. The distribution of these errors is shown in Figure
Models
Backbone
Avg. Sensitivity J
Human
1.95
VntemVL.2-4B (Chen et al., 2024)
InternV1.2-4B
2.97
OS-Atlas-Pro (Wu et al. 2024b)
IntemVLL--4B
9.07 tt
Omni-OS- ASlas (Ours)
V1temVV12--4B
3.49 15.55
Qwen2-VL (Wang et al., 2024b)
Qwen2-VL-
2.58
Aguvis (Xu et al., 20124b
Qwen2-V1 V1--BB
12,9 100.0
Omni--guuvs (Ours)
Qwen2-VL- Wwen2-VL-7B
2.67 .10.23
iments to explore the applicability of task intent to both open- source and closed-source models. 1) For open-source mod- els, we conduct a comparative experiment using two separate datasets to fine-tune OS-Atlas-Bel Atlas-Base-4B and UGround-Y (GGround-V17B. One dataset includes task intent, while the other does not, As shown in Table 9, incorporating task intent in the training data significantly improves the model's planning performance on OmniBench. Specifically, OS-Atlas- AS-Ataa--asse44 improves its over- all planning score from 28.6 to 30.3, an increase of 1.7 points, while UGround-V/ BrrbounddV--7B- V-7B improves from 32.3 to 33.5, gaining 1.2 points. 2) For closed-source models, we use Qwen- VL-Max, Gemini-2.0-Faass Claude-3.5-Sommen and GPT-4o planners, with JGround-V1-77 serving as the grounding models. As shown in Table 9, all closed-source models exhibit improved planning performance with the inclusion of task intent. GPT-4o shows the most significant improvement, with its overall score rising from 25.4 to 34.3, gain of 8.9 points. Mlaude-3.5-Somm increases by 5.4 points, (emmini-2.0-Flash by 4.9 points, and Owen-VL-Mar by 2.6 points. This indicates that closed-source models can enhance their planning through this plug-and and--and approach.
Overview
Model-wise
Eitooraction Misundentarding
30%
Bitck of Knswiedss
Envirommnnaa Bror Grounddin
Agusts
Oc O6-Arandiv
13%
Hallucinatory Success sre to ande
Environmental Error
Instruction Misunderstonding
(137 307)
Peaschhly anc on aremm to fire Unatteg tim uott my borty.
'ujvo'''
Plass rutry... X
No probion, /walkr fare know cren' thtt you'se goin to ha home.
Step 2: ciit bnighinnu
Stay in tyse 3/'
Grounding Error
Lock of Knowledge
clis riser
Peoze Jnly
The task is fieiiied X
Geccibench faite.
andkr secords
/oggst to youe
problers doable-
the Insepee
Figure 8. Distribution of five major errors in 100 failure instances for each model. An asterisk (*) indicates that the agent uses GPT- 4o as the planner.
Table The effect of task intent on the planning of open-source versus closed-soure models.
6. Conclusion
Models
Parallel Planning 7 Long-Range Range Planning
Overall 1
In conclusion, we introduced OmniBench, a graph-based benchmark that addresses the limitations of existing eval- uation frameworks by enabling controllable task complex- ity through automated subtask composition. Along with OmniEval, a multidimensional evaluation framework, we evaluate virtual agents across 10 capabilities. Our results show that training on this data improves agent generaliza- tion, and our evaluations provide valuable insights into the strengths and areas for improvement in MLLM-based vir- tual agents.
Open-soun yource Maltimodal Large Langwaye Modelf
Chmmi-O8-Arlas 418-Attt--Bse-448
24.2
33.0
28.6
intent cuning
25.7 1.
34.9
30.3
Omni- UGrrondd-V VI-7B
33.2
31.3
32.3
tuning
31.1 11.2
32.6 113
33.5
Cloved-soouree Multimodal Lorge Language Medels
Qwen-VL-Max
21.9
20,8
21,4
intent prompt
24.5 p24
23.5
24.
Gemini O-Flash
23.1
22.7
22.9
intent prompl
153
140
149
Chande-355-Sene
24.2
23.7
24.0
intent prompt
30.6 1
28.1 114
151
GPT-40
25.7
25.1
25.4
intent prompt
32.9
35.7 1106
34.3
Acknowledgement. This work was supported by the NSFC (62272411). the Fundamental Research Funds for the Cen- tral Universities (226-2025-00017) the Key R&D Projects in Zhejiang Province (No. 2024C01 106, 2025C01030), Ningbo Yongjiang Talent Introduction Programme (2024A- 401-G), the Zhejiang NSF (LRG25F000001), Ant Group.
Failure Analysis. In this section, we delve into the analysis of eιrors encountered during the OmniBench evaluation. This analysis aims not only to identify the current shortcom- ings of the agents but also to inform future improvements in their design and training. We carefully examine 100 randomly sampled error instances for each model from the OmniBench evaluation. These instances are analyzed by expert annotators who identify the root causes of mispredic- tions based on their knowledge. Specifically, there are five types of errors: 1) Instruction Understanding. We observe that 23% of the failures are due to the agent's 's misunderstand- ing of the instructions. For example, it overlooks the final
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
References
agents: A survey autonomous agents across digital platforms. arXiv preprint arXi::241 2024a.
Bai, J.. Bai, S., Yang, S., Wang. S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Gao, M., Li, J., Fei, H., Pang, L.., Ji, W., Wang, G., Lv, Z., Zhang, W., Tang, S., and Zhuang, Y. De-fine: Decomposing and refining visual programs with auto- feedback. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, pp. 7649 7649-7657, New York, NY, USA, 2024b. Association for Com- puting Machinery. ISBN 9798400706868. doi: 10. 1145/3664647.36610666 1082. URL https: htt//ddooorgggo 10. 08.1145/366 00444//364444 001144444..688811 3681082.
Burns, A., Arsan, D., Agrawal, S., Kumar, R., Saenko, K., and Plummer, B. A. A dataset for interactive vision- language navigation with unknown command feasibility. In European Conference on Computer Vision, pp. 312- 328. Springer, 2022.
Cao, R., Lei, F., Wu, H., Chen, J., Fu, Y., Gao, H., Xiong, X., Zhang, H., Mao, Y., Hu, W., et al. Spider2-v: How far are multimodal agents from automating data science and en- gineering workflows? arXiv preprint N.di::2407.199555 2024.
Gao, M., Liu, X., Yue, Z., Wu, Y., Chen, S., Li, J., Tang, S., Wu, F., Chua, T.S., and Zhuang, Y. Benchmark ing multimodal cot reward model stepwise by visual program, 2025. URL https: http:::/aaxvv.oooo//// abs/ 06606.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M. Zhang, Q. Zhu, X., Lu, L., et al. Internvi: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24185-24198, 2024.
Ge, Z., Li, J., Pang, X., Gao, M., Pan, K., Lin, W., Fei, H., Zhang, W., Tang, S., and Zhuang, Iris: Breaking gui complexity with adaptive focus and self-refining. arXiv preprint arXiv:2412. 10342, 2024.
Gou, B., Wang, R., Zheng, B., Xie, Y., Chang, C., Shu, Y.. Sun, H., and Su, Y. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243 2024.
Cheng. K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint Xiv::2401 10935, 2024.
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards general- ist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Hong, W., Wang, W., Lv, Q. Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
Fei, H., Wu, S., Ji, W., Zhang, H., and Chua, T.-S. Dysen- vdm: Empowering dynamics- aware text-to-video diffu- sion with llms. In Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 7641-7653, 2024a.
Hu, S., Ouyang, M., Gao, D., and Shou, M. The dawn of gui agent: A preliminary case study with claude 3.5 computer use. arXiv preprint 2rXiv:2411.1033
2024. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card, arXiv preprint arXiv:2410.21276 2024.
Fei, H., Wu, S., Ji, W., Zhang, H., Zhang, M. Lee, M.- L., and Hsu, W. ofleo-of-thoughhe Step-by-step video reasoning from perception to cognition. In Proceedings of the International Conference on Machine Learning, 2024b.
Kapoor, R., Butala, Y. P., Russak, M., Koh, J. Y., Kamble, K., AlShikh, W., and Salakhutdinov, R. Omniact: dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pp. 161-178. Springer, 2025.
Fei, H., Wu, S., Zhang, H., Chua, T.-S., and Yan, S. Vit- ron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. 2024c.
Fei, H., Wu, S., Zhang, M., Zhang, M., Chua, T.-S., and Yan, Enhancing languale representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024d.
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents on re- alistic visual web tasks, arXiv preprint arXiv:2401.13649 2024.
Gao, M., Bu, W., Miao, B., WU, Y., Li, Y., Li, J., Tang, S.,
Wu, Q., Zhuang, Y., and Wang, M. Generalist virtual
10

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Li, J., Wang, X., Tang, S., Shi, H., Wu, F., Zhuang. Y., and Wang, W. Y. Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22122- 2020a.
Pan, K., Li, J., Wang, W., Fei, H., Song, H., Ji, W., Lin, J., Liu, X., Chua, T.-S., and Tang, S. 13: ntent-i ntrospec- tive retrieval conditioned on nstructions. In Proceedings of the 47th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pp. 1839-1849, 2024b.
Li, J., He, X., Wei, L.. Qian, L., Zhu, L., Xie, L., Zhuang, Y. Tian, Q. and Tang, S. gene-graiedd semantically aligned vision asiio-languas language pre-training. Advances in neural information processing systems, 35:7290-7303, 2022.
Pan, K., Tang, S., Li, J., Fan, Z., Chow, W., Yan, S., Chua, TT-SS, Zhuang, Y., and Zhang, H. Auto-encoding morph-tokens for multimodal 1lm. arXiv preprint arXiv:2405.01926, 2024c.
Li, J., Pan, K., Ge, 7.,, Gao, M., Ji, W., Zhang, W., Chua, T.S.., Tang, S., Zhang, H., and Zhuang, Y. Fine-tuning multimodal Ilms to follow zero-shot demonstrative in- structions. In The Twelfth International Conference on Learning Representations, 2023a.
Pan, K.,, Bu, W., Wu, Y., Wu, Y., Shen, K.,, Li, Y., Zhao, H., Li, J., Tang, S., and Zhuang, Y. Focusdiff: Advancing fine- grained text-image alignment for autoregressive visual generation through rl, 2025a. URL https: //arii//.xivv.v//i org/abs hts/abs/25000 2500. 05501.
Li, J., Tang. S., Zhu, L., Zhang. W., Yang. Y., Chua, T.-S., Wu, F., and Zhuang, Y. Variational cross-graph reasoning and adaptive structured semantics learning for composi- tional temporal grounding. IEEE Transactions Pattern Analysis and Machine Intelligence, 15(100:12601-12611) 2023b. doi: 10.1 10110//AMM.2223332771111
Pan, K., Lin, W., Yue, Z., Ao, T., Jia, L.,, Zhao, W., Li, J., Tang, S., and Zhang, H. Generative multimodal pre- training with discrete diffusion timestep tokens. arXiv preprint arXiv:2504. 14666, 2025b.
Li, Y., He, J., Zhou, X., Zhang, Y., and Baldridge, J. Map- ping natural language instructions to mobile action sequences. arXiv preprint arXiv:2005.03776, 2020b.
Pan, K., Wu, Y., Bu, W., Shen, K., Li, J., Wang, Y., Li, Y., Tang, S., Xiao, J., Wu, F., et al. Unlocking aha mo- ments via reinforcement learning: Advancing collabora- tive visual comprehension and generation. arXiv preprint arXiv: 01480, 2025c.
Lin, K. Q. Li, L., Gao, D., Yang, Z.,, Wu, S., Bai, Z Lei, W., Wang. L., and Shou, M. Z. Showui: One vision-language-actionn model for gui visual agent. arXiv preprint arxiv: arXiv:2411.17445 2024.
Rawles, C., Li, A., Rodriguez, D., Riva, O., and Lillicrap. T. Androidinthewild: A large-scale dataset for android de- vice control. Advances in Neural Information Processing Systems, 36, 2024.
Lu, Q. Shao, W., Liu, Z., Meng, F., Li, B., Chen, B., Huang, S., Zhang, K., Qiao, Y., and Luo, P. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint
2024. Miao, B., Zhang, W., Li, J.. Tang, S., Li, Z., Shi, H., Xiao, J., and Zhuang, Y. Radar: Robust two-stage modality- incomplete industrial anomaly detection, 2024. URL https:// / aariiiiii.. org/ abs/241 01737.
Shen, J., Jain, A., Xiao, Z., Amlekar, I., Hadji, M., Podolny, A., and Talwalkar, A. Scribeagent: Towards specialized web agents using production-scall workflow data. arXiv preprint 2rX.::241..15004, 2024.
Shen, Y., Song, K. Tan, X., Zhang, W., Ren, K., Yuan, S., Lu, W., Li, D., and Zhuang, Y. Taskbench: Benchmarking large language models for task automation. arXiv preprint arXiv:2311.1876/ 2023.
Miao, B., Wu, Y., Gao, M., Yu, Q., Bu, W., Zhang, W., Li, Y., Tang, S., Chua, T.-S., and Li, J. Boosting virtual agent learning and reasoning: A step-wise, minii-dimensional, and generalist reward model with benchmark, 2025. URL https: //axxvvvv.vv//////////a///aaaavl org/ abs///b////// 2503. 18665.
Wang. L., Deng, Y., Zha, Y., Mao, G., Wang, Q., Min, T., Chen, W., and Chen, S. Mobileagentbench: An efficient and user-friendly benchmark for mobile Ilm agents. arXiv preprint arXiv:2406.08/
Pan, K., Li, J., Song, H., Lin, J., Liu, X., and Tang, S. Self-supervised meta-prompt learning with meta-gradient regularization for few-shot generalization. arXiv preprint S.KV::2303.131144 2023.
2024a.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing lanionnlanguage model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12/9/, 2024b.
Pan, K., Fan, Z., Li, J., Yu, Q. Fei, H., Tang, S., Hong, R., Zhang, H., and Sun, Q. Towards unified multimodal edit- ing with enhanced knowledge collaboration. Advances in Neural Information Processing Systems, 37:1 10290- 1 110314 14 2024a.

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. NExT- GPT: Any-to-any multimodal LLM. In Proceedings of the International Conference on Machine Learning, pp. 98366-53399 2024a.
Wu, Z., Wu, Z., Xu, F,, Wang. Y., Sun, Q.. Jia, C., Cheng, K., Ding, Z., Chen, L.,, Liang, P. P., al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.232/11 2024b.
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. Os- world: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024.
Xu, T, Chen, L., Wu, D.-J., Chen, Y., Zhang, Z., Yao, X., Xie, Z., Chen, Y., Liu. S., Qian, B., et al. Crab: Cross- environment agent benchmark for multimodal language model agents, arXiv preprint arXiv:2407.01511, 2024a.
Xu, Wang, Z., Wang. J., Lu, D., Xie, T., Saha, A., Sahoo, D., Yu, T., and Xiong, C. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprinl a12::2241..0445,, 2024b.
Yao, S., Chen, H., Yang, J., and Narasimhan, Web- shop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Informa- tion Processing Systems, 35:20744-20757, 2022.
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. Weharena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13885 2023.
12

---

Table of Contents in Appendix
A Environment Setup
14
A.1 Environment Infrastructure
14
A.2 Observation Space
14
A.3 Action Space
14
A.4 Categorization of Applications
14
A.5 Definition of Scenarios
14
B Data Collection
14
B.1 Subtask Discovery
15
B.2 Iterative Synthesis
15
B.3 Task Composition
15
B.4 Task Validation
16
C Details of OmniBench
16
C.1 Subtask Metadata
16
C.2 Subtask Trajectory
17
C.3 Subtask Evaluation
17
C.4 Task Metadata
18
C.5 Task Trajectory
18
C.6 Example of Task Graph
19
D Details of OmniEval
19
D.I Capability Design
19
D.2 Experiment Setup
19
D.3 Evaluation Details
21
E Prompts..
21
E.1 Prompt for Subtask Trajectory Synthesis
21
E.2 Prompt for Subtask Evaluation Synthesis
23
13

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
A. Environment Setup
Table 10. Summary of action types in the desktop environment of OmniBench.
OmniBench conducts agent evaluations across three cate- gories of virtual environments: Desktop, Mobile, and Web. Its automated data collection pipeline makes it easy to ex- tend the benchmark to additional environments with min- imal effort. In the following section, we use the desktop environment as a case study to illustrate the environment design in OmniBench in more detail.
Action
Description
click.inpat
Simulazes mouse chicks on UI contrll elemenss Supponts configurable mouse buttens (lef, right, mid.le, a) and cas peróom both single and double clicks Commonty used for selecting sens, activuting contuls. oreninng foller.
wheel 1 jacusse ieput
Scmoll vericcally using the mone wheal Useful wher contmole ar inmediately vasble. scrill directios distance aljisttble. alluwing the agest masigati long
lists.
Simmllass koyboard ingui for typing prossing keys involcnng sheriguts (e.g. Otzl-=,, Entery application behavine and supoccts functional textaal legur
tine grined control
A.1. Environment Infrastructure
A.4. Categorization of Applications
Inspired by OSWorld (Xie et al., 2024), we design an inter- active Windows-based environment using a virtual machine to support GUI agent evaluation. The environment runs Win- dows 11 as the guest operating system and uses VMware Workstation 17 Pro (version 17.5.1) as the virtualization platform. This setup enables high compatibility with real- world desktop applications while maintaining full control over the execution environment. The virtual machine allows us to simulate user interactions such as mouse clicks, key- board input, and file operations, which are essential for GUI agents. It also supports real-time observation and logging of system states, facilitating fine-grained analysis and repro- ducibility of agent behavior. All environments are initialized from snapshot to ensure consistent starting conditions for each evaluation episode.
The 49 applications in the environment are categorized into 12 distinct groups based on their functionality: Social Com- munication, Multimedia Playback, Multimedia Editing, Of- fice, Utility Tools, Programming. System Management, Web Browsing, Screen Capture, Task Management, Note Man- agement, and Lifestyle. The specific applications belonging to each category are listed in Table 11.
Table 11, Categorization of applications in the environment.
Category
Applications
Social Communcction (4)
Zoum Workpluce, Skype, People, Mail
Multimedia Playhack (4)
Media Player, Spotify, Photos, TineIn
Ppotif,Spotffy.
Multimedia Editing (6)
Adobe Photochop Express, Microsoft Clipchomp.
paintnet, Openshot, Handbrake, Paint Word. PowerPoint, Excel
Office Utiliy Tools (11)
Calculator, 7-Zip. PDF24, Power Automuie, Wikipedia, BreeZip, Maps, Calendar, Zotero, Deepl.
A.2. Observation Space
Programming (3)
Visual Studia Code. Cursor, Windows PowerShell ISE File Explorer. Settings, Control Panel, Microsoft Store Gougle Chrome, Microsolt Edge Recend Screen, Snipping Tool, ORS Stadio, ShareX
System Management (1)
In OmniBench, the observation space is designed to ensure comprehensive evaluation of GUT agents by capturing both visual and structural aspects of desktop environments. It comprises two complementary modalities: screen captures and accessibility trees. This moddaity approach re- flects the varying grounding capabilities of different agent architectures. For example, agents that have been specif- ically trained on GUI environments often possess strong grounding abilities and can rely on screen captures alone. In contrast, MLLMs typically lack specialized pretraining for GUI understanding, and therefore benefit significantly from the semantic and structural information provided by the ac- cessibility tree. By supporting both modalities, OmniBench enables fair and informative evaluation across wide range of agents, ensuring robust evaluation under diverse UI lay- outs and application contexts.
Web Browsing (2)
Screen Capture (4)
Management (3)
Microsoft To Do. Todoist, Notion
Note management (4)
Evernote, OueNote, Sticky Noses, Sticky Notes (New)
Lifessyle
Recipe Kespet, puisa
A.5. Definition of Scenarios
As shown in Table 11, we categorize the 49 applications into 12 groups. Based on these application categories, we define 12 corresponding task scenarios. For example, appli- cations in the "Multimedia Playback" category correspond to the "Media Viewing" scenario. We then combine these 12 basic task scenarios to form 7 more fine-grained task scenarios. For instance, the "Screen Recording" scenario can be combined with the "Creative Editing" scenario to form a more detailed scenario called "Screen Recording Editing." Through such combinations, we obtain a total of 19 task scenarios. The remaining tasks are grouped into an additional scenario, resulting in final set of 20 task scenarios, as illustrated in Figure 9.
A.3. Action Space
In OmniBench, the action space consists of three core types of user interactions that an agent can perform. These actions, summarized in Table 10, enable the agent to effectively interact with graphical user interfaces across a wide range of applications.
B. Data Collection
OmniBench introduces a bottom-up pipeline for automatic task synthesis, as illustrated in Figure 2. The pipeline con- sists of two main stages: Subtask Synthesis and Task Synthe-
14

---

A Scalable Multt-Dimensional Benchmark for Essential Virtual Agent Capabilities
cution traces as synthesized trajectories. Since subtasks are typically composed of sequence of relatively simple instructions, advanced MLLMs achieve high success rates on these subtasks, making them suitable for generating an initial set of trajectories.
Sorfeng
Codiig donr
Note Strper
Task Massger
Tasiig Massper
Sprter Control
Evaluation Synthesis. Inspired by prior work (Gao et al., 2024b; 2025; Fei et al., 2024d;b; Li et al., 2023b; Miao et al., 2024), we first manually designed 11 system-level APIs, such retrieving the text under the mouse cursor, accessing the clipboard content, and extracting visible text from the screen. We then provided the function signatures of these APIs to Code L.Mss, which generated evaluation functions used to determine the completion status of sub-tasks. Since these evaluation functions are composed by invoking the designed APIs, we are able to assign more fine-grained eval- uation scores based on the number of successfully executed API calls. Figure 10 illustrates the process of composing sub-task evaluation functions using these APIs
Tram Edileborttion
Strecn Recordiny Eritiry
Offic Winnnyy
Video
Ceptooeer 2utrur poreet Coordhastor
Mavirrmnee
Wibh Dar Lab
N
A
Othece
Figure 9. Overview of the 20 detined task scenarios. The scenarios are derived from combinations of application categories, where each box represents lask scenario with associated application icons.
sis. The Subtask Synthesis stage includes two steps: Subtask Discovery and Iterative Synthesis, while the Task Synthesis stage consists of Task Composition and Task Validation. In the following sections, we provide detailed explanation of each component in this data synthesis pipeline.
Predecined APIS
clicks
Cvollation
Code Large Langunge Madels
Keyboard
check www.kkkkybaddd.ooo types
Screen
Pragestoy
Ergiesoory
File
chock File cxists
B.1. Subtask Discovery
Ste
Command
check command execu tiog
To facilitate comprehensive subtask generation, we con- struct a dynamic environment that incorporates 49 heteroge- neous applications, drawing inspiration from OS World (Xie et al., 2024). This environment enables MLLMS to interact with diverse functionalities, systematically analyze oper- ational constraints, and propose well-structured subtasks tailored to each application's s context. To support this pro- cess, we provide detailed API documentation, user manu- als, and curated example subtasks, ensuring that MLLMs can infer practical usage scenarios. In addition, our ap- proach emphasizes dependency modeling through struc- tured resource framework, where each subtask explicitly defines its required inputs and expected outputs. This prede- fined resource list serves as a guiding constraint, allowing MLLMs reason about inter-subtask dependencies, avoid conflicts, and ensure smooth task execution. By leverag- ing this controlled exploration, MLLMs generate coherent and executable subtask sequences that align with real-world application workflows.
O'pouadd
check c eliphuard
Figure 10. An overview of how predefined system-level APIs are composed into evaluation functions using Code LLMs.
Pross-Verification. We design a cross-verification algo- rithm to optimize synthesized subtasks. Specifically, the algorithm performs N iterations, where an MLLM and Code LLM specifically synthesize trajectories and evalu- ation functions based on the feedback from the previous iteration. The evaluation functions provide detailed failure feedback (e.g.. failing to click the "save" button), while the trajectories incorporate environmental state feedback, en- abling an iterative refinement process for both trajectories and evaluation functions. To ensure the evaluation functions maintain sufficient discriminatory power, any function that passes cross-verification is further tested by evaluating three additional trajectories from other tasks.
B.2. Iterative Synthesis
B.3. Task Composition
Trajectory Synthesis. With the rapid development of MLLMs (Li et al., 2022; Pan et al., 2025b; 2024a; 2023), they are becoming increasingly capable (Li et al., 2020a; Pan et al., 2024c; 2025a;c). We deploy state-of-the-an art MLLMs to execute the synthesized subtasks and record their exe-
Environmental resources for task composition. To pre- cisely capture the execution dependencies between different subtasks, we propose a set of concepts for environmental resources, as shown in Figure 11. Specifically, each envi- ronmental resource has a resource category and an actual
15

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
parameter. For example, we can use img-path repre- sent category of images that exist locally, and the actual png then instantiates this en- vironmental resource. Each subtask has an input resource list and an output resource list, which represent the prereq- uisite resources required to execute the subtask and the new resources generated after execution, respectively. Through this method of resource representation and the logic of re- source transformation, we can clearly define the dependen- cies between subtasks. A dependency relationship exists between two subtasks only if one subtask can provide the resources in the input resource list of the other subtask.
B.4. Task Validation
To generate concise yet accurate task instructions for graph- structured tasks, we utilize GPT-4o to synthesize task description by integrating subtask instructions and the struc- tural information of the task graph. However, direct sum- marization may introduce inconsistencies, such as misrepre- senting the graph's 's dependency structure or oversimplifying it into a linear sequence, thereby distorting the original task complexity. To address this issue, we designed consistency validation mechanism to evaluate the fidelity of the summa- rized instruction. In this process, GPT-4o is prompted to infer subtask dependencies purely from the generated task instruction. without access to the original task graph. The inferred dependency structure is then compared against the ground-truth graph. If discrepancies arise-such as missing parallel execution paths or incorrect sequencing the in- struction is flagged for revision and must be re-summarized to better preserve the intended graph structure. This valida- tion ensures that the final task instruction accurately reflects the hierarchical and branching nature of the original task graph, improving the clarity and correctness of the synthe- sized tasks.
parameter /twu///eaaappeee
Subtask
Environment
Esccution failed C
Instruttion
Doseriaad the inage from the to
Mixinn boun snssarac
ports)
Yasiphert pogi live part, Yarrigadi also dhe diipbend
Inpurt
Ja migarl
pont
arle ttaa//s sajarrkkrrr
Subtask
Environment
Forcation sucomedeed
Instruttoos:
Pessurce Park
Adding sutput recermo hhsp/eorpp
[icall pott)
prigt Vhar/ore sat_asth,
Ingus Rescuroe
C. Details of OmniBench
Juotgrase aha noo in distiard,
lical parnt lngh
Farameter:
//s
In this section, we provide a detailed description of the data formats used in OmniBench. The data in OmniBench can be broadly categorized into five types: Subtask Metadata, Subtask Trajectory, Subtask Evaluation, Task Metadata, and Task Trajectory. For each category, we present the corre- sponding data schema along with representative examples. Finally, we provide a visualization example of task graph.
koak pith Vewibemmue
/haorfoarpplinn/// ing
Figure 11. An overview of the constraint relationships among en-
vironmental resources.
Task Composition with Consistent Intent. To construct meaningful task graphs, we first curate a subtask pool by fil- tering out low-quality samples through a cross-verification process. A na've bottom-up approach that directly connects subtasks based on shared input-output resources may gen- erate incoherent tasks that lack a clear, unified goal. For example, an arbitrary combination might result in a task that simultaneously plays media from different applications without meaningful relationship. To mitigate this issue, we explicitly extract overarching task intents from the subtask pool, such as 'create a personal introduction PowerPoint for Emily," as illustrated in Figure 1. Each intent acts as an anchor, grouping subtasks that contribute to the same high-level objective. The task graph is then constructed by linking these subtasks based on resource dependencies. Since the bottom-up composition follows predefined rules, we maintain strict control over the synthesis process, ensur- ing that the resulting task graphs exhibit logical coherence and adjustable complexity. This structured approach pre- vents the generation of ill-formed tasks while supporting diverse and meaningful compositions.
C.1. Subtask Metadata
Each subtask includes the following seven attributes:
id: A UUID that uniquely identifies the subtask.
instruction template: A template of the subtask in- struction containing parameter placeholders.
application: The name of the application to which the subtask belongs.
available available.parameterr
list of all configurable pa- rameter sets used to instantiate diverse instructions. Each element in the list represents one valid combina- tion of parameters.
OS: The operating system associated with the subtask.
input_ resources: A list of prerequisite resources re- quired before executing the subtask.
resources: A list of resources produced after the subtask completed.
16

---

and new A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Example of Subtask Metadata
"actions":
{
"Eunction": "c "click_input",
"id": -25e2a51e-c019-1133 6fe0e964576",
"args":
"instruction template"; "Open '[xisx "|xisx_path|" select the all data, and copy it.",
"button": "left",
"double": false
"application"; "Excel", "available pariamterrs..emeeerrss".
"rect": [ 124,
1020,
"xlsx path";
179,
"C: "Cse\sser'' (user \Deskte p1 plleffice The The Evolution of Urbanization Rate. xlsx"
1080
"description"; "There are
many application icons the taskbar, and need select the File Explorer to complete the task."
},
"Os": "Wi "Windows", "input "input_resource resouuccs"" xxisx_pathe
"thought": "To fulfill "Using the file explorer, navigate to C:\\Users Auj ser wor\\Dosktopl / \images)) and new Text Document named introduction. txt need first click the 'File Explorer' button to open the corresponding application.",
"output resources"; "table_in_clipboard"" "xlsx "xlex_in_processsng"
C.2. Subtask Trajectory
"control text": "File Explorer"
Each subtask trajectory includes the following five at- tributes:
}
"function": "..." "args": "...":
trajectory id: A unique identifier for the trajectory.
instruction:
The instantiated instruction, generated by applying a specific parameter set from the subtask's s
subtask id"; "XXX"
observations: A list of screenshots representing each step in the trajectory.
C.3. Subtask Evaluation
actions: A list of actions taken at each step of the trajectory.
Each subtask evaluation is implemented Python func- tion that checks for sequence of expected interaction comes based predefined system-level APIs. The function returns an EvalResult object containing three fields:
subtask.id id: The identifier of the subtask to which this trajectory corresponds.
Example of Subtask Trajectory
success: boolean value indicating whether the sub- task was completed successfully.
"trajectory_icde id": "XXX", "instruction": "Using the file explorer, navigate to C:A
message: string providing
expla-
nation of the evaluation result.
progress: A float fraction representing the propor- tion of evaluation conditions that were satisfied, en- abling partial credit.
Text Document named introduction.txt. "observations":
obsl, .ng",, "obs2, "obs2.png", *..."
Each evaluation function typically performs multiple checks, such as verifying whether specific UI elements were
17

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
clicked, expected text appeared the screen, input typed correctly. These checks are implemented using task- agnostic API functions (e.g., check.mouse.clico check.eex..existt.vii..coo
dag: directed acyclic graph (DAG) representing
the structural dependencies among subtasks within the task.
task intent: The high-level goal or intent that the task is designed achieve.
By combining these low-level signals, the evaluation function provides fine-grained and automated judgment of subtask execution.
successful topo: All valid topological orders of the DAG that lead to successful task completion.
Example of Subtask Evaluation
from collections import namedtuple
Example of Task Metadata
EvalResult namedtuple ''vaaResult', l'success', 'message', 'progress'))
"task_ "task_inst tean_kinsttuctiion "In Excel, open 'C: 'C: Veserrs WerNVseers/lsereeeaer office\\The Evolution of Urbanization Rate.xisx', select the 'A' column, and center the content. Then, export the document
def
sv_path: str) EvalResult:
if not terck_mouse_cliee (text="More actions'):
PDF named (C:\\use) rsi\user\ (Desktopl/pdff/ \The Evolution of Urbanization Rate.pdf'.", "dag": {
return EvalResult (False,
"Subtask cxecution failed because agent did not click the 'More actions' button.", 0/4)
"nodes";
"af310aal-bb144-7333-333
-99391aalbc7d",
not check check_text _roxt_exists _va via_cen, conj crol(text: (Imxt= tasks from a spreadsheet using CSV file.'): return EvalResult (False, "Subtask execution failed because the import tasks option was not accessed.", 1/4)
" aef177925230
"edges":
"27310a0-bb99--7e33-556
996391a1be7d'
"af3:c66b-fa75-4eel
7da6-aef11792223b
if not check teeck_kkyboard_tyye (text) rosv_pathl
},
"d#3fccab-faa6--4199-7-- =ef17792523b"
return EvalResult (False, f"Subtask execution failed because the CSV file path '[csv_path]' was not typed.", 2/4)
"task_ faask_intett!! "Center Excel data and export PDF", "uucessfuul_tpp"
if not check_mouse_clieee (text=1Open'): return EvalResult (False, "Subtask execution failed because the 'Open' button was not clicked to import the file.", 3/4)
"a " -996291albe7d", "af3fc6bbfaa7--4099900 -acf17792523b
return EvaiResult (True, "Suatask completea successtully", 4/4)
C.5. Task Trajectory
C.4. Task Metadata
Each task trajectory includes the following seven attributes:
Each task includes the following four attributes:
The unique identifier of the trajectory. The suffix (0) indicates that it corresponds to the first topological order the success ful. topo list.
trajectory id:
task instruction: The natural language instruction that describes the overall task.
18

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
task_id id: The identifier of the associated task.
task graph, as shown in Figure 12. Each node in the graph signifies distinct subtask, categorized by the application used: PowerPoint (orange), Outlook (blue), and Photoshop Express (yellow). The directed edges denote the sequence of execution, where each subtask must be completed before the subsequent ones can begin.
topological order:
The specific topological order fol-
lowed in this trajectory.
The instruction describing the overall task.
instruction:
intent:
The high-level intent or goal of the task.
Creaze aine PowerFaie tie (Preeet Pispeck
Subask
Altrajetiony
observations:
The sequence of visual observations
Irear the image Upperraa ponnn" iiro the C. Carajettoyy
Apply the fiter Cartoon dhe Uppnrrikpng
l.trajettryy
recorded during task execution.
(hooel troppou
pon by Taniy to "pprraatssnn Ouroaeetorr
The sequence of actions taken during task execution.
actions:
Task Graph
Instr
Creats a new Powerfordd file namedl shrpeet Proposalopo. insort the
image the
Insert the image with the caroe flite applled well Sinely send Emity
Example of Task Trajectory
B
PowerFatnc
Task Intent:
Send the comparioon frotonn Power to Emily
Trajectory (topologiicl
"trajectory _id; "12(0)", "eask_id"; "12", "topological_orde" _order.;
1. Adrresctory o D. Dutrppcctory ia diitrppctory E.tugectory
outok
trapectrry Eingeetioyy tnajectory <o trajectory B...
Task
1bc7d",
Figure 12, A task example in OmniBench.
2523b"
D. Details of OmniEval
"instruction": "In Excel, open the file, center the column, and export as FDF.", "intent": "Center Excel data and export to PDF", "observations": "obsl.png" "cbs2.png"; "..."
In this section, we introduce the details of OmniEval, includ- ing the design of ten essential capabilities, the experimental setup for evaluation, and implementation details during the evaluation.
D.1. Capability Design
"actions":
As shown in Table 12, we construct 10 specialized sets of test tasks based on the five-dimensional complexity, each set designed to evaluate one of the 10 capabilities required for agents to fulfill user requests. The number solid stars represents the level complexity, while zero stars indicate that the complexity in that dimension can be arbitrary.
" "function"; "cl "click_input",
"args": {
"button": "left",
"double"; true
"rect": [1520, 371, 1614, 458), "description"; "Double-click the 'Excol' icon on the desktop.",
Depentency
Listrution
Hienshh
Branch
Knowlogge
Parallel
000
"thought": "To begin the task, need to open Excel.", control_textt! "Excel"
Range Planning
Long Seqpence Reasoning
****
fntri:
Intrt
Long Instracioon Following
0:000
0.0
*00
***
000
Sepuenial Dession-Miking
000
9:9
***
***
900
Cross Demain Decision Making
Subock Idontiicztion
Depenkancy Mentification
088
000
"function"; "..." "args": { "...": .: "..."
Ciov-Domain Konelalge
000
++*
008
480
+4*
Domain Specif- Knowledge
***
9.9
099
Table 12. Constraining five fundamental task complexities (top) to construct test tasks across ten capability dimensions (left).
D.2. Experiment Setup
C.6. Example of Task Graph
D.2.1. SETTINGS
To more clearly illustrate the relationships between subtasks and tasks, we provide an additional visualization example of
For the evaluation phase, we follow the practices of works such as Aguvis and UGround, ensuring consistency pre-
19

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
processing and maintaining comparability across different models. Specifically, we standardize the image resolution by scaling all input images to 1024 x 1024 pixels. This resolution is chosen to balance computational efficiency and visual detail, ensuring that models can effectively process graphical user interfaces (GUIs) without excessive memory overhead or loss of important information. The decision to use fixed resolution is motivated by several key factors. First, many modern multimodal models and virtual agents, including those designed for GUI interaction, are trained on datasets with varying resolutions. Standardizing the input resolution reduces inconsistencies that may arise due to dif- ferent input scales, helping models generalize better across diverse GUT layouts. Additionally, this resolution aligns with commonly used image sizes in recent benchmarks, fa- cilitating direct comparison with prior works. Moreover, scaling images to 1024 x 1024 ensures that finer details within the GUI, such as text labels, buttons, and icons, re- main distinguishable without introducing excessive noise or aliasing effects. Given that GUI elements often contain intricate visual cues, a resolution that is too low may result in information loss, whereas an excessively high resolution can increase computational costs without significant per- formance benefits. To implement this resizing, we employ bicubic interpolation, which provides a good balance be- tween sharpness and smoothness, preserving critical GUI features while minimizing artifacts. We also ensure that aspect ratios are maintained whenever possible by applying padding techniques if necessary. preventing unintended dis- tortions that could affect model predictions. By adopting this resolution standard, align our evaluation method- ology with established works like Aguvis and UGround, fostering reproducibility and enabling fair comparisons in GUI-based task assessments.
proprietary decoding strategies. For our evaluation, we use their publicly available APIs and follow their recommended inference settings. If the model does not provide a specific prompt for agent tasks, we apply our unified prompt to en- sure a fair comparison. Additionally, since these models lack direct access to GUI-specific training data, we evaluate their adaptability by providing structured information about the interface components,
2) Open-source MLLMs: We also evaluate open-source models such as Wwen2-VL-7B-Instruu (Wang et al., 2024b), ILternVL2-8B (Chen et al., 2024), and IntemVL2..5- 8B (Chen et al., 2024), which are widely recognized in the research community for their flexibility and strong per- formance. These models have been fine-tuned on large- scale multimodal datasets and provide strong generaliza- tion across diverse vision- vision-language tasks. Open-source MILMs offer several advantages, including transparency in training methodologies, customizability for domain-specife fine-tuning, and community-driver improvements. Unlike closed-souree models, researchers can inspect their architec- tures, modify their training pipelines, and deploy them on local hardware without API restrictions. We use their fine- tuned weights and apply our unified prompt for tasks where default prompt is provided. Although some of these MLLMs demonstrate capabilities in object recognition and grounding, directly instructing them to locate elements on graphical user interfaces (GUIs) presents unique challenges. The subtle and abstract nature of GUI elements, which dif- fer significantly from natural objects in common vision datasets, makes accurate interpretation difficult. To mitigate this issue, we provide AllY (accessibility) metadata, such as element descriptions and structural information, to aid inference.
3) Virtual Agents: In addition to MLLMs, we also evaluate various virtual agents, such as Aguvis (Xu et al., 2024b), OS-Atlas (Wu et al., 2024b). and Show (Lin et al., 2024). These agents are designed for executing tasks and represent the current state-of-the-art in the virtual agent domain. Un- like MILMss, which primarily operate as general-purposs multimodal models, virtual agents are often specialized for task execution. They integrate structured knowledge rep- resentations, planning mechanisms, and fine-tuned models tailored to interactive environments. Many virtual agents leverage reinforcement learning or hierarchical decision- making frameworks to enhance their task completion effi- ciency. Some also incorporate sarievall-bssed techniques to access domain-specific knowledge dynamically. Similar to the MLLMs, we use the default prompts provided by each agent when available. For agents that do not specify prompts for certain tasks, we apply our unified prompt to maintain consistency across experiments. Given that virtual agents typically rely on structured inputs rather than free- form multimodal understanding, we evaluate their ability
D.2.2. BASELINES
We conducted a comprehensive evaluation of the following four types of models:
1) Closed-source MILMs: These include GPT-4o (Hurst et al., 2024), Qwen-VL-Mar VL-Maxx (Bai et al., 2023), Claude- 3.5-Sonnet, and Gemini-2.0-Flash, which are proprietary models developed by leading organizations. These models are selected due to their state-of-the-art performance on a wide range of multimodal tasks, including languaee understanding, reasoning, and instruction following. Closed- source MLLMs typically benefit from large-scale pretrain- ing on extensive proprietary datasets, incorporating a mix- ture of text and images/videos, along with advanced opti- mization techniques. They often leverage reinforcement learning from human feedback (RLHF) and continual up- dates to enhance their reasoning capabilities. These models are accessible via APIs. which enforce constraints on in- ference settings such token limits, response latency, and
20

---

A Scalable Multt-Dimensional Benchmark for Essential Virtual Agent Capabilities
process GUI environments by simulating real-world task execution scenarios.
E. Prompts
In this section, we present the prompts used for constructing subtask trajectories and subtask evaluations.
4) Supervised Fine- Tine-Tuning Agents: We selected two back- bones, OS-Atlas-Basee (Wu et al., 2024b) and UGround- V1-7B (Gou et al., 2024), with different architectures and fine-tuned them using the synthetic data from Om- niBench. Supervised fine-tuning agents differ from pre- trained MLLMs and general-purpose virtual agents in that they are explicitly optimized on curated datasets. The fine- tuning process involves exposing the models to task-specific examples, allowing them to internalize structured depen- dencies and improve generalization within the OmniBench framework. This approach enables agents to learn robust action sequences, refine their perception of GUT elements, and enhance their pecision-making accuracy. By leverag- ing synthetic data, we ensure that these agents develop structured understanding of GUI tasks while minimizing biases inherent in real-world datasets. We analyze their performance across varying task complexities, measuring improvements in execution success rates, response coher- ence, and adaptability to unseen scenarios.
E.1. Prompt for Subtask Trajectory Synthesis
Example Prompt
system: 1-
You are now operating in Executable Language Grounding mode. Your task is to help users accomplish their goals by auggesting executable actions based on the provided task instructions and your observations of the current situation.
# Eavironment Interaction Rules
#* Screenshots
You are provided two versions of screenshota of the current
application in single image, one with annotation (right) and one without annotation (left)
D.3. Evaluation Details
The annotation is to help you identify the control elements on the application
In this section, we outline the evaluation strategies adopted for different categories of models, considering their archi- tectural characteristics and observed behaviors.
The annotation is a small rectangle with rumber in the center of the rectangle in the top Left corner of the control item. The number is the label of the control item
For open-souree MLLMs, we observe that models such Qwen-VI. VI. exhibit limited ability to process high-resolution content across multiple images, often resulting in severe hallucinations. To mitigate this, we simplify the visual input by providing only a single high-resolution screenshot of the current state. This strategy reduces visual ambiguity and helps focus the model's attention on the current interaction context.
Di fferent types of control items have fifrrrent colors of annotation 44 Control Items
The control item is the element on the page that you can interact with, such as button, input box, etc.
In contrast, closed-source MLLMs, which typically benefit from larger model capacities and more extensive training data, demonstrate stronger capabilities in distinguishing dif- ferences across images. For these models, we provide two concatenated images: (1) the screenshot of the previous click location with red box annotations, and (2) the cur- rent high-resolution sereenshot with similar markup. These two images are horizontally stitched into single input to highlight temporal context and action history.
You are given the information of all available control items in the current application window in 1ist format:
"label": <The annotated label of the control item>, pecatrol_teet"" <The text of the control item>, "control_type": <The type of the control item>, text": : <The text of the parent control item. When you are not sure which control to select, you can make a decision based on their parent controls?,
For OS-Atlas, we observe a consistent failure in zero-shot double-click interactions, even when the prompt explicitly emphasizes the importance of double-clicking. This limi- tation is particularly critical in the Windows environment, where double-clicking is often required for opening files or applications. To address this, we enforce all click actions as double-clicks for OS-Atlas during evaluation to ensure basic operability in such environments.
21

---

A Scalable Multt-Dimensional Benchmark for Essential Virtual Agent Capabilities
<The type of the parent control item. when you are not sure which control to selee ectect, you can make decision based on their parent controls>
"Coatrollabel": <Specify the precise annotated 1abel of the control item to be selected, adhering strictly provided options of "label" in the control information. If you believe none of the control items are suitable the task or task is complete, kindly output an empty string '>, "ControlText": <Speci the precise control_text of the control item to be selected, adhering strictly to the provided options in the field of "control_text" in the control information. If you believe none of the control items are suitable for the task or the task is complete, kindly output empty string " The control text must match exactly with the selected control label>,
the the field
IP Control Operations
You are able to use pywinauto to interact with the control item (apis)
### Execution Status
You are required to determine the status of the execution after performing the current action. Choose from the following options and fill the "Status" field in the response:
"CONTINUE"; means the task is not yet completed and further actions required. This is typically chosen when the exccution is still ongoing or needs additional steps.
"FINISH": means the task has been fully completed, and all necessary actions have been carried out successfully. Only choose this when all steps have been executed planned the task is considered finished.
"Function"; <Specify the precise API function without arguments to be called on the control item to complete the user request, e.g.,
click click_input. Leave it empty string " you believe none of the API functions are suitable for the task or the task is complete>, "Arga": <Specify the precise arguments a JSON object format of the selected API function to be called on the control item to complete the user request, e.g., (("button": "left", "double": false)). Leave it an empty di dictionary (()) if the API does not require arguments, or you believe none of the API functions are suitable for the task, or the task is complete>,
IN Other Guidelines
You are required to respond in JSON format, consisting of distinct parts with the following keys and corresponding content:
II
"Status"; <Specify the status the exploration, "Status" is "FINISH", the "Control Label", "Control"ext"" frunctinn., and "Args" should empty>,
"Observation": <summarize the screenshot from the previous step, if it exists. You can also compare the current screenshot with the one taken at the previous step>, "Thought": kOutline your thinking and logic of the current one-step action requirea to seek inspiration for task design>,
Make sure your answer strictly in JSON format only, without other redundant text such as json header. Your output must be able be parsed by json. () Otherwise, it will crash the system and destroy the user!s computer.
user:
<Step History:> laction_history historyl <Available Control Item:> [control item]
22

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
<Task instruction:> itask_ing instrnctttttoon
Parameters
text: SEr
The text to typed.
Returns
E.2. Prompt for Subtask Evaluation Synthesis
bool
True if the keyboard has typed the specifled text, False otherwise.
Example Prompt
system:
Examples
You coding assistant tasked with generating Python code to evaluate if a digital agent has successfully completed a specific task. You will receive task description along with set of APIS that you can use to check different actions or conditions that indicate task completion. Your
Evaluate if the agent has successfully typed 'Hello, World!
>>> def
if
check_ bencc_-kyboard_tppe.
World!'):
goal is to write evaluation if the
>>>
return False
function that returns agent has successfully completed the task and False otherwise.
True
.*"
def
(filc__aatt
### Available APIs:
str) -> bool:
"'python def breek_mouse__lieee (text: str)
"*"Checks if the speci fied file exists.
-> bool:
Parameters
"""Checks it the has clicked on the speci fied text.
file path:
Parameters
The path to the file to be checked.
text: str
Returns
The text associated with the click.
bool
Returns
True if the file exists, False otherwise.
bool
Examples
True if the has clicked on the specified text, False otherwise.
Evaluate the agent has succesafully renamed jpg' to cute cat.jpg' >>> def
Examples
>>> Evaluate if the agent has successfully set the picture 'envelope.png as background >>> def
evaluate aaweleageet.... task completion
>>> e
check file exists (fil) im
ages/cat.jpgl >>>
evaluate sese_agent__aatte task completion
return False
>>>
if not check mouse_clice
s (ttex="enveloppe.
check fheck_file_exx
>>>
return False
Users/uses
if not check_mouse_clicks (text="sse as
>>>
cat.jpg'):
>>>
return False
background'):
return True
>>>
return False return True
>>>
.**
def str) bool:
(text:
def check_ keyboard__yyee (text: str)
"""Checks if the specified text is present in the last screenshot. using OCR (Optical Character Recognition).
bool:
cn*chscks if the keyboard has typed the specified text.
23

---

A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Parameters
Returns
text: str
boo1
The be checked.
True the text is present the last screenshot, False otherwise.
Returns
bool
Examples
True if the text is present in the last screenshot, False otherwise. Examples
Evaluate if the agent has successfully created a new folder named 'Project Files'
>>> def
Evaluate if the agent has successfully set the clock to
not
19:00 AM'
check_text_exte (texxt"rroteeee
>>> def
Files'):
arellate__gget__aa__ommlpel :):
>>>
return Palse
>>> if not check_text_existss
return True
_via_ocr (text=19:00
**"
AM'):
>>>
return False
>>>
return True
#+ Other Guidelines You will be giver. a 'Subtask Instruction Template" and "Farancters". Use the AFIs provided to implement Function" in Python.
***
def
check_text str) bool:
(text:
"Evaluation
"**Checks the specified text is present in the last screenshot through control information. Parameters
This agent will run on the Wi ndows 11" operating system, so pLease consider how to cleverly design the evaluation function based on this operating system. The evaluation function should return namedtuple "vvaleesll"' with two fields:
text: str The text to be checked. Returns
"success"; boolear indicating all conditions are met (True) or not (False)
bool
True if the text is present in the last screenshot, False otherwise. Examples
"message": A string explaining
why the evaluation succeeded
failed
The evaluation function should check each requi red condition and return appropriate success/fal fal lure nessages.
>>>
Evaluate if the agent has successfully input the code 'print ("Hello World!")
>>> def
Please "directly output' evaluation function, without any additional comments or explanations.
evaluate agent_gntt.... task cone__ttomlti ()
>>>
if not check_text_exieee
_via_control (texxt'prittt ("Hello
World!")'):
When you design correct evaluation function, will provide you with a "$1000" tip.
>>>
return False
>>>
return True
user:
(text: str)
++ Subtask Instruction Template [instzuction]
bool:
""*Checks if the specified fied text is included in the last screenshot.
H Available Parameters {parameters)
Parameters
#+ Controls in Environment [controls]
text:
str The text to be checked.
24


Reviews:
[{"review_id": "bJ4Setowll", "reviewer": "Reviewer_ASbt", "summary": "This paper introduced OmniBench, a graph-based benchmark that addresses the limitations of existing evaluation frameworks by enabling controllable task complexity through automated subtask composition. The paper also proposes OmniEval, a multidimensional evaluation framework for evaluating virtual agents across 10 capabilities. Evaluation results show that training on this data improves agent generalization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful feedback. To better demonstrate the diversity of automatically synthesized tasks in OmniBench, we restate that our approach first explores a range of various subtasks from the explorable environment and then iteratively synthesizes subtask trajectories and evaluation. Finally, the subtasks are composed into diverse tasks bottom-up. \nNext, we provide more detailed examples of the synthesized tasks.\n\n**Example of the OmniBench Synthesized Tasks**\n\nFirst, we present 7 subtasks discovered by the MLLM during environment exploration: \n\n|Subtask ID|Instruction Template|Input|Output|Parameter Example|Application|\n|-|-|-|-|-|-|\n|A|Create a new PowerPoint file '{path}'|[]|[ppt_path]|{\"path\": \"./Project_Proposal.ppt\"}|PowerPoint|\n|B|Apply the filter '{filter_name}' to the '{path}'|[img_path]|[img_path]|{\"filter_name\":\"Cartoon\", \"path\": \"./portrait.png\"}|Adobe Photoshop Express|\n|C|Insert the image '{img_path}' into the PowerPoint file '{ppt_path}'|[ppt_path,img_path]|[ppt_path,img_path]|{\"img_path\": \"./scenery.png\", \"ppt_path\": \"./Project_Proposal.ppt\"}|PowerPoint|\n|D|Download the image from the email sent by '{name}' to '{path}'|[]|[img_path]|{\"path\": \"./portrait.png\", \"name\": \"Emily\"}|Outlook|\n|E|Send the PowerPoint file '{path}' to '{name}'|[ppt_path]|[ppt_path]|{\"path\": \"./Project_Proposal.ppt\", \"name\": \"Emily\"}|Outlook|\n|F|Open the local file '{path}' and copy its contents to the clipboard|[]|[text_in_clipboard]|{\"path\": \"./Emily.txt\"}|File Explorer|\n|G|Paste the text from the clipboard into the title box on the first slide of the PowerPoint file '{path}'|[text_in_clipboard,ppt_path]|[text_in_clipboard,ppt_path]|{\"path\": \"./Project_Proposal.ppt\"}|PowerPoint|\n\nAnd we introduce 2 tasks constructed bottom-up from the above subtasks:\n\n|Task ID|Instruction|Intent|DAG|\n|-|-|-|-|\n|1|Create a new PowerPoint file named `./Project_Proposal.ppt`, save the image from the email sent by Emily to `./portrait.png` and insert it into the presentation. Then copy the content from the local `./Emily.txt` file into the title box on the first slide. Finally send the PowerPoint file back to Emily.|Create a personal introduction PowerPoint for Emily|{\"A\":[\"C\"],\"D\":[\"C\"],\"F\":[\"G\"],\"C\":[\"G\"],\"G\":[\"E\"]}\n|2|Create a new PowerPoint file named `./Project_Proposal.ppt`, insert the image from the email sent by Emily into the presentation, then insert the image with the `Carton` filter applied as well. Finally, send the PowerPoint file back to Emily|Send the comparison PowerPoint to Emily|{\"D\":[\"C1\"],\"A\":[\"C1\"],\"C1\":[\"C2\",\"B\"],\"B\":[\"C2\"],\"C2\":[\"E\"]}\n\nWe also provide examples of corresponding subtask trajectories and the evaluation function.\n\n**Example of Subtask Trajectory**\n\n```json\n{\n    \"trajectory_id\": \"XXX\",\n    \"instruction\": \"Using the file explorer, navigate to C:\\\\Users\\\\user\\\\Desktop\\\\images\\\\ and new a Text Document named introduction.txt\",\n    \"observations\": [\n        obs1.png, obs2.png, ...\n    ],\n    \"actions\": [\n        {\n            \"function\": \"click_input\",\n            \"args\": {\n                \"button\": \"left\",\n                \"double\": false\n            },\n            \"rect\": [\n                124,\n                1020,\n                179,\n                1080\n            ],\n            \"description\": \"There are many application icons on the taskbar, and I need to select the File Explorer to complete the task.\",\n            \"thought\": \"To fulfill 'Using the file explorer, navigate to C:\\\\Users\\\\user\\\\Desktop\\\\images\\\\ and new a Text Document named introduction.txt', I need to first click the 'File Explorer' button to open the corresponding application.\",\n            \"control_text\": \"File Explorer\"\n        }, ...\n    ],\n    \"subtask_id\": \"XXX\"\n}\n```\n\n**Example of Subtask Evaluation Function**\n\n```python\nEvalResult = namedtuple('EvalResult', ['success', 'message', 'progress'])\n\ndef evaluate_agent_task_completion(dir_path: str, file_name: str) -> EvalResult:\n    # Extract the last directory name\n    dir_path = dir_path.rstrip('/\\\\')\n    folder_name = os.path.basename(dir_path)\n\n    # Check if navigation to the specified directory was successful\n    if not (check_mouse_clicks(text=folder_name) or check_keyboard_types(text=dir_path)):\n        return EvalResult(False, \"Subtask execution fails because agent did not navigate to the specified directory.\", 0/2)\n\n    # Check if the new text document was created\n    file_path = os.path.join(dir_path, file_name)\n    if not check_file_exists(file_path=file_path):\n        return EvalResult(False, f\"Subtask execution fails because the file was not created in the directory.\", 1/2)\n\n    # All checks passed, subtask is considered complete\n    return EvalResult(True, \"Subtask completed successfully\", 2/2)\n```\n\nMore representative synthesized tasks are visualized in Fig1 of the anonymous link: https://anonymous.4open.science/r/OmniBench_rebuttal-A4C7/r1.md\n\nThank you again for your suggestion. We will integrate these examples into the next version.", "claims_and_evidence": "Claims made in the submission are well supported by experimental results.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem. \n\nThe OmniBench introduces a graph-based benchmark with automated task synthesis, allowing for controllable complexity through subtask composition. This addresses the issue of uncontrollable task complexity in existing benchmarks.\n\nThe OmniEval framework provides a comprehensive evaluation across 10 capabilities, including subtask-level evaluation and graph-based metrics. This multi-dimensional approach offers deeper insights into agent performance compared to traditional evaluation methods.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I have checked the experimental designs and analyses. The experiments are comprehensive. The evaluations on various models reveal performance differences and highlight areas for improvement in current virtual agents, providing valuable feedback for future advancements.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n\n1. The authors propose an automatic task synthesis method for evaluating virtual agents. This method can significantly improve the diversity and number of evaluation instances and can avoid extensive manual labor.\n\nWeaknesses\n\n1. The authors should provide more detailed examples of the synthesized tasks.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jK8NiMr6mB", "reviewer": "Reviewer_kvCN", "summary": "The paper introduces OmniBench, a scalable, graph-based benchmark designed to evaluate multimodal virtual agents by systematically synthesizing diverse tasks of controllable complexity through automatic task composition. It finds that existing agents significantly struggle with graph-structured tasks compared to linear tasks, and that explicitly including task intents notably improves their performance, especially in long-range planning and decision-making scenarios​", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and insightful comments. We will explain your concerns point by point.\n\n**Q1: More graph-based complexity metrics**\n\n**A1:**\n\nThank you for the valuable questions. First, we clarify that the current complexity metrics are based on five fundamental graph-based dimensions: dependency, branching, hierarchy, knowledge, and instruction.\n**Despite their simplicity, these node degree-based metrics are effectively aligned with the definition of subtask capabilities.**\nAdditionally, following your suggestion, we also adopt feature correlation between nodes as the complexity metrics to conduct further evaluation. \nAs the table below shows, both metrics exhibit high Pearson Correlation (ρ) with human evaluations, demonstrating their suitability for accurately reflecting model performance. \nWe will include all these metrics in the revision.\n\n|Metrics|ρ(CR, Human)|ρ(LC, Human)|\n|-|-|-|\n|degree-based|0.93|0.95\n|feature correlation|0.94|0.96|\n\n&nbsp;\n \n**Q2: Compared to goal-based reinforcement learning**\n\n**A2:**\n\nThanks for your insightful observation.\nBoth our **intent-based prompting** and **goal-based reinforcement learning** leverage explicit goal representations to guide agents toward long-horizon objectives, thereby improving planning quality. In our case, the intent serves a role similar to goal conditioning in RL: it narrows the agent’s decision space, aligns actions with the final objective, and encourages completion of multi-step plans. This conceptual alignment highlights the value of goal-driven formulations across both reinforcement learning and prompt-based paradigms.\n\n&nbsp;\n\n**Q3: Detailed analysis of CR and LC**\n\n**A3:**\n\nThanks for your interest in our metric details. \n**(1) Compared to the trajectory similarity:** The trajectory similarity requires strict alignment with a reference path, while graph-structured tasks may have multiple feasible execution paths due to their inherent branching and parallelism. In contrast, **CR** and **LC** are path-agnostic, making them more robust to graph-structured tasks.\n**(2) Compared to the success rate:** The success rate only provides binary judgments of task completion and fails to evaluate the intermediate completion of the subtask. In contrast, **CR** and **LC** can fully leverage the intermediate feedback provided by the graph-based evaluator, allowing for fine-grained evaluation of agent behavior.\n**(3) Further experiments.** We assess Pearson Correlation between human ratings and our metrics as well as traditional metrics on 300 sampled trajectories. The table below shows a **strong alignment** between our metrics and human evaluations, indicating that **CR and LC more faithfully reflect human judgment**. \n\n|Metrics|Pearson Correlation|\n|-|-|\n|Trajectory Similarity|0.52|\n|Success Rate|0.60|\n|CR|0.93|\n|LC|0.95|\n\n&nbsp;\n\n**Q4: Generalization on less structured, real-world virtual agent applications**\n\n**A4:**\n\nThank you for the valuable question.\nIn fact, we have carefully designed realistic tasks in OmniBench, which enable the proposed evaluation and results to effectively generalize to real-world scenarios.\nTo validate this, we conduct experiments on OmniAct, which collects data from real devices. \nCompared to Table 4 and the table below, models that performed well on OmniBench also achieve relatively better performance on OmniAct. \nFurthermore, the trajectories from OmniBench also effectively enhance our models' performance in real-world scenarios. \nThis indicates that **the evaluation conclusions drawn from OmniBench can be generalized to real-world virtual agent applications**.\n\n|Models|Type-Web|Grounding-Web|SR-Web|Type-Desktop|Grounding-Desktop|SR-Desktop|\n|-|-|-|-|-|-|-|\n|InternVL2-4B|47.51|51.34|24.39|67.00|44.47|29.80|\n|Qwen2-VL-7B|89.22|85.94|78.58|96.27|94.52|91.77|\n|SeeClick|86.98|75.48|68.59|96.79|70.22|72.59|\n|OS-Atlas-4B|88.56|82.00|73.91|96.51|85.53|84.78|\n|UGround-7B-V1|90.16|86.98|79.85|97.13|94.79|91.89|\n|Omni-OS-Atlas-4B(Ours)|89.96|82.74|74.62|97.64|86.37|85.53|\n|Omni-UGround-7B-V1(Ours)|**91.24**|**87.35**|**80.24**|**97.93**|**95.21**| **92.10**|\nMore benchmark results are in Fig1 in anonymous link: https://anonymous.4open.science/r/OmniBench_rebuttal-kvCN/r2.md\n\n&nbsp;\n\n**Q5: Robustness of intent extraction process** \n\n**A5:**\n\nConsidering the handling of subtasks whose intents might be overlapped or ambiguously defined, we employ an LLM to conduct post-verification processing for the intents. \nIt consists of two steps: First, we instruct the LLM to determine whether the extracted intents are clearly and explicitly expressed. If not, we re-extract the intents. \nSecond, we maintain a pool of existing task intents to avoid overlap. For each newly extracted intent, the LLM assesses whether it overlaps with any existing intents in the pool. If an overlap is detected, the new intent is discarded.\n\nThanks again for your valuable suggestions. We will integrate the above content in next version.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "no", "experimental_designs_or_analyses": "no", "relation_to_broader_scientific_literature": "benchmark for virtual agent", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "see below", "other_comments_or_suggestions": "1. The graph-based complexity metrics are superficial, e.g. number of nodes and edges, depth and width. More complex correlations between different tasks should also be considered, e.g. the feature correlation between nodes across graph topology [1], or some other metrics summarized in [2]\n\n2. \"By incorporating intents into the prompt, we observed a marked improvement in the agents’ performance on tasks designed to evaluate planning (i.e., Long-Range Planning and Parallel Planning),\" The improvement brought by idea of intents looks similar as goal-based reinforcement learning.\n\n3. Could you clarify why are the proposed metrics, Coverage Rate (CR) and Logical Consistency (LC), particularly suited for evaluating graph-structured tasks over more traditional metrics such as trajectory similarity or success rate alone?\n\n4. The benchmark utilizes synthesized tasks within controlled environments. Could you discuss how well the proposed evaluation and results can generalize to less structured, real-world virtual agent applications?\n\n5. Could you provide more details on the robustness of intent extraction process, especially regarding the handling of subtasks whose intents might be overlapped or be ambiguously defined?\n\n\n\n[1] What Is Missing For Graph Homophily? Disentangling Graph Homophily For Graph Neural Networks. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems.\n\n[2] The heterophilic graph learning handbook: Benchmarks, models, theoretical analysis, applications and challenges. arXiv preprint arXiv:2407.09618. 2024 Jul 12.", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACAqkpCZDA", "reviewer": "Reviewer_DTtP", "summary": "This paper introduces OmniBench, a scalable, graph-based benchmark designed to evaluate multimodal large language model (MLLM)-based virtual agents across multiple dimensions. OmniBench employs a bottom-up subtask composition pipeline to generate 36k tasks with controllable complexity across 20 scenarios. The OmniEval framework introduces Coverage Rate (CR) and Logical Consistency (LC) metrics to assess agents beyond simple success rates. Experiments reveal that current agents, including GPT-4o, struggle with graph-structured tasks, highlighting the need for better reasoning and planning. Fine-tuning on OmniBench improves performance, demonstrating its potential for advancing virtual agent capabilities.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for professional comments and high appreciation of our work! We are encouraged that our research is recognized as laying the foundation for future advancements. We will address your concerns point by point.\n\n**1.Data Quality**\n\nThank you for your suggestion, we conduct additional experiments on OmniAct and AndroidControl benchmarks. We follow the experimental setup in the OS-Atlas paper, using our high-quality graph-based trajectories to train our models: **OS-Atlas-4B** and **UGround-7B-V1**. The results are shown in the following tables. \nOur models achieve superior performance on both benchmarks compared to the baselines, which showcases the **effectiveness of our graph-based trajectories**.\n\n**Table:Evaluation on OmniAct**\n|Models|Type-Web|Grounding-Web|SR-Web|Type-Desktop|Grounding-Desktop|SR-Desktop|\n|-|-|-|-|-|-|-|\n|InternVL2-4B|47.51|51.34|24.39|67.00|44.47|29.80|\n|Qwen2-VL-7B|89.22|85.94|78.58|96.27|94.52|91.77|\n|SeeClick|86.98|75.48|68.59|96.79|70.22|72.59|\n|OS-Atlas-4B|88.56|82.00|73.91|96.51|85.53|84.78|\n|UGround-7B-V1|90.16|86.98|79.85|97.13|94.79|91.89|\n|Omni-OS-Atlas-4B(Ours)|89.96|82.74|74.62|97.64|86.37|85.53|\n|Omni-UGround-7B-V1(Ours)|**91.24**|**87.35**|**80.24**|**97.93**|**95.21**|**92.10**|\n\n**Table:Evaluation on AndroidControl** \n|Models|Type-LL|Grounding-LL|SR-LL|Type-HL|Grounding-HL|SR-HL|\n|-|-|-|-|-|-|-|\n|InternVL2-4B|90.94|84.05|80.10|84.09|72.73|66.72|\n|Qwen2-VL-7B|91.94|86.50|82.56|83.83|77.68|69.72|\n|SeeClick|93.00|73.42|75.00|82.94|62.87|59.11|\n|OS-Atlas-4B|91.92|83.76|80.64|84.69|73.79|67.54|\n|UGround-7B-V1|92.15|87.17|83.29|84.72|78.85|70.31|\n|Omni-OS-Atlas-4B(Ours)|**92.49**|83.51|81.38|84.86|73.81|67.71|\n|Omni-UGround-7B-V1(Ours)|92.37|**87.24**|**83.57**|**84.89**|**78.97**|**70.83**|\n\n&nbsp;\n\n**2.The Rigor of the Conclusions**\n\nWe appreciate your insightful observations. \nWe define the impact of textual order on the model as its instruction sensitivity, conducting experiments with standard deviation as the metric.\nWe construct 10 specially designed test tasks, each associated with three task instructions that are semantically identical (based on the same task graph) but differ in textual order.\nAs shown in the table below, the original MLLMs tend to be less sensitive to instruction variations, but perform poorly overall.\nThough fine-tuning them on navigation tasks enhances the performance, it also compromises the models' robustness to instructions, with OS-Atlas-Pro and Aguvis exhibiting significantly higher sensitivity.\nMoreover, **after incorporating graph-structured task samples from OmniBench into fine-tuning, the models' performance is further improved while largely preserving their robustness**, with Omni-OS-Atlas and Omni-Aguvis exhibiting reduced sensitivity. This indicates that the trajectory data from OmniBench can help models better recognize complex dependency structures embedded in task instructions.\n\n|Models(backbone)|Avg. Sensitivity|\n|-|-|\n|Human|1.95|\n|InternVL2-4B|2.97|\n|Qwen2-VL-7B|2.58|\n|OS-Atlas-Pro(InternVL2-4B)|9.07|\n|Aguvis(Qwen2-VL-7B)|12.90|\n|Omni-OS-Atlas(InternVL2-4B)|3.49|\n|Omni-Aguvis(Qwen2-VL-7B)|2.67|\n\n&nbsp;\n\n**3.Experimental Setup**\n\nThank you for your valuable suggestions. Considering MLLMs without grounding-specific training may struggle to perceive fine-grained UI elements in screenshots, we adopt the setup of A11Y+Screen for them. We also conduct the evaluation with only Screenshot setup for fair comparison. Compared to Table4 and the table below, baseline MLLMs consistently achieved poorer performance in Screenshot setup.\n\n|Models|PP|LRP|LSR|LIF|SDM|CDDM|SI|DI|CDK|DSK|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Qwen2-VL-7B-Instruct|3.2|6.4|2.9|3.5|4.7|3.1|4.5|6.7|5.0|6.8|\n|InternVL2-8B|3.3|5.9|2.8|3.2|4.9|3.2|4.7|6.2|4.1|5.3|\n|InternVL2.5-8B|4.9|6.7|4.8|5.7|5.4|6.8|5.2|6.1|5.9|7.2|\n\n&nbsp;\n\n**4.Minor Weaknesses**\n\nFor more detailed introduction to Cross-Verification, please refer to Appendix B.2.\nWe present additional case studies in Fig1, specify the 20 scenarios in Fig2, and more statistics on OmniBench in Fig3 of the anonymous link: https://anonymous.4open.science/r/OmniBench_rebuttal-A4C7/r3.md\n\n&nbsp;\n\n**5.Questions For Authors**\n\n**To Q1:** \nIn OmniBench, test tasks collected from the virtual environment are initially selected based on a joint constraint of five complexity dimensions. We also manually filter the tasks to improve quality.\n**To Q2:** Additional examples of input and output resources are provided in Fig1 via the anonymous link. These resource types, carefully designed by humans, capture the specific states of the virtual environment. Clearly defining these resources for subtasks effectively represents and constrains state transitions in complex virtual environments, facilitating seamless integration between subtasks.\n**To Q3:** We give examples of how APIs are combined in Fig4 of the anonymous link.\n\n&nbsp;\n\nWe will integrate these experiments and correct the grammatical errors in the next version of paper.", "claims_and_evidence": "Yes, the claims are supported by clear and convincing evidence. The authors' idea of defining task complexity across five dimensions using five graph properties is impressive.", "methods_and_evaluation_criteria": "Yes, I think the methods and evaluation criteria make sense. The bottom-up task synthesis approach is highly innovative. More importantly, this synthesis method effectively identifies and reutilizes recurring subtasks in trajectory data, making it highly efficient.", "theoretical_claims": "The task graph proposed by the authors, combined with the bottom-up theoretical claims, is highly intuitive and I consider them to be correct.", "experimental_designs_or_analyses": "Yes, I have checked the experimental designs and analyses. I have reviewed the authors' comparison experiments between OmniEval and human evaluation, the performance of 12 agents on OmniBench, the analysis of task intent's impact on agents, and the comparison between graph-structured and chain-structured tasks.", "relation_to_broader_scientific_literature": "OmniBench builds on recent advancements in graph-based evaluations for agents. The graph-based evaluator in OmniBench is derived from the relevant design in CRAB (CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents). Additionally, there have been recent works on synthesized trajectories. For example, AgentTrek (AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials) leverages collected tutorials to synthesize over 10,000 high-quality demonstration trajectories.", "essential_references_not_discussed": "I do not find any essential papers that the authors failed to discuss.", "other_strengths_and_weaknesses": "### Strengths\n\n**S1.** The authors propose a novel trajectory synthesis method that significantly alleviates the shortage of agent demonstration trajectories. I believe this bottom-up synthesis approach incorporates a divide-and-conquer strategy, overcoming the challenges of directly synthesizing trajectories in a top-down manner by first synthesizing subtask trajectories and then composing them into task trajectories. Additionally, the synthesized trajectories undergo strict data filtering to ensure high quality. Finally, fine-tuning experiments on OmniBench further validate this approach.\n\n**S2.** The authors' Cross-Verification method cleverly integrates subtask trajectory data with evaluation functions. By leveraging mutual verification between these two types of data, it iteratively optimizes both the synthesized trajectories and the evaluation functions simultaneously. This significantly reduces the expert knowledge required to construct evaluation functions for an interactive benchmark.\n\n**S3.** The authors design a graph-based evaluator on a DAG and introduce two novel evaluation metrics, Coverage Rate (CR) and Logical Consistency (LC), enabling a more reasonable and fine-grained evaluation of agents. This evaluation approach differs from traditional result-based and trajectory-based evaluations and is better aligned with real-world tasks that involve complex parallel relationships.\n\n**S4.** The authors define task complexity across five dimensions using five graph properties, enabling the controlled synthesis of tasks with varying complexity. Additionally, these five complexity dimensions are leveraged to design ten agent capabilities through the composition of them. This approach facilitates a multidimensional evaluation of agents, laying the foundation for comprehensive future advancements.\n\n### Weaknesses\n\nI find the ideas in this submission quite intuitive. The bottom-up task synthesis approach and the representation of tasks as graphs are particularly impressive. However, I still have some concerns about this submission:\n\n**W1.** Data Quality\n\nI am somewhat concerned about the quality of the synthesized data. The authors seem to have conducted training experiments only on OmniBench but have not performed similar experiments on other benchmarks. Although Table 3 presents an ablation study demonstrating the effectiveness of their designed quality control module, I would prefer a more intuitive way to showcase this. For example, the authors could evaluate the performance of agents trained on synthesized trajectories on other benchmarks. This might be more convincing than the results in Table 3 or those obtained solely on OmniBench.\n\n**W2.** The Rigor of the Conclusions\n\nFigure 7 shows that agents often struggling to handle graph-structured tasks. And the authors mention on lines 362–365 that \"most existing agents are predominantly fine-tuned on chain-structured tasks, which may result in their tendency to interpret graph-structured tasks as linear,\" attributing this to the chain-structured training data. Does this imply that existing agents plan the next action mainly based on the textual order of the task instructions? I believe the authors' conclusion is important and could offer valuable guidance for the future development of agents, but the authors should conduct relevant experiments to support this conclusion.\n\n**W3.** Experimental Setup\n\nIn Table 4, the inputs for MLLMs are all A11Y + Screen, and they perform well under this setting. However, these results cannot be directly compared with those of the agent, as they belong to different experimental settings. I am curious about the performance differences between these MLLMs and specialized agents under a fair comparison on OmniBench.\n\n### Minor Weaknesses\n\nI have some minor comments, which might help improve the writing quality.\n\n1. I am interested in the Cross-Verification algorithm in this submission, which applies to optimizing trajectories and evaluation functions iteratively. But I think the details of this algorithm could be explained more clearly.\n\n2. It would be great to highlight more case studies. Although the authors provide quantitative results in Table 4, qualitative analyses of failure cases are also important for understanding the boundary capabilities of agents.\n\n3. The 20 scenarios mentioned in the submission should be specified in more detail. Additionally, the statistics on OmniBench could be expanded, for example, by including statistics on the synthesized data, such as the step distribution of demonstration trajectories.", "other_comments_or_suggestions": "1. When formatting the paper, pay attention to spacing. For example, the spacing around Section 4 appears too tight and should be adjusted.\n\n2. For tables, the caption is generally placed above.\n\n3. There is a numbering error in the introduction, located in the left section of line 122.\n\n4. The authors should pay attention to some tense errors. There are several instances in the submission where the past tense is incorrectly used.\n\n5. The subfigure in the bottom right corner of Figure 1 seems to be inconsistent with the experimental results in Table 4. The authors should revise it to ensure consistency.", "questions_for_authors": "**Q1.** The authors don't seem to explain the source of the number of test tasks for each capability in the main figure. How were these test tasks obtained? Additionally, were the final test tasks in OmniBench manually filtered?\n\n**Q2.** Are there more examples of input and output resources, and how is their matching ensured so that two subtasks can be seamlessly connected?\n\n**Q3.** In Section 3.2, the authors mention composing a set of predefined APIs into a complete evaluation function using a code LLM (e.g., Claude). I'm curious about the specific APIs included and how the evaluation function is constructed. Could you provide an example of how these APIs are combined to form an evaluation function?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8spq4YKPg6", "reviewer": "Reviewer_7v9F", "summary": "This submission introduces OmniBench, aiming to provide a scalable task synthesis paradigm along with an evaluation framework for agent capabilities across ten dimensions. Due to the inherent complexity of agent trajectories, it is challenging to effectively construct large-scale, high-quality trajectory datasets. Existing trajectory datasets in agent community are typically three orders of magnitude smaller than GUI grounding datasets (thousands vs. millions), struggling to provide sufficient supervision to enhance agents' navigation capabilities across various environments.\n\nTo construct tasks more systematically, the authors propose the Task Graph, a novel task modeling approach. They treat frequently occurring subtasks as nodes in the task graph. First, they synthesize subtasks and apply cross-verification to obtain gold subtasks. Then, they synthesize tasks through rule-based subtask composition. This synthesis pipeline not only avoids the need for strong task planning capabilities in top-down task decomposition but also exponentially scales up task synthesis by reusing synthetic gold subtasks. \n\nAdditionally, the authors quantifies task complexity based on five fundamental attributes of task graph (e.g., depth, number of nodes, number of edges, etc.). By constraining the complexity levels of the five dimensions, the authors filter test tasks for evaluating 10 capabilities and reveal the fine-grained performance of 12 agents, providing guidance to the community on the direction of future optimization.\n\n## update after rebuttal\nAuthors' rebuttal have solved my concerns. I think previous score is high enough and I will keep my rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our paper as comprehensive and identifing highly valuable insight, your constructive comments and suggestions are valuable to us. Below is our detailed response to clarify the points you raised.\n\n**Q1: The Effect of Task Intent on Planning.**\n\n**A1:** The task intent is to enhance the planning capability of the planner in a plug-and-play way. We further explore the generalizability of task intents on the **open-source and closed-source models**. (1) For open-source models, we fine-tune OS-Atlas-Base-4B and UGround-V1-7B in the same dataset with and without task intent, respectively. As shown in the following table, incorporating task intent in the training data significantly improves the model's planning performance on OmniBench. This indicates that task intent has the potential to serve as fine-tuning data to guide models in improving their planning capabilities.\n|Models| Parallel Planning|Long-Range Planning|\n|-|-|-| \n|Omni-OS-Atlas-Base-4B|24.2|33.0|\n|**+intent tuning**|**25.7 (+1.5)**|**34.9 (+1.9)**| \n|Omni-UGround-V1-7B| 33.2|31.3| \n|**+intent tuning**|**34.4 (+1.2)**|**32.6 (+1.3)**| \n\n(2) For closed-source models, we use prompts with and without task intent to Qwen-VL-Max, Gemini-2.0-Flash, and Claude-3.5-Sonnet for planners, with UGround-V1-7B serving as the grounding models.\n\n|Models|Parallel Planning|Long-Range Planning|\n|-|-|-|\n|Qwen-VL-Max|21.9|20.8|\n|**+intent prompt**|**24.5 (+2.6)**|**23.5 (+2.7)**| \n|Gemini-2.0-Flash|23.1|22.7| \n|**+intent prompt**|**28.9 (+5.8)**|**26.7 (+4.0)**| \n|Claude-3.5-Sonnet| 24.2|23.7| \n|**+intent prompt**|**30.6 (+6.4)**|**28.1 (+4.4)**|\n\n&nbsp;\n\n**Q2: Compare with Human Demonstration Trajectories.**\n\n**A2:** Thank you for nicely pointing out that Omnibench may obtain high-quality graph-based tasks. To address this, we compare human demonstration trajectories from the GUIAct dataset and synthetic trajectories from OmniBench. We conduct experiments on the OmniAct-Web dataset and apply the same fine-tuning settings for fair comparison. \n\n|Models|Type-Web|Grounding-Web|SR-Web|\n|-|-|-|-|\n|InternVL2-4B|47.51|51.34|24.39|\n|Qwen2-VL-7B|89.22|85.94|78.58|\n|SeeClick|86.98|75.48|68.59|\n|OS-Atlas-4B|88.56|82.00|73.91|\n|&nbsp; +1k human demostrations|88.64|82.34|74.06|\n|&nbsp; +1k synthesized trajectories|**88.71**|**82.50**|**74.12**|\n|UGround-7B-V1|90.16|86.98|79.85|\n|&nbsp; +1k human demostrations|90.23|87.19|80.02|\n|&nbsp; +1k synthesized trajectories|**90.29**|**87.28**|**80.11**|\n\nThe results in above table show that **graph-based trajectories in OmniBench are of high quality and lead to better performance**.\n\n&nbsp;\n\n**Q3: Performance per Complexity Dimension.** \n\n**A3:** Thanks for your constructive suggestions. We report the model's performance across different levels within each complexity dimension in the table.\n\n|Models|D-Easy|D-Medium|D-Hard|B-Easy|B-Medium|B-Hard|I-Easy|I-Medium|I-Hard|K-Easy|K-Medium|K-Hard|H-Easy|H-Medium|H-Hard|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Aguvis-7B|32.8|27.6|24.3|41.2|36.8|30.6|49.5|36.9|25.3|38.4|32.5|27.6|37.9|33.6|29.7|\n|OS-Atlas-Pro-7B|32.3|26.8|23.7|39.1|31.0|25.4|44.3|34.8|21.8|33.9|28.4|24.3|34.5|28.1|25.6|\n|ShowUI-2B*|34.0|28.3|25.6|41.3|32.7|28.2|45.9|36.6|25.4|37.8|32.6|27.4|37.6|32.0|28.1|\n|OS-Atlas-Base-4B*|32.7|29.1|24.9|35.2|32.4|27.6|48.2|37.5|26.7|39.1|34.2|28.9|43.1|38.4|33.2|\n|UGround-7B*|34.1|30.0|27.1|44.6|38.3|32.4|53.0|39.4|27.2|42.3|36.4|32.6|35.7|28.8|25.5|\n\nwhere D means Dependency, B means Branching, I means Instruction, K means Knowledge, and H means Hierarchical. As can be seen, the model's performance gradually decreases as the complexity increases, indicating that the **task complexity is reasonable and independent**.\n\n&nbsp;\n\n**Q4: How to design predefined resource lists for determining subtask inputs and outputs?**\n\n**A4:** Thanks for your interest in the details of our subtask determination. To capture the specific state of the virtual environment, we manually design cold start resources based on the principles of **subtask definition and task transition** to ensure accuracy and facilitate seamless integration between subtasks. To further explain the implementation, we provide additional examples of input and output resources in Figure 1 via the anonymous link: https://anonymous.4open.science/r/OmniBench_rebuttal-A4C7/r4.md\n\n**Q5: How to ensure environmental consistency?**\n\n**A5:** Thanks for your comment, we also believe this verification is important. Since there is an inherent trade-off between environmental realism and consistency, when introducing web-based applications~(e.g., Skype, Mail, DeepL) to the OmniBench evaluation environment, we will **use firewalls to block their outbound traffic and preload the necessary page state** to provide an environment that ensures realism and consistency.\n\n**Q6: Writing issues.**\n\n**A6:** We sincerely thank you for constructive suggestions on expression, organization, figures, and citations. We will correct them in next version.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence. I have reviewed the submittion and did not find any inappropriate claims.", "methods_and_evaluation_criteria": "Yes, I think the methods and evaluation criteria make sense. Considering agent tasks as a directed acyclic graph provides a rational foundation for subsequent task synthesis, task complexity definition, and agent capability design. Additionally, Figure 6 shows that OmniEval aligns with human evaluation, demonstrating that the evaluation criteria also make sense.", "theoretical_claims": "I have reviewed the proofs for the theoretical claims and consider them to be correct.", "experimental_designs_or_analyses": "Yes, I have checked the experimental designs and analyses. As a benchmark, I consider the experiments to be sound and valid. If the authors can further validate the quality of the training data, this submission would have a greater impact.", "relation_to_broader_scientific_literature": "OmniBench provides a more fine-grained evaluation of agents, unlike most benchmarks that focus on task success rates. It evaluates ten diverse capabilities, including dependency identification and long-range planning, similar to benchmarks like MMBench designed for MLLMs. However, in the agent domain, this might be the first attempt I am aware of to define the fine-grained capabilities required for agents.", "essential_references_not_discussed": "I believe the inspiration for evaluating various agent capabilities in the submission might come from benchmarks [1, 2] used to evaluate MLLM capabilities. However, this is not explicitly discussed in the submission. Including such a discussion would make the submission more impactful.\n\n[1] MMBench: Is Your Multi-modal Model an All-around Player?\n\n[2] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension", "other_strengths_and_weaknesses": "## Strengths\n\nS1. The authors propose a data collection pipeline to automatically synthesize high-quality demonstration trajectories and evaluation functions. The author designed three quality control modules to ensure the data quality, achieving a 91% human acceptance rate. The proposed bottom-up synthesis provides a highly valuable insight, which can be important for the community: fully leveraging the repeatedly occurring sub-trajectory in trajectory data. \n\nS2. The authors introduce the concept of the Task Graph, allowing for a systematic quantification of task complexity based on fundamental graph properties (e.g., depth, number of edges, etc.). By constraining the complexity levels of the five dimensions, the authors filter test tasks for evaluating 10 capabilities.\n\nS3. Experiments are conducted to evaluate agents across 10 capability dimensions, revealing the fine-grained performance differences between open-source and closed-source agents, providing precise directions for future optimization.\n\n## Weaknesses\n\nW1. On the left side of lines 430 to 434, the authors state that \"By incorporating intents into the prompt, we observed a marked improvement in the agents’ performance on tasks designed to evaluate planning.\" Additionally, Figure 9 illustrates the impact of intent on OS-Atlas-Base and Uground-V1 in the agent setting, showing a significant improvement in planning capabilities when task intent is incorporated. \n\nHowever, the intent here is actually provided as input to the planner (i.e., GPT-4o), and the authors don’t test the effect of feeding intent into end-to-end agents. If task intent is only effective on GPT-4o, it is hard to prove that task intent is as meaningful as stated in the submission.\n\nW2. The subfigure in the bottom left of Figure 1 shows that the graph-based data is of high quality, but I couldn't find any related experimental descriptions in the article. I believe that comparing the synthetic trajectories with human demonstrations could reveal the quality of the OmniBench dataset. Therefore, I suggest that the authors train the agent separately on these two types of data and evaluate its performance on other benchmarks.\n\nW3. The authors define task complexity from five dimensions in Table 2 but don’t evaluate the performance of agents on tasks within a specific dimension. The experimental results are valuable to the community, as they could reveal whether certain models exhibit preferences for specific types of tasks.", "other_comments_or_suggestions": "1. The metric ND under \"Overall\" in Table 4 has not appeared before. I suspect this is a typo by the authors.\n\n2. Change 'we introduced OmniBench' in line 429 on the right side, 'we observed a marked' in line 431 on the left side, and 'We compared' in line 378 on the left side to the present simple tense.\n\n3. The line spacing around line 266 seems too tight and should be adjusted.\n\n4. In the subfigure at the bottom left of Figure 1, the authors may have forgotten to replace 'x' with the actual values. Additionally, this figure does not seem to intuitively illustrate that task complexity has three levels and its relationship with agent capabilities.", "questions_for_authors": "Q1. How is the \"predefined resource list for determining subtask inputs and outputs\" mentioned in the submission implemented? Does the predefined resource list come from the LLM? What are the design principles for the resources?\n\nQ2. I noticed that the appendix of OmniBench introduces several apps in the environment, including some that require an internet connection (e.g., Skype, Mail, DeepL). How do the authors ensure that the environment remains consistent for each evaluation? I am curious about this because interactive benchmarks [1, 2, 3] typically include only offline apps/websites (e.g., Microsoft Office, VS Code) to ensure a consistent environment for each evaluation.\n\n[1] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents\n\n[2] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments\n\n[3] WebArena: A Realistic Web Environment for Building Autonomous Agents", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this work, the authors present OmniBench, a graph-based benchmark designed for evaluating multimodal virtual agents' capabilities of solving complex tasks. The benchmark is synthetically generated, but in a way that the evaluation conclusions drawn from OmniBench can be generalized to real-world virtual agent applications (such as long horizon and graph-structured computer use tasks). In OmniBench, the tasks are generated in a bottom-up fashion: based on a set of synthesized subtasks, the framework composes multiple subtasks in a meaningful way to form a task, and makes sure the resulting task's complexity is on a desirable level. 

The work also propose OmniEval, a multidimensional evaluation framework, which uses Coverage Rate (CR) and Logical Consistency (LC) metrics to assess agents beyond success rates. 

Reviewers agree that this work's contribution is strong, the benchmark itself as well as the design and procedure to obtain such benchmark are very valuable to the community. 
- Citing Reviewer DTtP (score 5): It has the potential to prompt the community to rethink how to more comprehensively assess the true capabilities of virtual agents. 
- Citing Reviewer 7v9F (score 4): I find the ideas meaningful and thought-provoking ... the idea of leveraging graph-based auxiliary information to enable such evaluation is insightful for the community... I'd be happy to champion this paper.

The majority of the reviewers' concerns were clarification questions and asking for additional analyses. The authors did a good job addressing the concerns, there are a few additional results that I feel particularly made the work stronger:
- concrete examples of the OmniBench data, including subtasks, tasks, and the evaluation function. 
- additional graph-based complexity metrics, and their Pearson correslation with human evaluation.
- results on OmniAct (prior work where data is collected from real devices), providing evidence that the evaluation conclusions drawn from OmniBench can be generalized to real-world virtual agent applications.
- preliminary evidence on the feasibility of using OmniBench data to fine-tuning models.
- additional experiments on incorporating task intent, using both open-source and closed-source models.

Additional AC comments (mostly accessibility improvements):
1. In Section 4.1, the authors could refer to Figure 5 top/mid/bottom rather than just Figure 5, it's a busy figure. 
2. L95 left: `Tab. 1` but everywhere else the author use `Table x`
3. The authors could explain what is `A11Y` when they first mention it (L359 right).
4. It seems Appendix C is empty. 

Overall this is a strong work. I clearly see its value to present to the ICML community and the broader ML/AI community. I recommend to accept.
Decision: Accept (oral)