Computer Agent Arena: Toward Human-Centric Evaluation and Analysis of Computer-Use Agents
Authors: Bowen Wang
Xinyuan Wang
Jiaqi Deng
Tianbao Xie
Ryan Li
Yanzhe Zhang
Junli Wang
Dunjie Lu
Zicheng Gong
Gavin Li
Toh Jing Hua
Wei-Lin Chiang
Ion Stoica
Diyi Yang
Yu Su
Yi Zhang
Zhiguo Wang
Victor Zhong
Tao Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Computer-Use Agent
Visual Language Model
Human-in-the-loop
Evaluation
Primary area: datasets and benchmarks

Abstract: As Computer-Use Agents (CUAs) proliferate and grow increasingly capable, evaluation has become more challenging: static, manually curated benchmarks are narrow in domain, contamination-prone, and environment-heavy, and they diverge substantially from user-driven, real-world evaluation. We present Computer Agent Arena, an open-source platform for head-to-head CUA evaluation and a dynamic methodology that converts human preferences into structured feedback in realistic environments. The system (i) simulates real-world computer use via cloud-hosted, diverse, and dynamic environment initializations and customizations; (ii) ensures authentic, fair comparison by faithfully reproducing open-source CUAs and executing anonymously in matched, controlled environments; and (iii) extends evaluation beyond pairwise preference and correctness to capability- and behavior-oriented signals. Across 2,201 high-quality votes over 12 agents—spanning multi-app interactions, ambiguous instructions, and open-ended queries—we observe striking ranking reversals relative to static benchmarks. Further analysis shows that overall correctness mainly drives human preference; beyond that, agent-human interaction and self-correction boost user preference, even when overall task completion is comparable. Our error analysis reveals agent behavior errors, such as long-horizon memory and fine-grained action failures that static benchmarks fail to evaluate. We also contrast pure GUI agents with universal digital agents capable of tool use and coding, and discuss the trade-offs of these different design philosophies. We open source the full platform, collected dataset, and code of Computer Agent Arena to support future research on the evaluation and development of CUA.

Published as a conference paper at ICLR 2026
COMPUTER AGENT ARENA: TOWARD HUMAN- CENTRIC EVALUATION AND ANALYSIS OF COMPUTER- USE AGENTS
Bowen Wang"* t Xinyuan Wang" th Jiaqi Deng"
Tianbao Xie
Ryan Yanzhe Zhang
Junli Wang Dunjie Lu" Zicheng Gong"
Gavin Li
Toh Jing Hua
Ion Stoica Wei-Lin Chiang
Diyi Yang Yu Su Yi Zhang
Zhiguo Wang
Victor Zhong
Tao Yu th
The University of Hong Kong
Stanford University "The Ohio State University
University of California, Berkeley
"AWS AI
"University of Waterloo
ABSTRACT
As Computer-Use Agents (CUAs) proliferate and grow increasingly capable, eval- uation has become more challenging: static, manually curated benchmarks are narrow in domain, contamination prone, and environment-heeav. and they diverge substantially from user-driven, real-world evaluation. We present COMPUTER AGENT ARENA, an open-source platform for head-to-head CUA evaluation and a dynamic methodology that converts human preferences into structured feedback in realistic environments. The system (i) simulates real-world computer use via cloud-hosted, diverse, and dynamic environment initializations and customizations; (ii) ensures authentic, fair comparison by faithfully reproducing open-source CUAs and executing anonymously in matched, controlled environments; and (iii) extends evaluation beyond pairwise preference and correctness to capability- and behavior- oriented signals. Across 2,201 high-que high-quality votes over agents- Spanniing multi- app interactions, ambiguous instructions. and open-ended queries we observe striking ranking reversals relative to static benchmarks. Further analysis shows that overall correctness mainly drives human preference; beyond that, agent-human interaction and self-correction boost user preference, even when overall task com- pletion is comparable. Our error analysis reveals agent behavior errors, such long-horizon memory and fine-graat fine-grained action failures that static benchmarks fail to evaluate. We also contrast pure GUI agents with universal digital agents capable of tool use and coding, and discuss the trade-offs of these different design philoso- phies. We open source the full platform, collected dataset, and code of COMPUTER AGENT ARENA to support future research on the evaluation and development of CUA.
Website:
httpes//arona.xlaagg.e
Code:
Dataset:
INTRODUCTION
Recent advancements Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated significant potential for building Computer- Use Agents (CUAs) (Anthropic 2025c
OpenAl| 2025d Qin et al.) 2025 Guo et al. 2025 Team et al. 2025). Such agents can perform
various computer-uu use tasks, from web browsing professional applications (Xu et al. 2024b Xie
et al. 2024a Li et al. 2025a Drouin et al. 2024 Wang et al.) 2025). These systems are increasingly positioned for woal-world deployment, making human-centric evaluation grounded in user preferences, safety, and reliability in open-ended settings a prerequisite rather than an afterthought.
Currently, CUA evaluations predominantly rely on static online and offline benchmarks (Xie et al.
2024a Zhou et al. 2024 Xue et al. 2025 He et al. 2024 Xie et al. 2025b) containing human-
"Equal contribution. +Corresponding author.

---

Published as a conference paper at ICLR 2026
Preset desksop & initial
M
a
Websiee
Computer
Camputera
App Customize the inieal desktop
Cocligg
Computer
Mouse
Chrome
GDrive File
Computer
Gitub Gaeee Rapo
Upload Lecs File
Computers with scenarios
! Choose
Setup Initial desktop environment
Correct nott
"CYedt"
Yoao
Briai No
Agunt A
Agant
the desioo so Shol and àt
one better?
User
Ala better
bemer
The
Yoly" charnd
Safe
Evaluate
Write your task prompt
Agenc A GPI4o (20240006)
Step-wise feedback with
Agent B Chuck 1.5 Sonnet (20241022)
Agent identities revealed post-evaluation
4 Observe anonymized agents' execution
Figure 1: COMPUTER AGENT ARENA evaluation workflow: Users select an operating system (Windows/Ubuntu) and initialize the environment via preset or custom scripts. Two anonymized CUAs execute tasks simultaneously in parallel virtual machines, with their recorded visual trajectories presented user evaluation. User preferences and correctness judgments are collected to generate global agent rankings.
written computer tasks with manually designed reward functions. However, these benchmarks are increasingly unable to accurately assess CUA capabilities and systematically neglect human- and weal-world-centric evaluation. The static and limited task domain and evaluation environment leave them vulnerable to contaminationnoveriiie open-ended objectives. They also ignore personalization (different users value different outcomes and interaction styles), underestimate safety/privacy risks, and lack robustness to environment drift (software updates, network variability, unseen apps). Finally, their design often trades off authenticity for reproducibility, offering little guidance on fair head-to-head comparisons or scalable human- centric feedback collection.
and unable to capture real-world dynamism and
To address these limitations, we introduce COMPUTER AGENT ARENA, novel evaluation frame- work allowing evaluating CUAS on authentic computer tasks from real users and assessing agent performance through direct human feedback. To ensure diversity and authenticity, deploy cloud-hosted VM infrastructure with hundreds of prebuilt setups (c.g., preopened apps/web pages and files) and support user-defined initializations and customizations mimic realistic computer use. To ensure fair comparison, two anonymous CUAs execute the assrrproposed instruction in parallel within identical environments, rendering side-by-side execution trajectory for pairwise evaluation. The accumulated preferences converge to a stable leaderboard through Bradley-Tenee ranking model (Bradley & Terry 1952), following the Chatbot Arena (Chiang et al., 2024). In parallel, optional step-wiee wise evaluations of grounding errors, privacy violations. and self-correction behaviors are also collected to comprehensively evaluate CUAs capabilities.
We collected 2,201 filtered, high-quality votes from 1,058 users across 12 CUA models. The resulting leaderboard diverges sharply from static benchmarks-man
notably, several top performers on OSWORL (Xie et al. (024b) are inverted in our setting. Through analysis, we find this divergence stems from (i) broader, more heterogeneous task distribution and environment and (ii) human-centric evaluation criteria that emphasize process quality in realistic environments. We first did rigorous eтror study identifies systematic failure modes that static suites underexpose: (1) tool-integrated agents, despite excelling scripted benchmarks, often underperform in real-user tasks due to tool-selection and tool-use errors; (2) long-horizon horizon memory lapses and plan drift; (3) insufficient information seeking and underuse of clarification; and (4) fine-grained grounding/action-preeii
2

---

Published as a conference paper at ICLR 2026
errors. Beyond agent performance, preference analysis from a human-centric perspective shows that users evaluate the execution process, not just outcomes: agents earn preference through thoughtful planning, meaningful partial progress, error recovery, responsiveness. and judicious CALL USER queries, even when full completion not achieved. These effects are especially salient in open-ended subjective tasks, where correctness is inherently ambiguous and closely tied process quality. Together, the findings surface alignment signals beyond outcome correctness and clarify why agents can both fail more often and still (or fail to) carn human preference.
In summary, COMPUTER AGENT ARENA establishes human-centric methodology for evaluat- ing computer- use agents by converting real-world tasks and real-user preferences into structured signals and stable rankings. Our rigorous enror study diagnoses current agent shortcomings (e.g.. sool-selection/use errors, long-horizon memory failures and plan drift, and fine-grained action errors), while our human-centric preference analysis identifies what users actually care about in deploy- ment-process quality, responsiveness, judicious clarification queries, recovery from errors, and privacy awareness. Beyond revealing ranking reversals relative to scripted benchmarks, open- source the platform, reference implementations, and a large, multimodal, human-labeled preference dataset to facilitate CUA research. Rather than replacing benchmark-drin driven evaluation, work provides a complementary human-centric lens that should not be overlooked in agent design and deployment.
2 COMPUTER AGENT ARENA SYSTEM DESIGN
We design the COMPUTER AGENT ARENA evaluation system shown in Figure [ we detail the platform infrastructure, agent execution interface, and the ranking system that transforms pairwise evaluations into a global leaderboard.
In this section,
2.1 PLATFORM IMPLEMENTATION
Scalable online CUA evaluation requires infrastructure that (i) elastically serves many concurrent sessions, (ii) presents diverse, realistic software and web contexts, and (iii) guarantees fair comparison by running agents in matched environments. COMPUTER AGENT ARENA meets these requirements with a cloud-based stack that exposes fully interactive desktops to general users via a web interface.
Scalable. We extend OSWorld (Xie et al. 2024b) by packaging standardized AMI and deploying it on AWS EC2 behind a dedicated backend service. The service provides on-demand provisioning of preconfigured virtual machines (VMs), parallel allocation for crowd evaluations, and low- low-latency startup through a managed pool. Each session streams a native desktop via a VNC window in the browser, enabling real-time interaction without client-side installation (Figure I Step (1)).
Diverse and open-domain. To replicate authentic computer use, we curated 600+ distinct initializa- tions spanning both software and the web. Concretely, we sample popular sites from SimilarWeb and expand coverage through popular subdomains; we install mainstream applications from Microsoft Store and Snapcraft; and we preload 100+ heterogeneous files (e.g., docx, Py) instantiate realistic workflows [Figure[]] Step (2)). To reduce overfitting to fixed contexts, file-system contents are periodically refreshed. To further support customization, we provide quick-start tools that let users configure environments with minimal friction e.g., uploading files, pre-opening specific websites, cloning GitHub repositories, and applying initialization recipes (packages, datasets, browser profiles) in one click. These initializations and user-defined customizations keep scenarios flexible and aligned with real user needs.
Fair. For head-to-heal comparisons, two anonymized CUAs are instantiated in identical environ- ments: same AMI, software versions, initialization recipe, and seeded configuration. The backend records environment fingerprint (AMI ID, package hashes, and initialization spec) with each trial to support reproducibility checks. Agents execute in parallel to avoid temporal drift; their full trajectories are captured via built-in OBS services and rendered as synchronized replays the evaluation interface. Users then submit pairwise preferences and structured labels (e.g., correctness, safety, efficiency), producing comparable feedback across matched conditions (Figure [ Step 4).
Observable and frienffrindll. All interactions are logged at step level (screens, actions, timestamps) and compiled into visual summaries for rater inspection. The browser-basse workflow lowers the
3

---

Published as a conference paper at ICLR 2026
barrier for non-technical participants while preserving native desktop experience, enabling scalable, diverse, and fair human-centric evaluation at crowd scale.
2.2 AGENT IMPLEMENTATION
CUAs interact with computers through unified action space and API service to ensure gross-modes
compatibility (Xu et al. 2024ct Bai et al. 2025: Guo et al.) 2025; Xu et al. 2024b] Zhang et al.) 2025 Wang et al. (025). ). At each timestep, the agent receives a x280x desktop screenshot and outputs structured function call representing actions such as mouse movement, clicks, keyboard typing, scrolling, or special signals (DONE, FAIL, CALL_
Each session begins with a natural language instruction and proceeds as trajectory of state-action pairs until termination, with full action specifications in Appendix B.3 For model integration, we adopt official frameworks when available (e.g., OpenAI Operator OOpenAl, 2025d). Claude 3.7 Sonnet (Anthropic (025c)) and otherwise use standardized baseline agent that handles screenshot ingestion, prompting, function- call formatting, and environment interaction. This unified pipeline isolates model behavior and ensures fair comparison under identical constraints, including fixed step limits, response windows, and access to prior CoT outputs. To ensure fairness and reproducibility. all open-source CUAs instantiated verbatim from public repository, released checkpoints, default system prompts and tools, inference parameters (e.g., temperature, max-tokens), and tool schemas- so that differences in outcomes reflect model behavior rather than integration variance. This unified pipeline isolates the model, equalizes interfaces and resources, and enables opples-to-apples comparison under identical environments. The details of implemented CUA models are listed in Section 3.1
2.3
AGENT RANKINGS
Each evaluation in COMPUTER AGENT ARENA results in pairwise preference vote: two anonymized CUAs execute the same task in identically initialized cloud computers. and the user selects the better- performing agent declares tie. Following prior work human preference evaluation (Chiang
ct al. 2024] Chi et al. 2025 Li ct al. 2022bb Guertler et al. (025), we aggregate these votes into global leaderboard using an Elo ranking system derived from the Bradley-Terry model (Bradley &
Terry 1952).
Let æ; (m, mB) [M] denote the agent pair in comparison and y/s E {1, 0, 1 the correspond- ing user preference. Each agent m is assigned strength parameter 8m. and the probability that the left agent wins is modeled as:
exp(sm))
Pr(m- > mR) exp(3mL) exp(3mx))
We optimize the log-likelihood of all votes to estimate B, and convert scores to the standard Elo scale via:
Em = 400 login (w + 1000.
To ensure leaderboard stability, we compute 95% confidence intervals via bootstrap and rank agents by the lower bound of their interval. The Appendix C shows full optimization and leaderboard details.
EXPERIMENTS
We present experimental setup and leaderboard results, followed by analysis of task distribution, data validation and cross-benchmark comparisons.
3.1 EXPERIMENTAL. SETUP
Agent Models and Sampling. We evaluate 12 publicly accessible CUAs spanning three representa- tive groups (i) Proprietary computer-use agents with strong benchmark performance: Claude
4

---

Published as a conference paper at ICLR 2026
Sonnet AAthropicc 2025b), Claude 3.7 Sonnet (Anthropic. 2025a). UI-TARS-L5 1.5 QQnn al. 2025
Operator OppenAl| 2025d), and Claude 3.5 Sonnet (New) (Anthropic, 2024). (ii) Open-source CUAs:
OpenCUA-32B (Wang et al. 2025). Qwen 2.5 VL 72B (Bai et al., 2025). and CoAct-1 (Song et al. 2025); notably, CoAct-1 is a universal digital agent integrating API tool calls and code execution. (3) General foundation models. We additionally include strong general-purposs. multimodal models not previously tested in this setting (c.g., GPT-5 OOpenAl 2025b). GPT-4. (OpenAI 2025a), OpenAI 04-mini (OpenAT 2025c), and Gemini 2.5 Pro (Google DeepMind 2025)). All agents follow the unified protocol in Section 2.2 and are sampled with uniform probability during evaluation. For
quality control, agents with <10% correctness in their first 100 votes are removed to preserve data quality and user experience.
Crowdsourcing Evaluation We collected evaluation data from two sources: (1) public users on the COMPUTER AGENT ARENA platform and (2) paid crowd workers Prolific (Peer et al. 2021) a high-quality crowdsourcing platform. Prolific participants were pre-screened for prior experience with LLM tools (e.g., ChatGPT, Claude) with details in Appendix [B3.I] All users had to submit real computer- use tasks and evaluate anonymized agent trajectories via parwise preference judgments and other labels. To ensure that the resulting evaluations reflect realistic usage patterns, recruited annotators from diverse demographic hackgrounds, spanning multiple countries, educational levels, and professions, detailed statistics are reported Appendix B.3.1
Data Filtering To ensure data quality, we applied post-hoc filtering to remove duplicate instructions, off-topic queries (e.g., math problems, chit-chat), and tasks incompatible with GUI-based execution. For further validation of annotation consistency, we conducted an inter-annotator agreement study, sampling 100 human-labeled trajectories from dataset and assigning them to three different annotators for independent labeling. Krippendorff's scores were calculated for each label type, yiclding values of = 0.72 for preferences, = 0.78 for correctness, 0.68 for safcty, and 0.70 for efficiency. These results demonstrate moderateeto-ssrooe confirming the reliability and consistency of our labeled data. Full details of the filtering procedure, including these consistency results, provided in Appendix [B.32]]
level of agreement among annotators,
3.2
MAIN RESULTS
Rank
Model
Elo
Votes
Correct Rate
1700
Clande Sunnet
1167
416
52.0%
Claude 3.7 Sonnet
1140
507
52.3%
1100
CoAct-1"
Operator
1064
511
37.4%
1000
OpenCUA*
1023
100
38.5%
Claude 3.5 Sonnet
1023
425
35.8%
900
GPT-5*
1002
34.3%
o4-mini
895
266
15.4%
500
Qwen 2.5 Instruct
Claure TTTP Cluree Scrnet
pa-ma Mi- Coo, Dpwn 2.5 soumet
Onnn 2.5 VC 728 Instrss
Germer
11
GPT-4.1
837
432
8.6%
12
Gemini 2.5 Pro
829
11.8%
CUA Model
(a) COMPUTER AGENT ARENA Icaderboard with Elo scores, vote counts, and correctness rates.
(b) Bootstrapped Elo scores with 95% confidence in- tervals.
Figure 2: Model performance on COMPUTER AGENT ARENA based on 2,201 pairwise user votes. (a) presents the leaderboard; (b) shows bootstrapped Elo score distributions. (* denotes models with limited votes; results will be updated in later versions.)
We collected a total of 3,418 evaluation votes, including 1,773 from public users and 1,645 from Prolific participants. After filtering invalid submissions and removing low-performing models, we retained 2,201 high-quality preference votes for analysis. In total, the platform engaged 1,058 unique users, comprising 821 Prolific participants and 237 public users.
Leaderboard Results. Figure 2 shows the COMPUTER AGENT ARENA leaderboard based 2,201 pairwise votes from 1,058 users across 12 CUA models. Claude Sonnet 4 and Claude 3.7 Sonnet dominate the rankings with clear Elo margin, followed by UI-TARS-L.5 and OpenAl Operator.
5

---

Published as a conference paper at ICLR 2026
In contrast, general-purgge pureess models like GPT-5 and Gemini 2.5 Pro rank lower, suggesting that
strong multimodal capabilities do not necessarily translate to robust computer-use performance. We also compare the COMPUTER AGENT ARENA leaderboard with four public benchmarks: OSWorld. WebArena (Zhou et al. 2024), Web VebVogr (He et al. (024) and Online-Mind2Weh (Xue et al. 2025).
Figure 3a summarizes how existing models rank across all five leaderboards.
Cross-8endhmark
OSWorld Ablation: Out vs. In-domain Tasks
Cluud Sonnet 4
1050
3.7 Sonnel
UI-TARS:115
Operater
Operatoo
UHARS15
2
1008
Sonnel (Now)
Owen 2.5 VL 728 Instrust
Sonnet
Gemini 2.5 Pro (Expertmenta) ntal))
CAA
Us/Wrrld
Weblesyage?
Onlline kond2edd
Out-of-domain
In-domain
(a) Relative ranks of 12 models across COMPUTER
(b) Elo scores on COMPUTER AGENT ARENA after AGENT ARENA and four public benchmarks. Green splitting 1,000 tasks into OSWorld-In-domaii (left) higher (better);
lower; blank not evaluated.
and OSWorld -OOD (right) subsets.
Figure 3: (a) Fross-benchmant comparison highlights rank inversions among top CUAs. (b) Ablation confirms rankings shift once evaluation moves away from OSWorld templates.
Statistical Validation We validate that observed model differences not artifacts of sampling variance using three complementary tests (details in Appendix D). Bootstrapped Elo scores provide narrow 95% confidence intervals. Permutation tests, implemented by repeatedly shuffling preference labels to form a null distribution, confirm that pairwise win-rate differences are highly significant (p 0.01) with medium-to-large effect sizes (Cohen's simulating effect detection under varying sample sizes, shows that with typical data (>~200 votes per model pair) the probability of detecting medium effects (AFlo x 50) exceeds 0.9. Together, these results demonstrate that the ranking gaps reported in Figures 10b significant and robust.
0.5). Power analysis, conducted by
IIa and 5 are statistically
Task distribution strongly affects leaderboard rankings, revealing the fragility static bench- marks. To examine how task distribution influences agent rankings, we perform an ablation study on the COMPUTER AGENT ARENA dataset. Using GPT-4o as semantic classifier, we sampled 1,000 user-submitted tasks either in-domain out-on-domain. After manual verification, Elo scores for each subset separately. As shown Figure 3b leaderboard orders shift markedly: while Claude 3.7 Sonnet remains top, models like UI-TARS- 1.5 rise under in no-oomain tasks. These findings highlight that static benchmarks often overestimate agent performance by overfitting to narrow task distributions. In contrast, COMPUTER AGENT ARENA captures broader task landscape through crowd-sourced inputs, which is a more reliable reflection model robustness in real-world usage.
recompute
4 ANALYSIS
We analyze COMPUTER AGENT ARENA from four perspectives: task diversity, preference, agent behavior, and error study. This allows us to characterize how CUAs are evaluated in real- world scenarios and to identify key divergences from static benchmarks.

---

Published as a conference paper at ICLR 2026
Instruction Length
Open-ended Tacks
PCA Projection of Task Instructions
100
800
40
400
20-
200
0-
Instruction Perplexity
Trajectory Length
Pappoett (M
2000
1500
Arp. Sergh
1.0
Compukertgetteerr
OSWord
500
Werb/rens Webleyager
(a) PCA projection of task instructions across bench- (b) Instruction-level statistics, where "CAA" denotes marks.
COMPUTER AGENT ARENA.
Figure 4: Comparison of task distributions across four CUA benchmarks. Figure (a) shows the semantic projection of task instructions via PCA, highlighting coverage differences. Figure (b) shows instruction length, open open-endedness, perplexity, and trajectory length across benchmarks, with COMPUTER AGENT ARENA ("CAA") exhibiting higher ambiguity and longer interaction sequences.
4.1
TASK ANALYSIS
Tasks in COMPUTER AGENT ARENA feature broader semantic and domain coverage than prior benchmarks. We analyze the semantic and linguistic characteristics of task instructions across COMPUTER AGENT ARENA, OSWORLD, WEBARENA, and WEB VOYAGER. As shown in Figure a &CAA projection indicates that COMPUTER AGENT ARENA occupies broader and less clustered semantic space, suggesting greater topical diversity. We further compare instruction-level features, including instruction length, apen-endedness, unigram perplexity, and reference- trajectory Figure shows that COMPUTER AGENT ARENA tasks are shorter on average (mean 17 words) yet substantially more umbiguous--reflected by nearly double the perplexity- lack canonical ground-truth answer. This pattern reflects key trend: real users frequently issue concise but underspecified queries, which require agents to infer implicit intent and reason iteratively rather than converge on a fixed end state.
refeeerce-traeectoryy length.
likely to
4.2 USER PREFERENCE ANALYSIS
We analyze behavioral features and outcome labels from COMPUTER AGENT ARENA to identify the factors that truly shape user preferences.
1.0
Win Rate
Correctness is leading predictor user preference, while execution length and latency have negligible im- pact. Building on the prior observation that users value coherent execution, we next quantify how correctness and efficiency metries correlate with user preference. Leverag- ing explicit correctness labels from our evaluation pipeline, we observe a strong linear relationship between model cor- rectness and win rate (Figure Tla indicating that task success is the most influential factor in shaping user judg- ments. To isolate the effect of execution efficiency, we analyze sessions where both agents were marked "correct" and compare their step counts and average latencies. As shown Figure Ilb neither factor shows consistent influence on user preference users do not sys- tematically prefer faster shorter executions when outcome quality is held constant. Overall, these
64
022
60
Number Querles
Figure 5: Win rate grouped by number of CALL USER queries. Moderate query- ing (1-2 times) leads to higher user pref- erence, while excessive or no querying lowers win rate.

---

Published as a conference paper at ICLR 2026
results suggest that while concise responsive behavior may offer slight advantages, correctness remains the dominant signal guiding user preferences in current CUA systems.
User preferences are shaped more by turn-level completeness rather than by final state. We conduct a quantitative case study of 100 selected cases to better understand why cop-performing agents diverge in real-world evaluation. Our analysis reveals that, unlike static benchmarks which assess agents solely based on final states, users on COMPUTER AGENT ARENA often base their preferences on holistic execution quality, As illustrated in Example in Appendix |] to complete the task may still be preferred if they demonstrate clear intent understanding, partial progress, or thoughtful error recovery and self-correction attempts. This preference is especially evident in open-ended tasks with no unique solution, where users value adaptive and coherent behavior over rigid outcome success. These findings highlight the importance of evaluating how an agent performs, not just what it accomplishes, emphasizing the need for turn-level assessment in realistic human-agent evaluation.
agents that fail
Agent-human interaction improves agent preference, while excessive queries hinder perfor- mance. Beyond exccution correctness, buman-agen interaction plays a key role in shaping pref- erences. Unlike prior benchmarks that treat CUAs as fully autonomous systems (Xie et al. 2024a Zhou et al. 2024), COMPUTER AGENT ARENA enables an explicit human-agent interface via the CALL USER action (Sec.2.2) Figure (/swss that moderate querying-tyyicallyy one two calls- correlaesss with the highest win rates, whereas both zero and excessive queries are associated with lower user preference. This inverted U-shaped pattern suggests that users value well-timed clarification, especially for underspecified tasks, yet tend to penalize over-reliance on human input. These results indicate that effective interaction not merely about asking questions, but about demonstrating autonomy and judgment- -tannttt-titssssteeee users increasingly expect from capable CUAs.
4.3 TOOL-INTEGRATED VS. PURE GUI CUAS
Recent CUA work emphasizes nool-aummented architectures that invoke code interpreters or APIs to improve success on structured tasks. Although such agents often surpass GUI-only models on scripted benchmarks, our human-centric evaluation reveals different pattern in real use. For example, CoAct-1 (Song et al. (205) tool- and coding-integrated agent based on OpenAl o3 (OpenAT 2025ch state-of-the-ar results on OSWORLD-VERS VERIFIED (60. 1% success), yet exhibits substantially lower performance in COMPUTER AGENT ARENA, particularly on non-technical tasks. This contrast underscores a benchmark-reall real-world gap: scripted suites tend to reward deterministic tool pipelines, whereas real user instructions are often underspecified and not always tool-benefiting.
The evaluation results of CoAct-1 show two salient characteristics in our evaluation: (1) On tasks natu- rally suited to tooling (e.g., code debugging, technical workflows), CoAct-1 1 is competitive among tech-savvy raters. (2) On general-purposs tasks, tool selection frequently misfires, producing errors invisible in the GUI replay and leading to lower user preference. Notably, its successful trajec- tories are very short (mean R 3 steps), suggesting that tool calls can solve narrow problems efficiently but generalize poorly across open-ended tasks. We trace the gap two factors: (i) Tool-selection bias, where agents over- iver-invoke coding tools on tasks better served by direct GUI actions; (ii) Error amplification, where tool calls produce opaque, nur-surfaced failures that undermine interpretability and user trust.
Our findings indicate that adding more tools to CUAs does not inherently translate into better real-world performance; misapplied tooling can degrade both accuracy and user satisfaction. Two implications follow: (1) Tool definition and selection matter. Future designs should develop adaptive policies that decide when, what, and how to invoke tools- abstaining, clarifying with the user, or falling back to direct GUI actions. (2) Mind the senchmark-real-wootee gap. Benchmarks that overweight tool-centric pipelines risk misrepresenting user needs. By exposing these discrepancies, COMPUTER AGENT ARENA offers a lens to study tool-use strategies and their usability impact in open-ended environments.
8

---

Published as a conference paper at ICLR 2026
4.4 ERROR ANALYSIS
While quantitative metrics such as Elo and correctness provide global view of model performance, they cannot reveal the nuanced behavioral failures that limit agent usability in realistic settings. A key advantage of COMPUTER AGENT ARENA is that human preference annotations surface diverse error modes that scripted benchmarks fail to capture. Many failed trajectories stem from common issues such grounding errors (e.g., mis-clicking button) and planning errors (e.g., lacking clear execution strategy), which reflect base model limitations and can be mitigated with training. Beyond these expected cases, however, we identify three insightful error types that are subtler, more detrimental, and harder to expose with scripted tasks:
Long-horizon memory failures. In tasks requiring long-horizon context or repetitive workflows, CUAs often lose track of key information after many steps (e.g., converting multiple files or compiling season-long statistics). Even state-of-the-an models such as Claude 4 Sonnet frequently drift or forget intermediate goals.
Information awareness. Real-world queries are often underspecified, omitting details such file paths or intention slots. Purely GUI-based agents typically issue speculative commands rather than clarifying uncertainties, leading to compounding errors. By contrast, models like Claude 4 Sonnet and Operator effectively invoke CALL_USER resolve ambiguity, underscoring the importance of interactive mechanisms.
Fine-grained action failures. Even when intent is clear, execution breaks down due to fine- grained control errors, including mishandled scrolling, clicks on non-interactive elements, faulty text editing (e.g., appending instead of replacing). These mistakes, sitting between planning and grounding, often derail entire tasks and highlight the need for new training signals on low-level action precision.
Together, these categories show how COMPUTER AGENT ARENA functions as systematic error discovery pipeline. By surfacing nuanced failure modes beyond correctness, our analysis provides actionable insights for improving memory, uncertainty handling, and fine-grained grounding in future CUAs.
RELATED WORK
Computer-Uss Agent Benchmarks Recent advancements have fostered diverse benchmarks and
evaluation frameworks for assessing computer-use agents. Within GUI agent contexts, evaluations
mainly target scenarios involving web navigation (Zhou et al. 2024 Deng et al. 2023 Koh et al.
2024 Drouin al. 2024) and computer-use (Xie et al. 2024a] Bonatti al. 2024 Davydova et al.
2025 Xie et al. (222aa.. OS-level benchmarks, such as OSWorld- Verified (Xie et al. [202aa] (025b)
and Windows Agent Arena (Bonatti et al., 2024), provide rule-based evaluations of desktop agents performing various GUI tasks. Recent works like AgentCompany (Xu et al.] (024a), AndroidWorld (Rawles et al., 2024), Onldne-Mind2Web (Xue et al. 2025). AgentNetBench (Wang et al. (025) ex- tend evaluations toward longer horizons, leveraging interactive and multi-modal scenarios. Although these frameworks have significantly contributed toward standardized assessments, they remain largely script-based and static. Our platform, COMPUTER AGENT ARENA, further expands upon these ideas by crowd-sourcing authentic user-defined tasks and emphasizing dynamic, human-centin evaluations of computer-use agents.
Human-centric Evaluation Human preference evaluation has become increasingly significant for tasks lacking clearly defined ground truths, such as dialogue agents and interactive applications. Plat- forms like Chatbot Arena (Chiang et al. (204)) popularized large-scale, pairwise human comparisons for evaluating language models, using Bradley-Terry -easr-bbased Elo ranking systems. This paradigm was subsequently adapted to domains like programming assistants (Copilot Arena (Chi et al. 2025)), large audio models (Li et al. 2025b), and text-based gaming agents (TextArena (Guertler al. 2025 These arenas highlight the value of integrating human judgment into model assessment, enabling nuanced distinctions that conventional benchmarks scripted evaluations might overlook. Despite their success, existing preference-basee based platforms primarily focus on isolated, context- indeeeeinneeenndent interactions without grounding evaluations within users' real environments. By bridging the gap

---

Published as a conference paper at ICLR 2026
between evaluation and deployment, our platform lays the groundwork for more ecologically valid assessments and provides a foundation for future research on agents operating in the wild.
CONCLUSION
We present COMPUTER AGENT ARENA, a human-centric framework for evaluating Computer- Use Agents (CUAs) that enables scalable, diverse, authentic, and fair head-to-head comparisons in realistic, dynamic environments. We collect 2,201 filtered, high-quality votes across specialized CUA models and seneral-purposs foundation models on diverse, open-ended tasks, uncovering insights that static, scripted suites overlook. In particular, our error study surfaces systematic shortcomings of current CUAs (e.g.. tool-selection/use errors, long-horizon memory and plan drift, fine-grained action/grounding issues), while our human preference analysis shows that users value process quality, agent-human interaction, recovery from errors, and privacy awareness. We open-source the evaluation platform, reference agent implementations, and a large, multimodal, human-labeled preference dataset facilitate future research on CUAs. Ultimately, we underscore the centrality of human-centric perspective for Al agent design and deployment in real-world use.
ACKNOWLEDGEMENTS
We thank Graham Neubig, Daniel Fried, Shuyan Zhou, Pengcheng Yin, Hao Zhang for their helpful feedback on this work. This project was supported by gift funding from AWS Al and support from the ECS (27212023) and Areas of Excellence Scheme (AoE/E-601/24-N) from RGC of Hong Kong
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work involves the collection of human preference data through both public participation and paid crowdworkers. All participants provided informed consent prior to participation: public users agreed to the consent form before using the COMPUTER AGENT ARENA platform, and paid annotators recruited via Prolific were required to read and accept an explicit consent statement before starting tasks. Paid annotators interacted only through anonymized accounts, and no personally identifiable information was collected or stored. For the final dataset release, we applied a sensitive data filtering process to remove submissions containing personal or inappropriate content, ensuring that all data remain anonymized and
We acknowledge that crowdsourced data may contain demographic and cultural biases. To assess fairness, we conducted demographic study annotators, which confirmed a broad distribution across countries, education levels, and professional backgrounds (Appendix B.3.I).
The dataset and platform are released solely for academic research purposes. They are intended to advance evaluation methodology and alignment for computer-use use agents, and must not be used for surveillance, discriminatory decision- miiiiiinmmaiing.... or other potentially harmful applications. This study conducted in accordance with institutional ethical guidelines, and all authors confirm compliance with the ICLR Code of Ethics. We declare that there are no conflicts of interest or external sponsorships influencing this work.
REPRODUCIBILITY STATEMENT
We committed ensuring the reproducibility and transparency of work. All code for the COMPUTER AGENT ARENA platform, including frontend, backend, evaluation logic, and ranking scripts, will be released under open-source license. To guarantee consistent runtime environments, we additionally provide pre-built Amazon Machine Images (AMIs) for both Windows and Ubuntu, and plug-and-pll and-play agent hub for integrating external models and enabling third-party leaderboard participation (Sec. [.2 and Appendix B.2). Detailed descriptions of data filtering, annotation protocols, and demographic studies are provided Appendix [B3.2 and Appendix [B.3.T] ensuring transparency in data collection and quality control. We also report evaluation cost and throughput analyses in the Appendix H confirming that the framework is both affordable and scalable for community use. Together, these efforts provide reproducible end-to-end pipeline for benchmarking, training, and behavioral diagnostics of computer-us use agents in realistic environments.

---

Published as a conference paper at ICLR 2026
REFERENCES
Anthropic.
Claude computer use.
3 5 models and computer- use 2024.
Accessed: 2025-05-03.
Anthropic. Claude 3.7 sonnet and claude code. Technical report, Anthropic, Feb 2025a. URL
http:s/s/www...nnrroooo..//.oo// 3 sonnet Online; accessed 2025-
09-25.
Anthropic. Introducing claude 4, may 2025b. htt_tttp///w.aattrrpoo.....o.o/p Elaude-4 Accessed 23 June 2025.
Anthropic. Claude's extended thinking.
http://www.atthrppcccooo
A.siblee-oxtennee--hiiii 2025c. Accessed: 2025-05-03.
Shuai Bai, Keqin Chen, Xuejing Liu. Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li. Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, abs/2502. 13923, 2025.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Fender C. Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale. ArXiv preprint, 2024. URL
htttps://apt www.seettitcciii. ar. www.oorgg//oooo...... 2726000411
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. URL htt.:: www.nticscccoocooceooooco
org/Corpus 125209808
Canonical Ltd. Snap store install linux apps using snaps, May 2025, URL https://
snapcraft.io/stcom Snapcraft.io Snap Store home page, accessed 10 May 2025.
Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, and Ameet Talwalkar. Copilot arena: platform for code Ilm evaluation in the wild, 2025. URLhttps::
abs 2502. 09328
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Iims by human preference, 2024. URL httpttts//aaxiv..oom
abs abss/2033 2203. 04 132
Mariya Davydova, Daniel Jeffries, Patrick Barker, Arturo Márquez Flores, and Sinéad Ryan. Os- universe: Benchmark for multimodal gui-navigation agents, 2025. URL htt.:htttss//aaae
org/abs/ 2505. 03570
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-281111 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. URL/atts:://axx org/ abs 0000////03.133 07718
Google DeepMind.
Gemini 2.5 pro: Our most advanced reasoning model, 2025. http:///cccccccloooooooooooeeeea gemi htt.rrooeoo.iiiiiionoaonaaaaaaaa
Available at
https:
Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. Textarena, 2025. http://tsse//aaxx.oo///////a///p
12

---

Published as a conference paper at ICLR 2026
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang. Jiawei Wang, Jingji Chen, Jingjia Huang. Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian. Shen Yan, Shixiong Zhao, Shuai Peng. Shuangye Li, Sihang Yuan, Sijin Wu, Tianheng Cheng, Weiwei Liu, Wenqian Wang, Xianhan Zeng. Xiao Liu, Xiaobo Qin, Xiaohan Ding, Xiaojun Xiao, Xiaoying Zhang, Xuanwei Zhang, Xuehan Xiong. Yanghua Peng. Yangrui Chen, Yanwei Li, Yanxu Hu, Yi Lin, Yiyuan Hu, Yiyuan Zhang, Youbin Wu, Yu Li, Yudong Liu, Yue Ling, Yujia Qin, Zanbo Wang, Zhiwu He, Aoxue Zhang, Bairen Yi, Bencheng Liao, Can Huang, Can Zhang, Chaorui Deng. Chaoyi Deng, Cheng Lin, Cheng Yuan, Chenggang Li, Chenhui Gou, Chenwei Lou, Chengzhi Wei, Chundian Liu. Chunyuan Li, Deyao Zhu, Donghong Zhong, Feng Li, Feng Zhang. Gang Wu, Guodong Li, Guohong Xiao, Haibin Lin, Haihua Yang, Haoming Wang. Heng Ji, Hongxiang Hao, Hui Shen, Huixia Li, Jiahao Li, Jialong Wu, Jianhua Zhu, Jianpeng Jiao, Jiashi Feng. Jiaze Chen, Jianhui Duan, Jihao I Liu, Jin Zeng, Jingqun Tang, Jingyu Sun, Joya Chen, Jun Long, Junda Feng, Junfeng Zhan, Junjie Fang. Junting Lu, Kai Hua, Kai Liu, Kai Shen, Kaiyuan Zhang, Ke Shen, Ke Wang. Keyu Pan, Kun Zhang, Kunchang Li, Lanxin Li, Lei Li, Lei Shi, Li Han, Liang Xiang. Liangqiang Chen, Lin Chen, Lin Li, Lin Yan, Liying Chi, Longxiang Liu, Mengfei Du, Mingxuan Wang, Ningxin Pan, Peibin Chen, Pengfei Chen, Pengfei Wu, Qingqing Yuan, Qingyao Shuai, Qiuyan Tao, Renjie Zheng, Renrui Zhang, Ru Zhang, Rui Wang, Rui Yang, Rui Zhao, Shaoqiang Xu, Shihao Liang, Shipeng Yan. Shu Zhong, Shuaishuai Cao, Shuangzhi Wu, Shufan Liu, Shuhan Chang, Songhua Cai, Tenglong Ao. Tianhao Yang. Tingting Zhang. Wanjun Zhong, Wei Jia, Wei Weng, Weihao Yu, Wenhao Huang, Wenjia Zhu, Wenli Yang. Wenzhi Wang. Xiang Long, XiangRui Yin, Xiao Li, Xiaolei Zhu, Xiaoying Jia, Xijin Zhang, Xin Liu, Xinehen Zhang, Xinyu Yang. Xiongeai Luo, Xiuli Chen, Xuantong Zhong, Xuefeng Xiao, Xujing Li, Yan Wu, Yawei Wen, Yifan Du, Yihao Zhang, Yining Ye, Yonghui Wu, Yu Liu, Yu Yue, Yufeng Zhou, Yufeng Yuan, Yuhang Xu, Yuhong Yang. Yun Zhang. Yunhao Fang, Yuntao Li, Yurui Ren, Yuwen Xiong, Zehua Hong, Zehua Wang, Zewei Sun, Zeyu Wang, Zhao Cai, Zhaoyue Zha, Zhecheng An, Zhehui Zhao, Zhengzhuo Xu, Zhipeng Chen, Zhiyong Wu, Zhuofan Zheng, Zihao Wang, Zilong Huang, Ziyu Zhu, and Zuquan Song. Seed1.5-vi technical report, 2025. URLbttpa
arxiv. org abs htt.s/orraaaa///5...oom 220..
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024. URL/attss: httl:ttps///aaii.ooom org
www./aggg//////oo/o///.////eeaee abs 0000s/244022 22112
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. URL 24C1.1364
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screemspot-pro: Gui grounding for professional high-resolution computer use, 2025a. httt/tttt///aaattt wttp://gggga///5...00033
Minzhi Li, William Barr Held, Michael J Ryan, Kunat Pipatanakul, Potsawee Manakul, Hao Zhu, and Diyi Yang. Mind the gap! static and interactive evaluations of large audio models, 2025b. URL.
https: /ttt:///xxxii org////g//..... 2502 15919
Microsoft Corporation. Microsoft store - download apps, games more for windows pc, May 2025. htt.hhttss///apte.. m crosoft.. wwrosoft.com/hotoe hom/home" 7h1 = =en- US6g1=U5 Microsoft Store home page, accessed 10 May 2025.
OpenAI. Introducing gpt-4.1 in the api. Technical report, OpenAI, Apr 2025a. URL https Online; accessed 2025-09-25.
OpenAl. Introducing gpt-5. Technical report, OpenAL, Aug 2025b. URL https://openai com//ndeex/itroddeiing -attaaiiininng-t---0 gpt Online; accessed 2025-09-23.
OpenAl. Introducing opendaaiinggooenaiii o3 and 04-mini. Technical reportt OpenAl, Apr 2025c. URL. https Online; accessed 2025-09-
25. //tpen/i..oooooeerrrrrrrraaaaaad
OpenAI. Operator, wtt.littsss/ppeee..com/eeeeeeeee
13

---

Published as a conference paper at ICLR 2026
Eyal Peer, David Rothschild, Zak Evernden, Andrew Gordon, and Ekaterina Damer. Mturk, prolific or panels? choosing the right audience for online research. SSRN Electronic Journal, 01 2021. doi: (0.2139/ssm.3764448.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang. Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL bttpst/:: www.iv.org/aaaa. 1232
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop. Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents, 2024. httphtt::// /ttpps://aaaiiii http:///wxxxw..g..a/a/ abs hwwpgg/bb//440... 14575
Similarweb Ltd. Similarweb: Ai-powered digital data intelligence solutions, May 2025. URL htttps://we /wwww/wwwoo/mmmmm simi wwr.aaaawwbbb..... com/ Corporate website, accessed 10 May 2025.
Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, and Caiming Xiong. C 1: Computer-using agents with coding actions, 2025. URL URL httpp::/axxxi....... www./xiww.....aaaaoooooooeeeeeae abs/ 2508 03923
Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, Michael Stern, Brian Clarke, Landon Goldberg, Theodore R. Sumers, Jared Mueller, William McEachen, Wes Mitchell, Shan Carter, Jack Clark, Jared Kaplan, and Deep Ganguli. Clio: preserigne insights into real-world ai use, 2024. wtt URL htt.:s://aaat::. wttp://.rrrrbbb///..eeae
13678
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025.
Xinyuan Wang. Bowen Wang, Dunjie Lu, Junlin Yang. Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhenman Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang. Heng Wang. Diyi Yang. Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foundations for computer-use agents, 2025. URL http:::ts:///ar/ wwtp:/vv..rr...ee erg/ wbw.g/////aaeeeeeeeeeeeeeee /b////////288//. 2tt./22000001232333
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024a.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. ArXiv preprint, 2024b. URL https
//arxiv. org/ abs/ 2404 07972
Tianboo Xie, Jiaqi Deng. Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025a. http:/tsss//.....rgggppppa/
13227
Tianbao Xie, Mengqi Yuan, Danyang Zhang, Xinzhuang Xiong, Zhennan Shen, Zilong Zhou, Xinyuan Wang, Yanxu Chen, Jiaqi Deng. Junda Chen, Bowen Wang, Haoyuan Wu, Jixuan Chen, Junli Wang, Dunjie Lu, Hao Hu, and Tao Yu. Introducing asworld-verified xlang.ai, ai, July 2025b. URL https: ang htw.g.../bllg//o/oo/o/og o-wp:///ooooll--.iiiieieeeee
14

---

Published as a conference paper at ICLR 2026
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang. Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany Benchmarking llm agents on consequential real world tasks, 2024a. URL httphhtsss//axxii..rrrrgeeeeeeee abtp//bs/ 2. 14161
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang. Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. arXiv preprint arXiv:2412.09600 2024b.
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454 2024c.
Tianci Xue, Weijian Qi. Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprini 251::2504.013882 2025. URL Att.:://///......./////aaaaa org.abs//r///////oo/oooo/../ss 204. 01382
Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, and Dongmei Zhang. Api agents vs. gui agents: Divergence and convergence, 2025. URL https:tppp....m htttt:tpp//aaat http://awwwv..././...ee
2503 11069
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In International Conference Learning Representations, 2024.
15

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS IN APPENDIX
LLM Usage Statement
17
System Details
17
B.1
Platform Implementation Details
17
B.2
Initial Setup Details
18
B.3
Agent Implementation Details
19
B.3.1
Crowd-soureing Data Collection Details
25
B.3.2
Evaluation Data Filtering Procedure
26
C Elo Ranking Method
26
Statistical Validation of Rankings
27
D.I
Bootstrap Confidence Intervals
27
D.2
Permutation Tests and Effect Sizes
................................
27
D.3
Power Analysis
28
D.4
Inter-Annotator Agreement (IAA)
28
D.5
Noise Sensitivity Analysis
28
D.6
Step-Limit Ablation Study
28
D.7
Conclusion
28
Analysis Details
28
Setup Distribution Analysis.
28
Generalization ................................
29
E.3
OSWORLD Ablation Study
30
Task Distribution Analysis Details
31
E.S
Instruction and Trajectory Metric Computation
32
E,6
Setup Distribution Analysis.
33
Topic Clustering Details
33
Agent Behavior Analysis Details
34
Software and Infrastructure Resources
35
H Reproducibility and Cost Analysis
35
Additional Results
35
Case Study Analysis
35
16

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
We provide disclosure of how LLMs were used in the preparation of this work. LLMs were employed as a genprae-purpose assistive tool in the following ways:
Writing support and refinement: LLMs were used to assist with phrasing, grammar checking, and stylistic polishing of the manuscript to improve clarity and readability. All technical content, claims, and arguments were conceived, verified, and finalized by the authors.
Data analysis assistance: LLMs were occasionally used to help with exploratory data analysis (e.g., generating Python code snippets for statistical tests, summarizing results of bootstrapping agreement studies), with all outputs verified and validated by the authors.
LLMs were not used for the ideation of research questions, experimental design, or the generation of novel scientific claims. All conceptual contributions, dataset collection, experiments, and final interpretations are the sole responsibility the authors. We emphasize that LLMs are not contributors or authors, and that the authors take full responsibility for the correctness and integrity of all content reported in this paper.
B
SYSTEM DETAILS
B. PLATFORM IMPLEMENTATION DETAILS
can you use the snip tooal to crop the imagg that concains the computer desktop only?
can you use the snip tooll to crup the imaiga that contains the computer diecktop only?
Computer- Use Agent
Conversation
Computer- Agent
Skop Corvetitoon
Full
Step 4/$
Step 2/3
4181
2z0x
Ci
Move 1s, d0 and Drag (1277,79) 78) second.
jand, [30)
Figure 6: User interface of the COMPUTER AGENT ARENA platform. Users are presented with two side-by-side VNC desktops representing anonymized CUA executions. Task instructions appear at the top, while trajectory replays and voting options are shown below.
To support scalable evaluation CUAs under realistic conditions, we extend the OSWorld image and build private Amazon Machine Image (AMI) deployed on AWS EC2. This AMI serves as the base environment for launching on on-demand virtual machines. Upon receiving a task execution request backend service either allocates two pre-warmed instances or dynamically
from the frontend, launches new EC2 instances depending on current load. Each evaluation session involves two such virtual machines to facilitate parallel execution and pairwise comparison.
17

---

Published as a conference paper at ICLR 2026
For real-time human interaction, enable WebSocket Secure (WSS) communication to expose VNC (Virtual Network Computing) interfaces. These interfaces are embedded in the frontend using an iframe-based VNC viewer, allowing users directly observe and interact with the virtual desktop in real time. This mechanism ensures a high-fidelity, low-latency connection, even under concurrent usage. To ensure performance robustness, the infrastructure supports multi-region deployment and load balancing, which minimizes cold start latency and improves responsiveness during high-traffic periods.
Figure 6 provides visual overview of the COMPUTER AGENT ARENA evaluation interface. On the frontend, we implement the entire user interface in React and TypeScript. We leverage built-in OBS (Open Broadcaster Software) services installed within the VM to record agent trajectories. Specifically, we capture key frames before and after each agent-issued action (e.g., mouse click, keystroke, scroll) and compile these frames into animated .i files. These gifs are rendered sequentially on the user interface to form an interpretable execution timeline. Each frame annotated with overlays (e.g., bounding boxes for clicks, keyboard icons for typing) to help users understand the agent's interaction logic at each step.
Correctriss A
Correct
Correctness
Conrect
Partially Wrong
Partially
Wrong
Whichh one
Ais
Tielboth
better?
Advanced Evailuation [Optional)
Submit Results
Figure 7: Evaluation form interface shown after users review the agent trajectories. Users select preferred agent (or tie), mark correctness labels for each agent, and can optionally provide comments.
To support pairwise evaluation, the anonymous agent trajectories rendered side-by-side. When agents invoke the CALL_USE action during execution, the frontend renders a user-facing message box present the agent's clarification query. After watching the side-by-side execution trajectories, users are directed to the evaluation form (Figure 7). This form collects pairwise preference judgments by asking users indicate which agent performed better whether both agents were equally good or bad. Additionally, users assign correctness labels to each agent (Correct, Partially Correct. or Wrong), allowing for fine-grained accuracy analysis. Optional text fields also enable users to explain their decisions or report abnormal behavior. All evaluation artifacts, including task metadata, screen recordings, and user responses, are logged for subsequent analysis and benchmarking.
B.2 INITIAL SETUP DETAILS
To ensure realistic diversity across evaluation environments and reduce duplication, we implement a structured pipeline for configuring initial virtual machine states. Following the design philosophy of OSWorld (Xie et al., 2024a), we predefine large number environment setups through scripted configuration files. When user initiates evaluation session, the backend randomly selects and applies one of these predefined scripts to both virtual machines, ensuring a synchronized and comprehensive starting state.
Website Configuration. We curate a diverse pool of popular websites by crawling the top 100 high-traffic domains from SimilarWeb (Similarweb Ltd., 2025). After manual filtering to exclude login-gated or geoblocked domains, we expand the pool by collecting up to 10 subdomains per level site. These subdomains injected into the browser history opened startup tabs within the VM, providing agents with richer content structures and deeper navigation targets during execution.
18

---

Published as a conference paper at ICLR 2026
Table Summary of Initial Environment Setup Components
Category
Source
Example Types
Count
Websites
SimilarWeb
Wikipedia, YouTube, GitHub en. enkiiippdaa. studio youtubo.ouubccomm LibreOffice, VLC, VS Code _docx, -xlsx, .pptx, -ddf, -py, .md
89
Subdomains
Manual Expansion
472
Applications
Microsoft Store & Snapcraft
12
Files
Synthetic Corpus
97
Application Configuration. To simulate realistic usage patterns, we preload the virtual machines with over 20 frequently used desktop applications. These are sourced from Microsoft App Store (Microsoft Corporation 2025) and Snapcraft (Canonical Ltd. 2023), spanning categories such as document editing, media playback, messaging, and terminal utilities. Each application is installed system-wide and accompanied by prewritten startup scripts to ensure reproducibility across instances.
File System Setup. We populate the virtual machine file system with over 100 heterogeneous
files, covering common formats such as docx, xxsxx, -pptx, pdf, .py, and .md. These files simulate working desktop environment and provide agents with realistic artifacts manipulate during task execution. To avoid agent overfitting, we update the file content monthly and apply randomized filenames and folder structures at each session launch.
User Customization Interfaces. Beyond static setups, implement suite of "quick-start" APIs that allow users to tailor the VM environment. Specifically:
upload_frl file allows users to upload local files directly to both VMs.
open_webs tes enables users to preload specific URLs into browsers.
cicne_repo automates the cloning of GitHub repositories via single command.
These tools enhance fidelity and flexibility in modeling users' native computing environments, making the evaluation experience more aligned with realistic desktop usage scenarios.
B.3 AGENT IMPLEMENTATION DETAILS
Interaction Protocol. We model the agent-computer interaction as sequential decision process:
(I, (Sa, ao); (#1, d1)..................... (st,at) ax))
Here, I the initial task instruction provided by the user; 8; the screenshot of desktop at step i; is the action taken by the agent at that step. The agent conditions its decisions 7, the current observation 8t, and a history buffer of n past state-action pairs (n = 5 by default), allowing limited memory and context tracking.
Observation. Each observation 8; is full-color PNG screenshot of the entire virtual desktop, captured at a fixed resolution of x These raw pixels are encoded and sent to the agent model without any additional metadata structured UI representations.
We do not provide DOM trees, accessibility layers, or pre-parsed UI elements, due to their inherent latency and inconsistency across software stacks. By enforcing vision-only interface, we ensure that agents are robust to general GUI variation and layout changes.
Action Space. Our platform adopts a structured action format modeled after the PyAutoGUI API. The action space includes:
Mouse Interactions: click (x, y), double_click (x, y), move (x, y),
scroll (amount)
Keyboard Interactions:
type (text), press (key), hotkey ( (kkeyl, key2,
....)
19

---

Published as a conference paper at ICLR 2026
Meta Actions: DONE (terminate when task is complete), FAIL (terminate if task is achievable), CALL_USER (request user feedback or clarification)
All actions are serialized into a JSON-compatible function-call format that supports easy logging, replay, and debugging.
Action Execution Workarounds. To ensure consistent and reliable agent behavior across diverse operating systems and software environments, introduce key code-level workarounds that address common compatibility and reliability issues in action execution.
On some systems, the pyautogui. write and typewrite functions sufter from unpredictable wehavior-especialle for non-English input or special characters. To improve compatibility and transform these commands into clipboard-
ensure multi-language text entry behaves consistently, based pasting operations. Specifically, we use the pyperc 1p module to set clipboard contents and then simulate ct rt+llv hotkey to inject the text. The original clipboard contents are saved and restored to minimize side effects. The transformation logic is shown below:
def workaround pywp:ooouudppttttopu.u.u.u. wrrit (originorii 8tx) str:
import medee, wwweeeddeeaaa.eeeeeee splitlines (ksepends True)
lines criginsl
pattern re. compilel
=" by-pupppyauto.ull (writel (trite/=ypewrittnt sypewrite) \srl (Ss (
has_write any spattern.maton (line) for 1ine in linea)
if has_write: transformei_lines
("import pyperclipln',
periginal_cl-pheee pyperclip.paata \ala")
linea:
match cestern.match. (line)
match
Saansoormed_iiee agpend
else:
indent, - wpw.:/ee....ppdddeeeeee append (f"(inden pyperrl oopy (f ((repr (text)) An")
match. groupa)
transformed_ httprmed__ineeeepppee
Lranstormed_ append [indentt time aleep (0. \n*)
transformed_1ine wwweeddd_ieeeeepeeeeeeeeeeeeeeee append (f"(""""))) (indent pyyuutoqud hotkay ccrrl'' "ttrl,"" "tt' An")
transtormed www.rreee___eeee_ee_eeeeee aspend (f" (indent pypercllp.copp [tlginaa__llppoardd An")
return "* jojjin (tan
return original_cods
def Horkaround workaround_pyyartooe wwwpppyyytuuuoooooo _orool torigina leriggnnl__ooo _oode: str) -> strt import
inne criginal__ooa spl iplitiina fnes (keepends
patters:
re.compile (r pyautegut, sccoolllsscllsooooooooo (iss (Ns (Ad+)ssd lav (1di) Aax, Nex (1d)) \s+A) \e*51)
has_scroll" any (pat:ern.match (line) for line in lines)
if has_scrall:
transformed_ lines ["inport pyautogui\n")
lines:
match mattern.matoh (1ine)
if match is None: transformed_ liness wtw.::iiesese......... agpend
alse:
x, match.groops
trans Liteedd_e_seeseeeeeedee
append ("" (iinfiident hpyautogui www.oegguiiooooooeeee move moveTo!(x)) ((x), (y)) An")
transformed append (indent paaeannnpyauttuu....m ((llicks) An")
return 'a (Lrnn lines)
return original_code
Listing I: PyAutoGUI Workaround
In certain environments, pwautogui.ccmmm scroll behaves inconsistently when used with three ar- guments (e.g., scroll (clicks, x, y) ), especially on Windows or within virtual desktops. To ensure coordinate- aware scroll behavior works uniformly, split the scroll into two atomic operations: first, moving the mouse to the intended location, then performing the scroll. The adjusted transformation is shown below:
20

---

Published as a conference paper at ICLR 2026
These workarounds enable robust agent behavior under diverse conditions and reduce errors arising from OS -Se-pciific inconsistencies. The transformation logic is applied automatically to all agent- generated code prior to execution within the virtual environment.
Execution Interfaces.
Agents interact with the virtual machine through two API calls:
get_observation () - Returns a base64-encoded PNG screenshot
step (action) Executes the given action.
The interface is stateless and HTTP-based, allowing deployment across different VM backends and containerized setups.
Model Integration. We support two modes of model integration:
Official Agent Implementation: For models such OpenAl Operator Clanee-Sonnet- CU, the agent logic is implemented and maintained by the model provider. We interface with their inference server and wrap the input/output into our standardized protocol, translating their tool calls into unified action format.
2. Baseline Agent Implementation: For open-source models or checkpoints without official implementations, offer default baseline wrapper. This wrapper is designed to handle screenshot ingestion, instruction prompting, function call parsing, step-wise planning, and tool-use execution across Ubuntu and Windows environments. The logic also includes configurable termination criteria, retry-on-failure, and error reflection.
All logs, actions, screenshots, and intermediate CoTs are saved for every session to support evaluation and visualization. The framework is open-sourced and modular, allowing easy integration of new models.
Baseline Prompt Implementation. The default inference prompt used in our platform structured as follows:
System Prompt for Baseline Agent Ubuntu
You are agent performing desktop tasks instructed, with knowledge of computers and internet access. Your tool calls will control mouse and keyboard actions on a computer.
### Task Farameters:
Insttrucctionn
inst_innttrcction
**Resolutions:: (resalution)
*ormat Ubuntu
**System Password*:: 'password' (for sudo rights if needed)
### Ubuntu-Specišic Instrustions:
-+Desktoop Path**: //ome/userr/Desttqp
## Observation Information:
Each step provides observation that includes screenshet with these characteristics:
If previcus mouse actions didn't achieve the expected result, do not repeat then, especially the last one adjust the coordinate pased the new screenshot
Do not predict multiple clicks once. Base each action en the current screenshot; do not predict actions for elementa or events not yet visible in the acreenshot.
Launching applications may take some time to appear the desktop. If the screenshot indicates that the correct application has already been elicked, not click it again-wait for it to open instead.
## Tool Call Information:
21

---

Published as a conference paper at ICLR 2026
You are provided with computer-use tools which are defined in the tools clearly, you can use them to perform the task. REQUIRED: Accurate Positioning Base each click/move/drag as precisely as possible the screenshot coordinates. Use visual cues approximate the exact location the target.
REQUIRED: Wait between multiple actiens.
### Response Structure:
Current observation analysis
Results any previous actions
Any adjustments needed based on feedback The tool call you will use
### Requi rements:
MUST USE THE PROVIDED TOOLS EVERY RESPONSE TO PERFORM THE TASK IN COMPUTER. ONLY SKIP TOOL IF YOU HAVE FINISHED THE TASK. You are computer-use agent please keep executing en the computer until the user's query is completely resolved, before ending your turn and yielding back the user. Only terminate your turn when you are sure that the problem is solved.
You MUST plan extensively before each function call, and reflect extensively
the outcomes of the previous function calls. DO NOT do this entire process by making function calls only, this can impair your ability to solve the problem think insightfully.
The system prompt implementation for Windows platform is similar to the Ubuntu platform, only replacing Chatform-Specife Instructions'.
System Prompt for Baseline Agent Windows
You are an agent performing desktop tasks as instructed, with knowledge of computers and internet access. Your tool calls will control and keyboard actions computer.
## Task Parameters:
#*Inttrucioon: instruiint
**Resolution+:: (resolution)
**Platform*:: Windows
### windoxs-$peci Fic Instructions:
**Desktop Fath*:: 'Cr\\Users' Raminisstttin (Desktop"
++Open Terminal Command**: "WintR", then type "end" and press
Enter"
**Application Launch**: Desktop applications require a double- click to open
### Observation Information:
Each step provides an observation that includes a screenshot with these characteristics:
If previous mouse actions dian't achieve the expected result, do adjust the coordinate
not repeat them, especially the last pased on the new screenshot
Do not predict multiple clicks at ence. Base each action en the current screenshot; do not predict actions for elements or events not yet visible in the screenshot.
Launching applications may take some time to appear the desktop. If the screenshot indicates that correct application has already been clicked, do not click it againwar wait for it to open instead.
22

---

Published as a conference paper at ICLR 2026
### Tool Call Information:
You are provided with computer-use tools which are defined in the tools clearly, you can use them to perform the task. REQUIRED: Accurate Positioning precisely as possible on the screenshot coordinates. cues to approximate the exact location of the target. REQUIRED: Wait between multiple actions.
Base each click/move/drag visual
## Response Structure:
Current observation analysis
Results of any previous actions
Any adjustments needed based on feedback
The tool call you will
### Requirements:
YOU MUST USE THE PROVIDED TOOLS IN EVERY RESPONSE TO PEREORK THE TASK IN COMPUTER. ONLY SKIP TOOL IF YOU HAVE FINISHED THE TASK. You computer-use agent please keep executing the completely resolved, before ending your turn and yielding back to the user. Only terminate your turn when you are sure that the problen is solved.
computer until the users query
You MUST plan extensively before each function call, and reflect extensively
the outcomes of the previous function calls. DO NOT this entire process by making function calls only, as this can impair your ability to solve the problem and think insightfully.
Function Call Tools Definition for Baseline Agent
tools =
"type": "function",
"function":
"name":
"desktop "desktoopaaut.matiiie auttomtiionn""
"description"; "Perform desktop automation actions like mouse movements, clicks, keyboard input, and more.", "parameters":
"type": "object", "properties"; {
"action";
"type": "string",
"enum": "key", "hold_key",
"type",
"nouse_move",
"Isft_mouse_down",
"left_mouse_up",
"left_click",
"left_click_érag""
"right_click",
"niddle_click",
"double_click",
"triple_click",
"scroll",
"wait",
"fail",
"done",
"sarr_user"
23

---

Published as a conference paper at ICLR 2026
"description";
"The action to perform. Available actions include keyboard input, mouse movements, clicks, and system operations. call_user is used to call the user for further query."
"coordinate":
"type": "array",
"itens": {
"type": "integer"
b, "description" "The (x, y) pixel coordinates for mouse actions. Required for house_move and left_click_dral and scroll actions
h "duration": {
"type": "integer",
"description": "Duratien in seconds for held key and wait actions."
b eecroll_amount";
"type": "integer",
"description": "Number of 'clicks' to scroll. Required for scroll action."
"scroll_ direotionn:
"type": "string",
"enum": ["up", "cown", "1eft", "right"], "description": "Direction scroll. Requi red for scrol1 action."
h "start coordinate": {
"type": "array", "items": "type": "integer"
b. "description": "Starting (x, y) coordinates for
drag actions. Required for Left_click_drag,
b,
"text":
"type": "string",
"description": "Text input for type, key, and hold_key actions. also be used with click or scroll actions to hold down keys while performing the action."
b,
"required"; ["action")
Agent Sampling To ensure a meaningful and balanced evaluation of CUA performance, curated a pool of 12 models from two categories: (1) public CUA implementations that have reported results on existing computer-use benchmarks, and (2) general-purpose vision-langueet models (VLMs) with sufficient instruction-folloiin capabilities, included assess their zero-shot utility in realistic computer environments.
During each evaluation session, we sample two agents uniformly at random from the available pool. To prevent sampling bias, newly added agents temporarily upweighted until their total vote count reaches parity with existing models. This ensures fair exposure in pairwise comparisons and stabilizes Elo ranking calculations.
24

---

Published as a conference paper at ICLR 2026
To safeguard the quality of user experience and filter out underperforming models, apply lightweight correctness- basssd screening policy: if any model receives fewer than 10% correctness ratings window of 100 user votes, it is automatically retired from the pool and excluded from further evaluation. This policy mainly targets general VLMs lacking GUT grounding, whose performance may be unreliable in visual desktop environments.
Through this hybrid sampling and filtering strategy. we maintain both the diversity and reliability of agent comparisons while preserving fairness across the evaluation process.
B.3.1 CROWD-SOURCING DATA COLLECTION DETAILS
Consent and Ethical Compliance Before participating, all users- both Prolific-based and pub- lic-wery required to review and agree to digital consent form that outlined the nature and scope of data collection. The consent form clearly specified:
The purpose of the study: to evaluate Al agents in realistic computer- use scenarios.
The types of data collected: task instructions, screenshots, agent actions, user votes, and optional comments.
The intended use of the data: for academic research and potential release under permissive license (e.g., CC-BY 4.0).
Voluntary participation and withdrawal: participants could exit at any time without providing a reason, and without loss of compensation or access.
Anonymity and privacy: participants were instructed not to include any personally identifi- able information in their submissions.
All study procedures were reviewed and deemed exempt by our institutional ethics board. For the public deployment, the terms of service shown on the platform homepage served as a binding usage agreement. The ToS reiterated the research purpose, prohibited inappropriate use, and informed users that submitted data may be reused under open academic licenses.
Evaluation Crowdsourcing To collect human preference data for evaluating CUAs, we deployed study channels: (1) unpaid public users on the COMPUTER AGENT ARENA platform, and (2) paid participants recruited via the Prolific platform (Peer et al. 2021), a reputable crowdsourcing service widely used for academic research.
For the Prolific-based study, we designed a structured evaluation task which participants were asked to: (i) submit real-world computer-use tasks they personally about, and (ii) evaluate anonymized trajectories from two competing agents using a pairwise preference interface. Before starting, all participants were required review and agree digital consent form clearly describing the purpose of the study, data collected (task instructions, votes, optional comments), and their right withdraw at any time without penalty.
Participants were pre-screened through Prolific's internal filters to ensure prior familiarity with large language models (c.g., ChatGPT, Claude), ensuring informed judgments. In accordance with the Prolific platform's 's fair pay policy, we paid all participants with the price of 66/hour, which meets or exceeds the legal minimum payment requirement on the platform.
In parallel, unpaid users accessing the public version of COMPUTER AGENT ARENA were offered free usage of the platform and invited to contribute evaluation data on a voluntary basis. These users also interacted with the same anonymized evaluation interface and followed the same task submission and voting protocols.
In total, we collected 682 valid evaluation votes from Prolific, contributed by 481 unique participants who successfully completed at least one voting task. These users spanned a range of demographic backgrounds and were geographically distributed across English-speaking regions. Their contribu- tions accounted for approximately one-third of the final evaluation dataset and provided a reliable baseline of high-quality, vetted user judgments.
To quantify the resulting diversity, we conducted a post-task demographic survey of our Prolific annotators, with the following results:
25

---

Published as a conference paper at ICLR 2026
Participants spanned 20+ countries across North America, Europe, and Asia
47% held a bachelor's degree or higher, with the rest covering high school and associate-level education
Employment status included full-time (32%), part-time (21 2)), and student (26%) popula- tions
Age distribution was broad (mean 33.2 years, SD 9.6)
Ethnic backgrounds included White (54%), Asian (13%), and Black (8%)
These findings suggest that annotator pool reflects wide range of real-world user perspectives. We believe the resulting benchmark captures realistic and representative computer-use behaviors across a broad population.
B.3.2 EVALUATION DATA FILTERING PROCEDURE
To ensure the reliability and relevance of the evaluation data, we performed a multi-stage post-hoc filtering process combining automated heuristics with manual review. The goal was to retain only high-quality, executable, and task-relevant sessions for leaderboard ranking and agent behavior analysis.
Automated Filtering
We first applied set of automatic filters to remove:
Duplicate or near-duplicate task instructions, based on normalized string matching and embedding similarity using CentenceTransformmer (cosine similarity 0.98).
Off-topic or irrelevant queries, such
Non-computer-use tasks (e.g.. "What is 2 2 2?", "Tell me a joke")
Chit-chat or open-ended dialogue prompts
Content requesting hallucinated data (e.g.. "Give me fake email address")
Tasks incompatible with GUI-based execution, including those requiring internet accounts, payment, or system-level changes (e.g., rebooting).
Manual Review To further improve data quality, we conducted a manual verification over the
filtered dataset:
3 human reviewers (including of the authors) independently reviewed randomly
sampled evaluation sessions (totaling ~600) to ensure task executability and validity.
Sessions with technical problems-sut as browser rendering failures, agent output trunca- tion, or corrupted trajectory logs logs flagged and removed.
Votes with both agent outputs missing, or showing formatting artifacts (e.g., empty responses, code-only dumps without interaction), were discarded.
Resulting Dataset After filtering, we retained 2,201 high-quality pairwise votes, each associated with valid computer-use task and two successfully executed agent trajectories.
C ELO RANKING METHOD
To construct dynamic and interpretable leaderboard based human preference votes, adopt
Elo-style ranking system grounded in the Bradley-Terry model (Bradley & Terry 1952), as widely used in other pairwise evaluation settings (Chiang al. 2024 Chi et al. 2025).
Each evaluation session results in a comparison outcome between two anonymized agents m and mp. We denote the i-th pairwise comparison result as:
z; = (m: mf), Vi E (1,0, 1)
where y; = indicates the left agent wins, the right wins, and } denotes a tie (including both "tie" and "tie (both bad)" responses, which make up roughly 33% of all votes).
26

---

Published as a conference paper at ICLR 2026
We assign each agent real- valued strength score 8m, and model the win probability of the left agent as:
exp(2mt)
Pr(m) > m exp(3m:) exp(2m;)
We then compute 3 E RM by maximizing the (regularized) log-likelihood over all recorded votes:
((3) =
log Pr(ss [z
where ties are modeled as the average of the two win probabilities. For interpretability, we convert the strength scores to standard Elo ratings via:
Fm 400 log t0 (e + 1000,
such that all agents begin with an Elo score of 1000 and adjust as evidence accumulates.
To stabilize the leaderboard and report uncertainty, we compute 95% confidence intervals using bootstrapping (1,000 resamples over logs). Agents ranked by the lower bound of their Elo confidence interval to ensure conservative movement and mitigate volatility when differences are not statistically significant.
To better contextualize the Elo leaderboard results, we include several supplementary visualizations that capture vote distribution, model correctness, and head-to-head performance dynamics. As shown in Figure [4] each CUA model received reasonably balanced number of votes, with newly added models temporarily boosted in sampling frequency ensure coverage. Figure rep the user-assessed correctness rate for each agent, reflecting how often their outputs were deemed valid across all tasks.
To examine inter-model interactions in more detail, Figure 16 presents a pairwise heatmap of vote counts between agent pairs, illustrating the evaluation density across matchups. Building on this, Figure 17 shows the empirical win rates between agents, highlighting not just the aggregate strength but also performance asymmetries in specific matchups- instance, some models consistently outperform certain others while struggling elsewhere.
Lastly, Figure 18 visualizes the pairwise win probabilities predicted from our fitted Elo model, offering smooth, interpretable estimate of how likely each agent is to win in head-to-head comparisons. These analyses collectively validate the robustness of our ranking method and reveal fine-grained interaction patterns that go beyond overall Elo scores.
D STATISTICAL VALIDATION OF RANKINGS
To ensure that the ranking differences reported in COMPUTER AGENT ARENA are robust and not artifacts of sampling variance or annotator subjectivity, we conducted a series of statistical validation experiments. Below we detail the methodology, results, and conclusions.
D.1 BOOTSTRAP CONFIDENCE INTERVALS
We applied bootstrap resampling (1,000 iterations) over the collected pairwise preference votes. Elo scores were recomputed in each resample, and we report bias-corrected and accelerated (BCa) 95% confidence intervals. These intervals, visualized in Figure 2b) quantify uncertainty in model rankings. Notably, top-ranked models (e.g., Claude Sonnet Claude 3.7 Sonnet) maintain non-overlapping intervals, supporting the reliability of observed differences.
D.2 PERMUTATION TESTS AND EFFECT SIZES
To evaluate whether observed pairwise win-rate differences could arise by chance, we performed permutation tests with 5,000 random shuffles of preference labels. This generated null distribution of Kendall's 7 correlations and pairwise win rates. The observed global ranking achieved 7 = 1.0 compared null mean of 0..044 (std 0.276), yielding p 10-1 1, In addition, conducted
27

---

Published as a conference paper at ICLR 2026
two-sided 2-tests all model pairs using wootstrap-estimate Elo and standard errors. After Nolm-Bonnferron correction, 95% comparisons remained significant at a = 0.05. We also report effect sizes using Cohen's d, which consistently indicated medium-to-large effects in the most critical comparisons (e.g., Claude 3.7 Sonnet vs GPT-4.1).
D.3 POWER ANALYSIS
We performed a power analysis assess whether the collected number of votes is sufficient to detect meaningful effects. Assuming conservative medium effect size (Á Elo 50, corresponding to 57% win rate), our dataset with n 1, 661 votes provides 0.90 power to detect differences at a 0.05. This analysis confirms that the study is well-powered for medium effects, though smaller differences between closely-ranked models may require additional votes.
D.4 INTER ENTER-ANNOLN ANNOTATOR AGREEMENT (IAA)
To quantify annotation reliability, randomly sampled 100 trajectories and had them independently labeled by three annotators for multiple attributes: preferences, correctness, safety, and efficiency. Agreement was measured using Krippendorff's
Preferences: a 0.72
Correctness: a 0.78
Safety: a 0.68
Efficiency: a 0.70
These results indicate moderate-to strong agreement even for open-ended GUI tasks, validating the consistency human judgments.
D.5 NOISE SENSITIVITY ANALYSIS
To examine robustness under noisy annotations, randomly perturbed 10-30% of votes and recomputed Elo scores. Kendall's 7 correlation with the original ranking remained high: = 0.87 (10% noise), 0.76 (20%), and 5 0.61 (30%). This suggests that rankings remain stable under moderate subjectivity, though extreme noise can affect closely ranked models.
D.6 STEP-LIMIT ABLATION STUDY
Since agent trajectories are capped at 50 steps, we investigated whether this limit biases rankings against hong-horizon strategies. We truncated 100 pairwise trajectories to the first 15 steps and recollected votes from three annotators. The recomputed Elo scores correlated with the original ranking p 0.71 (Spearman), indicating moderate sensitivity trajectory depth. This suggests that step limits may underrepresent agents with strong long-term planning ability.
D.7 CONCLUSION
Together, these analyses provide convergent evidence that our reported ranking differences are statistically significant, robust to sampling variance, and supported by consistent human judgments. While sensitivity analyses highlight areas for future partteeaillllally around closely ranked models and long-horizon tasks-the overall findings are reliable within the current evaluation design.
E ANALYSIS DETAILS
E. SETUP DISTRIBUTION ANALYSIS.
To better understand how real users define computer-use tasks, we analyzed the distribution of initial environment setups selected on COMPUTER AGENT ARENA. The most frequently chosen setups involved Chrome browsers, LibreOffice applications, and custom file uploads. closer inspection of Chrome- based sessions further reveals that users commonly visited domains related to web search
28

---

Published as a conference paper at ICLR 2026
Domain Distribution for Chrome URLs
14.0%
count
42.5%
150
100
100
50
Chrore
LitrOffcc
Googl Workquaee
splodded File
amozon ccmm
poogle corn
gitruut comm
Boca apple com
dopn, com
ebary bwitch org votp fuuggingg fusigingtece sigiiggteecce co
keople com
tae, nytimen com
stack fectrode com
cockkooerioon com
fike com Codeesa sefesanation.
com
GING
Windows
Top-Level Domains
(a) distribution of user-selected environment sctups.
(b) Top domain categorics accessed under Chrome setups.
Figure 8: User behavior analysis on COMPUTER AGENT ARENA. (a) Environment setup distribution. (b) Domain categories in web browsing tasks.
(e.g., Google, Bing), e-commerce (e.g., Amazon, eBay), and technical resources (e.g., GitHub, Stack Overflow). These patterns indicate that real users tend evaluate CUAs utility-driven work- flows-suue such as web browsing, information retrieval, and document editing biiting-rathhr leisure-oriented applications, which highlights practical focus in current CUA expectations: users prioritize performance in high-frequency, productivity- tasks, while evaluation casual or exploratory domains remains underexplored.
rather than on creative
E.2
GENERALIZATION ANALYSIS
Agent Generalization Score
GenScore vs Arena Score Correlation
0.530
bebla 0.00)
0.406
U8-1AR5-1.5
GenScore
Arina Score
Cluude
D.348
0,092
Claude Sonnet
Claide 3.7 Sonnet
UN-ARRS11.5
Claude 3.5 Sonner
Oper ator
Gwenn 2.5 728
0.012
GPT44.
0.008
0.005
640
of-mini
2.5 720
Gamini 2.5
GPT-4.1
GenScore (Generalization Score)
(a)
(b)
Figure 9: Generalization across topics. (a) Bar plot of GenScore for the top agents (sorted descending). higher GenScore indicates both strong mean correctness and low variance across task categories. (b) Scatter of GenScore versus the arena-wide Elo-based Arena Score; we observe strong correlation (r = 0. 0.949, = 0.0001), suggesting that success on the crowdsourced platform tightly tracks an agent's 's cross-domain generalization ability.
While Elo captures pairwise preferences, it does not tell us whether CUA is consistently good across heterogeneous task types. Because COMPUTER AGENT ARENA crowdsources tasks from real users, the resulting distribution is highly diverse (Sec. TOa). An agent that overfits few niches will underperform in the arena as whole. We therefore introduce a generalization score (GenScore) that rewards both high accuracy and uniformity across topic categories.
29

---

Published as a conference paper at ICLR 2026
Using the topic taxonomy derived by our clustering pipeline (Fig. 10a we compute, for each agent and topic E T, a correctness rate
# correct tasks of in t
Ta.
# total tasks of a in t
GenScore definition. topics. We compute:
Let r = (fatheT be the smoothed rate vector of agent a across K = [7]
Σ Pa.t
Ha K
E(far - MMo),
CVa = da P4a
0a = V K
K
HM, Σ, 1/fatt
ma = min Patt.
We then define
GenScore = HM, x max(0, 1 CV. x
me H4
This multiplicative form (i) uses HM to punish low-prrfonning topics, (ii) rewards uniformity through (1- (L-CV), and (iii) explicitly guards the worst-case via m./Pa. Maa-
Figure 9a shows GenScore for top agents; Fig. 96 visualizes its relationship with the platform-wide Elo-based Arena Score. We observe strong Pearson correlation (r 0.949, p 0.0001) between GenScore and Arena Score. This suggests that success on COMPUTER AGENT ARENA is tightly linked an agent's ability generalize across many real-world task types, rather than excelling in a small subset. Concretely, models such as Claude 4 Sonnet and Claude 3.7 Sonnel achieve both high mean correctness and low variance across topics. yielding the highest GenScores. contrast, lower-ranked models show not only lower average correctness but also high -nspersion-indicattiin pronounced weaknesses on certain categories (e.g., spreadsheet or document-editing tasks).
COMPUTER AGENT ARENA's 's crowdsourcing pipeline naturally yields broad, utility-driven task mix, agents are implicitly required to be robust generalists. GenScore makes this requirement explicit and provides a compact diagnostic: models that score well on the platform also maintain balanced performance across categories. Going forward, we argue that reporting such generalization metrics metrics-alongside global fittertter reflects what users actually need from CUAs in the wild.
CUA performance varies substantially by task category, with open-ended tasks amplifying model differences. To examine task-specific performance, we cluster user-submitted instructions into six semantic categories using the Cilo pipeline (Tamkin et al. (022)). We then recompute Elo scores per category across all models (Figure TOp, and observe substantial Elo margins depending on task type. Open-ended categories like Information Retrieval and Content Creation yield the largest performance gaps, Conversely, while more deterministic categories like UI Navigation and OS Operations yield tighter Elo differences. These findings underscore the importance of evaluating CUAs across a broad spectrum of task types. Uniform benchmark scores may obscure large model disparities that only emerge under complex, interactive, and underspecified conditions prrccieos-prrciiellyy the kinds of tasks real users often submit.
E.3 OSWORLD ABLATION STUDY
To examine the impact of task distribution on agent performance, we randomly sampled 1,000 user- submitted tasks from COMPUTER AGENT ARENA and used GPT-4o-mini to classify them according to semantic similarity to OSWorld. We prompted the model with a binary choice prompt comparing each task to curated of OSWorld exemplars. As shown in Table only 20.7% of tasks were deemed in-domain, while the majority (79.3%) were classified as out-of-domain, highlighting the broader topical spread of real user inputs in COMPUTER AGENT ARENA.
30

---

Published as a conference paper at ICLR 2026
Hive archical Tank Categury Gistrlbuioo
Claude Sornel
Ui-TUR--1
Clivado Sonne
Opunator
Obann
Hubding qual Onlinn Act arm
Information
Orille Insmraii
Cl Scone
Vage sERi2
Content
Speratane
Ceerzon
Daiida Custeiiaatin
Infommaion Rebteral
Technical and Domain Spacifo
Content Craatio
Navigasion
0S Operations
Miccalianmos
(a) Hierarchical topic clusters of user-submitted tasks.
(h) Elo scores of models across task categories.
Figure 10: Analysis of task taxonomy and model performance by category. Figure (a) shows sunburst visualization of hierarchical task categories derived via topic modeling. Figure (b) presents model Elo scores grouped by task type, illustrating how task taxonomy amplifies performance differences across CUAs.
Table 2: Task distribution by semantic similarity to OSWorld (based on GPT-4o-mini classification).
Category
Count
Percentage
In-Domain Tasks (similar to OSWorld)
207
20.7%
Out-of-Domain Tasks (diverse / open-ended)
793
79.3%
Total
1,000
100.0%
OSWorld Ablation Study Classification Prompt
Compare the following task with the set of tasks from OSWerld. Task to analyze: "[task)"
Here is a whole set of osworld tasks:
bosoorld_task_cxampees
Rate the simi larity of the task to analyze with the OSWorld tasks scale:
Similar (the task very similar to some of the tasks in OSWorld, maybe just a little different)
2 Not similar (the task is different from tasks in OSWorld)
Provide only the number
your answer.
E.4 TASK DISTRIBUTION ANALYSIS DETAILS
To compare task distributions across benchmarks, we sample task instructions from COMPUTER AGENT ARENA, OSWorld (Xie et al. 2024a), WebVoyager (He et al. 2024), and WebArena (Zoou et al. 2024j. We embed all instructions using the text fextteembedding-- 3-smal1 model and project them into two-dimensional space via Principal Component Analysis (PCA) visualize semantic coverage. This highlights the broader and less clustered semantic spread of real user instructions in COMPUTER AGENT ARENA.
31

---

Published as a conference paper at ICLR 2026
(Wimeer Louct Conredt
40.0%
Clebk
Correct
Dap Ciberraa t-mmaaa viner VOnE maou
Thee Offirrncs Distibuton (Minner minus Correc
20.0%
Oven 2. Foond
Oreni
20.0%
80.0%
100.0%
Rabe
(a) Scatter plot of agent win rate and correctness rate.
(b) Distribution of step and lateney differences for agent pairs with both marked correct.
Figure 11: User preference analysis based on agent behavior. Figure (a) shows positive correlation between correctness rate and user win rate across agents. Figure (b) illustrates that differences in trajectory length and response latency do not significantly affect user preference in correctness controlled settings.
E.5 INSTRUCTION AND TRAJECTORY METRIC COMPUTATION
To compare the complexity and ambiguity of user instructions across benchmarks, we compute four metrics uniformly for all tasks: instruction length, proportion of open-ended tasks, unigram perplexity, and average trajectory length.
Instruction Length. We tokenize each task instruction using standard whitespace and punctuation rules and count the number tokens. The results are aggregated to compute the average instruction length per benchmark.
Open-Ended Task Detection. To estimate the proportion of open-ended or subjective tasks, we employ GPT-4o-mini classifier using the following prompt:
Open-ended Task Classification Prompt
You are an expert at analyzing computer-use task instructions to determine if they are open-ended or subjective tasks.
A task is considered open-ended or subjective if:
1. has multiple valid solutions or approaches
2. The success criteria
not strictly objective
3. requires creative or subjective judgment
4. The outcome can vary based on personal interpretation
5. the answer cannot be evaluated by simple rule-based judgement
Please analyze the given task instruction and respond in JSON format with:
"is jis_ope_ended"" true/false, "confidence"; 0-1,
"explanation"; "brief explanation of your reasoning"
For each task, we collect the binary open-endedness decision and compute the share of positive cases within each benchmark.
32

---

Published as a conference paper at ICLR 2026
Unigram Perplexity. We smoothed unigram model measure the lexical ambiguity of task instructions. The corpus-wide token distribution is first collected, and perplexity for each instruction is computed
Perplexity(x) exp
log Pr(w))
t=1
where Pr(w;) count( (w; +1
total+VV with V denoting the vocabulary size. Higher perplexity indicates more
unpredictable and context-dependent language, suggesting more ambiguous under-specified input.
Trajectory Length. For COMPUTER AGENT ARENA, we compute the trajectory length of each agent-task task interaction as the number executed steps (actions). We include only trajectories marked as correct by human evaluators. For OSWorld and Web Voyager, we extract golden trajectory lengths as reported in their official benchmarks. WebArena is excluded from this comparison due its lack of standardized trajectory ground-truth.
E.6 SETUP DISTRIBUTION ANALYSIS.
Opmaemee FOR
Domain Distribution for Chrome URLs
70
count
92.5%
60
60
50
2014
40
20
Deiane CIPP
Relaaded
ansazon couts
ponple com
gittcci
apple com
bwit oarhh
couyy com
beenn kogglle corn
seaci avwrtow..cco fow..com
vilce bac fnstacart com
blsembere q.com
Rytimes cath ona, comm Stebib com velg, com
atriows Puin
VLC
YoUTUbw
Top-Level Domains
(a) distribution of user-selected environment setups.
(b) Top domain categories accessed under Chrome setups.
Figure 12: User behavior analysis on COMPUTER AGENT ARENA. (a) Environment setup distribution. (b) Domain categories in web browsing tasks.
To better understand how real users define computer-use tasks, we analyzed the distribution of initial environment setups selected on COMPUTER AGENT ARENA. The most frequently chosen setups involved Chrome browsers, LibreOffice applications, and custom file uploads. A closer inspection of Chrome-based sessions further reveals that users commonly visited domains related to web search (e.g., Google, Bing), e-commerce (e.g., Amazon, eBay), and technical resources (e.g., GitHub, Stack Overflow). These patterns indicate that real users tend to evaluate CUAs on utility-driven work- flows such web browsing, information retrieval, and document editing dditing-rthher rather than creative or leisure-oriented applications, which highlights practical focus current CUA expectations: users prioritize performance in high-frequency, productivity-cent tasks, while evaluation in casual or exploratory domains remains underexplored.
E.7 TOPIC CLUSTERING DETAILS
To investigate how model performance varies by task type, we apply unsupervised topic modeling to group subm-ttb instructions into semantic categories.
Following the Clio pipeline (Tamkin et al. 2024). we first encode all task instructions using Sentence- BERT embeddings. These embeddings are then projected using PCA for dimensionality reduction.
33

---

Published as a conference paper at ICLR 2026
We perform initial coarse-grained clustering with KMeans, using the Elbow Method to select the optimal cluster count based on within-cluster variance. To refine results, we apply agglomerative hierarchical clustering over the K.Maans centroids using Ward's linkage method, which enables the construction dendrogram for topic hierarchy visualization.
Each final cluster is then manually reviewed to assign interpretable labels. This process results in six-category taxonomy:
Information Retrieval (e.g., search, browse, summarize web content)
Content Creation (e.g., writing, editing, preparing slides)
Technical and Domain-Specific Tasks (e.g.. code debugging, finance tools)
UI Navigation (e.g.. click, type, scroll workflows)
OS Operations (e.g., file renaming, screenshot, task switching)
This hierarchical clustering approach provides a flexible way to capture semantic variation across real user instructions, allowing us to conduct category-speciff evaluations and better understand the strengths and weaknesses of different CUAs across diverse task profiles.
Logt
preade
Win daseen to gayz (-18 prouudo Roud 113, Br-0.004
Redere jiriping
-00
20
(a) Impact of repeated action ratio on performance.
(b) Impact of reflection frequency on performance.
Figure 13: Relationship between agent behavior patterns and performance metrics.
F AGENT BEHAVIOR ANALYSIS DETAILS
To better understand how users perceive different agent behaviors, we analyze two specific patterns observable from execution trajectories: repetition and reflection.
Repetition Ratio. We define the repetition ratio as the proportion of actions within a trajectory that belong contiguous segments near-identical steps. Specifically, scan the action sequence (01, og, (r) and mark segments where three or more consecutive actions share high structural and parametric similarity -eg. repeated scrolls or identical clicks. Repetition often reflects failure to adapt or unclear intent from the model.
Reflection Frequency. We count the number of steps whose Slain-of-Though (CoT) outputs contain introspective or self-corrective language (e.g., "think", "retry", "mistake", "adjust"). These steps are labeled as reflective, indicating the model engaging self-monitoring or explicit reasoning before issuing commands.
Regression Analysis. For both behavioral metrics, we perform linear regression against model win rate and correctness. Repetition negatively correlates with preference and correctness, confirming that users tend to penalize agents that exhibit stuck or redundant behavior. Conversely, reflection is positively associated with both outcomes, suggesting that agents that appear deliberate and thoughtful are better received, even if they do not fully solve the task.
This analysis complements the visual trends shown in Figure E.7 and supports the design implication that lightweight behavioral signals can serve as useful proxies for assessing agent robustness and user trust.
34

---

Published as a conference paper at ICLR 2026
G SOFTWARE AND INFRASTRUCTURE RESOURCES
Our platform and experiments were built using combination of open-source and proprietary software components. Below, we list the key software tools and dependencies used throughout the development and deployment of COMPUTER AGENT ARENA:
AWS EC2: Hosts the virtual machines used for agent execution and screen recording.
AWS S3: Stores large artifacts such task logs, screen recordings, and output traces.
OBS: OBS software for recording agent trajectories.
Vision-Languuae Models (VLMs)): Accessed via OpenAl, Google, Anthropic APIs; local inference used for selected open models (e.g., MiniGPT-4, LLaVA)
Frontend: React (TypeScript), TailwindCSS for styling
Backend: FastAPI + PostgreSQL for logging, orchestration, and user/session management
Data Analysis and Visualization: Python (NumPy, pandas, matplotlib, seaborn, scikit- learn)
custom components, orchestration scripts, and evaluation UIs will be released as part of our open-sourced codebase to facilitate reproducibility and community adoption.
H
REPRODUCIBILITY AND COST ANALYSIS
Full Reproducibility Commitment. We release all assets under an open-source license, including:
End-to-end platform code (frontend/backend, evaluation logic, ranking scripts)
Pre-built Amazon Machine Images (AMIs) for both Windows and Ubuntu, ensuring consis- tent virtual machine environments
A plug-and-play agent hub for integrating external models and enabling third-party leader- board participation
Evaluation Cost and Throughput. We provide detailed breakdown of the per-evaluation cost:
Each comparison runs two agents on t3. .medi um EC2 instances for approxi- mately 15 minutes, costing about S0.02 USD per session.
Compute:
API usage:
Using Claude 3.7 Sonnet reference, each session consumes about $0.72
USD.
Annotation: Paid annotators are compensated at $0.50 USD per qualified evaluation.
Therefore, the total estimated cost per evaluation is approximately $1.24 USD.
Summary. We believe COMPUTER AGENT ARENA delivers a reproducible and affordable evalua- tion pipeline with high community utility for benchmarking, training, and behavioral diagnostics.
I ADDITIONAL RESULTS
J CASE STUDY ANALYSIS
To complement our quantitative evaluation, present a qualitative case study showcasing how real users interact with agents in complex, multi-step computer-use tasks. The example below illustrates a scenario where the user requests the agent to encrypt PDF document using specific password. This task requires not only navigating a real desktop interface but also verifying the success of encryption through file reopening.
35

---

Published as a conference paper at ICLR 2026
Count
5.0
435
434
373
Battle Count
100
Quee 2.5 VL 728 fostruet
Cluude 3.7 Sunnet
Operrato
Claude 3.5 Sentet (New)
Ul MA-TARS-1 1.5
GPT-4.1
Gambd 2.5
agent
Figure 14: Number of votes for each CUA model.
80.0%
Corredt Rate
40.0%
19.2%
37.0%
20.0%
19
14.4%
I6
Cleudo 3.7 Sonnet
U1-TARS-1.5
Oberator
Cludo 3.5 Sonnet (Now)
Owen
Gemini 2.5 Pro
GPT-411
Jnserect
Agent
Figure 15; Aggregate correctness rate for each agent judged by users.
36

---

Published as a conference paper at ICLR 2026
agent1
Qwen 2.5 V. 728 Instruct
Claudo 3.5 Sornet (New)
Coude 3.7 Sonnet
Gemini 2.5 Pro
Operator
1.5
GPT-4.1
64-minn
Cloude 3.7 Sonnet
86
76
54
80
Owen 2.5 VL 728 Instruct 63
76
65
75
55
71
78
70
Operator
48
agento
(New)
50
UI-TARS-L.5
76
51
33
40
GPT-4.1
76
55
45
55
66
50
28
20
04-mini
48
10
Figure 16: Pairwise battle count heatmap: each cell (1,j) indicates the number of votes conducted between agent, and agentz.
Model Opponent
Cleude 3.5 Sonnet (New)
Owen 2.5 VL Instrict
Claude 3.7 Sonnet
Gemini Pro
Operator
MI-TARS-1.5 1.5
GPT-4.1
minl
Cloude 3.7 Sornet
0.75
0.87
Operator 0.32
0.52
0.52
0.83
0.86
0.88
0.91
0.8
UI- UI-TARS-1.5 0.35
0.46
0.74
0.93
0.7
Model A: Winner
0.6
Owen 2.5 VL 728 Instruct
0.40
0.60
0.4
o4-mini
0.15
0.14
0.13
0.24
0.53
0.58
0.76
0.3
GPT-4.1
0.05
0.13
0.33
0.64
0.2
Figure 17: Pairwise win rate heatmap: each cell (i, indicates the proportion of wins where Model A (row) outperformed Model B (column) based on user preferences.
37

---

Published as a conference paper at ICLR 2026
Average Win Rate Againstt All Other Agents (Assuming Uniform Sampling and No Ties)
0.7
0.58
Average
0.4
0.1
Cleadee Sonnar
Operator
Up TART---133
Clode 3.5 Sonnut (Now)
Given 2.5 728 Insruct
GPT-4.1
Genoei 2.5 Pro
Model
Figure 18: Average win rate of each agent against all others under uniform sampling and no ties.
The case highlights several key agent behaviors: interpreting user intent, identifying relevant UI elements within real application (PDF-XChange Editor), and executing sequential steps with feedback loops (e.g., reopen verify). Such real-world interactions are difficult assess through scripted benchmarks but are essential for evaluating robustness and usability in practical settings.
This case exemplifies how our platform supports in-the-wild evaluation by embedding agent actions, user instructions, and full desktop context into unified assessment flow.
Example 1:
Can you help encrpyt this pdf tile with password "jhsvejvakj"2 want this contract file to keep secret, please help me do and PLEASE HELP ME VERIFY THE ENCRYPTION BY OPENING IT' AGAIN.
Agent I: UI-TARS-1.5 (Lose)
Agent 2: Claude 3.7 Sonnet (Win)
Agent
Partial Correct
Agent 2: Correct
Step
Step 1
10.08
Thougle: In Fromt me PDF docoment ticke "Project Timeline: Roadrop:" and pusswrr protect begin,
Thought: The nerr asking enerypt PDF file that to he Project want verily
Timeline Readmap password "jhsvejsakk". encryption works by respening the file. Let me help this task.
upper corner. which the fins step in eneryping the file, From there, I will be sble in find the options related to enerypion settings.
Action
In PDF-XClaage Editor, which the application conently open. nead to: I. Save the PDF with password encryption Closs the file 3. Reopen to verify the encryption
Prisllogii.cicck44...8 click447 702, 55 385, buttons (Mff'
Step2
encryption option Hile > Save As with security options.
or possibly
38

---

Published as a conference paper at ICLR 2026
with password, you XChange Editor open with your Project Timeline Roadmap document. Let me guide you through the sieps encrypt it and then verily the encryption works. Hiss, take scresnshot the
help you encrype
Action
prantoge screenshott)
Step 2
Thought: jost opened the
curverlly storching export
optica. Based on my experience, the eneryption foature roaally tacked away within the export function, so I'm going to click export the left sideher now.
Tirecline
Action
Pyssilogni click(49 689, 593,626 burkoe="left")
Phen
Prene
Proer
Step
Me--1
Thought:
notice there's "Protect" tah the menu bac. Ler's click that first
the eneryption options:
Action
pyantogu click/200, button= Jett')
Step 3
Thougl:: Let's another screenshot the protection options:
Action
pyautogu screensbut))
Thought: I took close look the contents of the export menu, and realized that this option doesn't actually provide PDF encyyrion festure close camrent expect mens
Step 4
rettink our approach, searching for the cornect eneryption option in the File mema.
Action
pysuioogi click(6261887 187, hutoos-"let"")
Step
(The screanshot is the same as the previoes one)
Project Readivpp
Thougir: Let me summarize the current sinuation: I'm currently working in the PDF XChange Editor. and I just spent time looking for encryption feature expoct menu, can't find the option the File menu, thought might be good idea to check the "Protect" button in the toother the of the interface. This button should help find fusctionality add password pentection PDR.
looking
wrong place.
11.55
Action
Thought: sce the Protect open, but don't dinect password encrypiion option this ribbon. by looking the "Security Properties" sectior:
Pynutogui clicia 209.317, 56.374 buston left'))
Step
Action
pyautogui click/963, 115, buston=
Step 5
Thougla: Let's take another screenshot what options uppear:
Action
pyaulogui screenshut))
Step 6
Thoughr: lookeed through File dadn't find encryption optine, nociced a 'Decumest Security Propenties" button the which etactly what I nzeded. Clicking un it opened the PDF encryption settings windre, allowing me continae configuring passward protection for the document.
Action
Pyostogui, www.loggi.ciccco6t.....4 969999964999 122.637, buttoos 'left')
Step
15.15
Thoughtt Great! see we're the Document Properiees dialog box and "Security"
39

---

Published as a conference paper at ICLR 2026
alteady left panel. up password encryptio:
"Change Settings... to
Action
Psattogu click/177, 200, button=
Step 7
Thouglr: Let's ses what security options we have now:
Action
Psaulogui screenshut))
Step 8
nest step is set the encryption method. To make this file more secure, nead to locate the drvedown menu labeled "Securiyy Method" and then choose appropriate encryption method
Thought: can dreument propertios window right now,
This way, effecrively safeguard this importurr
documtent.
Action
Pysulogui clicki 709. 565, 200.769, bution="left
Step1
Vili
Thought: I notice that even though I clicked "Change Settings..." the dialog dida't V change. elicking the dropdoww next "Security Method:" where
curreutly says "No Security::
Action
pyantoga click/827, 200, bustoo="left" "loft"
Step 9
I've finally reached the ctucial Mtpp selecting encryption methud. dropdown Security Method, there are several eneryptoon schemes wailable, Considering both security und computiblity. I believe the Passwerd Security optim is the most suitable choice, allows for precise control option pooceed setting up password protection.
Thoughh: Afler going thruuhh all
Thought: Let's what security options are available:
Action
prantogu.srresshott
permissions. Nest, I'll click this
Step 10
Action
Pyallogui 0lcck/636.25,
lleft")
Step& 8
Thought:
dropdown menu security options.
select
"Passworl Security":
Action
Peanlogei click671,, 245, bustom="left "left")
need fill the passward "jhsvejvak"" in the "Require password open the
Thoughu: I noticed that the passward socurity settings window is now open.
Step 11
document" hhis passward crucial step for protecting the documeas, confirm
it's important input accuntely. After that, the password again, as this will ensure that the document's eneryption security is properly establisted.
Thouglr: Now let's see we get pussworr dialog:
Action
peanlogei screenshot()
Action
Pyastogui, click 333.297, buttoss 'left')
Step 12
Step
(The screanshot is the same as the previoos one.)
and the interface wailing input password "jhsvejvakj. The passwerd labeled "Require password open the documeut" scems to be specifically designed for prosecting document access. am focusing on entering this passward accuraeely. is crucial for security the documeni.
Thouglr:: currently
process
40

---

Published as a conference paper at ICLR 2026
Action
Pysologui clickr45,, cwcck4g
168, 392.637, bution="leff
Step 10
(The screenshot the
one.)
noticed that the paseword inpur labeled "Decument Open Passoven"" enptt.. should click on this box und type in "Jhsoojvak"" as the document opening setting the
Thought
the password in
poesword, Once I've completed this I'll be able
passwont.
Action
Pysstogui 014,,4476625, hutkoe="leff"
Step 11
Thought: Exeellent!
doenmeut" Enter the paseword "Jhsvejaak" in the Docament Open Paesword
for "Reguire password open
fiekl 3, Confirm the poseword the Confirm fiekd Let's start checking the "Require passwond" option:
as
one)
good me start over need to set the paseward in the 'Document Open Password" box, This password will used to open the document, so should enter "jhevejvakj" and fleish step.
Thought: jiut sccidentally clicked wrong spot,
Action
pountage click,367, buttate"left
need fellow sotting permission password.
Step 13
Action
the Decument Password field:
P Pyslttgui www.ooguu.ccccc6444885333 845, 637, leett")
Action
Step 12
prantage click/755, button
(The screenshot the same os previeus one)
Step 14
Thoughr: It szems like there's a litte hiccup in the pasewood actup process. I'm staring at the enagyy input box, the screen, which supposed be for the This
Let'stype password "Jusvejokk"":
documents opening
hut the system ise't respandlnng the "Dacument Open
Action
souse lag the insertaee. Password" imput box again make suze it's activated so can continae entering the password.
write "Jhavsvvaki"
Step 15
Action
Pynutogui click 700.621, 309,56, button- "lett")
Thougir: Now clckk the Cunfirm pussword field:
Step 13
Action
click,755, 335, buston"lleff"
(The screenshot is the same as the previsus one.)
Thought: The passward impul box fimally and ready Now. need to the specified pasaword "jhvvejsukj" It's importani input correcily since this is the fins stpp in settnn the password to protect the document.
Step 16
Thought: And the confirmation preswond:
Action
Action
Pyastogui typewriil jeavejvakj')
mannlogei "hovgiaa))))
Step 14
Step 17
(The screemshot the same as the previous one.
Thought: Now let'kllcck "OK" apply the poseword security:
Thought Let me sseess the aination here I'm the paseaordd security settings interface, the imput bux sill greyed out and unresponsive. It seems that the Open
Action
click(762, 629, buttome"left")
system
bit isput again octivate it. Only can ester passwerd "jheeejokk""
click
Step 18
Action
Py gyuaogu..cksse6ek.... clc.ooo
"lfft"-leeft)
Thought: Now lat's click "OK" the Document Properties dialing settings:
Step 15
Action
(The screemshot the samo as the previous one.
paantoge click/851, 585, buttat-
I've alrendy eatered the password inpuz interface, and now need to impur "jhveejakk" into the "Document Open Paseword" Seld. This step cracial passoced
Thoughr:
Step 19
key to open enerypted docnment eater each character accurately.
security, doccstent.
on this
Action
Action
Py Pysstogul typewrtoe""sssveeeeee jasvejoaky')
clicke45, 55, button="lef")
Step 16
Step
(The screenshot the as previous one.)
Thought: Let's take screenshot to see the File menu options:
Thougir: Lmm, something feels off here. The password box right frunt of me. bui I've nied zew times and can't seem get the password entered coneetly. like the input lield fully activated. more inpur-rcady stare, and thes I'll be able to enter the passwoed "Jhveejakk" sasothly..
Action
Upon cleser "Docunentt Open Paseword"
Prautoga screessho)))
an
Step
Action
Pysstogui.ciccttttoo clc1 332.308, button (eftt)
Step 17
41

---

Published as a conference paper at ICLR 2026
(The screenshot the same as previsos one.)
Thought: found myself stuck in a cycle of capeatedly entering the paeswoed, and this was clearly potmleess Taking step back. I realized that the current securily settings window was quite chattered and hard to navigste, makinn it difficult for uperate effectively.
here. close this window and mevalinle clearer method for encryping PDF So, decidded to click the close burton in the upper right corner and restert the entire encryption
wasting
more
Action
Proslogui click/933, wwi.k/9.iclick/993....
168, 33626, busio==llefi
Thought:
see we notification that saya "Socurity sottings
applied until the docnment is saved." Let's click "OK" on this notification:
Action
Plantage click917,, 389, buttoo=
Step 18
Step 22
Thought:
on Document Properties dialog:
Action
Poantoga click/851, 585, buttoo="lettt)
Step 23
Thought: try's to save document. click the menu again:
Action
prantgg click:45, button='lett)
Step 24
Thought: screenshot the
just closed peevious window, and staring document propertes interface. Frook quick glance at the drapdows main lcheled "Secuity Method," which should allow me to select the appropriate encryption option. wasting wrong place
Thought:
Action
paanlogai screenshuti
make don't on dropdoun meau.
incoerectly
go shead
Step 25
Action
Pysitouui wlwckcc0c9.6666 769, butto-"left"" "lett")
Step 19
Open
Thougha:
open. click "Save"
with the security settings
Action
around propertios window the docement diaally found the right place! the dropdown memn under Secuiity Method, there several optices, and now noed to select Password Security to oäd proswved
Thought:
just
pennlogei clicke40, 149, bution="leti "lel")
Step 26
protection the PDE option eleady visible, I'm going right now stst coufiguring the encryption sottings.
Thouglo: Let's take sercenshot to see any confimmation dialeg appears:
Action
Action
Pa Pyoutogul click6677.2444
llett')
peanlogai screenshot))
Step 20
Step 27
42

---

Published as a conference paper at ICLR 2026
Project lireeline Hoadinup
Thougft:
see that doceiment heen "(Secured)" in the tille bas. This indicates that the password security has been
showing
Thought: I'm now looking the inserface for passwoed security setiings, and it's time the password "jhsvejvakjj" Let first check the 'Ducumeni Open Password" ingut box to make
successfilly applied. Now, let's verify that the eneryptivn works by clowing and
roopening the file. First, let's close docament by clicking "x" the
selected stane, that type on the keyboard, the characters go directly into the box, With this prepuration. I'm confident ihat I won't encounier any issues like I did last time.
document
Action
Action
pyaulogei 192ck, 187, bustom= "lef')
wwwp:ppiiillcci0oo...0o.o 509.90 56, button=llett
Step 28
Step 21
Thougit: Let's if we need confimm closing:
(The screemshot is the same as the previous one)
Action
pynnlogei screemshutt)
Thought looks like the passward input fiekd is all set now. I'll ahead and "jhsvejvak""
continue with encryption process. Since the previous steps have been completed properly, this siep should go smoothly is well.
opening pasewced for the document,
Step
Action
pysuttogii typowrioo (yppwrict (havejokk))
Step 22
(The screanshot is the same as the previoes one.
Thought: After several attempes just now, realized that simply retyping the password wurking, look
calm down and analyse situation: system hasn't registered
of but seems my entries correctly. It looks like the issuc is that the input box hasa't 't been activated properly. So. should click on the "Document Open Password" inpui box in
ingut box right
dast make This time, need be panient
inpur, then get right the fins cry.
"jhsve
Action
Thought: Ghent, the decument now closed. Now let's reopan verify the
py pysutogui
www.loggiicicck/6633..33 clickr663 310,549 buttons (left")
passwoed protection works. Let's click the File
Action
Step 23
pyantogu click:45, button- 'lett'
(The screenshot the same as previoos one.
Step 30
The password field is right in frunt of me, and after several astemppes earlier, I've figured out the cornect appreach. Now. I need to enter the passworr do it in
Thougir:
Thouli: Let's take screenshot the File
'jhooejvaj"" into the Decument Open Paeeword box, making focus and complete
go. The system just waiting for to input step sccurately.
Action
pyantogu screcnshott)
Action
pyastogui wyw.wuuiitppppwiiteeoo (itssissa)))
Step 31
Step 24
(The screanshot is the same as the previous one.)
Opon
Thought have now arrived at the confimmation step for the password setup. It's hit annoying that the password hot not responding, However, based previons experieneo, activated, need to click on the "Document Open Presword" box gain focus, which will allow me to enter the password, Once this step is completedd, we can proceed with enerypting the docament.
neually due the input
being properly
Action
pyyutogui clickt 364.72, 359.011, buttos- "ifft"
Step 25
(The screemshot is the same as the previous one)
Thought: Perfect! I can the document in the Recent Files list. Let's click
open li::
Thanght: that checkhox restricting editing printing is already possward inpur
Action
checked, which good. However, main focus right
field, as the PDF encryprion can't procead without the preeword being set. 1 me concentrate on the "Docmment Open Passwoed" box and click oa so con enter password.
mannlogai clicko673, 225, bitton="lef")
Step 32
Thought: Let's screenshot possword prompt appears:
43

---

Published as a conference paper at ICLR 2026
Action
Action
pyoulogui click(663885 (pw.logpiclccck666.566
851, 353,297, bution=
pylnllgei screemshutt)
Step 26
Step 33
(The screenshot is the
previous one.)
Thought: I'm cumenily selling the confimmation pasoward step, interface feel a hit aluggish, However, reminds me that the key encrypting a PDF lies in these two passwoni imputs. Even though the system is respunding slowly. I need passwced field. Once this step finished, the ontire eneryption process will nearly complete.
Opch
parient and complete entry "jhevejakk"" in the confirmation
Action
Pyaslogui wyw.:upiiypppwiitttece. (iaossiai))
Step 27
(The screenshot the same as the previous one)
Thonght: Hmm. let me analyze the current simuttion. I'm stuck in the PDF encryttion settings interface, where (here's a passwced input box that I just can't realised
Thought: Perfect! We see that the document password protected Iheme's prompt asking password. enser the password ("hooejval''
I've been stuck mindlesdly repeating inpur, but never actually getting any where. Now I've come to understond the probleme the input bos is there. hut it hase'' been activated property, which is why ispur isn't heing repistered. So here's what I'll dic I'll
right.
vicious syclo-just
Action
"Document Open Passward" box first make receive ingut, then I'll enter "Jhsvejvak"" socurately, Sometimes, reall pays to bea bit parient and nekle problems step by step.
to
peuntoge elick6670, 371, huttam="left") psanlogui writer "jsvvivkkk"
Step 34
Action
Pyustogui ppuulouu.clicssaa5....77 clc.)16857714, 331.319, bumou-lett"
Thougho: Now click "OK" to submit the passward:
Step 28
Action
Pyanlogei click1658, 413, "leff"="left))
(The screemsoot is the same as the previous one)
Thought: I've came to realize that I've been repeating the same mistake over and again. Each time. simply type thought to actual
Step 35
paseword without giving hand. problems quite simple:
input isa't activoted! It's like milking to thin air, no mattrr how many times try, it's all in vain. Right now, need to pause for a momest and make sure activate the "Pocament Open Passwand" ingut properly. Only the passward extry process. Let's take it step by stop and flest resolvv the issue of the input box.
Thougir, Let's take a screeushot to see the document opeus:
Action
begin
pyantagu screenshott)
Step 36
Action
Though: Perfect! The documert has successfully opesed after entering the cornect password. We can see that it still shows "(Secured)" in the tile bur, conferming pasewerd peotection is working properly.
Pyastogul ppautogu..ccccut6o...kkm click166638851111 310,549, butto=
Step 29
Action
Summary What We Did:
password protection your Project Timeline Roadmap PDF with the password" favejraky" 2. We saved the doenment with the secuuity settings applied 3. We verified that the protection works by: Closing the document Reopening which peompsed for passward Successfully entering the paseward to sccess the protected document
We successiully
Your P1I now encrypted and can only be apened by somcone who knows the passward "jhsvejvakj" The encryption uses 256 bit AES security. which strong encryption standard. Anyane trying
this document will view contents.
enter conrect password they won't there mything else you'd like me to help you with reganding this docmmenn or its security?
Thought: The previous steps were a bit chaotic, and after malyzing the simuttion,
it seems I've fallen into trap. I've been repeating the passaar entry process
without addressinn the
bock, realized that
property
clicked the DDcument Open Passward" input Now, passward security
settings window still open, and I nead to click on the isput hox to setivite it
fisst. Ouly then will be able to enter" jhsve jaayy as the password cnable the document encryption feature.
Action
Example 2:
Due to work requirements, the following changes need to be made to the document. All "challenges" are changed to "difficultics". and the changed text is highlighted in red for my convenience to view.
Agent 1: o3 (Lose)
Agent 1: OpenAl Computer-Use Preview (Win)
44

---

Published as a conference paper at ICLR 2026
Agent 1: Partial Correct
Agent 1: Partial Correct
Step
Step 1
Iou
Thought Planning tindtsplace task The user wants to reglace all occurrenees of "challenges" with "diflicllree" in a Libreiilee Writer document, highlighhing the
Thougla; Replacung" "calleenges' with "difficutiie" red
Action
I'm thinking
Find Replace dialeg (Chl+H) replazement
up: find "challenges," replace is rod. Altematively. I could find "diffeatiee" aftrward and style it rod. For now, I'll open Find & Replace and praceed from there. It sounds strighhforward!
"difaculties""
Psantage hotieyy'sn 'eti',, ''til,""")) 'h")
Step 2
Action
import Pynitogui impvet time pyantogui JeyDonn ctrl)) gulkkyDDownl (yautognlkeyUgg(hh) pyutngu.eeytiieenm
peanto-
Step 2
Action
tlmeps)
Step 3
Action
import pyautogui gui.typewrite( challenges', interval=0.01)
import
time
pyauto-
Bi
Step3 3
Action pyautogui hotkey( ctrl', 'f')
Step 4
Thougir:: Planning replace action
change "challenges' to Find Replace dialog, highlight
difficallies" LibreOficce
text red. The immediate sext step is fill in the Replace field with "ditficultis." I'll also open the Format option set the text color red, and then click Replace
All. Essentially, I'll need to click the Replace enty box
"ifficuliies"
docusing on approtimate coordinates within the dizlog. Let's proceed with
that click action.
Action
import pyantogui import pyantogniclicks51. 254)
Step 4
Thoupis: Reattempinng shortout for Replace dialog
Action
pyaulogui hotkey" 'etr'. 'b')
45

---

Published as a conference paper at ICLR 2026
Step 5
Thought: Exccuting find replace plansing to replace instancess "challenges" with "difecahies" and hightight the replacements in red. First, I'll use the Find & Replace dialeg, ensring "challenges" is in the Find field. For the
Replace field, type "difliculties" Next, need using Formot... button. After that, click 'Replace it will only change "challenges" and not "challenge "I Lat's get started! Adjjstingg formatiing settings In the screenshor, I notioed that the Fontit... buttin is greyed after all,
choracter color It's crucial note
Thougia: Dismissing bunner access toels
Action
pyaulogun c21ck/1218, 143, butto=="left')
clickable! I'll by ryping "difheulties" Replace input. Then, I'll click Format... botton choose I'll change the Foot color to After contlrming. I'll hit Replace All to apply the
Step 6
Fifects, where
changes, The curser visible in the Replace field, indicating it's active, so can go ahoul type" diffiicliiios.
Action
import pyautogui import yyuilogailtppeeriic. hypewrite difficuliies' interval=0.00)
Step 5
Thougla: Opening TFind and Replace" dislog
Action
Pyanlogei hotkey''ati ""W"
Step 7
'Sarch Styles and replace text with rod color. seems like need to change replaced words to roll instead of highlighting the backgrand. think I'll ma the 'difficulties" the 'Replace' Geld. then replace
Thuught: Checking formoning aptions I'm looking into
Find and Replace feature
specify font color in formatting, Wait, button for Replice looks greyed out.
to clarity that for using "Format..."
option Planning sest formatsing I'm figuring out how to highlight changed text while replacing the Formal option available, makes sense set it beffore doing Replace
both simultanxussl. clicking "oomoo...." in the Replace Gildd to set to red. Looking the screemshos, I'll nced estimate the cocedinates for clicking the buttons. The Find/Replace window shows the format bution around (620. 501 bui might need to adjust if coundinutes!
stop
duesa't right away. Let's stay flexible
Action
import pysutogui import pyullogu.mmweelolB. duration=O..)
Thougia: Opening "Find and Replace" dialog box
Action
Step 6
pyaulogui click/77, buiton='eef') leef')
Step 8
Bi oud
3
Action
import pyautogui import time crautogui.click(66
505)
Action
clepp))
Step?
Step 9
46

---

Published as a conference paper at ICLR 2026
Tou
Thought: Execuring find replace The screessee shows the 'Find place' dislog in LibrelOffie Writer. The user wants to change "chillegges" to "difficulties and highlight the changes red. The Find and Replace fields pessurablly already I
Action pyautogui. click(( (1,, 319, button='left')
configue the "Format" button. approximate button's coordintees hy clicking seoms conterad, around (618, 497), before clicking, Nest, I'll proceed to click "Replace All' after setting the format.
dialog,
Step 10
Action
import py-ulogui import time pyanlogui (61c 501)
Step
Action
time sleep()
Step
Thought Adjusting Find & Replace settings looks like I clicked the option when nrying open Find Replace, ending up "Replace with format- ting" dislog instead, current goal is to replace instances "challenges" with "difficultie" and highlight them in red using LibreOffice Writer need to check if highlighring means chinging the font color, and I'll set the font color the format options. I'll
"Replace with" field "difficutties" adjusi the formatting accondingly. Setting formatting for replacement have "Replaee with formatring" window open, which helpful. Now, I noed to ser the click the 'Fom.
Foni Effictts by choosing red for the font color. The next the top and click window: savigate the tab socond My mouse turger will he aronn the "Fom Efflctt"" aren, rpprotimottlly ai coundimates 450, 182. Let's procsed with that!
Eiffcce""
Action
Emport pyautogui import time pyantngui gpartngni.mwwTTeee moweT 181, dumaioon0..5)
Step
Thougfa: Inputting find and replace ferms
Action
pratioga.ch clicks 159, buston=
Step 12
Thought: Adjusting text in L.bbeffffice To help the user replace "chullenges" with "difficulties' and highlight those changes in red, need open the Find open.
Replace dialog with Cirl-H. Right likely because the user domble-cliekdd the "Foemut" button, First, should close that modal to relurn to the main dialog, where I can set "Tind" to "chullenges" and RRppacee" to "difficulties " I'll also check "Mutch case" and possibly "Whole color for changes To highlight
formatiing
Action pyautogui typeerite cchallenges'
words. 'Settin to adjust the Foot Color under the "Replace"" area. Right now, the Font Effectss tab appears to be selected, but I'm secing options that suggest the Font tab might be active instead. nzed to check if the F'om Effects tab is indeed active. The goal is the Colur red. So, my step
changed
need
Step 13
navigase Foot Eflictt to click the Effects tib.
Action
import pyantogui import pyantogni.clckkeo 181)
47

---

Published as a conference paper at ICLR 2026
Step 10
Thele
Iou iecereeeer
Action
srautoouuicllcc(499, 252, button='left')
Thought: Planning text replacement in LibreOffice need help with the user's 's nek of replacing all occurennes of "chillengee" with "difficsiiies" in Jibeefhiice Writer and highlighting these changes red. We've 've opened the Find Replace dialog by pressing Chrl-H, which shows formaning details, The color dreddown is currently
Step 14
none My nest step to sslest from dropslow to ensure the replaced lest appears in red. I need to carefullly courdinate this action based on the scroenshot's liyout Selecting font color in LibreOffiie I'm
dropdown's location roughly at coordinates (522, 238) on the scroenshut. The srrow puints to the top of the drupdown, n. which looks to be about 150 pixels wide. After this, I'll wamt to select the red color, which may require another click. So, the next
estimating
click that drupdown list based on the screemshot's resolution of 1280x720, The 'Foot color" field is almoot ventered, amoud 460 pixels wide. I'll coordinute my clicks accorilingly
Action
import pyaulogui import time cruukoguiclick(51.. 238)
Step 11
Action pyautogui typewritel difficulties"
Step 15
Thele
Thought: Inepecting grid coorlinutes I'm taking look at the screemshot In the grid the third row looks to have yellowish calom, while the fourth has reds. want focus
red cell, possibly around about 338, 12x12 pisels, calcolltte the coordinates for the red cell row 3, column 6 be (586, 374), Now, I'll plaa the action click at that point.
column 3. I'll estimate
wordinstes,, starting from
each is ruughly
Action
Thougfa: Clicking "Fumnal" set highlightig
import pyautogui import time pyantou..cickic56. 374)
Action
Step 12
psauloge wautiog..lickke6S.. clickr658, 497, buston=
Step 16
Boay Blow
Thought: Replacing text in LibreOflice The user instructed me to replace "chal- lenges" "difficulties" and highlight
red. I've opened LibreOffice Wrise and pressed Curltll access Find Replace, with fommatinng dialog, where I can castomize the fomaat for the replacement. The Foor Effects tab shows Light Red 4, which good for highlightn.. need consimm this red color and ingly Coofiimingg formatting changes nsed
'Replace
Action time.sleep(1)
the Find and Replave fields accund-
bonoa- right corner of the "Replace with formatting" window, It looks like there are buttons for "OK" and "Canoel" My cursor is probubly in the Color field, I fjust need do left click OK to contirm the changes. should allow fiaaiie the fommating sotting I'm adjusting get done!
OK button
Step 17
48

---

Published as a conference paper at ICLR 2026
Action
import pyoulogui import time pyauloguii (ltkk//wwwwiikk888.... 644)
Step 13
Thoupfa: Selecting highlighting formatring
Action
Pynnlogui clicko889, 182, bustom="eft
Step 18
Thouphr: Setting lext replasement The user wants replace highlight the changes
"challonges" with "ithccuties"
I've opened
Find & Replace dialog and lilled in the fields with the approprizue wurds. There's d Tormat..." button in the dislug for setting the iest culor. I see it's highlighted but hasn't been configured can confimm changes/Prpprriii Prepprrin I see the "Hocmar.... button near the bottom right. to proceed. need to click this button, set the forz color to red, and then confirm with OK before using "Replaze All," I'm estimating the voordinates for the click since the screenshox size is 12800x720 estimation. The highlight indicates it's ready to be clicked, so I'll execute the leftchick those coordimates.
need click this button to the fom color to
fommating the screenshos,
button seems
around x 6201,
on my
Action
impert pyantogui import time pyanlogai. click(631), 5211
Step 14
Thought: Selecting red color for highlighting
Action
penntage click(733, 219, hurton-"lefft)
Step 19
Action import pyautogui import time 333 501)
Step 15
Thougla:
Selecting standted red shade
Action
pynnlogai clicke464, 344, bustom= "left")
Step 20
Thiught: Esceuting replacement I'm octing desktop automation agent. and the taak is to change uccureenees "cchlllngee" to "difficullies" while high-
lighting the changel test in red. When I triggeed the Find & Replace dislog LO Writer. I found that the 'Replave with furmatting" tub upen. Now, "Replace" fields and set the font coker Light The next step simply to click OK tn coefine the formatting
specify 'Find'
Thougla: Comeeting selection to red shnde.
procced to complete the replacement Devernining click coordimutes I'm reviewing screenshut that displays the 'Rephace with formalting"" window, and see the right
Action
OK button locnted the
awand conrdiratess 665. think this is good upprosimation for the location since is looks reasomably accunte: My next step is to execute left click at that position to confirm the formatting changes. It's straightforwrd action, just following the visual cues the scrcenshut.
peaniage click4733, 363, hurton-"left")
Step 21
49

---

Published as a conference paper at ICLR 2026
Action
import pyoulogui import time httpu/olooiiicccc88555.. 645)
Step 16
Thoupfa: Selecting rod color, confirming choice
Action
Pynnlogai clicke472, 335, bustom=lefft "lef(")
Step 22
Thoughr: Planning formalling steps The wanks know the next step after secing scccesshot the LibreOffiie and Replace dialog. The
changing the foot color of the replacement text to red. First, Ill click the Formas bullun the dialog. It should be located at the bultom center, around coordinttes from the screenshot. After that, I'll navigate to the red the dialog to respond after the click.
(628, 520) based
Edfects and
I'll
Action
import pysutogui import time pysuiogui. click/33....3 520)
Step 17
Thought: Changing color selection red.
Action
pountoge click(483, 366, burton=""efft")
Step 23
Thought: Clarifying Find and Replace Steps I see that we need to zeplace "hhal- lenges" with "diffiuuties" Libreliftie Writer, and hightight the changes in red. Fins, I'll the Find Replace
Replace bux. should "chilenges" the Find box light
"difficulties"
font
focmatiing options. Once tha's sot, I can click "Meplace All" Afteraard I'll close the character formatting dialog to complete the process. Confirming Formatting up effectively. Insted jusi hitting Enter,' since pressing an unwanted action. The OK button located near the bottom center of the window, approsimately coordinates (850, 640). I'll contirm my click there, and window closes, can Replace tool, Let's this plan!
Changes I need wrop this in Format
it's safer
might
click 'Hoplace All' Find
Thougls: Selecting rod paint instead orange
Action
Action
import pyautogui import time pyautogu.click/851 641)
pynnlogai clicke473, 327, buttom="lfft "left")
Step 18
Step 24
Thought: Approximating button cook Bocdinatts looking screenchot size of 1250x720, Socusing on the dialog positioued around 361. The Format button is
Thougla: Selecting red sbade, confirming choice
Action
near the hotiom right of the dialag. I'll estimate coordinates: the button scems
For the vertical pasitioninng dialog top appears be around y=112, while the Focmat button might around y-480 to y-515 and x around 650. So, my next action is to loft-liek the Formsr button apply the toot formatting.
highlighted just
the bonom
peumigge click500, 326, hurton-"leftt)
Step 25
50

---

Published as a conference paper at ICLR 2026
Action
import pyoulogui import time httpuloooiiccccckii0.... 512)
Step 19
Thoupfa: Selecting prodefinnd dark shade
Action
Pynnlogui clickc482, 365, bustom="left "left)
Step 26
Thoughr: Analyzing test replacementt
The user has screenchot from LibreOtfice Writee, focusing the and Replace dialog,
liffctts
tab showing x Light Red & font color. They's 'e siming to replace "challenges" with "difficultie" while highlighing the new iest in red. The nest steps involve coe-
faming the formatting by clicking "OK" Replace dialog replace instances, Checking the coordinates the bottenn right. I should approximaze them within the given
Replace with formatting window,
then setting up the
screenshot resolution. Confirming fommating action The Replace with lormasting window is ceniered, and the OK butiom the botiom right. My step button confirm changes. This will finalize focmatting for the text replacement. It's strighttoowwadd action. but important for
cleu;
ensuring ev evythhing sei up corretlly for the user's 's needs. Onue I click OK, can proceed execute replacement.
Action
import pyaulogui import wtw.uloouu.cccck8555. 612)
Step 20
Thought: Contirming user's 's preferred red shade
Action
ponntoge screesshot))
Step 27
Thougir: Execusing replacement LibreOffice The wamts change instasees "challenges" "difficulties" their documest, and they need that text highlighted in red. Since the document is already open LibreOltice Writer, see that the Find & Replace window is propared with the correct terms. font color that. quickly check for the highlighted text Choosing text fammatting options I realize that the phrase ghaed in red" could mean either font color or background color. Since the mentioned highlighting. I'll for the font color red avoid confusion. through
need click the Formar button, 'Replace
Four Effects
and
apply
changes.
Thougla: Howering over rol shides, confirming solection
Action
pynnlogai moveTrx-64, 344)
focas that. Next. I'll click the Fonuat butoo, which is at approximattly (626, 520) based on my screenchot estimation. This should help me apply the red font color to the replaced text effectively.
LibreOffice allows me
Format menu,
Step 28
Action
import pyantogui import pyantogni.cick/iat click/626, $20)
Step 21
Action
pyautogui. wosittogui.moveITe To(489, 322)
Step 29
51

---

Published as a conference paper at ICLR 2026
Action
import pyautogui import time }yautogui.clcc((55)) 333)
Step 22
Thouglt: Tnputtingg values
Action
Pynnlogui click 353, buston=lefft) "lef""
Step 30
Thought: Analyzing find and replace steps I need to sel up the Find & Replace
dialog to find "challeegges and replace it with difficuliies" maybe considering case senclivitty since theme might he plarals Afterand, I want to conferm format- ting by
small window, which has the peeviously selected Light Red format. I'll also identify the approtimate coondinates for the OK button it seems be around (857, 644) relutive the screetabot. So, I'll need to click that Approcimaaing bution coordinates I'm estimating the coordinates for window. The border
be and the right boeder maybe $50 ps, making the window width about 500 ps. So, I'm chinking the OK butson is likely around (805, 646), Aliernatvveyy ely. checking the botiom
OK button
360 ps.
hutioes could help. soums like the Reset butlon buson could around 800 and 645. use these coordinates for clicking.
740.
Cancel button at 880,
Action
import py pauttggu import time prsuutgui.clcc/8800 645)
Action
Step 23
pyautogui.coo pyautogui.click(005, 9905, 356, button='left')
Step 31
Thouglt: Planning text roplasement need to replace every instance of "chal- lenges" with "diffiulties" in LibreOffiice Writer and highlihtt the changes in light red. ser "Find and Replace" dinlog with formalting font color to light After that. ensure "Find' tield "challenges" and the 'Replace" field has "difficulties" It's essentaal to seleci 'Replace all" and then click OK apply the formatting. Finally, I'll proceed with the replacement.
Nest. I'll cunfirm
Action
pyautogui.hotkee ('ctrl, "anl','' 'a')
Action
Step 32
impert pyautogui import time claplnggiiccck//8hh... 614)
Step 24
Action
cnuutogui.typewritte
Thoughu: Flanning text replacement steps I've got LibreOffice Wiiser open with the Find and Replace dialog ready. The 'Find' field his "challenges" and the has "difficultiss." The stop highlight red. see need click the Format bution under Replace, choose Font Eifects, and set the font color red. I'll need to click this button at the bottom center of dialog, which I've noted blue focus border. The approximate courdinates 520).
Step 33
"Replace"
changed
52

---

Published as a conference paper at ICLR 2026
Action
import pyoulogui import time httpulloouicccccciii.... $20)
Step 25
You hereeeeeer
Action
Crautoouuicllcc(000 381, button='left')
Step 34
Action
import pyautogui import time yyautogui.clic((64)))
521)
Step 26
Action
peautogui.hotkey.. 'a')
Step 35
Thoughr:: Detailing find and replaze steps I need to describe the nest step for replacing "challenges"
"difficuities" and applying red color formatting "Replace
LibreOfiice Writer. screenshot indicates dialog is open, and it appears have multipee canegoey rabs like Fent, Font Effecis, and Poution. The Foot lnffects tab is likely active. showing the "Kont Color' which indicates that the chosen color currenly Light Red 4. Now, fent
formatting"
Explaining test replavement steps to change the replaced text to red since the user's requirement is to highlight it that way. In the 'Replace with formatting" dialog, we should confirm the color selection. The step click OK apply ester "challenges"
need
Red color. After
the Find Replace dialog, and "dfficuliies' as the text to replace it with. We should ensure the search is asselitsenstivve and then click Replace All. Let's this into clear stops for the
Action
import pyautogui import time pyantogni.cicki865, 644)
Action pyautogui typewrite( ('O)
Step 27
Step 36
Action
import pyautogui import time yyautogui.click(655 519)
Action
pyautogui.click(907,. 411, button="left')
Step 28
Step 37
53

---

Published as a conference paper at ICLR 2026
You mevereeee
Thought: Cheffggarig and replace nead replace all instances of "chal- lenges" with "diffiulties" and highlight the changes in red. The user specifled to find "chaillengs"" and replace with "äiffiuultis." In LibreOffiie Writrr, can font color red during replacement. Replace options, so I'll
Action pyautogui.holkey' ('ttll, 'a')
the "Fornat... button to opea formstting options. Then choose Foot Efficiss set the fort color to red, and fimilly click 'Replaoe All. "Plonning mouse click cocedinates need to plat use mousc click troud the conndinates (613,
Step 38
the Replace field. I'm estimating right spot as about 610c520 based on screenchot. serrss like specifying left click that Fonuat button will help mo proceed more securately. Hopefully. that gel to the nexi without any hiccups!
access Fornet bumon
Action
pyautogui import time pyantggai (lankog (1ick(6442, 520)
Step 29
Action pyautogui pyautogui.typewriee(
Step
Action import pyautogui import time pyautogui.click(664 645)
Step 30
Thougia: Applying red color, clicking "OK"
Action
prautogan prattoge.click077. click/875, 644, buston= 'lott')
Thought: Adjasing text formating The user is working in LitroOffice Writer and wants to replace "challenges" with "difficutties" " hiihiighinng the change in red. see there's 's Format bution setting
Find Replace
dislog, It's important ensure we select the cornect color. The screenshot shows light pink color instead of red which could mean the user picked the wrung shade corlier. Let's ahead and the Furmal uption again select red (4FF0000) Proparing
which is currently highligheed. Clicking this bution will open the Character Formas dialog. I'll need to simulate a left-click the upprosimatte coordinutes of the buttio. Based on the seneenhoo, I estimate the coordinates to be around (640. 513) because button
siep
bottom censer of dialeg. I'll prepare the tool for lefl-click these coandinates.
Action
import pysutogui import pyautogm. clcck///..cc/...0.0m0m0 513)
Step 31
54

---

Published as a conference paper at ICLR 2026
Thuught Serting replacement fonnatting naer wants to replace "challenges" with "difficlities" and highlight the changes red, so I'm opening Find and
Replace in I LibreOffice Writer. I've entered "challenges" the text find and "difiiuttiee"
replacements replazeent. Next, have the fammarting highlight
"Forzut..." button ovailable set the font coler ned for the replaceent test, focusing on geting the coordinates
I'll
right click correctly.
Action
py-ulogui import time pyanlogui (62, 525)
Step 32
Action
import pyautogui import time pyautogui.clic((66. 6466
Step 33
Action
import pyautogui import time pyautogui.click(666 520)
Step 34
55

---

Published as a conference paper at ICLR 2026
Action
import pyautogui import time pyantoguiccicck(500 640)
Step 35
Thought: Updating text formatting need to change all instanees of "challenges' "difficulties" and highlight those changes in red. I've opened the Find and Replace festare and entered the wonds cornecll. need make replacement
formar Replace
click Gield, and the dialug. set the font volor rod. After that. can click Rephare All to finalize the changes Plazning formatting sieps I've noticed that the Formar button highlighted in blue, indicasing it's potentially selected but basn't been then novigate tab to choose the font color red, This seems strigghffrwaadd I'll plan my newt steps, starting with that lins click on the Format button. It's all abous getting those
clicked yet.
click Formal
Foet Effects
settings right for the best results!
Action
import pyaulogui import pyaulogui.tick/o click/666i........... 521)
Step 36
Action import pyautogui import time vyautogui.click(66 521)
Step 37
Blow
Thowghr: Handling Find and Replace I've pot the LO Writer open and I need replace 'challenges" pressing ChrlHH where I've set the font color to Light Red 4, Now. I liced to confimm this and close the furnasting window, then enter "challenges" in the search bux and ditiicutties
"diffiulties. highlighing the section fontatiin Replace" dialng
in After
applied before clicking located bottom right the sub dialog. positioned between Reset and Cancel. plan to click OK. estimating cocrdimates be around (866.455) based on the screenshor dimensions of 1280x720. The button the third one from monitor encare that action registers
replacement
OK Conftrming action ideasified
bettom
row. I'll peocoed correctly.
56

---

Published as a conference paper at ICLR 2026
Action
import pyoulogui import time (8t. 645)
Step 38
Thought: Determining lest replagement format I need to replace all occurences "challenges" "difficulties" highligtt the Find and Replace window. The TFndd' field already filled with "calllngges' and 'Replace" has "ifficullies' Now, must set the formatting to rod. I'mm unsure if "highlighted"
text
opened
the color should be or the background color should highlight. Typically. "highligh"" refers background options may cluriyy Pinalizng text replacement settans The replacement section shows "diffculties" and a highlight color of KGB,255, 215. 215), which looks like light red pink. This suggests that the highlightng set. Next, need
formatting
already be 'Roplace
exceuse changes. All' button situate near the right side of the window. I should ensure that the background highlighted in the correct colur pertups adjusting it to true red if needed. Let's
"Replace
Formar option contiem
the color Adjusting highlight is already
highlight settings might simpler accepe Looking at the snippet, the "Keplace" area shows "difficuttics" with a highiight color of RGBB225, 215, 215). This scems lighter than typical strong red. should adjust standand red color insteod. plan click Furmat, then go exceute 'Roplace All." piopoint coordinates for the Format button and proceed with the click.
Font Effectts select red,
Action
impoet pyautogui import time pyantogui.licki665. 512)
Step 39
Thought: Changing text in docanten I've got Librelffcee Writer document open, and my task to replace "challenges" with "difficuties" while highlighting the changed red.
Find Replace dialog up, need to specify
"Find: challenges" and "Replace: difficuhies" Fox the fomsatting, the format button to set the font color red. Once that's done. I'll select Replace All. I'm checking that fommat button to ensure I'm clicking the right one Setting font color documeut this,
click
the Fommat... button the fout Replace section. This action
rod the documest.
open the Character Format mena where can choose the red foregroand color. I'll need to sare I'm clicking the rightt button the correct cooedinates to ennare opens properly. Once that's done, proceed
replacement ensaly.
Action
import pyantogui import pyantoguiclick660) 523)
Step 40
57

---

Published as a conference paper at ICLR 2026
Thought: Configuring test replacement I'm working replacing all instances
of "challenges" with "ifficahies" and changing the font color to red. opened
the Replace with Foematting window and saw that the Fost BlTects tab selected, which Light Red 4 the
New noed finalise formatiing
by clicking test. Then I'll check "Find All' or "Replace All" for a complete update. Sedite.Clile Clicking the OK button Alright, next action click the OK hitton at the betton
dialog.
I'll up search terns ceplace
the window. sinused there. I'm estimating need to sim
chocked screenshooe, and
that button. I'll go
center of ahead and do that now. This should con finiin my formatting choiess and allow me the nest the process. Let's get this done!
Action
pyantogui import time clakkoggiiccick(866... 645)
Step 41
Thoogla: Serting text color in Find and Replace The user wants me clange all of "challlegge" "diffiulliee" and highlight the changes red. opened the Find and Replace dialog, to encare that the changed text is highlighed rod. There's a "Foemer. huttin
the fields are set cornectly. Now, noed
I need click to the font color, I'll locate and click this button to adjustt the suggest
The
middle of window.
Action
import pyautogui import wywulogui.dickk633.. 518)
Step 42
Thoughr: Analyzing and Replace interface see "Replace" in immsuccni overlay, which might obstructing curlier clicks. Let's smaly ne "Find and Replace" window more closely. At the top. there are fields for Find and Replace, followed buttons Find AlL Find Previous. Find Nexi. Replace. Replace The "Other options" grop his chockboxes, are buttons Attrihttes Formar, Format, and Close There's small drop down arnow nexi to the Replace field for formatting settings simmlar to the format button Writer 7.6. Extimiting dialog layout I'm estimating the dinlog's positioning,
110) 258), making the width about 300. There' an arow on the right. likely at 1715, 258) After clicking this arow. expect a drep-down with "Format. to appeart though might be small. could also click "Foemat..." from the botion row. 256) using
top- arcund
Replace
plan click arow
and then for next
action.
Action
import pyantogui import pyantognisdick(66.. 256)
58

---

Published as a conference paper at ICLR 2026
Step 43
Thought: Analyzing the scroenshot task The user provided a sereenhet wants me to help with n took involving replacing "challenges" with "difficulties" and highlighting the changes in red. They's 've opened the Find and Replave dialog in LibreOffice Writer, which
fommttiing options ready go.
stop noeds confiming the sorth "chullenges" and inparting "dificclties" us the replacement temn. I'll make sure to apply the Light Red 4 oolor for the changes(Cvefimen
formatting choices Now that the formatting dialog immediate next step need to return to the Find & Replace dialog and set the search and replace terms. So, first, I'll click the OK button at the bottom right of the Replace with
OK
selection. After
fommatinng" dialog. I'm estimating the coundinates for the click, focusing thas
Once that's done. I'll
Action
import pyantogui import pyantognicfic///e 643)
Step 44
Thought: Adjusting LibreOffice
wocking Librelffice Writer "difficuttie"" highlightinn the chaiges in
and needs replace "challenges" red. I've opened the find and replace window with the terms already filled in. I'll need to click the Format buttonn set the chometer foot coler rof by navigning through options. However. I'mn wondering of just changing changing the barkgroadd, setting the fomt color to red foel like the simpler choiee.
request
highlight
color. Since highlighting typically means
Action
Import pyautogui import time pyantogui.cick(65,55 518)
Step 45
Thoughr: Clarifying text replacement I realize I need replace all instances of "challengss" with "ds "ifficutties and highlight them in red. opened the Find & Replace dinlog and notieed "Find" field "challenges" while "Replace" theld shows "difiiaaties"
light pink color
seoms the back-
ground was previonly set using the Formar button, but need to adjust it for proper red highlighting, specifically with the RGB value of 255,0.0 to achieve that Setting highlight color lned red.
highlight color Color Highigghtin. The
replaced
option libeefffice user wants the text to he highilghhed in red, which nsmally implics background color, However. I'm also considering that they might jusi profer the font issel to Since manual suggests highlight typically yallow, I'm interpeeting "highlight rod" as bockground tighlight. So, adjust Replace foruu
59

---

Published as a conference paper at ICLR 2026
ensure reffects change progesty solecting red highlight option. to click on the Format butiontEiimmiing cialog position m uyynng to approximate the location of the dialog box. li uppears to be ceniored borizonially at unound 580, 518. extmating, and has width center at uround 540, 300. After looking at the screenshot dimensions. I'll sot my click roughty (00l), 510) target the Formut button effectively. Time so click!
the dialeg 400 ps.
400 ps
butoo
near betton
Action
import pyautogui import time pyantogui.clck.. clickc99...8888 520)
Step 46
Though:: Executing find and replaze I'm currently working in LibreOtlice Writer The user's request reglace instanees of "challesges"
"difficuttie"
and Replace window and have the formatting bot ready to gu. mentions "Light Red 4." which works for purposes. need contem the color by clicking the "OK" button in the commating dislog. Once do that. proceed with the replacement.
highlight those changes red. opened
Action
import pyantogui import pyantoguiclick/667.cm 644)
Step 47
Thungle Planning replaceeentt need make changes replacing "diffiultiee" and highlighting that changed
instances of "challenges"
red. The Find and Replace dialog his the conrect fiekds ready, with "challenges" and "difficlties" set up. Now, I need to formar the replacement rest show in I'll click the Format... button which cureenly highlighted. appenximate the coodinates for clicking around (615, 517),
my
that!
Action
Emport pyantogui import pyantognicdcck((25, $15)
Step 48
Jwar
Thoughe: Planning the steps related to changing all occurrncces of challlegee" to" difficulties' and highlightng the chinges in red. They's ve opened the Find and Replace window, which shows formatting option for the replaceeents see that this sccvedary "Format" excellent.
asked anilyte
selection window, 'Foat Color' step to confimm the replacement and apply this formatting ensure the text correctly updinted. Confeeming formur changes I need to click the "OK" button on
Light
60

---

Published as a conference paper at ICLR 2026
window apply Replace. I'll cnter challenges" as the term to find and "dithcullacs' to teplace It seems the Find and Replace window is uvershadowed by the Formal window, making look faded. "OK" cantim. esimating cocrdinates since it's not fully slear. Let's procecd Confirming the action I see the buttoms "Resst." "UK"" and "Cancel" in the screenshot, and I'm estimating the coocdinate "OK" button
botton of Format window,
perform left click replacement text. I'll regain focus on the main window. Then should respond with the apprograte next steps. Let's keep moving durward!
(847, 641), continu Light
Action
pyautogui import time 2link/g. 641)
Step 49
Action
import pyautogui import time pyautogui.click(66 515)
Step 50
Thought: Checking Find and Replace sctup The nser wants to replace "challenges" with "difficutties" and highlight them in red. I've opened the Find and Replace tool
"Find: chillenges" and "Replace: diffieuliies
apply highlight foematting there's a 'Format... " buttoa nest to the Keplace field that I'll seed to click. see something about RGI,(255, 215, 215) the cereenhot, indicating that the light highlight night dirady selected. highiaght formatting
replacement
Writes,
dmuble-check that formatting (Senting options "Autibutes... and "Formar. Replace meaa There's a button for "Romtot.." that I need to click to open the Chanettr Format dinlog for setting the highlight color rod. The user wants the four color red.
click solect Color, and Then, I'll confirm and click Keplace All. Let's started by clicking that Forms button!
highlighted, which
button, savipute Foot
Action
import pysutogui import time vouulogiicicck627..oo 522)
61


Reviews:
[{"review_id": "vNQFgaY5tC", "reviewer": "Reviewer_GEBQ", "summary": "The paper presents a platform for evaluating computer use agents (CUAs) via head-to-head evaluation strategies with real users. The platform provides preset initialized computer systems with apps in various start states. The users write task prompts against this environment and get to observe and evaluate the agent performance on the task. These preferences from more than 1000 users on about 1200 tasks are used to provide ELO style rankings of a large number of models. The results surface significant differences in human preference compared to previously observed rankings on existing CUAs evaluation environments (e.g OSWorld). Further analyses shows specific difficulties in these dynamic real user evaluations compared to static fixed set evaluations, as well as some insights in to the type of agent behavior that humans prefer.", "questions": "See weaknesses above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a timely and much needed effort in evaluating computer user agents. The head-to-head evaluation using real users at scale provides strong empirical evidence and fine-grained information that can help distinguish between different model performance, and highlight where the models fail. The overall methodology is sound and has been done at scale. The framework looks extensible---can be easily used for other specific types of evaluation considerations.", "weaknesses": "I am not able to identify any significant weaknesses for this work. \n\n- The evaluation shows that the tasks being generated are diverse. It would help to either describe how this diversity comes about in the process. Figure 1 lays out the general process but some more details on the process could help clarify whether the scenarios within computers are somehow systematically guaranteed to be diverse or if the diversity comes from what the users do with the same scenarios. \n\n- While I appreciate the apples-to-apples comparison with standardized settings, it would also help to have tried the best settings for at least one or two models to see what is the best performance that could have been obtained. This is not a major issue and one that future work can fix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3kpj5Cpkmv", "reviewer": "Reviewer_xYzD", "summary": "The authors present a new benchmarking platform for agentic systems that interact with computers (virtual machines). In doing so, they find that existing models perform differently from what would be expected from previously reported results.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- With the increasing amount of models gaming evaluations, it's always good to have more benchmarks.\n- I like the direction of moving beyond easily codified evaluations and relying on more direct preferences. This has been the trend since InstructGPT, but many works regress on this.", "weaknesses": "- This feels a little weak on contributions. I would have expected some kind of algorithmic insights to go along with a benchmark for a conference paper. I would have expected to see something like this in a dedicated benchmark track.\n- It seems like the main advantage of the benchmark is that it has more open-ended tasks. But are the other benchmarks with fewer open-ended tasks already more or less solved? Why do we need more open-ended tasks?\n- I would want to see some more discussion relating to the shortcomings of other benchmarks here. I'm struggling a bit to understand why existing benchmarks are meaningfully inadequate here. This is really the only thing holding me back from accepting it.\n- Adding humans in here makes the evaluations markedly more stochastic. It isn't, after all, the only way of doing more holistic evaluation (see, e.g., LLM-as-a-Judge or Agent-as-a-Judge). Some discussion of this or alternative ways of automating the human judges in the future could be useful. Maybe the humans are not needed at all if correctness is guiding human evaluations?\n- Why are the mouse movements defined as actual movements and not clicks at certain points? This is more related to my own curiosity here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0d3xy8K8kT", "reviewer": "Reviewer_hCqQ", "summary": "This paper presents COMPUTER AGENT ARENA (CAA), an open-source platform for evaluating Computer-Use Agents (CUAs) in realistic computer environments using human preference–based judgments. CAA moves beyond static and contamination-prone benchmarks such as OSWorld or WebArena by supporting dynamic, side-by-side evaluation of agents in identical cloud-hosted virtual machines. Each session runs two anonymized agents that perform the same user-defined task, and human participants rate their outcomes and behaviors. The collected preferences are aggregated into an Elo leaderboard through a Bradley–Terry model, with optional stepwise labels that capture correctness, safety, and self-correction.\n\nThe authors gather 2,201 high-quality votes from 1,058 users across 12 agents, including commercial, open-source, and multimodal models. Results show significant ranking reversals compared with static benchmarks: models strong on OSWorld often perform worse on CAA. Human preference is driven mainly by correctness, but users also value process quality such as self-correction, reasoning clarity, and judicious use of CALL_USER queries. Both excessive and absent user interactions reduce satisfaction. Error analysis highlights failure types that static benchmarks overlook, including long-horizon memory drift, insufficient clarification, and fine-grained GUI control errors. The paper also releases the platform, dataset, and codebase to support open and reproducible human-centric evaluation of CUAs.", "questions": "1. **Task coverage and diversity**: How do you quantify the coverage of crowd-sourced tasks across domains and difficulty? \n2. **Influence of user expertise**: Did the authors analyze whether technical versus non-technical raters show different preference patterns? This could inform CUA design for different audiences.\n3. **Complementary metrics**: Can CAA report additional task-level signals such as time-to-completion, error counts, or privacy violations to assist agent diagnosis?\n4. **Score interpretation**: Given the high correlation between Elo and GenScore (r ≈ 0.95 in Appendix E.2), what distinct insight does GenScore provide beyond the main leaderboard?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. **Originality and significance**: CAA introduces a new evaluation paradigm for computer-use agents. It extends the “Arena” concept pioneered by Chatbot Arena into the a more complex, multimodal, and stateful domain of GUI-based computer interaction. By converting crowd-sourced, real-world tasks and pairwise human preferences into structured feedback, CAA establishes a scalable, human-centric framework for CUA assessment. This contribution is conceptually innovative and practically significant, redirecting focus from benchmark scores to real user satisfaction and behavioral quality.\n2. **Technical quality and system design**:  The system demonstrates great engineering practice. The cloud-based virtual machine infrastructure provides elastic scaling, hundreds of preset setups, and user-defined customizations for authentic computer use. Agents run in matched environments with identical AMI fingerprints and software versions, ensuring fair comparison. The ranking method is statistically sound and is supported by bootstrapping, permutation tests, and power analysis. It delivers reproducible Elo scores with confidence intervals. The paper also reports strong inter-annotator agreement and transparent filtering procedures.\n3. **Human-centric insight and analytical depth**: The analysis section is insightful, and I highly appreciate it. It shows that users value execution process quality —such as clarity, responsiveness, and error recovery —more than speed or execution time. The observed “inverted U-shaped” relationship between the number of CALL_USER queries and win rate offers actionable guidance for designing interactive agents. Moreover, the comparison between tool-integrated and pure GUI agents reveals an important benchmark gap. Tool-centric agents perform well on scripted benchmarks but often struggle with open-ended real-world tasks due to tool misuse and hidden failures. These findings represent valuable behavioral diagnostics for the field.\n4. **Clarity and communication**: The paper is well-written and illustrated. The methodology is described with approachable language, and the paper also offers full reproducibility details. This clarity significantly enhances the paper’s readability and utility to the community.", "weaknesses": "1. **Limited task coverage and representativeness**: Although the authors collect over 2,000 votes, some models receive relatively few comparisons, which may affect ranking stability. Tasks are crowd-sourced and primarily English-language, so the distribution may underrepresent non-English and specialized domains such as scientific or enterprise workflows. Explicit coverage metrics for task diversity would strengthen the evaluation.\n2. **Cost and scalability**: The framework is efficient for benchmarking but remains costly for large-scale data collection (about $1.24 per vote). Scaling to preference-tuning datasets might require further automation or active sampling methods.\n3. **Human-centric definition and interactivity**: While CAA is human-judged, the evaluation is not truly interactive because raters review replays rather than engage in real-time turn-taking. This limits its scope as a “human-in-the-loop” system. Future extensions could integrate live interaction to capture dynamic feedback and adaptation.\n4. **Task bias and environmental variability**: Despite identical AMIs and matched virtual machines, minor runtime differences such as network latency or software updates could influence agent behavior. Moreover, crowd-generated tasks may skew toward common desktop operations rather than professional applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x67KtHGgy9", "reviewer": "Reviewer_rdUN", "summary": "The paper introduces Computer Agent Arena, a platform where users can perform pairwise evaluation of agents on user-provided tasks. Analysing data generated in this way, they find that static benchmark performance does not necessarily correlate with user preference on these more ambiguous and open-ended tasks.", "questions": "- I find the poor performance of GPT-5 and Gemini 2.5 Pro quite surprising. Do you have hypotheses or analyses that suggest what the cause may be?\n- You mention that long-horizon memory errors and fine-grained action failures are both common. Can you give per-agent error rates in these categories?\n- Could you train a learned preference model on your data, incorporating correctness, latency, number of queries, etc, that would allow your human data to be reused in an offline setting (i.e. without requiring reranking for a quick oversight of a new model)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- A highly configurable environment allowing for diverse user-defined tasks\n- Quick start tools reduce friction when initialising new tasks\n- Very in-depth analysis in body and appendices\n- Likely relevant for significantly longer than static benchmarks", "weaknesses": "- The number of votes is fairly small when spread over the task distribution.\n- Evaluating agent traces and setting up new tasks is significantly more demanding than simply asking questions (as in LLMArena), which may limit the number of users willing to perform comparisons and hence the future relevance of the platform.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)