SCUBA: Salesforce Computer Use Benchmark
Authors: Yutong Dai
Krithika Ramakrishnan
Jing Gu
Matthew Fernandez
Yanqi Luo
Viraj Prabhu
Zhenyu Hu
silvio savarese
Caiming Xiong
Zeyuan Chen
Ran Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Computer-Use Agents
Enterprise Benchmark
CRM
Vision Language Model
Primary area: datasets and benchmarks
TL;DR: We propose SCUBA, a benchmark designed to evaluate computer use agents on the customer relationship management (CRM) tasks. using the real Salesforce platform.

Abstract: We introduce SCUBA, a benchmark designed to evaluate computer-use agents on customer relationship management (CRM) workflows within the Salesforce platform. SCUBA contains 300 task instances derived from real user interviews, spanning three primary personas—platform administrators, sales representatives, and service agents. The tasks test a range of enterprise-critical abilities, including Enterprise Software UI navigation, data manipulation, workflow automation, information retrieval, and troubleshooting. To ensure realism, SCUBA operates in Salesforce sandbox environments with support for parallel execution and fine-grained evaluation metrics to capture milestone progress. We benchmark a diverse set of agents under both zero-shot and demonstration-augmented settings. We observed huge performance gaps in different agent design paradigm and gaps between the open-source model and the closed-source model. In the zero-shot setting, open-source model powered computer-use agents that have strong performance on related benchmarks like OSWorld only have less than 5\% success rate on SCUBA, while methods built on closed-source models can still have up to 39\% percent task success rate. In the demonstration-augmented settings, task success rates can be improved to 50\% while simultaneously reducing time and costs by 13\% and 16\%, respectively. These findings highlight both the challenges of enterprise tasks automation and the promise of agentic solutions. By offering a realistic benchmark with interpretable evaluation, SCUBA aims to accelerate progress in building reliable computer-use agents for complex business software ecosystems.

Published as a conference paper at ICLR 2026
SCUBA:SALESFORCE COMPUTER USE BENCHMARK
Yutong Dai, Krithika Ramakrishnan, Jing Gu, Matthew Fernandez, Yanqi Luo, Viraj Prabhu, Zhenyu Hu, Silvio Savarese, Caiming Xiong, Zeyuan Chen, Ran Xu Salesforce Research
(yutong.dai) kkrithika, zeyuan. chen, ran. ran.xu ran.
ABSTRACT
We introduce SCUBA', benchmark designed to evaluate computer-use agents on customer relationship management (CRM) workflows within the Salesforce platform. SCUBA contains 300 task instances derived from real user interviews, spanning three primary
-astnnnss-patffrmm administrators, sales representatives, and service agents. The tasks ing Enterprise Software UI navigation, data manipulation, workflow automation, information retrieval, and troubleshooting. To ensure realism, SCUBA operates in Salesforce sandbox environments with support for parallel execution and fine- grained evaluation metries capture milestone progress. We benchmark diverse set of agents under both zero-shot and demonstration-augnee observed huge performance gaps in different agent design paradigms and gaps between the open-source model and the closed-source model. In the zero-shot setting, open-source model powered computer-use agents that have strong perfor- mance related benchmarks like OSWorld only have less than 5% success rate on SCUBA, while methods built on closed-soune -ource models can still have up to 39% task success rate. In the demonstration -agggettttid-emmeeeeeeeedee settings, task success rates can be improved to 50% while simultaneously reducing time and costs by 13% and 16%, respectively. These findings highlight both the challenges of enterprise tasks automation and the promise of agentic solutions. By offering realistic benchmark with interpretable evaluation, SCUBA aims to accelerate progress building reliable computer-use agents for complex business software ecosystems. The code-base
range of enterprise-critical abilities, includ-
settings. We
Data
List Listvien view
For the opportunity Cloud Boftware Deal, Schedule an event for neeting with the client to discuss implementation timeline. Update the opportunity stage to Value Propos tion.
9%
Permieston Sota
Model
User
6%
18%
Aeprovall Prosoms
admin
57%
Distboard
Repoott
service 18%
2u
sales
Encileet Managahnnt
25%
Entitlements
CPQ
Depertantion
Cose Manus Manusgemment
27%
27%
mroom gontats
Configue abonn
Q&A
20%
(a) Tasks distribution in different domains.
(b) One (simplified) agent trajectory on a task.
Figure 1: SCUBA tasks, environment, and agent trajectory preview.
'The code-hase is available at https: //gi thub. www http:/aallsffcce htt.r//c/e/eeeaaaeeeeeeee//ea SCUBA.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The advancement of large vision-language models (VLMs) has sparked increasing interest in turn- ing these models into autonomous agents automate different tasks, ranging from coding and deep (re)search to workflow automation (Yang et al., 2025a; Zhang et al., 2024b; Google, 2025; Claude, 2024: OpenAI, 2025c). One particular direction is to make these VLM-powered agents automate complex workflows via graphical user interfaces (GUIs) (Qin et al., 2025; Wang et al., 2025b; Ope- nAI, 2025c; Claude, 2024). Despite significant progress, the adoption of these agents enterprise software environments remains challenge and the performance of these agents is not fully under- stood. For performance, we mean more than success rates since in real business deployment, latency and costs are also decisive.
A primary obstacle the lack of benchmarks that accurately reflect the multifaceted nature of enterprise software and workflows. Seminal benchmarks WebArena (Zhou et al., 2023) and osWorld (Xie et al., 2024) are influential in evaluating agents on Web navigation and generic desk- top application tasks, but they do not capture the complexities and challenges of using enterprise- grade platforms and tasks. WorkArena series (Drouin et al., 2024; Boisvert et al., 2024) are proposed to address this issue, with the environment and tasks built upon the ServiceNow plat- form. However, these tasks are often limited to service use cases, CRMArena series (Huang et al., 2025a;b) are built on the Salesforce platform, however, the tasks are mainly designed for tool use agents (Zhang et al., 2025a) and are based on information retrieval, i.e., the tasks do not write or change data records and settings in the environment. To bridge these gaps, we introduce SCUBA (Salesforce Computer Use BenchmArk), comprehensive and realistic benchmark designed to rig- orously evaluate computer-use agents on customer relationship management workflows. The com- parison with related works is made in Table 1. Our key contributions summarized below.
1. SCUBA is realistic. The tasks are derived from user interviews, covering the critical business functions of platform administration, sales, and customer service. These tasks aim to evaluate spec- trum of agents' abilities, including enterprise software UI navigation, data manipulation, workflow automation, information retrieval, and troubleshooting in the live Salesforce sandbox environments.
SCUBA provides culti-dimensional evaluation harness. Each task is paired with a rule-based evaluator, which not only tells whether the task is completed successfully, but also offers fine-graine milestone score, also known as the process reward (Zhang et al., 2025c). Based on the milestone scores, the evaluator also provides details on which part(s) of the task the agent failed to perform. SCUBA also provides metrics on the latency metrics and costs metrics.
3. SCUBA is shipped with knowledge articles and human demonstration, which can be utilized to improve agents' performance. We leverage human demonstration to simultaneously increase the task success rates and lower the latency and costs for a collection of agents.
4. SCUBA is efficient to use. We developed an asynchronous evaluation pipeline, with which the full evaluation can be finished in less than 90 minutes?.
Table 1: Comparison on enterprise- tareeetttt-s-----eeeee benchmarks. SCUBA uniquely combines all these aspects, while other benchmarks remain narrower scope.
Benchmark
Instances Templates)
human unnotations?
Rewards?
Read&Write Tasks?
More Service Tasks?
WorkAcens
19912 (33)
X
X
BorkArena**
(341)
CRMArens
1170 (9)
X
X
CRAAcens
2140 (22)
X
X
SCUBA
(60)
/
2
RELATED WORK
2.1 COMPUTER- USEE AGENTS
Due to the rapid growth of the relevant works on this topic, we do not attempt conduct a compre- hensive review and direct the interested readers to the survey (Zhang et al., 2024a) and the references
This is for the self-served models. For API-hased models, the bottleneck the API limits.
2

---

Published as a conference paper at ICLR 2026
therein for more details. Instead, we focus recent development of computer-use agents powered by lasion-language models (VLMs), where the main imputs are the screenshot of the environment, and the outputs sequence of executable actions, like clicking, typing, even code blocks.
There are mainly three paradigms for developing computer-use agents of this kind. The first paradigm is to train foundational VLM with the computer-use one of its capabilities. Rep- resentative works include Claude family (Claude, 2025) and Quen2. 5-VL (Bai et al., 2025). The computer-use ability is achieved by leveraging the VLMs' visual perception&grounding pability and the tool-use capability. Usually a computer-use toolbox is provided to regulate the model's action space, i.e., the output can only be valid executable actions. The second paradigm is to build single native GUI agent, where the model is trained to have visual perception, visual grounding, reasoning, planning, action generation, and memory capabilities. Aguvis (Xu et al., 2025), UI-TARS family (Qin et al., 2025; Wang et al., 2025a), and CpenCUA (Wang et al., 2025b) belong to this category. The third paradigm is to build an agentic framework by offloading different capabilities such as visual grounding, reasoning, and planning to dedicated agents, which is in con- trast to the second paradigm. Since the agentic framework admits special design, such as crafting special visual grounding agents (Agashe et al., 2024; 2025; Yang et al., 2025c; Xie et al., 2025) or enhancing GUI actions with coding and API calls (Song et al., 2025; Lai et al., 2025), this paradigm tends to outperform other benchmarks like OSWorld (Xie al., 2024).
ca-
2.2 COMPUTER- USE BENCHMARKS
Challenging benchmarks are proposed to evaluate agents' capabilities from different dimensions, and the number continues to grow. We focus on benchmarks that are computer-use computer-use-centri and satisfy the following requirements: 1) the environments are interactive, where the agent is free to explore; 2) tasks can be tested with real desktop operating systems and/or desktop browsers; 3) reliable rule-based evaluators are available. These restrictions lead to the following classification.
Browser Native Benchmarks. MebArena and WebArea are seminal benchmarks, where agents are required to navigate through simulated websites to perform tasks such as infor- mation retrieval, social media posting, and shopping. To account for the UI gap and modern website design in real-world websites, REAL(Garg et al., 2025) replicates the functionality and UI of widely used consumer platforms, for example, e-commerce, and professional networking. series (Drouin al., 2024; Boisvert et al., 2024) and Amazon-Boon -Bnch (Zhang al., 2025b) adopts the real sandbox of production-gran related tasks the ServiceNow platform while Amazon-Benee capability on various functionalities offered by the Amazon shopping website.
borkArenaa
websites to build tasks. Wor MorkArena focus on the service-
puts emphasis testing agents'
OS Desktop Native Benchmarks. OSWorld (Xie al., 2024) is the first real computer envi- ronment, using 9 desktop applications to cover a variety of computer tasks, such as manipulation of slides, documents, images, and excel tables. It mainly focuses on the Ubuntu OS. Follow-up works WindowsAgent Aweeesesssnttttenaaaaaaa (Bonatti et al., 2024), macOSWorle (Yang et al., 2025b), and MMBenca-GUI (Wang et al., 2025c) extend the efforts cover tasks on Windows and MacOS and expand the evaluation dimensions to include the safety and efficiency.
3 SCUBA BENCHMARK
SCUBA is an interactive environment built on the Salesforce platform, a widely used cloud-based enterprise platform for CRM workflows. SCUBA is designed to cover three major types of tasks that are highly valuable for automation. Each task is paired with an initialization script and rule-based evaluation methods. Details are discussed in the following sub-sections.
3.1 ENVIRONMENT
Realism is one of the key requirements when designing SCUBA, therefore, we opt to use a sandbox developer org offered by Salesforce?. Its is identical to production orgs and free-tier features are sufficient to support all tasks. However, the pursuit of realism also brings unique challenges for
'The public can sign up for sandhox org via the link https: //www//wwww/////////////e// sai eeaffrceee... com com/form/ form/ leveloper-se gnupp/ ?d-pb
3

---

Published as a conference paper at ICLR 2026
resetting the environment. For benchmarks like WebArena (Zhou et al., 2023) and USWorld Xie et al. (2024), resetting can be done by restarting Docker containers. For a Salesforce sandbox devel- oper org, resetting the org to its initial state is equivalent to creating a new one, which is prohibitive for supporting large-scale evaluation. Therefore, we address this via taking a snapshot of the initial states of the org by downloading set of configuration files. During the resetting phase, we compare the differences of the initial states of org and modified states after an agent run. We then revert the configuration files that are changed by the agent their initial states. This means that the reset is done on the task level since we know what configurations are expected to be changed by agents.
Parallelism is another key requirement to allow efficient agent interaction and evaluation. There- fore, we build infrastructures support this. Users can choose to spin up either desired number of browser sessions or OS desktops (running in the Docker containers) concurrently. So, a large collection of agents can be assigned dedicated environments and can be tested asynchronously on SCUBA to save the time.
Observation & Action Space. The environment supports screenshots, accessibility trees, and flat- ten DOM object strings as the primary observation for the agent. All primitive actions supported by Playwright and PyAutoGUI can be used to interact with the environment.
3.2 TASKS
This version of SCUBA contains 300 test cases, covering daily workflows from three types of per- sonas, namely, the platform administrator, the sales role, and the service role. The tasks are sourced by interviewing real Salesforce platform users, including the solution engineers, org administrators, sales development representatives, business development representatives, and account executives. After collecting the cases, we prepare the tasks by associating each with one knowledge arti- cle on Salesforce Trailhead a learning platform that provides tutorials on how to use Salesforce platform. The distribution of tasksis visualized in Figure 1(a).
3.2.1 1
CORE ABILITIES
The tasks are designed to test the following abilities of the agents.
Enterprise Software UI understanding. Navigating on enterprise software to find correct pages to complete the task can already be very challenging for a daily computer user. The UI design is typically more minimalistic and prioritizes robustness and efficiency. Usually, formal training with detailed on-boarding materials is required to help users become familiar with the Salesforce platform. Therefore, agents need to have strong enterprise software UI understanding, navigation, and interaction abilities. For example, the task below requires human agent to navigate to 8 different pages with 31 steps to complete. "Create a territory model with the name "by Finance Industry" and with the rule name "Industry, Active, and Type". The selection criteria are "Industry equals Finance", "Account equals Active", and "Type is not equal to Channel/Partner Reseller or Installation Partner". If the sales territory is not enabled, enable it first."
2. Information Retrieval & Textual Reasoning. Service and Sales users usually want to quickly look up some quick facts buried in thousands of records. We build tasks test agent's ability to achieve this goal. One sample task looks like the following. "I'm reviewing the call transcripts to extract useful information. Please examine the call between [Person A] and [Person BJ, and tell me within how many days does the [Person A] wish to have the installation done. Respond with the number only." " In this query, [Person A] and [Person B] are made up user records, but we choose to mask out to avoid violation of the double-blind policy.
3. Salesforce Data Manipulation. One of the core functionality of the Salesforce platform is about managing data securely. Tasks on the data model, metadata, and data records creation/update/da tion need to be handled with correct privilege. The Salesforce platform provides a set of features to ensure the standards obligation. We design tasks like the following to test agents' such ability. "Add a formula field with the name "Days to close" to the object Opportunity. It tracks the number of field days until Opportunity Closes. This newly added field should only be visible to Marketing User."
/
readthedocs. ic hetp//n/aatesee "https://trai salesforoe.com/
4

---

Published as a conference paper at ICLR 2026
4. Business Workflow Build & Execution, One critical usage of Salesforce platform is to build and automate some established business workflows, where things must be done in a prescribed order following certain policy. One sample task is "Configure the Approve Opportunity Amount process final approval actions by adding a new action with the name 'approve when successful', which sets the amount approval status field value to 'Approved'. Similarly, add new action under the final rejection actions with the name 'reject when faile'', which sets the field value to 'Rejected'. If the amount approval status field does not exist in the Opportunity object, create it first."
5. Q&A for troubleshooting. Our survey reveals that platform administrators and human service agents need ground on the org's data to answer client questions. Therefore, we design tasks to test if the agents can answer questions faithfully. A sample task test this ability is as follows. "A user is assigned with Account Access permission set. Will the user be able to delete any records under the Account Object? Based on the Salesforce org's 's object settings, only answer with "yes" or "no"."
3.2.2 TASK CONSTRUCTION DETAILS
In this sub-section, we describe the pipeline for the task construction.
Cfficully Duiriuttion by Splii (No madiood 1025)
"Sown" 0.71 Thakus labed Mumbera (Aoow Besl Barwi]] sditad saear
"aak
18.2%
"qarr" "Sel us susuu cal D'nnarrs tre Inclids Alco Boo and Soll Derned sa members..
D'ellatoo" Grease quece withram Lxapper"
lovo relweai
plea-st "sarver UM11
tnu. 'wigerr
RELEALLATTEE inw.rreddddaaaa.. tarell Lratadort, kiclude Yeur tarrevdel
41,9%
$3.7%
"yeewo wous (
tou Coill
"eeuess P'aneeliedgeer"
wwiger::2
Erail actiicaioon sertio
"grund, murt Lof
Lucpport
41.0%
Jype "Laae"
('etestow: Ventes (Mice tutt Sota Seree editdd the
"eeaal secthra"
19.00
("Mbor Bes" Sur
(o..aaonn Mix. wiihe::225
fon uouetiuul
Yon, Bxio"
admin
Service
Figure 2: Left: Difficulty distribution by splits. Middle: A sample task configuration files. The light blue section contains basic information on the task; the light green section docuements details for initialization; the light orange section highlights the inputs used for rule-based evaluation; the gray section can be used to manipulate the environment before and after agent run, leaving the freedom configure different initial states and perform post-process. Right: A sample evaluation result. light purple section indicates if the task is successful. The light yellow section lists detailed milestone scores and rubrics.
Phase 1: Task Template Creation. For each workflow obtained from the user research, we build template for it, e.g., "Configure the organization-wide defaults for the {object_ _amme) object with visibility -isiiilliss for internal users, visibility {external. for external users, and allow users higher in the role hierarchy." " Each template paired with one knowledge arti- cle from the Trailhead. A complexity level from the set (easy, medium, hard] is assigned to the task template based on UI actions complexity and unique abilities required to finish the task. The distribution of difficulty levels by splits is visualized in Figure 2.
Phase 2: Task Queries Creation. For each task template, we generate five distinct task queries by filling in the template with different values. These values are deliberately selected so that queries from the same template vary in difficulty. For instance, one query might require extra scrolling or alphabet-basce scarching to locate the target data, while another might demand keeping track of more previously deselected check boxes. Then, the 5 instances are paraphrased by GPT-5 to mimic the real human queries and increase the language diversity. Finally, two authors manually check all paraphrased queries to ensure language diversity, wording accuracy, and ethics compliance. If any paraphrased query failed meet the standards, the authors would rewrite it before adding it the benchmark.
7 The values within the curly brackets be replaced with the valid values. For the example used here, the link is https://trailbet salesforce com/content/ ea rn wwtrmmnuolee//eeeee /etps//eemmmpppppm httpe///eeemmmmmmmattttttiiool settpp_mmmt_conff pepcee_.
/ 1ex_ px__mpp impp weppleeeettinonnn
5

---

Published as a conference paper at ICLR 2026
Phase 3: Task Initialization & Evaluation. Most of the tasks in SCUBA require prerequisites. These prerequisites include, but are not limited to, synthesizing and uploading data records that the tasks depend on and enabling the org's settings such that certain permissions are granted to allow tasks to be completed. An example can be found in the middle subplot of Figure 2. We also manually craft evaluation methods for each task query using Salesforce's 's APIs, for example, Metadata APIs and Tooling APIs 10 The evaluator not only offers binary 0/1 success score, but also provides milestone scores (process reward) with rubrics. This provides insights into the particular parts where the agent failed during execution. An example can be found in the right subplot of Figure 2.
Phase 4: Human Annotation In the final phase, all instances are sent to the internal annotation team to annotate. Each annotator is provided with an internal annotation tool, review tool, and a set of task instances. The annotation process serves several purposes. 1) Quality Control. This process ensures that all task instances are valid so that they can be completed by human. In addition, this process helps to test whether the results of the evaluators match with the human expectation. 2) Enhance the experiments. Annotated trajectories are used to to potentially improve agents' perfor- mances (discussed in Section 4). More details on annotation process can be found in Appendix F.
EXPERIMENTS
4.1 SETUPS
Since both browser-use agents and computer-use agents can be used to finish the tasks in SCUBA. Here we describe the setups.
Table 2: Comparisons of the setups for two types of agents. A total of 9 agents are tested.
Agent Type
Observation Space
Action Space
Backbone Models
GPT-5, 03, Gemini-2.
Claude-4-sonned
Browser-Use
SOM + DOM texts
Table
Ul UI-TARS-1 55, OpenCUA, Agent-S2.5, OpenAI-CUA, Souude---sonne
Computer- Use
Screenshot
Table 5
Each task of SCUBA can be formulated as a partially observable Markov decision S is the environment states space and A is the agent's action space. T(s'Is, a) S S x A [0, 1] is the state transition probability function that describes the conditional probability of changing to state s' E S given the action a E AA and the state s E S. is the observation space and 22 s',a) O x S x A 10, 1] is probability function describing the conditional probability of observing d' after taking the action a E S and moving to the states s' E S. Denote the agent (a generative model) and the task query Q. At the time stamp f, the task for the agent is to sample action as a+41 ~ xaa H,, Q) with H, = (oo, a1 01, at ). The goal is to maximize the rewards.
Formalization.
process
R(s,a) ()) S + [0, 1] is a reward function.
Observation (O) & Action (A4) Space. For browser-use agents, we choose the Set-of-Mark (Yang et al., 2023) (SOM) and DOM texts as the observation o to approximate the environment state s. SOM places bounding boxes with numeric tags on the interactive elements the screenshot of the browser's s viewport. Each DOM text contains the index, attribute, and label of an element. The observation o for computer-use agents is just the screenshots of the entire OS desktop. Please refer to Appendix B for examples. The action space for browser-use (19 actions) and computer-use agents (15 actions) are described in the Appendix C. We remark that, since different computer-use agents may have different action space by design, we eventually map them to a unified space in Table 5.
Backbone models (T)) Under the constraints of access to different API services and the computation resources to serving models, we make the following decisions about the back- bone models to test. For browser-use agents, we adopt GPT-5 (OpenAI, 2025a), o3 (Ope- nAI, 2025b), C1 Claude-4-sone 2025). For computer-use agents, since there are many candidates, we choose performant ones according to the osWorld leader board. Under the foundational VLM category, we
(Claude, 2025), and Gemini-2.55 5-pro (Comanici et al.,
hnttps://developpet /tt http://ss..sppper..........c///s meta meta/meta_intro htm waw.s http://////////e/eoessss///e//// o h
meta/

---

Published as a conference paper at ICLR 2026
choose Claude-4-sonne (Claude, 2024) with the computer toolbox. Under the native GUI agent, we choose 1L-TARS--1.-11 (Qin et al., 2025), OpenCUA-7B (Wang et al., 2025b), OpenAI-CUA (OpenAI, 2025c), and GUI-Ow1-7B (Ye et al., 2025). Under the agentic frame- work category, we choose Agent-s2.. (Agashe et al., 2025) and Mobi eAgentV3 (Ye et al., 2025) (paired with GUI-Ow1-7R)
Finally, we summarize the above discussion in Table 2 for easy comparison.
4.2 IMPLEMENTATION DETAILS
Table 3: Performances of various agents on the SCUBA. All metrics are averaged accross all in-
stances. next each column name represents the larger value, the better and vice-versa for J.
Method Name
Milestone Score
Success f
Time (min)
Steps
Tokens (k) 4
Costs ($)
Zero-shol Setting
Browser-Use Agents
GPT-5
51.33%
19.31
22.9
245.24
0.55
0.65
45.67%
21.37
26.74
237.16
0.56
Gemi ni-..5-ro
0.47
31.00%
7.27
15.61
134.94
0.24
Claude-4-sonna
0.56
34.67%
11.25
24.12
353.47
1.16
Computer Use Agents
11-JARS-1.5-77
0.10
2.67%
6.14
35.43
183.69
OpenCUR-7B
0.05
0.67%
20.48
31.21
133.88
GUT-Ow1-7B
0.05
1.00%
12.96
34.05
167.6 64
OpenAI-CUA
0.29
16.00%
5.80
31.32
193.83
0.59
-laidd-44sonnee (--munnnttttompueer)
Claude
0.48
27.00%
8.02
35.24
457.77
1.44
MobilcAgent93
0.11
3.10%
21.47
41.96
755.63
1.17
0.58
39.005h
25.13
34.23
597.92
1.17"
Demonstration Augmentted Setting
Browser-Use Agents
GPT-5
0.75
53.85%
17.76
21.33
232.76
0.51
03
0.69
50.00%
17.05
25.33
236.63
0.55
Gemini-2.G-Pre
0.71
46. 15%
10.02
19.48
177.13
0.31
Claude-4-sennes
46. 15%
11.21
Computer- Agents
3I-CARS-1.55 --575
0.24
9.16%
6.52
37.65
213.07
OpenCUA-7R
0.12
5.38%
18.86
33.57
159.11
GUI-Ow1-7B
0.12
2.70%
11.18
36.81
181.50
OpenAI-CUA
28.85%
5.29
27.29
0.52
Claude- Claude-4-sonnee (-smpo-tttttteetmutt
0.67
47.69%
7.33
32.05
424.90
1.34
MobileAgentv3
0.18
7.30%
19.96
42,98
830.85
1.17"
Agent-S2.5(W/PPP
0.57
40.38%
24.63
34.43
626.26
1.19"
We not report the inference prices for self-hosted models.
This does not include the costs of serving UI DI-TARS-1. 1..AARS-1.-777
5-78 as the grounder used in Agent-$2.
Agent Implementation. We primarily focus on testing two types of agents, Browser Use agents and Computer-Use agents. The implementation of the agent loop is adapted from the implementation proposed in browser-use r-se (Müller & Zunič, 2024) and OSWorld (Xie et al., 2024). We made significant changes the DOM parser provided by the browser detecting all interactive elements on the Salesforce platform and handling iframe and cross-origin contents, which results better quality SOM sereenshots and DOM texts. Without these changes, agents run with browser- use would perform significantly worse compared to the number reported in Table We also modify OSWorld's agent implementation to better handle tasks from SCUBA, for example, system prompts.
to improve the quality of
Asynchronous Parallel Inference Infrastructure. SCUBA supports asynchronous parallel eval- uation to reduce the overall evaluation time. The asynchronous support for prowser-use agents is straightforward by spinning up desired number of browsers in the headless mode and assigning each agent to a dedicated browser with one task. For computer-use agents, the asynchronous infer- pipelines starts by creating fixed number of Docker containers remote server with each

---

Published as a conference paper at ICLR 2026
container running Ubuntu system. Whenever the docker container is not occupied by an agent, we immediately assign an agent with a task to it. The container first reverts the system to the initial snapshot. then the agent runs with fresh desktop and new task. Since there are more tasks than Docker containers, this allows the full utilization of resources.
The maximum allowed steps, i.e., the number of predictions made by the agent is set 50. This is justified by the average steps used from the
Parameters.
human annotation, and we allow extra step buffers.
The maximum execution time allowed per task is 90 minutes. We terminate the agent execution loop,
etooe Artion
when either 1) agent thinks the task is done, 2) the
maximum steps budget reached, or 3) the max- imum time budget is reached. The resolution of the screenshot is set to 1920 x 1080 by following OSWor 1d. The generation temperature is set to 1.0 since some models can only support the tempera- ture to 1.0, e.g., GPT-5. We also empirically found that setting the temperature to 1.0 prevents the agents from getting stuck on some particular pages. The agent observation H, is the concatenation of the actions performed up to the time stamp plus the current screenshot.
8u.03
mu orng
unres 15
Clnde cut
Aged s2.
Openttl Openticua
Claude Soover
By Gamelz
Figure 3: Average number of atomic tions used by different methods. Within each method, the left corresponds the zero-shot setting and the right is the demonstration- augmented setting. The gray dashed lines is the human average.
For extended details on the implementation, and metric calculation, please refer to Appendix G.
4.3
MAIN RESULTS
Agents are tested in two settings, 1) zero-shot, where only the task query is used, and 2) demonstrationn-uggeeee augmented. where the task query is combined with human demonstration for the similar or the same task. Considering the second setting is mainly motivated by the observation that agents may lack necessary domain knowledge about the UI design and funetionalities of the Salesforce platform. Like using tutorials to train real person to the Salesforce platform, we prepare the demonstration for agents" !! The results for both settings are summarized in Table 3. Based on the experiment results, we present some observations and analysis.
Demonstration is a effective way to improve performance metrics. As shown in Table 3. The human demonstration can be effective strategy increase the agents' performance metries across the boards. It can consistently improve task success rates, while lowering time consumption, number of steps, token consumptions, and costs. However, there are few exceptions. For computer-use agents UI-TARS-2 1. 5-7-B and OpenCUA-7B, we see that while task success rates are improved, some other metrics also increase. We attribute these to the fact that these two agents cannot use the human demonstration effectively. Also, we empirically found that UI-TARS-1 5-7-B tends to provide answers for Q&& questions in Chinese characters, which affects its performance. For browser-use agents, there is one exception with Gemini- 5- We attribute the increases in the number of steps and costs to the fact that the human demonstration has room to be improved since we observe that this agent can exploit some shortcuts that are absent from the human demonstration. is worth mentioning that the number of steps may not capture the nuances that some agents can generate multiple atomic actions with one prediction. We further add metric called # atomic actions (defined in the action space) and visualize it in Figure 3. Lastly, we can see from Table and Table 6 (in the appendix), agents have varying abilities leverage the human demonstration, and if the agents' tasks success rates are already very high, e.g, GPT-5, then the task success rates improvement may not be that significant.
Design patterns: What type of agent should I use? For enterprise use cases, while having high success rates is crucial, costs and latency are also important factors. To answer the question, we visualize the agents' performance metrics in Figure 4. Methods falling into the green zone are good
Please refer to Appendix D for more details. We's 've conducted additional cost-constrained setting. The details are in Appendix
8

---

Published as a conference paper at ICLR 2026
Amr
Figure 4: Left: Costs Success Rate. Right: Time Success Rate. Orange squares and blue circles represent the browser-use agents and computer-use agents' performance metrics under zero- shot setting, The arrow points the performance metries under the aeemonstration-augmmenn The arrow that points to top-left are desired, since it means improvements. Green zone means low costs/latency and high success rates zone; vice versa for the red zone. Some methods are omitted in the left plot they overlapped (costs are zero).
setting.
candidates, since they have relatively high task success rates and low latency / costs. In general, we observe that computer-use agents have lower latency than browser-use agents at the cost of lower success rates. The high latency of browser-use agents comes from two parts, the API service response time and the multi-agent agent framework design 13 If latency is not concern, browser- use agents can be a better solution. The high task success rates seen in browser-use agents can be attributed to the combination of the stronger planning ability of the foundation models, richer observation space, and the more efficient action space design used in browser-use agents. Overall, we find that the browser-use agent powered by Gemi the balance among the task success rates, latency, and costs.
(with demonstration) strikes
Directions to improve the computer-use agents. Although browser-use agents have higher tasks success rates, the success built upon the tons meticulous customiza- tion of DOM parser for the Salesforce platform. The original DOM parser shipped with browser-use framework failed to capture some types of elements, making many tasks infeasible. On the other hand, computer-use agents only require screenshots, which is easier to acquire robustly. Also, if comparing the performance of Claude-4-a 4-onnet under both computer- use and browser-use settings, the performances are close, hinting that the computer-use agents still have their places. To further improve the computer-use agents, we should improve their generalization ability as shown in Figure 5, The performance drop is significant when moving from OSWorld to SCUBA. When analyzing the trajectories of computer-use agents, found that planning and grounding are still the main causes of task failure. Planning be partially addressed by providing the demonstration, and the primary source of the task success rate improvement. On the other hand, the grounding issue still persists, i.c., the agent failed to correctly predict the coordinates of elements with which tends to interact. However, the agents can make multiple predictions on the coordinates and use majority-voting approach (Yang et al., 2025c), the grounding accuracy might also be improved. More qualitative analysis and discussion on solutions with examples can be found in Appendix H.
Intermncs 0500ttae5OM
Agsm
Clasb-CU
CJI0w-73
Spans-CUA
OpenduA-TO J47605-15-1
Figure 5: Performance drop when moving from OSWorld (50 steps) to SCUBA (without demon- stration).
5 DISCUSSION
While strive provide comprehensive evaluation the computer-use agents' performance on the customer relationship management (CRM) workflows, with a special focus on the Salesforce Platform. This benchmark still has several limitations worth discussing.
The performance for Agent-82. -2.5 with GPT-5 5 also validates this claim.

---

Published as a conference paper at ICLR 2026
The rule-based evaluators only check whether the tasks are completed, yet fail to cap- ture whether the tasks are completed in a trustworthy and safe manner. To be precise, the evaluators in SCUBA are similar to OSWorld, which only check whether the envi- ronment's final state matches the prescribed states, yet can not detect whether the agent sneakily modified deleted task-irrelevant data. Ideally, the robust approach detecting the agent's ' unexpected or unsafe behavior is to compare the environment states before and after the agent runs. However, this approach is prohibitive since the environment state files are huge for the Salesforce Platform, and we can not download all of them, even deplet- ing the daily API quota. So, comparing the before-and- is not feasible. A compromised solution is to understand the failure modes of the agents and add case-by-case detection. We have the annotators inspect trajectories generated by a subset of agents and instructed the annotators to inspect whether the agents made any unin- tended changes to the environment, like adding data not asked for or accidentally deleting irrelevant data. We indeed observed that the agent may accidentally alter some relevant data fields that the task does not ask for. To address it, we add task-specific checks to capture this behavior. Admittedly, this is neither scalable nor comprehensive.
environments' state approach
SCUBA only a subset of enterprise workflows. Enterprise workflow involves many more scemarios than the tasks in this benchmark. According to our survey, a good portion of workflows involve multiple platforms. For example, sales agent might use HubSpot for lead generation, and then use Linkedln Sales Navigator for decision role confirmation, and finally manage lead information in the Salesforce platform. Even just within the Salesforce platform, there are additional tasks from the survey not included in SCUBA, for example, lead scoring and AI-driven insights. These tasks are not included in SCUBA because either the functionalities used in the workflows are paid features or there are no freely available sandbox environments (and the subscription fee for some platforms can be significant).
6 CONCLUSION
In this work, we introduced SCUBA to evaluate computer-use agents on realistic CRM tasks within the Salesforce platform. SCUBA's primary contribution is its focus on realistic enterprise-grade workflows across diverse business roles, addressing a critical gap left by existing benchmarks. Our experiments revealed two significant discoveries. First, browser-use agents' observation and ac- tion space design can better leverage the foundation VLMS' capabilities and outperform specialized computer-use agents at the current stage. Also, a substantial performance gap exists between current open-source and closed-soures models. Second, we showed that augmenting tasks with demonstra- tions a highly effective strategy, improving success rates while simultaneously reducing task time and cost. These findings highlight both the current limitations and the promising future directions for developing capable autonomous agents for complex business software, especially on how to leverage the unstructured documents and futorials more effectively since structured human demonstration is not always available.
We thank Chien-Sheng (Jason) Wu and Kung-Hsiang (Steeve) Huang for shar- ing their insights on developing the CRMArena benchmark series (Huang et al., 2025a;b). We are also grateful to Nesrine Yakoubi, Fabriana Louisa Pita, Michael Thuo, and Caitlyn Cline for their annotation support.
Acknowledgment
Ethics statement The authors confirm that we have carefully reviewed and adhered to the ICLR Code of Bthics in this work.
Reproducibility statement Our codebase is accessible via this link to ensure reproducibility. However, audience should be aware that the Salesforce developer org sandbox is an evolving envi- ronment and is maintained by Salesforce. Therefore, there might be breaking changes we cannot control and might make some evaluators or tasks problematic, thus hurting reproducibility. The authors cannot address this by fixing certain version of the sandbox. Also, the authors have no control over the closed-source model's API service. However, the authors will strive for the best efforts to make the results reproducible and promise to actively maintain the benchmark. If some tasks become invalid due to the environment change, we will replace them with similar tasks.
"https: thub. com les forceA Resea Siaaearch/SCURe
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like human. arXiv preprint arXiv:2410.08164, 2024.
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li. and Xin Eric Wang. Agent s2: A compositional ceneralist-specialiis framework for computer use agents. arXiv preprint arXiv:2504. 00906, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++;: Towards compositional planning and reasoning- based common knowledge work tasks. Advances in Neural Information Processing Systems, 37:5996-6051 2024.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu. Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.0822
2024.
Team Claude. Introducing computer use, new claude 3.5 sonnet, and claude haiku. https: 2024. Accessed:
/ //www. anthropic. wttpoopp.........oooooooooooeeee www.///www////. aaw----mmmmee---d--ooo------ee 2025-09-07.
Team Claude. Introducing claude 4. htttss//www.anthrrpc..coo////// 2025. Accessed: 2025-09-08.
4/,
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXhh:2507.06261 2025.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks? International Conference on Machine Learning. pp. 11642-11662. PMLR, 2024.
Divyansh Garg, Shaun Van Weelden, Diego Caples, Andis Draguns, Nikil Ravi, Pranav Putta, Naman Garg, Tomas Abraham, Michael Lara, Federico Lopez, al. Real: Benchmarking autonomous agents deterministic simulations of real websites. arXiv preprint 2rXf::2504111543, 2025.
Team Google. Introducing gemini deep research. https: http///emmnnii/ii//ee///e//..... wwwgle/oveoieeeee.ee.ee http://////////.....oooooo//o/oo
deep researchs 2025. Accessed: 2025-09-08.
Kung-Hsian Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. Crmarena: Understanding the capacity of Ilm agents to perform professional erm tasks in realistic environments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3830-3850, 2025a.
Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, Divyansh Agarwal, Prafulla Kumar Choubey, Yixin Mao, Silvio Savarese, Caiming Xiong, and Chien-Sheng Wu. Crmarena-proy Holistic assessment of Ilm agents across diverse business scenarios and interactions. arXiv preprint Ar0iv:2505 18878, 2025b.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
11

---

Published as a conference paper at ICLR 2026
Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. Computerrl: Scaling end-to-end online reinforcement learning for computer agents. arXiv preprint arXiv:2508. 14040, 2025.
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed frame- work for emerging {AI] applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18), pp. 561-577, 2018.
Magnus Müller and Gregor Zunič. Browser use: Enable ai to control your browser, 2024. URL https: //ggttbb..ttbb/u hwtmm//wwwwwwueeeeeeeeeeeeeeeeee
OpenAl. Gpt-5 system card. Technical report, OpenAL, 2025a. URL https://cdh..ore openai http://tt---ssstsee cardd cdr.. System Card.
OpenAl. Openai o3 and 04-mini system card. Technical report, OpenAL, 2025b. URL
https:/ cdn openai. /ttp:pp///211111-0000000 saraa-----mmii-systeeee-e
pdf. System Card.
OpenAl.
https: attps://openai.coomm usweeer--uiinggggeeeeee 2025c. Accessed: 2025-09-07.
Computer-using
agent.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioncering automated gui interaction with native agents, arXiv preprint arXiv:2501.12326 2025.
Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, et al. Coact-l: Computer-using agents with coding actions. arXiv preprint arXiv:2508. 033332 2025.
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning, arXiv preprint wwXiv:2509.02444 2025a.
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianboo Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, al. Opencua: Open foundations for computer-use agents. arXiv preprint arXiv:2508.0912 09123, 2025b.
Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang. Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, et al. Mmbench-gui: Hierarchical multi-platfora evaluation framework for gui agents. arXiv preprint 1xXiv:2507.19478 2025c.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's 's transformers: State-of-the-ard natural language processing. arXiv preprint arXiv: 2019.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024.
Tianbao Xie, Jiaqi Deng. Xiaochuan Li, Junlin Yang. Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. Scaling computer-use grounding via user interface decomposition and synthesis. arXiv preprint arXiv:2505. 13227, 2025.
Yiheng Xu, Zekun Wang, Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. In Forty-second International Conference on Machine Learning, 2025.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 200:2550500388 2025a.
12

---

Published as a conference paper at ICLR 2026
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.11441, 2023.
Pei Yang, Hai Ci, and Mike Zheng Shou. macosworld: A multilingual interactive benchmark for gui agents, arXiv preprint arXiv:2506.041:: 2025b.
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, et al. Gtal: Gui test-time scaling agent. arXiv preprint arXiv:2507.0579, 2025c.
Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, et al. Mobile-agent-v3: Fundamental agents for gui automation. arXiv preprint arXiv:2508. 15444 2025.
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et al. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279 2024a.
Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Api agents vs. gui agents: Divergence and convergence. arXiv preprint VrXiv:2503.11066 2025a.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprini arXiv:2401.07339, 2024b.
Xianren Zhang, Shreyas Prasad, Di Wang, Qiuhai Zeng, Suhang Wang, Wenbo Yan, and Mat Hans. A functionality-groondee benchmark for evaluating web agents in e-commerce domains. arXiv preprint arXiv:2508. 15832, 2025b.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons developing process reward models in mathematical reasoning. arXiv preprint a0X 2025c.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307.13885 2023.
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
The large language model used the manual script proofreading to detect grammatical errors and rephrase some overly long sentences. As described in the paper, the LLM (GPT-5) is used to paraphrase the task queries and summarize human annotations. VLMs (see a list them in Table 2) used in the experiments.
B AGENT OBSERVATION SPACE
We provide an example of SOM + DOM texts used as the observation space for the agents running with Efrowser-usee
Satap
Pejert Mnarc
Stattd
TD Co
calaries..
beean Lo0ee
Recently Used
strrew tock
Deftnior
(a) Set-of-Marks Screenshot used for the browser agents
My Besoaads
Wuild Opolee
9266K
(b) Sereenshot of the entire desktop used for the computer-use agents
Figure 6: Different types for screenshots.
Sample of DOM texts
[0]<a Skip to Navigation/>
[1]<a Skip to Main Content/> Setup
[2] [2]<input text; combobox; false; Search Setup/> Search Setup
[3] <b]ttotoon Favorite this itemsThis item doesn't support faverites/>
14

---

Published as a conference paper at ICLR 2026
14] <butten Favorites list/>
[5] <a ;button; false>G1abal Actions/>
[6] <button Guidance Center/>Guidance Center
[7] <b]tbttn Salesforce Help/>
[8] :button; false>Setpp>>>>>>
[9ttttnn Notisications/> new notifications
[10]<button View profile/>
[11]<button false>App Launcher/> Setup
[12]<a tab>Home/>
[]3]<a tab>Objeet Manager/>
[14]<a
Manager List/> Quick Find
[15]<input Quick Find; search/>
[16]<1i Setup HomestrresitemDxxannd
[17]<a Setup Home/>
[18]<1i Service Setup Assistant; ErsistantttreeiiemeExaaaaa
<] Service Setup Assistant/>
[20]<15 treeitem; Comnerce Setup Assistant 2aaksttnnt>xpan//>>
omit for brevity
C AGENT ACTION SPACE
C.I BROWSER-USE AGENTS
Browser-use agents is implemented via the browser-use framework with our extensive modifi- cation on the DOM parser, the action space is described in Table
Table 4: The action space of the browser-use agents.
Action
Primitive Actions
pooso,ou uz lruit st, new sab: bool)
Novigate given URL the currest tab depending on new tab
Navignte the provious page
holding input, tixerindex: int, sext str, cleas existing: bool) aoelt th abitih id str)
wleeen dby--iiiil indexiidxx:
to open new hackground Input cext into element at the gives index: optionally clear existing text first Swirch to the browser tab identified by uis
as
gives
Close
identifled the
sarollldows.
num, pages:
element index; list
upidown a given Scrall to text in the current page.
of pages, optionally specitic
wcroll to text (lestistr)
sensskkysskeyy str)
Send keystrokees to the active clemzat
get get.drogdow.. opt io
option
dropdown
given
lect drogdowe opt cotindex: int, text: wallifecanddi int)
dropdown option
Wait for specifted seconds. Caa be used to was watil the page is fully loaded. Signal that the took is done, with optional files to display
doneclest:
bool, files display: lis(str)
Manipulatonn
wtw...dt........................ (indexcint, patieatri
Upload file to internctive element with file path Write or append content in tile system. file Replace old. with system
c.sto..rioffiic.amee contentistr, append: bool)
xeplacs.rile ststfille name: old str:
Additional Toolboves
search.googi.o
Perform Google seands with the given query page LLM
ructuree. elquerystte pape. himbstr)
semantic
C.2 COMPUTER-UP USE AGENTS
Our implementation is built upon and extends the implementation of OSWorld (Xie et al., 2024). Therefore, the action space of usmputer-use agents the same as the one used in OSWorld. Please refer Section A.3 in Xie al. (2024) for details. We repeat it in Table 5 for completeness.
D CONSTRUCTION OF HUMAN-DEMONSTRATION
To construct the demonstration, summarize the high-level plan with UI elements operations by distilling the human trajectories. turn human trajectories into demonstration examples by prompting the GPT-5 to summarize it into actionable plans. One example is provided below.
15

---

Published as a conference paper at ICLR 2026
Table 5: The action space of the computer-use agents.
Action
Description
Primitive Actions
teex y-y, puraatoornun.eeconn
cuenlinases numus comuseconde
draghe bexk, yey, derat ton-num seconds)
Doag moree to XY cocddnttes over nun seconds seconds. Click at XY coorliattes with specified number clicks and mocuse button.
elick (sek, yey, clicksenun,of_ell Ficlkkk,, Maltoon"leef
rigntClick (exx y-t)) yeY]
Right-elick XY cvordiattes.
Midfle-elick
covedinates.
boe,, y-Y]
Double-click
coontintess.
rrtplellikkeenn y-Y]
Triple-lick at XY cocelimites.
scrall [erruntt stosanalll, xex, ya*)
Scroll the mouse wheel by given amount XY coordinates.
(oeX, y-y, butionn Nect/
XY cooddnntess.
mouselip beek, y-t, pocton="lete"
Relesse mouse
coorliinttss
typewr ihe fatry interve ledel sy.secs)
Type text with optional delay herween keysrokes
hotkey
Press coméirstion of keys simulaneuudly
keyDown (key_name)
keyllp keyll (keyuname) Done
key. Signal that the task is done
The task prompt "Create a Queue with the name "Shoe Case Support". Only send email to members under the distribution list "support.shoe shope @ papaltd.com". Add the Object "Customer". Include "Role: Customer Support, North America" as the Member.".
Sample Human Trajectory
The task is successful. The trajectery is follows:
STEP:1 Perform the action [click] on the element: Log In.
STEP:2 Perform the action [click] on the element: Setup.
STEP: Ferform the action [click] en the element: Sctup.
STEP:4 Ferform the action [click]
the element: Jsers.
STEP:5 Ferform the action [click] on the element: Queues.
STEP:6 Perform the action [click] on the element: New Queue.
STEP:7 Perform the action [write] on the element: Textbox for Queue
Label with the text: Shoe Case Support.
STEP:8 Ferform the action [write]
the element: Textbox for Queue Email with the text: sxsport.shoeppppaaldd.comm
STEP:9 Perform the action [scroll down].
STEP:10 Ferform the action [click] on the element: Customer.
STEF:11 Ferform the action [click] the element: Add to Selected Objects.
STEP:12 Perform the action [write] en the element: Dropdown for Queue Members with the text: Role.
STEP:13
Perform the action [click] on the element: Role: Customer Support, Nerth America.
STEP:14
Ferform the action [click] en the element: Add to Selected Members.
STEP:15 Perform the action [click] on the element: Save.
Sample Demonstration Summarized from Human Trajectory
Successful plan:
Log in and open Setup.
Navigate to Users > Queues.
Click New Queue.
Configure the queue:
Queue label: Shoe Case Support.
Queue Email:
Add supported object:
Select Customer and click Add to Selected Objects.
Add menbers:
In Queue Members, choose Role, select Role: Customer Support, North America, and click Add to Selected Menbers.
save the queue.
16

---

Published as a conference paper at ICLR 2026
For each template, crafted two demonstration examples. As result, two samples are tested with exact demonstration and three samples are tested with the similar demonstration. This setting reveal two important abilities of the agents: 1) Among given set of the demonstration, can the agent identify and follow the exact demonstration to finish the task? 2) Can the agent leverage similar successful demonstration complete the current task? The results presented in Table 6.
E METRICS BREAKDOWN FOR THE DEMONSTRATIONNAAGGEENTT SETTING
setting, we provide two demonstration examples. As a result, two task instances are tested with the exact demonstration examples but the agent still needs to figure out which one to use (corresponds to "w/ demo." column in Table 6). This simplifies the test by setting the recall (of the relevant demonstration) to 100%. The remaining three task instances are tested only with the similar demonstration examples (corresponds to "w/o demo." column in Table 6).
In the demonstration-augnmi
One can make the following observations.
For task instances under the "w/o demo." category, the demonstration examples consis- tently improve milestone score, task success rate, and time consumption metrics. The steps, tokens, and costs metrics are also in general improved only with a few exceptions.
For task instances under the "w/ demo." category, the demonstration examples helps with the methods performance if their performance under the zero-shot setting is not high.
Different models show different capabilities in utilizing the demonstration, with the closed- source models being significantly capable than the open-source models.
Table 6: Performance metries breakdown. The percentage number within the parentheses is the change with respect to the zero-shot setting. All numbers are averaged across samples. We do not report the costs for self-served models. The costs for Agent S2. 5(W/GPT-S 5) does not account for the cost of serving UI OI-TARS-7-2 as grounding model.
Mothod Name
Milestonc Score
Success Rate
Time (min) 1
w/ demo
w/o demo.
demo.
wlo demo.
w/ demo.
who demo.
Browser Use Agents
GPT-5
0,75 (-1.(%)))
0.76 (1.08%)
(-1.27%
0.56 (8.77%)
(-13.849))
(-4.22%)
03
0.69 (1.01%)
0.69 (7.55%)
0.50 (8. 82%)
0.50 (14.29%)
(-23
18:20 1-1.2.9-9..3%% 23%%
Gemin1-2.5-Per 5-P2o
0.72 (60.16%)
0,70 (29, 36%)
0.47 (70.73%)
0.45 (0%.
9.76 (-61 399%)
10.37 (-54.12%)
claude-6-sonne
(16.30%)
(12.34%)
0.42 (26.00%)
(26.67%)
(1.01%)
11.17 (-8.00%)
Computcr-Use Use Agents
U1-TARS-1.5--
0.27 (243. 38%)
0.21 (98.91%)
0.13 (1800.00%)
0.05 (100.00%)
6.62 (5.62%)
6.38 (-4.09))
Open:OUA-7B
0.13 (156 29%)
0.11 (207.20%)
0.05 (300.00%)
0.05 (oo)
18.79 (-10.9366)
18.94 (-7. 68%)
OpenAT-CUA
0.51 (59, 945%)
0.48 (54.01%)
0.31 (64,29%)
0.26 (52.63%)
4.97 (-13.50%%
5.71 (-8. 58%)
Claude- (alll----ooo w-w----ooneeeeee (computer))
0.66 (25.29%)
(22.36%8)
0-48 (60.009))
(-4.50%)
Agent-82. 5(WGPT-5)
0.56 (-8.90%
0.59 (3,291))
0.38 (-13.85%)
0.44 (28.95%)
24.18 (-2.61%)
25.23 (-9.58%)
Method Namo
Steps
Tokens (k)
Costs (S)
w/ demo
w/o demo.
w/ demo.
wlo demo.
w/ demo.
w/o demo.
Browscr-ssc Aguats
GFT-5
(-19.3-1
23.14 (-4336%)
216.49 (- (-19991%)
254,60 (-1.03%)
0.47 (-14.82%)
0.57 (-2.35%
03
24.44 (-6.28%)
26.54 (-7.42%)
227 82 (-2.27%)
248 (-1.92%))
0.54 (-2.03%)
0.59 (-1.98%)
Genini-2.5-Bco
(25)469)
(35.246))
172.28 (27.94%)
(46.72%)
0.30 (26.05%)
(45.05%)
Clauda-4- 4-sonne-
23.99 (-0.25%)
23.82 (-6.44%)
368. 36 (3.54%)
358.39 (.8..99-44199 (-4.19%)
1,21 (3.25%)
1.18 (-4.63%)
Computer-Use Agents
UI-TARS-1.9-1E
39,18 (7.32%)
37.32 (+3.07%)
213.83 (14.69%)
212.05 (5.15%)
OpenCUA-75
33.56 (4.52%)
33.60 (6.27%
158.77 (15 44%)
159. 36 (17.07%)
OpenAI-CUA
25.34 16%)
29.91 (11.44))
159.25 09.299 (- (1165339)
188 89 (-10.2900 29%)
0.49 (-16.64%)
0,58 (-10.43%)
Claude-1 CLaude-I-zonnetlee 1-2onne:
30.55 (-13.69%%)
33.66
(-3.00%)
(-12.80%)
(-3.88%)
Agent-E2, 52.t-52..40 22.55(1TT755)
34.33 (0.75%)
34.56 (-9.55%))
620.41 (5.75%)
634,11 (-7.58%)
1.17 (1.91%)
1.21 (-8.3%%
F ADDITIONAL DETAILS ON THE ANNOTATION PROCESS
Each annotator first needs to pass the on- on-boarding tests in order be qualified for annotation. This process includes learning the basic usage of the Salesforce platform, the annotation and review tools. The annotation tool uses action space similar to that described in Table 4 (only with primitive
17

---

Published as a conference paper at ICLR 2026
actions). Due to the action space design, we also used the DOM parser developed for browser- agents to put bounding boxes with the number tags on interactive elements on the Salesforce web pages. In this way, annotators can prescribed actions with element indices to finish the tasks. This design has two advantages: 1) the collected trajectories are free from redundant and noisy user actions; 2) ensure action space design is self-complete and sufficient for agents to use.
Once the annotator is qualified, it starts the annotation process by first reading the knowledge article understand the intention of the task templates and correct execution steps. Once the annotation is completed, the trajectories are sent for review to ensure final quality. Any rejected trajectories are sent to different annotators for rework.
G EXTENDED IMPLEMENTATION DETAILS
Agent Implementation. We primarily focus testing two types of agents, Browser-Use agents and Computer-Use agents. The implementation of the agent loop is adapted from the implementation proposed browser-use (Müller & Zunič, 2024) and OSWorld (Xie et al., 2024). We made significant changes in the DOM parser provided by the browser-use to improve the quality of detecting all interactive elements on the Salesforce Platform and handling Iframe and cross- cooss-origin contents, which results in better quality of SOM screenshots and DOM texts. Without these changes, agents run with browser-use framework would perform significantly worse compared to the number reported in Table 3. We also modify the OSWorld's agent implementation to better handle tasks from SCUBA, for example, system prompts.
For the prediction service, we either directly use the API services or serve the models with the vLLM (Kwon et al., 2023) whenever it's possible. If we self-served the model, we create 8 replicas of the vLLM online inference endpoints, each occupying GPU. The load balancing is implemented in round-robin fashion. There is only exception with the OpenCUA-7B since there's no vLLM support yet. We tried different solutions and finally choose to serve the model with ray (Moritz et al., 2018) 8) with inference engine from Huggingface's Transformers (Wolf et al., 2019). We created model instances with each on one GPU and the load balancing is handled by the ray.
Asynchronous Parallel Inference Infrastructure. As mentioned in Section 3, SCUBA supports asynchronous parallel evaluation to reduce the overall evaluation time. The asynchronous support for browser-use agents is straightforward by spinning up a desired number of browsers in the head- less mode and assigning each agent to a dedicated browser with one task. For computer-use agents, the asynchronous inference pipelines starts by creating desired number of Docker Containers on a remote server with each container running a Ubuntu system. Whenever the environment is not in use, we immediately assign an agent with task to it. The container first reverts the system to initial snapshot, and the agent runs with fresh desktop. Since we have more tasks than agents, this allows the full utilization of the docker containers.
this paper, the remote server is a n2-standard-96 instance (96 vCPUs and 384 GB memory). If the agent is served with the vLLM, the full evaluation can be done in less than 90 minutes with 40 parallel containers.
To give a full picture of the agents' performance, we design the following metrics, to cover the accuracy, efficiency, and costs, three arguably the most important aspects when developing the agents in the enterprise number between 0 and I) and Task Success(O for failure and for success), which is described in Section 3.2.2. In the latency dimension, we designed two metrics. The Time metric captures the wall clock time required to finish one task. This only captures the actual users' experience, but not the full picture. Therefore, we also use the metrics #Steps to capture more information. The #Steps metric counts the number of prediction made. The nuance is that with one prediction, some agents may predict multiple actions while the others may only predict one action at time. This metric does not distinguish the difference. Lastly, we also report the costs to finish the task. We only provide this number for the paid API service. Specifically, for OpenAI -CUA, it's $3.00/M input tokens and $12.00/M output tokens; for Claude sonnet, it's $3.00/M input tokens and S15.00/N output tokens; for CPT-5, it's $1.25/M input tokens and $10.00/M output tokens; for o3, it's $2.00/M input tokens and S8.00/M output tokens; for Gemini-2 1..11..5-pro,, it's $1.25/M input tokens and $10.00/M output tokens. We also use #Tokens to record the total token consumption. The more token it consumes, the overall time would be proportionally longer and more costly.
Metrics
cases. In the accuracy dimension,
the Milestone Score (a
18

---

Published as a conference paper at ICLR 2026
H QUALITATIVE ANALYSIS
In this section, we provide qualitative analysis of the common failure modes and potential remedies.
Observation 1: Grounding is still a persistent issue for the computer-use agents. In the ex- ample shown in Figure 7, the agent supposed click the "Add" button (right arrow). The Claude-4- -onnee-s--ontttt-t--------- (computer)'s thought is "Perfect! Now I can see "Case" in the Available Objects list. Let me click on "Case" to select it". However, the predicted action is cl iick (378, 693), whose coordinate is on the text "Add" instead of the button (visualized as the red dot). On the contrary, when Claude- 4 sonnet used with browser-use only needs to select the element id. Its prediction becomes click ck.element (index=121) which is correct. One addi- tional grounding failure is provided in Figure
18888
(a) Screenshot for Claude- waw.ud---snnneteeee sonnee
(b) Screenshot for -ouuee---ooneetbrrowss
Figure 7: Different grounding difficulty.
(a) Screenshot for - soonnettooouuuute on (b) Screenshot for Claude- 4- --oonnetootnttutue on step 9
step
Figure 8: Agent selects the wrong cell.
Potential solution. To address the grounding issue for computer-use agents, one potential solution is to ask the models to make multiple predictions (with confidence) about the coordinates and do a majority voting or weighted average. For example, in the example shown Figure 9, at the step 27, the agent's 's thought is "I noticed an "Edit" button at the top of the page, ... it's important to first click this Edit button to enter the detailed settings page." " The initial predicted coordinate was wrong, After multiple attemps, the agent final get the coordinate right the step 34. This idea is aligned with the recent work ( (Yang et al., 2025c). We also want to emphasize that the SOM is not the ultimate solution to address the grounding issue. The out-of-box DOM parser from br browser- cannot detect the add button (index 121) in Figure 9 (b), making the task impossible to be finished. Similar issues happen with other websites and having universal DOM parser that works for all websites is very challenging.
Observation 2: Ineffective history managing strategy can mislead computer-use agents track- ing progress. The history H; at the timestamp used for computer-use agents is the concatenation
19

---

Published as a conference paper at ICLR 2026
(a) step 27
(b) step 34
Figure 9: Multiple attempts can get the right coordinate.
of the past actions and the latest screenshot. This can be problematic and further hurts the task suc- cess rates when grounding accurate. Take the example from Figure 7 again elaborate.
(a) Step
(b) Step
Figure 10: Incorrectly track the progress.
plo
(a) Step 28
(b) Step 29
Figure 11: Additional example of incorrectly track the progress. The agent search the object after 8 steps.
At the step 28, the agent failed to click the "Add" button. But the action click (378, 693) is added the history of the step 29, therefore, the agent's s thought became "Perfect! can see that "Case" is now highlighted in blue, and looking at the Selected Objects section on the right, I can see no longer shows "-None-". Now need to add Alice Bob as member to the queue. Looking at the Queue Members section at the bottom, I can see "User: Alice Bob" in the Available Members list. Let me click on "User: Alice Bob" to select it." (See Figure 10 (b)). We can see that even the agent did not successfully click the "Add" button, it thought the sub-task example could be found in Figure 11, where the agent search for the same object in step 8 and step 16.
done. Another
20

---

Published as a conference paper at ICLR 2026
Potential solution. Reflection is effective way to alleviate this phenomenon. Agent-S2. 5 also did not predict the coordinate correctly in this step. However, it has reflection mechanism to output something like. "(Previous action verification) Previous attempts add "Case" via the Add button and double-click did not move it to Selected Objects. The Selected Objects list still shows "-None-"" With this reflection, the Agent- 5 finally managed click the "Add" button and complete the task successfully. browser use also has the similar design to instruct the agent to evaluation if previous goal was achieved or not.
Observation 3: Human Demonstration is correct yet not necessarily optimal. Recall from Ta- ble 3 and Table 6, there are some cases the human demonstration also increase the latency and costs despite the performances are greatly improved. One reason is that the agent simply does not take enough steps to finish the tasks under the zero-shot setting. The increase in latency and costs can not be avoided. However, there are other cases where the human demonstration is not efficient and the agent can take shortcuts under the zero-shot setting.
(a) zero-shot setting: Step
(b) demonstration-auggeeee augmented setting; Step 4
Figure 12: Agent take different actions under the zero-shot and demonstrationnauggennne settings.
(a) Demmonstratioo--agmeentt setting: Step 4
(b) Demoosttaaioon-augmmeetl setting; Step 5
Figure 13: Agent still can not choose the correct searching box even with demonsttaaion--aggmendd settings.
As shown in the Figure 12, the sub-task is to navigate to the "Report" page. Under the zero-shot setting, the agent issued the action click element (index=58). which clicks setting, the agent predicted lecckeeleeeet....m ement (index=28) - input.text (i (index=98 -> click_e lement (index-105) to achieve the same goal. This differ- ences precisly originates from the human demonstration, where the annotator exactly did " Log in Salesforce and open the App Launcher. Search for and open Reports," And it turned out that the annotator learned this from the knowledge article in the Trailhead website. Another example is presented in Figure 13.
the "Report" tab. However, under the demonstration-augmeenn a sequence of actions as click_e
Potential solution. This is an interesting research direction on how to adapt the knowledge article to the agent's performance. We leave this for future research.
21

---

Published as a conference paper at ICLR 2026
Observation 4: Agents can easily fall into navigation dead ends and fail to recover. As il- lustrated in Figure 14 and Figure 15, even strong agents can accidentally enter irreversible states during multi-step computer-use tasks. In Figure 14, the agent unexpectedly launches new browser window, thereby losing its previous login state and becoming unable to proceed with the remaining steps. Similarly, in Figure 15, the agent opens new tab and subsequently closes the entire browser, leaving itself in an unrecoverable state at step 4. These behaviors reveal that current agents lack robust mechanism for escape actions-they do reliably detect when they are deviating from the expected workflow, nor do they possess the ability to navigate back to safe state once such devia- tions occur. Unlike performance issues that stem from insufficient reasoning or grounding, dead-end states introduce stricter form of failure: even with correct planning afterward, the task becomes impossible complete due to earlier irreversible actions.
Potential solution. A promising direction is developing navigation-reooverabliiyy mechanisms for web agents. These include (1) state-change detection modules that can identify when the UI or browser state diverges from the intended trajectory, (2) explicit "rollback" or "safe-state restoration" policies that recover previous login states or reopen closed contexts, and (3) training demonstrations that emphasize recovery behaviors rather than only optimal trajectories. Designing such robust cape strategies is an important open challenge for building reliable, real-world computer-use agents, and we leave comprehensive exploration of this direction to future work.
Google
(a) The agent started browser and loses the login (b) The agent could not go back through the remaining status, step
steps, step 50
Figure 14: Example of dead end where the agent left the login status.
(a) The agent opened a new tab, step 2
(b) The agent got confused and further closed the whole browser, step 4
Figure 15: Example of dead end where the agent left the login status.
Observation 5: Agents often lack essential domain knowledge to navigate enterprise environ- ments. As shown in Figure 16, even when the interface is clearly visible, the agent may fail to locate critical UI entry points due missing domain-specific knowledge. In this example, the agent repeatedly searches for the Salesforce "Setup" page but still cannot identify the correct navigation path path-even even after 36 steps. This failure is not due to grounding or perception issues: the relevant UI elements are present, and the agent correctly recognizes the surrounding content. Instead, the agent lacks an priori understanding of common enterprise platform conventions (e.g., Salesforce's
22

---

Published as a conference paper at ICLR 2026
gear icon leading to Setup, object manager locations, admin navigation flows). Without such prior knowledge, the agent explores the interface in a trial-and-ers manner, wasting many steps and often failing to reach the correct page at all. This highlights fundamental gap between generic web-navigation ability and the specialized domain knowledge required for realistic enterprise tasks.
Potential solution. One direction is incorporate domain- priors into the agent-e.g- curated Salesforce navigation knowledge, task-specific demonstrations, or symbolic UI maps that specify where essential admin features reside. Another promising approach is to pretrain or finetune on large-scale enterprise interaction logs so the agent internalizes common UI patterns. Overall, integrating domain knowledge remains key challenge for scaling computer-use agents to real en- terprise workflows.
B
(a) The agent does not know where to open setup page (b) The agent still does not know where to open setup in Salesforce website, step 10
page in Salesforce website, step 36
Figure 16: Example of lack Salesforce domain knowledge.
Observation 6: Agents struggle to recover once an early mistake is made. As shown in Fig- 18, even small early mistake can render the remainder of the trajectory ineffective. In this example, the agent accidentally saves incorrect form information at step 10, but instead of recog- nizing the inconsistency or attempting revise the entry, it continues executing future steps under the assumption that the earlier action was correct. By step 50, the agent is still unable to correct the form, resulting in a failure despite otherwise reasonable planning. This showcases common weakness in current computer-use agents: they lack a mechanism for error introspection the abil- ity to detect, revisit, and amend earlier steps when the downstream state diverges from the expected trajectory. Unlike grounding navigation issues, this failure mode is particularly severe because the agent's 's plan may be logically correct, yet the initial incorrect state makes success impossible without rollback or self-correction.
Potential solution. A promising direction is to incorporate action- level verification and self-correction loops, enabling the agent to pe- riodically re-evaluate whether prior actions produced the intended UI state. This could include explicit state-validation checkpoints, rollback strategies, or memory-based action traces that allow the agent to detect when its current state is inconsistent with the task goal. Developing reliable recovery mechanisms is crucial for en- abling robust nong-horizon enterprise tasks, and we consider this an important area for future work.
Karget
I
TEST SETTING
Figure 17: Budget limit per task on the final task success rates. The "default" means under the 50 steps budget used in Table 3.
In this section, we consider budget-constrained test setting to study different method's performance since cost is an important fac- tor when deploying agents in the enterprise scenario. The results are shown in Figure 19. We chose browser- use agents with the demon- stration setup for the testing purpose it is the most performant setup according to Table 3. As shown in the Figure 19, all method's success rates are improved by allowing more budgets.
23

---

Published as a conference paper at ICLR 2026
(a) The agent saved the wrong form information, step (b) The agent not able correct the form until the 10
end, step 50
Figure 18: Example of lack of Salesforce domain knowledge.
Figure 19: Performances of various agents on the SCUBA. All metrics averaged accross all
instances. f next to each column name represents the larger value, the better and vice-versa for J.
Method Name
Milestone Score f
Success Rate
(min)
Steps
Tokens
Costs (S)
Demonstration- -waertrrratii--uuunneeeeee Setting with Browser- Agents
budget 50.5 per task
GPT-5
0.65
47.69%
9.64
14.93
125.98
0.30
63
0.60
40.70%
10.02
18.06
139.09
Gem! n1-2.5-PM Pro
0.61
39.77%
6.29
16.69
129.20
Slaude-4-soane
0.41
23.85%
5.52
13.73
149.01
budget $1.0 per task
GPT-5
0.71
52.40%
12.33
17.64
168.56
0.38
03
0.67
46.54%
14.81
23.55
209.47
0.49
Gemini-2.5-pe
0.67
44.40%
7.54
20.24
171.22
0.23
Claude-4-soanet
0.58
38.08%
9.14
18.32
224.51
0.75
Judicious readers might notice that in Figure 19 the average costs of some agents far below the budget limits. This is because some tasks require fewer steps than others, and even under the default budget setting (termination by steps), the average costs also not exceed $1. To support this claim, we visualize the distribution of costs per task for Claude sonnet in Figure 20.
Dhit Ehis taki (hagerdds
Oad DISTANEEnn Jelytwtii
230
670
04
Figure 20: Cost Distribution per tasks for
using browser-uss framework.
24


Reviews:
[{"review_id": "QoWptkdVmQ", "reviewer": "Reviewer_QXNn", "summary": "SCUBA establishes a comprehensive and enterprise-grounded framework for evaluating computer-use agents on real CRM workflows. It captures the full spectrum of enterprise challenges (UI navigation, data manipulation, workflow automation, and troubleshooting) that prior benchmarks such as OSWorld and WorkArena fail to encompass.", "questions": "- Line 261 mentions the use of ChatGPT — could the authors specify which version was employed?\n\n - Line 258 states that each task template is filled five times. \\Were these fillings done manually or LLM-generated? Additionally, in the example where “one query might require extra scrolling or alphabet-based searching to locate the target data, while another might demand keeping track of previously deselected checkboxes,” were these variations manually created?\n\n- I am uncertain how demonstration examples were obtained for each test sample. Were they directly taken from the two demonstrations associated with the corresponding task template? If so, how should we interpret Line 811, which states that “two samples are tested with exact demonstrations and three samples are tested with similar demonstrations”?\n\n- Since the authors have shown that adding human demonstrations to every test sample is not always optimal, I am curious about potential gains from incorporating simple memory mechanism, similar to the agent memory module implemented in MMINA.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The overall presentation of the benchmark—covering dataset construction, filtering, and annotation—is exceptionally coherent and methodically executed. The benchmark is clearly grounded in realistic business workflows and real-world use cases.\n\n- The paper demonstrates clarity in explaining implementation details, e.g., the description and justification of the chosen inference engine.\n\n- I especially appreciate that the benchmark provides rule-based evaluators with milestone (process) rewards, enabling multi-dimensional performance assessment.", "weaknesses": "Personally, I find this benchmark meaningful and thoughtfully described. My comments here are merely minor suggestions: (1) The authors might consider evaluating a broader range of open-source models [1,2,3]. It would also be worthwhile to update UI-TARS to the latest UI-TARS-2B version. I understand this may not have been feasible before the ICLR submission deadline, but such updates could be included in the camera-ready version.\n\n---\n\n[1] Aria-UI: Visual Grounding for GUI Instructions\n\n[2] Mobile-Agent-v3: Fundamental Agents for GUI Automation\n\n[3] MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HX7bwfc2dd", "reviewer": "Reviewer_X9Qy", "summary": "This paper introduces SCUBA, a new benchmark designed to evaluate computer-use agents on realistic customer relationship management (CRM) workflows within the Salesforce platform. It contains 300 task instances derived from real user interviews, covering roles like administrators, sales representatives, and service agents. SCUBA features a sandbox-based Salesforce environment, rule-based milestone evaluation, and demonstration-augmented evaluation for assessing enterprise agents' performance. The experiments benchmark nine agents (browser-use and computer-use paradigms) under zero-shot and demonstration-augmented conditions, revealing substantial performance gaps between open- and closed-source models, as well as between paradigms.", "questions": "How do the authors plan to maintain task validity when Salesforce updates its sandbox environment or API schema?\nWhat percentage of failures stem from grounding vs. reasoning vs. action space limitations? Could process rewards help isolate these error modes quantitatively?\nWill SCUBA support online fine-tuning or reinforcement learning in the sandbox (e.g., for ComputerRL-style training)?\nGiven the availability of fine-grained rewards and high-quality labeled trajectories, why did the authors not conduct fine-tuning or reinforcement learning on open-source models to build a stronger GUI agent baseline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Timely and highly relevant: The paper tackles an important gap—evaluating GUI-based LLM agents on enterprise-grade workflows, rather than consumer or synthetic tasks.\n- Strong realism and comprehensiveness: SCUBA operates in real Salesforce sandbox environments, using realistic personas and tasks drawn from actual user interviews.\n- Clear methodological contributions: Task construction pipeline (template → paraphrase → initialization → evaluation) is well-detailed and reproducible.Fine-grained, interpretable milestone scoring system provides richer signals than binary success rates.", "weaknesses": "- Insufficient qualitative error analysis: While quantitative metrics are extensive, more detailed examples of failure cases (e.g., grounding errors, navigation dead ends) would better illuminate where agents struggle.\n- Demonstration quality underexplored: The authors note some demonstrations are suboptimal (e.g., missing shortcuts), but the process for ensuring demonstration consistency and coverage could be better justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PzqB3P0Jmb", "reviewer": "Reviewer_YpFg", "summary": "This paper introduces SCUBA (Salesforce Computer Use Benchmark), a new benchmark designed to evaluate computer-use agents on Customer Relationship Management workflows within the Salesforce platform. SCUBA contains 300 task instances derived from real user interviews, covering three primary personas: platform administrators, sales representatives, and service agents. It tests a wide range of enterprise-critical abilities, including UI navigation, data manipulation, workflow automation, information retrieval, and troubleshooting.", "questions": "1. The paper highlights a dramatic performance drop for agents moving from OSWorld to SCUBA, attributing it to planning / grounding capabilities. This analysis could be deeper. Does this failure simply indicate a domain shift, or does it suggest that benchmarks for general desktop applications (like OSWorld) are poor proxies for the challenges of complex, web-based, stateful enterprise applications?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The benchmark runs in live Salesforce sandbox environments and uses fine-grained evaluation metrics that capture milestone progress, not just final success. This indicate some progress, as typical benchmarks like OSWorld[1] and ScienceBoard[2] only provide a 0/1 final success.\n2. Tasks are sourced from real user interviews, ensuring practical relevance and representativeness. The evaluation is conducted within a live Salesforce sandbox env, offering higher fidelity than simulated platforms.\n3. The study systematically evaluates both zero-shot performance and a \"demonstration-augmented\" setting. It empirically validates that providing demonstrations is an effective strategy.\n\n[1] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments https://arxiv.org/abs/2404.07972\n\n[2] ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows https://arxiv.org/abs/2505.19897", "weaknesses": "1. While advances have been made in the eval metrics, the SCUBA's infrastructure is wholly dependent on the Salesforce \"sandbox\" environment. This constitutes a significant threat to the long-term validity and reproducibility of the benchmark and imposes a heavy burden on the maintainers.\n2. The current scale is relatively limited for capturing the vast complexity of an enterprise platform like Salesforce. Furthermore, existing works like AndroidWorld[3] and ScienceBoard[2] commonly employ randomized parameters to initialize the scenarios faced by the scaled agent tasks.\n3. The task distribution is heavily skewed (57% for the 'admin'), which may limit the benchmark's ability to evaluate agent performance comprehensively across all core business roles.\n4. For the experimens, the claim “The high task success rates seen in browser-use agents can be attributed to the combination of the stronger planning ability of the foundation models, richer observation space, and the more efficient action space design used in browser-use agents\". This comparison between \"Browser-Use\" and \"Computer-Use\" agents seems to be methodologically unbalanced.\n    a. The browser agents (or cuas leveraging a11ytrees [6]) receive a more structured observation space (SOM + DOM texts), which inherently poses no challenge for grounding.\n    b. In contrast, other computer-using agents (e.g., SeeClick[3], OS-Atlas[4], UI-TARS[5]) take raw screenshots, which necessitates explicit grounding.\n5. Minor: SCUBA is deeply coupled with Salesforce. While this provides depth in specific business scenarios, it sacrifices breadth. Conclusions drawn from this benchmark regarding UI nav and DOM-related stuffs may be difficult to generalize directly to other platforms.\n\n[3] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents\n\n[4] SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents, ACL 24\n\n[5] OS-Atlas: A Foundation Action Model For Generalist GUI Agents\n\n[6] UI-TARS: Pioneering Automated GUI Interaction with Native Agents\n\n[7] OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis https://arxiv.org/abs/2412.19723", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "maH0dsf9yM", "reviewer": "Reviewer_Hddk", "summary": "The paper presents SCUBA, a benchmark for evaluating computer-use agents on enterprise-grade CRM workflows within the Salesforce platform. Unlike prior browser-oriented or service-specific benchmarks, SCUBA emphasizes realism through sandboxed Salesforce environments, task instances sourced from user interviews across administrator/sales/service personas, and fine-grained rule-based evaluation with milestone scores. The authors benchmark a wide range of representative agents under zero-shot and demonstration-augmented settings.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Tasks are grounded in real enterprise workflows and executed in genuine Salesforce sandboxes rather than synthetic mockups, addressing a major gap left by OSWorld/WebArena-like benchmarks.\n\nThe inclusion of process rewards, latency and cost metrics, and rule-based evaluators provides more diagnostic insight than binary success metrics commonly used in prior work.", "weaknesses": "1. As acknowledged, benchmark stability and long-term reproducibility depend on third-party platform updates, which may undermine comparability over time. Is it possible to provide a version-frozen evaluation environment to improve the reusability of the results?\n\n2. The paper reports cost/time for agent inference, but does not quantify the annotation cost or the engineering overhead associated with constructing and maintaining SCUBA.\n\n3. The reported improvements (e.g., from demonstration augmentation) are presented as absolute numbers. Could the authors conduct a statistical analysis to verify the robustness of these gains?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pNQ2Sujijg", "reviewer": "Reviewer_JVzK", "summary": "This paper introduces a new benchmark for evaluating AI agents that use computers in only one website. The authors design a set of tasks, a rule-based evaluation system, and several case studies. The work involves notable engineering effort, but the paper lacks clear novelty and convincing evidence of realism.", "questions": "With rule-based evaluation, how to control side effects of the auto-execution. E.g. The model rm -rf all files before clicks a correct button? (maybe this is an extreme case)\n\nWhy use Q&A for troubleshooting as an additional type of agent execution？Maybe it should use some execution to find that why \"yes\" or \"no\" but not directly \"only answer yes or no\"?\n\nStrong models like GPT and Claude pass near 50% of the tests, how to prove that the reasons of fails are the ability of the model or agent, but no an unreasonable task setting?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The benchmark demonstrates notable implementation effort on a real web interface, reflecting practical interest in evaluating agent actions.\n\nThe paper clearly describes the task setup and examples, which helps readers understand the benchmark design.\n\nThe focus on web-based computer use provides a concrete starting point that could be extended to broader environments in the future.", "weaknesses": "The claim of “realistic” tasks is not well supported. Many prior benchmarks also make this claim without proper evidence.\nSeveral important analyses and examples remain in the appendix. Some should be moved to the main text to support the main arguments.\n\nThe evaluation method is underexplained. The paper mentions rule-based evaluators but gives no clear description, pseudo-code, or examples. In Figure 2, the element ground_truth_dict appears, but its role in evaluation is unclear.\n\nThe rule-based evaluation raises safety concerns for automatic execution. It is unclear how potential side effects (e.g., unsafe file operations) are prevented.\n\nThe “Q&A troubleshooting” tasks look inconsistent with the main task types. Asking models to answer “yes or no” is unstable and not representative of real computer-use behavior.\n\nThe example with a 31-step task does not seem realistic. Long tasks like this are unreliable and should not be split into smaller tasks without showing how performance transfers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)