CoAct-1: Computer-using Multi-agent System with Coding Actions
Authors: Linxin Song
Yutong Dai
Viraj Prabhu
Jieyu Zhang
Taiwei Shi
Li Li
Junnan Li
silvio savarese
Zeyuan Chen
Jieyu Zhao
Ran Xu
Caiming Xiong
Venue: ICLR 2026 Poster (2026)
Keywords: Computer-using Agent
Multi-gent System
LLM Agent
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose CoAct-1, a hybrid multi-agent system that combines GUI control with direct code execution, achieving SOTA performance and improved efficiency on OSWorld and WindowsAgentArena by dynamically delegating tasks to GUI or coding agents.

Abstract: Autonomous agents that operate computers via Graphical User Interfaces (GUIs) often struggle with efficiency and reliability on complex, long-horizon tasks. While augmenting these agents with planners can improve task decomposition, they remain constrained by the inherent limitations of performing all actions through GUI manipulation, leading to brittleness and inefficiency. In this work, we introduce a more robust and flexible paradigm: enabling agents to use coding as an enhanced action. We present CoAct-1, a novel multi-agent system that synergistically combines GUI-based control with direct programmatic execution. CoAct-1 features an Orchestrator that dynamically delegates subtasks to either a conventional GUI Operator or a specialized Programmer agent, which can write and execute Python or Bash scripts. This hybrid approach allows the agent to bypass inefficient GUI action sequences for tasks like file management and data processing, while still utilizing visual interaction when necessary. We evaluate our system on the challenging OSWorld and WindowsAgentArena benchmark, where CoAct-1 achieves a new state-of-the-art success rate of 60.8% on OSWorld and 52.5% on WindowsAgentArena, significantly outperforming prior methods. Furthermore, our approach dramatically improves efficiency, reducing the average number of steps required to complete a task to just 10.15 on OSWorld, compared to 15 for leading GUI agents. Our results demonstrate that integrating coding as a core action provides a more powerful, efficient, and scalable path toward generalized computer automation.

Published as a conference paper at ICLR 2026
CoACT-1: COMPUTER-USING MULTI-AGENT SYSTEM WITH CODING ACTIONS
Linxin Song", Yutong Dai", Viraj Prabhu", JJeyy Zhang", 3 Taiwei Shi Li Junnan Li
Silvio Savarese*, 2 Zeyuan Chen", Jieyu Zhao', 1 Ran Xu', Caiming Xiong"
'University of Southern California SSalesforce University of Washington
Code: http:p///.attttoooo///aaaaaaaaaa
ABSTRACT
Autonomous agents that operate computers via Graphical User Interfaces (GUIs) often struggle with efficiency and reliability on complex, long-horizon tasks. While augmenting these agents with planners improve task decomposition, they main constrained by the inherent limitations of performing all actions through GUI manipulation, leading to brittleness and inefficiency. In this work, we introduce a more robust and flexible paradigm: enabling agents to use coding as an en- hanced action. We present CoAct-1, novel multi-agent system that synergistically combines GUI-based control with direct programmatic execution. CoAct-1 fea- tures Orchestrator that dynamically delegates subtasks to either conventional GUI Operator or a specialized Programmer agent, which can write and execute Python or Bash scripts. This hybrid approach allows the agent to bypass ineffi- cient GUI action sequences for tasks like file management and data processing, while still utilizing visual interaction when necessary. We evaluate our system on the challenging World and Windows/ggenAArens benchmark, where CoAct-1 I achieves a new state-of-the-art success rate of 60.8% on OSWorld and 52.5% on AinnttssAgeeAAree significantly outperforming prior methods. Furthermore, our approach dramatically improves efficiency, reducing the average number steps required complete task to just 10.15 on OSWorld, compared to 15 for leading GUT agents. Our results demonstrate that integrating coding as a core action provides more powerful, efficient, and scalable path toward generalized computer automation.
INTRODUCTION
Recent advancements in computer-using agents have primarily focused on operating through Graphi- cal User Interfaces (GUIs). While these GUI agents, powered by languae (action) models (Li
2023 Deepmind 2025a by OpenAl 2024, 2025a) Qin al., 2025 Xie al., 2025a Yang et al. 2025). have demonstrated the ability to perform variety of tasks, they often struggle with long-horizon planning and interactions in environments with dense GUI elements. For example, routine tasks in Office productivity software often involve long and intricate sequence of precise GUI operations, such as locating specific table within a multi-sheet spreadsheet, filtering it based on complex criteria, copying the results, and saving them as finding all image files a nested directory structure, resizing them to specific dimensions, and then compressing the entire directory into single archive brittle and inefficient when solved via GUT actions such as clicking and dragging. In these scenarios, existing agents often struggle with visual grounding ambiguity (c.g., mistaking visually similar icons or menu items) and the cumulative probability of errors over long-term interactions. A single mis-click or misunderstood UI element can derail the entire task.
al.
CSV file. Similarly, tasks like
To address these challenges, prominent line of research has focused on augmenting GUI agents with dedicated high-level planners. Approaches such as GTA-1 (Yang al. (025) and other modular
Work done during the internship at Salesforce.

---

Published as a conference paper at ICLR 2026
systems (Yang et al., 2024; Xu et al. 2024| Agashe et al. 2024:
2025) utilize powerful language
models like OpenAl o3 (OpenAl 2025b) to decompose a user's 's high-level goal into a sequence of more manageable subtasks. This herarchical decomposition can improve performance complex, multi-step problems by providing a structured plan. However, this paradigm does not fundamentally address the inefficiency and brittleness associated with exclusive reliance on GUT-based execution. Even with the high-level planning, the agent still needs to navigate menus, click buttons, and type into fields, even for operations that could be accomplished more directly and reliably through programmatic means. This leaves the system susceptible to planning uncertainty, visual perception errors, and the integration challenges between high-level planning and low-level action generation.
In this work, we advocate for and instantiate a more flexible and powerful action space. We propose a hybrid approach that combines the intuitive, human-like strengths of GUT manipulation with the precision, reliability, and efficiency of direct system interaction through code. We introduce CoAct-1 (Computer-using Multi-agent System with Coding Actions), a novel multi-agent system composed of three specialized agents: Orchestrator, Programmer, and GUI Operator. A high-level Orchestrator serves as the central planner, decomposing the user's goal and determining the appropriate modality for each subtask. Based on this analysis, it assigns the task to one of two distinet execution agents: Programmer agent, which writes and executes Python or Bash scripts for backend operations like file management, data processing, or environment configuration; agent that performs frontend actions like clicking buttons and navigating visual interfaces. This dynamic delegation allows CoAct-I to strategically bypass inefficient GUI sequences in favor of robust, single-shot code execution when appropriate, while still leveraging visual interaction for tasks.
GUI Operator, a VLM-basedd
Our experimental analysis provides strong evidence for the advantages of this hybrid design. On the OSWorld and SindowsAgeetAree benchmark, CoAct-l establishes a new state-of-the-art achieving an overall success rate of 60.76% and 52.50%, respectively. This marks a significant improvement over leading baselines like Agent S2.5 (55.98%) on OSWorld. The performance gains are most pronounced categories where programmatic control is highly advantageous. For instance, Cale (70.21%) multi-application (47.88%) and VS Code (78. (8) tasks, our Programmer's ability in exccuting precise scripts lcads to substantial gains over the strongest GUI-only methods. Beyond improving success rates, our dual-modality approach dramatically enhances operational efficiency. By replacing long, error- error-prone click sequences with concise code, CoAct-1 I solves tasks in an average of just 10.15 steps on OSWorld, a stark contrast to the 15 steps required by agents like GTA-1. This efficiency underscores the potential of our approach to pave a more robust and scalable path toward generalized computer automation.
2 RELATED WORK
Screen parsing and visual grounding A first line of work focuses on perceiving and grounding GUI elements directly from pixels, without relying on DOM or accessibility hooks. OmniParser learns screen-parsing primitives for pure vision-based understanding (Lu et al., 2024). On the grounding grounding), Aria-Ul (instruction grounding over GUTs), and UGround (universal GUI grounding) map language to actionable screen locations (Cheng et al. 20024] Yang et al., 2024 Gou al. 2024). OS-Arlas trains a foundation action model to generalize across diverse interfaces (Wu et al.; 2024). Dedicated grounding evaluations such as ScreenSpot-Prr further benchmark grounding under professional, higl-resolution settings (Li et al. 2025).
side, SeeClick (instruction-to-taree
Native end-to-end GUI agents A third thread trains native agents that unify perception, reasoning, and action single model. UI-TARS and OpenCUA exemplifies this approach with unified action space for mouse/keyboard operations across apps, eschewing hand-crafted controllers (Qin et al. 2025) Wang et al. 2025a b). AGUVIS pushes toward unified, pure-vision GUT agents that generalize across interfaces (Xu al. 2024).
Modular prounerrgrounder agents A second strand explicitly separates what to do from where/how to act screen: a language planner proposes subgoals while visual model grounds each step. Representative systems include SeeClick and OS-Atlas (Cheng et al., 2024; Wu et al., 2024). GTA-1 strengthens this two-stage paradigm via test-time scaling: sampling multiple candidate actions and using an MLLM judge to select among them, improving robustness on bigh-resolution, cluttered UIs (Yang et al. 2025). Other related open frameworks such as Agent-S / Agent-S2 and AutoGen
2

---

Published as a conference paper at ICLR 2026
provide reusable infrastructures for multi-agen agent orchestration and tool calling (Agashe al. 2024
2025
Song et al.) 2025al Zhang et al. 2024; Wu et al. 2023 Zhang et al. 2023 2025b).
Hybrid agentic frameworks Beyond GUI-only interaction, several agentic systems compose tools and APIs on the fly to extend capabilities at run time, Examples include UFO-2 (Zhang et al. 2025a
Py Vision (Zhao et al. 2025). BeyondBrowsing (Song al. (2225)) and ALITÄ (Qiu et al. [0025]
which, while not restricted to GUI/CUA, share the principle of dynamically constructing and invoking tools.
3 COMPUTER-USING AGENT WITH CODING AS ACTIONS
User
Task
Subtask
Subtesk
Programmer
Orchestrator
GUI Operator
Summary.
Summary..
Operation
Result
Screenshhot
Screenshot
BB
Screonshat
Operating System
Figure I: Multi- system design for our CoAct-l. This multi-age agent system includes a Programmer that interact with the operating system through multi-round coding. This multi-agent system includes an Orchestrator, which serves as high-level planner that decomposes goals and delegates subtasks appropriate execution agent, a Programmer, which interacts with the operating system through multi-round coding, and a GUT Operator, which leverages vision-language capabilities to perform visual interface actions.
In this work, introduce new system-interactive action:
coding, replace part of the redundant and brittle GUI actions. Unlike summarizing APIS or SDKs from each application or website, we focus on enabling agents perform free-form coding solve computer-use problems guided by a strong language model. Specifically, we design a multi-agent system, CoAct-1, that introduces a new agent, Programmer, capable of interacting with the OS through a boding-obbervaane An Orchestrator serves as the high-level controller, determining whether to assign the subtask to the Programmer or the GUI Operator. The overall framework illustrated in Figure
circle.
3.1 PROBLEM DEFINITION
formalize the problem of general computer control as an interactive decision-making pro- cess. At each timestep t, the agent observes the computer environment (primarily consists of a screenshot) o, E O, and takes an action a, AA according to a policy π [a,l He, Here, H, (01, 01:--0000011 0 represents the historical context, and G is the user's high-level goal provided in natural language. Learning an effective policy is particularly challenging when the action space A is restricted to low-level GUI operations. Complex tasks, such managing nested files or processing spreadsheet data, can require long and intricate sequences of GUI actions. This makes the process inefficient and highly susceptible to error propagation, where single mis-check can derail the entire task.
To address this limitation. we introduce a hybrid action space that integrates direct programmatic control. We augment it to the standard GUI action space, denoted as A Aqui U Acode- An action a, E AGUI involves the direct manipulation of the graphical interface (e.g.. mouse clicks, keyboard typing). In contrast, an action a, E Acode consists of a Python or Bash script that interacts directly with the operating system's 's backend. This allows the agent to perform complex operations like file manipulation or data processing in single, robust step, effectively bypassing brittle and inefficient GUI sequences. In CoAct-1, the policy π is implemented hierarchically. A high-level Orchestrator acts meta-policy, Torch, which analyzes the current subtask and delegates it to one of two specialized executor policies: a GUI Operator that implements тदu for actions in AçuI- or a Programmer that implements Code for actions in Acode-
3

---

Published as a conference paper at ICLR 2026
Could youlhelp me unaip downioaded oxtension flle from hhwwc/lsse/Drsstttgg hww.hm//ssereetttoool and configure in Chrome''s oxtensions?
Programmer
Orchestrator
GUI Operator
Stbttakk
Round call,
Subhtask
Undip the fll hhomeluse iDesknep! hol Exterstonnn dtp mo Destio
hommiiuser Desktbeett on on ats Inomeinnneeetttg
Configure the Unpacked Extens on Chrome
1. do Desitop
code
Fio 2071 DeanssiseeeDstgghhheeeeeeee
Navigating extennionn page WAIT
184)
ueenDesntop octo "Dosstop directory
LLM Summary
actai "Cument cieeclory: Siowti"
Abund cal.gu agent
Task: Contigure the Unparked Extensioo in Chrome
Code outputt
Bcreenshot
OUI Operator Sunmmary
Google
IPP arcteve founci huttEoursiin..0 Archive he heloEraaskkn..p
[Omiitedd Muttiple Tuma)
Fetraction cumpleted succeesriall. Extraction contanta:....
and dicway dicovayconectt connecty.
TERMINATE
Figure 2: Illustration of CoAct-l workflow. Given task, Orchestrator choose either call Programmer or GUT Operator to solve subtask. A programmer can interact with the OS by coding, and GUT Operator can interact with the OS by performing GUT operations.
3.2 MULTI--GENT SYSTEM DESIGN FOR COMPUTER USF
Our multi-agen agent system is the architectural instantiation of the hierarchical policy outlined in the problem definition. It comprises three specialized agents- the Orchestrator, Programmer, and GUI Operator that collaboratively generate the action trajectory to solve the user's goal G. Each agent establishes a dedicated conversation to perform its role.
Orchestrator: The Orchestrator embodies the high-level meta-policy, Torch- It is responsible for task decomposition and dynamic planning based on the full history of observations H, and the overall goal G. The Orchestrator does interact directly with the OS. Instead, its primary function is to select the best specialized sub-policy, TCode or TGUI. to execute the current subtask. Upon completion, the Orchestrator receives summary of the execution process and new observation o++1 (a screenshot along with summary reflecting the current system state) to inform its next decision. If it determines that the overall goal G is met, outputs a termination signal.
The Programmer implements the specialized policy #Code responsible for generating actions a, Açode- Upon receiving subtask from the Orchestrator, initiates a multi-round conversation with code interpreter. It generates Python Bash scripts, which are executed by the interpreter. The feedback, consisting the code execution results, allows Programmer to reflect and refine their code until the subtask is solved. The Orchestrator provides the Programmer with sufficient context, such as file paths or window information inferred from H,, to ground its code generation.
Programmer.
GUI operator. The GUI agent is a vision-language action model that implements the GUI-based policy Taun for generating actions a, A.uu- Similar to the Programmer, the GUI Operator engages multi-round interactive loop to complete its assigned subtasks. In each step of this "perception- action" loop, the agent takes the current screenshot and the subtask instruction input generate a single GUI action (e.g., a mouse click or keyboard input). A GUT action interpreter executes this action on the OS, which in turn provides new screenshot as observational feedback. The GUI Operator uses this new visual information to decide on its subsequent action, continuing this eycle until its subtask complete.
3.3 WORKFLOW AND MEMORY DESIGN
The hierarchical policy implemented by CoAct-1 necessitates a structured workflow and memory system manage the flow of information between agents. This design ensures that each agent has the necessary context without being overwhelmed by irrelevant details from other parts of the task. The overall workflow is illustrated in Figure
4

---

Published as a conference paper at ICLR 2026
Workflow The workflow begins when the Orchestrator, acting the meta-policy Torch delegates subtask to an appropriate executor agent: either the Programmer or the GUI Operator. The selected agent then engages in its own multi-round interactive loop solve the subtask, generating detailed conversational history of its actions and the environment's responses. Upon completing the subtask, this detailed history is processed by dedicated summarizer model. This model condenses the entire interaction into a concise summary that captures the key actions taken and the final outcome. This summary, along with the final screenshot representing the new state of the environment, then returned to the Orchestrator. This handoff mechanism provides the Orchestrator with condensed, high-level update to its overall task history H,-
Memory design for task decomposition CoAct-1 employs a hierarchical and isolated memory structure:
Orchestrator Memory: The Orchestrator maintains the long-term, primary memory, which corresponds to the historical context H, from our problem definition. It consists of the initial user goal G and the sequence of summaries and screenshots received from the executor agents after each completed subtask. This aggregated history provides the context for all high-level planning decisions.
Executor Memory: The Programmer and GUI Operator each maintain short-term, working memory that is active only for duration their assigned subtask. This memory contains the "instance conversation history" of their multi-round interaction with the OS.
To ensure modularity and focus, these memories are isolated; the agents do not share their conver- sational histories directly. Furthermore, once executor completes its subtask and reports back to the Orchestrator, its working memory is cleared. This reset mechanism critical, as it allows the executor agents to focus entirely the context of the subtask they receive without being influenced by prior, irrelevant interactions.
4 EXPERIMENTS
4.1 BENCHMARK DATASETS
We evaluate C on OSWorld (Xie et al.) 2024) and WindowsAgentArena (Bonatti et al. 2024). Both are scalable real-computer testbed that exposes an OS (Windows or Ubuntu) to an agent through pixel streams and an OS shell interface. OSWorld comprises 369 tasks, while AindowsAgentArena comprises 154 tasks. These task span common productivity tools, IDEs, browsers, file managers, and multi-application workflows, thereby challenging both vision- language grounding and long-horizon planning in heterogeneous GUI environments.
4.2 BASELINES
We compare CoAct-1 with two categories of computer-usine using agents, the end-to-end models and agentic methods. These baselines represent the forefront of GUl-based task automation.
End-to-end model An end-to-end model takes user instructions and OS sereenshots as input and outputs corresponding actions by pure inference without any agentic workflow. OpenAl o3 (OpenAl 2025b) is a cutting-edge reasoning model from OpenAl that excels at multi-step problem-solving and versatile, context-aware assistance. OpenAl CUA 4o (OpenAl 2025al uses vision and reasoning to interact with graphical user interfaces, controlling the mouse and keyboard to perform tasks. It is the technology behind services like Operator and ChatGPT agent. UI-TARS (Qin et al. [025) UI- TARS introduces fully end-to-end, screenshot-only native GUI agent model that unifies perception, reasoning, memory, and action. OpenCUAA(2B)) (Wang et al. 2025b) introduces fully open-source framework, including scalable data collection tool, the first large-scale multi-OS computer-use task dataset, a reflective chain-of oh-tthttht reasoning pipeline, and strong vision- -sionnlanguage agent models.
The agentic method encompasses single- and multi-agent systems with diverse structures, such as blanner-grounder cammerr-ullgrrondder. etc. These baselines primarily focus on enhancing the ground-level ability of language model improve computer-base performance.
Agentic Method
5

---

Published as a conference paper at ICLR 2026
Jedi-7B w/ o3 (Xie et al.
2025b) We refer the term Jedi to Qwen2.5-VL trained on the Jedi dataset. The author plugs Jedi into an agent stack to translate high-level plams into pixcl-perfect OSWorld, WindowsAgentArenn. and multiple grounding
GUI actions, achieving large gains benchmarks. GTA-/ w/ 03 ( (Yang et al. 2025) The GUI Test-time Scaling Agent (GTA-1) is a GUI agent that addresses the challenges of planning ambiguity and action grounding in high-resolution interfaces. To improve accuracy, this agent employs a "test-time scaling" strategy, where it generates multiple possible actions and utilizes Judge model select the optimal one. It also leverages the GRPO to train powerful grounder. Agenr S2.5 w/ 03 (Agashe et al. p025 Agent S2.5 is compositional
framework in which a planner generates high-level subgoals, multiple grounders executes them while delegating GUI-element localization to visual, textual, and structural experts via Eexture-of-Grrunndiig and both levels proactively replan after every subgoal to remain robust to changing screens.
Besides the above powerful baselines, we also add Agent $ (Agashe et al. 2024) and NAVI (Bonatti et al. 2024) as baselines for WindowsAgentArena.
4.3 IMPLEMENTATION DETAILS
Environment We test the C Linux with extended RESTful server from OSWorld. Specifically, we implement a remote code interpreter that can take long Python and Bash scripts as input and return the execution result back to the sender. On the other hand, for each task, OSWolld will establish an initial state, such as opening a set of apps or specific websites, or downloading the specified files to a specified location, etc. After the initial state is ready, we will take a screenshot as the initial input along with a user task to CoAct-1 and baselines.
CoAct-1 settings We implement 1 using AG2 (Wu et al. 2023). In CoAct-l, we adopt OpenAI o3 for Orchestrator and OpenAl 04-mini for Programmer. For the GUT Operator, we use OpenAl computer-use caaputer-use-preveeee use-preview, a Dision-language action model finetuned by OpenAl for computer use, as the backbone model. We use the 04-mini as the summarizer for summarizing the conversation history between the Programmer and the Orchestrator. We set the maximum round I for the Programmer to 20, the maximum step K for the GUI Operator to 25, and the maximum round J for the Orchestrator to 15. Therefore, the number of system interactions, i.e, the number of steps, for CoAct-1 I is upper bounded to 375 (but in all cases, as shown in Figure 3d CoAct-1 1 will early stop before 150 steps). More details are in Appendix C
Evaluation Protocol We evaluate our method with the rule-based evaluator provided by OSWorld and WindowsAgentArena. Internally, every evaluator is expressed as a Boolean expression built from 134 atomic, execution-based evaluators that the authors handcrafted for the benchmark. For a given task, the benchmark composes these atoms with logical AND / OR operators, so a "pass" might require, for instance, (file exported AND MD5 matches) AND (email sent True).
4.4 RESULTS
Our experimental results, detailed in Table and Table 2, unequivocally establish CoAct-1 as the new state-of-the-art across two challenging, real-world computer operation benchmarks. The findings validate our core hypothesis: integrating programmatic actions alongside traditional GUI manipulation provides more robust, efficient, and generalizable paradigm for computer automation.
Performance on OSWorld On the comprehensive OSWorld benchmark Tabll 1). CoAct-1 demon- strates superior performance and efficiency. achieves a final success rate of 60. 76% within the 150-step limit, creating significant margin over the strongest contemporary agentic frameworks, including Agent $2.5 w/ o3 (55.98%) and GTA-1 w/ o3 (53.07%). The strength of our hybrid architecture is not only in its peak performance but also in its consistency across different task complexities. At the 100-step mark, CoAct-l already leads with a 59.93% success rate, surpassing the final scores of all other baselines. The advantages of our approach are particularly pronounced in categories where programmatic control is most effective. CoAct-1 achieves top-tier performance in or file system interactions, scoring 64.80% in Office, 75.00% in OS, and 47.87% in Multiple Apps. This exceptional performance in domains that are historically brittle for pure-GUI agents underscores the efficacy of delegating backend operations to the Programmer agent,
tasks requiring complex data

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of the state-of- oftte-ofthheart methods the OSWorld (Xie 2024) verified benchmark. We split the results by steps and show the approach type from LibreOffice Cale, LibreOffice Impress, and LibreOffice Writer. Daily tasks include tasks from Chrome, Thunderbird, and VLC. Professional includes tasks from GIMP and VSCode. We report the success rate (%) as the evaluation metrie for each type of task, and mark the best result of each step budget in bold and the best result overall step budgets underlined bold.
second column. Office tasks include tasks
Agent
(104 tasks)
Daily (78 tacks)
Professional tasks)
tasks)
Multiple Apps tasks)
Avg.
15 steps
OpenAl
1.45
8.02
12.29
UI-TARS-IA 1.5 (7B)
27.19
27.99
61.45
34,78
5.38
25.76
OpenAl CUA 4o
27.17
37.65
41.22
45.83
10.75
26,01
OpenCUA
26.69
33.25
51.57
43.48
10.41
28.12
Agout $2.5 w/
42.85
44.61
57.10
70.83
17.82
38.98
Jedi-7B w/
45.84
$7.49
60.95
50.00
20.43
42.37
CoAct-1
47.18
42.30
47.74
66.67
23,82
39.81
50 steps
OpenAT
11.50
19.78
UI-TARS-1.5 (7B)
26.51
OpenAl CUA
23.56
38.43
52.09
70.83
15.86
OpenCUA
30.06
42.31
58.28
47.83
16.79
33.76
GTN-1-7R w/ o3
48.58
52.31
77.84
58.33
37.05
48.59
Jedi-7B w/ o3
50.10
65.25
68.65
54.17
34,97
50.65
Agent $2.5 w/.5./oom
52.81
55,80
75.42
75.00
39.53
54.21
CoAct-1
62.91
59.43
69.89
70.83
42.37
56.38
100 steps
OpenAT
17.23
26.29
UL-TARS-155 (7B)
OpenAl CUA
25.00
39.19
OpenCUA
30.06
38.89
60.70
$2.17
18.10
33.84
Jodi-7B w/ 63
47.89
64.37
75.92
50.00
35.27
50.98
GTA-1-7B w/ o3
55.68
64.74
61.20
62.50
38.34
$3.07
Agent $2.5 w/
$4.23
$5.80
75.42
75.00
44.06
55.98
CoAct-1
64.80
61.60
71.82
75.00
47.87
59.93
150 steps
CoAct-1
64.80
66.51
71.82
75.00
60.76
Table Comparison of the stateoofthhe-a methods on the WindowsAgent/Arena (Bonatti al. 2024 split the results by steps and show the approach type in the second column. Office tasks include tasks from LibreOffice Calc and LibreOffice Writer. Web tasks include tasks from Chrome, and Microsoft Edge. Windows System includes tasks from settings and Filc Explorer. Windows System includes tasks from Settings and File Explorer. Windows Utils includes tasks from Clock, Windows Calculator, Notepad, and Microsoft Paint. We mark the best result in bold.
report the success rate (%) as the evaluation metric for each type of
Method
Office433 tasks)
Web:30 30 tasks)
Windows System:24 24 tasks)
VSCude tacks)
VLC1211 tasks)
Windows Utibs tasks)
Agent
13.3
29.2
19.1
22.2
NAVI
30.3
sieps)
sepp)
CoAce-1 stopa)
which can execute precise and reliable scripts for tasks involving spreadsheets, file manipulation, and cross-application data flows.
Performance on WindowsAgentAreem We further evaluated CoAct-l on the WindowsAgentAremn benchmark (Talle 2). The results show that our framework successfully transfers its capabilities different operating system, again achieving state-of-the-art performance by substantial mar- gin. CoAct-1 attains an overall success rate of 52.5%, which is remarkable improvement over prior leading methods like Agent S2 (29.8%) and NAVI (19.5%). The performance breakdown on WindowsAgent. ArenA further reinforces our central claim. CoAct-1 shows commanding strength in system-level and utility-based tasks, achieving standout scores of 83.3% in Windows System and 77.7% in Windows Utils. These categories, which involve interacting with file explorers, system settings, and other native utilities, are ideally suited for the script-based actions of the Programmer. Furthermore, CoAct-1's 's performance demonstrates clear and effective scaling with an increased step
7

---

Published as a conference paper at ICLR 2026
Average Steps Solved Tasks
GUI Coding Action Steps Overall Tasks
Number of tasks in bin
Coding actions
14.90
GUI actions
Average steps 88
60
10.15
10
40
6665
GTA-1
U1-TARS(78: OpenAi CUA 40
CoAct-1
75
100
125
150
Method
Total steps per task
(a) Average steps per passed task with 100 step budget, showing CoAct-1 is significantly more efficient than other SOTA agentic frameworks like GTA-1.
(b) Distribution of tasks by total step count, illustrating the ratio of coding to GUT actions and showing that coding helps reduce the total action steps.
Pass Fail Ratio Split by Total Steps
1.0
w/ and w/o Coding Action Step in the Passed Tasks
Number of tasks in bin
80
Fail
Pass
Preportion of steps
w/ Coding Coding
0.8
60
0.4
40
0.2
2325
20
Librofftce Cale Mulli Apps
05
VSCode
Libreffiie Writer
LbrrOffice impress
VLC
Thundorbin
Gimp Corome
12.0
2
0
25
50
75
100
125
150
Domain
Total steps per task
(c) Breakdown passed tasks by application domain, highlighting that coding actions are most frequently applied in complex domains like LibreOffice Calc, Multi-Apps, and direct OS interaction.
(d) Pass/fail ratio split by total steps, demonstrating that the failure rate is positively correlated with the number of actions required.
Figure 3: CoAct-1 Efficiency and Step Modality Analysis.
budget, rising from 21.4% 15 steps to 52.5% 100 steps, highlighting its capacity to solve more complex, long-horizon problems.
In summary, the consistent, state-of-thed the-trt performance across two distinct and challenging bench- marks confirms that CoAct-1's 's hybrid agentic architecture represents significant advancement toward creating more capable and reliable autonomous agents for general computer use.
4.5 DISCUSSION
Efficiency analysis The analysis of CoAct-1's operational efficiency, illustrated in Fiiuuee reveals that our hybrid approach is substantially more efficient than leading GUI-only agents. This efficiency is a key factor in its improved success rate. As shown in Figure 3a CoAct-1 solves tasks with an average of 10.15 steps. This represents significant improvement
other high-performing agents like GTA-1, which requires 15.22 steps, and UI-TARS, which needs 14.90 steps on average. While OpenAI CUA 4o averages fewer steps (6. 14), its overall success rate much lower compared to CoAct-l's 's (31.40% v.s. 59.93% on 100 steps). This indicates that CoAct-l's efficiency is coupled with greater effectiveness. The source of this efficiency lies in the strategic use of coding actions. Figure sbpporrss this by showing that coding actions help keep the total steps per task relatively low. This efficiency crucial for robust performance, Figure 3c shows that coding is particularly beneficial in complex domains like "LibreOffice Calc", "Multi-apps", and direct OS interactions, where a large proportion of tasks are solved with code. A single script can replace a long and error-prone sequence of GUI clicks, streamlining the workflow. Figure 3d illustratss clear trend: tasks that require more actions are more likely to fail. By reducing the total number of steps, the hybrid approach not only accelerates task completion but also minimizes the opportunities for error.
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance of CoAct-1 with different backbone model for each participant agent. Powerful Orchestra- tor significantly help improve the performance on OSWorld.
GUI Operator
Orchestrator
Programmer
Performance
04-mini
04-mini
43,43
OpenAl CUA
58.72
o3
04-mini
60.76
Table 4: Ablation study on the performance CoAct-1's components. We compare the full hybrid system against agents restricted to using only the Programmer (pure coding) and only the GUI Operator (pure GUI). The results highlight that the integrated CoAct-I system significantly outperforms either single-modality agent, demonstrating effectiveness of its hybrid approach.
CoAct-1
Office
Daily
Professional
OS
Multiple Apps
Avg.
Avg. Steps
w/ Programmer w/ GUT Operator
40.88
16.17
53.06
62.50
29.63
35.73
1.14
43,50
58.80
69.38
79.16
35.68
50.68
11.20
64.80
66.51
71.82
75.00
47.87
60.76
10.15
The ability to dynamically select the most appropriate action- direct coding command or GUI interaction-is fundamental to the enhanced efficiency and reliability of CoAct-l.
CoAct-1 with different backbone We investigated the impact backbone model selection for each agentic component of CoAct-l on the OSWorld benchmark, with results presented in Table 3 Our analysis reveals that the overall system performance is highly sensitive to the reasoning and cntructioonfollowiinn capabilities of the models chosen for the Orchestrator and Programmer roles. When utilizing 04-mini for both the Orchestrator and Programmer, alongside the OpenAl CUA 4o the GUI Operator, the system achieved a performance of 43.43%. A significant performance enhancement to 58. 72% is observed when a more powerful model, o3, is used for both the Orchestrator and Programmer. This underscores the critical role of a sophisticated high-level planner and capable code generator in the system's 's success. The highest performance of 60.76% was achieved with heterogeneous configuration: employing o3 for the Orchestrator, 64-mini for the Programmer, and retaining OpenAl CUA 4o for the GUI Operator. This configuration suggests an optimal balance, leveraging the powerful reasoning o3 for task decomposition and delegation, while benefiting from the specialized capabilities of 04-mini for code generation. These results highlight that enhancing the capabilities of the Orchestrator and Programmer yields the most substantial performance gains, validating modular design and demonstrating the benefits of strategically allocating powerful models to roles with high reasoning demands.
Pure GUI action VS pure coding action To isolate each agent modality and validate our hybrid design, conducted an ablation study comparing the full CoAct-1 system with agents restricted to single action type. Results in Table 4[show the combined approach is superior. Programmer-only agent (pure coding) achieved 35.73% success, highlighting that many tasks require GUI interaction beyond scripting. It was highly efficient, averaging just 1.14 steps per success, confirming the directness of programmatic actions. A GUI Operator-only agent (pure GUT) achieved higher 50.68% success rate, handling more task types. but required 11 1.20 steps per task. The full CoAct-1 model, integrating both modalities, achieved 60.76% success with 10.15 steps on average, demonstrating the synergy of our architecture: the Orchestrator exploits the Programmer's efficiency for backend tasks and the GUT Operator's 's versatility for visual navigation, yielding robust system.
5 CONCLUSIONS
In this work, we introduced CoAct-l, a novel multi-agent system designed to address the inherent inefficiency and brittleness of agents that rely exclusively on GUI manipulation. Our multi-agent system features Orchestrator that dynamically delegates subtasks GUI Operator Program- mer. Our extensive evaluation on the OSWorld and WindowsAgentAree benchmark confirms the effectiveness of this approach. CoAct-1 achieves state-of-the-ar success rate of 60.76%

---

Published as a conference paper at ICLR 2026
OSWorld and 52.5% on WindowsAgentArena. significantly outperforming previous leading methods. The performance gains were particularly pronounced categories involving OS-level interactions, multi-application workflows, and other tasks where the Programmer agent could leverage direct programmatic execution.
REFERENCES
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that computers like human. arXiv preprint 2rXiv:2410008166 2024.
Saaket Agashe, Kyle Wong. Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang, Agent s2: A compositional generalist-specialss framework for computer use agents. arXiv preprint arXiv:2504.00006, 2025.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal agents scale, 2024. URL https: wawii.iiiiv.ooca org. /abs/
24C9.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprini arXiv22401.10933, 2024.
Deepmind. Introducing gemini 2.0: our new ai model for the agentic era. Technical report, -ttp::eeeeeea--------aa------ee- wttp::/www..aaaaaarerroeeeeeeeea
Deepmind, 2025a URL
deepmind//
2024 httpa//ee022200111et-tat-a--
Deepmind Gemini 2.5: Our most intelligent ai model. Technical report, Deep- http///oooleeeemmmrrddd.d.
mind, 2025b. URL mach march
gemin ni model wwt-thiikinggn updttein-upptttss
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.
arXiv preprint arXiv:2410.05243, 2024. URL/https://axxiie org. abs 2410.42002200 2410.05211
Jia He, Mukund Rungta, David Koleezek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. Does prompt formatting have any impact on I1m performance?, 2024. URL https///////xi
org/ abs 2411. 10541
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, ICML' 233. JMLR. org, 2023.
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981, 2025.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408. 00203, 2024.
OpenAl. Gpt-4o system card, 2024.
OpenAI. Computer-using agent: Introducing universal interface for ai to interact with the digital world. 2025a. URL htt.ittps::/
com http////owwww//cooooooeeeeouseno
OpenAI. Openai o3 and 04-mini system card.
Technical report, OpenAl, 2025b. http:://dd...oo http:://cco..ppaaaaccoooo///p//p 020
URL
13-and-o4-min mini- www.imissiieeccrrreeeee pdf System Card.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 2025.
10

---

Published as a conference paper at ICLR 2026
Jiahao Qiu, Xuan Qi, Tongcheng Zhang. Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, Xing Zhou, Dongrui Liu, Ling Yang, Yue Wu, Kaixuan Huang. Shilong Liu, Hongru Wang, and Mengdi Wang. Alita: Generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution, 2025. URL https: //arxiv org/abs/2500 2505. 20286
Linxin Song, Jiale Liu, Jieyu Zhang, Shaokun Zhang, Ao Luo, Shijian Wang, Qingyun Wu, and Chi Wang. Adaptive in-oonversation team building for language model agents, 2025a. URL
all///ss//22/ 22000./220.11111 19425
Yuegi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api- baied web agents,
2025b. httplttt////xii..ooooo http//aaxivvioro. abs 241./2400..0 16464
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint w:Xiv:2509.02544 2025a.
Xinyuan Wang, Bowen Wang. Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang. Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, et al. Opencua: Open foundations for computer-use agents. arXiv preprint arXiv:2508.09) 09123, 2025b.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen 11m applications via multi-agent conversation, 2023. URL
https://arxit org/ abs//aa///// 2308 08155
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang. Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents, arXiv preprint urXiv:2410.232
2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. (tttttts:: org.iiioorrraaae /awp/////////
Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang. Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang. Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025a. URL
13227
Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, al. Scaling computer-use grounding via user interface (tl::2505.13227 2025b.
decomposition and synthesis. arXiv preprini
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint 24X.:2412.004555 2024.
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Caiming Xiong, and Junnan Li. Gtal: Gui test-time scaling agent, 2025. htlttttss///aattt http:/iwww.rg/b////0/000 2557/
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.1622 2024.
Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, Liqun Li, Yu Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Ufo2: The desktop agentos, 2025a. httL/htttaa//rrat//
org/ abs/250/ 14603
Jieyu Zhang, Ranjay Krishna, Ahmed H. Awadallah, and Chi Wang. Ecoassistant: Using Ilm assistant more affordably and accurately, 2023.
03046

---

Published as a conference paper at ICLR 2026
Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. Offline training language model agents with functions as learnable weights, 2024. URL
https://arxiv: org/ abs /tss//202.11333
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which agent causes task failures and when? on automated failure attribution of Ilm multi-agent systems, 2025b. URL https 7/ //axxiv. abs /2005. 00212
Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Ming Li, Qilong Wu, Kaipeng Zhang, and Chen Wei. Pyvision: Agentic vision with dynamic tooling, 2025. URL Stl.https: https:://axiv..rggg///oo wwtprxiwworggga///ooo///
2507.0792
A LLM USAGE STATEMENT
We used a large language model (OpenAI's o3 and GPT-5) as a general-purpose writing assistance tool. Its role was limited to sentence- and paragraph-level polishing, including improving clarity, grammar, and flow. The authors developed all research ideas, analyses, results, and conclusions. The model did not generate new content, perform literature review, or contribute to the conceptual framing of the paper.
B ETHIC STATEMENT
Our research aims to advance computer automation for beneficial and productive purposes, but we acknowledge the potential for dual-use and associated risks.
An agent with the ability to execute code (Python or Bash) and manipulate GUI could be leveraged for malicious activities not properly constrained. To mitigate this risk during our research, all experiments were conducted within secure, isolated, and virtualized benchmark environments (OSWorld and WindowsAgeetAArena) This ensures that the agent's 's actions are sandboxed and cannot affect real-world systems or data. We advocate that any future deployment of such agents in live environments must incorporate robust security protocols, strict permission controls, and mechanisms to prevent the execution of harmful code.
Security and Misuse:
The agent's operation relies on observing the sereen via screenshots, which real-world scenario could contain sensitive or personal information. In this work, we only the data provided within the benchmark tasks, which do not involve real user data. For any future applications, is imperative to implement strict data handling policies and Sawwacy-rresevving techniques to protect user confidentiality.
Data Privacy:
C REPRODUCIBILITY STATEMENT
Reproduction of CoAct-l requires an accurate use of specific OpenAl models and prompts. Please review the following details ensure the accurate performance of work.
C.I MODEL USAGE AND ENVIRONMENT SETTING
Model Usage In this work, 03-2025-04-18 for Orchestrator, 14-mini-2025-044 for Programmer, and computer. sourced model can also be adopted by CoAct-1 if meets the following requirements:
preview-22 2025-0222000000 03-11 for GUI operator. Any open-
For Orchestrator The Orchestrator requires multi-mon modality input (image and text) to process all screenshots from the OS for planning. It also requires a strong reasoning ability across different modalities (See Table 3 reasoning ability will largely affect the final performance).
For GUI Operator The GUI Operator be replaced with any open-sourced vision language action models (VLA) like UI-TARS (Qin et al.] (025) or OpenCUA (Wang et al.] (025b). or planner-grounder approaches like GTA-1 (ang al. 2025) and Agent S2 (Agashe et al. 2025)
12

---

Published as a conference paper at ICLR 2026
CoAct-1 require the GUI Operator to have the following two abilities: (1) instruction following ability out of the grounding task (like when to terminate), and (2) computer use ability, including clicking, dragging, ryping, and hotkeys. Unfortunately, we have not been allocated enough GPU resources for testing these models' performance on CoAct-1 in this work, and you can expect a reduction in average steps and a performance improvement when switching stronger model for the GUI Operator.
For Programmer Language model for Programmer should have a strong ability to write Python and Bash (or Powershell in Windows) for solving computing problems. Note that we didn't provide any API or SDK for programmers; however, this can be an extension to your approach for better performance on app-specific tasks, such as Chrome and Thunderbird.
Environment The OS environment is also a significant challenge when evaluating computer- using agents. To reproduce our experiment results, all input screenshots to the Orchestrator should accurately reflect the well-initialized system state. Therefore, we recommend waiting 60 seconds after the VM starts before capturing the OS screenshot to ensure the screenshot includes all necessary information for Orchestrator to plan.
C.2 PROMPT DESIGN FOR CoACT-1
The performance of any modern large language model agent system can be largely affected by its prompt or template design (He et al. 2024). In CoAct-I, despite the rules for general behavior like chain-of-thouuhh and verification, we notice that the GUI Operator has a significant hallucination rate when performing self-checks, while the Programmer's file modifications do not reflect in the OS, the Orchestrator cannot capture and plan for the next step. We mitigate these limitations in our prompt design by allowing Orchestrator to check the result independently and reload the file modified by the Programmer. We put the prompt used for WSWorld and WindowsAgentArremn in Table 5
also design rules based the model limitations. Specifically,
Table 6 Table 7
Table 8 and Table
D WHEN&WHY COACT-1 FAILS? INSIGHT FOR FUTURE WORKS
To better understand the capabilities and limitations of CoAct-1, this section analyzes failure cases observed during evaluation. Generally, errors in task completion arise from four primary challenges: high-level, ambiguous queries, reflection errors, and hallucinations.
High-level query A high-level query one where the user's s instruction does not directly map to sequence of actions. Instead, it requires the agent first to infer the user's 's underlying intent and the broader context before it can devise a solution. For instance, one task in the VSCode domain instructed the agent: "Please help me modify the setting of VSCode to keep my cursor focused on the debug console when debugging in VSCode, instead of automatically focusing back on the Editor." " In this scenario, the Orchestrator delegated the task to the Programmer. The Programmer attempted to find the relevant setting by searching for keywords like "debug" and "console". However, it failed to make the conceptual leap that the debugging process relates to "breakpoints." " Consequently, it overlooked the correct setting, "JocusEditorOnnraak
leading to the task's 's failure. This case highlights limitation in the agent's ability to reason about concepts that are not explicitly mentioned in the query.
Ambiguous query An ambiguous query user request that is vague or omits critical information necessary for successful task completion. Resolving ambiguity often requires the agent to correctly infer the user's intent, which can also involve safety considerations. An example of this occurred in VSCode task with the instruction: "Please help me modify VSCode setting to hide all " "_yccche____ folders in the explorer view." The Orchestrator assigned this subtask to the Programmer. The Programmer successfully identified the need to modify settings file but incorrectly altered the workspace-speciite settings instead of the global user settings. This misinterpretation of the query's 's scope resulted in the task failing. This illustrates the challenge the agent faces in disambiguating the user's 's intent when multiple valid interpretations exist.
Reflection error Reflection is crucial mechanism for verifying the task completion process in CoAct-1. In our system design, only the final state of the OS will be returned to the orchestrator after the GUT operator completes the task. This issue yields the final error if the GUT Operator makes some
13

---

Published as a conference paper at ICLR 2026
middle-state errors and the operation covers them. For example, when performing spreadsheet operations, the GUI Operator may mistype in a cell (let's say a cell in row A), then scroll down the spreadsheet and stop there. In this case, method will return screenshot of the OS that excludes the error value in row A. This error cannot be captured by the Orchestrator and will cause the task to fail due the unexpected operation
Hallucination As one of the most important topics the large language model era, hallucination also appears as common reasons that cause CoAct-l failures. All agents in CoAct-1 will hallucinate when task-solving, and the most significant hallucination comes from the Orchestrator and GUI Operator. The Orchestrator will provide error plan, usually including advanced forecasting, that affects the GUI Operator and Programmer. For example, the Orchestrator may predict the content of an unopened website and instruct the GUT Operator to work on the non-existing content. The GUI Operator will also hallucinate the reasoning process and imagine that it has already completed the assigned task. this work, we mitigate hallucination from Orchestrator by prompting the Orchestrator to perform verification more frequently and cross-verify the results between the GUI Operator and the Programmer.
System Prompt for Orchestrator (Part 1)
Today is (today]. ).
## Your Role
You are responsible for completing computer-based task, step by step, using the tools provided.
You are working on a (system info system.
#### Step-by-Stee Process
1. Describe the Screenshot Carefully review and clearly describe the screenshot's content.
2. Plan the Task
Create detailed, step-by-step plan to solve the task. List all user requirements, including exact file names, file paths, and any other specifics in the output (not the thinking).
Execute the Instructions
Think carefully and follow the user's instructions exactly. Do not make any changes not requested by the user (such as renaming files or changing file content).
You must apply all the changes the computer. If the task is impossible (e.g., missing files, wrong environment), reply with INFEASIBLE end the conversation.
For file operations (like modifying spreadsheets), you MUST try the Programmer (call programmer) first.
When the user ask you to create new sheet in spreadsheet, always name it sequencially. For example, 'Sheetl', 'Sheet2', etc.
Verify the Result
ALWAYS check the result through the screenshot by yourself. You can let a GUI Operator to navigate to the correct location for you. After the GUI Operator complete, the screenshot will automatically be returned to you.
If you used the Programmer to modify file, have the GUI Operator reopen the file to see the updated results.
Ensure that the result meets all user requirements.
All the things out of the user's instructions should not be changed.
Table S: System Prompt Orchestrator (part 1).
14

---

Published as a conference paper at ICLR 2026
System Prompt for Orchestrator (Part 2)
##
Tools You Can Use
### Programmer (call programmer)
Can run Python or Bash code to perform most file or system tasks.
Needs clear environment description and detailed task instructions.
Can use any Python package you specify.
After modifying file, ALWAYS verify every change by yourself. You can let GUI Operator to navigate to the correct location and check the result by yourself. If something is wrong, tell the Programmer to fix it.
Programmer will return summary of its task solving process after completing the task. No screenshot is provided after the Programmer completes the task.
### GUI Operator (call gui operator)
Can interact with the GUI by clicking exact position, scrolling, dragging, typing, and using hotkeys.
Require detailed task description.
The GUI Operator may not able to complete your task in 100% of accuracy and often make mistakes.
Have 25-step limit, each step is a single OS interaction (one click, one hotkey/typing action, etc.).
Do not let the GUI Operator to do any result check. You need to do it by checking the screenshot yourself.
I will return screenshot that reflect the final state of the computer after completing the task.
You don't need to prompt the GUI Operator do this.
Note: Only call ONE tool (call_ programmme or call qqui_poperatoo per reply.
Table 6: System Prompt for Orchestrator (part 2)-
System Prompt for GUI Operator
# Your role
You can control the computer by clicking, scrolling, dragging, and typing. Think carefully and execute the user's step-by-step instructions.
# Credentials
The user's password appears.
Use it when a system password prompt
# Operating rules
Keep apps open at the end of the task.
If the UI doesn't appear, perform a brief, deterministic retry (e.g., refocus and re-click).
Do not close the window, minimize the window unless told to do so.
# Response protocol
When you think the requested task is completed or cannot be completed, reply exactly: 'TERMINATE: <1. detailed description of what you currently. As detailed as possible. 2. What you did to complete the task or why this task cannot be completed.>"
Table 7: System Prompt for GUT Operator.
15

---

Published as a conference paper at ICLR 2026
System Prompt for Programmer
# Your role
You are the lead programmer. Solve the user's task step by step using the terminal (supports Python and Bash).
Your username 'user"; the sudo password is 'CLIENT PASSWORD'.
The terminal streams real-time execution output.
# Coding format
Submit one fenced code block only, labeled with its language:
"bash
# Your Bash script here
To use sudo, follow this pattern:
echo CLIENT SEITT_PSSSWRDD sudo -S <your commands>
or
"* "pyyhhon
# Your Python code here
# Do not use: name_ == main_ ": (it will suppress output)
# Requirements
File names: Do not rename files or change extensions during any file operation unless the user explicitly asks.
Code fence language: Every fenced block must specify the language ("bash' 'python'); otherwise you will receive 'unknown language unknown'.
Siingle block: Wrap all code in one code block-do not split your submission across multiple blocks.
Spreadsheets: When editing spreadsheets, ensure every value is written to the intended cell and preserve the original formatting (fonts, colors, sizes, etc.).
Dependencies: Before importing or using package, check whether it is installed: if not, install it in your submission.
Observability: Print intermediate results aid debugging. for example, the value you modifying.
Final review: Before completion, carefully inspect your result by writing test cases and confirm that nothing outside the user's instructions has changed.
Table 8: System Prompt for Programmer.
16

---

Published as a conference paper at ICLR 2026
Prompt for LLM Summarizer
# Programmer <>> Terminal Log Summarizer (No Timeline/En//Nex Actions)
Summarize Programmer Terminal logs for the Orchestrator they can decide the next step immediately. Orchestrator's task: '(task]'
Role:
Execution history: '(chat_ cccha_historyy (prompts + outputs).
## Output
1) Summary (2-4 lines) task, what tried, current status, why (cite key log lines / cxit codes).
2) Commands (deduped)
"bash
# unique commands in run order; annotate repeats (xN)
3) Terminal excerpts
"ttext
minimal evidence: head(-10)... ... [truncated N lines] ... tail(~10) # always include full error traces and return codes
4) Artifacts Side effects files/dirs changed (paths + purpose); installs/miggaaionn. Spreadsheet: list cells/ranges edited and confirm formatting preserved.
5) Errors / Blockers precise messages exit codes; likely root cause from logs (no speculation).
6) Verification what checks passed (tests, file existence, row counts); what still needs verification (e.g., reopen file and confirm cell Y).
## Rules
EEvidenceffisss first, no speculation.
Deterministic truncation (head/tail; note omitted lines); always include error stacks.
Call out deltas (what changed intended).
Keep it tight: bullets > prose.
Table 9: Prompt for LLM Summarizer.
17


Reviews:
[{"review_id": "Wn7XWHzyeG", "reviewer": "Reviewer_PXQ7", "summary": "CoAct-1 proposes a hybrid approach for autonomous computer agents that leverages both GUI manipulation and programmatic scripting to execute tasks more efficiently and robustly. The system introduces three specialized agents—Orchestrator, Programmer, and GUI Operator—to dynamically switch between GUI actions and Python/Bash scripting based on task requirements. The approach is evaluated on benchmarks like OSWorld and WindowsAgentArena, demonstrating significant performance improvements in success rates and task completion efficiency over traditional GUI-only systems.", "questions": "How does CoAct-1 handle tasks that require multiple rounds of interaction with both the GUI and programmatic code? Is there a risk of inefficient back-and-forth between agents?\n\nWhat improvements are planned for enhancing the system’s robustness in handling ambiguous or complex user instructions, especially in non-structured environments?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Hybrid Approach: The combination of GUI manipulation and coding as an action creates a powerful, adaptable agent system. The Orchestrator’s dynamic task delegation helps maximize the efficiency of both visual interactions and direct system manipulation.\n\nState-of-the-Art Performance: CoAct-1 sets new benchmarks for both OSWorld (60.8% success rate) and WindowsAgentArena (52.5% success rate), outperforming existing methods like Agent S2.5 and GTA-1 by significant margins. The system excels in tasks requiring complex file management, data processing, and cross-application workflows.\n\nFlexibility: By incorporating both GUI and code-based actions, CoAct-1 can handle a broad range of tasks with varying complexities, from simple GUI interactions to complex backend system operations. This flexibility is especially beneficial for applications requiring precise or multi-round task execution.", "weaknesses": "Complexity: The reliance on three distinct agents (Orchestrator, Programmer, and GUI Operator) introduces significant system complexity. While the multi-agent framework allows for high flexibility, it also makes the system harder to manage and debug, especially in real-world applications where the agents may not always coordinate perfectly.\n\nAdditionally, I believe that a more in-depth discussion about the differences between this work and other hybrid frameworks would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HiX0zo7xPx", "reviewer": "Reviewer_x4pG", "summary": "The paper introduces of CoAct-1, a multi-agent system that combines both a programmer agent and GUI agent. The system contains the following module:\n\nAn Orchestrator, which serves as a high-level planner to orchestrate tasks.\nA GUI Operator, a standard VLM agent that performs visual actions like moving mouse, and using keyboard.\nA Programmer, a specialized agent that can write and execute Python or Bash scripts to perform backend operations like file management or data processing.\n\nThe main idea is that the Orchestrator can dynamically delegate a subtask to the most efficient agent, leveraging the advantage of both GUI agent and programming agent.\nThe authors evaluate CoAct-1 on the OSWorld and WindowsAgentArena benchmarks. They show the state of the art performance and a improved efficiency.\n\n1. CoAct-1 achieves a new SOTA success rate of 60.8% on OSWorld and 52.5% on WindowsAgentArena, significantly outperforming prior GUI-only methods.\n\n2. The hybrid approach reduces the average number of steps required for a successful task on OSWorld to 10.15, compared to ~15 steps for leading GUI-only agents.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well written. The idea of a hybrid computer use agent is novel. The experiment is solid and covers different OS platforms. The overall presentation is good. The authors are also able to demonstrate the success rate increase and steps reduced from this CoAct-1 clearly over the current SoTA agents in the OSWorld benchmark and the WindowsAgentArena benchmark. The authors provide a detailed example of a user task and provide the detailed prompt of each module, which is useful for the reader to enhance understanding. This work meets the quality requirements for this conference.", "weaknesses": "Overall this is solid work and I don't have any specific questions or concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "98da1RxZpZ", "reviewer": "Reviewer_T8jB", "summary": "This paper presents CoAct-1, a multi-agent framework to tackle long-horizon computer-use tasks. By combining an orchestrator, a GUI agent, and a programming agent, this work achieves state-of-the-art results on OSWorld and WindowsAgentArena.", "questions": "1. CoAct-1 doesn't perform particularly well with a very limited step budget (15 steps), while this work has expressed that the use of a coding agent saves steps. How would you explain this contradiction?\n\n2. In Figure 1, why does the Programmer agent return a screenshot as a summary?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. Good engineering results and practical value. This work achieved SOTA (at the time of ICLR submission) on two OS use benchmarks, covering both Linux and Windows environments.\n\n2. Clear methodology and good writing. Ablation studies are well-designed and insightful. Full prompts, model versions, and detailed implementation notes are provided, demonstrating a high level of transparency.", "weaknesses": "1. Limited research contribution. The paper reads more like a well-written technical report. The approach primarily combines existing and widely-used components (coding, visual, a multi-agent orchestrator) without proposing fundamental methodological advances. It has limited differentiation from general tool-use agents.\n\n2. The OSWorld task distribution is skewed towards Office tasks, specifically, LibreOffice Calc and LibreOffice Writer, where programming agents could greatly outperform pure visual agents. It is unclear if this approach would be particularly beneficial to other daily tasks, and if so, why.\n\n3. Lack of experiments on different models. This work primarily uses OpenAI CUA 40 as the GUI operator and has only explored o3 and o4-mini as the orchestrator and programmer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhwcFLeXom", "reviewer": "Reviewer_KLTJ", "summary": "This paper presents CoAct-1, a multi-agent system that combines GUI operations with programmatic execution for autonomous computer control tasks. The system comprises three specialized agents: an Orchestrator (for task decomposition and delegation), a Programmer (for writing and executing Python/Bash scripts), and a GUI Operator (for vision-based interface interactions). On the OSWorld and WindowsAgentArena benchmarks, CoAct-1 achieves success rates of 60.8% and 52.5% respectively, significantly outperforming existing methods while reducing average steps from 15 to 10.15.", "questions": "- When the Programmer's code produces errors or unexpected results, how does the system recover? Is there a rollback mechanism? Can the Orchestrator detect Programmer failures and replan?\n\n- What are the API costs？\n\n- Can CoAct-1 generalize to web automation (e.g., Selenium) or mobile application control? What modifications are needed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an important and practical problem regarding the brittleness and inefficiency of pure GUI-based agents on complex, long-horizon tasks. Introducing coding as an augmented action space for computer-using agents is an innovative and promising research direction that offers a new paradigm for general computer automation.\n\n- The multi-agent architecture is well-designed, with the Orchestrator dynamically delegating subtasks to either the Programmer or GUI Operator, effectively combining efficient programmatic execution for backend operations (file management, data processing) with GUI interaction for frontend tasks. The hierarchical memory design (isolated working memory vs. long-term planning memory) effectively prevents context pollution.\n\n- Comprehensive evaluation on two challenging real-world computer operation benchmarks demonstrates significant performance improvements (4.78% over the best baseline on OSWorld, 22.7% on WindowsAgentArena). The efficiency analysis is thorough, clearly showing the advantages of code actions over GUI action sequences, particularly in Calc, VS Code, and multi-application tasks.", "weaknesses": "- Imprecise Action Space Definition: The paper defines the action space as A = A_GUI ∪ A_Code, but these two action types have fundamentally different granularities and abstraction levels. A single Python script may be equivalent to hundreds of GUI actions; this simple set union representation obscures the actual complexity and asymmetry.\n\n- Evaluation is limited to two benchmarks, both on Linux/Windows desktop environments. Validation on web applications, mobile platforms, or other operating systems is missing. As to the baseline,  add comparison with Cradle[1] for evaluation.\n\n- Comparisons with baselines are not entirely fair: CoAct-1 uses the latest o3 model while some baselines use 7B open-source models. Comparisons with equivalent-scale models should be provided.\n\n[1] Tan, Weihao, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou et al. \"Cradle: Empowering foundation agents towards general computer control.\" arXiv preprint arXiv:2403.03186 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)