Agent S: An Open Agentic Framework that Uses Computers Like a Human
Authors: Saaket Agashe
Jiuzhou Han
Shuyu Gan
Jiachen Yang
Ang Li
Xin Eric Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Vision and Language Model
Agents
Retrieval Augmented Generation
GUI
Large Language Models
Agent Computer Interface
Primary area: applications to robotics, autonomy, planning

Abstract: We present Agent S, an open agentic framework that enables autonomous interaction with computers through Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. Agent S addresses three key challenges in automating computer tasks: acquiring domain-specific knowledge, planning over long task horizons, and handling dynamic, non-uniform interfaces. To this end, Agent S introduces experience-augmented hierarchical planning, which learns from external knowledge search and internal experience retrieval at multiple levels, facilitating efficient task planning and subtask execution. 
In addition, it employs an Agent-Computer Interface (ACI) to better elicit the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the baseline by 9.37\% on success rate (an 83.6\% relative improvement) and achieves a new state-of-the-art. Comprehensive analysis highlights the effectiveness of individual components and provides insights for future improvements. Furthermore, Agent S demonstrates broad generalizability to different operating systems on a newly-released WindowsAgentArena benchmark. Code available at https://github.com/simular-ai/Agent-S.

Published as a conference paper at ICLR 2025
AGENT S: AN OPEN AGENTIC FRAMEWORK THAT USES COMPUTERS LIKE A HUMAN
Saaket Agashe; Jiuzhou Han; Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang Simular Research
Can you help me calculate the total sales, average monthly sales, and generate visualizations?
Human
Online Web Search
Select the calls containing all the required data the on the Slandard tooiber
Agent-Computer Interface
Narrative Memory
agent lype(" Sype(==UU((Sheet (eMM(Shee11A222200) A22:A20)")
did similar tasks in doing calculation LibreOffice Calc. Use the formuls SUM to
The total sales have been calculated
Abslnclive Task Experience Retieval
Knowledge
Sublnak Essrutio
agent,drag drag_and drop(ele id1, ele _e_id) cells
selected ...
Episodic Memory
- Hierarchical Planning
and the cell.... agent.type()
Subtask
the sales Subtask Calculate the average sales Subtask Create chart to visualize
Subtack Exparience Retrieval
Planning
agent.drag_sen sar___ddddoggg drop0
Agent S
Figure 1: Agent uses a computer like a human to solve diverse desktop tasks on different systems.
ABSTRACT
We present Agent S, an open agentic framework that enables autonomous in- teraction with computers through Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. Agent S aims to address three key challenges in automating computer tasks: acquiring domain-specific knowledge, planning over long task horizons, and handling dynamic, non-uniform interfaces. To this end, Agent introduces experience-augmente hierarchical planning, which learns from external knowl- edge search and internal experience retrieval at multiple levels, facilitating ef- ficient task planning and subtask execution. In addition, employs an Agent- Computer Interface (ACI) better elicit the reasoning and control capabilities of GUT agents based on Multimodal Large Language Models (MLLMs). Eval- uation on the OSWorld benchmark shows that Agent outperforms the base- line by 9.37% on success rate (an 83.6% relative improvement) and achieves a new state-of-the-art dividual components and provides insights for future improvements. Further- more, Agent S demonstrates broad generalizability to different operating sys- tems on newly-released WindowsAgentArren benchmark. Code available at /attps://gitnne com http//imullass ai/Agent-S
Comprehensive analysis highlights the effectiveness of in-
INTRODUCTION
"The digital revolution is far more significant than the invention of writing even of printing."
-Douglas Engelbart, The Inventor of Computer Mouse
Since its invention, the mouse has been controlled by humans for interacting with computers. But does it really have to be? Autonomous Graphical User Interface (GUT) agents offer the promise of
"quull contributions.

---

Published as a conference paper at ICLR 2025
solving very specific and highly varied user queries suriees--uuhaa such as data entry, scheduling, and document creation for individual users, and streamlining operations in commercial settings- in the most gen- eral way: through direct UI interaction using the mouse and keyboard. Moreover, by eliminating the need for constant manual interaction, these agents not only boost efficiency but also improve accessibility, empowering individuals with disabilities to interact with technology in new, trans- formative ways. Recent advancements in Multimodal Large Language Models (MLLMs), such as GPT-4o (OpenAl 2023) and Claude (Anthropic) 2024), have laid the foundation for the develop- ment of GUI agents for human-centred interactive systems like desktop OS (Xie et al. 2024 Bonatti 2024).
et al.
However, automating computer tasks presents significant challenges. First. the vast range of constanty-eeolvving applications and websites requires the agent to possess specialized and up-to- date domain knowledge and the ability to learn from open-world experience. Second, complex desktop tasks often involve long-horizon, multi-step planning with interdependent actions that must be executed in specific sequence. The agent must, therefore, create clear plan with interme- diate subgoals and track task progress. Third, GUI agents must navigate dynamic, non-uniform interfaces, processing large volumes of visual and textual information while operating within vast action space. This involves distinguishing between relevant and irrelevant elements, accurately in- terpreting graphical cues, and responding to visual feedback during task execution.
In this paper, we present Agent S, a new agentic framework that tackles these challenges towards the goal of using computers like human. First, to enhance the GUI agent's capabilities in solving diverse, long-horizon desktop tasks with specific domain knowledge, we propose an Experience- Augmented Hierarchical Planning method. This approach leverages Online Web Knowledge and past experiences stored in Narrative Memory to decompose the complex, long-horizon task into a structured plan of manageable subtasks (see Figure T.. Online Web Knowledge provides up-to-date external knowledge about specific applications, allowing the agent to adapt to frequently chang- ing software and websites. Narrative Memory contains high-level, abstractive task experiences from past interactions, equipping the agent with contextual understanding for effective task planning. The agent monitors task completion progress, and during each subtask execution, retrieves detailed, step-by-step subtask experience from Episodic Memory to dynamically refine its actions and contin- uously improve its planning ability. Successful subtasks and the full task experience are evaluated, summarized, and stored in episodic and narrative memory to enable continual improvement.
Furthermore, we introduce a specific language- centrie Agent-Computer Interface (ACI) (Lieberman & Selker 2003) as an abstraction layer to improve grounding. safety, and ef- ficiency for MLLM-based GUI agents. The ACI defines an interaction paradigm by (1) dual-input strategy using visual input for understanding environmental changes together with an image-augmented accessibility tree for precise element grounding; (2) a bounded ac- tion space of language-based primitives (e.g..
60%
OSWorld
Agent
40%
80%
20%
13.5%
12.3%
14.3%
10%
63%
click (element.id) ) that are conducive to MLLM common-senss reasoning and generate environment transitions at the right temporal resolution for the agent to observe immediate and task-relevant environment feedback.
Operating System
Office
Dail
Professional
Worddow
Figure 2: Agent S vs.
OSWorld Agent results across five broad computer task categories.
Our approach shows a remarkable improvement in the overall performance of Agent S the OSWorld benchmark (OpenAI] (023) (from 11.21% to 20.58%, with a relative improvement of 83.6%), establishing the new state-of-the-art results. The detailed comparison is shown in Fig- ure 2 which demonstrates consistent improvements by Agent S across five broad computer task categories over the OSWorld agent. We also evaluate our Agent S on a concurrent work- WindowsAgentArena (Bonatti et al. 2024), where we observe a performance improvement from 13.3% to 18.2% on an equivalent setup without any explicit adaptation. The improvement demonstrates the broad generalizability of Agent S to different operating systems. We detail the component-wist improvements introduced by the proposed strategies through ablation studies and
2

---

Published as a conference paper at ICLR 2025
present comprehensive error analysis of our Agent S framework. In summary, contributions are four-fold:
We introduce Agent S, agentic framework that integrates experience-uugmente hicrar- chical planning, sclf-supervised continual memory update, and an Agent-Complln Interface for MLLL-based GUI agents perform complex computer tasks.
We propose an experience-augmented hierarchical planning method that uses experience from external web knowledge and the agent's 's internal memory to decompose complex tasks into exe- cutable subtasks.
We extend the concept of an ACI to GUI agents, allowing MLLM-bare based agents to operate com- puters more precisely using a set of high-level, predefined primitive actions.
We conduct experiments on different operating systems in the OSWorld and WindowsAgentAie benchmarks, where the results show that Agent S achieves new state-of oftt--of-hh-arrr performance.
2 RELATED WORK
MLLM Agents. The advent of Multimodal Large Language Models (MLLMs) has led to a host of works that utilize them as a reasoning backhone in Agentic Systems (Sumers et al. 2024). These
Agents augment LLMs with Memory, Structured Planning (Wang et al., 2023 [Shinn et al.) [023] Weng et al. 2023). Tool Use (Schick et al.] 2023 Shen et al. 2023 Patil et al. (023) and the ability to Act in external environments Park et al. (2023). These agents have shown promise in domains ranging from embodied simulators (Liang et al. 2023 Song et al. 2023) to video games (Wu et al.)
2023 Wang et al. (2024) and scientific research (Bran et al. 2023). For Software Engineering (Hong Qian et al. 2024) in particular, Yang et al. (2024) proposed an Agent-Compute Interface (Lieberman & Selker 2003) for MILM agents understand and act more efficiently and reliably.
et al. 2024
Our work extends and integrates these individual modules into a new MILM agent framework for computer control.
GUI Agents. MLLM agents have been applied to execute natural language instructions in both web and OS environments. Early research concentrated on web navigation tasks, utilizing MLLMS to teract with web interfaces (Gur et al. 2024 He et al. [0224] Kim et al. 2023 Shaw et al.) 2023 Putta
ct at.]] Recently, the focus has shifted to OS-level environments, leading to the development of (2024) and AgenttwsAAgetAeeeeeene Bonatti
benchmarks and frameworks such as WSWlrl Xie et al.
2024 and AndroidWorld (Rawles et a al. (2024) for desktop control, and DiGIRL (Bai et al.
2024 for mobile environments. These OS-level tasks offer broader control capabilities beyond the limitations of single-browser contexts in web navigation. Methodologically, earlier GUI agents employed behavioral cloning with reinforcement learning (Humphreys et al.) (022). in-context tra- jectory examples (Zheng et al. (2024b), state-dependent offline experience (Fu et al., 2024b), and reusable skill generation (Wang et al., 2024). Contemporaneous work on GUI agents for video games and OS (Wu et al. 2024 Song et al. 2024 Tan et al.] 2024) propose varying instances of cognitive architectures (Sumers al. 2024). Our work contributes unique modules such as experience-augmentne hicrarchical planning and ACI for GUI control, integrated with novel con- tinual memory update framework.
Retrieval-Augmented Generation (RAG) for AI Agents. RAG [an et al. 2024) improves the reliability of MLLM inference by augmenting the input with reliable and up-to-date external knowl- edge. Similarly, MLLM agents benefit from retrieving task exemplars (Kim et al. 2024), state-aware guidelines (Fu et al. 2024a), and past experiences (Kagaya et al.) (024). Our use of experience for augmentation differs in three ways: 1) our hierarchical planning leverages both full task experience and subtask experience; 2) the full task experience summarized into an abstractive textual reward for subtask planning; 3) the subtask experience is assessed and annotated by self-evaluator before being stored in memory.
3 AGENTS S
The GUI-based Operating System control tasks be formalized Partially Observable Markoy Decision Process (POMDP). defined as M = OS, TRR, where the OS state space, O is the observation space (natural language instructions, screenshots, accessibility trees,
3

---

Published as a conference paper at ICLR 2025
Manager
Knowlediee
Narrative Experience
Narrative Memory
Workars Wo..
Episodic
Worker
Experience
Evaluator
Context Fusion
Agont
Trajectory Reflector
Episodic Memory
Subtask Planner
Action
Sounded Action Space
Wis
ID-Grounding
Sp- SSp
Action Generator
OCit
User Teak
Augmentaation
Desktop Environment
Observation
Figure 3: Overview the Agent S framework. Given task T, and initial environment observation 0o. the Manager conducts experience-augmennee hierarchical planning using web knowledge and narrative memory to produce subtasks 80 82-
For each 8,, Worker w, draws from episodic memory to generate action a, time t. which is executed by the ACI to return the next immediate observation 0++1. self-evaluation module closes the loop by storing the summarized subtask and full-task trajectories in narrative and episodic memory.
the state transition function, and R S A + [0, 1] is the reward function. Agent S, illustrated in Figure B is novel framework that integrates three main strategies in a closed loop to tackle such complex GUI-based operating system control tasks: experience-augmentee hierarchical planning, continual update of narrative and episodic memory, and an Experience-augmeen hierarchical planning allows Agent S to break down complex tasks into man- ageable subtasks. This enables both high-level planning and low-level execution to draw from ex- ternal web-based experience and internal task-specific experience. A continual process of storing and retrieving self-evaluated task experience in narrative and episodic memory enables Agent S to improve over time and adapt to changes the open- desktop environment. The ACI ensures grounding by providing vision- asiinnauggmented accessibility tree observation containing all valid GUT elements and constraining the agent's 's chosen action to a bounded discrete space valid actions. Below, we describe each component and its integration in detail.
etc.), .A the action space (clicks, keys, etc). T S A
Interface for precise perception and action on GUIs.
3.1
HIERARCHICA PLANNING
In hierarchical planning Manager decomposes high-level task T into sequence of subtasks {Si, S2,..., S,). ager assigns these subtasks Workers, which execute them by performing low-level actions. Each Worker operates in a localized decision mabssion-making loop, selecting actions at each timestep based on its observation. The hierarchical structure allows the Manager focus on planning while Workers handle execution.
where each subtask S; is more granular and feasible for execution. The Man-
3.1.1 MANAGER: FUSING EXTERNAL KNOWLEDGE AND INTERNAL EXPERIENCE FOR PLANNING
The Manager is the primary plan generator module our system. It receives a task T, from the user and the initial environment observation Og (Annotated Accessibility Tree Screenshot) from the ACI as input. The manager formulates an observation-awwr query Q based on the user instruction and its observation in "How to do X" format. This query used for two types of

---

Published as a conference paper at ICLR 2025
retrieval. First, the query is used for Online Web Search through Perplexica Search Engine to get external knowledge. Then the same query is used to retrieve a similar task experience summary from the Manager's own Narrative Memory M,. The retrieval is based on the similarity of the query embedding.
The Narrative Memory includes summaries of both successful and failed trajectories with specific actions removed as abstractive full task experience En... The success/failure is evaluated by the Self-Evaluator S module (described in Subsection (.1.3) without any human feedback or ground truth information. This two-step retrieval provides the Manager with both the general and specific domain knowledge required to plan for the task. The outputs of the retrieval process are fused into a single fused guideline using the Experience Context Fusion submodule, represented formally as:
Q LL, Oo), En_ Retrieve(Mns Q), Krused = LLM (Mx (Q), Kweb)
Keeb = Retrieve Q)
The fused knowledge Krused is then used by Subtask Planner submodule of the Manager for- mulate a detailed, topologically sorted queue of subtasks (s9...8n) that can accomplish the user instruction. The manager also generates associated context Cs. for each subtask 8, which includes additional information useful to accomplish the subtask.
3.1.2 WORKER; LEARNING FROM SUBTASK EXPERIENCE AND TRAJECTORY REFLECTION
The subtasks (s).....) generated by the Manager G are executed sequentially by Worker modules (wg..wn)) Each Worker can take multiple time steps within one episode to complete a subtask s;- Firstly, the combination of the User Task Tu. the subtask and the contextual information C.. are used as a query to retrieve similar subtask experience Es, from the Worker's Episodic Mem- ory. The Episodic Memory is indexed by the concatenation of the task query, the subtask, and the contextual information (Q, st, C..)). based on the similarity of the embedding. As opposed to Narrative Memory, Episodic Memory includes a complete plan with specific grounding actions and only summaries from the subtask trajectories designated as DONE or successful by a Worker. Ad- ditionally, a Trajectory Reflector submodule T R, is associated with each worker. This submodule observes the entire episode as the worker is executing the subtask and provides reflective advice to the ageetthhelpin it think of alternative strategies and avoid repetitive actions.
F.s: Retrieve (Tu, 8i, Cs,))
The subtask experience E,, and the reflection is used by the Action Generator submodule inside a Worker to generate a single structured response- consisting of a previous action status check, ob- servation analysis, semantic next action and grounded next action. This structured response allows the agent to generate a templated chain-of thihuggh Wei et al. y Yao et al. 20233 for improved reasoning and results in single grounded action aj. This action passed to the ACI which imple- ments it in the Desktop Environment. Once the worker reasons that the subtask has been completed, generates a special grounded action DONE which signals the successful end of the subtask. The worker can also optionally generate FAIL signal, in which case the hierarchical operation is reset and the Manager replans new set of subtasks based on the intermediate environment configuration.
3.3.3 SELF-EVALUATOR SUMMARIZING EXPERIENCES AS TEXTUAL REWARDS
The Sell-Evaluator responsible for generating experience summaries as textual rewards r for the Manager and Worker modules. In the case of the successful end of an episode signaled by the Worker with a DONE signal, the evaluator observes the complete episode and generates learning in the form of summarization of the strategy used by the worker to complete that subtask. This strategy is fed back into the Worker's episodic memory Mc. In the case of the end of the complete usev-provided task, indicated either by the successful completion of all subtasks by the maximum number of steps limit, the evaluator generates a learning signal in the form of the summary of the entire task completion process. This summary is fed back and saved in the narrative memory M,, of the Manager. This process of Observations, Hierarchical Action Generation, and Rewards in the form of textual summaries update the internal memories of the Manager and Worker mirrors classic Hierarchical Reinforcement Learning process - but uses Retrieval as a learning strategy.
https://ggttuu..comm/
5

---

Published as a conference paper at ICLR 2025
Web Knowledge
Web Knowledge
Agent
A
Task Generator
Inference Task
Exploration Tasks
Summerizztion
Retrieval
Summerization
Narrative & Episodic Mammory
Memmory
(a) superviuer Exploration
(b) Continual Memory Update
Figure 4: The pipeline of memory construction and update, which contains two phases: Self- supervised Exploration and Continual Memory Update. The initial Narrative & Episodic Memory constructed through some randomly curated tasks during the exploration phase, and then is up- dated based on the inference tasks continually.
3.2 MEMORY CONSTRUCTION AND UPDATE
Initial Memory Construction via Self-supervised Exploration. To bootstrap Narrative M,, and Episodic Memories Mc, Agent S conducts self-supervised exploration on a of synthetically gen- erated tasks (see Figure R). We utilize two methods to create two types of random exploration tasks: environmenntinddependee
tasks and environment-awarr tasks. For crviroamennt-nddependenn tasks, we leverage task generator to generate the top 50 most common tasks from the various applications used OSWorld (Xie et al.] 2024) and WindowsAgent For environment-aware tasks, take the initial environments of the tasks in OSWorld and Win- AewsAgentArena and prompt a Task Generator to generate a different task based on the environ- ment. Both types of tasks consist of the exploration tasks. Then only taking web knowledge Kweb and collect the full task (Narrative Experience F,,) and subtask experiences (Episodic Experience Eg) for the narrative and episodic memories. The key stored in narrative memory M,, is the query Q and for episodic memory Mc. the key is query Q concatenated with subtask information sQ,
(Bonatti et al., 2024).
Agent S these tasks by
8,, C.,)) Through this process, the initial memory constructed.
Continual Memory Update. As our Agent S interacts with new tasks, it continually updates the Narrative Memory M, and Episodic Memory Mc. illustrated in Figure Thus even after the initial exploration is completed, the agent continues to learn as it encounters and attempts newer, more novel tasks. This process enables our agent learn even during inference and retrieve the learned knowledge to tasks effectively.
3.3 AGENT-COMPUTER INTERFACE
Current desktop environments designed to accommodate two distinct user types: (1) human users, who can perceive and respond subtle visual changes in real-time, and (2) software pro- grams, which execute predefined tasks through scripts and Application Programming Interfaces (APIs). However, these interfaces are inadequate for MILMM agents tasked with GUI control and ma- nipulation the fundamental keyboard-mon they respond in slow, discrete time intervals, lack an internal coordinate system, and cannot effi- ciently process fine-grained feedback after each minor mouse movement keyboard input. Draw- ing inspiration from the ACI developed for Software Engineering agents (Yang et al., 2024), we propose the creation of novel ACI bridge the gap between the unique operational constraints of MILMM agents and the requirements of open-ended GUI-control tasks.
level. These agents operate on a different paradigm:
Perception and Grounding. Grounding GUIs is the task of identifying the target element e, based on the natural language description of the element é. Agents need to understand this nat- ural language directive and find the exact element corresponding to this description. Current MLLMs can effectively reason about certain elements and features in image, but they cannot directly ground and pinpoint specific elements in images as they lack an internal coordinate sys- tem. In GUI manipulation, agents need to constantly interact with fine UI elements, and previous works have shown that grounding a significant bottleneck in these agents (Xie et al. 2024 Zheng et al. 2024a). Desktop environments, however, provide easily parseable Accessibility Tree with

---

Published as a conference paper at ICLR 2025
coordinate information about almost every element in the UL Thus, ACI design incorporates dual-input strategy with different purposes for each input. The image imput is used by the agent to observe salient details about the environment-
popups, button states, checking if a previous action worked, and reasoning about the next step. The accessibility tree input is used for reason- ing about the next step and, more importantly, grounding specific elements in the environment. To achieve this, we tag each element in the accessibility tree with unique integer tags which can be used by agents when referring these elements. Furthermore. while previous works seek augment the image with information from the accessibility tree (Xie et al., 2024 Zheng et al. 2024a Bonatti al. 2024) using Set-of-Mart Prompting, we augment the tree with details from the image. To achieve this, we run an OCR module on the image and parse textual blocks from the screenshot. We then add these blocks to the accessibility tree as interactable UI elements they do not already exist in the tree. To check for existing elements, we perform an IOU (Intersection over Union) match with all elements in the tree.
Constrained Action Space with Concurrent Feedback. Desktop automation has traditionally relied on APIs and scripts, but adopting these as actions would imply an unbounded combinato- rial action space of arbitrary executable code. This is unsuitable for keyboard mooooer--ouse-leeelll GUI automation agents because it compromises safety and precision. Code blocks can contain multiple sequential actions, leaving the agent with neither control over nor feedback from individual steps. To ensure that actions generated by agents are safely and reliably relayed to the environment and produce clear and timely feedback, our ACI design incorporates a bounded action space. This space includes primitive actions like click, type, and hotkey (detailed in Appendix [..]]. Agents can refer to different elements by their tagged IDs, and the ACI translates the ( (primitive- ID information into executable Python code. Furthermore, the agent is allowed to perform only one discrete action at each time step, so it can observe immediate feedback from the environment. These actions are also coarse enough to account for the slow, stateless nature of MLLMs, e.g., the agent can directly move to and click an element instead of moving the mouse in small increments.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Benchmarks. We evaluate Agent S on OSWorld (Xie et al. 2024). a benchmark for testing the multimodal agents' capability of executing wide range of computer tasks in a real computer envi- ronment. This executable environment allows free-form keyboard and mouse control of real com- puter applications, including OS, Office (LibreOffice Calc, Impress, Writer), Daily (Chrome, VLC Player, Thunderbird), Professional (VS Code and GIMP), and Workflow (tasks involving multiple apps). In addition, we also evaluate the generalization of Agent on WindowsAgentArenm (Bonatti al. 2024), contemporaneous benchmark in the Windows operating system.
Settings & Baselines. Since the OSWorld benchmark contains 369 tasks on Ubuntu, for the backbone model of Agent S, we leverage GPT-4o and Claude-3-Sonnet, respectively. For Win- Arws we test all 154 tasks on GPT-4o. We use the PaddleOCR]]oollii toolkit as our OCR tool in augmenting accessibility trees for grounding. The embedding model for the retrieval we use is -it--emmbedding---ms-iii Agent S takes the accessibility tree and screenshot as inputs, so we also use the reported results in OSWorld (Xie et al.) (024) and WindowsAgentArenn (Bonatti et al. 2024) with same input setting as basclines. The OSWorld bascline MMAgent takes the accessibility tree and sereenshots as input for spatial grounding to generate the action with coordinates each step. The WindowsAgentArees
baseline NAVI (Bonatti et al., (024) utilizes an accessibility tree, OCR, and Proprictary models to process the screenshot and create Set-of-Marks as input. Its action space includes constrained set of primitives but allows multiple actions to be chained together.
4.2
MAIN RESULTS
OSWorld. Table I shows the performance comparison between Agent S and the baseline models,
evaluated across the whole OSWorld test set. We compare with MMAgent Xie et al. 2024). CRA- DLE Tan et al. (2044). Open Interpreter, and the FRIDAY Agen(Wu et al. (2024). For the GPT-4o
http:://giituuu..cmm//eet
7

---

Published as a conference paper at ICLR 2025
Table 1: Main results of Successful Rate (%) on the OSWorld full test set of all 369 test examples. All agents except CRADLE use Accessibility Tree + Screenshot as inputs. FRIDAY and OpenInter- preter were evaluated by third parties.
Agent
MLLM
OS
Office
Daily
Profess.
Workflow
Overall
MMAgent
Claude-3-opus
12.50
3,57
5.27
8.16
1.00
4.41
MMAgent
Gemini-Pro-le
12.50
3,58
7.83
8.16
1.52
5.10
CRADLE
GPT-4o
3,58
16.67
6.55
20.41
5,48
7.81
OpenInter*
GPT-4o
8.94
Friday*
GPT-4o
11.11
MMAgent
GPT-4o
41.67
6.16
12.33
14.29
7.46
11.21
MMAgent
GPT-4V
16.66
6.99
24.50
18.37
4.64
12.17
Agent S
Claude-3.5-somnet
41.66
13.83
30.46
32.65
9.54
20.48
Agent S
GPT-4o
45.83
13.00
27.06
36.73
10.53
20.58
Thunderbird Heservnd
(a) Open Account Settings: agent.click(44, "left")
(h) Remove the Account: agent.click(86. "leftt)
(c) Remave the Account: agent.click( (49, I, "left")
Figure 5: A successful example of the Thunderbird task: "Help remove the account 'anonym- x2024 @ outlook.com". For space concern, (a) (b) (c) demonstrate the screenshots, current subtasks, and grounding actions at steps and 6, respectively.
model, Agent S achieves an overall success rate of 20.58% nearly doubling the performance of the best corresponding MMAgent baseline (11.21%). Agent consistently outperforms the baselines in the "Daily" and "Professional" tasks, where it reaches 27.06% and 36.73% success rates, respec- tively, compared to the best baseline results of 24.50% and 18.37%. These tasks are commonly used in daily life involved with knowledge- intensive professional applications, which benefit more from the retrieval augmentation of Agent S. Both Blaude-3.5-Somit and GPT--4o outperform the baseline versions across the majority of tasks. The results demonstrate the enhanced capability of Agent S in handling diverse and complex tasks more effectively than the baseline approaches.
Qualitative Examples. In Figure 5 we illustrate an example of a task from the Thunderbird app from OSWorld: Help me to remove the account "anonym-x2024 outlook.ccmm" ". Agent S completes tasks by interacting with the desktop through combination of actions. More qualitative examples are demonstrated in Appendix D.I
4.3 ABLATION STUDY
To demonstrate the effectiveness of individual modules of Agent S, we stratified sampled a subset of 65 instances, test from the full test set for the ablation study. Considering the inference cost, we utilized GPT-4o as the L.M backbone for all ablation studies for both the baseline and Agent S.
Learning from experience enhances the domain knowledge of GUI agents. The Experiential learning process Agent S involves searching web knowledge, retrieving full task experience from narrative memory and retrieving subtask experience from episodic memory. To assess the impact of different components, we systematically remove each component and observe performance changes across different task categories. The results are shown in Table 2 Learning from universal experi- ence available as web knowledge allows Agent S to make informed plans across a wide range of tasks and has the most significant impact. The learning from Narrative and Episodic memories synergies effectively with web retrieval, and the results detail how their ablation affects the agent's ability to handle complex tasks, underscoring the value of experiential learning. These results demonstrate that each component plays a critical role in enhancing the agent's 's domain knowledge. Removing
Thhe test_small set provided by the OSWorld codebase is too small and imbalanced (only 39 examples in total and 2 the OS category) for practical evaluations. Thus, we sample larger and more balanced subset.
8

---

Published as a conference paper at ICLR 2025
Table 2: The ablation study of experience-auummette hierarchical planning in OSWorld test
The metric is Successful Rate (%)
Method
0S (6)
Office (17)
Daily (16)
Profess. (10)
Workflow (16)
Overall (65)
buseline (OSWorld Agent)
33.33
5.88
12.50
10.00
6.25
10.77
Agent S
50,00
11.76
37.50
40.00
12.50
26.15
W/o Web Knowledge
16.60
11.76
24.49
30.00
6,25
16,80
-/o Narrative Memory
33.33
11.76
36,99
20.00
12.50
21.41
w/o Episodic Memory
33.33
5.88
25.00
30.00
12.50
18.46
w/o All
33.33
5.88
18.75
20.00
6,25
13.85
Baseline
10.77%
Agent
26.15%
Baseline Retrieval
12.31%
Agent Memory Update
23.18%
Agent (ACI-only)
12.31%
Agent 5
Self-Fvalusie
18.46%
Agent (ACI Retrieval)
20.00%
Agentt
Agent S wr Exploration
25.38%
10
10
1'5
20
25
(%)
(16)
Figure 6: Ablation of ACI in World test
Figure 7: Ablation of the memory update mech- anism in OSWorld test
all three components (wlo All) degrades the performance significantly, revealing the importance of learning from experience in the design.
ACI elicits better reasoning abilities of LLMs and supports better agentic learning. Figure presents the results of the ablation study on the ACI module. Comparing the baseline with Agent S (ACI-only f highlights the enhanced reasoning abilities achieved by incorporating ACI. Addi- tionally, we examined the impact of ACI on agentic learning by integrating the Experiential learning process. For the baseline, adding Experiential learning slightly improved overall performance. How- ever, when added to Agent S (ACI-only). the performance improved significantly, demonstrating ACI's effectiveness in enhancing agentic learning.
Hierarchical Planning supports long-horizon workflows. The (ACI-only + Experiential Learn- ing) setup in Figure 6 shows Agent S performance without Hierarchical Planning, and the observed performance drop (26. 15% to 20.00%) compared to the full Agent underscores the importance of Hierarchical Planning in modeling long-horizon workflows. The effect of hierarchical formulation becomes pronounced in the presence of Experiential learning as the Manager can generate more detailed and accurate plans in the subtask planning stage.
Exploration, Continual Memory Update and Sell-Evaluator are indispensable for memory construction. Our agent collects experience in phases initially during the self-supervised ex- ploration phase and then continually as it interacts with new examples (See Figure 8). To assess the effectiveness of these two learning stages and further examine our Self-evaluator which stores experience as summaries instead of unfiltered trajectories we run the ablation shown in Figure 7 Removing exploration limits memory updates to the inference phase only. Removing the contin- ual memory update means we only use the memory obtained from the exploration phase without subsequent updates. Removing the self-evaluator involves replacing summarized experiences with the original full trajectories. The results shown in Figure 7 reveal that ablating both the continual memory update and supervsddd exploration phases results a performance drop, with the self- supervised exploration being much more impactful. The ablation of the Self-Evaluator further shows the benefits using summarized trajectories instead of full trajectory exemplars for planning.
4.4 ERROR ANALYSIS
We performed a thorough error analysis on tasks that Agent S failed within testsws of the OS- World. There are three types of errors that we observed: (1) Planning Error: A planning error occurs when the agent generates unsuitable plans for a task, including inaccuracies in the plan, misleading subtask information, misalignment of subtask sequence with task requirements. (2) Grounding Error:
grounding error arises when the agent fails to accurately interact with target elements
"This version of Agent excludes Hierarchical Planning to better study the effects of ACI in isolation.

---

Published as a conference paper at ICLR 2025
Table 3: The statistic of Error Rate (%) on testaud of OSWorld that Agent S failed to complete.
Error Metrie
os
Office
Daily
Profess.
Workflow
Overall
Planning Error
66.67
25,00
30.00
66.67
28.57
34.69
Grounding Error
0.00
75.00
50.00
66.67
35.71
53.06
Exccution Error
33.33
87.50
100.00
66.67
71.43
79.59
Subtask Failure
16.67
58.47
62.82
33.61
70.43
57.17
despite their visibility and the application of correct reasoning. This includes incorrect element se- lection or inaccurate coordinate selection due to the inherent limitations of our action space (e.g., selecting the center instead of more precise part of the element). (3) Execution Error: An execu- tion error emerges when the agent makes incorrect decisions or fails to adjust its behavior during task execution. This includes repetitive actions, diverging from subtask goals, delays in transitioning between subtasks or violating established protocols by combining multiple actions into one.
Statistic Results of the Errors. We analyzed Agent S's trajectory for each failed task, identifying error types based on the definitions provided. A single task may contain multiple errors. We also cal- culated the Subtask Failure Rate, which measures the average percentage of failed subtasks relative to total attempts, and the Error Rate, which reflects the proportion tasks exhibiting a specific error type. As shown in Table execution and grounding errors are the most common across various task categories. A case study of eггor occurrence can be found in Appendix D.2
4.5 GENERALIZATION TO DIFFERENT OPERATING SYSTEMS
We test the Agent S framework with no modification on WindowsAgentArena (Bonatti et al. 2024), Windows OS benchmark released contemporaneously with work. We compare Agent S with the similar configuration with GPT-4o as the MLLM backbone, Accessibility Tree + Image as the input, and parsing with OCR. As shown in Table 4 Agent S outperforms the Navi agent without any adaptation to the new Windows environment.
Table 4: Results of Successful Rate (%) on WindowsAgentArena using GPT-4o and Image Ac- cessibility Tree input on the full test set of all 154 test examples.
Method
Office
Web Browser Windows System Coding
Media & Video Windows Utils Overall
NAVI NoVatBonati al. 2024
0.0
20.0
29.2
9.1
25,3
0.0
13.3
Agent S
0.0
13.3
45.8
29.2
19.1
22.2
18.2
CONCLUSION
In this work, we present Agent S-A novel framework for developing fully Autonomous Graphical User Interface (GUI) agents that perform wide range of user queries by directly controlling the keyboard and mouse. Through the Agent S framework, we show the benefits of Learning from Expe- rience for Taik-tree GUT agents. We also discuss the concept of Agent Computer Interface for the GUI domain, arguing in favour of an abstraction layer that allows MI.LM agents to perceive and reason language level with rich and continuous feedback. By leveraging Experience-Auumentee Hierarchical Planning, Online Web Knowledge, and an Agent-Computer Interface (ACI), Agent demonstrates SOTA performance on the OSWorld benchmark and generalizability across different operating systems. We demonstrate the potential of MLLM agents to learn from external sources and through direct interaction with the environment, without any human or environmental feedback in the GUI agents domain, thus opening discourse zero-shot, agentic methods for GUI agents.
Future Work. A key metric that has been unaddressed existing work on MILM agents for computer control, including ours, is the number of agent steps and wall clock time required for task completion. While our work focuses on achieving significant improvement future work can consider shortest-path navigation formulation of GUI control and evaluate the Pareto- optimallty of various agents on the dimensions time and accuracy. In our work, we use the state-oftth ofetthe-rt GPT-4o and Claude-3.5 ideas of experiential learning and Agent Computer Interface for smaller, open-source source MLLMs which could be fine-tuned to bridge the gap.
task performance,
models. However, future work extend the
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Anthropic. The claude 3 model family: Opus, sonnet, haiku, Anthropic Blog, 2024. URL https:
7/api. www.ssemaattsss.eee semanticcccolaaz org./CorpussD::2222333333
Hao Bai, Yifci Zhou, Mert Cemri, Jiayi Pan, Alane Suhr. Sergey Levine, and Aviral Kumar. Di- girl: Training ---d device-control agents with autonomous reinforcement learning. CoRR, abs/2406. 1 1896, 2024. doi: 10.48550/ARXIV2240006 1896. URL https::///o org/10. 48550/arXiv 2406 11896
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal OS agents at scale. CoRR, abs/2409.0826 2024. doi: 10. BtLhhtps::// / /wii..i...//// org/10..//.. 48550/ arxiv 2409.08264
48550/ARXIV.2009.00 082644
08264
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools, 2023. URL
https://arxii: org/ abs /twp///////32 2304 05376
Wenqi Fan, Yujuan Ding, Liangbo Ning. Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey RAG meeting llms: Towards retrieval-augmented large language models. In Ricardo Baeza- Yates and Francesco Bonchi (eds.), Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pp. 6491-6501 ACM, 2024. doi: 10.114573335528.66777 27775278..7777
URL https://
doi 1.w.orrg11...1. 000.0..114333333
3671470
Yao Fu, Dong-Ki Ki Kim, Jackyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bac, and Honglak Lee. Autoguide: Automated generation and selection of state-aware guidelines for large language model agents. CoRR, abs 2024a. doi: 10448550/ARXI12240000885 URL httth: org. 485..455550 48550/ wttp:55//a.ivv.... 240....8770088888 08978
Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of state-aware guidelines for large language model agents. CoRR, hhbs///443.00975 2024b. doi: 10.48500AARXI.2203.08777 URL https:://doo /tt.///o..rrr11....5/000//1/0
2403.08978
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Bck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openrevies www.:eett/f...... forum? d= rumve!
d=
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 6864-6890, Association for Computational Linguis- tics, 2024. doi: 10. 11011//5/111244AA--LLOOO v1 v1/2024 www.allll.oon.nmm 371
URL htttt:://ooiii http://o.....11101.11111111
Sirui Hong, Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang. Ceyao Zhang. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative framework. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:
//openzeview.no
Peter Conway Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy P. Lillicrap. A data-driven approach for learning control computers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML, 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of

---

Published as a conference paper at ICLR 2025
Proceedings of Machine Learning Research, pp. 146669488 PMLR, 2022. URL https://
proceedings.c swr.gg.mme.. wwr.. pwwss/ves2ee.. / http://wwww..uumm...a.....aaa.aa
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Ki- nose, Koki Oguri, Felix Wick, and Yang You. RAP: retrieval- planning with contextual memory for multimodal LLM agents. CoRR, abs/2402.03610, 2024. doi: 10.48550/ARXIV22422.. 03610. URL https://ddet www.oo..oo..ooooo.oooooo 48550/ arwiv 2402. 03610
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URLhttp://papett nips. cctpap::ppppppppppee Sactaape__feeeeee wtt.:////pppee///////aaaaa/
html
Minsoo Kim, Victor S. Burszlyn, Eunyee Koh, Shunan Guo, and Seung-won Hwang. Rada: Retrieval-augmented web agent planning with 1lms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 13511-13525 Association for Com- putational Linguistics, 2024. doi: 10.86655W12222..NININGS
URL https:
7/doi. oww. 10. 18633 2224. inddi ngs acl. 802
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference
Robotics and Automation, ICRA 2023, London, UK, May 29 June URL https:
2, 2023, pp. 9493-9500. IEEE, 2023, doi: 10. (011011/R44888)
7/doi. org/ 10. 09/ T1/// 2023. 10160597
Henry Lieberman and Ted Selker. Agents for the user interface. Handbook of agent technology, pp- 2003. 2003.
OpenAL. GPT-4 technical report. CoRR, abw/2303.08877, 2023. doi: 10.45550/ARXXV22203..774 URL
2ttp://0...55////....33333 08774
Joon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. Sean Follmer, Jeff Han, Jürgen Steimle, and Nathalie Henry Riche (eds.), Proceedings of the 36th Anmual ACM Symposium on User Interface Software and Technology, UIST 2023, San Francisco, CA, USA, 29 October 2023- November 2023, pp. 2: 2:22. ACM, 2023, doi: 10.1145/358618. 3606763. URL /ttp:::::/ddooooo http://doo..ogg//////// /0000//S..././tpp///oooooo/////0
11411114441 / 366
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. CoRR, abs/2305. 15334, 2023. doi: 1044550/ARXIV.205.1522 URL /tttts:://oi....om
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent Q: advanced reasoning and learning for autonomous Al agents. CoRR, abs/2408.07 07199, 2024. doi: 10-8550/ARX1.22488001999 URL 18550/arXiv> 200.v..4008.01333
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang. Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Commu- nicative agents for software development. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 15174-15186 doi: ac1 ong 810
Association for Computational Linguistics, 2024,
URL
https://do1 www.doi.orgg//o 1w..g///0000 11w11165311122
Christopher Rawles, Sarah Clinckemailie,, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E. Bishop, Wei Li, Folawiyo Campbell-Ajaal Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy P. Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. CoRR, abs/2405. 10..005.455555 14573, 2024. doi: 40.48550/ ARXIV.2405.14577 htl.httss:/ h/tttps://dd.... .org/ / 48150/048555/5a/ 2
12

---

Published as a conference paper at ICLR 2025
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Ham- bro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves to use tools. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neu- ral Information Processing Systems 36: Annual Conference cessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: http://aapes..nnsss.coomm dB42425e 0.9222444477003335220555000-0000
Neural Information Pro-
htmm
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to UI actions: Learning to follow instructions via graphical user interfaces. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:::pttpps.....ppppppp http://ppprr...ps.cccpppppppppep httpp///.....p.......p...e/eaaaa
html
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- ginggpt: Solving Al tasks with chatgpt and its friends in hugging face. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: http:://apppss.e http::////paapesapss.ccooooaaaea 7Tc3:aa061222003ff10ttboooo.eee.
html
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:://apprrs..s...cccooo http://ers.ppps.cccpppep../. 134587700772223555866233100000.0
html
Chan Hee Song, Brian M. Sadler, Jiaman Wu, Wei-Lun Chao, Clayton Washington, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In IEEF/CVF International Conference Computer Vision ICCV 2023, Paris, France, October 1-6, 2023, pp. 2986-2997. IEEE, 2023. doi: 10. N0110///CVV510002222320
URL https:
7/doi. org 10. 1 1109/ ICCVI 2023. 00280
Zirui Song, Yaohang Li, Meng Fang, Zhenhao Chen, Zecheng Shi, Yuan Huang, and Ling Chen. Mmac-copilot: Multi-modal agent collaboration operating system copilot. CoRR, doi: 10445550/ARXIV.2444.8000 48550/arXiv.24m 240..
abs/2404. 2024.
httph:t/////oo /ttt:::://doooor/11... org/10.
Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architectures for language agents. Trans. Mach. Learn. Res., 2024, 2024. URL https:
//openreview. forum?? d= 62Cvfl
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia. Ziluo Ding, Boyu Li, Bohan Zhou, Jun- peng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yu- jie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering foundation agents towards general computer control, 2024. URL
attps::/arxxr org/ abs 2403. 03186
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language mod- els. Trans. Mach. Learn. Res., 2024, 2024.
net net/forum?idt
e
13

---

Published as a conference paper at ICLR 2025
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V. Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL (rrum?id=1PLINNII
net./
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 De- cember 9, 2022, 2022. URL URLhttp:: htpp:///pppppppppppppsss.. nips cC/ wwwppppppppppppeppep..ee.ee....a files/ www.ee/pppeeeeeeem pptpp/pppppr/////////// html
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics; EMNLP 2023, Singapore, December 6-10, 2023, pp. 2550-2575. Association for Computational doi: 10. (T15S////2223.FIIIIIII prg/10.1880 /
Linguisties, 2023.
EMNLP.167. URL https://doi.
Yue Wu, Shrimai Prabhumoye, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom M. Mitchell, and Yuanzhi Li. SPRING: GPT-4 out-performs RL algorithms by studying papers and reasoning. CoRR, abs/2305. 15486, 2023. doi: 10.48550ARRXI22305 15486. URL
https::/doii wwwp:/............ 1.0.0.4855000 48540 2305. 15486
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement.
CoRR, abs/2402.07456. 2024. doi: ID.4550/AAXXX.22222044666 htRpsttps: //ooi /rg/
10. 0188550/ar1110 24w.xii..442222 2402. 07456
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao. Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. CoRR, abs/2404.00997. 2024. doi: 10. http:::/dooooooooo 4.0.0.8850000 / ar 2404. 07972
4SS50/AARXIV22444.07992
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. CoRR, abs/2405.15793, 2024. doi: 10.455500ARXXI.2405..555
[tLttttts: org/
17..4850/aaxiv.2440 24777
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL htt.https:: http://ppeneeeiieeee /
net feseeffooumee forum? id=wE fo v1uYUl-X
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a. URL
nett/ooun?id-pieece
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-ss-eemppla
prompting with memory for computer control. In The Twelfth International Conference on Learn- ing Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL httpss:/oppntee
net wttp://iwwwweetonuueteemm
14

---

Published as a conference paper at ICLR 2025
A AGENT-COMPUTER INTERFACE
Lineeriedd Tine with IDs, Missing elements from DCR, current app info and Screenshat az OBSERVATION
Agent Computer
60%
Interface
Full accessibility Tree and Screenshot as inputs
0o
agent.click(177,15 "lett,
Action Space
Agent
ID-Grounding
pystuogui click(45e 342, button-"lett'..)
Bounded action space provided as callable functions with arguments allmwing immeddate feedback
OCR Augmentation
Desktop
Environment
agent.elick(ID, type,
agent.type(ID. text,
Figure 8: The gereenshot-based observations, and bounded action space for GUI interaction. It enables the agent to ground elements using unique integer tags and augment the accessibility tree with OCR-parsed elements from screenshots. Actions restricted to discrete primitives (e.g., click, type) to ensure precise and safe interactions with the desktop environment, providing immediate feedback after each step.
Interface (ACT) for Agent S integrates a linearized accessibility tree,
A.1 BOUNDED ACTION SPACE
To facilitate the agent's ss accurate and effective task execution, define constrained action space, which simplifies the action selection process, making easier for the agent to ground its decisions in well-structured of operations. As summarized in Table 5 each action type has certain parameters and detailed in description.
Table 5: Agent Action Space, Descriptions, and Arguments.
Action Details
Agent Action
Description
Arguments
click type scroll
Click on an element.
element id, num clicks, button type, hold keys text, element id, overwrite, enter
Type text into an element.
Scroll within element.
element id. clicks keys
betkey
Press hotkey combo.
hold.and.preas
Hold keys and press others.
hold keys, press keys drag from id, drop_on id, hold keys
drag.and.drop
Drag and drop between elements.
save.to.buffer
Save text to buffer for later use.
text
switch applications
Switch to another app.
app code
wait
Wait for some time.
time
done
Mark task as success.
None
fail
Mark task failure.
None
A.2 ABLATIONS ON AGENT COMPUTER INTERFACE
The incorporation of Retrieval-as-Learmin method enhances the performance of both the Baseline and Agent models, with notably greater impact observed for Agent as shown Table
15

---

Published as a conference paper at ICLR 2025
Table The detailed result of ACI ablation study on testaub OSWorld. The backbone model of baseline and Agent S is GPT-4o.
Success Rate (%)*
Method OS (6)
Office (17)
Daily (16)
Profess. (10)
Workflow (16)
Overall (65)
Baseline
33.33
5.88
12.50
10,00
6.25
10.77
Retrieval
00.00
00.00
25.00
30.00
6.25
12.31
Agent (ACI-only)
16.60
5.88
18.75
20.00
6.25
12.31
+ Retrieval
33.33
11.76
31.25
30.00
6.25
20.00
B DETAILED RESULTS ON OSWORLD AND WINDOWSARENA
Table 7: Detailed success rates of baseline and Agent S using GPT-4o on OSWorld, divided by apps (domains): OS. LibreOffice Cale, LibreOffice Impress, LibreOffice Writer, Chrome, VLC Player, Thunderbird, VS Code, GIMP and Workflow involving with multiple apps.
Success Rate (%)
Method
OS
Cale
Impress
Writer
VLC
TB
Chrome
VSC 30.43
GIMP 0.00
Workflow
Baseline
1.67
4.26
6.81
8.70
9.50
6.67
15.22
7.46
Agent S
45.84
2.13
15.34
30.42
30.06
40.00
21.74
52.17
23.08
10.53
Table 8:
Detailed success rates Agent using GPT-4o on WindowArena, divided by apps (do- mains): Chrome, Microsoft Edge, VS Code, Notepad, LibreOffice Calc, Settings, Windows Calc, Clock, VS Code, Microsoft Paint, File Explorer, LibreOffice Writer, VL.C Player.
Success Rate (%)
Method
Chrome
Msedge
VSC
Notepad
Lib_Cale
Settings
Win_Cale
Clock
Paint
Writer
VLC
Agent
17.65
29.17
0.00
80.00
0.00
0.00
19.05
C
EXPERIENCE-AUGMENED HIERARCHICAL PLANNING
Observation-Awart Query
The Manager formulates query Q based the user task T, and
initial observation Op:
LLM Oo)
Narrative Memory Storing Full Task Experiences The narrative memory is indexed using an obscrvation-awarc query Q formulated by the Manager. It is represented as:
M.(Q) Save(Mn, Q)
where M, represents the narrative memory, and Q is the query generated based on the user task and initial observation Oo-
Episodic Memory Storing Successful Subtask Experiences The episodic memory is used by
Workers to execute subtasks and is indexed using the full User Task T., subtask s;, and contextual information C.,;
Mc 8,, C..) Save(Me, (Ta, 85, Ca.))
Where M. represents the episodic memory.
16

---

Published as a conference paper at ICLR 2025
C.I MANAGER: FUSING EXTERNAL KNOWLEDGE AND INTERNAL EXPERIENCE FOR PLANNING
External Knowledge Retrieval the Perplexica search engine:
The query is used retrieve external knowledge Kest using
Kexi = ReReeiereeeearr Web, Q)
Fusion of Internal Experience and External Knowledge The internal narrative memory expe-
rience (Q) and external knowledge Kex are combined using the Experience Context Fusion module:
Kfused MLLM(MM.(M Kex)
Subtask Planning The fused knowledge Kuseed used by the Manager to generate a queue of subtasks (5o, .,, 8,) and associated contexts (Cso: Cay.... Cs..):
(150, Coods (61, Coid...., (8ns Con)) MLLM K (useed)
C.2 WORKER: LEARNING FROM SUBTASK EXPERIENCE AND TRAJECTORY REFLECTION
Subtask Execution Each Worker ui retrieves subtask experience by querying the episodic memory Mc:
F.s: Retrieve(Mes (Tu, Cst))
Trajectory Reflection
The Worker reflects on the entire episode using Trajectory Reflector TR;
Reflection Rt
This reflection helps the Worker refine its strategies.
Action Generation Using the retrieved subtask experience Ex,, the Worker generates a structured response for grounded action ay:
aj MLLM( observation, Reflection)
Subtask Completion
The Worker signals the end of a subtask either through DONE or FAIL:
DONE, if subtask completed successfully
status
FAIL, if subtask fails
C.3 SELF-EVALUATOR: GENERATING SUMMARIZED EXPERIENCES AS TEXTUAL REWARDS
Episodic Experience Update If a Worker completes subtask, the Self-Evaluator S generates an Episodic Experience Fe; as a summary of the strategy used:
R4 = S(Eiisodee))
This experience is saved back into the episodic memory, indexed by the task T., subtask s;, and contextual information C.,:
M. Save(Mo, (Tu, 8i, Css);
17

---

Published as a conference paper at ICLR 2025
Narrative Experience Update When the entire task is completed by the Manager G, the Self- Evaluator generates a task completion reward ry, which is saved into the narrative memory, indexed by the observation-aware query Q formulated by the Manager.
E.,, =S(G(T,)) Mn Save(Mn> Q, Fm..
C.4 ABLATIONS ON LEARNING
The results presented in Table demonstrate the critical role played by both the Continual Learning component and the Self-Evaluaton in enhancing the performance of Agent S.
Table 9: The detailed result of axperience-augmentee hierarchical planning ablation study on testaub of OSWorld. The backbone model of baseline and Agent S is GPT-4o.
Success Rate (%)
Method
OS (6)
Office (17)
Daily (16)
Profess. (10)
Workflow (16)
Overall
Agent
50.00
37.50
40.00
12.50
26.15
w/o Continual Memory Update
33.33
11.76
37.50
30.00
12.50
23.08
w/o Self-Evaluator
33.33
5,88
31,25
20.00
12.50
18.46
w/o Self-superviser Exploration
33.33
5.88
25.00
20.00
6.25
15.38
D SUPPLEMENTARY EXAMPLES FOR QUALITATIVE ANALYSIS
Here we present additional examples of successful and failed tasks as supplements to the qualitative analysis in $4.2 Furthermore, we provide more detailed error analysis complement $4.4
D.1
SUCCESS EXAMPLES
In this section, we present successful task examples from variety of domains.
Google
(a) Open Terminal: agent.elick(24, 1, "left")
Open Terminal:
agent.clickk (13, 1, "left")
Google
Google
Google
(c) Navigate Home Directory: agent. syent,typettext= ed/home/user', enter=True)
(d) Clone Repository:
(c) Clone the Repository: agent. agent.wait() wait(1)
agent. (tttttttpttttt git clone
http://gtthub..oo/laaae al/instructor-embeddae enter=True)
Figure 9: A successful task of Multi apps. The task instruction is: Please help me clone the repo "https://github. https:///iitubb.om//laggg wiwarn/aaiinnuuoo-eeetttee of the subtask and its corresponding grounding action.
to fhomeluser. Each caption contains the plan
18

---

Published as a conference paper at ICLR 2025
(a) Click on the Tools menu: agent.click(38, left)
(b) Click on the Options... item: agent.click(53, 1, left)
(c) click on Load/Save category: agent.click(207, 1, left)
(d) Double-click on Load/Save cat- egory: agent.click(207. left)
(e) Click on the General sub- option:
(1) Change the time to minutes: sgent.,ype((30., write=True)
"3". over-
agent.click(208, left)
Figure 10: An example of LibreOffice Impress. The task instruction is: Enable auto-save every 3min for me, so that don't need to hit Ctrl-S that much. Each caption contains the plan and its corresponding grounding action.
(a) Initiate Find and Replace: agent.hotkey(( ctrl',
(b) Enter the Words:
"otle 'h'])
agent.type("tent 181)
(c) Enter the Words: "gent.type""test" 200)
(d) Enter the Words: seennttype""tex""" 181, overwrite= True)
(c) Enter the Words: agent.click(200)
(I) Enter the Words: agent.type(" typst""est""" 200)
(g) Replace All Instances: agent.click(206, "left")
(h) Replace All Instances: agent.click(266, 1, "left")
Figure 11: A successful task of VSCode. The task instruction is: Please help me change all the places in this document that say "text" to "test". Each caption contains the plan of the subtask and its corresponding grounding action.
19

---

Published as a conference paper at ICLR 2025
(a) Select the First Two Paragraphs: agent.click(SS, 1, "left")
(b) Select First Two Paragraphs: agent.hold and
(c) Select the First Two Paragraphs: agent.click(SS, 1, "left")
anddssses(( press(('shitt) J. ())
(d) Select the First Two Paragraphs: agent.drag.am and aat.aaddddddp... drop(55, 57)
(e) Open Line Spacing Menu: agent.click(151, "left")
(I) Set Double Spacing: agent.click(179, 1,"left")
Figure 12: A successful task of LibreOffice Writer. The task instruction is: Make the line spacing of first two paragraph into double line spacing, Each caption contains the plan of the subtask and its corresponding grounding action.
(a) Open the Files application: agent.click(20, 1. "left")
(b) Navigate to the Trash: agent.click(23, 1, "left")
(c) Select the poster file: agent.click(139, 1, "left"
(d) Restore the poster file: agent.click(135, 1, "left")
(e) Restore the poster file: agent.click(138, 1, "left")
(I) Restore the poster file: agent.done(28, 1. "lee""""
Figure 13: A successful task OS. The task instruction is: am currently using an Ubuntu system, and I have wrongly deleted poster of party night. Could you help recover it from the Trash? Each caption contains the plan of the subtask and its corresponding grounding action.
Although the agent successfully completes the tasks depicted in Figure still issues present in its execution trajectories. For instance, during the task Figure the agent incorrectly enters the word into the wrong field at Figure (c), although this mistake corrected promptly. Furthermore, in the course of the task demonstrated in Figure [2 the agent exhibits inappropriate actions at Figure 12 (a)) Additionally, while performing the task depicted in Figure [3] the agent fails to recognize the completion of the task at Figure T3 (d), subsequently at- tempting to recover an already existing file on the desktop Figure (e) These issues highlight the inherent challenges in achieving consistently reliable behavior, even when tasks are nominally completed.
12
there
D.2 DETAILED ERROR ANALYSIS AND FAILURE EXAMPLES
In this section, we analyze the sources execution errors as defined in $4.4. followed by presenting several examples of failed tasks, each with detailed error analysis provided for the respective case. Empirically, Grounding and planning emrors often directly lead to execution errors (c.g., failing to interact with the correct target element can result in repetitive actions, and incorrect planning
20

---

Published as a conference paper at ICLR 2025
Grounding-indai induaeed
51%
Beft-induodd
Planning induoed
Figure 14: The error sources of the overall 39 execution errors.
messages can lead to wrong decisions while performing the task). We reviewed all 39 execution errors in errors on fest of OSWorld that Agent failed to complete, as shown Figure [4] and found that 46% were caused by planning grounding errors. This indicates that reducing these errors, particularly grounding errors, which frequently cause repetitive actions, could significantly improve performance.
During the task in Figure 15 the agent simultaneously makes planning, execution, and grounding errors. First, the inaccurate planning information in Figure [5] (a) suggests typing '1' instead of 'No. I' in the cell constitutes a planning error, leading the agent to type the incorrect value. Additionally, the agent's attempt drag the fill handle from 'B2' to 'B23' in Figure (b) fails due to the selection of erroneous elements and coordinates, which can be classified as a grounding enror Furthermore, the agent continues to try execute the subtask 'Drag the Fill Handle with repetitive actions in Figure L3 (f)) overlooking the prior grounding eπror and being unable to correct its behavior timely, which indicative of an execution error.
Another type of planning error emerges while the agent executing the task shown in Figure 16 The plan generated by the agent is flawed, as incorporates irrelevant subtask "Updating of Chrome", which does not pertain to the intended goal. Additionally, the resulting subtask sequence incorrect, as it erroneously prioritizes such subtask, illustrated in Figure [6 (c)(d. This fundamental planning deficiency propagates into an execution error, preventing the agent from successfully turning off the extension, demonstrated in the subsequent figures.
The failed task depieted in Figure 17 illustrates scenario where the agent makes a grounding error, which subsequently leads to an execution error. After adding the Alpha Channel, the agent attempts select the Fuuzzy Select Tool' from the toolbox target the background. However, instead of selecting the correct element (represented by the magic wand icon), the agent consistently grounds the incorrect element, 'Activity', located at the top-left corner. This misselection brings the system to its 'Overview' state. The agent then switches back to GIMP but continues to incorrectly select Activity', mistakenly identifying as the 'Fuzzy Select Tool'. This repeated incorrect action is demonstrated in Figure (g)) It is evident that the agent fails to correct its behavior promptly when facing this issue, which can considered an execution error stemming directly from the initial grounding error.
21

---

Published as a conference paper at ICLR 2025
select the Starting Dell: click
first call under "See No.")-
inter the starting suaber: Type in cell '92".
Handlel
betton right comerr of cell oz artil you see
(a) Planning Information: The in- formation marked in red is wrong.
(b) Drag the Fill Handle from 'B2' to 'B23': Handle from 'B2' to 'B23':
(c) Drag the Fill
agent.drag and drop(56, 87)
agent.drag.ant -angggaan...o drop((2,, 387)
(d) Drag the Fill Handle from 'B2' 'B23': Handle from 'B2' to 'B23': Handle from 'B2' 'B23': agent.drag.am drddppp 342)
(e) Drag the Fill
(I) Drag the Fill
agent.drag.and drr2ppp 387)
agent.drag.and. dnddopp(72 342)
Figure 15: An failed task of LibreOffice Calc. The task instruction is: Fill the Sequence Numbers as "No. #" in the "Seq No." column
(a) Open Chromium Settings: agent.click(43, 1, "left")
(b) Open Chromium Settings: agent.click(442.. 1, "left")
(c) Update Chromium: (oonttcllic(133) 1, "left")
(d) Update Chromium: agent, wait(5)
(c) Tum Off Extensions: (eenttclicc(125) 1, 'left")
(f) Turn Off Extensions: agent.elict click(68,))) 1, "left")
(g) Turn Off Extensions: agent.click(68, 1, "left")
(h) Turn Off Extensions: agent.click(62, 1, "left")
(i) Turn Off Extensions: agent.click(40, 1, "left")
Figure 16: An failed task of Chrome. The task instruction is: Can you help me clean up my computer by getting rid of all the tracking things that Amazon might have saved? I want to make sure my browsing is private and those sites don't 't remember me. Each caption contains the plan of the subtask and its corresponding grounding action.
22

---

Published as a conference paper at ICLR 2025
(a) Close Color Profile Dialog: agent.click(167, 1. "left")
(b) an Alpha Channel: agent.click(11, 1, "left")
(c) Add Alpha Channel: agent.click(181, 1, "left")
(d) Add Alpha Channel: agent.click(180, 1, "left")
(e) Select the Background: clent,,llic((O, "left")
(f) Select the Background: ngentsswichhappliiattnnn "ggmpp
(g) Select the Background: agent.click(1. 1, "left")
(h) Select the Background:
(i) Select the Background: agent.click(2. 1, "left""
agentsswich.aapliccctonns gimp")
Figure 17: An failed task of GIMP. The task instruction is: Could you make the background of this image transparent for me? Each caption contains the plan of the subtask and its corresponding grounding action.
23


Reviews:
[{"review_id": "q51Qr6zHIZ", "reviewer": "Reviewer_meL5", "summary": "The paper introduces Agent S, a groundbreaking framework designed to automate complex, multi-step tasks on computers through Graphical User Interface (GUI) interaction for human usage. Agent S tackles the challenges of acquiring domain-specific knowledge, planning over long task horizons, and navigating dynamic interfaces by employing experience-augmented hierarchical planning, which leverages both external web knowledge and internal experience retrieval. It also utilizes an Agent-Computer Interface (ACI) to enhance the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). The framework demonstrated significant improvements in task success rates on the OSWorld benchmark and showed broad generalizability to different operating systems, setting a new state-of-the-art in autonomous GUI agent performance.", "questions": "1. Could you elaborate on how Agent S differentiates between when to retrieve external web knowledge versus when to leverage internal memories? How is this decision balanced?\n2. How does Agent S handle the cold start problem, especially when it encounters a task that neither memories has prior experience with? Could you explain the strategies for quickly adapting to new tasks?\n3. Your error analysis indicates a high rate of execution errors. Could you present a more detailed breakdown of the types of execution errors encountered and discuss potential improvements to the Action Generator to reduce these errors, especially in complex, long-horizon tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Agent S stands out for its task automation through experience-augmented hierarchical planning. This method harnesses external web knowledge and draws upon internal memories, enabling the agent to decompose complex tasks into executable subtasks.\n2. The introduction of the Agent-Computer Interface (ACI) is a notable strength of Agent S. This interface serves as a critical abstraction layer that facilitates precise perception and action in GUI environments. By defining a bounded action space with language-based primitives and incorporating a dual-input strategy, ACI enhances the agent's ability to ground actions and receive immediate environmental feedback. This innovation allows Agent S to operate more effectively and efficiently, setting a new standard for MLLM-based GUI agents.", "weaknesses": "1. The paper does not address the scalability and efficiency of the framework when handling a large volume of tasks or more complex workflows. There is a need to evaluate how the agent performs under increased load and whether the hierarchical planning and memory update mechanisms can scale without compromising the speed and accuracy of task completion.\n2. The framework's performance could potentially falter in scenarios where reliable web knowledge is scarce or when there are frequent, rapid changes in application interfaces that outpace the web's ability to update corresponding information.\n3. The paper acknowledges a high rate of execution errors, indicating that Agent S may struggle with decision-making and behavior adjustment during task execution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNXRE2fAxs", "reviewer": "Reviewer_YYG1", "summary": "The paper proposes Agent S, a computer agent framework which can directly interact with computers through GUI interface. The proposed framework mainly consists of three components, a manager to manage external and internal memory for subtasks planning, a worker to complete subtasks with an episodic memory trajectory reflector, and a self-evaluator to summarize task experiences. The authors also propose an agent-computer interface as an abstraction layer. The proposed framework is evaluated on OSWorld and seen an increase of 9.37% success rate.", "questions": "1. What is the average time cost to complete a task using OpenAI API-based models and self-hosted models?\n2. Does the continued memory growth consume more LLM input context tokens? What is the average context length?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The performance of the proposed framework on OSWorld benchmark is quite good.\n2. The proposed framework is well-engineered and the evaluation is systematic.\n3. The presentation and visualization of the paper is good.", "weaknesses": "1. It would be unfair to compare the framework only to the baseline from OSWorld, which is a benchmark paper, not a methodology paper.\n2. The self-supervised exploration process is not realistic in actual deployments and I believe it will lead to overfitting.\n3. The ACI proposed in this paper can only act on selected elements in the accessibility tree, which somewhat sacrifices flexibility for performance because you cannot click on every coordinate of the screen.\n4. While the framework is well designed, it doesn’t introduce much new. For example, it’s quite obvious that doing some early exploration, using external resources, and OCR would help. In addition, these processes, as well as subtask planning and self-evaluation would significantly slow down the task, which is already quite slow, and would cost more OpenAI tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KbDFKbN1mG", "reviewer": "Reviewer_KnHm", "summary": "This paper presents Agent S, a novel framework for GUI-based operating system control that integrates three key strategies: experience-augmented hierarchical planning, continual memory update, and an Agent-Computer Interface (ACI). The framework introduces an effective memory mechanism with initialization and continuous update algorithms, demonstrating state-of-the-art performance on computer use benchmarks like OSWorld. A notable contribution is the carefully designed ACI that addresses the unique challenges of MLLM agents interacting with desktop environments.", "questions": "1. **Citation Format Issues**: References to OSWorld (Xie et al., 2024) and WindowsAgentArena (Bonatti et al., 2024) need clarification as they appear to be forward citations\n\n\n2. **Figure Organization**: Have the authors considered combining Figures 3 and 4 for better presentation, or focusing Figure 4 more specifically on memory aspects?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novel and Effective Memory Mechanism**: Introduces a well-designed memory system with both narrative and episodic components\nProvides clear algorithms for both initial memory construction and continuous updates\nDemonstrates a complete closed-loop system with practical effectiveness\n\n\n2. **Insightful Analysis of Agent-Computer Interaction**: Deep analysis of fundamental challenges in MLLM-based computer control\nIdentifies key issues like discrete time response, lack of internal coordinate systems, and inefficient feedback processing\nAddresses the limitations of traditional API/script-based automation approaches\n\n\n3. **Innovative ACI Design**: Proposes a dual-input strategy combining visual and accessibility tree information\nImplements bounded action space with concurrent feedback\nSuccessfully bridges the gap between MLLM agents and GUI control requirements\n\n\n4. **Strong Empirical Results**: Achieves SOTA performance on established benchmarks\nProvides comprehensive experimental validation", "weaknesses": "1. **Limited Problem Definition**: The paper could benefit from a more detailed introduction to computer automation tasks\nKey concepts like planning, execution, and grounding could be better explained for readers new to the field\n\n\n2. **Presentation Issues**: Some overlap between Figures 3 and 4 that could be consolidated or better differentiated\nTechnical details of the ACI implementation could be more thoroughly described", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmdRhBZTMx", "reviewer": "Reviewer_7E6Z", "summary": "This paper introduces a new multimodal large language model agent for GUI control, called Agent S. Its main feature is the ability to complete various tasks in a GUI interface through direct keyboard and mouse control. Comparing with other agents, Agent S incorporates an experience-augmented hierarchical planning method, which enhances the agent's ability to decompose tasks based on trajectories stored in memory. Experiments demonstrate that this framework is versatile, capable of executing various GUI-oriented tasks across different systems(Ubuntu and Windows).", "questions": "First, I have some doubts about the generalizability of accessibility trees. Do all GUIs have accessibility trees, and do systems like Ubuntu and Windows have similar forms of accessibility trees? I hope the article can clarify these points. Secondly, I would like the article to compare Agent S with different agents, evaluating whether Agent S demonstrates better experimental results compared to existing GUI agents. Lastly, the ablation study does not adequately cover all points of contribution. An experiment should clarify the performance of Agent S when solely relying on image input without ACI.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The advantages of this paper are that it has a clear line of thought, smooth transitions, and is easy to follow. The main architecture diagram of the agent is concise and clear, allowing for a straightforward understanding of the information flow. The selection of experimental environments is well-considered, taking into account both Ubuntu and Windows systems, which proves the effectiveness and generalization of the framework.", "weaknesses": "However, the experiments in this paper are not sufficiently comprehensive. First, the baselines used are not enough, and in the comparison results of the main experiments, MLLMs are used simply instead of MLLM agents. The experimental results based on single MLLMs that only input images and accessibility trees are not convincing enough. Existing agents such as Cradle and Claude 3 perform well using only keyboard and mouse inputs without requiring additional accessibility trees. As a result, the third contribution of this paper also limits the applicability of Agent S and raises doubts about whether all GUIs provide accessibility tree inputs and whether such inputs are necessary. Thus, the contributions of this paper may seem insufficiently innovative. \n\nreference：\n[1]Tan W, Zhang W, Xu X, et al. Cradle: Empowering Foundation Agents towards General Computer Control[C]//NeurIPS 2024 Workshop on Open-World Agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)