Agent S: An Open Agentic Framework that Uses Computers Like a Human

## Abstract

Abstract We present Agent S, an open agentic framework that enables autonomous interaction with computers through a Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. Agent S aims to address three key challenges in automating computer tasks: acquiring domain-specific knowledge, planning over long task horizons, and handling dynamic, non-uniform interfaces.
To this end, Agent S introduces experience-augmented hierarchical planning, which learns from external knowledge search and internal experience retrieval at multiple levels, facilitating efficient task planning and subtask execution.
In addition, it employs an Agent-Computer Interface (ACI) to better elicit the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs).
Evaluation on the OSWorld benchmark shows that Agent S outperforms the baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves a new state-of-the-art.
Comprehensive analysis highlights the effectiveness of individual components and provides insights for future improvements.
Furthermore, Agent S demonstrates broad generalizability to different operating systems on a newly-released WindowsAgentArena benchmark. Code available at https://github.com/simular-ai/Agent-S .

## 1 Introduction

“The digital revolution is far more significant than the invention of writing or even of printing.”

— Douglas Engelbart, The Inventor of Computer Mouse

Since its invention, the mouse has been controlled by humans for interacting with computers. But does it really have to be?
Autonomous Graphical User Interface (GUI) agents offer the promise of solving very specific and highly varied user queries—such as data entry, scheduling, and document creation for individual users, and streamlining operations in commercial settings—in the most general way: through direct UI interaction using the mouse and keyboard.
Moreover, by eliminating the need for constant manual interaction, these agents not only boost efficiency but also improve accessibility, empowering individuals with disabilities to interact with technology in new, transformative ways.
Recent advancements in Multimodal Large Language Models (MLLMs), such as GPT-4o (OpenAI, ) and Claude (Anthropic, ), have laid the foundation for the development of GUI agents for human-centred interactive systems like desktop OS (Xie et al., ; Bonatti et al., ).

However, automating computer tasks presents significant challenges.
First, the vast range of constantly-evolving applications and websites requires the agent to possess specialized and up-to-date domain knowledge and the ability to learn from open-world experience.
Second, complex desktop tasks often involve long-horizon, multi-step planning with interdependent actions that must be executed in a specific sequence. The agent must, therefore, create a clear plan with intermediate subgoals and track task progress.
Third, GUI agents must navigate dynamic, non-uniform interfaces, processing large volumes of visual and textual information while operating within a vast action space. This involves distinguishing between relevant and irrelevant elements, accurately interpreting graphical cues, and responding to visual feedback during task execution.

In this paper, we present Agent S, a new agentic framework that tackles these challenges towards the goal of using computers like a human.
First, to enhance the GUI agent’s capabilities in solving diverse, long-horizon desktop tasks with specific domain knowledge, we propose an *Experience-Augmented Hierarchical Planning* method. This approach leverages Online Web Knowledge and past experiences stored in Narrative Memory to decompose the complex, long-horizon task into a structured plan of manageable subtasks (see Figure 1).
Online Web Knowledge provides up-to-date external knowledge about specific applications, allowing the agent to adapt to frequently changing software and websites.
Narrative Memory contains high-level, abstractive task experiences from past interactions, equipping the agent with contextual understanding for effective task planning.
The agent monitors task completion progress, and during each subtask execution, it retrieves detailed, step-by-step subtask experience from Episodic Memory to dynamically refine its actions and continuously improve its planning ability.
Successful subtasks and the full task experience are evaluated, summarized, and stored in episodic and narrative memory to enable continual improvement.

Figure: Figure 2: Agent S vs. OSWorld Agent results across five broad computer task categories.
Refer to caption: x2.png

Furthermore, we introduce a specific language-centric *Agent-Computer Interface (ACI)* (Lieberman & Selker, ) as an abstraction layer to improve grounding, safety, and efficiency for MLLM-based GUI agents. The ACI defines an interaction paradigm by (1) *a dual-input strategy* using visual input for understanding environmental changes together with an image-augmented accessibility tree for precise element grounding; (2) *a bounded action space* of language-based primitives (e.g., click(element_id)) that are conducive to MLLM common-sense reasoning and generate environment transitions at the right temporal resolution for the agent to observe immediate and task-relevant environment feedback.

Our approach shows a remarkable improvement in the overall performance of Agent S on the OSWorld benchmark (OpenAI, ) (from 11.21% to 20.58%, with a relative improvement of 83.6%), establishing the new state-of-the-art results. The detailed comparison is shown in Figure 2, which demonstrates consistent improvements by Agent S across five broad computer task categories over the OSWorld agent. We also evaluate our Agent S on a concurrent work—WindowsAgentArena (Bonatti et al., ), where we observe a performance improvement from 13.3% to 18.2% on an equivalent setup without any explicit adaptation. The improvement demonstrates the broad generalizability of Agent S to different operating systems.
We detail the component-wise improvements introduced by the proposed strategies through ablation studies and present a comprehensive error analysis of our Agent S framework.
In summary, our contributions are four-fold:

- •
We introduce Agent S, a new agentic framework that integrates experience-augmented hierarchical planning, self-supervised continual memory update, and an Agent-Computer Interface for MLLM-based GUI agents to perform complex computer tasks.
- •
We propose an experience-augmented hierarchical planning method that uses experience from external web knowledge and the agent’s internal memory to decompose complex tasks into executable subtasks.
- •
We extend the concept of an ACI to GUI agents, allowing MLLM-based agents to operate computers more precisely using a set of high-level, predefined primitive actions.
- •
We conduct extensive experiments on OSWorld to show the effectiveness of individual components of Agent S, establishing new state-of-the-art on automating computer tasks. Besides, we demonstrate its generalizability across different operating systems on WindowsAgentArena.

## 2 Related Work

MLLM Agents.  The advent of Multimodal Large Language Models (MLLMs) has led to a host of works that utilize them as a reasoning backbone in Agentic Systems (Sumers et al., ). These Agents augment LLMs with Memory, Structured Planning (Wang et al., ; Shinn et al., ; Weng et al., ), Tool Use (Schick et al., ; Shen et al., ; Patil et al., ) and the ability to Act in external environments Park et al. (). These agents have shown promise in domains ranging from embodied simulators (Liang et al., ; Song et al., ) to video games (Wu et al., ; Wang et al., ) and scientific research (Bran et al., ).
For Software Engineering (Hong et al., ; Qian et al., ) in particular, Yang et al. () proposed an Agent-Computer Interface (Lieberman & Selker, ) for MLLM agents to understand and act more efficiently and reliably.
Our work extends and integrates these individual modules into a new MLLM agent framework for computer control.

GUI Agents.
MLLM agents have been applied to execute natural language instructions in both web and OS environments. Early research concentrated on web navigation tasks, utilizing MLLMs to interact with web interfaces (Gur et al., ; He et al., ; Kim et al., ; Shaw et al., ; Putta et al., ). Recently, the focus has shifted to OS-level environments, leading to the development of benchmarks and frameworks such as OSWorld Xie et al. () and WindowsAgentArena Bonatti et al. () for desktop control, and DiGIRL (Bai et al., ) and AndroidWorld (Rawles et al., ) for mobile environments. These OS-level tasks offer broader control capabilities beyond the limitations of single-browser contexts in web navigation.
Methodologically, earlier GUI agents employed behavioral cloning with reinforcement learning (Humphreys et al., ), in-context trajectory examples (Zheng et al., ), state-dependent offline experience (Fu et al., ), and reusable skill generation (Wang et al., ). Contemporaneous work on GUI agents for video games and OS (Wu et al., ; Song et al., ; Tan et al., ) propose varying instances of cognitive architectures (Sumers et al., ). Our work contributes unique modules such as experience-augmented hierarchical planning and ACI for GUI control, integrated with a novel continual memory update framework.

Retrieval-Augmented Generation (RAG) for AI Agents.
RAG (Fan et al., ) improves the reliability of MLLM inference by augmenting the input with reliable and up-to-date external knowledge. Similarly, MLLM agents benefit from retrieving task exemplars (Kim et al., ), state-aware guidelines (Fu et al., ), and past experiences (Kagaya et al., ). Our use of experience for augmentation differs in three ways: 1) our hierarchical planning leverages both full task experience and subtask experience; 2) the full task experience is summarized into an abstractive textual reward for subtask planning; 3) the subtask experience is assessed and annotated by a self-evaluator before being stored in memory.

Figure: Figure 3: Overview of the Agent S framework. Given task $T_{u}$ and initial environment observation $o_{0}$, the Manager conducts experience-augmented hierarchical planning using web knowledge and narrative memory to produce subtasks $s_{0},\dotsc,s_{n}$. For each $s_{i}$, Worker $w_{i}$ draws from episodic memory to generate an action $a_{t}$ at time $t$, which is executed by the ACI to return the next immediate observation $o_{t+1}$. A self-evaluation module closes the loop by storing the summarized subtask and full-task trajectories in narrative and episodic memory.
Refer to caption: x3.png

## 3 Agent S

Agent S, illustrated in Figure 3, is a novel framework that integrates three main strategies in a closed loop to tackle complex GUI-based operating system control tasks: experience-augmented hierarchical planning, continual update of narrative and episodic memory, and an Agent-Computer Interface for precise perception and action on GUIs. Experience-augmented hierarchical planning allows Agent S to break down complex tasks into manageable subtasks. This enables both high-level planning and low-level execution to draw from external web-based experience and internal task-specific experience. A continual process of storing and retrieving self-evaluated task experience in narrative and episodic memory enables Agent S to improve over time and adapt to changes in the open-world desktop environment. The ACI ensures grounding by providing a vision-augmented accessibility tree observation containing all valid GUI elements and constraining the agent’s chosen action to a bounded discrete space of valid actions. Below, we describe each component and its integration in detail.

### 3.1 Experience-augmented Hierarchical Planning

#### 3.1.1 Manager: Fusing External Knowledge and Internal Experience for Planning

The Manager $G$ is the primary plan generator module in our system. It receives a task $T_{u}$ from the user and the initial environment observation $O_{0}$ (Annotated Accessibility Tree + Screenshot) from the ACI as input. The manager formulates an observation-aware query $Q$ based on the user instruction and its observation in a “How to do X” format. This query is used for two types of retrieval. First, the query is used for Online Web Search through Perplexica Search Engine(^1^11[https://github.com/ItzCrazyKns/Perplexica](https://github.com/ItzCrazyKns/Perplexica)) to get external knowledge. Then the same query is used to retrieve a similar task experience summary from the Manager’s own Narrative Memory $M_{n}$. The retrieval is based on the similarity of the query embedding.

The Narrative Memory includes summaries of both successful and failed trajectories with specific actions removed as *abstractive full task experience* $E_{n_{u}}$.
The success/failure is evaluated by the Self-Evaluator $S$ module (described in Subsection 3.1.3) without any human feedback or ground truth information.
This two-step retrieval provides the Manager with both the general and specific domain knowledge required to plan for the task. The outputs of the retrieval process are fused into a single fused guideline using the Experience Context Fusion submodule, represented formally as:

$$ $\displaystyle Q$ $\displaystyle=\text{LLM}(T_{u},O_{0})$ $\displaystyle K_{\text{web}}$ $\displaystyle=\text{Retrieve}(\text{Web},Q)$ $\displaystyle E_{n_{u}}$ $\displaystyle=\text{Retrieve}(M_{n},Q)$ $\displaystyle K_{\text{fused}}$ $\displaystyle=\text{LLM}(M_{n}(Q),K_{\text{web}})$ $$

The fused knowledge $K_{\text{fused}}$ is then used by Subtask Planner submodule of the Manager to formulate a detailed, topologically sorted queue of subtasks $\langle s_{0}...s_{n}\rangle$ that can accomplish the user instruction. The manager also generates associated context $C_{s_{i}}$ for each subtask $s_{i}$ which includes additional information useful to accomplish the subtask.

#### 3.1.2 Worker: Learning from Subtask Experience and Trajectory Reflection

The subtasks $\langle s_{0}..s_{n}\rangle$ generated by the Manager $G$ are executed sequentially by Worker modules $\langle w_{0}..w_{n}\rangle$. Each Worker can take multiple time steps within one episode to complete a subtask $s_{i}$. Firstly, the combination of the User Task $T_{u}$, the subtask $s_{i}$ and the contextual information $C_{s_{i}}$ are used as a query to retrieve similar subtask experience $E_{s_{i}}$ from the Worker’s Episodic Memory.
The Episodic Memory is indexed by the concatenation of the task query, the subtask, and the contextual information $\langle Q,s_{i},C_{s_{i}}\rangle$, based on the similarity of the embedding. As opposed to Narrative Memory, Episodic Memory includes a complete plan with specific grounding actions and only summaries from the subtask trajectories designated as DONE or successful by a Worker.
Additionally, a Trajectory Reflector submodule $TR_{i}$ is associated with each worker. This submodule observes the entire episode as the worker is executing the subtask and provides reflective advice to the agent—helping it think of alternative strategies and avoid repetitive actions.

$$ $\displaystyle E_{s_{i}}$ $\displaystyle=\text{Retrieve}(M_{e},\langle T_{u},s_{i},C_{s_{i}}\rangle)$ $$

The subtask experience $E_{s_{i}}$ and the reflection is used by the Action Generator submodule inside a Worker to generate a single structured response - consisting of a previous action status check, observation analysis, semantic next action and grounded next action. This structured response allows the agent to generate a templated chain-of-thought Wei et al. (); Yao et al. () for improved reasoning and results in a single grounded action $a_{j}$. This action is passed to the ACI which implements it in the Desktop Environment. Once the worker reasons that the subtask has been completed, it generates a special grounded action DONE which signals the successful end of the subtask. The worker can also optionally generate a FAIL signal, in which case the hierarchical operation is reset and the Manager replans a new set of subtasks based on the intermediate environment configuration.

#### 3.1.3 Self-Evaluator: Summarizing Experiences as Textual Rewards

The Self-Evaluator $S$ is responsible for generating experience summaries as textual rewards $r$ for the Manager and Worker modules. In the case of the successful end of an episode signaled by the Worker with a DONE signal, the evaluator observes the complete episode and generates learning in the form of a summarization of the strategy used by the worker to complete that subtask. This strategy is fed back into the Worker’s episodic memory $M_{e}$. In the case of the end of the complete user-provided task, indicated either by the successful completion of all subtasks or by the maximum number of steps limit, the evaluator generates a learning signal in the form of the summary of the entire task completion process. This summary is fed back and saved in the narrative memory $M_{n}$ of the Manager. This process of Observations, Hierarchical Action Generation, and Rewards in the form of textual summaries to update the internal memories of the Manager and Worker mirrors a classic Hierarchical Reinforcement Learning process - but uses Retrieval as a learning strategy.

Figure: Figure 4: The pipeline of memory construction and update, which contains two phases: Self-supervised Exploration and Continual Memory Update. The initial Narrative & Episodic Memory is constructed through some randomly curated tasks during the exploration phase, and then it is updated based on the inference tasks continually.
Refer to caption: x4.png

### 3.2 Memory Construction and Update

##### Initial Memory Construction via Self-supervised Exploration.

To bootstrap Narrative $M_{n}$ and Episodic Memories $M_{e}$, Agent S conducts self-supervised exploration on a set of synthetically generated tasks (see Figure 4). We utilize two methods to create two types of random exploration tasks: environment-independent tasks and environment-aware tasks. For environment-independent tasks, we leverage a task generator to generate the top 50 most common tasks from the various applications used in OSWorld (Xie et al., ) and WindowsAgentArena (Bonatti et al., ). For environment-aware tasks, we take the initial environments of the tasks in OSWorld and WindowsAgentArena and prompt a Task Generator to generate a different task based on the environment. Both types of tasks consist of the exploration tasks. Then we run Agent S on these tasks by only taking web knowledge $K_{\text{web}}$ and collect the full task (Narrative Experience $E_{n}$) and subtask experiences (Episodic Experience $E_{e}$) for the narrative and episodic memories. The key stored in narrative memory $M_{n}$ is the query $Q$ and for episodic memory $M_{e}$, the key is query $Q$ concatenated with subtask information $\langle Q,s_{i},C_{s_{i}}\rangle$. Through this process, the initial memory is constructed.

Continual Memory Update.
As our Agent S interacts with new tasks, it continually updates the Narrative Memory $M_{n}$ and Episodic Memory $M_{e}$, as illustrated in Figure 4. Thus even after the initial exploration is completed, the agent continues to learn as it encounters and attempts newer, more novel tasks. This process enables our agent to learn even during inference and retrieve the learned knowledge to new tasks effectively.

### 3.3 Agent-Computer Interface

Current desktop environments are designed to accommodate two distinct user types: (1) *human users*, who can perceive and respond to subtle visual changes in real-time, and (2) *software programs*, which execute predefined tasks through scripts and Application Programming Interfaces (APIs).
However, these interfaces are inadequate for MLLM agents tasked with GUI control and manipulation at the fundamental keyboard-mouse level. These agents operate on a different paradigm: they respond in slow, discrete time intervals, lack an internal coordinate system, and cannot efficiently process fine-grained feedback after each minor mouse movement or keyboard input. Drawing inspiration from the ACI developed for Software Engineering agents (Yang et al., ), we propose the creation of a novel ACI to bridge the gap between the unique operational constraints of MLLM agents and the requirements of open-ended GUI-control tasks.

Perception and Grounding.
Current MLLMs can effectively reason about certain elements and features in an image, but they cannot directly ground and pinpoint specific elements in images as they lack an internal coordinate system.
In GUI manipulation, agents need to constantly interact with fine UI elements, and previous works have shown that grounding is a significant bottleneck in these agents (Xie et al., ; Zheng et al., ).
Desktop environments, however, provide an easily parseable Accessibility Tree with coordinate information about almost every element in the UI. Thus, our ACI design incorporates a dual-input strategy with different purposes for each input. The image input is used by the agent to observe salient details about the environment—such as popups, button states, checking if a previous action worked, and reasoning about the next step. The accessibility tree input is used for reasoning about the next step and, more importantly, grounding specific elements in the environment.
To achieve this, we tag each element in the accessibility tree with unique integer tags which can be used by agents when referring to these elements.
Furthermore, while previous works seek to augment the image with information from the accessibility tree (Xie et al., ; Zheng et al., ; Bonatti et al., ) using Set-of-Mark Prompting, we augment the tree with details from the image. To achieve this, we run an OCR module on the image and parse textual blocks from the screenshot. We then add these blocks to the accessibility tree as interactable UI elements if they do not already exist in the tree. To check for existing elements, we perform an IOU (Intersection over Union) match with all elements in the tree.

Constrained Action Space with Concurrent Feedback.
Desktop automation has traditionally relied on APIs and scripts, but adopting these as actions would imply an unbounded combinatorial action space of arbitrary executable code. This is unsuitable for keyboard-mouse-level GUI automation agents because it compromises safety and precision. Code blocks can contain multiple sequential actions, leaving the agent with neither control over nor feedback from individual steps.
To ensure that actions generated by agents are safely and reliably relayed to the environment and produce clear and timely feedback, our ACI design incorporates a bounded action space. This space includes primitive actions like click, type, and hotkey (detailed in Section A.1). Agents can refer to different elements by their tagged IDs, and the ACI translates the $\langle$ primitive - ID $\rangle$ information into executable Python code.
Furthermore, the agent is allowed to perform only one discrete action at each time step, so it can observe immediate feedback from the environment. These actions are also coarse enough to account for the slow, stateless nature of MLLMs, e.g., the agent can directly move to and click an element instead of moving the mouse in small increments.

## 4 Experiments

### 4.1 Experimental Setup

Benchmarks.
We evaluate Agent S on OSWorld (Xie et al., ), a benchmark for testing the multimodal agents’ capability of executing a wide range of computer tasks in a real computer environment. This executable environment allows free-form keyboard and mouse control of real computer applications, including OS, Office (LibreOffice Calc, Impress, Writer), Daily (Chrome, VLC Player, Thunderbird), Professional (VS Code and GIMP), and Workflow (tasks involving multiple apps). In addition, we also evaluate the generalization of Agent S on WindowsAgentArena (Bonatti et al., ), a contemporaneous benchmark in the Windows operating system.

Settings & Baselines.
Since the OSWorld benchmark contains 369 tasks on Ubuntu, for the backbone model of Agent S, we leverage GPT-4o and Claude-3-Sonnet, respectively. For WindowsAgentArena, we test all 154 tasks on GPT-4o. We use the PaddleOCR(^2^22[https://github.com/PaddlePaddle/PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)) toolkit as our OCR tool in augmenting accessibility trees for grounding. The embedding model for the retrieval we use is text-embedding-3-small. Agent S takes the accessibility tree and screenshot as inputs, so we also use the reported results in OSWorld (Xie et al., ) and WindowsAgentArena (Bonatti et al., ) with same input setting as baselines. The OSWorld baseline takes the coordinates-based accessibility tree and screenshots as input for spatial grounding to generate the action with coordinates at each step. The WindowsAgentArena baseline NAVI (Bonatti et al., ) utilizes an accessibility tree, OCR, and Proprietary models to process the screenshot and create Set-of-Marks as input. Its action space includes a constrained set of primitives but allows multiple actions to be chained together.

### 4.2 Main Results

OSWorld.  Table 1 shows the performance comparison between Agent S and the baseline models, evaluated across the whole OSWorld test set. For the GPT-4o model, Agent S achieves an overall success rate of 20.58%, nearly doubling the performance of the best corresponding baseline (GPT-4o with 11.21%). Agent S consistently outperforms the baselines in the “Daily” and “Professional” tasks, where it reaches 27.06% and 36.73% success rates, respectively, compared to the best baseline results of 12.33% and 14.29%. These tasks are commonly used in daily life or involved with knowledge-intensive professional applications, which benefit more from the retrieval augmentation of Agent S. Both Claude-3.5-Sonnet and GPT-4o outperform the baseline versions across the majority of tasks. Claude-3.5-Sonnet even performs better than GPT-4o in “Daily” and “Professional” tasks. The results demonstrate the enhanced capability of Agent S in handling diverse and complex tasks more effectively than the baseline approaches.

**Table 1: Main results of Successful Rate (%) on the OSWorld full test set of all 369 test examples.**
| Method | OS | Office | Daily | Profess. | Workflow | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Claude-3 | 12.50 | 3.57 | 5.27 | 8.16 | 1.00 | 4.41 |
| Gemini-Pro-1.5 | 12.50 | 3.58 | 7.83 | 8.16 | 1.52 | 5.10 |
| GPT-4V | 16.66 | 6.99 | 24.50 | 18.37 | 4.64 | 12.17 |
| GPT-4o | 41.67 | 6.16 | 12.33 | 14.29 | 7.46 | 11.21 |
| Agent S w/ Claude-3.5 | 41.66 | 13.83 | 30.46 | 32.65 | 9.54 | 20.48 |
| Agent S w/ GPT-4o | 45.83 | 13.00 | 27.06 | 36.73 | 10.53 | 20.58 |

##### Qualitative Examples.

In Figure 5, we illustrate an example of a task from the Thunderbird app from OSWorld: Help me to remove the account ”anonym-x2024@outlook.com”. Agent S completes tasks by interacting with the desktop through a combination of actions. More qualitative examples are demonstrated in Appendix D.1.

Figure: (a) Open Account Settings: agent.click(41, 1, “left”)
Refer to caption: extracted/5914178/figures/appendix/success/Thunderbird/step_0.png

### 4.3 Ablation Study

To demonstrate the effectiveness of individual modules of Agent S, we stratified sampled a subset of 65 instances, $test_{sub}$(^3^33The test_small set provided by the OSWorld codebase is too small and imbalanced (only 39 examples in total and 2 in the OS category) for practical evaluations. Thus, we sample a larger and more balanced subset.) from the full test set for the ablation study. Considering the inference cost, we utilized GPT-4o as the LLM backbone for all ablation studies for both the baseline and Agent S.

**Table 2: The ablation study of experience-augmented hierarchical planning in OSWorld $test_{sub}$. The metric is Successful Rate (%).**
| Method | OS (6) | Office (17) | Daily (16) | Profess. (10) | Workflow (16) | Overall (65) |
| --- | --- | --- | --- | --- | --- | --- |
| baseline (OSWorld Agent) | 33.33 | 5.88 | 12.50 | 10.00 | 6.25 | 10.77 |
| Agent S | 50.00 | 11.76 | 37.50 | 40.00 | 12.50 | 26.15 |
| - w/o Web Knowledge | 16.60 | 11.76 | 24.49 | 30.00 | 6.25 | 16.80 |
| - w/o Narrative Memory | 33.33 | 11.76 | 36.99 | 20.00 | 12.50 | 21.41 |
| - w/o Episodic Memory | 33.33 | 5.88 | 25.00 | 30.00 | 12.50 | 18.46 |
| - w/o All | 33.33 | 5.88 | 18.75 | 20.00 | 6.25 | 13.85 |

Learning from experience enhances the domain knowledge of GUI agents.
The Experiential learning process of Agent S involves searching web knowledge, retrieving full task experience from narrative memory and retrieving subtask experience from episodic memory. To assess the impact of different components, we systematically remove each component and observe performance changes across different task categories. The results are shown in Table 2. Learning from universal experience available as web knowledge allows Agent S to make informed plans across a wide range of tasks and has the most significant impact.
The learning from Narrative and Episodic memories synergies effectively with web retrieval, and the results detail how their ablation affects the agent’s ability to handle complex tasks, underscoring the value of experiential learning. These results demonstrate that each component plays a critical role in enhancing the agent’s domain knowledge. Removing all three components (w/o All) degrades the performance significantly, revealing the importance of *learning from experience* in the design.

Figure: Figure 6: Ablation of ACI in OSWorld $test_{sub}$.
Refer to caption: x5.png

ACI elicits better reasoning abilities of LLMs and supports better agentic learning. Figure 7 presents the results of the ablation study on the ACI module. Comparing the baseline with Agent S (ACI-only)(^4^44This version of Agent S excludes Hierarchical Planning to better study the effects of ACI in isolation.) highlights the enhanced reasoning abilities achieved by incorporating ACI. Additionally, we examined the impact of ACI on agentic learning by integrating the Experiential learning process. For the baseline, adding Experiential learning slightly improved overall performance. However, when added to Agent S (ACI-only), the performance improved significantly, demonstrating ACI’s effectiveness in enhancing agentic learning.

Hierarchical Planning supports long-horizon workflows.
The (*ACI-only + Experiential Learning*) setup in Figure 7 shows Agent S performance without Hierarchical Planning, and the observed performance drop (26.15% to 20.00%) compared to the full Agent S underscores the importance of Hierarchical Planning in modeling long-horizon workflows. The effect of hierarchical formulation becomes pronounced in the presence of Experiential learning as the Manager can generate more detailed and accurate plans in the subtask planning stage.

Exploration, Continual Memory Update and Self-Evaluator are indispensable for memory construction.
Our agent collects experience in two phases - initially during the self-supervised exploration phase and then continually as it interacts with new examples (See Figure 4). To assess the effectiveness of these two learning stages and further examine our Self-evaluator which stores experience as summaries instead of unfiltered trajectories we run the ablation shown in Figure 7. Removing exploration limits memory updates to the inference phase only. Removing the continual memory update means we only use the memory obtained from the exploration phase without subsequent updates. Removing the self-evaluator involves replacing summarized experiences with the original full trajectories. The results shown in Figure 7 reveal that ablating both the continual memory update and self-supervised exploration phases results in a performance drop, with the self-supervised exploration being much more impactful. The ablation of the Self-Evaluator further shows the benefits of using summarized trajectories instead of full trajectory exemplars for planning.

### 4.4 Error Analysis

We performed a thorough error analysis on the tasks that Agent S failed within $test_{sub}$ of the OSWorld. There are three types of errors that we observed: (1) *Planning Error*: A planning error occurs when the agent generates unsuitable plans for a task, including inaccuracies in the plan, misleading subtask information, or misalignment of subtask sequence with task requirements.
(2) *Grounding Error*: A grounding error arises when the agent fails to accurately interact with target elements despite their visibility and the application of correct reasoning. This includes incorrect element selection or inaccurate coordinate selection due to the inherent limitations of our action space (e.g., selecting the center instead of a more precise part of the element).
(3) *Execution Error*: An execution error emerges when the agent makes incorrect decisions or fails to adjust its behavior during task execution. This includes repetitive actions, diverging from subtask goals, delays in transitioning between subtasks or violating established protocols by combining multiple actions into one.

**Table 3: The statistic of Error Rate (%) on $test_{sub}$ of OSWorld that Agent S failed to complete.**
| Error Metric | OS | Office | Daily | Profess. | Workflow | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Planning Error | 66.67 | 25.00 | 30.00 | 66.67 | 28.57 | 34.69 |
| Grounding Error | 0.00 | 75.00 | 50.00 | 66.67 | 35.71 | 53.06 |
| Execution Error | 33.33 | 87.50 | 100.00 | 66.67 | 71.43 | 79.59 |
| Subtask Failure | 16.67 | 58.47 | 62.82 | 33.61 | 70.43 | 57.17 |

Statistic Results of the Errors. We analyzed Agent S’s trajectory for each failed task, identifying error types based on the definitions provided. A single task may contain multiple errors. We also calculated the Subtask Failure Rate, which measures the average percentage of failed subtasks relative to total attempts, and the Error Rate, which reflects the proportion of tasks exhibiting a specific error type. As shown in Table 3, execution and grounding errors are the most common across various task categories. A case study of error occurrence can be found in Appendix D.2.

### 4.5 Generalization to Different Operating Systems

We test the Agent S framework with no modification on WindowsAgentArena (Bonatti et al., ), a Windows OS benchmark released contemporaneously with our work.
We compare Agent S with the similar configuration(^5^55The best-performing agent in WindowsAgentArena is based on an internal closed-sourced model that was trained for GUI grounding and is not accessible outside of Microsoft now, so we choose a similar configuration with ours for fair comparison.) with GPT-4o as the MLLM backbone, Accessibility Tree + Image as the input, and parsing with OCR.
As shown in Table 4, Agent S outperforms the Navi agent without any adaptation to the new Windows environment.

**Table 4: Results of Successful Rate (%) on WindowsAgentArena using GPT-4o and Image + Accessibility Tree input on the full test set of all 154 test examples.**
| Method | Office | Web Browser | Windows System | Coding | Media & Video | Windows Utils | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| NAVI(Bonatti et al., ) | 0.0 | 20.0 | 29.2 | 9.1 | 25.3 | 0.0 | 13.3 |
| Agent S | 0.0 | 13.3 | 45.8 | 29.2 | 19.1 | 22.2 | 18.2 |

## 5 Conclusion

In this work, we present Agent S—A novel framework for developing fully Autonomous Graphical User Interface (GUI) agents that can perform a wide range of user queries by directly controlling the keyboard and mouse. Through the Agent S framework, we show the benefits of Learning from Experience for Task-oriented GUI agents. We also discuss the concept of an Agent Computer Interface for the GUI domain, arguing in favour of an abstraction layer that allows MLLM agents to perceive and reason at a language level with rich and continuous feedback. By leveraging Experience-Augmented Hierarchical Planning, Online Web Knowledge, and an Agent-Computer Interface (ACI), Agent S demonstrates SOTA performance on the OSWorld benchmark and generalizability across different operating systems. We demonstrate the potential of MLLM agents to learn from external sources and through direct interaction with the environment, without any human or environmental feedback in the GUI agents domain, thus opening a discourse on zero-shot, agentic methods for GUI agents.

Future Work.
A key metric that has been unaddressed in existing work on MLLM agents for computer control, including ours, is the number of agent steps and wall clock time required for task completion.
While our work focuses on achieving significant improvement in task performance, future work can consider a shortest-path navigation formulation of GUI control and evaluate the Pareto-optimality of various agents on the dimensions of time and accuracy. In our work, we use the state-of-the-art GPT-4o and Claude-3.5-sonnet models. However, future work can extend the ideas of experiential learning and Agent Computer Interface for smaller, open-source MLLMs which could be fine-tuned to bridge the gap.

## Appendix A Agent-Computer Interface

### A.1 Constrained action space

To facilitate the agent’s accurate and effective task execution, we define a constrained action space, which simplifies the action selection process, making it easier for the agent to ground its decisions in a well-structured set of operations. As summarized in Table 5, each action type has certain parameters and detailed in description.

**Table 5: Agent Action Space, Descriptions, and Arguments.**
| Agent Action | Action Details |  |
| --- | --- | --- |
| Description | Arguments |  |
| click | Click on an element. | element_id, num_clicks, button_type, hold_keys |
| type | Type text into an element. | text, element_id, overwrite, enter |
| scroll | Scroll within an element. | element_id, clicks |
| hotkey | Press a hotkey combo. | keys |
| hold_and_press | Hold keys and press others. | hold_keys, press_keys |
| drag_and_drop | Drag and drop between elements. | drag_from_id, drop_on_id, hold_keys |
| save_to_buffer | Save text to a buffer for later use. | text |
| switch_applications | Switch to another app. | app_code |
| wait | Wait for some time. | time |
| done | Mark task as success. | None |
| fail | Mark task as failure. | None |

### A.2 Ablations on Agent Computer Interface

The incorporation of Retrieval-as-Learning method enhances the performance of both the Baseline and Agent S models, with a notably greater impact observed for Agent S, as shown in Table 6.

**Table 6: The detailed result of ACI ablation study on $test_{sub}$ of OSWorld. The backbone model of baseline and Agent S is GPT-4o.**
| Method | Success Rate (%) ↑ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| OS (6) | Office (17) | Daily (16) | Profess. (10) | Workflow (16) | Overall (65) |  |
| Baseline | 33.33 | 5.88 | 12.50 | 10.00 | 6.25 | 10.77 |
| + Retrieval | 00.00 | 00.00 | 25.00 | 30.00 | 6.25 | 12.31 |
| Agent S (ACI-only) | 16.60 | 5.88 | 18.75 | 20.00 | 6.25 | 12.31 |
| + Retrieval | 33.33 | 11.76 | 31.25 | 30.00 | 6.25 | 20.00 |

### A.3 Ablations on Learning

The results presented in Table 7 demonstrate the critical role played by both the Continual Learning component and the Self-Evaluator in enhancing the performance of Agent S.

**Table 7: The detailed result of experience-augmented hierarchical planning ablation study on $test_{sub}$ of OSWorld. The backbone model of baseline and Agent S is GPT-4o.**
| Method | Success Rate (%) ↑ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| OS (6) | Office (17) | Daily (16) | Profess. (10) | Workflow (16) | Overall (65) |  |
| Agent S | 50.00 | 11.76 | 37.50 | 40.00 | 12.50 | 26.15 |
| - w/o Continual Memory Update | 33.33 | 11.76 | 37.50 | 30.00 | 12.50 | 23.08 |
| - w/o Self-Evaluator | 33.33 | 5.88 | 31.25 | 20.00 | 12.50 | 18.46 |
| - w/o Self-supervised Exploration | 33.33 | 5.88 | 25.00 | 20.00 | 6.25 | 15.38 |

## Appendix B Detailed Results on OSWorld and WindowsArena

**Table 8: Detailed success rates of baseline and Agent S using GPT-4o on OSWorld, divided by apps (domains): OS, LibreOffice Calc, LibreOffice Impress, LibreOffice Writer, Chrome, VLC Player, Thunderbird, VS Code, GIMP and Workflow involving with multiple apps.**
| Method | Success Rate (%) ↑ |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OS | Calc | Impress | Writer | VLC | TB | Chrome | VSC | GIMP | Workflow |  |
| Baseline | 1.67 | 4.26 | 6.81 | 8.70 | 9.50 | 6.67 | 15.22 | 30.43 | 0.00 | 7.46 |
| Agent S | 45.84 | 2.13 | 15.34 | 30.42 | 30.06 | 40.00 | 21.74 | 52.17 | 23.08 | 10.53 |

**Table 9: Detailed success rates of Agent S using GPT-4o on WindowArena, divided by apps (domains): Chrome, Microsoft Edge, VS Code, Notepad, LibreOffice Calc, Settings, Windows Calc, Clock, VS Code, Microsoft Paint, File Explorer, LibreOffice Writer, VLC Player.**
| Method | Success Rate (%) ↑ |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Chrome | Msedge | VSC | Notepad | Lib_Calc | Settings | Win_Calc | Clock | Paint | File | Writer | VLC |  |
| Agent S | 17.65 | 7.69 | 29.17 | 0.00 | 0.00 | 80.00 | 0.00 | 50.00 | 0.00 | 36.84 | 0.00 | 19.05 |

## Appendix C Experience-augmented Hierarchical Planning

##### Observation-Aware Query

The Manager formulates a query $Q$ based on the user task $T_{u}$ and initial observation $O_{0}$:

$$ $Q=LLM(T_{u},O_{0})$ $$

##### Narrative Memory – Storing Full Task Experiences

The narrative memory is indexed using an observation-aware query $Q$ formulated by the Manager. It is represented as:

$$ $M_{n}(Q)=\text{Save}(M_{n},Q)$ $$

where $M_{n}$ represents the narrative memory, and $Q$ is the query generated based on the user task and initial observation $O_{0}$.

##### Episodic Memory – Storing Successful Subtask Experiences

The episodic memory is used by Workers to execute subtasks and is indexed using the full User Task $T_{u}$, subtask $s_{i}$, and contextual information $C_{s_{i}}$:

$$ $M_{e}(T_{u},s_{i},C_{s_{i}})=\text{Save}(M_{e},\langle T_{u},s_{i},C_{s_{i}}\rangle)$ $$

Where $M_{e}$ represents the episodic memory.

### C.1 Manager: Fusing External Knowledge and Internal Experience for Planning

##### External Knowledge Retrieval

The query $Q$ is used to retrieve external knowledge $K_{\text{ext}}$ using the Perplexica search engine:

$$ $K_{\text{ext}}=\text{Retrieve}(\text{Web},Q)$ $$

##### Fusion of Internal Experience and External Knowledge

The internal narrative memory experience $M_{n}(Q)$ and external knowledge $K_{\text{ext}}$ are combined using the Experience Context Fusion module:

$$ $K_{\text{fused}}=\text{MLLM}(M_{n}(Q),K_{\text{ext}})$ $$

##### Subtask Planning

The fused knowledge $K_{\text{fused}}$ is used by the Manager to generate a queue of subtasks $\langle s_{0},s_{1},\ldots,s_{n}\rangle$ and associated contexts $\langle C_{s_{0}},C_{s_{1}},\ldots,C_{s_{n}}\rangle$:

$$ $\{\langle s_{0},C_{s_{0}}\rangle,\langle s_{1},C_{s_{1}}\rangle,\ldots,\langle s _{n},C_{s_{n}}\rangle\}=\text{MLLM}(K_{\text{fused}})$ $$

### C.2 Worker: Learning from Subtask Experience and Trajectory Reflection

##### Subtask Execution

Each Worker $w_{i}$ retrieves subtask experience $s_{i}$ by querying the episodic memory $M_{e}$:

$$ $E_{s_{i}}=\text{Retrieve}(M_{e},\langle T_{u},s_{i},C_{s_{i}}\rangle)$ $$

##### Trajectory Reflection

The Worker reflects on the entire episode using a Trajectory Reflector $TR_{i}$:

$$ $\text{Reflection}=TR_{i}(\text{trajectory})$ $$

This reflection helps the Worker refine its strategies.

##### Action Generation

Using the retrieved subtask experience $E_{s_{i}}$, the Worker generates a structured response for a grounded action $a_{j}$:

$$ $a_{j}=\text{MLLM}(E_{s_{i}},\text{observation},\text{Reflection})$ $$

##### Subtask Completion

The Worker signals the end of a subtask either through DONE or FAIL:

$$ $\text{status}=\begin{cases}\text{DONE},&\text{if subtask completed successfully}\\ \text{FAIL},&\text{if subtask fails}\end{cases}$ $$

### C.3 Self-Evaluator: Generating Summarized Experiences as Textual Rewards

##### Episodic Experience Update

If a Worker completes a subtask, the Self-Evaluator $S$ generates an Episodic Experience$E_{e_{i}}$ as a summary of the strategy used:

$$ $R_{s_{i}}=S(\text{Episode}_{i})$ $$

This experience is saved back into the episodic memory, indexed by the task $T_{u}$, subtask $s_{i}$, and contextual information $C_{s_{i}}$:

$$ $M_{e}\leftarrow\text{Save}(M_{e},\langle T_{u},s_{i},C_{s_{i}}\rangle,r_{s_{i}})$ $$

##### Narrative Experience Update

When the entire task is completed by the Manager $G$, the Self-Evaluator generates a task completion reward $r_{T}$, which is saved into the narrative memory, indexed by the observation-aware query $Q$ formulated by the Manager:

$$ $E_{n_{u}}=S(G(T_{u}))$ $$

$$ $M_{n}\leftarrow\text{Save}(M_{n},Q,E_{n_{u}})$ $$

## Appendix D Supplementary Examples for Qualitative Analysis

Here we present additional examples of successful and failed tasks as supplements to the qualitative analysis in §4.2. Furthermore, we provide a more detailed error analysis to complement §4.4.

### D.1 Success Examples

In this section, we present successful task examples from a variety of domains.

Figure: (a) Open Terminal: agent.click(24, 1, ”left”)
Refer to caption: extracted/5914178/figures/appendix/success/multi/step_1.png

Figure: (a) Click on the Tools menu: agent.click(38, 1, left)
Refer to caption: extracted/5914178/figures/example_figures/1.png

Figure: (a) Initiate Find and Replace: agent.hotkey([’ctrl’, ’h’])
Refer to caption: extracted/5914178/figures/appendix/success/VSCode/step_1.png

Figure: (a) Select the First Two Paragraphs: agent.click(55, 1, ”left”)
Refer to caption: extracted/5914178/figures/appendix/success/writer/step_1.png

Figure: (a) Open the Files application: agent.click(20, 1, ”left”)
Refer to caption: extracted/5914178/figures/appendix/success/OS/step_1.png

Although the agent successfully completes the tasks depicted in Figure 8 9 10 11 12, there are still issues present in its execution trajectories. For instance, during the task in Figure 10, the agent incorrectly enters the word into the wrong field at Figure 10 (c), although this mistake is corrected promptly. Furthermore, in the course of the task demonstrated in Figure 11, the agent exhibits inappropriate actions at Figure 11 (a)(b)(c). Additionally, while performing the task depicted in Figure 12, the agent fails to recognize the completion of the task at Figure 12 (d), subsequently attempting to recover an already existing file on the desktop at Figure 12 (e)(f). These issues highlight the inherent challenges in achieving consistently reliable behavior, even when tasks are nominally completed.

### D.2 Detailed Error Analysis and Failure Examples

In this section, we analyze the sources of execution errors as defined in §4.4, followed by presenting several examples of failed tasks, each with a detailed error analysis provided for the respective case.
Empirically, Grounding and planning errors often directly lead to execution errors (e.g., failing to interact with the correct target element can result in repetitive actions, and incorrect planning messages can lead to wrong decisions while performing the task). We reviewed all 39 execution errors in errors on $test_{sub}$ of OSWorld that Agent S failed to complete, as shown in Figure 13, and found that 46% were caused by planning or grounding errors. This indicates that reducing these errors, particularly grounding errors, which frequently cause repetitive actions, could significantly improve performance.

Figure: Figure 13: The error sources of the overall 39 execution errors.
Refer to caption: x7.png

During the task in Figure 14, the agent simultaneously makes planning, execution, and grounding errors. First, the inaccurate planning information in Figure 14 (a) suggests typing ’1’ instead of ’No. 1’ in the cell constitutes a planning error, leading the agent to type the incorrect value. Additionally, the agent’s attempt to drag the fill handle from ’B2’ to ’B23’ in Figure 14 (b) fails due to the selection of erroneous elements and coordinates, which can be classified as a grounding error. Furthermore, the agent continues to try to execute the subtask ’Drag the Fill Handle’ with repetitive actions in Figure 14 (c)(d)(e)(f), overlooking the prior grounding error and being unable to correct its behavior timely, which is indicative of an execution error.

Another type of planning error emerges while the agent is executing the task shown in Figure 15. The plan generated by the agent is flawed, as it incorporates an irrelevant subtask “Updating of Chrome”, which does not pertain to the intended goal. Additionally, the resulting subtask sequence is incorrect, as it erroneously prioritizes such subtask, as illustrated in Figure 15 (c)(d). This fundamental planning deficiency propagates into an execution error, preventing the agent from successfully turning off the extension, as demonstrated in the subsequent figures.

The failed task depicted in Figure 16
illustrates a scenario where the agent makes a grounding error, which subsequently leads to an execution error. After adding the Alpha Channel, the agent attempts to select the ’Fuzzy Select Tool’ from the toolbox to target the background. However, instead of selecting the correct element (represented by the magic wand icon), the agent consistently grounds to the incorrect element, ’Activity’, located at the top-left corner. This misselection brings the system to its ’Overview’ state. The agent then switches back to GIMP but continues to incorrectly select ’Activity’, mistakenly identifying it as the ’Fuzzy Select Tool’. This repeated incorrect action is demonstrated in Figure 16(e)(f)(g)(h). It is evident that the agent fails to correct its behavior promptly when facing this issue, which can be considered an execution error stemming directly from the initial grounding error.

Figure: (a) Planning Information: The information marked in red is wrong.
Refer to caption: extracted/5914178/figures/error_figures/plan.png

Figure: (a) Open Chromium Settings: agent.click(43, 1, ”left”)
Refer to caption: extracted/5914178/figures/appendix/failure/chrome/step_1.png

Figure: (a) Close the Color Profile Dialog: agent.click(167, 1, ”left”)
Refer to caption: extracted/5914178/figures/appendix/failure/gimp/step_1.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could you elaborate on how Agent S differentiates between when to retrieve external web knowledge versus when to leverage internal memories? How is this decision balanced?
2. How does Agent S handle the cold start problem, especially when it encounters a task that neither memories has prior experience with? Could you explain the strategies for quickly adapting to new tasks?
3. Your error analysis indicates a high rate of execution errors. Could you present a more detailed breakdown of the types of execution errors encountered and discuss potential improvements to the Action Generator to reduce these errors, especially in complex, long-horizon tasks?
soundness: 3
strengths: 1. Agent S stands out for its task automation through experience-augmented hierarchical planning. This method harnesses external web knowledge and draws upon internal memories, enabling the agent to decompose complex tasks into executable subtasks.
2. The introduction of the Agent-Computer Interface (ACI) is a notable strength of Agent S. This interface serves as a critical abstraction layer that facilitates precise perception and action in GUI environments. By defining a bounded action space with language-based primitives and incorporating a dual-input strategy, ACI enhances the agent's ability to ground actions and receive immediate environmental feedback. This innovation allows Agent S to operate more effectively and efficiently, setting a new standard for MLLM-based GUI agents.
summary: The paper introduces Agent S, a groundbreaking framework designed to automate complex, multi-step tasks on computers through Graphical User Interface (GUI) interaction for human usage. Agent S tackles the challenges of acquiring domain-specific knowledge, planning over long task horizons, and navigating dynamic interfaces by employing experience-augmented hierarchical planning, which leverages both external web knowledge and internal experience retrieval. It also utilizes an Agent-Computer Interface (ACI) to enhance the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). The framework demonstrated significant improvements in task success rates on the OSWorld benchmark and showed broad generalizability to different operating systems, setting a new state-of-the-art in autonomous GUI agent performance.
weaknesses: 1. The paper does not address the scalability and efficiency of the framework when handling a large volume of tasks or more complex workflows. There is a need to evaluate how the agent performs under increased load and whether the hierarchical planning and memory update mechanisms can scale without compromising the speed and accuracy of task completion.
2. The framework's performance could potentially falter in scenarios where reliable web knowledge is scarce or when there are frequent, rapid changes in application interfaces that outpace the web's ability to update corresponding information.
3. The paper acknowledges a high rate of execution errors, indicating that Agent S may struggle with decision-making and behavior adjustment during task execution.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: 1. What is the average time cost to complete a task using OpenAI API-based models and self-hosted models?
2. Does the continued memory growth consume more LLM input context tokens? What is the average context length?
soundness: 3
strengths: 1. The performance of the proposed framework on OSWorld benchmark is quite good.
2. The proposed framework is well-engineered and the evaluation is systematic.
3. The presentation and visualization of the paper is good.
summary: The paper proposes Agent S, a computer agent framework which can directly interact with computers through GUI interface. The proposed framework mainly consists of three components, a manager to manage external and internal memory for subtasks planning, a worker to complete subtasks with an episodic memory trajectory reflector, and a self-evaluator to summarize task experiences. The authors also propose an agent-computer interface as an abstraction layer. The proposed framework is evaluated on OSWorld and seen an increase of 9.37% success rate.
weaknesses: 1. It would be unfair to compare the framework only to the baseline from OSWorld, which is a benchmark paper, not a methodology paper.
2. The self-supervised exploration process is not realistic in actual deployments and I believe it will lead to overfitting.
3. The ACI proposed in this paper can only act on selected elements in the accessibility tree, which somewhat sacrifices flexibility for performance because you cannot click on every coordinate of the screen.
4. While the framework is well designed, it doesn’t introduce much new. For example, it’s quite obvious that doing some early exploration, using external resources, and OCR would help. In addition, these processes, as well as subtask planning and self-evaluation would significantly slow down the task, which is already quite slow, and would cost more OpenAI tokens.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. **Citation Format Issues**: References to OSWorld (Xie et al., 2024) and WindowsAgentArena (Bonatti et al., 2024) need clarification as they appear to be forward citations


2. **Figure Organization**: Have the authors considered combining Figures 3 and 4 for better presentation, or focusing Figure 4 more specifically on memory aspects?
soundness: 3
strengths: 1. **Novel and Effective Memory Mechanism**: Introduces a well-designed memory system with both narrative and episodic components
Provides clear algorithms for both initial memory construction and continuous updates
Demonstrates a complete closed-loop system with practical effectiveness


2. **Insightful Analysis of Agent-Computer Interaction**: Deep analysis of fundamental challenges in MLLM-based computer control
Identifies key issues like discrete time response, lack of internal coordinate systems, and inefficient feedback processing
Addresses the limitations of traditional API/script-based automation approaches


3. **Innovative ACI Design**: Proposes a dual-input strategy combining visual and accessibility tree information
Implements bounded action space with concurrent feedback
Successfully bridges the gap between MLLM agents and GUI control requirements


4. **Strong Empirical Results**: Achieves SOTA performance on established benchmarks
Provides comprehensive experimental validation
summary: This paper presents Agent S, a novel framework for GUI-based operating system control that integrates three key strategies: experience-augmented hierarchical planning, continual memory update, and an Agent-Computer Interface (ACI). The framework introduces an effective memory mechanism with initialization and continuous update algorithms, demonstrating state-of-the-art performance on computer use benchmarks like OSWorld. A notable contribution is the carefully designed ACI that addresses the unique challenges of MLLM agents interacting with desktop environments.
weaknesses: 1. **Limited Problem Definition**: The paper could benefit from a more detailed introduction to computer automation tasks
Key concepts like planning, execution, and grounding could be better explained for readers new to the field


2. **Presentation Issues**: Some overlap between Figures 3 and 4 that could be consolidated or better differentiated
Technical details of the ACI implementation could be more thoroughly described

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: First, I have some doubts about the generalizability of accessibility trees. Do all GUIs have accessibility trees, and do systems like Ubuntu and Windows have similar forms of accessibility trees? I hope the article can clarify these points. Secondly, I would like the article to compare Agent S with different agents, evaluating whether Agent S demonstrates better experimental results compared to existing GUI agents. Lastly, the ablation study does not adequately cover all points of contribution. An experiment should clarify the performance of Agent S when solely relying on image input without ACI.
soundness: 3
strengths: The advantages of this paper are that it has a clear line of thought, smooth transitions, and is easy to follow. The main architecture diagram of the agent is concise and clear, allowing for a straightforward understanding of the information flow. The selection of experimental environments is well-considered, taking into account both Ubuntu and Windows systems, which proves the effectiveness and generalization of the framework.
summary: This paper introduces a new multimodal large language model agent for GUI control, called Agent S. Its main feature is the ability to complete various tasks in a GUI interface through direct keyboard and mouse control. Comparing with other agents, Agent S incorporates an experience-augmented hierarchical planning method, which enhances the agent's ability to decompose tasks based on trajectories stored in memory. Experiments demonstrate that this framework is versatile, capable of executing various GUI-oriented tasks across different systems(Ubuntu and Windows).
weaknesses: However, the experiments in this paper are not sufficiently comprehensive. First, the baselines used are not enough, and in the comparison results of the main experiments, MLLMs are used simply instead of MLLM agents. The experimental results based on single MLLMs that only input images and accessibility trees are not convincing enough. Existing agents such as Cradle and Claude 3 perform well using only keyboard and mouse inputs without requiring additional accessibility trees. As a result, the third contribution of this paper also limits the applicability of Agent S and raises doubts about whether all GUIs provide accessibility tree inputs and whether such inputs are necessary. Thus, the contributions of this paper may seem insufficiently innovative. 

reference：
[1]Tan W, Zhang W, Xu X, et al. Cradle: Empowering Foundation Agents towards General Computer Control[C]//NeurIPS 2024 Workshop on Open-World Agents.

Meta-review: This paper introduces Agent S, a novel framework for controlling GUI-based operating systems, leveraging three core strategies: experience-augmented hierarchical planning, continual memory update, and an Agent-Computer Interface. 

The proposed framework is comprehensive and involves the key components of for LLM agents. The work performs comprehensive evaluations across multiple benchmarks and demonstrates the framework’s effectiveness. 

Although the proposed framework achieved state-of-the-art performance at the time of submission, the number of baselines is somewhat limited, potentially due to the relative novelty of the benchmarks.

Decision: Accept (Poster)