Agent S: An Open Agentic Framework that Uses Computers Like a Human

Authors: Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang
Primary area: applications to robotics, autonomy, planning
Keywords: Large Vision and Language Model, Agents, Retrieval Augmented Generation, GUI, Large Language Models, Agent Computer Interface

Abstract:
We present Agent S, an open agentic framework that enables autonomous interaction with computers through Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. Agent S addresses three key challenges in automating computer tasks: acquiring domain-specific knowledge, planning over long task horizons, and handling dynamic, non-uniform interfaces. To this end, Agent S introduces experience-augmented hierarchical planning, which learns from external knowledge search and internal experience retrieval at multiple levels, facilitating efficient task planning and subtask execution. 
In addition, it employs an Agent-Computer Interface (ACI) to better elicit the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the baseline by 9.37\% on success rate (an 83.6\% relative improvement) and achieves a new state-of-the-art. Comprehensive analysis highlights the effectiveness of individual components and provides insights for future improvements. Furthermore, Agent S demonstrates broad generalizability to different operating systems on a newly-released WindowsAgentArena benchmark. Code available at https://github.com/simular-ai/Agent-S.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730127477256

thread_type:
official_review

<thread_object>
creator: reviewer_7e6z

<content>
summary:
This paper introduces a new multimodal large language model agent for GUI control, called Agent S. Its main feature is the ability to complete various tasks in a GUI interface through direct keyboard and mouse control. Comparing with other agents, Agent S incorporates an experience-augmented hierarchical planning method, which enhances the agent's ability to decompose tasks based on trajectories stored in memory. Experiments demonstrate that this framework is versatile, capable of executing various GUI-oriented tasks across different systems(Ubuntu and Windows).

strengths:
The advantages of this paper are that it has a clear line of thought, smooth transitions, and is easy to follow. The main architecture diagram of the agent is concise and clear, allowing for a straightforward understanding of the information flow. The selection of experimental environments is well-considered, taking into account both Ubuntu and Windows systems, which proves the effectiveness and generalization of the framework.

weaknesses:
However, the experiments in this paper are not sufficiently comprehensive. First, the baselines used are not enough, and in the comparison results of the main experiments, MLLMs are used simply instead of MLLM agents. The experimental results based on single MLLMs that only input images and accessibility trees are not convincing enough. Existing agents such as Cradle and Claude 3 perform well using only keyboard and mouse inputs without requiring additional accessibility trees. As a result, the third contribution of this paper also limits the applicability of Agent S and raises doubts about whether all GUIs provide accessibility tree inputs and whether such inputs are necessary. Thus, the contributions of this paper may seem insufficiently innovative. 

reference：
[1]Tan W, Zhang W, Xu X, et al. Cradle: Empowering Foundation Agents towards General Computer Control[C]//NeurIPS 2024 Workshop on Open-World Agents.

questions:
First, I have some doubts about the generalizability of accessibility trees. Do all GUIs have accessibility trees, and do systems like Ubuntu and Windows have similar forms of accessibility trees? I hope the article can clarify these points. Secondly, I would like the article to compare Agent S with different agents, evaluating whether Agent S demonstrates better experimental results compared to existing GUI agents. Lastly, the ablation study does not adequately cover all points of contribution. An experiment should clarify the performance of Agent S when solely relying on image input without ACI.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730362782386

thread_type:
official_review

<thread_object>
creator: reviewer_knhm

<content>
summary:
This paper presents Agent S, a novel framework for GUI-based operating system control that integrates three key strategies: experience-augmented hierarchical planning, continual memory update, and an Agent-Computer Interface (ACI). The framework introduces an effective memory mechanism with initialization and continuous update algorithms, demonstrating state-of-the-art performance on computer use benchmarks like OSWorld. A notable contribution is the carefully designed ACI that addresses the unique challenges of MLLM agents interacting with desktop environments.

strengths:
1. **Novel and Effective Memory Mechanism**: Introduces a well-designed memory system with both narrative and episodic components
Provides clear algorithms for both initial memory construction and continuous updates
Demonstrates a complete closed-loop system with practical effectiveness


2. **Insightful Analysis of Agent-Computer Interaction**: Deep analysis of fundamental challenges in MLLM-based computer control
Identifies key issues like discrete time response, lack of internal coordinate systems, and inefficient feedback processing
Addresses the limitations of traditional API/script-based automation approaches


3. **Innovative ACI Design**: Proposes a dual-input strategy combining visual and accessibility tree information
Implements bounded action space with concurrent feedback
Successfully bridges the gap between MLLM agents and GUI control requirements


4. **Strong Empirical Results**: Achieves SOTA performance on established benchmarks
Provides comprehensive experimental validation

weaknesses:
1. **Limited Problem Definition**: The paper could benefit from a more detailed introduction to computer automation tasks
Key concepts like planning, execution, and grounding could be better explained for readers new to the field


2. **Presentation Issues**: Some overlap between Figures 3 and 4 that could be consolidated or better differentiated
Technical details of the ACI implementation could be more thoroughly described

questions:
1. **Citation Format Issues**: References to OSWorld (Xie et al., 2024) and WindowsAgentArena (Bonatti et al., 2024) need clarification as they appear to be forward citations


2. **Figure Organization**: Have the authors considered combining Figures 3 and 4 for better presentation, or focusing Figure 4 more specifically on memory aspects?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730435032454

thread_type:
official_review

<thread_object>
creator: reviewer_yyg1

<content>
summary:
The paper proposes Agent S, a computer agent framework which can directly interact with computers through GUI interface. The proposed framework mainly consists of three components, a manager to manage external and internal memory for subtasks planning, a worker to complete subtasks with an episodic memory trajectory reflector, and a self-evaluator to summarize task experiences. The authors also propose an agent-computer interface as an abstraction layer. The proposed framework is evaluated on OSWorld and seen an increase of 9.37% success rate.

strengths:
1. The performance of the proposed framework on OSWorld benchmark is quite good.
2. The proposed framework is well-engineered and the evaluation is systematic.
3. The presentation and visualization of the paper is good.

weaknesses:
1. It would be unfair to compare the framework only to the baseline from OSWorld, which is a benchmark paper, not a methodology paper.
2. The self-supervised exploration process is not realistic in actual deployments and I believe it will lead to overfitting.
3. The ACI proposed in this paper can only act on selected elements in the accessibility tree, which somewhat sacrifices flexibility for performance because you cannot click on every coordinate of the screen.
4. While the framework is well designed, it doesn’t introduce much new. For example, it’s quite obvious that doing some early exploration, using external resources, and OCR would help. In addition, these processes, as well as subtask planning and self-evaluation would significantly slow down the task, which is already quite slow, and would cost more OpenAI tokens.

questions:
1. What is the average time cost to complete a task using OpenAI API-based models and self-hosted models?
2. Does the continued memory growth consume more LLM input context tokens? What is the average context length?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648142100

thread_type:
official_review

<thread_object>
creator: reviewer_mel5

<content>
summary:
The paper introduces Agent S, a groundbreaking framework designed to automate complex, multi-step tasks on computers through Graphical User Interface (GUI) interaction for human usage. Agent S tackles the challenges of acquiring domain-specific knowledge, planning over long task horizons, and navigating dynamic interfaces by employing experience-augmented hierarchical planning, which leverages both external web knowledge and internal experience retrieval. It also utilizes an Agent-Computer Interface (ACI) to enhance the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). The framework demonstrated significant improvements in task success rates on the OSWorld benchmark and showed broad generalizability to different operating systems, setting a new state-of-the-art in autonomous GUI agent performance.

strengths:
1. Agent S stands out for its task automation through experience-augmented hierarchical planning. This method harnesses external web knowledge and draws upon internal memories, enabling the agent to decompose complex tasks into executable subtasks.
2. The introduction of the Agent-Computer Interface (ACI) is a notable strength of Agent S. This interface serves as a critical abstraction layer that facilitates precise perception and action in GUI environments. By defining a bounded action space with language-based primitives and incorporating a dual-input strategy, ACI enhances the agent's ability to ground actions and receive immediate environmental feedback. This innovation allows Agent S to operate more effectively and efficiently, setting a new standard for MLLM-based GUI agents.

weaknesses:
1. The paper does not address the scalability and efficiency of the framework when handling a large volume of tasks or more complex workflows. There is a need to evaluate how the agent performs under increased load and whether the hierarchical planning and memory update mechanisms can scale without compromising the speed and accuracy of task completion.
2. The framework's performance could potentially falter in scenarios where reliable web knowledge is scarce or when there are frequent, rapid changes in application interfaces that outpace the web's ability to update corresponding information.
3. The paper acknowledges a high rate of execution errors, indicating that Agent S may struggle with decision-making and behavior adjustment during task execution.

questions:
1. Could you elaborate on how Agent S differentiates between when to retrieve external web knowledge versus when to leverage internal memories? How is this decision balanced?
2. How does Agent S handle the cold start problem, especially when it encounters a task that neither memories has prior experience with? Could you explain the strategies for quickly adapting to new tasks?
3. Your error analysis indicates a high rate of execution errors. Could you present a more detailed breakdown of the types of execution errors encountered and discuss potential improvements to the Action Generator to reduce these errors, especially in complex, long-horizon tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732269684588

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of responses and revisions

comment:
We are grateful to all the reviewers for their insightful comments and suggestions for our work. We are pleased that the reviewers found our framework effective and achieving strong performance on benchmarks (**YYG1**, **KnHm**, **7E6Z**); that our experimental evaluation is systematic and comprehensive (**YYG1**, **KnHm**); that our Agent-Computer Interface (ACI) design for GUI control is innovative (**meL5**, **KnHm**); that our novel memory mechanism is effective (**meL5**, **KnHm**); and that the paper is well-presented and easy to follow (**YYG1**, **7E6Z**).

We have made the following revisions based on Reviewers’ feedback:

1. Explanations for Problem definition and hierarchical planning have been added to Section 3, and additional context about UI element grounding is added to Subsection 3.3 to facilitate understanding for new readers
2. Table 1. has been updated to represent that the baselines used in the paper are based on the MMAgent, and other agent baselines (CRADLE, OpenInterpreter, and FRIDAY) have been added. 
3. Figure 4 has been updated to focus more on memory construction and learning
4. Figure 8 has been added to the Appendix to illustrate the Agent Computer Interface 

We have also added additional results in comments to the Reviewers to answer their questions and address their concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Agent S, a novel framework for controlling GUI-based operating systems, leveraging three core strategies: experience-augmented hierarchical planning, continual memory update, and an Agent-Computer Interface. 

The proposed framework is comprehensive and involves the key components of for LLM agents. The work performs comprehensive evaluations across multiple benchmarks and demonstrates the framework’s effectiveness. 

Although the proposed framework achieved state-of-the-art performance at the time of submission, the number of baselines is somewhat limited, potentially due to the relative novelty of the benchmarks.