OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning

## Abstract

Abstract Large language models (LLMs) and large multimodal models (LMMs) have shown great potential in automating complex tasks like web browsing and gaming. However, their ability to generalize across diverse applications remains limited, hindering broader utility. To address this challenge, we present OSCAR : O perating S ystem C ontrol via state- A ware reasoning and R e-planning. OSCAR is a generalist agent designed to autonomously navigate and interact with various desktop and mobile applications through standardized controls, such as mouse and keyboard inputs, while processing screen images to fulfill user commands. OSCAR translates human instructions into executable Python code, enabling precise control over graphical user interfaces (GUIs). To enhance stability and adaptability, OSCAR operates as a state machine, equipped with error-handling mechanisms and task-driven re-planning, allowing it to efficiently adjust to real-time feedback and exceptions. We demonstrate OSCAR ’s effectiveness through extensive experiments on diverse benchmarks across desktop and mobile platforms, where it transforms complex workflows into simple natural language commands, significantly boosting user productivity. Our code will be open-source upon publication.

## 1 Introduction

Large Language Models (LLMs)(Ouyang et al., ; Achiam et al., ; Dubey et al., ) and Large Multimodal Models (LMMs)(Li et al., ; Team et al., ; Liu et al., ; Reid et al., ) have demonstrated exceptional performance on tasks requiring complex reasoning (Liang et al., ; Srivastava et al., ; Suzgun & Kalai, ), particularly when combined with advanced planning techniques (Wei et al., ; Wang et al., ; ) and external tools (Yang et al., ; Liu et al., ). These model-centric agents show revolutionary potential for automating real-world tasks such as web browsing (Gur et al., ), gaming (Krzywinska, ), and software development (). However, despite impressive results, these agents struggle to generalize across different applications due to variations in observation and action spaces.
In real-world scenarios, workflows often involve switching between applications and interacting with diverse graphical or command-line interfaces. This raises an intriguing and practical question: can we build a generalist agent capable of following user instructions across various applications using standardized operating system (OS) controls like mouse and keyboard inputs, while processing screen outputs?

Recent work has explored graphical user interface (GUI) control on mobile devices, with a focus on smartphone GUI understanding (You et al., ; Fan et al., ; Wu et al., ) and task automation (Yang et al., ; Guan et al., ; Zhang & Zhang, ; Wang et al., ). For desktop computers, existing approaches simulate tasks in black-box systems like AAA games (Tan et al., ) and office workflows (Wang et al., ). Some methods extend this to general OS control via visual question answering and human action trajectories (Hong et al., ; Chen et al., ; Cheng et al., ). However, these systems often lack real-time feedback from the OS and struggle to adapt dynamically when task execution fails.
Without a grounded executable environment, these methods fall short in real-world scenarios, where real-time feedback and adaptive action adjustment are crucial for navigating new GUI environments, similar to human behavior. Recently, new executable environments (Zheng et al., ; Xu et al., ; Xie et al., ) have emerged, offering dynamic feedback and enabling agents to modify their actions on the fly, paving the way for more autonomous, adaptive agents in OS control tasks.

Figure: Figure 1: Comparison of agent action sequences in static and dynamic OS environments for the task of opening and printing ’Report.docx’. The static environment (left) requires a fixed action trajectory and fails if the agent deviates. The dynamic environment (right) allows for alternative action trajectories, enabling the agent to adapt and successfully complete the task using different valid methods.
Refer to caption: x2.png

As depicted in Figure 1, consider an agent tasked with opening “Report.docx” and printing it.
In a static environment, the agent must follow a predetermined sequence of actions—clicking the Start Menu, searching for “Report.docx”, opening the file, and printing using the “Print” option in Word.
Any deviation from this sequence, such as navigating directly to “C:$\backslash$Documents$\backslash$” directory, opening the file, and printing using the Ctrl+P shortcut, results in failure because the static environment cannot accept multiple valid solutions.
In contrast, a dynamic environment allows the agent to adapt its actions based on real-time feedback, successfully completing the task using various valid methods.
This example highlights the importance of adaptability in real-world settings, where agents must handle unforeseen changes or errors. To address this limitation, we propose leveraging a large multimodal model (LMM) to develop a generalist agent capable of interpreting user commands, interacting with graphical user interfaces (GUIs), and adjusting its strategy in response to real-time feedback.

To achieve this, we identify three key challenges in building such a generalist agent for dynamic executable environments: 1) Unified Control Interfaces: The agent must seamlessly operate standard input methods like mouse and keyboard across various applications. This involves executing precise actions such as mouse movements, clicks, scrolling, and using keyboard shortcuts (e.g., Ctrl+C for copying content), all based on visual inputs; 2) GUI Grounding: The agent needs to interpret the screen and accurately identify relevant elements, such as buttons, menus, or text fields. For example, when instructed to perform a web search, it must locate and interact with the search box by correctly grounding the user instructions to the on-screen components; 3) Exploration-Based Simulation and Re-planning: Similar to how humans navigate unfamiliar software interfaces, the agent must have the ability to explore and adjust its plan dynamically. This includes retrying actions, handling exceptions like software crashes, and adapting its strategy based on real-time feedback from the system.
By addressing these challenges, we aim to develop a robust agent capable of navigating a wide range of computer applications in a flexible and reliable manner. This dynamic interaction between the agent and the operating system—driven by real-time feedback—forms the foundation of our approach, moving beyond the limitations of static, pre-scripted workflows.

In this paper, we introduce OSCAR, a general-purpose agent designed to autonomously interact with dynamic OS environments through code-centric control. OSCAR generates executable Python code to directly interface with the OS, enabling semantically clear and precise actions, ensuring broad applicability across diverse tasks.
To enhance GUI understanding, OSCAR augment screen observation with visual grounding and semantic grounding inputs by leveraging the OS window API to extract interactable elements and their spatial layout.
OSCAR operates as a state machine, continuously looping through planning, action, and re-planning to handle execution failures and system exceptions. To optimize efficiency, we incorporate task-driven re-planning, allowing the agent to adjust specific tasks rather than entire workflows, minimizing overhead and enhancing adaptability in dynamic environments.

We validated OSCAR’s effectiveness and generalizability across diverse benchmarks involving both desktop and smartphone OS environments. On the GAIA (Mialon et al., ) benchmark, OSCAR outperformed previous methods, achieving a 28.7% average success rate, with a notable 13.5% success rate on the most complex Level 3 tasks, nearly doubling the prior state-of-the-art performance. On the OSWorld (Xie et al., ) and AndroidWorld (Rawles et al., ) benchmarks, OSCAR consistently surpassed other agents, achieving a 24.5% success rate on OSWorld, and 61.6% on AndroidWorld, demonstrating superior adaptability across real-time dynamic OS tasks.
These results highlight OSCAR’s advancement in transforming tedious tasks into natural language commands, showcasing its adaptability and strong general-purpose capability.

## 2 Methodology

In this section, we introduce OSCAR, an intelligent agent designed for general-purpose control and navigation within operating systems. As illustrated in Figure 2, OSCAR operates as a state machine (Girault et al., ; Yannakakis, ), enabling it to handle dynamic OS environments through systematic state transitions. This framework allows OSCAR to efficiently process user instructions, observe the environment, plan and execute actions, and verify outcomes, while managing potential OS exceptions.
We now detail the state transition process, highlighting how OSCAR integrates GUI grounding, task-driven re-planning, and code-centric control in each operational state.

### 2.1 Formulation of State Transitions

[Init $\rightarrow$ Observe].
In the [Init] state, OSCAR awaits user instructions. Upon receiving a command, the system transitions to the [Observe] state to begin processing the input. This is the starting point for each task, and the agent returns to this state after completing or terminating a task.

[Observe $\rightarrow$ Plan].
After receiving the user’s request, OSCAR captures a screenshot of the current environment and interprets it by performing GUI grounding detailed in Section 2.2. This involves identifying screen elements, such as buttons and input fields, to understand the user interface context. The system then transitions to the [Plan] state.

[Plan $\rightarrow$ Execute, Plan $\rightarrow$ Verify].
In the [Plan] state, OSCAR generates an action plan based on the current screenshot, user instructions, context memory, and any previous verification feedback from the OS (if available). As detailed in Section 2.3, it utilizes task-driven re-planning to invoke the model backend and determine the next action.

- •
If more actions are needed to complete the task, OSCAR stores the planning results and generated actions in the context memory and transitions to the [Execute] state to interact with the operating system via executable Python code.
- •
If no further actions are necessary (the whole task completion is indicated), OSCAR transitions directly to the [Verify] state.

[Execute $\rightarrow$ Plan, Execute $\rightarrow$ Observe $\rightarrow$ Plan].
In the [Execute] state, the Python code is executed to interact with the operating system. There are two possible outcomes:

- •
If execution fails due to invalid code (e.g., syntax errors or attempts to access non-existent GUI elements), OSCAR transitions back to the [Plan] state, incorporating the interpreter’s error message for re-planning.
- •
If execution succeeds, OSCAR first transitions to the [Observe] state to capture a new screenshot, reflecting the updated state of the environment. Subsequently, OSCAR moves to the [Plan] state to plan the next action based on the new context.

Figure: Figure 2: Illustration of the state machine model used in OSCAR. The model consists of multiple states—[Init], [Observe], [Plan], [Execute], [Verify], [Success], [Fail], [Reset], and [Error]—and handles transitions between them. Transitions are triggered by user request, planning completion, verification results, or OS errors, allowing for dynamic interaction with the environment.
Refer to caption: x3.png

[Verify $\rightarrow$ Success, Verify $\rightarrow$ Plan, Verify $\rightarrow$ Fail].
In the [Verify] state, OSCAR runs evaluation scripts to validate the outcomes of the executed actions. These scripts check system or application settings and analyze file content to confirm that the intended tasks were successfully completed. Based on the results, OSCAR either transitions to the [Success] state if verification passes or returns to the [Plan] state if it fails. If the failure exceeds the allowed maximum number of attempts, OSCAR transitions to the [Fail] state.

[Success $\rightarrow$ Init].
If the task verification passes, OSCAR enters the [Success] state, signaling successful task completion and notifying the user. The system then transitions to the [Init] state, ready to process the next user query.

[Fail $\rightarrow$ Reset].
If the task cannot be completed after the maximum number of allowed attempts, OSCAR transitions to the [Fail] state, notifying the user of the failure and then transitioning to the [Reset] state.

[Plan $\rightarrow$ Error, Execute $\rightarrow$ Error, Verify $\rightarrow$ Error, Error $\rightarrow$ Reset].
OSCAR transitions to the [Error] state when a critical system exception or crash occurs, such as a local model backend failure or when too many files or processes are open in the OS. In this state, the task is terminated, and the user is notified of the error. User intervention may be required to resolve the issue before OSCAR transitions to the [Reset] state.

[Reset $\rightarrow$ Init].
In the [Reset] state, OSCAR restores the operating system to its pre-query configuration by terminating processes and closing file handlers. Once the reset is complete, OSCAR returns to the [Init] state, ready to process the next user query.

In a nutshell, the state machine architecture of OSCAR introduces continuous feedback loops, enabling dynamic interaction and error recovery, which enhances its robustness in dynamic OS environments.
Additionally, unlike previous methods that relied on linear action sequences and re-planning from scratch (Yang et al., ; Zhang et al., ; Wu et al., ), OSCAR’s state machine integrates real-time verification feedback for fine-grained, task-driven re-planning, significantly improving efficiency and adaptability.
Most importantly, its modular state transitions allow for flexible generalization across diverse OS environments, such as desktop and smartphone OS.

### 2.2 GUI-grounded Observation

While LLMs exhibit strong capabilities in understanding general visual information and grounding in broad domains, feeding a screenshot into the model to facilitate planning and output control over the screen remains insufficient. This insufficiency stems from the fact that GUI images differ significantly from natural images (Cheng et al., ), as they are densely packed with text and diverse interaction elements, such as icons and widgets, often rendered at a small scale relative to high-resolution screens. As a result, it is difficult for models to accurately locate all interaction elements and understand GUI semantics. For instance, both and could represent a settings icon, depending on the application.

To address this, we introduce a dual-grounding observation approach to enhance GUI understanding, i.e. incorporating both visual grounding and explicit semantic grounding.
Firstly, we leverage a Set-of-Mark (SoM) prompting (Yang et al., ) technique to enhance GUI visual grounding. SoM prompting, a visual prompting technique that adds marks to image regions to significantly improve LMM performance on fine-grained vision tasks.
Specifically, we utilize native window API to extract the Accessibility (A11Y) tree, a kind of structural representation providing the location, properties, and states of UI components (Consortium, ).
Based on the A11Y tree, we extract precise numerical coordinates of UI elements and map them into bounding boxes to generate SoM visual prompts (Figure 3).
The A11Y tree offers greater precision and robustness than the commonly adopted detection+OCR pipeline (Gao et al., ; Wang et al., ), particularly in complex screens with numerous UI elements where OCR often fails (see Section 3.1 for ablation analysis).

In addition to visual grounding, we further enhance GUI understanding through explicit semantic grounding by adding descriptive labels to key elements, such as: (ID: 14, Label: Start, X1: 0.35, Y1: 0.95, X2: 0.38, Y2: 1.00). These labels not only offer semantic descriptions of UI components but also facilitate code-centric control by allowing precise references to elements (e.g. by element ID).

By combining the screenshot with dual-grounding observations, OSCAR can not only grasp the overall layout and context of the GUI, but also focus on relevant areas of the screen, while flexibly referring to specific elements when needed.
This approach significantly enhances GUI understanding, ensuring robust and efficient task execution in dynamic OS environments.

Figure: Figure 3: Illustration of GUI-grounded observation in OSCAR, which includes original screenshot, semantic grounding input, and set-of-mark (SoM)-based visual grounding input.
Refer to caption: x6.png

### 2.3 Task-driven Re-planning

Interacting with dynamic environments for open-ended tasks has been well-studied in domain-specific agents, such as those agents in Minecraft (Wang et al., ; ) and data analysis (; ). Iterative planning with exploration in self-instructed task curricula has proven effective, as agents adjust their plans based on environmental feedback. These methods typically involve two stages: exploration and deployment. During the exploration phase, agents comprehensively interact with the environment to gather knowledge and experience. In the deployment phase, agents apply the learned strategies from exploration to operate and navigate new environments.

However, while navigating dynamic operating systems shares the goal of determining feasible action sequences for complex tasks, it introduces significant efficiency challenges, as agents must respond promptly to user requests. The plan-after-fully-exploration approach is inefficient for OSCAR in these contexts.
To balance efficiency and effectiveness, we introduce task-driven re-planning, while storing action trajectories and planning results in context memory to summarize and leverage past experiences. Specifically, we draw inspiration from plan-and-solve prompting (Wang et al., ; Zhang et al., ), a planning-based chain-of-thought (Wei et al., ) approach that simplifies complex tasks by breaking them into a hierarchy of sub-tasks and mapping them into executable actions.
As shown in Figure 4, we instantiate this concept as two-level planning.
Level 1: Decompose user instructions into tasks using standardized operating procedures (SOPs) (), improving clarity in task decomposition.
Level 2: For each task, generate actions step-by-step, interleaving planning and execution within OSCAR’s state machine.

Figure: Figure 4: Illustration of task-driven re-planning and code-centric control in OSCAR. Based on the current observation, context memory, and real-time OS feedback from execution or verification, OSCAR generates a refined task list and determines the next action. The action refers to GUI elements using semantic grounding input and includes executable Python code to control the OS, such as clicking the “Start” button (id=14 in Figure 3) and launching applications.
Refer to caption: x7.png

A significant advantage of task-driven re-planning is fine-grained self-refinement (Shinn et al., ; Tao et al., ), i.e. when negative feedback is received from dynamic evaluation in the state transition of [Verify] $\rightarrow$ [Plan], OSCAR can re-plan only specific tasks, rather than re-planning the entire workflow or just the current action. This approach improves planning efficiency by enabling fine-grained re-planning of tasks. It also helps avoid error propagation (Zhang & Zhang, ), where incorrect actions early on prevent successful completion of user requests, regardless of how well subsequent actions are planned.
For example, in a workflow involving multiple applications—extracting information from a Word document, observing a figure in Photos, and summarizing content in PowerPoint—each task requires several interactions. Errors in earlier tasks, such as copying text or capturing an image, will propagate and result in incorrect summaries in PowerPoint.

Formally, the complete prompt input for invoking the model is summarized in Figure 4, which includes user request, context memory, GUI-grounded observation and feedback from both execution and verification phases. The full version of system prompt can be found in the Appendix B.

### 2.4 Code-based Action

As portrayed in Figure 4, leveraging the textualized SoM from observed screenshots, OSCAR can easily refer interaction elements on the screen using element ID or numerical coordinates. This allows OSCAR to generate code to control these elements with logically clear semantics. To operationalize OSCAR’s action space, we employ the widely-used PyAutoGUI library (^1^11https://pyautogui.readthedocs.io/) for mouse and keyboard control. This library enables various mouse behaviors (movement, left-click, right-click, double-click, scroll) and keyboard interactions (single key presses, key shortcuts). Further details are summarized in Table 5.

## 3 Experiments

Benchmarks. We evaluate OSCAR on real-world workflow automation benchmarks involving complex user requests. The first benchmark is GAIA (Mialon et al., ), which consists of 466 question-answering (QA) structured into three levels: Level 1 includes simple tasks requiring no more than five steps; Level 2 involves more complex tasks with 5-10 steps and multiple tools; and Level 3 presents advanced tasks requiring over 10 actions and tool usage.
The second benchmark is OSWorld (Xie et al., ), an interactive dynamic environment with real-time OS feedback. It includes 369 tasks covering OS settings, office software, daily applications (e.g. Chrome), professional tools (e.g. VSCode), and multi-application tasks. Without a gold-standard reference action sequence, the environment allows for multiple valid solutions, which are evaluated through dynamic execution testing—verifying modified files or displayed text content in windows.
Additionally, similar to OSWorld, AndroidWorld (Rawles et al., ) provides a dynamic smartphone OS environment with 116 tasks spread across 20 diverse applications, and human annotated difficulty level: easy, medium, hard.
*Please refer to Appendix D and Appendix E for more experiments on the GUI understanding and static navigation benchmark.*

**Table 1: Real-world workflow results on the GAIA benchmark using the exact match metric. Since MMAC does not publicly release their code, we report MMAC’s results as stated in their paper and use the same base model (i.e. GPT-4-turbo ) in all of the baseline models, for a fair comparison.**
| Model | Level 1 | Level 2 | Level 3 | Average |
| --- | --- | --- | --- | --- |
| GPT-4-turbo | 9.7 | 6.9 | 0.0 | 5.5 |
| GPT-4 plugins | 30.3 | 9.7 | 0.0 | 13.3 |
| UFO | 36.9 | 16.1 | 5.4 | 19.4 |
| FRIDAY | 40.9 | 20.1 | 6.1 | 22.4 |
| MMAC | 45.2 | 20.8 | 6.1 | 24.0 |
| OSCAR | 47.0 | 25.6 | 13.5 | 28.7 |

Baselines. We compare OSCAR with seven generalist agents designed to handle dynamic OS feedback. For the desktop OS environment, we include Cradle (Tan et al., ), UFO(Zhang et al., ), FRIDAY (Wu et al., ), and MMAC (Song et al., ). For the smartphone OS environment, we evaluate against M3A (Rawles et al., ), AppAgent (Yang et al., ), and Mobile Agent (Wang et al., ). Implementation details of OSCAR and these baselines are provided in Appendix B.

Results. Table 3 summarizes the results on the GAIA benchmark, where OSCAR achieves the best performance across all three levels of workflow complexity. In particular, for Level 3 tasks, OSCAR significantly outperforms previous methods, achieving 13.5% compared to MMAC’s 6.1%, demonstrating the effectiveness of OSCAR’s task-based planning.
Additionally, as shown in Table 3, OSCAR consistently surpasses other methods across various applications in dynamic desktop OS environments. In challenging tasks involving multiple applications, OSCAR achieves a 12.9% success rate, outperforming the multi-agent baseline, UFO, which leverages dual agents to coordinate workflow decomposition and execution.
When adapting OSCAR’s action space to a mobile environment, as shown in Table 3, it achieves better average performance than the two-phase approach (comprehensive exploration followed by execution) of AppAgent, particularly in the medium and hard subsets, highlighting the effectiveness and efficiency of OSCAR’s task-driven re-planning.

### 3.1 Ablation analysis

We conduct ablation analysis on the individual components of OSCAR, including GUI-grounded observation and various planning techniques. Specifically, we first compare our GUI-grounded observation against baseline that omits the dual-grounding input, i.e. feeding raw screenshots as input. Additionally, we replace A11Y tree-based extraction with a Detection+OCR pipeline.

Figure: Figure 5: Decomposed performance of different ablation baselines. Scores are re-scaled using max-min normalization for each capability to improve clarity.
Refer to caption: x8.png

For the baselines in planning techniques, we replace our task-driven re-planning with state-of-the-art methods used in multi-step decision-making tasks, particularly for long action sequences. These include ReAct (Yao et al., ), plan-and-solve (Wang et al., ), and chain-of-action (Zhang & Zhang, ).

The results of different baselines on the OSWorld benchmark are illustrated in Figure 5. We have the following observation:
1) Both GUI-grounding and task-driven re-planning significantly enhance performance. Specifically, raw screen input without GUI grounding and direct prompts without fine-grained planning achieve only 70% and 80% of OSCAR’s full performance, respectively.
2) The Detection+OCR pipeline is less effective than the original A11Y tree-based method, particularly on the subset of professional tools with numerous UI elements, where it only marginally outperforms raw screenshot input. Furthermore, the Detection+OCR method introduces additional processing time, reinforcing the A11Y tree as the superior choice for dynamic OS environments.
3) Advanced planning strategies can significantly enhance workflow performance. For instance, ReAct and Chain-of-Action achieve results that are comparable to OSCAR in daily application and office software scenarios.
4) However, without considering real-time OS feedback and efficient re-planning, ReAct and Chain-of-Action struggle in professional software and multi-application scenarios, highlighting OSCAR’s advantage in adapting to dynamic OS environments.

### 3.2 In-depth Analysis

Instance-level analysis on planning efficiency.  To better understand why OSCAR achieves superior performance, particularly in dynamic OS environments, we take a closer look at the final success rate results and conduct an instance-level analysis for both successful and failed user requests on the OSWorld benchmark. Specifically, for the successful cases with OSCAR, we track the number of re-planning occurrences before verification failures exceed the allowed maximum number of attempts i.e. the upper bound for re-planning is the maximum number of allowed attempts. We also track the total action steps taken and the ratio of the successful action path length to the total steps, serving as a proxy for the action matching score in dynamic environments, where no reference action path exists as it does in static environments (Rawles et al., ). It is used to quantify the planning and execution efficiency in the fail-and-re-planning setting, is also referred as process score (PS) by Wang et al. (), or as completion rate (CR) by Zhang et al. ().

For failed cases, following Xu et al. (), we categorize failures into three classes: False Completion (FC), where the agent incorrectly believes the task is completed; Reach Step Limit (RSL), where the agent reaches the maximum step limit without completing the task; and Invalid Action (IA), where the agent produces outputs that do not follow instructions, including invalid formats, nonexistent actions, or incorrect action parameters. Since OSCAR can handle invalid actions and false completions through execution and verification feedback, i.e. [Execute] $\rightarrow$ [Plan] and [Verify] $\rightarrow$ [Plan] state transitions, FC and IA errors do not occur in OSCAR. We further analyze a subclass of RSL, where re-planning generates the same task list or action trajectory that has already been marked as a verification failure in previous attempts. We refer to this subclass as Redundant Re-plan (RR). For comparison, we also analyze these metrics for FRIDAY, the most competitive baseline in dynamic OS environments, as shown in Table 3.

OSCAR requires fewer re-planning attempts. As shown in Figure 6, in the successful requests, over 80% of the samples using OSCAR required fewer than 3 re-planning attempts, whereas in FRIDAY, more than 50% of the successful samples needed 3 to 4 re-planning attempts

Figure: Figure 6: Planning efficiency analysis of successful cases.
Refer to caption: x9.png

(the maximum allowed re-planning attempts in our experiments is 4, after which the case is deemed a failure).
This distribution highlights OSCAR’s efficiency advantages, as it leverages task-driven re-planning to focus on high-level task lists and perform fine-grained adjustments, rather than re-planning the entire workflow. These findings align with our goal of adapting to dynamic OS feedback while improving efficiency.

OSCAR’s re-planning includes smaller, more efficient steps.
The proxy action matching score indicates that OSCAR consistently takes smaller, more efficient steps during re-planning, while FRIDAY’s score worsens as the number of re-planning attempts increases. This efficiency is due to OSCAR’s ability to learn from previous trials, using the stored task list and action history in its context memory to optimize subsequent task lists and action trajectories upon receiving verification failure feedback.

OSCAR’s failure cases involve less redundant re-planning.
As shown in the failure case statistics in Table 4, although OSCAR may not always complete the user request within the allowed attempts,
its re-planning typically avoids repeating previously explored steps. In contrast, FRIDAY’s tendency to re-plan the entire workflow frequently (52.8%) results in generating an action trajectory that has already been verified as unsuccessful. This finding complements the success case results, where most of OSCAR’s successful cases required only 1-2 re-planning attempts.

Qualitative examples. As illustrated in Figure 7, OSCAR effectively handles complex requests involving multiple applications, i.e. OS$\rightarrow$Office$\rightarrow$OS$\rightarrow$Daily, showcasing its flexible and effective planning capabilities.
Please refer to Appendix F for more qualitative examples.

## 4 Related Works

GUI agents.  LLM and LMM-based agents (Wang et al., ; Xie et al., ; Madaan et al., ), equipped with advanced planning module (Xu et al., ; Shinn et al., ), have been developed across various environments, including robotics (Driess et al., ; Zitkovich et al., ), web browsing (Yao et al., ; Gur et al., ), gaming (Fan et al., ), software development (Yang et al., ), automating benchmark construction (Liu et al., ), data analysis (), and AI for science (Xiao et al., ).
Although recent all-in-one agent development platforms (Wu et al., ; Xie et al., ; ) have been released, most of these agents operate within specific domains, limiting their broader applicability. Among them, GUI agents—capable of interacting with various desktop and smartphone GUIs like human users—offer broader applicability in automating real-world workflows (Mialon et al., ). Some agents are continually pre-trained (Cheng et al., ) or fine-tuned (Chen et al., ) on GUI-specific data. Others simulate GUI control in sandbox environments, such as AAA games (Tan et al., ) or office workflows (Wang et al., ), which require internal application-specific APIs to interact with the environment.
In a broader context, some agents interact with basic OS APIs but are often designed for static, pre-defined environments (Reed et al., ; Hong et al., ) without grounding in real-time executable environments. Other agents follow linear action sequences and perform re-planning from scratch (Yang et al., ; Zhang et al., ; Wu et al., ) when verification fails, lacking fine-grained re-planning strategies, which makes them less efficient in real-world scenarios.
Motivated by these limitations, we design OSCAR to handle real-time dynamic OS feedback using an efficient, state-aware, task-driven re-planning strategy.

Synergizing LLMs and LMMs with OS.  Beyond GUI agents, another line of work explores integrating LLMs and LMMs with OSs in two key areas: 1) optimizing or tuning traditional OS functions using LLMs, and 2) integrating LLMs into OS kernels (LLM as OS) to serve as system-level interfaces, facilitating local agent operations and deployment.
The former includes optimizing CPU load balancing (Li et al., ), improving storage access (Wu et al., ), and identifying and repairing code vulnerabilities (Islam et al., ). The latter focuses on OS-level hardware adaptation and resource management (Kamath & Yadalam, ) as well as agent-level resource scheduling and sharing (Mei et al., ; Zhuo et al., ), such as managing agent memory and enabling efficient communication among multiple heterogeneous agents sharing the same model back-end.
Unlike these approaches, OSCAR functions as a generalist GUI agent, acting as an OS co-pilot to enhance user experience and productivity.

Figure: Figure 7: Qualitative results when processing user request “*Could you please convert ‘Pre.pptx’ to video and play it with VLC?*” on the OSWorld benchmark. Some intermediate steps and other regions of the screenshot have been omitted for clarity.
Refer to caption: x10.png

## 5 Conclusion

In this work, we introduced OSCAR, a generalist agent designed to autonomously navigate and interact with dynamic OS environments using a code-centric control framework. By leveraging task-driven re-planning and GUI-grounded observations, OSCAR demonstrates robust adaptability and effectiveness across both desktop and smartphone OS tasks. Our experiments on real-world workflow automation benchmarks, including GAIA, OSWorld, and AndroidWorld, showed that OSCAR outperforms existing methods, achieving significant improvements in task success rates, particularly for complex, multi-step tasks. Ablation studies further confirmed the importance of key components like GUI-grounded observation and task-driven re-planning for enhancing performance and minimizing redundant re-planning. Overall, OSCAR offers a versatile, efficient solution for automating workflows, making it a powerful tool for improving productivity in dynamic OS environments.

## Appendix A Overview

In the Appendix, we present:

- •
Implementation details in Appendix B.
- •
Baseline details in Appendix C.
- •
Experiments on GUI understanding benchmarks in Appendix D.
- •
Experiments on static GUI navigation benchmarks in Appendix E.
- •
Additional qualitative results in Appendix F.

## Appendix B Implementations

Observation space.
In dynamic OS environments, we extract Set-of-Mark (SoM) using native system APIs to obtain the Accessibility (A11Y) tree, as described in Section 2.2. For ablation study in Section 3.1 and other benchmarks without a dynamic OS environment, as described in Appendix D and Appendix E, i.e. only providing a screenshot, we employ an Detection+OCR pipeline to extract SoM. Specifically, we follow Gao et al. (); Wang et al. () and use YOLO-v8 (Reis et al., ) and Google OCR () to parse the GUI into SoM visual prompts, serving as auxiliary inputs for screen observation.

Action space.
The action space of OSCAR in desktop OS and smartphone OS is summarized in Table 5. This action space is used in the dynamic OS environments, i.e. OSWorld (Xie et al., ) and AndroidWorld (Rawles et al., ). For the GUI understanding benchmark described in Appendix D and the static GUI navigation benchmark in Appendix E, we adapt the action space to meet the benchmark requirements, i.e. free-form answering text format in the GUI understanding benchmark, and structural output including predefined action types and selected elements or location coordinates.

Base model.
To ensure a fair comparison, we set the base model of OSCAR and all baseline models to GPT-4o, i.e. gpt-4o-2024-05-13, except for the results on GAIA in Table 3, which are based on GPT-4-turbo, i.e. gpt-4-turbo-2024-04-09, since the baseline MMAC (Song et al., ) does not publicly release their code and their results are based on GPT-4-turbo. The temperature of response generation is set to 0.1 to reduce the variance in text generation. We provide 8 in-context demonstration examples to help the model better understand the instruction. These examples do not include a screenshot but provide a description of the current screen. All baselines are also provided with 8 in-context demonstrations to ensure a fair comparison. The full version of system prompt are provided in Figure 10 and Figure 11.

Experiment setup.
We conduct evaluation experiments on 2 A100 GPUs. Since fine-tuning the base model is not involved and it is accessed via API, the GPU is mainly required for the Detection+OCR pipeline. As this pipeline is efficient on CPU machines, all experiments can also run on regular Windows 11 machines with WSL virtualization support, which is used for encapsulating the development and test environments in Docker containers. The maximum number of allowed attempts per run is set to 4. We report the average results across 4 runs for each model on each benchmark.

**Table 5: The formulation of action space of OSCAR to navigate in desktop OS (top part) and smartphone OS (bottom part).**
| Action | Parameter | Description |
| --- | --- | --- |
| move | id: int | Move the mouse cursor to the GUI element labeled with id |
| (x: float, y: float) | Move the mouse cursor to given coordinate (x, y) |  |
| single_click | – | Click the left button of mouse at current position |
| double_click | – | Click the left button twice of mouse at current position |
| right_click | – | Click the right button of mouse at current position |
| scroll | dist: int | Scroll the mouse wheel with distance dist |
| drag | (x1: int, y1: int, x2: int, y2: int) | Hold and drag the mouse cursor from (x1, y1) to (x2, y2) |
| press | key: str | Press given key or keyboard shortcuts in current window |
| write | text: str | Write down the given text in current window |
| tap | id: int | Tap on the GUI element labeled with id |
| (x: float, y: float) | Tap the screen on given coordinate (x, y) |  |
| long_tap | id: int | Press and hold the GUI element labeled with id |
| (x: float, y: float) | Press and hold screen on given coordinate (x, y) |  |
| swipe | (id: int, dir: str, dist: float) | Swipe on an element labeled with id in a given direction dir (up, down, left, right) and distance dist. |
| swipe | (x: int, y: int, dir: str, dist: float) | Swipe from the coordinate (x, y) on the screen in a given direction dir (up, down, left, right) and distance dist. |
| write | text: str | Write down the given text in current text field |

**Table 6: Baselines for comparison with OSCAR, categorized by general-purpose out-of-the-box (OOTB) vs. specialized fine-tuned (FT) base LMMs and their target GUI environment (desktop or smartphone OS). OSCAR uniquely navigates both environments with real-time OS feedback.**
| Agent | Base Model | Desktop OS | Smartphone OS | Dynamic Feedback |
| --- | --- | --- | --- | --- |
| Auto-GUI (Zhang & Zhang, ) | OOTB | ✗ | $\checkmark$ | ✗ |
| SeeAct (Zheng et al., ) | OOTB | $\checkmark$ | ✗ | ✗ |
| CogAgent (Hong et al., ) | FT | $\checkmark$ | $\checkmark$ | ✗ |
| SeeClick (Cheng et al., ) | FT | $\checkmark$ | $\checkmark$ | ✗ |
| GUICourse (Chen et al., ) | FT | $\checkmark$ | $\checkmark$ | ✗ |
| AppAgent (Yang et al., ) | OOTB | ✗ | $\checkmark$ | $\checkmark$ |
| Mobile Agent (Wang et al., ) | OOTB | ✗ | $\checkmark$ | $\checkmark$ |
| M3A (Rawles et al., ) | OOTB | ✗ | $\checkmark$ | $\checkmark$ |
| WebAgent (Gur et al., ) | FT | $\checkmark$ | ✗ | $\checkmark$ |
| FRIDAY (Wu et al., ) | OOTB | $\checkmark$ | ✗ | $\checkmark$ |
| UFO (Zhang et al., ) | OOTB | $\checkmark$ | ✗ | $\checkmark$ |
| MMAC-Copilot (Song et al., ) | OOTB | $\checkmark$ | ✗ | $\checkmark$ |
| Cradle (Tan et al., ) | OOTB | $\checkmark$ | ✗ | $\checkmark$ |
| OSCAR | OOTB | $\checkmark$ | $\checkmark$ | $\checkmark$ |

## Appendix C Baseline Details

We employ four types of baselines for a comprehensive and fair comparison with OSCAR, categorized along two orthogonal dimensions: 1) whether the baseline is based on general-purpose out-of-the-box LMMs, or specialized LMMs that have been continually pre-trained (without human annotations) or fine-tuned (with curated human annotations) on GUI-specialized data, and 2) the target GUI scenario, whether the agent is developed for desktop OS or smartphone OS. These baselines are summarized in Table 6. To the best of our knowledge, OSCAR is the first agent capable of navigating both desktop and smartphone OS environments while responding to real-time OS feedback.

## Appendix D GUI Understanding Benchmark

Benchmarks and Evaluation.  To testify OSCAR whether possess a robust understanding of various GUI scenarios, including different OS platform and multi-window interactions, we firstly evaluation OSCAR on a comprehensive GUI understanding benchmark - GUI-World(Chen et al., ). GUI-World covering six GUI scenarios across Desktop OS and Smartphone OS and formulated as a visual question-answering task. Specifically, Given one or multiple screenshots, the agent outputs a summarized caption, layout description, and GUI elements, or infers relations between screenshots. Following Chen et al. (), we evaluate performance using automatic metrics for natural language generation, such as BERTScore(Zhang et al., ) and LLM-as-a-Judge methodology (Liu et al., ; Zheng et al., ), or accuracy metric for multiple-choice questions.

Results.  As shown in Table 7, we observe that: 1) OSCAR achieves the best GUI understanding performance across five types of GUI domains, except for websites, where the state-of-the-art agent uses an advanced parser to extract HTML as input. When HTML text is provided to OSCAR as additional input, it also demonstrates state-of-the-art performance in website GUI understanding. This success can be attributed to OSCAR’s GUI-grounded observation, which we further analyze in Section 3.2. 2) Fine-tuning on domain-specific data slightly compromises performance in more general domains. For example, the Web Agent achieves 83 on iOS GUI, significantly lower than its state-of-the-art performance of 93 on website GUI. 3) The average performance difference among the agents is marginal, highlighting the strong single-step GUI understanding capability of the base model, GPT-4o, used in our experiments.

**Table 7: Quantitative results on the GUI-World benchmark covering six types of GUI domains.**
| Model | Software | Website | XR | Multi | iOS | Android |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MC | Free | MC | Free | MC | Free | MC | Free | MC | Free | MC | Free |  |
| SeeAct | 93.9 | 4.328 | 91.1 | 4.167 | 90.6 | 4.031 | 90.1 | 4.172 | 84.8 | 3.750 | 92.3 | 3.865 |
| Auto-GUI | 94.8 | 4.422 | 90.5 | 4.131 | 89.0 | 3.904 | 88.0 | 4.073 | 84.0 | 3.666 | 91.4 | 3.742 |
| CogAgent | 94.4 | 4.322 | 87.5 | 3.976 | 90.1 | 4.031 | 88.3 | 4.086 | 88.7 | 4.193 | 93.6 | 4.056 |
| SeeClick | 91.0 | 4.083 | 88.5 | 4.038 | 89.5 | 3.893 | 89.7 | 4.176 | 88.2 | 4.078 | 94.3 | 4.124 |
| GUICourse | 92.4 | 4.156 | 88.9 | 4.038 | 90.3 | 4.057 | 88.6 | 4.111 | 88.4 | 4.168 | 92.9 | 4.058 |
| AppAgent | 86.5 | 3.644 | 84.3 | 3.805 | 90.8 | 4.159 | 89.5 | 4.176 | 90.6 | 4.398 | 95.0 | 4.326 |
| Mobile Agent | 88.6 | 3.822 | 86.0 | 3.877 | 90.6 | 4.047 | 89.7 | 4.199 | 91.1 | 4.482 | 94.9 | 4.230 |
| M3A | 88.1 | 3.799 | 84.1 | 3.803 | 92.1 | 4.270 | 94.1 | 4.456 | 89.6 | 4.278 | 93.5 | 4.127 |
| WebAgent | - | - | - | - | - | - | - | - | - | - | - | - |
| FRIDAY | 95.1 | 4.406 | 89.4 | 4.090 | 87.7 | 3.662 | 86.5 | 3.991 | 85.2 | 3.768 | 92.0 | 3.845 |
| UFO | 94.4 | 4.352 | 91.0 | 4.182 | 91.4 | 4.159 | 89.8 | 4.179 | 84.8 | 3.778 | 90.6 | 3.649 |
| MMAC | - | - | - | - | - | - | - | - | - | - | - | - |
| OSCAR | 96.4 | 4.509 | 89.2 | 4.035 | 94.4 | 4.551 | 95.5 | 4.527 | 92.7 | 4.585 | 96.4 | 4.524 |
| OSCAR+HTML | - | - | 92.2 | 4.235 | - | - | - | - | - | - | - | - |

## Appendix E Static GUI Navigation Benchmark

Benchmarks and Evaluation.  We evaluate OSCAR on GUI navigation benchmarks involving multi-step decision-making in pre-defined interaction episodes, which includes widely adopted datasets such as Mind2Web(Deng et al., ) (Desktop OS) and AITW(Rawles et al., ) (Smartphone OS). These benchmarks consist of high-level task descriptions, gold reference sequences of actions, and corresponding observations in HTML and screenshots. Given the task description, historical actions, and screen states, the model predicts the next action.
Borrowing setting from Cheng et al. (); Rawles et al. (), we evaluate performance
using the Step Success Rate (both the selected element and predicted operation are correct), Task Success Rate (all steps are correct), and a screen-wise action matching score (the number of correct steps divided by the total number of steps). Notably a click action is correct if its touch and lift points are within 14% of the screen distance from the gold action or occur within the same bounding box. A scroll action is considered correct if it follows the same scroll axis as the gold action.

Results.  Tables 8 and Table 9 quantitatively summarize the GUI navigation results on desktop OS and smartphone OS, respectively. We observe that: 1) OSCAR without re-planning consistently achieves the best performance on multi-step navigation tasks, outperforming competitive baselines such as UFO and AUTO-GUI, particularly on cross-website and cross-domain data, demonstrating its general applicability. 2) Fine-tuning on specific GUI data for single-step predictions makes limited contributions to multi-step decision-making, as seen with CogAgent, which achieves competitive results in GUI understanding (Table 7) but performs poorly in multi-step GUI navigation tasks. A possible explanation is that domain-specific fine-tuning increases the probability of hallucinated actions when intermediate feedback is not available from static environments (Qiao et al., ).

## Appendix F Qualitative Results

Figure 8 and Figure 9 present qualitative results of OSCAR’s on the daily application and professional tool, respectively.

**Table 8: Desktop OS GUI navigation results on the Mind2Web benchmark in terms of element accuracy (Ele.Acc), Operation F1 (Op.F1) and step success rate (Step SR).**
| Model | Cross-Task | Cross-Website | Cross-Domain |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ele.Acc | Op.F1 | Step SR | Ele.Acc | Op.F1 | Step SR | Ele.Acc | Op.F1 | Step SR |  |
| SeeAct | 31.8 | 89.3 | 29.6 | 25.5 | 85.0 | 20.4 | 26.6 | 87.3 | 23.6 |
| CogAgent | 31.1 | 88.6 | 28.8 | 25.6 | 84.8 | 20.4 | 27.1 | 87.7 | 24.2 |
| SeeClick | 28.3 | 86.9 | 25.5 | 21.4 | 80.5 | 16.4 | 23.3 | 85.1 | 20.9 |
| GUICourse | 31.8 | 89.6 | 29.6 | 26.4 | 85.7 | 21.2 | 27.8 | 88.4 | 25.0 |
| WebAgent | - | - | - | - | - | - | - | - | - |
| FRIDAY | 31.3 | 89.4 | 28.8 | 27.2 | 86.0 | 22.2 | 28.4 | 89.0 | 25.5 |
| UFO | 33.5 | 90.1 | 31.3 | 27.2 | 86.2 | 22.1 | 27.9 | 88.4 | 24.8 |
| MMAC | - | - | - | - | - | - | - | - | - |
| OSCAR w/o Re-plan | 35.5 | 92.4 | 33.9 | 29.6 | 88.3 | 24.5 | 29.8 | 90.0 | 26.5 |

**Table 9: Smartphone OS GUI navigation results on the AITW benchmark in terms of action matching score.**
| Model | General | Install | GoogleApps | Single step | Webshopping |
| --- | --- | --- | --- | --- | --- |
| Auto-GUI | 67.9 | 76.7 | 71.2 | 84.5 | 70.5 |
| CogAgent | 61.0 | 72.0 | 64.7 | 73.8 | 65.0 |
| SeeClick | 54.2 | 66.7 | 54.6 | 63.5 | 57.6 |
| GUICourse | 64.1 | 73.5 | 66.3 | 78.0 | 66.2 |
| AppAgent | 58.1 | 70.7 | 59.9 | 71.7 | 61.5 |
| Mobile Agent | 59.5 | 71.4 | 61.4 | 73.9 | 63.8 |
| M3A | 65.4 | 75.7 | 68.0 | 82.9 | 68.0 |
| OSCAR w/o Re-plan | 71.4 | 78.7 | 74.8 | 88.6 | 73.0 |

Figure: Figure 8: Qualitative results when processing user request “*Please open Notepad, create a new file named “draft.txt”, type “This is a draft.”, and save it to the Documents folder.*”.
Refer to caption: x11.png

Figure: Figure 9: Qualitative results when processing user request “*Install the pylance extension in VS Code.*”.
Refer to caption: x12.png

Figure: Figure 10: Part 1/2 of system prompt of OSCAR.

Figure: Figure 11: Part 2/2 of system prompt of OSCAR.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - What makes OSCAR outperform other baselines mostly (since many methods also have their feedback loop)?
- Will a fine-tuned LM be eventually required for the robustness in real-world scenarios beyond those benchmarks? To what degree is a user/expert required to supervise the agent's operation?
- Is plan -> execution -> error -> re-plan enough even for complex tasks or sampling-based methods such as evolutionary search (but costly and redundant) should be also considered.
soundness: 3
strengths: - This work shows a robust and efficient OS agent by resorting to a general purposed language models without fine-tuning. To cope with GUI recognition difficulties (beyond general vision tasks), SoM techniques are applied to improve reliability.
- Many baseline comparisons and detailed implementation are also provided.
summary: - This work presents an LLM+LMM-based agent to solve Desktop/Mobile OS tasks, including key modules such as GUI elements recognition and SoM prompting, plan/re-plan state machine and code execution for general mouse/keyboard control.
- Experiments on multiple, both static and dynamic, benchmarks demonstrate the proposed framework's efficacy and generalizability.
weaknesses: - Discussion on safety may be helpful since the agent has control over an OS.
- Although it uses a state machine and re-planning, OSCAR lacks a self-improvement mechanism. Do you have more insights on representing states, e.g. system errors/task failures, beyond pure text language?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In the dual-grounding observation approach, how do you add the labels to each element to provide explicit semantic grounding?
2. What does "evaluation scripts" in section 2.1 refer to? Do you leverage the task-specific evaluation scripts from OSWorld? You may make it more clear about how you evaluate OSCAR on each of the three benchmarks.
3. As you allow 4 attempts per run for the agent to finish the task, is it crucial for a better performance and higher score on the benchmark? Do they always learn from failures and need more attempts to do better?
4. A more detailed error analysis is needed to specifically analyze the reasons for failure cases of reaching step limits, as this may help further improve OSCAR's error handling mechanism.
soundness: 4
strengths: 1. The use of a task-driven re-planning strategy allows OSCAR to adjust its actions based on real-time feedback, which enhances its ability to correct itself, do the state-aware reasoning and complete complex tasks autonomously.
2. OSCAR is evaluated against a diverse set of benchmarks, demonstrating superior performance in both desktop and smartphone environments, which underscores its generalizability and effectiveness.
3. The state transition process in OSCAR allows for systematic handling of tasks by structuring operations into distinct phases, such as observation, planning, execution, and verification. This structured approach enhances error recovery and adaptability by enabling the agent to assess and resolve issues at each state
summary: The paper presents OSCAR, a generalist agent designed to autonomously navigate and interact with both desktop and mobile applications using standard controls like mouse and keyboard. The agent aims to interpret user commands, interact with graphical user interfaces, and adapt its strategies based on real-time feedback. OSCAR is constructed as a state transition process and integrated with a GUI dual-grounding observation and task-driven re-planning. Further, the authors evaluate OSCAR on three digital task benchmarks, OSWorld, GAIA and AndroidWorld, outperforming current SOTA AI systems in completing digital tasks.
weaknesses: 1. I think it would be better to give some detailed cases to explain how the state transition works to complete the task and handle the errors.
2. There could be more discussions about how to fit OSCAR into different OS envs and generalize it to more benchmarks and systems.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: No questions
soundness: 3
strengths: 1. OSCAR demonstrates strong performance on evaluated benchmarks, achieving an average success rate of 28.7% and a success rate of 13.5% on the most challenging Level 3 tasks in the GAIA benchmark.
2. OSCAR implements a state machine with well-defined behaviors to manage complex, dynamic environments effectively.
3. Grounding GUI images with semantic symbols offers a more systematic approach for VLM to interpret the current state of the OS environment.
summary: This work introduces OSCAR, a generalist agent that autonomously navigates and interacts with desktop and mobile applications through standard controls like mouse and keyboard inputs. OSCAR’s framework includes state transitions that dynamically adapt to various environments and are equipped with error-handling mechanisms. OSCAR also addresses the challenge of VLM’s difficulty in interpreting GUI screenshots by grounding them with semantic symbols. Finally, OSCAR incorporates task-driven replanning for efficient real-time adjustments based on feedback and exceptions. OSCAR is evaluated on GAIA, OSWorld, and AndroidWorld benchmarks. The results show that OSCAR significantly outperforms other baseline methods.
weaknesses: The [Verify] state is crucial to OSCAR’s framework; however, its exact functionality remains unclear. For instance, what occurs when [Verify] yields a false positive, such as validating an invalid plan? Can the system recover from these failures? Conducting an in-depth analysis of the [Verify] step's failure modes and success rate would significantly enhance the paper's quality.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See Weaknesses above.
soundness: 2
strengths: 1. The proposed agent framework is formulated as a state machine, making it clear to follow.
2. OSCAR outperforms several baselines in both desktop and mobile environments.
summary: This paper introduces OSCAR, a computer agent framework mainly focused on planning and aimed to generalize across diverse applications. The experiments demonstrate OSCAR’s effectiveness on GAIA, OSWorld, and AndroidWorld.
weaknesses: 1. The proposed modules (task-driven re-planning, context memory, GUI observations, and code actions) are not novel. Many closely related methods are not discussed, e.g.:
- Kim et al. "Language models can solve computer tasks." NeurIPS 2023.
- Sun et al. "Adaplanner: Adaptive planning from feedback with language models." NeurIPS 2023.
- Zheng et al. "Synapse: Trajectory-as-exemplar prompting with memory for computer control." ICLR 2024.

2. The experiments only ablate GUI observations and the replanning module. The analysis claims that the improvements come from “fewer re-planning attempts” and “smaller, more efficient steps”, but it is unclear how the proposed planning module contributes to this compared to the baselines.

3. In Table 3, OSCAR outperforms baselines in medium and hard cases but is worse in easy cases, for which the authors should analyze the potential reasons.

4. It is unclear how OSCAR leverages “real-time” feedback from the OS. Additional experiments on AgentStudio, a more complex, dynamic environment, are helpful.

5. The limitations of OSCAR are not discussed.

6. Minor: there are many typos, e.g.:
- line 300, “i.e. when” -> “i.e., when”; line 332, “(e.g. Chrome)” -> “(e.g., Chrome)”; and so on.
- missing or abundant space in Section 1 “(LLMs)(Ouyang …”, “(LMMs)(Li …”, Section 3 baselines “UFO(Zhang et al., 2024a)”, Table 1 caption “(i.e. GPT-4-turbo )”, etc.

I will raise my score if the authors resolve the above concerns.

Meta-review: **Summary**: This paper presents OSCAR, a generalist agent designed for autonomous interaction with desktop and mobile operating systems. OSCAR uses state-aware reasoning and task-driven re-planning to dynamically interact with graphical user interfaces (GUIs) through mouse and keyboard inputs. The agent processes screen observations, translates user instructions into executable Python code, and adapts to errors using a state machine framework. OSCAR’s robustness and adaptability are validated through extensive experiments on benchmarks such as OSWorld, GAIA, and AndroidWorld, where it outperforms current state-of-the-art systems. The paper highlights OSCAR’s ability to generalize across dynamic OS environments and handle real-time feedback effectively.

**Strengths**:
- All reviewers praise the thorough and diverse evaluations across dynamic OS benchmarks, including OSWorld, GAIA, and AndroidWorld. OSCAR achieves state-of-the-art performance, demonstrating its ability to adapt to various tasks and outperform existing agents like FRIDAY and UFO.
- Reviewers recognize OSCAR’s dual-grounding approach for GUI understanding, combining visual and semantic grounding, as a significant advancement. This enables the agent to interpret complex user interfaces more accurately and robustly, enhancing task execution.

**Weaknesses**:
- Safety concerns: Reviewer `TSTo` raises concerns about safety, particularly when an agent has autonomous control over operating systems. The authors admit that safety (and scalability) issues are the biggest limitations of the paper, and provide more discussion in the rebuttal. While the authors discuss virtualization and safeguards, a more thorough exploration of potential misuse and real-world deployment risks would strengthen the paper.
- Scalability concerns: Reviewer `TSTo` questions whether OSCAR’s current plan-execute-replan cycle is sufficient for highly complex tasks. Exploration-based methods like evolutionary search could provide additional robustness but were not fully considered in this work.
- Generalization to other OSs: Reviewer `Bipc` points out that adapting OSCAR to other operating systems and benchmarks requires additional effort. While the authors clarify that the modular design aids generalizability, further examples or discussions would improve confidence in OSCAR’s scalability.
- Lack of self-improvement mechanisms and error analysis discussion: Reviewer `TSTo` notes the lack of advanced self-improvement mechanisms and reviewer `Bipc` requests more discussion on error analysis. While OSCAR handles task failures well through state-driven re-planning, it does not learn from cross-task experiences or explore structural improvements beyond text-based state representation.

**Recommendation**: Based on the reviewers' feedback and discussion, the paper offers an innovative, well-executed solution for dynamic OS control tasks. While safety considerations and generalization to new environments remain areas for improvement, the authors' rebuttal addressed these concerns adequately. In my opinion, OSCAR’s strong performance, technical contributions, and experimental validation make it a valuable addition to the field, and my vote is for Accept as a poster.

Decision: Accept (Poster)