OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning

Authors: Xiaoqiang Wang, Bang Liu
Primary area: applications to robotics, autonomy, planning
Keywords: Large Language Model, Autonomous Agent, Graphical User Interface

Abstract:
Large language models (LLMs) and large multimodal models (LMMs) have shown great potential in automating complex tasks like web browsing and gaming. However, their ability to generalize across diverse applications remains limited, hindering broader utility. To address this challenge, we present OSCAR: Operating System Control via state-Aware reasoning and Re-planning. OSCAR is a generalist agent designed to autonomously navigate and interact with various desktop and mobile applications through standardized controls, such as mouse and keyboard inputs, while processing screen images to fulfill user commands. OSCAR translates human instructions into executable Python code, enabling precise control over graphical user interfaces (GUIs). To enhance stability and adaptability, OSCAR operates as a state machine, equipped with error-handling mechanisms and dynamic task re-planning, allowing it to efficiently adjust to real-time feedback and exceptions. We demonstrate OSCAR’s effectiveness through extensive experiments on diverse benchmarks across desktop and mobile platforms, where it transforms complex workflows into simple natural language commands, significantly boosting user productivity. Our code will be open-source upon publication.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730046970322

thread_type:
official_review

<thread_object>
creator: reviewer_cncy

<content>
summary:
This paper introduces OSCAR, a computer agent framework mainly focused on planning and aimed to generalize across diverse applications. The experiments demonstrate OSCAR’s effectiveness on GAIA, OSWorld, and AndroidWorld.

strengths:
1. The proposed agent framework is formulated as a state machine, making it clear to follow.
2. OSCAR outperforms several baselines in both desktop and mobile environments.

weaknesses:
1. The proposed modules (task-driven re-planning, context memory, GUI observations, and code actions) are not novel. Many closely related methods are not discussed, e.g.:
- Kim et al. "Language models can solve computer tasks." NeurIPS 2023.
- Sun et al. "Adaplanner: Adaptive planning from feedback with language models." NeurIPS 2023.
- Zheng et al. "Synapse: Trajectory-as-exemplar prompting with memory for computer control." ICLR 2024.

2. The experiments only ablate GUI observations and the replanning module. The analysis claims that the improvements come from “fewer re-planning attempts” and “smaller, more efficient steps”, but it is unclear how the proposed planning module contributes to this compared to the baselines.

3. In Table 3, OSCAR outperforms baselines in medium and hard cases but is worse in easy cases, for which the authors should analyze the potential reasons.

4. It is unclear how OSCAR leverages “real-time” feedback from the OS. Additional experiments on AgentStudio, a more complex, dynamic environment, are helpful.

5. The limitations of OSCAR are not discussed.

6. Minor: there are many typos, e.g.:
- line 300, “i.e. when” -> “i.e., when”; line 332, “(e.g. Chrome)” -> “(e.g., Chrome)”; and so on.
- missing or abundant space in Section 1 “(LLMs)(Ouyang …”, “(LMMs)(Li …”, Section 3 baselines “UFO(Zhang et al., 2024a)”, Table 1 caption “(i.e. GPT-4-turbo )”, etc.

I will raise my score if the authors resolve the above concerns.

questions:
See Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730413429033

thread_type:
official_review

<thread_object>
creator: reviewer_1zyj

<content>
summary:
This work introduces OSCAR, a generalist agent that autonomously navigates and interacts with desktop and mobile applications through standard controls like mouse and keyboard inputs. OSCAR’s framework includes state transitions that dynamically adapt to various environments and are equipped with error-handling mechanisms. OSCAR also addresses the challenge of VLM’s difficulty in interpreting GUI screenshots by grounding them with semantic symbols. Finally, OSCAR incorporates task-driven replanning for efficient real-time adjustments based on feedback and exceptions. OSCAR is evaluated on GAIA, OSWorld, and AndroidWorld benchmarks. The results show that OSCAR significantly outperforms other baseline methods.

strengths:
1. OSCAR demonstrates strong performance on evaluated benchmarks, achieving an average success rate of 28.7% and a success rate of 13.5% on the most challenging Level 3 tasks in the GAIA benchmark.
2. OSCAR implements a state machine with well-defined behaviors to manage complex, dynamic environments effectively.
3. Grounding GUI images with semantic symbols offers a more systematic approach for VLM to interpret the current state of the OS environment.

weaknesses:
The [Verify] state is crucial to OSCAR’s framework; however, its exact functionality remains unclear. For instance, what occurs when [Verify] yields a false positive, such as validating an invalid plan? Can the system recover from these failures? Conducting an in-depth analysis of the [Verify] step's failure modes and success rate would significantly enhance the paper's quality.

questions:
No questions
</content>
</thread_object>

</thread>
<thread>
creation date:
1730566867645

thread_type:
official_review

<thread_object>
creator: reviewer_bipc

<content>
summary:
The paper presents OSCAR, a generalist agent designed to autonomously navigate and interact with both desktop and mobile applications using standard controls like mouse and keyboard. The agent aims to interpret user commands, interact with graphical user interfaces, and adapt its strategies based on real-time feedback. OSCAR is constructed as a state transition process and integrated with a GUI dual-grounding observation and task-driven re-planning. Further, the authors evaluate OSCAR on three digital task benchmarks, OSWorld, GAIA and AndroidWorld, outperforming current SOTA AI systems in completing digital tasks.

strengths:
1. The use of a task-driven re-planning strategy allows OSCAR to adjust its actions based on real-time feedback, which enhances its ability to correct itself, do the state-aware reasoning and complete complex tasks autonomously.
2. OSCAR is evaluated against a diverse set of benchmarks, demonstrating superior performance in both desktop and smartphone environments, which underscores its generalizability and effectiveness.
3. The state transition process in OSCAR allows for systematic handling of tasks by structuring operations into distinct phases, such as observation, planning, execution, and verification. This structured approach enhances error recovery and adaptability by enabling the agent to assess and resolve issues at each state

weaknesses:
1. I think it would be better to give some detailed cases to explain how the state transition works to complete the task and handle the errors.
2. There could be more discussions about how to fit OSCAR into different OS envs and generalize it to more benchmarks and systems.

questions:
1. In the dual-grounding observation approach, how do you add the labels to each element to provide explicit semantic grounding?
2. What does "evaluation scripts" in section 2.1 refer to? Do you leverage the task-specific evaluation scripts from OSWorld? You may make it more clear about how you evaluate OSCAR on each of the three benchmarks.
3. As you allow 4 attempts per run for the agent to finish the task, is it crucial for a better performance and higher score on the benchmark? Do they always learn from failures and need more attempts to do better?
4. A more detailed error analysis is needed to specifically analyze the reasons for failure cases of reaching step limits, as this may help further improve OSCAR's error handling mechanism.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731079253845

thread_type:
official_review

<thread_object>
creator: reviewer_tsto

<content>
summary:
- This work presents an LLM+LMM-based agent to solve Desktop/Mobile OS tasks, including key modules such as GUI elements recognition and SoM prompting, plan/re-plan state machine and code execution for general mouse/keyboard control.
- Experiments on multiple, both static and dynamic, benchmarks demonstrate the proposed framework's efficacy and generalizability.

strengths:
- This work shows a robust and efficient OS agent by resorting to a general purposed language models without fine-tuning. To cope with GUI recognition difficulties (beyond general vision tasks), SoM techniques are applied to improve reliability.
- Many baseline comparisons and detailed implementation are also provided.

weaknesses:
- Discussion on safety may be helpful since the agent has control over an OS.
- Although it uses a state machine and re-planning, OSCAR lacks a self-improvement mechanism. Do you have more insights on representing states, e.g. system errors/task failures, beyond pure text language?

questions:
- What makes OSCAR outperform other baselines mostly (since many methods also have their feedback loop)?
- Will a fine-tuned LM be eventually required for the robustness in real-world scenarios beyond those benchmarks? To what degree is a user/expert required to supervise the agent's operation?
- Is plan -> execution -> error -> re-plan enough even for complex tasks or sampling-based methods such as evolutionary search (but costly and redundant) should be also considered.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**: This paper presents OSCAR, a generalist agent designed for autonomous interaction with desktop and mobile operating systems. OSCAR uses state-aware reasoning and task-driven re-planning to dynamically interact with graphical user interfaces (GUIs) through mouse and keyboard inputs. The agent processes screen observations, translates user instructions into executable Python code, and adapts to errors using a state machine framework. OSCAR’s robustness and adaptability are validated through extensive experiments on benchmarks such as OSWorld, GAIA, and AndroidWorld, where it outperforms current state-of-the-art systems. The paper highlights OSCAR’s ability to generalize across dynamic OS environments and handle real-time feedback effectively.

**Strengths**:
- All reviewers praise the thorough and diverse evaluations across dynamic OS benchmarks, including OSWorld, GAIA, and AndroidWorld. OSCAR achieves state-of-the-art performance, demonstrating its ability to adapt to various tasks and outperform existing agents like FRIDAY and UFO.
- Reviewers recognize OSCAR’s dual-grounding approach for GUI understanding, combining visual and semantic grounding, as a significant advancement. This enables the agent to interpret complex user interfaces more accurately and robustly, enhancing task execution.

**Weaknesses**:
- Safety concerns: Reviewer `TSTo` raises concerns about safety, particularly when an agent has autonomous control over operating systems. The authors admit that safety (and scalability) issues are the biggest limitations of the paper, and provide more discussion in the rebuttal. While the authors discuss virtualization and safeguards, a more thorough exploration of potential misuse and real-world deployment risks would strengthen the paper.
- Scalability concerns: Reviewer `TSTo` questions whether OSCAR’s current plan-execute-replan cycle is sufficient for highly complex tasks. Exploration-based methods like evolutionary search could provide additional robustness but were not fully considered in this work.
- Generalization to other OSs: Reviewer `Bipc` points out that adapting OSCAR to other operating systems and benchmarks requires additional effort. While the authors clarify that the modular design aids generalizability, further examples or discussions would improve confidence in OSCAR’s scalability.
- Lack of self-improvement mechanisms and error analysis discussion: Reviewer `TSTo` notes the lack of advanced self-improvement mechanisms and reviewer `Bipc` requests more discussion on error analysis. While OSCAR handles task failures well through state-driven re-planning, it does not learn from cross-task experiences or explore structural improvements beyond text-based state representation.

**Recommendation**: Based on the reviewers' feedback and discussion, the paper offers an innovative, well-executed solution for dynamic OS control tasks. While safety considerations and generalization to new environments remain areas for improvement, the authors' rebuttal addressed these concerns adequately. In my opinion, OSCAR’s strong performance, technical contributions, and experimental validation make it a valuable addition to the field, and my vote is for Accept as a poster.