Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
Authors: Saaket Agashe
Kyle Wong
Vincent Tu
Jiachen Yang
Ang Li
Xin Eric Wang
Venue: COLM 2025 (2025)
Keywords: Computer Use
GUI Agents
Multimodal Large Language Models
Planning
Grounding
Vision
TL;DR: State-of-the-art results on Computer Use using a framework of Generalist and Specialist modules.

Abstract: Computer use agents automate digital tasks by directly interacting with graphical user interfaces (GUIs) on computers and mobile devices, offering significant potential to enhance human productivity by completing an open-ended space of user queries. However, current agents face significant challenges: imprecise grounding of GUI elements, difficulties with long-horizon task planning, and performance bottlenecks from relying on single generalist models for diverse cognitive tasks. To this end, we introduce Agent S2, a novel compositional framework that delegates cognitive responsibilities across various generalist and specialist models. We propose a novel Mixture-of-Grounding technique to achieve precise GUI localization and introduce Proactive Hierarchical Planning, dynamically refining action plans at multiple temporal scales in response to evolving observations. Evaluations demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance on three prominent computer use benchmarks. Specifically, Agent S2 achieves 18.9% and 32.7% relative improvements over leading baseline agents such as Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation. Moreover, Agent S2 generalizes effectively to other operating systems and applications, surpassing previous best methods by 52.8% on WindowsAgentArena and by 16.52% on AndroidWorld relatively. Code available at https://github.com/simular-ai/Agent-S.

Published as a conference paper at COLM 2025
Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, Xin Eric Wang Simular Research
Abstract
Computer use agents automate digital tasks by directly interacting with graphical user interfaces (GUIs) on computers and mobile devices, offering significant potential to enhance human productivity by completing an open- ended space of user queries. However, current agents face significant chal- lenges: imprecise grounding of GUT elements, difficulties with long- long-horizon task planning, and performance bottlenecks from relying on single gener- alist models for diverse cognitive tasks. To this end, we introduce Agent S2, a novel compositional framework that delegates cognitive responsibil- ities across various generalist and specialist models. We propose a novel Mixture-of-Gronndin technique to achieve precise GUI localization and in- troduce Proactive Hierarchical Planning, dynamically refining action plans at multiple temporal scales in response to evolving observations. Evalua- tions demonstrate that Agent $2 establishes new state-of-the-ae (SOTA) per- formance three prominent computer use benchmarks. Specifically, Agent S2 achieves 18.9% and 32.7% relative improvements over leading baseline agents such Claude Computer Use and UI-TARS the OSWorld 15-step and 50-step evaluation. Moreover, Agent S2 generalizes effectively to other operating systems and applications, surpassing previous best methods by 52.8% on WindowsAgentAnee and by 16.52% on WorloidWorld relatively.
Code available https:/ /gi thub. com/ si simular-ai ai/geett-/AAgn-SS..
1 Introduction
Computer-use agents are autonomous AT agents that can directly interact with graphical user interfaces to complete user requests. They function as intel- ligent intermediaries between humans and their digital tools in the most intu- itive way: direct keyboard and mouse control. They can be generally used across host of applications and web- sites that humans utilize without the need for specific APIs and protocols. While there has been substantial re- cent progress in developing computer use agents (Agashe et al., 2024; Qin et al., 2025; Anthropic, 2024; OpenAI, 2025; Bonatti et al., 2024), it remains largely unsolved problem, with the best- performing agents lagging significantly behind humans (e.g., around 40% perfor- mance gap on OSWorld (Xie et al., 2024)).
Agent S2 (Ours)
36
32.6
OpenAl CUA
30
Agent S
Success Rate
Claude-3.7 Sonnet
24.6
22.7
UHTARS
22.0
20
20.6
Claude-3.5
19.7
Sonnet
15.5
15
14.9
10
Steps
Steps
Figure 1: Agent S2 achieves new SOTA results (Success Rate) on computer use tasks on both 15-step and 50-step evaluation in OSWorld.
"qqual contributions.

---

Published as a conference paper at COLM 2025
Current computer-use agents face three core limitations. First, they struggle to accurately ground textual descriptions of GUI elements to precise pixel-level coordinates. Second, they have difficulty handling long-horizon tasks, especially in the presence of background distractions, interruptions, and evolving user contexts and observations. Third, most methods rely solely on generalist models to perform diverse tasks such as planning, action generation, and grounding, leading to performance bottlenecks. While generalist models offer broad capabilities, they often underperform compared specialist models in domain- specific subtasks, ultimately constraining the overall performance of computer-uss agents.
To address these challenges, we introduce Agent S2, a compositional framework designed to delegate cognitive tasks across different generalist and specialist modules. First, Agent S2 tackles grounding bottlenecks via a novel Mixture of Grounding mechanism, where the agent reasons about subgoals and routes actions to specialized grounding experts for precise GUT localization across diverse applications. In addition, we propose a Proactive Hierarchical Planning method that dynamically adjusts and refines action plans at multiple temporal scales based on new observations. This enhances adaptability compared to passive or reactive planning methods, which either rigidly adhere to predetermined script or only adjust after encountering execution failures. Overall, Agent S2 functions as a compositional and hierarchical system, distributing responsibilities across modules specialized in high- level reasoning, low-level execution, and fine-grained grounding, avoiding the limitations of monolithic approaches that rely solely on training or fine-tuning single generalist model.
Our framework, Agent S2, achieves new state-of-the-art (SOTA) performance across multi- ple computer use benchmarks. Specifically, Agent S2 achieves 27.0% (t 18.9%)' and 34.5% (t 32.7%) on the OSWorld benchmark's (Xie al., 2024) 15-step and 50-step evaluations, respectively, highlighting its effectiveness and scalability. Moreover, Agent S2 generalizes effectively to other benchmarks, achieving new SOTA results with 29.8% (t 52.8%) accuracy on WindowsAgentArees
(Bonatti et al., 2024) and 54.3% (1 16.5%) accuracy on Android- World (Rawles et al., 2024b). Through comprehensive ablation studies, we highlight the improvements from our Mixture of Grounding strategy and the benefits of Proactive Plan- ning over conventional reactive planning methods. We also analyze how scaling compute and time steps enhances performance and provide extensive error analysis identifying current limitations and potential strategies for future improvements. Furthermore, our experiments validate that strategically com compost posing generalist and specialist models, even when each is slightly suboptimal on its own, can outperform the best monolithic models.
We summarize our contributions as follows:
1. We introduce Agent S2: a new compositional, hierarchical framework for computer use agents that effectively delegates reasoning, execution, and grounding responsibilities across various generalist and specialist modules.
2. To address key limitations in existing computer use agents, we introduce Mixture of Grounding for resolving the grounding bottleneck and Proactive Hierarchical Planning
for dynamic replanning in response to evolving observations and state changes.
3. We demonstrate that Agent S2 achieves satt--of-the-ate performance across multiple operating system benchmarks for Computer use and smartphone use tasks: OSWorld, and AndroidWorid.
4. Extensive ablation studies further demonstrate the effectiveness of the Mixture of Ground-
ing and Proactive Hierarchical Reasoning for compositional frameworks. We also show detailed thematic analysis of the emergent behaviors demonstrated by our agent with increased compute and time.
2 Background
Computer Use Tasks and Benchmarks. In computer use, agents interact with digital envi- ronments by executing desktop actions to fulfill user instructions. These tasks are inherently multimodal and can be formally described as Partially Observable Markov Decision
I represents relative increase with respect to leading baselines UI-TARS and Claude Computer Use for OSWorld, UI-TARS for AndroidWorld, and NAVI Agent for WindowsAgentArena.
2

---

Published as a conference paper at COLM 2025
Process (POMDP), defined as M = = (S, where S is the state space (current state of the desktop), O is the observation space (instructions, screenshots, accessibility trees, etc.), A is the action space (e.g., click, type, etc.), T S x A S is the state transition function, and R. : x A + [0, 1] is the reward function. For example, a user might request the agent to "change their default search engine". The agent must perceive the current screen - either through an image or an accessibility tree and execute sequence of actions to complete the given task.
Benchmarking multimodal agents for computer use has become a growing area of research. Early benchmarks (Deng et al., 2023; Rawles et al., 2023) utilized offline evaluations requir- ing agents to follow a specific action sequence for success, while recent online benchmarks use functional evaluation scripts. OSWorld (Xie et al., 2024) provides a structured envi- ronment for desktop control in Ubuntu, encompassing tasks like file management and document/ing image editing. WindowsAgentAreen. (Bonatti et al., 2024) extends these tasks to parallelizable Windows environment, and Android (Rawles et al., 2024a) includes mobile U1 navigation tasks. Related benchmarks like Spider2-V Cao et al. (2024) also use functional evaluation, focusing automating data science and engineering workflows. Other benchmarks like ScreenSpot (ByteDance, 2024) and VisualWebBench (Liu et al., 2024) focus explicitly Visual Grounding in UI environments.
Recent works have approached these computer use tasks along two primary directions: (1) Training monolithic, generalist models on UT interactions and grounding data, and (2) Composing multiple models into hierarchical frameworks and creating a pipeline of planning, execution, and grounding.
Monolithic Methods for Computer Use Agents. Monolithic methods employ a single model to handle all aspects of computer use: planning, execution, and grounding. These systems require models exhibit two critical abilities: (1) System-2 reasoning, which allows the generation of long-term plans and short-term actions, and (2) UI Grounding, which requires locating interactable elements by picking precise coordinate from screenshots or specific element from accessibility trees.
Several monolithic models post-trained on GUI tasks have emerged as native agents (Xu et al., 2024; Qin et al., 2025; Anthropic, 2024; Hong et al., 2023; Sun et al., 2025). Recent works explore strategies like Monte Carlo Tree Search (Yu et al., 2024), and learning from interactions (Su et al., 2025). AgentStore Jia et al. (2024) uses MetaAgent that prese- lects application-specife agents(s) to perform a full task/subtasi based on special tokens. Generalist approaches like Anthropic (2025) have even achieved state-of-the-art results on computer use tasks. Yet, monolithic methods have inherent drawbacks, as fine-tuning for specialization often diminishes broader capabilities (Yosinski et al., 2014; Luo et al., 2023), and assembling diverse, large-scale datasets for reasoning and visual grounding is expensive and time-intensive. Furthermore, relying on single model may not optimally address the distinct needs of planning, execution, and grounding since these tasks can benefit from different model strengths.
Hierarchical Methods for Computer Use Agents. Hierarchical methods have gained popularity as a way to overcome the limitations of monolithic approaches, by decoupling the cognitive processes of planning, execution, and grounding. In hierarchical methods, at the high-level (time step T), Manager M generates plan for instruction I, breaking it into coherent subgoals: I go, S1,-..... ......g -N. Each subgoal specifies an intermediate objective of the task. At the low-level (time step I), Worker W sequentially performs atomic actions (click, type, etc.) to iteratively complete each subgoal gi: 8i - ag, .q.,
Recent studies demonstrate hierarchical planning's effectiveness when combined with knowledge augmentation and continual learning (Agashe et al., 2024). Wu et al. (2024a) developed executor skill libraries complementing hierarchical decomposition. They per- form Reactive planning for correction, updating subgoals only a failure condition is seen. Wang & Liu (2024) uses verification with hierarchical planning. Tan et al. (2024) builds a Multi-agent framework for distributing responsibilities to multiple models. Other frameworks have explored modular approach by explicitly separating planning from
3

---

Published as a conference paper at COLM 2025
Manager
M
M
M
9
9t.
9
W
Worker
W
MoG
Mixture of Grounding
McG
Gt
G2
G3
MoG
a
and
a..5
Figure 2: The Agent S2 framework. composes generalist planning modules, Manager M and Worker W, with specialist grounding experts to complete complex, long-horizon computer use tasks. Please refer to Section 3 for detailed explanation.
visual grounding. They train models (Wu et al., 2024b; Gou et al., 2024; Yang et al., 2024) for UT element grounding and then further pair them with general-purpose language mod- els such as GPÎ-4o for planning. However, hierarchical systems also face key challenges. Workers managing both action generation and element grounding can become performance bottlenecks. Additionally, many planning methods are reactive and struggle to handle unexpected environmental changes, limiting overall robustness.
Agent S2: Compositional Grounding and Planning for Computer Use
Our Agent S2 framework, depicted in Figure 2, aims to address complex computer tasks by composing generalist hierarchical planning modules with specialist grounding modules. The key components of our framework are the Manager M, the Worker W, and the Mixture of Grounding Experts (MoG). The Manager M operates at higher semantic space, decomposing task into list of high-level subgoals gr. The worker W operates on lower semantic level, generating natural language actions to complete the topmost subgoal. Agent S2 uses a Mixture of Grounding strategy, where the Worker W routes its actions to the correct specialist module among the grounding experts {G effectively addressing the grounding bottleneck. Additionally, Agent S2 utilizes a Proactive Hierarchical Planning strategy where the Manager M updates its list of remaining subgoals after the completion of each individual subgoal to adapt to newer observations, while the Worker W routes its action to new expert after each action. Since computer use tasks require highly specialized domain knowledge about various applications and requests, Agent S2 also uses the knowledge base from Agent S (Agashe et al., 2024), featuring high-level task interaction experience, low-level subgoal interaction experience, and contextual web knowledge.
3.1 Mixture of Grounding
Operating user interfaces involves navigating wide range of applications like menus, canvases, spreadsheets, etc. The inability to precisely and robustly locate various regions of interest on a screen forms a crucial bottleneck in current computer use agents. To efficiently handle precise UI element localization, Agent S2 introduces Mixture of Grounding (MoG), which forms the Specialist part of our framework. Analogous to Lexture--ofExpeeee (Jacobs et al., 1991), the Worker W in our framework acts as gating mechanism and routes each generated action to the correct grounding expert. The grounding expert then generates the pixel-level coordinates. This allows the worker to focus on reasoning while distributing the cognitive load of grounding to the appropriate expert.
4

---

Published as a conference paper at COLM 2025
M
M
M
g1 ga
9
ga
g
9=
g
g g2 g3
9 9
ga
ga
m
W
W
W
W
W
W
Reactive Planning
Proactive Planning
Figure 3: Comparison between Reactive and Proactive Planning. Proactive planning re- evaluates and updates the remainder of the plan after every subtask, while reactive planning adheres to fixed plan and only revises it in response to subtask failures.
Formally, at each execution step, the Worker W receives a subgoal gi alongside the latest environmental observation 0,. Policy Tw generates atomic action a necessary for executing 8;, each accompanied by language descriptor specifying its target location. After deciding the atomic action At, the Worker delegates the grounding task to corresponding expert among the following modules. The selection of the grounding expert dynamically decided by the Worker module at each execution step.
Visual Grounding Expert. The visual grounding expert takes as input the current observa- tion screenshot paired with language description d of a specific point in the image and generates the precise low-level coordinates y) that represent d. This description visual grounding allows Agent S2 to rely solely on screenshots as input, eliminating the need for bulky accessibility trees and HTML. Most importantly, the visual grounding expert enables Agent S2 to act on any point on the screen rather than being restricted to selecting only high-level elements, greatly expanding the scope of possible interactions. Furthermore, the worker W can progressively refine the descriptions it provides to the Visual Grounding expert to self-correct actions over time.
Textual Grounding Expert. While visual grounding models like UGround (Gou et al., 2024) and UI-TARS (Qin et al., 2025) have shown impressive precision, a class of problems that still poses challenge is fine-grained text grounding, such as generating coordinates perfectly aligned with the edge of a word or sentence. To address this limitation, we use Optical Character Recognition (OCR), which is conventional way of locating characters in textual documents and paragraphs. In addition to the current observation screenshot o, the Textual Grounding expert also takes two phrases p1 and p2 as input. P1 and p2 are the exact word sequences from the start and end of the span of interest. The Textual Grounding expert uses OCR output the span coordinates (Xstart, Ystart), and (Xend
Structural Grounding Expert. Another category of grounding bottleneck involves locating elements in spreadsheets and tabular content. Since spreadsheet cells can be stretched and squeezed to arbitrary sizes, and translating the table can change the starting position of the rows and columns, grounding on tabular data remains a significant challenge. To overcome this limitation and ensure precise grounding in tabular UI elements, the Structural Grounding expert takes a dictionary of "cell": "value") mapping and programmatically updates the content of the corresponding cells. The structural grounding expert can take multiple cells, even entire rows, columns, or tables, as input and update them all at once, allowing both reliable and faster grounding for structured data.
3.2 Proactive Hierarchical Planning
Computer use tasks can often span long horizons involving multiple apps, screens, and long series of observations. The initial state often contains partial information needed to address the user's 's query. Moreover, background apps and pop-ups introduce significant noise, and the susceptibility of Multimodal LLMs to UI noise (Ma et al., 2024) further complicates the problem. Therefore, Agent S2 incorporates Proactive Hierarchical Planning,
5

---

Published as a conference paper at COLM 2025
Framework
Model
15-step
50-step
UI-TARS (Qin et al., 2025)
UI-TARS-72B-STT
18.7
18.8
UI-TARS (Qin et al., 2025)
UI-TARS-72B-DPO
22.7
24.6
OpenAl CUA (OpenAl, 2025)
OpenAl Operator
19.7
32.6
Aria-UI (Yang et al., 2024)
GPT-4o
15.2
Aguvis-72B (Xu et al., 2024)
GPT-4o
17.0
Agent S (Agashe et al., 2024)
GPT-4o
20.6
Agent S2 (ours)
GPT--o
21.1
26.6
Claude Computer Use (Anthropic, 2024)
Claude-3.5-Sonnet new
14.9
22.0
Agent (Agashe al., 2024)
Claude-3.5-Sonme
20.5
Agent S2 (ours)
Claude-3.-SSonne new
24.5
33.7
Claude Computer Use (Anthropic, 2025)
Mlaude-37--Sonmtt
15.5
26.0
Agent S2 (ours)
Claude-3.7-Sonme
27.0
34.5
Table 1: Success Rate (%) on OSWorld across different agent frameworks and foundation
models. Agent S2 achieves new state-of-the-ar results on OSWorld for both 15- and 50-step evaluations with multiple foundation models, such as (laude-3.5-Sonnet, Claude-3.7-Sonnee and GPT-4o. All agents use only screenshots as input, except Agent S, which uses both the accessibility tree and screenshots.
which replans and reasons at both levels of hierarchy (Manager and Worker) over different temporal scales. Unlike reactive planning approaches, which only update their plans after failure (see Figure 3), proactive planning allows Agent S2 to update its plan after completing every subgoal, effectively adapting to evolving observations and recontextualizing the user query while maintaining context from previous subgoals to reduce susceptibility to noise.
At each high-level time step T, given a user instruction and the current observation Op, the Manager M generates plan, which is sequence of subgoals {g 82. .3....r8i. ..). The Worker W then takes the first subgoal g1 8í and begins executing it. To do this, at each low-level time step f, the Worker follows its policy Zw to pick actions a, and routes the action to the appropriate grounding expert explained in Section 3.1. After several low-level steps, the Worker concludes the subgoal gi ag, a0..,,..,,4 with either SUCCESS or FAILURE, returning control to the Manager. T'hen, the Manager takes the prior subgoals 82.88'2 ., J, the latest observation o,, and the original instruction I as input. The context from the previous subgoals allows the Manager to bootstrap and connect its thinking to the original task while allowing it to incorporate the new observations. Based on these prior subgoals and the latest observations, generates a new set of subgoals g". g.) The first subgoal from this updated list becomes the next Worker objective, 82 8 This process continues required, with Manager refining subgoals until instruction resolved.
4 Experiments
4.1 Experimental Setup
Benchmarks. We run our main experiments on the OSWorld (Xie et al., 2024) benchmark consisting of 369 real-world computer use tasks across the following categories: OS, Office (LibreOffice Calc, Impress, Writer), Daily (Chrome, VLC Player, Thunderbird), Professional (VS Code and GIMP) and Workflow (tasks involving multiple apps). We further evaluate Agent S2 on WindowsAgentArne Windows operating system. To generalize beyond computer use, we test the Android- World (Rawles et al., 2024b) benchmark with 116 Smartphone use tasks across 20 real-world Android applications. For ablation studies, utilize subset of OSWorld, consisting of 65 examples sampled from the OSWorld environment, stratified by categories.
(Bonatti al., 2024) with 154 tasks executed on the
Baselines. Across each benchmark, mainly compare with screenshot-input baselines. For OSWorld, we compare our method with OpenAl CUA/ / Operator (OpenAI, 2025), Claude Computer Use (CCU) with 3.5-Sonnet and 3.7-Sonnet (Anthropic, 2024), and UT-

---

Published as a conference paper at COLM 2025
Model
OS
Daily
Office
Professional
Workflow
Overall
GPT-40
50.00
30.70
18.97
51.02
14.93
26.62
Claude-3.5-Sonne (new)
58.33
48.44
29.06
51.02
13.46
33.71
Claude-3.7-Sonnet
50.00
49.73
25.64
57.14
18.21
34.47
Table 2: Categorized Success Rate (%) of Agent S2 on the OSWorld 50-step evaluation. We
report results with various MLLMs as Manager and Worker.
Method
OS
Daily
Office
Professional
Workflow
Overall
Aria-UI (GPT-4o)
25.00
25.71
9.58
20.41
8.55
15.15
Agent S2 (Qwen2255V1--222-hnntrcu)
50.00
29.42
5.98
42.86
4.49
18.29
Agent (GPT-40)
45.84
27.06
13.00
36.73
10.53
20.58
Agent $2 (GPT-4o)
37.50
25.57
48.98
10.89
21.12
Agent (Claude-3.5-Sonne) (new))
45.84
35.82
15.52
57.14
5,00
24.50
Agent S2 (Claude-3.-Sonmee)
41.67
35.62
20.51
65.31
5.94
27.04
Table 3: Categorized Success Rate (%) on the OSWorld 15-step evaluation across various
agent configurations.
"Uses Atree addition screenshot input.
TARS-72B-DPO (Qin et al., 2025). To standardize the evaluation and test for scalability, we show our results both 15-step and 50-step evaluation. For WindowsAgentArena, we compare with the Navi Agent (Bonatti et al., 2024) Omniparser (Lu et al., 2024). Notably, this result uses both the accessibility tree and screenshot as input, while we only require the screenshot. Lastly, for AndroidWorld we compare with UI-TARS-72B-FTT (Qin et al., 2025) and GPT-4o + Aria-Ul (Yang et al., 2024).
Implementation Details. For the Mixture of Grounding experts, Agent S2 uses UI-TARS- 72B-DPO as the visual grounding expert, Tesseract OCR (OCR, 2025) as the textual ground- ing expert, and Universal Network Objects (UNO) (Unotools, 2025) interface the structural grounding expert. The backbone models evaluated include Claude-3.7-Sonnee Claude- 3.5-Sonnet (new), and GPT-4o (Specifically, versions Eessonnet-20241000 and gpt-40-2024-08 2024--0000)) The best results on Computer use tasks are from evaluations with Sruude-3.7-Soneed with its thinking mode enabled to fully leverage the reasoning capabilities in long-horizon tasks. For evaluations on OSWorld and Win- dowsAgentArena, test tasks that reach the allocated step limit before the agent signals task completion are considered failures. For AndroidWorld, we use Agent S2 in a worker-only setting due to shorter horizon tasks. In all environments, we use screenshots and action history as input.
claude-
4.2 Main Results
OSWorld. Table 1 presents Agent S2's performance on the OSWorld Benchmark. Agent $2 with Claude-3.7-Sonnel or 3laude-3.5-Sonnet (new) achieves new SOTA,, outperforming all other results on both 13-step and 50-step evaluations. Notably, Agent S2 with Claude- 3.5-Sonne (new) relatively outperforms Claude Computer Use with Claude-3.7-Sonnet by 58.1% on 15-step and 29.6% on 50-step evaluations, illustrating the advantages of modular, hierarchical frameworks computer use. Table 2 provides a further detailed breakdown of performance across OS- World's categories for 50-step evaluation, where Agent S2 demonstrates high effectiveness on OS, Daily, and Professional tasks across all backbone models. It also delivers competitive performance on Office tasks, a historically et al., 2025). Surprisingly, Agent S2 with challenging category (Agashe et al., 2024; Su Sonnet variant in the Office category. Closer analysis indicates that it relies on Textual and Structural Grounding experts almost twice as often, highlighting the benefits of Mixture of Grounding. Figure 4 shows an example of Agent S2 in action, where it resorts to alternate grounding expert and then replans based step performance of Agent with various underlying based models and against previous baselines demonstrating consistently better performance.
monolithic generalist modules in long-horizon tasks like
(new) surpasses the 3laude-3.7-
observation. Table 3 compares the 15-
7

---

Published as a conference paper at COLM 2025
Method
Model
Office
Web
Win. System Coding
Media & Video
Win. Utils
Overall
S
GPT-40
0.0
13.3
45.8
29.2
19.1
22.2
18.2
NAVI
GPT-4V-1006
0.0
27.3
33.3
27.3
30.3
8.3
19.5
Agent S2 (Ours)
Claude-3.7-Sonnet
7.0
16.4
54.2
62.3
28.6
33.3
29.8
Table 4: Success Rate (%) on the WindowsAgentAAenna test set (within 15 steps). Note that both Agent S and NAVT use screenshots and accessibility trees, while our agent only takes screenshots as the input. Agent S2 sets new SOTA on WindowsAgeetAAren..
Subgoal: Select Last Paragraph
Subgoal: Select Last Paragraph
Subgoal: Apply Formatting
Step 1: click(("Th last paragraph..")
Step 2:
select spant select_span( Eaah student..". "The essay writing..")
Step 3: elick(" "F Format in menu bar.." )
Rouled to Visual Grounding Expert
Routed to Textual Grounding
Routed to Visual Grounding Expert
Figure 4: Agent 52 attempts to use Visual Grounding Expert to select a paragraph, then self-corrects and uses Textual Grounding Expert for span selection. After completing the subgoal, it replans from the new state and starts working on new subgoal.
WindowsAgentArena. Agent S2's success also carries over to the WindowsAgeetAAeena benchmark, which features Computer use tasks in the Windows Operating system. Table shows that Agent S2 significantly outperforms the previous best agent NAVI (Bonatti et al., 2024) by 52.8%. The performance improvement is consistent across 4/6 categories. Notably, it is able to perform well on Windows-specific tasks, demonstrating generalization across operating systems.
4.3 Ablation Study
Mixture of Grounding improves subtask completion rate and thus the overall success.
Figure 5 illustrates the performance improvement provided by the Mixture of Grounding strategy, especially when provided more steps. Specifically, MoG increases success rate from 27.69% to 30.77% at shorter horizons (15 steps) and, more prominently, from 33.85% to 38.46% at longer horizons (50 steps). To further study the benefit of individual experts, we extract subset of OSWorld examples where the agent routes actions to the Textual or Structural Grounding expert. We then re-evaluate those examples without the corresponding expert to measure their impact on successful subtask completion. When removing the textual grounding expert, the subtask success rate drops from 70.6% to 65.2%, and when removing the structural grounding expert, the subtask success rate drops from 73.7% to 69.4%.
Visual Grounding forms the foundation of screenshot-only agents and is used in every single example task by our framework. We conduct a 15-step evaluation of the various grounding models as our Visual Grounding expert, as reported in Figure 6. In general, incorporating specialist model that has better UI grounding capabilities into broader framework yields better performance on computer use tasks. More importantly, we observe that smaller specialist models, such as UI-TARS-7B-DPO and (GGround-i--79 can outperform large generalist models like Claude-3.7-Sonnet when employed within modular framework that balances cognitive load effectively.
Proactive planning enables self-correction and contextualization with new observa- tions. Our ablation study in Figure also demonstrates the efficacy of proactive planning, revealing performance improvement of +4.62% at 15 steps and +6.15% at 50 steps com- pared to reactive planning. This increase in success rate confirms that proactive hierarchical planning substantially enhances task completion by adapting to evolving observations.
8

---

Published as a conference paper at COLM 2025
40
Agent S2
38.46
U-TARRS-78-DPO
30.77%
Success Rate (%6
MoG
w/o PHP
UI-TARS-78-DPO
29.23%
33.85
32.31
UGround-V1-78
24.61%
37
30
27.69
Caude-3.7-Somme
24.61%
26
30
35
25
Success Rate (%)
15
50
Max Steps Allowed
Figure 6: 15-step performance of Agent S2 with different visual grounding models. Small open-sourced models can outperform large close-sourced reasoning models.
Figure 5: Ablation study on Mixture Grounding (MoG) and Proactive Hierarchi- cal Planning (PHP).
Adaptive Navigation
Planning
34.5%
41.0%
Interaction
Adaptive
31.0%
Grounding
20.5%
13.8%
Backwards Correction
Infeasible
10.3% 10.3%
17.9%
10.3%
Navigation
Other
Complexity
Interaction
Figure 7: Contributing factors of 15-step to 50-step scaling success.
Figure Failure categories on subset of the OSWorld benchmark.
Similar to the improvement brought about by the Mixture of Grounding strategy, Proactive Hierarchical Planning is also more beneficial with more time steps.
Agent S2 scales with more compute and steps. To gain insight into the specific behaviors that enable Agent S2 to scale at test-time, extract the subset of 29 OSWorld examples where Agent S2 fails on a 15-step evaluation but succeeds on a 50-step evaluation. As shown in Figure 7, our analysis reveals four primary behaviors that improve Agent S2's performance with additional steps. First, we observe that the most common behaviors are (1) Adaptive Navigation, where the agent explores multiple approaches to find certain element or navigate to certain page, and (2) Adaptive Interaction, where the agent interacts with the same element or page in different ways. These two behaviors show that the modular Agent S2 framework self-corrects and explores alternative approaches during inference, owing to its Proactive Hierarchical Planning abilities. Furthermore, also intersperses various grounding experts to refine its interactions with the UT, as demonstrated in Figure 4. Other major contributing factors include: (3) Backwaard Correction, where the agent corrects small errors missing interactions from previously completed subgoals while solving the current subgoal, which is promoted by the contextualizaaton of user queries with new observations during Proactive Planning; and (4) Task Complexity, which includes tasks where perfectly optimal agent or human would require more than 15 steps to complete. This analysis further solidifies the role of Proactive Hierarchical Planning and Mixture of Grounding in enabling our agent to improve with more time steps.
4.4 Error Analysis
Figure 8 displays the frequency of each failure type and offers insight into the current bottlenecks Agent S2. We conduct error analysis on a subset of 65 samples from OSWorld through qualitative evaluation. For each example, cause of failure. We observe the following failure modes: (1) Planning failures, where the manager formulates inadequate plan, typically due to inaccurate /noss subtask
report the theme of the first major

---

Published as a conference paper at COLM 2025
information misalignment with the task requirements. (2) Grounding failures, where the grounding expert produces inaccurate coordinates for the provided language description. (3) Interaction failures, where the worker is unable to successfully manipulate an element, reflecting a lack of domain knowledge on GUT interactions. (4) Navigation fallures, where the worker struggles to find certain element, suggesting deficiencies in layout understanding and navigation. (5) Infeasible tasks, for which the agent is unable to predict this infeasibility. Although previous works (Agashe et al., 2024; Xie et al., 2024) report grounding as main cause of failure, we observe that Agent S2 maintains relatively lower rate of grounding errors, while planning failures are now the most frequent. Furthermore, interaction and navigation failures are less common, which strengthens the findings from Figure 7 that Agent S2 adapts over longer horizons through fest-time exploration.
4.5 Generalization to Smartphone Use
Method
Base Model
SR (%)
Owen2- VL-2B InfiGUlAAent (Liu al., 2025)
Owen2-VL.
OPT-4Turoo + Wodroid (Rawles al., 2024b)
GPT-4 Turbo
30.6
GPT-4o + Ponder&Press (Wang al., 2025)
GPT-4o
34.5
GPT-4o + UGround (Gou et al., 2024)
GPT-4o
44.0
CPT + Aria UI (Yang et al., 2024)
CPT
44.8
UI-TARS-72B-S5 (Qin et al., 2025)
UI-TARS-722
46.6
Agent $2 (Ours)
GPT-4o
54.3
Table 5: Success rate (SR) comparison on the AndroidWorld benchmark for smartphone use.
We also perform a generalization study of Agent 52 on AndroidWorld (Rawles et al., 2024a) for smartphone use. Table 5 shows that Agent S2 outperforms the previous state-of-the-art method by a large margin of 16.5% relatively, which validates the strong generalizability and modularity of Agent S2.
Conclusion
We introduce Agent S2, compositional framework integrating generalist and specialist models for high-level reasoning, low-level execution, and detailed grounding. Our Mixture of Grounding approach enlists a team of experts for precise grounding across diverse appli- cations, while Proactive Hierarchical Planning refines plans and contextualizes observations based on user instructions. We show that Agent S2 achieves state-of-the performance two computer benchmarks and one smartphone use benchmark. We also perform ablation studies and error analysis to highlight the roles of each component in our system.
Acknowledgments
We extend our sincere thanks to Tianbao Xie, Yujia Qin, Shihao Liang, and Zhiyong Wu for their engaging discussions on computer use agents.
References
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent S: open agentic framework that uses computers like human. CoRR, abs/2410.08164 2024. doi: 10.48550/ 20555//AAAXX.2210.011666. URL https: //doi. org/10. 48550/arXiv. 2410. 08164.
Anthropic. Claude computer use.
2024.
URL
1--5mooellssaadd-ommptee--u
Anthropic. Claude's extended thinking. 2025. URL
seiilleeextende--hhinke
10

---

Published as a conference paper at COLM 2025
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal OS agents at scale. CoRR, abs/ 240000409,,, 2024. doi: 1044850/ARRXV.24099000 48550/arXiv. 2409. 08264.
URL https://doi.org/
ByteDance. Screenspot benchmark. 2024. URL screenspot.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida I. Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, and Tao Yu. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Anmual Confer- ence Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, De- cember 10 - 15, 2024, 2024. URL http: /ttper/apppppsaaaaaea nips. cc/paperppppppccccppcc. aaappprrfllsssllaaaaaaaaaaasesse www.fl//lpppeeeeeee pttpppappe2222 hatee//////////h//////// 26F71567cd33444 htt.d//61000aaaee6eeeee----eeeee
and and_Bhenhaarks Track. html.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, Decem- ber 10 16, 2023, 2023. URL http::/ nips. cc/paper pappaprrfilleeee wwwes//aperrrrreeeeeeepereaaaeee paper 222///2222//////// 5950bf290a1 57 570 bf98882128 Abtrratttttattttaeeeeeeeeeeeeeee and Benddheehhaarss html.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. CoRR, abs/2410.05244 2024. doi: 10.48550 ARXIV2440000244. URL https://doi org/10. 48550/arXiv 2410. 05243..
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for GUI agents. CoRR, abs/ fbbs/2311.0814 2023. doi: 10..8550/AAA /A////2XXX..20090901 URL httpps://dnt httss:///dooooo//..// wtt.g//...8550//// 48550/ arr/v...vvvvvv/v. . 2312. 08914.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey F. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):99-87, 1991. doi: 10.162/NEEC..1991.31..
URL https:///ooi o 1162/neco. 1991.3.1.7 3.1.79.
Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, and Zhiyong Wu. Agentstore: Scalable integration of heterogeneous agents as specialized generalist computer assistant. CoRR, abs 2410.18603, 2024. doi: 10.48550/ ARXIV.2410.18603. URL https: wtt..//..00........0//////// 2410. 18603.
48550/arXiv.
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal lime evolved in web page understanding and grounding? CoRR, abs abbs/2404.05955, 2024. doi: 10.48550 A5550///ARXI..244405555.5. URL https://doi. org/10. 48550/ atv. 2404. 05955.
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist GUI agent with native reasoning and reflection. CoRR, abs/ 0tt//2551.045555 2025. doi: 10.48550/ ARXIV:2501.0457 URL http::: www..o//r.....5//rraaaa.e. 48550/arXiv 2501. 04575.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based GUI agent. CoRR, abs/ 2408.00203, 2024. doi: (0.48550/ARXXI222220 000003 URL
https://doi. org/10. 48550/ Xav./v...vvvvv ww0...........00
11

---

Published as a conference paper at COLM 2025
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. CoRR, abs 2ww.0000077200 2023. doi: 020550//RXV..2300000000 URL https://doi.orgg 48550/arXiv.. 2308.08747.
Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. Caution for the environment: Multimodal agents are susceptible environmental distractions. CoRR, abs
2024. doi: 0048850/ARXIV.2400002222 URL https:
//doi.org/1e 48550/////////ii 2408. 02544.
Tesseract OCR. Tesseract open source ocr engine. 2025. URL https://gi http:///gttttu.... com/
OpenAL. Operator: 2025. URL https:
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. UI-TARS: pioneering automated GUI interaction with native agents. CoRR, abs abs 22111122111222....11 2025. doi: 10.48550/ ARXIV.2501. 12326. URL https: //doi. org/10. 405110..445/8888 48550/arXiv 12326.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P. Lillicrap. An- droid in the wild: A large-scale dataset for android device control. CoRR, abs ab002.07.,, 2023. doi: 10.48550/ 00.4550/ARXIVV297.10000 ARXIV22307.10088. URL https: //doi. http://do.ooog///////oo 01.11...855500 208850/arXiv.2300 10088.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E. Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy P. Lillicap,, and Oriana Riva. An- droidworld: A dynamic benchmarking environment for autonomous agents. CoRR, abs/2405.14577 14573, 2024a. doi: 10.48550/ ARXTV.2405.1457. URL https:///oi.oor//1... 48550/arXiv. 2405.144210 14573.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E. Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy P. Lillicrap, and Oriana Riva. An- droidworld: A dynamic benchmarking environment for autonomous agents. CoRR, abs/2400.14577, 2024b. doi: 10.48550/ AD055//AXXX..2905455553 URL https://doi.org// 48550/arXiv. 2405. 14573.
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Arik. Learn- by-interact: A data-centric framework for self-adaptive agents in realistic environments. CoRR, / 10893, 2025. doi: 10.48550/ 448500/ARXTV.5001111 ARXTV.2501.1089 URL https://doi..or org/10. 48550/arXiv. 2501. 10893.
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. Os-genesis: Automating GUT agent trajectory construction via reverse task synthesis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 5555-5579. Association for Computational Linguistics, 2025. URL https://aclantiolog...rr////////
1olg-long... 277/
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering foundation agents towards general computer control, 2024. URL https::/arxxv.org//aaae
03186.
12

---

Published as a conference paper at COLM 2025
Unotools. Unotools 0.3.3. 2025. URL /ttttt/ss//ppppooo/////tt/tttttt
Xiaoqiang Wang and Bang Liu. OSCAR: operating system control via state-aware reasoning and re-planning. CoRR, abs / 2410.18963, 2024. doi: 10.48550/ AA00///21XX211018666. URL https://doi org/10. 48550/ 2410. 18963.
Yiqin Wang, Haoji Zhang, Jingqi Tian, and Yansong Tang. Ponder & press: Advancing visual GUT agent towards general computer control. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 1461-1473. Association for Computational Linguistics, 2025. URL https: https://aclantholggg/ 0025. findings-acl. 76/.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. CoRR, abs /200000007 2024a. doi: 10.48550/ ARXIV.2402.027755 URL
https://doi.on wwwp://...///.......2..... www.rriiwiii...7.2....
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: A foundation action model for generalist GUI agents. CoRR, abs 2210..2312,, 2024b. doi: 10.48550/ ARXIV.2410.232118 URL https: http://doo.ooog//// arvv50/ or850//.rgg//..48//rkkkkk 2410. 23218.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environ- ments. CoRR, abs/2404.07972, 2024. doi: 10.48550/ ARXIV.2404.0772. URL https: //doi.org/10 /ttp//00.8555//rxivv 2404. 07972.
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous GUT interaction. CoRR, abs/ 2442.0444, 2024. doi: 00.4550/ARRXIV244222 //doi.org//1 418.44855///aiiv 2412.04454.
URL https:
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for GUT instructions. CoRR, abs/ 2et./2222..16666 2024. doi: 01.45550/ARXIV2222
URL ttttps://dof: http://ooooooo......// 48550/8rXiv a.taaaiivv...1... 2412. 16256.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27: Anmual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3320-3328, 2014. URL https: pttpps:://pooeeiiiggss... neurips. cc/paper
Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. Exact: Teaching Al agents to explore with reflective-mets and exploratory learning. CoRR, 22100.20022 2024. doi: 10.48550 A200022412211000555 URL https: //doi. www.orgg//.../. 48555//ar555/// 2410. 2410 02052.
13

---

Published as a conference paper at COLM 2025
A
Domain Specific Results on OSWorld
Model
OS
Gimp
Code
TB
Writer
Calc
Impress
Chrome
VLC
Multiapps
Overall
Claude-3.5
58.33
38.46
65.22
60.00
43.48
25.53
25.53
41.19
57.87
13:46
33.71
Claude-3.-SSonne
50.00
50.00
65.22
73.33
34.77
25.33
21.28
41.19
51.99
18.21
34.47
GPT-4o 50.00
42.31
60.87
33.33
34.77
10.64
19.57
28.15
35.29
14.93
26.62
Table 6: Success rate (%) of Agent S2 on 50 step evaluation in OSWorld, divided by domains: OS, GIMP, VS Code, Thunderbird, LibreOffice Writer, LibreOffice Calc, LibreOffice Impress, Chrome, VLC, and Multiapps. We also report performance using different models the Manager and Worker backbones.
Method
OS
Gimp
Code
TB
Writer
Cale
Impress
Chrome
VLC
Multiapps
Overall
Aria-UI (GPT-4o)
25.00
19.23
21.74
26.67
8,70
4.26
15.32
23.80
30.06
8.55
15.15
Agent (GPT-44)
23.08
2.13
15.34
20.58
Agent 52 (GPT-4o)
37,50
50.00
47.83
26.67
21.73
8.51
10.64
23.80
29.41
10.89
21.12
Agent S2 (Claude-3.-SSomnee
45.84
50.00
40.00
30.42
17.39
6.38
36.84
29.41
24.50
Agent S2 (Claude-3.
41.67
61.54
69.57
33.33
34.77
19.15
14.89
34.67
40.22
5.94
27.04
Table 7: Success rate (%) of Agent S2 and other baselines 15 step evaluation in OSWorld, divided by domains. We also report performance using different models as the Manager and Worker backbones.
B App Specific Results on WindowsAgentArena
Chrome
Edge
Code
Notepad
Lib.Calc Settings
Win.Calc Clock
Paint
File
Writer
VLC
Overall
17.08
15.38
62.50
50.00
4.17
100.00
0.00
50.00
33.33
42.11
10.53
28.57
29.81
Table 8: Success rate (%) of Agent $2 with Claude-3.7-Some on WindowsAgentArrens divided by apps: Chrome, Microsoft Edge, VS Code, Notepad, LibreOffice Calc, Settings, Windows Calc, Clock, Microsoft Paint, File Explorer, LibreOffice Writer, VLC Player.
C Agent S2 Action Space
To make it easier for the agent to interact with the environment, create structured action space using a function-calling inspired interface that streamlines the action selection and parameter specification process. Table 9 summarizes each action type, along with their specific parameters and a brief description of their usage.
Agent Action
Action Details
Description
Arguments
click
Click on an element.
element description, num clicks, bution hype, hold keys element._dsccippton description, text, overwrite, enter element deneentdescription, clicks, shift keys
type
Type text into an element.
scroll
Scroll within an element.
hotkey
Press a hotkey combo.
hold prrsss
press
Hold keys and press others.
kold keys, press keys
drag and drop
Drag and drop between elements.
element element description element description 2, hold koys text
save_to_ knowleege
Saves data to per-task memory.
Switch another app.
app name
highlight text text_span
Highlights text span.
starting phrase, ending phrase
set cell values
Sets tabular cell values.
cell values, app name, sheet name time
wait
Wait some time.
done
Mark subtask success.
None
fail
Mark subtask failure.
None
Table 9: Agent $2 action space, descriptions, and arguments.
14

---

Published as a conference paper at COLM 2025
D Case Studies on OSWorld
We present supplementary examples on OSWorld to demonstrate the efficacy of Proactive Planning and Mixture of Grounding. Some examples have less focal portions of the trajectory omitted in order to better highlight key behaviors of Agent S2. Figure 9 and Figure 10 showcase grounding experts action, while Figure 11 displays Proactive Planning.
Tiipe last porograph oseating will eapponallle discuaion ensire paragraph.
With the fost selected. apply stciletleoogh Bensatinng using one these mathide
las to Format meon - Teet Strborhmugh
UUe kistoard abertrot Chel-D spen Chararter dialeg.
then wdec the "Pont Effecta' tub and check the "Sinloetherrgg" sprion
Step Bebcc the lest pangrraph agrnt.elirt elurkiTte last poragrapt starting with 'Each stubict will reapsnsibl Ge dacusien leuding clase.." 8, "ed"")
Step Belact the See parsgnghh speut hight out.appn: bo rosprralll for one disezssion leading in clams during the semestee.' The eaay writing and prsearting comprooee 10% of firal grade.)
student will
Look for ehilerhrough button the fomoating
tolher visiibed
applidd
Plan 3: cuu ace that the iast parngruph has already bozn selected dighhighted seangt), praceed dineclly to applying wnristhhough formarting.
"Apply sleikechavugh bernsilinng" 1 using theae methoda::
Shelkchreogh
kestoard
open disles.
mixet
*Stniletheergg" cptlon
didl dropdows "adt")
Hight dirk an the soloctd tost and chosse "Chorarice.." from the cantooi mona, then go to "Funt Biiscte teb and chock "Stribettrsight
L.brOificee
"Clink clarwhee=** the darumost deadect the afte the stribathrough has been applisd.
Click "Sinibedrrcugh"
www..iiiAAttreeelnnng...o optios in the test submanu', "ad""
bolk
applying ctribchrrogh hematting hare eusoerhilly tompkeed The last paragraph now abona ariasthmugh tematting (test with line through 20)
The only remsining stop is
apons desclett the striketluoigg firastting.
Step Deseleet space decument sgrrtt ama. away from weleted 1, "aa"")
Figure 9: Agent S2 demonstrates textual grounding. The task instruction is: think the last paragraph redundant so want add strike-through on words in the last paragraph. Can you do this for me? Agent S2 intelligently utilizes the textual grounding expert to perform a precise word-level selection of text.
15

---

Published as a conference paper at COLM 2025
Plan
Creane caluzn and citle "radee in call 31 the columo header
Weak
and fill handll aquare letteun-iigit corner dnen copy
weeks
salnseg7D)
Frafit" udded,
oppeoprintely detailod, keep
"Vlater prafft firmala**: Ealer lbe Genula =B5 eill colalalatt grott Sur Woek
"*Oogy fommila all rowett
Step agentiaci valuox""D2 C27, "adifio", "Sheett")
Byg
Click end tins the fill Lanalk: fomall square
betton eight cornee call Dio doan call Dtt the
focmolaa
column, addedd the "Prufie" header, entored the fire proftt fecmull,, IIl aaaiaue with the remsining stop, The fulurs enitavii commett and appespetanaly detalle, en hoop ix:
"CCoay thoug
fomall square
Step Sereel agent.ling The betbom-right D2; DEI Proft calumet)
Step Cgent.click!eell contuining 11517.2. Sad")
Step
apwtt mase 50500111)
Step cell values
samtieeeed valuna C28--2 wisse wBL-C1 "DS" 0886-00 DD mBD. "p8
"aBs-s; DDS =Bocos; Dair WBBCC.... "Dir -111-11 "affloe" Niheee"
Figure 10: Agent S2 demonstrates structural grounding. The task instruction is: Add new columm named "Profit" and calculate the profit for each week by subtracting "COGS" from "Sales" in that column. Agent 52 adaptively interacts with the "Profit" column and chooses to use the structural grounding expert to perform cell-level operations.
16

---

Published as a conference paper at COLM 2025
Plan I:
Oyen the "dettings" cuplicatase
the Sectings windoo losk for and click on "Prwer sinduu
Power vcting ponel,
opticn
Click un lee lozale undled mosi "Dii asceu wheu insetive tum OF?
(tep agent lop-3cl cocner
Step Type "Settngg' aprittyre
"eslings orcrnsile-Tou
Plan
aleealy
"Din
insrtion option
5. Elick on the toggl extrch noxt "Dire screnn when insetive turn OFF
Step 3: Powee optass opeann Te0")
donn Fowez Powce willings yead", Fala)
Plan R-Adher comining the cureett view, don't spectlic "Dim srnwer when traeriee oprion with nour name the Power acttings. Hare dhe reviaed plan:
L Click on "Doplass" in the ieft siddher of the Settings window. Look for any option minned scroes dimming actomatio irightneee adjuctment
If found Duplays, check if them might similar with diffceent
Step dicpe"Daspppp sideher window", "adk
Plan The Dipplays porel include oprions for dimming when inadiee, Harr's the revised plan:
Click ou Powerr" in the sodchar rotura Power suttinge
lod. for
"Fower Opcions" section. Pay special attuntion any option mumiion "dinming." "trighrnwes"" "alank" The "iin serocu whes inscisee setting night be umno difioeetly
Step 4 Click the Power optict qocckclck option the leit adobar of the Pectings window"" "lot")
Lt setiing related to dimming the seroen when inacive Gound, click du bogals wiitch tura OFF
Plan Adlee commnnig
options, Entiie appsare
Leok the "Smmen Blankc option inder Power Seving Opesone, Click the drspdoonn
sebec "Nevee", the
longest time perkol option N''eser lis's availalle
2. thers are any udditionll gower settingg that apoeas below, armil down
other scmen dirrming oetions
dianinng uptica are 'Beeeu oostads daaming
Step drogdown menn cledd
Step longret sption
setting liiey
behavior this
option"
Plan 4: con Powce octings alwody adjeated "Semen Ilank" 12 minutes Aftee exrmining the aviiatle vctlings.
minates
1. Check the Seneen Blank drspdcers har a "Neeve sprion: Click on the droydonn mon next to "Seecu Blank" upain and "Nevee avniable
the
Senil down check the additional dinming below the visikie
Step Scrall Etrough the drspown sendi Sireen Blank dnsplown nunu than aloming time optisns"
Slep 16. Click the Nover oplion anm.cbok" NackeNeeve optors buttten despolon monu". "ket""
Figure 11: Agent S2 demonstrates replanning. The task instruction is: Could you set the 'Dim screen when inactive' to off in settings? Even though the 'Dim screen when inactive' option is not found in verbatim, Agent S2 proactively replans and reasons to find the correct setting.
17

---

Published as a conference paper at COLM 2025
E
Case Studies on WindowsAgentArena
We present an example on WindowsAgentArenna for a qualitative analysis in Figure 12.
dick deaktig and click "Now" Then click "Crate
Sharoou".
la the "Cestee Shorical" visunl, click "Bewsee.
"his PC: "Doumeens"" foler, solect
Type "Prjectt create
Step 1. Right dids Deakluy crnt.diccellgghtt thick empey area on the desttop", 1, "righ()
Step Click (he "Bicn" upplan agent apentchhk)) chell New option the contret menu. 1,, 'et")
Navigate PC,, "Documezis"
"Prjjctt"
a Type "Pojecta Sharouu to create now abseicur.
Step Sheetres sgent.diic/"Tee splion muzu drapdowrz',
Plan Navigato "This PC"
Click the "Deenmente" falber.
"Decumente',
Type "Pojects crenic
Step Open Settings Tgeni,dick""Tee Browae butten masigite füdde currect faldac", "Jaf
Step Click Decumente faldee tidrr Decuments fibdee estry in the browoer dinbig list., 1.'d" l.1x"")
"Pojects" falder
Than,
Selot the tost Gold under "Oyp the beation of the Rens""
3. Type Prnjecte Shomout and click "Fnichh"
Step di Open the Projecta Seider ageot.diicki"Tee Projoct nider lis with yullow fulder ioso". 1, "ladk"
Step Click OX After Selecting "Pojjett" apei elokr Berwoo for Files Faliaes dialeg", 1, 168")
"Prjects- "Shrriou" suem-type The text fiild that curmenlly containe "Porecte" "Pogjett Shortnt", Troae
Figure 12: Agent S2 on the WindowsAgentArena environment. The task instruction is: Create shortcut the Desktop for the folder named "Projects" that is located in the Documents folder. Name the shortcut "Projects Shortcut". Through consistent replanning, Agent $2 is able to dynamically update its plan and revise its current subtask in more detail.
18

---

Published as a conference paper at COLM 2025
F
Case Studies on AndroidWorld
We present an example on Wodro for qualitative analysis in Figure 13.
nictur
Hap heres
Call yoart feoding contere
Q
IS
M
@
+
Step I: apon (Canttttt Contacta")
Step 2: clicet "Cuntacta tab es the bethon right of the serwer")
Step & Clack"Craate Create new contact at top the contarte het.")
Stop 4: Fiin Name Eoldd"" Throe")
Cruaie contact
Create porocc
Croase contact
Tuaal
Collac
q
Y
S
H
L
d
B
TOS
103
Step Last Name Beid?"
Step Pree Suidd"
Step 7 Labe feeld showingg Mobile with dropdrare))
Step elicss"WWok" Aropdoom meno""
"Taylor")
*206 -335 30030")
Figure 13: Agent S2 the AndroidWorld mobile environment. It utilizes open, touch, and type interactions to complete the instruction "Go to the new contact screen and enter the following details: First Name: Grace, Last Name: Taylor, Phone: Work".
Phone Label:
19


Reviews:
[{"review_id": "Nb2rkhrASU", "reviewer": "Reviewer_YT8u", "summary": "This paper proposes Agent S2, a GUI agent framework with two main components: Proactive Hierarchical Planning, which replans at every step, and a Mixture-of-Grounding Experts to achieve more precise GUI grounding. The method is tested in three GUI environments: OSWorld, WindowsAgentArena, and AndroidWorld, and achieves state-of-the-art performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is overall well written, accompanied by clean and intuitive figures.\n- The proposed “Proactive Hierarchical Planning,” which replans at every step, is sound, as it is often impossible to generate a perfect plan at the beginning in a partially observable environment.", "reasons_to_reject": "The major weakness of this paper is the significant lack of details and experiments, elaborated below:\n- **Unclear whether the improvement comes from a better framework design or simply from using a better base LLM**: Among the baselines in Table 1, only Claude Computer Use (CCU) shares the same base LLMs (Claude-3.5-Sonnet (new) and Claude-3.7-Sonnet) as the proposed method, while most other baselines use GPT-4o, which performs worse than Claude-3.5-Sonnet (new) and Claude-3.7-Sonnet as shown in Table 2. Instead of a single “Method” column, I would suggest the authors split this into separate “Framework” and “Model” columns for a more rigorous comparison.\n- **Unclear whether the improvement comes from a better framework design or from a more tailored action space**: Table 8 shows that the authors added several actions to make it easier for the agent to interact with the environment. Are the current baselines using the same action space? How does this tailored action space affect agent performance in OSWorld, WindowsAgentArena, and AndroidWorld?\n- **Lack of fine-grained baseline comparison**: While I appreciate the detailed breakdown of success rates in Table 2, it provides little insight into how Agent S2 performs compared to baselines at a finer level. I would suggest the authors compare the categorized success rates with those of the baselines as well.\n- **Insufficient baselines for some environments**: There are currently only 2-3 baselines for WindowsAgentArena and AndroidWorld. I would suggest the authors include more baselines for a more comprehensive comparison.\n- **Unclear cost implications**: Agent S2 replans at every step, which raises concerns about additional cost. How expensive is Agent S2 compared to the baselines (e.g., in terms of the number of LLM calls, number of input/output tokens, and time to finish a task)?\n- **Limited novelty**:\n  - The Proactive Hierarchical Planning component is essentially just replanning at every step given new observations.\n  - The “Mixture-of-Grounding-Experts” approach is equivalent to the tool-using paradigm, where each expert functions as a tool. In this paper, three tools are considered: a VLM for visual grounding, `tesseract` for OCR, and `unotools` to interact with OpenOffice.org/LibreOffice. The `unotools` is tailored to the OSWorld environment, which may explain the strong performance on office-related tasks (Table 2 and L271).\n- Other minor concerns:\n  - Exact prompts and model versions (GPT-4o and Claude models) should be included for reproducibility.\n  - Details on how the error analysis was conducted should be provided.\n\nIn summary, I am concerned that the lack of analysis and experimentation makes it difficult to identify the true source of accuracy gains, which may arise from trivial factors.", "questions_to_authors": "None.", "extra_scores": {}}, {"review_id": "9khUtfsCSa", "reviewer": "Reviewer_u6Sn", "summary": "This paper presents Agent S2, a compositional generalist-specialist framework for computer use agents. The framework is illustrated in Figure 2, which comprises generalist planning modules, a manager, workers, and specialist grounding experts to handle complex computer use tasks. The authors conducted experiments on the OSWorld benchmark, consisting of 369 computer use tasks across multiple categories. The results show Agent S2 with Claude-3.7-Sonnet performs best compared to other baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed framework with a combination of multiple models for visual, textual, and structure grounding experts together with a strong LLM, shows improvement on the computer use tasks\n- Good experiments with ablation studies and error analysis\n- Potential to generalize to smartphone use", "reasons_to_reject": "The proposed framework is a combination of various models with a highly abstract description. It is not clear what the key innovation of the proposed work is.\n- Lack of detailed ablation studies to show the key contribution of each model in the framework, especially if we replace Claude with an open-source LLM.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FothOvUSUB", "reviewer": "Reviewer_Cqe8", "summary": "This paper presents Agent S2, a modular framework for GUI agents that integrates generalist and specialist models to improve performance on complex GUI tasks. The framework introduces two key components: a Mixture of Grounding strategy that delegates GUI localization to specialized experts (visual, textual, and structural), and Proactive Hierarchical Planning, which enables dynamic replanning based on evolving observations. Agent S2 achieves good results on multiple real-world benchmarks including OSWorld, WindowsAgentArena, and AndroidWorld.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a new GUI agent, Agent S2, that integrates generalist planning with specialist grounding components. This includes a Mixture of Grounding (MoG) system and Proactive Hierarchical Planning, addressing bottlenecks in GUI agents.\n\n- Agent S2 achieves state-of-the-art results across multiple real-world benchmarks—OSWorld, WindowsAgentArena, and AndroidWorld—outperforming baselines such as Claude CUA, UI-TARS, and GPT-4o-based systems.\n\n- The paper includes detailed ablation studies and failure case breakdowns, giving transparency into the contribution of each component.", "reasons_to_reject": "- I do not think this paper have any good reason to be rejected, altough I think the only part can be improve is the system’s multiple moving parts: manager, worker, three grounding experts, and multiple LLMs—add complexity. This may make deployment or real-time interaction more challenging without significant engineering effort.", "questions_to_authors": "1. How will you handle the deployment or substitution of specialist modules (e.g., grounding experts) in environments where those specific tools may not be available? \n\n2. Additionally, the Agent S2 framework involves a fairly complex architecture with multiple coordinated modules. How do you envision this system being deployed or maintained in real-world settings, particularly with respect to integration complexity, latency, and robustness across components?", "extra_scores": {}}, {"review_id": "aQmoJyQbSI", "reviewer": "Reviewer_Ygqb", "summary": "This paper proposes Agent S2, a compositional framework for computer use agents for GUI applications on desktops and mobile. Unlike single agents that rely on a single generalist model, Agent S2 decomposes cognitive responsibilities across generalist planning modules and specialist grounding experts. It introduces two key techniques: Mixture of Grounding, which enables precise GUI element localization by routing actions to the appropriate grounding expert, and Proactive Hierarchical Planning, which continuously updates task plans based on evolving observations. Agent S2 achieves good results on three dynamic benchmarks. It also shows strong some generalization and improved capability on long-horizon tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a compositional generalist-specialist architecture that addresses some limitations of existing GUI agents. It propose Mixture of Grounding and Proactive Hierarchical Planning, which are practically useful for computer use scenarios.\n2. Agent S2 achieves new SOTA on highly challenging OSWorld and AndroidWorld dynamic benchmark.\n3. Beyond main experiments, authors provide sound ablations and failure analyses, clearly attributing gains to new architectural choices.", "reasons_to_reject": "1. While the paper presents a compelling system design, many of the contributions appear to stem from engineering efforts rather than research. In particular, several design choices seem tailored to the OSWorld benchmark.\n2. The proposed Proactive Hierarchical Planning, IMO, involves re-generating the plan after completing each subgoal. However, similar ideas have already appeared in earlier works such as OS-Copilot[1]. Moreover, the multi-step planning capability could also be acquired through training, e.g., OS-Genesis[2], without requiring the added complexity of a multi-agent framework.\n3. Regarding experiments: the Structural Grounding Expert updates sheetcells directly, and the Textual Grounding relies on OCR for fine-grained word/line-level localization. This causes the system to circumvent some core challenge of precise pixel-level clicking, which baseline agents like TARS and OpenAI CUA must fight grounding directly. This may introduce bias in the comparison, as Agent S2 is operating with privileged settings (IMO, this discrepancy would likely confer an even greater advantage on benchmarks like Spider2V[3]).\n\n[1] OS-Copilot: Towards Generalist Computer Agents with Self-Improvement\n\n[2] OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis\n\n[3] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?", "questions_to_authors": "1. The routing strategy of Mixture-of-Grounding module lacks clarity, whether expert selection is fixed or dynamically switched (as in AgentStore), and whether this is based on hand-crafted heuristics?\n\nI will discuss with the authors further.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a modular approach that decomposes cognitive tasks across generalist planning modules and specialist grounding experts to improve performance on complex GUI tasks. Reviewers recognized the novelty of the Mixture-of-Grounding strategy and Proactive Hierarchical Planning, and noted the strong empirical results demonstrating state-of-the-art performance across multiple benchmarks. The framework’s ability to generalize across different operating systems and applications was also highlighted positively. Concerns were raised about the clarity regarding the source of improvements and the impact of a tailored action space that may simplify tasks relative to baselines. Some reviewers questioned the novelty of the planning approach, noting similarities to prior work, and noted the complexity and engineering effort involved in the multi-component system. Authors clarified that improvements hold when controlling for base models, providing ablation studies showing the contribution of individual components and added actions, and emphasizing the generality of the action space and grounding experts. Reviewers appreciated these clarifications, with some raising their scores accordingly. Overall, the paper was viewed as a strong contribution with sound methodology and significant empirical validation, with some reservations about novelty and complexity. I agree with reviewer consensus towards acceptance given the demonstrated advances and thorough author responses.
Decision: Accept