SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience

## Abstract

Abstract Repurposing large vision-language models (LVLMs) as computer use agents (CUAs) has led to substantial breakthroughs, primarily driven by human-labeled data. However, these models often struggle with novel and specialized software, particularly in scenarios lacking human annotations. To address this challenge, we propose SEAgent, an agentic self-evolving framework enabling CUAs to autonomously evolve through interactions with unfamiliar software. Specifically, SEAgent empowers computer-use agents to autonomously master novel software environments via experiential learning, where agents explore new software, learn through iterative trial-and-error, and progressively tackle auto-generated tasks organized from simple to complex.
To achieve this goal, we design a World State Model for step-wise trajectory assessment, along with a Curriculum Generator that generates increasingly diverse and challenging tasks. The agent’s policy is updated through experiential learning, comprised of adversarial imitation of failure actions and Group Relative Policy Optimization (GRPO) on successful ones.
Furthermore, we introduce a specialist-to-generalist training strategy that integrates individual experiential insights from specialist agents, facilitating the development of a stronger generalist CUA capable of continuous autonomous evolution. This unified agent ultimately achieves performance surpassing ensembles of individual specialist agents on their specialized software. We validate the effectiveness of SEAgent across five novel software environments within OS-World. Our approach achieves a significant improvement of 23.2% in success rate, from 11.3% to 34.5%, over a competitive open-source CUA, i.e ., UI-TARS.

## 1 Introduction

“A new generation of agents will acquire superhuman capabilities by learning predominantly from experience.” silver2025welcome

— David Silver, Richard S. Sutton

With the rapid development of large vision-language models (LVLMs) touvron2023llama; grattafiori2024llama; bai2025qwen2; wang2024qwen2; gpt4; claude-3-7; team2023gemini, computer use agents (CUAs) claude; operator; qin2025uitars; lin2024showui; wu2024atlas have not only emerged but also demonstrated increasing practical utility. By leveraging the powerful perception and reasoning capabilities of LVLMs, these agents can interpret screenshots as visual inputs and operate computers via keyboard and mouse actions.
Despite their promising capabilities, current CUAs qi2024webrl; putta2024agent; deng2023mind2web; he2024webvoyager; bai2024digirl; lu2024gui primarily depend on costly human-curated datasets deng2023mind2web; chen2024guicourse; wu2024atlas; kapoor2024omniact; li2024effects, which are typically derived from demonstrations lu2024gui; zhang2023you; gur2023real; rawles2023androidinthewild; zhang2024agentohana or video tutorials in the wild xu2024agenttrek. However, new software continuously emerges and existing software may regularly be updated, often in the absence of annotated human data. It is both necessary and timely to enter an era that emphasizes learning from experience silver2025welcome in CUA domain. In this paper, we aim to enable CUAs to autonomously explore unfamiliar software environments and evolve into experts without relying on human supervision.

Figure: Figure 1: SEAgent enables computer use agents self-evolving in novel environments by autonomously exploring and learning from their own experiences without human intervention. The specialist-to-generalist training strategy further enhances the development of a strong generalist agent.
Refer to caption: x1.png

To address this challenge, we propose SEAgent, an agentic self-evolving framework in which Computer Use Agents (CUAs) are exposed to previously unfamiliar software environments and engage in autonomous exploration and experiential learning, as illustrated in Fig. [1](https://arxiv.org/html/2508.04700v2#S1.F1). Enabling such self-evolution requires addressing two key challenges: (1) generating executable tasks within unfamiliar software environments, and (2) accurately assessing task success and precisely identifying the step at which failure occurs.
To this end, we introduce a World State Model for environmental state captioning and step-wise trajectory assessment, together with a Curriculum Generator powered by a continuously updated software guidebook memory to generate increasingly diverse and challenging tasks, thereby establishing a curriculum learning paradigm. The agent’s policy is optimized through experiential learning from both failures and successes, combining adversarial imitation of failure actions and Group Relative Policy Optimization (GRPO) on successful ones.

Given the critical role of reward accuracy, we conduct extensive evaluations and observe that existing reward models of computer use tasks fall short in terms of judgment precision and reward density. Leveraging the enhanced long-context processing capabilities of advanced LVLMs, we input the agent’s full trajectory of states into the reward model and fine-tune a reward model, World State Model, using Qwen2.5-VL bai2025qwen2, substantially narrowing the gap with commercial models such as GPT-4o gpt4 with +7.5% improvement in precision compared to baseline model in evaluating CUAs’ trajectories on AgentRewardBench lu2025agentrewardbench, enable World State Model to provide high quality step level reward signals in self-evolving agentic system.

Moreover, SEAgent enables agents to evolve into either single-software specialists or multi-software generalists. To overcome the limitation that directly training a generalist underperforms compared to specialists, inspired by zhang2024buildingspecializedgeneralistai, we introduce a novel specialist-to-generalist training strategy, which even surpasses the performance of individual specialists on their respective software applications.

We perform extensive experiments of SEAgent built on UI-TARS qin2025uitars and evaluated on five professional software applications from OSWorld xie2024osworld. SEAgent with the specialist-to-generalist strategy significantly improves the UI-TARS qin2025uitars from 11.3% to 34.5%. Furthermore, SEAgent with the specialist-to-generalist strategy (34.5%) outperforms both specialist RL (32.2%) and generalist RL (30.6%) by a substantial margin, demonstrating the effectiveness of the specialist-to-generalist paradigm. In general, SEAgent offers a promising approach for developing more powerful and versatile computer-use agents without human involvement.

## 2 Related Work

#### Agent for Computer Use.

With the recent advances in LLMs and LVLMs touvron2023llama; grattafiori2024llama; liu2023visual; bai2025qwen2; wang2024qwen2, which enable human-level perception and reasoning capabilities, the development of agents for computer use has garnered significant attention hu2024agents; hong2024cogagent; cheng2024seeclick; nguyen2024gui; lin2024showui. These agents either rely solely on structured text inputs qi2024webrl; nakano2021webgpt; putta2024agent; lai2024autowebglm; ma2023laser or, in a more human-like manner, use multi-modal inputs such as screenshots combined with textual conditions hong2023cogagent; lin2024showui; wu2024atlas; operator. Although they have been extensively studied and show strong performance on in-domain benchmarks lu2024gui; zheng2024gpt; liu2024visualwebbench; li2025screenspot; cheng2024seeclick, computer use agents still lag significantly behind human-level performance in simulated environments xie2024osworld; rawles2024androidworld; koh2024visualwebarena; zhou2023webarena. This gap highlights the challenges posed by the multi-dimensional demands on LVLMs, including grounding, decision-making, and reasoning. Some approaches address this by decomposing tasks into specialized expert models gou2024navigating; wan2024omniparser and employing agent collaboration agashe2024agent; agashe2025agent; liu2023bolaa; zhang2025appagent through prompt engineering yan2023gpt; he2024webvoyager; zhang2024android; wang2023voyager; wu2024copilot. However, improvements from these training-free methods remain limited without fine-tuning. In this work, we explore the next phase of computer use agents, where a pretrained agent is fine-tuned to learn from its own experience, enabling self-evolution on novel, specialized software without human annotations.

#### Reinforcement Learning for LLMs/LVLMs.

Previous scalable training efforts for LLMs and LVLMs touvron2023llama; grattafiori2024llama; liu2023visual; bai2025qwen2; wang2024qwen2; xing2025scalecap; sun2024bootstrap3d; sun2024xpromptuniversalincontextimage; ding2025mm have primarily relied on supervised fine-tuning (SFT) liu2023visual; wei2022chain. Analogous to imitation learning or behavior cloning in reinforcement learning (RL), SFT trains models to produce desired outputs based on labeled data, making it heavily dependent on high-quality human-curated procedures. Recently, DeepSeek-R1 guo2025deepseek demonstrated strong reasoning capabilities via Group Relative Policy Optimization (GRPO) shao2024deepseekmath using verifiable rewards. Earlier works ouyang2022training; ziegler2019fine; rafailov2023direct have also employed RL for single-turn optimization from human feedback. However, in agentic scenarios such as computer usage—where feedback is sparse with reward signals often results from multi-step interactions—it becomes crucial to introduce stable, step-level reward signals. Prior RL approaches for agents bai2024digirl; qi2024webrl; zhou2024archer; zhai2024fine; carta2023grounding have fine-tuned their own critic models for advantage estimation schulman2015high, either using output reward models (ORMs) trained on labeled data or adopting Direct Preference Optimization (DPO) rafailov2023direct based on interaction data putta2024agent; qin2025uitars. In this work, we investigate various strategies for constructing high-performing reward models for CUAs and find that full-process-based analysis yields more accurate evaluations with fine-grained reward signals compared to training dedicated critic models for advantage estimation as done in bai2024digirl; qi2024webrl or with filtered behavior cloning pan2024autonomous; chen2020bail.

## 3 Methods

Problem Formulation. The objective of SEAgent is to establish a training pipeline enabling the Computer Use Agent (CUA) to autonomously explore its environment (Sec. [3.1](https://arxiv.org/html/2508.04700v2#S3.SS1)) and progressively self-evolve on novel software applications via reinforcement learning from experience (Sec. [3.2](https://arxiv.org/html/2508.04700v2#S3.SS2)). Specifically, the SEAgent pipeline comprises three primary components: an Actor Model $\pi$ performing exploratory actions to accomplish these tasks, and a World State Model $\mathcal{M}_{state}$ describing the current environment state and evaluating the success or failure of executed actions, and a Curriculum Generator $\mathcal{M}_{task}$ that continuously proposes more diverse and challenging exploration tasks:

(1) Actor Model $\pi$: The policy $\pi(a|s_{t},I)$ defines the probability of taking action $a$ at time step $t$, conditioned on the current state $s_{t}$ and the overall task instruction $I$.

(2) World State Model $\mathcal{M}_{state}$: This component is a fine-tuned Large Vision-Language Model (LVLM) responsible for providing detailed descriptions of environment states. It also evaluates each step of the trajectory executed by the Actor Model $\pi$, producing trajectory judgement $\mathcal{J}$ which indicates whether the task has been successfully completed. Joint training with state change captioning $\mathcal{C}$ of the software GUI has been shown to enhance judgment accuracy, as shown in Table [1](https://arxiv.org/html/2508.04700v2#S4.T1).

(3) Curriculum Generator $\mathcal{M}_{task}$: This component utilizes a powerful Large Language Model (LLM) to automatically generate novel exploration tasks. It also maintains and updates a software guidebook $U$ based on the state change captioning $\mathcal{C}$ and the trajectory judgement $\mathcal{J}$ provided by $\mathcal{M}_{state}$ during interactions. The gradually enriched guidebook $U$ enables $\mathcal{M}_{task}$ to progressively generate increasingly diverse and challenging tasks in a curriculum learning fashion.

SEAgent can be applied to enable the self-evolution of a computer-use agent, either as a specialist for a single software or as a generalist across multiple software. However, we observe that direct training for generalist agents is suboptimal. We introduce a specialist-to-generalist training strategy, which achieves even better overall performance than training multiple generalist agents, as discussed in Sec. [3.3](https://arxiv.org/html/2508.04700v2#S3.SS3).

### 3.1 Autonomous Exploration with Self-evolving Curriculum

Figure: Figure 2: SEAgent autonomous exploration and experiential learning pipeline. Guided by tasks generated by the Curriculum Generator, the Actor Model is updated according to step-level rewards from the World State Model through verifiable reward functions tailored for different action types.
Refer to caption: x2.png

Autonomous exploration is essential for enabling the Computer Use Agent (CUA) to develop proficiency in novel software applications that are previously unseen or poorly understood. This process involves addressing two key challenges: (1) generating executable tasks within unfamiliar software environments, and (2) evaluating task completion success and pinpointing the specific step at which failure occurs. To tackle these challenges, we introduce two novel components: the World State Model $\mathcal{M}_{\text{state}}$ and the Curriculum Generator $\mathcal{M}_{\text{task}}$. These components jointly support a self-evolving curriculum paradigm, which facilitates the autonomous generation of increasingly diverse and challenging tasks.

The self-evolving curriculum paradigm pipeline is structured into $P$ sequential phases. Before the first phase, a set of initial tasks targeting basic GUI operations is generated (details provided in Sup. [C.1](https://arxiv.org/html/2508.04700v2#A3.SS1)). In each phase, these tasks are executed and step-wise evaluated. The resulting judgments and descriptions of the exploration trajectories are fed into the Curriculum Generator $\mathcal{M}_{\text{task}}$, which updates a self-maintained software guidebook $U$. Leveraging this updated guidebook and the current capabilities of the CUA, the generator then produces more diverse and challenging tasks for subsequent phases. The following outlines each step of the process in detail:

(1) Task initiation: The initial state of the unfamiliar software is provided, typically in the form of screenshots of its basic GUI interface. The World State Model $\mathcal{M}_{\text{state}}$ performs dense captioning of the GUI elements, including button detection and OCR-based recognition. These detailed captions are passed to the Curriculum Generator $\mathcal{M}_{\text{task}}$, which generates an initial set of task instructions $\mathcal{I}_{0}=\{I_{0}^{(1)},I_{0}^{(2)},\cdots\}$ along with an initial software guidebook $U_{0}$ for the software.

(2) World state judgment: In the $p$-th phase of Auto Exploration, the Actor Model $\pi_{p}$ executes tasks based on the instructions in $\mathcal{I}_{p}$. Each execution is evaluated by the World State Model $\mathcal{M}_{\text{state}}$, which provides feedback $\mathcal{J}_{p}=\{J_{p}^{(1)},J_{p}^{(2)},\cdots\}$ for each step within the operation trajectory. In addition, it generates a detailed description of GUI state changes based on captured screenshots, denoted as $\mathcal{C}_{p}$.

(3) Task self-evolving: Based on the outcomes $\mathcal{J}_{p}$ and $\mathcal{C}_{p}$, the Curriculum Generator $\mathcal{M}_{\text{task}}$ produces a more challenging task set $\mathcal{I}_{p+1}$ and expands the agent’s knowledge boundary by updating the software guidebook to $U_{p+1}$. The detailed prompting process is illustrated in Fig. [8](https://arxiv.org/html/2508.04700v2#A9.F8). This iterative update can be formalized as:

$$ $U_{p+1},\mathcal{I}_{p+1}=\mathcal{M}_{\text{task}}(U_{p},\mathcal{I}_{p},\mathcal{J}_{p},\mathcal{C}_{p})$ (1) $$

Here, $U_{p+1}$ serves as a more comprehensive software guidebook memory, while $\mathcal{I}_{p+1}$ represents a more challenging task set tailored to the current capabilities of the Actor Model $\pi_{p}$. Examples of $\mathcal{I}_{p}$ are provided in Fig. [4](https://arxiv.org/html/2508.04700v2#S4.F4), where the Actor Model $\pi$ demonstrates curriculum learning by handling increasingly complex tasks across different phases $p$. Illustrations of $U_{p}$ across various software applications are provided in Sup. [J](https://arxiv.org/html/2508.04700v2#A10). Comparison with previous methods murty2025nnetnavunsupervisedlearningbrowser; murty2024bagelbootstrappingagentsguiding; sun2024genesis on task generation are detailed in Sup.[C.2](https://arxiv.org/html/2508.04700v2#A3.SS2)

(4) Autonomous RL Training: Through iterative reinforcement learning, the Actor Model $\pi_{p}$ is updated based on its execution of the instruction set $\mathcal{I}_{p}$, guided by evaluation feedback $\mathcal{J}_{p}$ and a set of action-specific verifiable functions $\mathcal{R}_{\text{verifer}}$. The resulting policy $\pi_{p+1}$ is then used as the actor in the subsequent phase. Further details are provided in Sec. [3.2](https://arxiv.org/html/2508.04700v2#S3.SS2).

### 3.2 Reinforcement Learning from Experience

The World State Model $\mathcal{M}_{state}$ provides step-level reward signals for reinforcement learning. Unlike previous reward models for CUA qi2024webrl; bai2024digirl; putta2024agent; pan2024autonomous; lu2025agentrewardbench,
our $\mathcal{M}_{state}$ model takes the entire trajectory of states and actions, $\mathcal{H}=\{(s_{0},a_{0}),(s_{1},a_{1}),\ldots\}$, as input. It classifies each action $a$ as either $a_{F}$ or $a_{T}$, where $a_{F}$ indicates an incorrect action leading to failure or redundant loops, and $a_{T}$ represents a correct action that contributes to successful progression without redundancy. The curated prompt used for judgment is depicted in Fig. [7](https://arxiv.org/html/2508.04700v2#A9.F7). For historical states that result in $a_{T}$, we encourage CUA to reinforce these actions through verifiable rewards defined by a set of functions $\mathcal{R}_{\text{verifer}}=\{r_{dist}\}$. Conversely, for states leading to $a_{F}$, we penalize them using negative KL divergence with adversarial imitation.

Adversarial Imitation for Failure Action Punishment.
To explicitly encourage the policy to diverge from failure-inducing behaviors, we employ a contrastive log-ratio loss based on a reference failure action $a_{F}$. This objective encourages the policy to sample actions $a$ that minimize alignment with the failure action $a_{F}$:

$$ $\mathcal{L_{\text{AI}}}(\pi_{\theta})=\mathbb{E}_{\nu}\left[-\log\frac{\pi_{\theta}(a\mid s,I)}{\pi_{\text{ref}}(a_{F}\mid s,I)}\right]$ (2) $$

This formulation serves as an adversarial imitation signal. By maximizing divergence from this distribution, the agent is trained to explore alternative action distributions that deviate from those leading to failure, particularly in complex GUI interaction scenarios. Notably, this loss shares a similar form with DPO rafailov2023direct but only the negative part.

Verifiable Rewards for Correct Action Encouragement.
To more effectively guide the policy toward correct actions $a_{T}$, we adopt Reinforcement Learning with Verifiable Rewards (RLVR) guo2025deepseek; shao2024deepseekmath, which has recently shown success in enhancing language models on tasks with objectively verifiable answers, such as mathematics shao2024deepseekmath, and more recently, counting and grounding in the vision-language domain liu2025visual; shen2025vlm; meng2025mm. After labeling the correct step $(s,a_{T})$ using the World State Model, we apply Group Relative Policy Optimization (GRPO), computing the relative advantage of each response based on its reward:

$$ $A^{(i)}=\frac{r^{(i)}-\text{mean}(\{r^{(j)}\}_{j=1}^{G})}{\text{std}(\{r^{(j)}\}_{j=1}^{G})},\quad i=1,\cdots,G.$ (3) $$

As we design distinct reward signals for different action types, we define the reward function between a predicted action $a$ and the ground-truth action $a_{T}$ as:

$$ $r^{(i)}=r(a^{(i)},a_{T})=\mathbb{I}\left(\text{type}(a^{(i)})=\text{type}(a_{T})\right)+r_{\text{dist}}(a^{(i)},a_{T}),$ (4) $$

where $\mathbb{I}(\cdot)$ is the indicator function that returns 1 if the predicted action and ground-truth action are of the same type, and 0 otherwise. The distance-based reward term $r_{\text{dist}}(a^{(i)},a_{T})$ is defined according to the specific action type: for click actions, it is computed based on the normalized L1 distance between the clicked coordinates; for drag and select actions, it is computed using the Intersection over Union (IoU) between the predicted and ground-truth bounding boxes; and for type actions, it is determined by the character-level BLEU score between the predicted and ground-truth text. All $r_{\text{dist}}$ rewards are normalized to the range $[0,1]$ to ensure consistency across different action types. A comprehensive list of $r_{\text{dist}}(a^{(i)},a_{T})$ definitions for various action types is provided in Tab. [8](https://arxiv.org/html/2508.04700v2#A7.T8). The final loss of GRPO is directly adopted from shao2024deepseekmath:

$$ $\displaystyle\mathcal{L}_{\text{GRPO}}(\pi_{\theta})$ $\displaystyle=-\mathbb{E}_{(s,I)\sim\mathcal{D},\{a^{(i)}\}_{i=1}^{G}\sim\pi_{\text{ref}}(\cdot\mid s,I)}$ (5) $\displaystyle\Bigg{[}\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|a^{(i)}|}$ $\displaystyle\sum_{t=1}^{|a^{(i)}|}\Big{\{}\min\Big{(}r_{t}^{(i)}(\theta)A^{(i)},\text{clip}(r_{t}^{(i)}(\theta),1-\epsilon,1+\epsilon)A^{(i)}\Big{)}-\beta\,D_{\text{KL}}^{(i,t)}(\pi_{\theta}\|\pi_{\text{ref}})\Big{\}}\Bigg{]},$ $$

$$ $\displaystyle\text{where}\quad r^{i,t}(\theta)=\frac{\pi_{\theta}(a^{(i)}|s,I)}{\pi_{\theta_{\text{ref}}}(a^{(i)}|s,I)}\text{ and }D_{\text{KL}}^{i,t}(\pi_{\theta},\pi_{\text{ref}})=\frac{\pi_{\text{ref}}(a^{(i)}|s,I)}{\pi_{\theta}(a^{(i)}|s,I)}-1-\log\frac{\pi_{\text{ref}}(a^{(i)}|s,I)}{\pi_{\theta}(a^{(i)}|s,I)}.$ $$

Similar to shao2024deepseekmath; guo2025deepseek, advantage $A$ is weighted on the whole reasoning token logits to encourage free form thinking for performing action and planning.

The final training loss is defined as a weighted combination of positive and negative action samples, i.e., correct actions $a_{T}$ and incorrect actions $a_{F}$:
$\mathcal{L}(\pi(\theta))=\mathcal{L}_{\text{GRPO}}+\gamma\mathcal{L}_{\text{AI}}$.
We set $\gamma=0.2$ during training, and the rationale for this choice is discussed in the ablation study presented in Sup. [F](https://arxiv.org/html/2508.04700v2#A6).

This strategy is shown to be more effective in Sec. [4.2](https://arxiv.org/html/2508.04700v2#S4.SS2) compared to Generalized Advantage Estimation (GAE) schulman2015high-based RL methods qi2024webrl; bai2024digirl, as the more powerful reward model $\mathcal{M}_{state}$ provides accurate step-level reward signals by leveraging the entire episode trajectory $\mathcal{H}$ from a global perspective.

### 3.3 From Specialist to Generalist.

Achieving a generalist agent capable of operating across multiple software platforms is an ambitious and valuable goal. We first attempted to train such a generalist directly using the proposed SEAgent framework across all software environments. However, this approach led to suboptimal performance compared to specialized agents, as the actor struggled to learn effectively in the multi-software environment.

We thus introduce a specialist to generalist strategy, as illustrated in Fig. [1](https://arxiv.org/html/2508.04700v2#S1.F1). Specifically, we first train software-specialized agents via SEAgent on individual environments, allowing each to master a specific application. These specialists are then distilled into a single generalist model through supervised fine-tuning (SFT) on synthesized successful trajectories. Finally, the generalist is refined via SEAgent on multiple software. This generalist, now equipped with better reasoning, planning abilities, and software-specific commonsense, achieves significantly improved performance, outperforming both the SEAgent via direct general RL and the performance combination of multi-specialists as in Table [2](https://arxiv.org/html/2508.04700v2#S4.T2).

## 4 Experiments

### 4.1 Benchmark of Reward Model for computer use agent.

Providing CUA agents with reliable reward signals is crucial for enabling self-evolution in agentic systems, consisting of an actor (CUA) and a judge model, especially when interacting with unfamiliar software environments. Recent work, AgentRewardBench lu2025agentrewardbench, proposes to evaluate the precision of reward models by assessing the accuracy of judge predictions on web-based tasks using trajectories from diverse agents.
Building upon AgentRewardBench lu2025agentrewardbench, we further extend the evaluation beyond web tasks to a broader set of PC software environments. Specifically, we evaluate on all 339 feasible tasks from OSWorld xie2024osworld, using rule-based criteria for determining success or failure. Trajectories are sampled from UI-TARS qin2025uitars and Gemini-2.5-Pro google2025gemini25preview, and rule-based evaluation is used as ground-truth supervision. We then compute the confusion matrix by comparing the predictions of different reward models against these labels.

The judge strategy in AgentRewardBench lu2025agentrewardbench relies solely on the final state and the associated action history. However, it is more natural and reliable for a judge model to consider the entire trajectory when assessing task success, rather than focusing only on the final state. For example, consider the task of booking a flight to London. A final state message such as "Your flight ticket has been successfully booked." does not confirm whether the correct date and time were selected, which can lead to compromised judgment accuracy.

However, we observe that current open-sourced LVLMs do not perform well under this more holistic evaluation strategy. As shown in Fig. [3](https://arxiv.org/html/2508.04700v2#S4.F3), feeding additional historical screenshots into Qwen2.5-VL bai2025qwen2 significantly degrades its Average Precision (AP), diverging notably from GPT-4o hurst2024gpt on the same curated prompt detailed in Fig.[6](https://arxiv.org/html/2508.04700v2#A9.F6). We attribute this performance drop to the insufficient pretraining of Qwen2.5-VL on long sequences of high-resolution screenshots, which likely pushes it toward the limits of its 32K context length.

**Table 1: Precision and Negative Predictive Value (NPV) on AgentReardBench lu2025agentrewardbench and OSWorld xie2024osworld with last screenshot only (LS) or entire process screenshots (ES) as input. World State Model closes the gap with commercial model supporting full process high resolution screenshots as input. The co-training with screenshot change description (CD) improves judgment precision.**
| Model | Input | AgentRewardBench | OS-World-Full | Prof/Office |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Precision | NPV | Precision | NPV | Precision | NPV |  |  |
| GPT-4o hurst2024gpt | LS | 68.1 | 92.3 | 46.3 | 88.2 | 40.5 | 81.0 |
| ES | 72.1 | 92.2 | 74.6 | 95.2 | 70.4 | 85.3 |  |
| Qwen2.5-VL-72B bai2025qwen2 | LS | 64.5 | 94.2 | 41.5 | 86.9 | 31.7 | 78.7 |
| ES | 26.2 | 83.0 | 26.8 | 83.0 | 25.6 | 76.6 |  |
| Qwen2.5-VL-7B bai2025qwen2 | LS | 64.1 | 90.3 | 37.3 | 85.2 | 31.8 | 79.0 |
| ES | 25.4 | 83.8 | 20.0 | 81.7 | 23.5 | 76.0 |  |
| World State Model (w/o CD) | ES | 69.1 | 88.5 | 71.1 | 88.4 | 65.0 | 81.1 |
| World State Model (w/ CD) | ES | 71.6 | 91.2 | 73.9 | 90.5 | 69.3 | 82.0 |

To address this issue, we propose a distilled model based on Qwen2.5-VL-7B, referred to as World State Model, which conducts step-by-step screenshot analysis to produce final judgments. The training process for World State Model is detailed in Sup. [A.2](https://arxiv.org/html/2508.04700v2#A1.SS2.SSS0.Px2), using a dataset of 0.86K GPT-4o hurst2024gpt generated evaluations on trajectories with dense GUI change descriptions, exclusively from Chrome within the OSWorld xie2024osworld environment. Despite being trained solely on Chrome data, World State Model exhibits strong generalization to both other professional software in OSWorld and the external AgentRewardBench lu2025agentrewardbench benchmark. This demonstrates that the model learns transferable judgment patterns rather than overfitting to the specifics of a single application, thanks to the diversity and quality of step-level annotations in the training data.

Figure: Figure 3: The Average Precision on AgentRewardBench lu2025agentrewardbench, where GUI-Judge exhibits an improvement in AP as the number of input middle states increases, showing a similar trend to that of the closed sourced GPT-4o hurst2024gpt when compared with its base model.
Refer to caption: x3.png

We evaluate World State Model and our full-process screenshot-conditioned strategy on AgentRewardBench lu2025agentrewardbench, as well as on agent trajectories from OSWorld xie2024osworld. As shown in Tab. [1](https://arxiv.org/html/2508.04700v2#S4.T1) and further analyzed in Fig. [3](https://arxiv.org/html/2508.04700v2#S4.F3), World State Model achieves state-of-the-art performance among open-sourced models, significantly narrowing the gap with GPT-4o hurst2024gpt. More importantly, it exhibits a similar performance trend to GPT-4o when conditioned on historical screenshots. Despite being trained on a relatively small dataset, World State Model is explicitly encouraged to capture the sequential dependencies among historical screenshots and to perform step-by-step reasoning for final judgment. Serving as our foundation reward model, World State Model provides reliable, step-level reward signals that support downstream policy learning. In line with our agentic system design—which emphasizes the evolution of the actor agent with full open-sourced models—we intentionally avoid relying on GPT-4o hurst2024gpt API calls for judgment during training and inference (also due to inefficiency). More details of World State Model is supplied in Sup.[A](https://arxiv.org/html/2508.04700v2#A1).

### 4.2 Self evolution of GUI Agents

**Table 2: Success Rate (SR) on OSWorld xie2024osworld. SEAgent demonstrates strong performance after reinforcement learning from experience. In addition to evolving on separate software, a new General Model achieves better performance after another iteration of SEAgent. *Indicates specialist agents trained separately for each software with ensembled results. All results are averaged over three runs.**
| Model | VScode | GIMP | Impress | VLC | Writer | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Human Performance | 73.9 | 73.1 | 80.9 | 70.6 | 73.9 | 74.5 |
| GPT-4o hurst2024gpt | 4.35 | 3.85 | 6.77 | 16.1 | 4.35 | 7.08 |
| GPT-4V gpt4 | 0.00 | 7.69 | 2.52 | 18.3 | 4.35 | 6.59 |
| Gemini-Pro-1.5 team2023gemini | 0.00 | 11.5 | 13.2 | 6.53 | 8.71 | 7.99 |
| Claude3.7 Sonnet anthropic2025claude37systemcard | 18.8 | 24.4 | 10.6 | 27.5 | 17.4 | 19.7 |
| Gemini-Pro-2.5 google2025gemini25preview | 21.7 | 26.9 | 9.92 | 25.5 | 24.6 | 21.7 |
| UI-TARS-7B-DPO lu2024gui | 13.0 | 23.1 | 4.26 | 11.8 | 4.35 | 11.3 |
| UI-TARS-72B-DPO lu2024gui | 18.8 | 25.6 | 6.38 | 15.7 | 8.70 | 15.0 |
| DigiRL bai2024digirl (Specialized RL)* | 21.7 | 32.1 | 12.8 | 23.5 | 18.8 | 21.8 |
| WebRL qi2024webrl (Specialized RL)* | 27.5 | 29.5 | 10.6 | 25.5 | 15.9 | 21.8 |
| SEAgent (Specialized RL)* | 37.7 | 38.5 | 22.0 | 33.3 | 29.0 | 32.2 |
| DigiRL bai2024digirl (General RL) | 21.7 | 35.9 | 12.1 | 19.6 | 15.9 | 21.0 |
| WebRL qi2024webrl (General RL) | 20.3 | 32.5 | 9.93 | 21.6 | 14.5 | 19.6 |
| SEAgent (General RL) | 36.2 | 39.7 | 19.9 | 31.4 | 26.1 | 30.6 |
| SEAgent (General SFT) | 30.4 | 37.2 | 18.4 | 31.9 | 20.3 | 27.9 |
| SEAgent (Specialist-to-Generalist) | 40.5 | 42.3 | 22.7 | 35.3 | 31.8 | 34.5 |

Figure: Figure 4: Self-evolved task instructions and success rate (SR) curves across different software. Tasks are progressively upgraded by the Curriculum Generator without human intervention, based on the evolving capabilities of the Actor Model at different training phases.
Refer to caption: x4.png

#### Models Before Self-Evolution.

Our self-evolving system is initialized with three locally deployed models: UI-TARS-7B-DPO qin2025uitars as the Actor Model, World State Model as the step-level reward model, and Qwen2.5-72B yang2024qwen2 as the Curriculum Generator for task evolution with software guidebook memory. We conduct experiments on five professional and office-related software applications from OSWorld xie2024osworld. As shown in Tab. [2](https://arxiv.org/html/2508.04700v2#S4.T2), the initial actor agent demonstrates limited performance on these software environments, achieving an average success rate of 11.3% only.

#### Evolution Process Details.

At beginning, we provide World State Model with the initial GUI state of the novel software. The Curriculum Generator then generates the first software guidebook and a set of basic tasks (illustrated in Fig.[5](https://arxiv.org/html/2508.04700v2#A3.F5)). This yields an initial instruction set $\mathcal{I}_{0}$, averaging 150.2 instructions, which are executed by the Actor Model. The resulting trajectories are evaluated by World State Model and parsed into an average of 1361.5 multi-turn conversation pairs (detailed statistics are in Sup.[H](https://arxiv.org/html/2508.04700v2#A8)).
We then perform reinforcement fine-tuning (RFT) following the methodology described in Sec. [3.2](https://arxiv.org/html/2508.04700v2#S3.SS2). Training is conducted for 1k iterations on 8 NVIDIA A100 80GB GPUs, with $G=8$, a batch size of 16, and a learning rate of $2\times 10^{-5}$, scheduled via cosine decay. This evolution process is repeated iteratively for three phases using the same training configuration.

#### Specialist Evaluation.

For a fair comparison with previous reinforcement learning methods bai2024digirl; qi2024webrl, we adapt their training strategies to the UI-TARS qin2025uitars model. Specifically, we initialize the actor agent from UI-TARS-7B-DPO and, instead of providing step-level reward signals, We evaluate its executed trajectories with binary success or failure outcomes using World State Model.
A separate critic model is also initialized from UI-TARS-7B-DPO, with additional random initialized MLP layers taking the LLM’s hidden states as input to regress value predictions. This critic is trained to perform advantage estimation based on Generalized Advantage Estimation (GAE) schulman2015high. The loss functions follow the same configurations as in bai2024digirl; qi2024webrl. Both the critic and the actor agent are trained iteratively using the same phased reinforcement fine-tuning (RFT) process, where the Curriculum Generator continually generates new curriculum-style tasks.

As shown in Fig. [4](https://arxiv.org/html/2508.04700v2#S4.F4) and Tab. [2](https://arxiv.org/html/2508.04700v2#S4.T2), we train separate actor agents for five different software applications. Our approach, denoted as SEAgent (Specialist), achieves strong performance compared to previous reinforcement learning methods such as DigiRL bai2024digirl and WebRL qi2024webrl. We attribute this improvement to the use of World State Model, which provides fine-grained, step-level reward signals derived from a comprehensive understanding of the full history of states and actions. This contrasts with previous approaches that rely on separate critic models—typically initialized from the actor itself—to estimate advantages from sparse, final success/failure signals. Furthermore, the curriculum of task instructions generated by the Curriculum Generator, as illustrated in Fig. [4](https://arxiv.org/html/2508.04700v2#S4.F4), validates the effectiveness of our autonomous learning framework. These tasks progress from simple to complex based on the actor’s evolving capabilities, enabling it to gradually specialize in each target software environment. Based on the observed evolution curves, we set the number of training phases to three, as performance gains saturate beyond that point.

#### From Specialist to Generalist.

After training five strong software specialists, we pursue generalization using the methodology described in Sec. [3.3](https://arxiv.org/html/2508.04700v2#S3.SS3). Specifically, we collect task instructions generated during each specialist’s training phase and use them to prompt the respective specialists for execution. A total of 3.5K successful trajectories, along with their corresponding reasoning traces, are distilled into a new base model (UI-TARS-7B qin2025uitars) via supervised fine-tuning (SFT). This distilled model is then further optimized through reinforcement learning (RL) across all five software environments.

As shown in Tab. [2](https://arxiv.org/html/2508.04700v2#S4.T2), the resulting generalist model surpasses the performance of the individual specialist ensemble, demonstrating the effectiveness of a specialization-first strategy for achieving generalization. By learning from a broad range of software tasks, the generalist improves its reasoning and decision-making capabilities, acquiring transferable commonsense knowledge across domains.

**Table 3: Ablation of different configurations and their corresponding VScode success rates on OSWorld xie2024osworld. Using World State Model as the reward model yields significant performance gains. We further compare different training strategies including supervised fine-tuning (behavior cloning), GRPO, and Adversarial Imitation (AI).**
| Qwen2.5VL-72B | World State Model | SFT (BC) | GRPO | AI | VScode SR |
| --- | --- | --- | --- | --- | --- |
|  |  |  |  |  | 13.0 |
| ✓ |  | ✓ |  |  | 10.1 |
| ✓ |  |  | ✓ |  | 11.6 |
|  | ✓ | ✓ |  |  | 23.2 |
|  | ✓ | ✓ |  | ✓ | 30.4 |
|  | ✓ |  | ✓ |  | 34.8 |
|  | ✓ |  | ✓ | ✓ | 37.7 |

#### Ablation Study of Specialist Training.

In Tab. [3](https://arxiv.org/html/2508.04700v2#S4.T3), we present an ablation study on the effectiveness of various components in our training pipeline, using the success rate on VSCode from OSWorld xie2024osworld as the evaluation metric.
First, we ablate the use of the World State Model for reward signal generation. Its high precision in judging the success or failure of the actor agent’s actions—compared to using a base model—is shown to be essential for effective self-evolution.
In addition to reward quality, reinforcement fine-tuning (RFT) also proves critical. Compared to direct supervised fine-tuning (behavior cloning), RFT encourages more diverse and exploratory reasoning patterns under verifiable rewards, enabling more generalized task planning.
Finally, incorporating adversarial imitation to penalize critical failure-inducing actions allows the CUA to learn from its mistakes, yielding additional performance gains. This highlights the importance of learning not only from successful behaviors but also from failure signals.

## 5 Conclusion

In this work, we introduce SEAgent, an autonomous Computer Use Agent (CUA) exploration system that learns from its own experience on specific software. Powered by a robust World State Model that provides step-level reward signals, and a carefully designed reinforcement learning framework that encourages free-form reasoning through trial and error, the CUA is able to evolve into a specialist for individual software platforms. Furthermore, a specialist-to-generalist training strategy enables the development of a strong generalist agent capable of operating across multiple software environments. Given that computer software constitutes a highly regularized virtual world, we believe this work can inspire future research on agentic systems in both gaming and real world embodied environments.

Limitations and future work. While promising, our work still has several unresolved limitations. Firstly, our self evolving agent system is bounded by GUI-Judge to provide reliable reward signal instead of real signal from environment. As its still challenging to learning from sparse reward signal in complex environment. Secondly, though we tested on relatively complex and novel software like libreoffice-tools and GIMP. The task is still relatively simple as it only takes a human expert less than 20 step to accomplish. How to adapt the system to achieve hours-long workflow in even more challenging software used by real human expert are thus interesting future directions.

## References

- (1)
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang.
Agent s: An open agentic framework that uses computers like a human.
arXiv preprint arXiv:2410.08164, 2024.
- (2)
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang.
Agent s2: A compositional generalist-specialist framework for computer use agents.
arXiv preprint arXiv:2504.00906, 2025.
- (3)
Anthropic.
Claude computer use.
2024.
- (4)
Anthropic.
Claude 3.7 sonnet system card.
[https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf](https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf), 2025.
- (5)
Anthropic.
Claude’s extended thinking.
2025.
- (6)
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar.
Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning.
Advances in Neural Information Processing Systems, 37:12461–12495, 2024.
- (7)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
arXiv preprint arXiv:2502.13923, 2025.
- (8)
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer.
Grounding large language models in interactive environments with online reinforcement learning.
In International Conference on Machine Learning, pages 3676–3713. PMLR, 2023.
- (9)
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al.
Guicourse: From general vision language models to versatile gui agents.
arXiv preprint arXiv:2406.11317, 2024.
- (10)
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross.
Bail: Best-action imitation learning for batch deep reinforcement learning.
Advances in Neural Information Processing Systems, 33:18353–18363, 2020.
- (11)
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu.
Seeclick: Harnessing gui grounding for advanced visual gui agents.
arXiv preprint arXiv:2401.10935, 2024.
- (12)
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su.
Mind2web: Towards a generalist agent for the web.
Advances in Neural Information Processing Systems, 36:28091–28114, 2023.
- (13)
Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, and Jiaqi Wang.
Mm-ifengine: Towards multimodal instruction following.
arXiv preprint arXiv:2504.07957, 2025.
- (14)
Google DeepMind.
Gemini 2.5 Pro Preview (03-25).
[https://deepmind.google/technologies/gemini](https://deepmind.google/technologies/gemini), 2025.
- (15)
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su.
Navigating the digital world as humans do: Universal visual grounding for gui agents.
arXiv preprint arXiv:2410.05243, 2024.
- (16)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.
- (17)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
arXiv preprint arXiv:2501.12948, 2025.
- (18)
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.
A real-world webagent with planning, long context understanding, and program synthesis.
arXiv preprint arXiv:2307.12856, 2023.
- (19)
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu.
Webvoyager: Building an end-to-end web agent with large multimodal models.
arXiv preprint arXiv:2401.13919, 2024.
- (20)
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al.
Cogagent: A visual language model for gui agents.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024.
- (21)
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang.
Cogagent: A visual language model for GUI agents.
CoRR, abs/2312.08914, 2023.
- (22)
Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al.
Os agents: A survey on mllm-based agents for general computing devices use, 2024.
- (23)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
arXiv preprint arXiv:2410.21276, 2024.
- (24)
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov.
Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web.
In European Conference on Computer Vision, pages 161–178. Springer, 2024.
- (25)
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried.
Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.
arXiv preprint arXiv:2401.13649, 2024.
- (26)
Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al.
Autowebglm: A large language model-based web navigating agent.
In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5295–5306, 2024.
- (27)
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua.
Screenspot-pro: Gui grounding for professional high-resolution computer use.
arXiv preprint arXiv:2504.07981, 2025.
- (28)
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva.
On the effects of data scale on computer control agents.
arXiv e-prints, pages arXiv–2406, 2024.
- (29)
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou.
Showui: One vision-language-action model for gui visual agent.
arXiv preprint arXiv:2411.17465, 2024.
- (30)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
Advances in neural information processing systems, 36:34892–34916, 2023.
- (31)
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue.
Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?
arXiv preprint arXiv:2404.05955, 2024.
- (32)
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al.
Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents.
arXiv preprint arXiv:2308.05960, 2023.
- (33)
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang.
Visual-rft: Visual reinforcement fine-tuning.
arXiv preprint arXiv:2503.01785, 2025.
- (34)
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo.
Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices.
arXiv preprint arXiv:2406.08451, 2024.
- (35)
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J Pal, and Siva Reddy.
Agentrewardbench: Evaluating automatic evaluations of web agent trajectories.
arXiv preprint arXiv:2504.08942, 2025.
- (36)
Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu.
Laser: Llm agent with state-space exploration for web navigation.
arXiv preprint arXiv:2309.08172, 2023.
- (37)
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al.
Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning.
arXiv preprint arXiv:2503.07365, 2025.
- (38)
Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee.
Bagel: Bootstrapping agents by guiding exploration with language, 2024.
- (39)
Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D. Manning.
Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild, 2025.
- (40)
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.
Webgpt: Browser-assisted question-answering with human feedback.
arXiv preprint arXiv:2112.09332, 2021.
- (41)
Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al.
Gui agents: A survey.
arXiv preprint arXiv:2412.13501, 2024.
- (42)
OpenAI.
GPT-4 technical report.
CoRR, abs/2303.08774, 2023.
- (43)
OpenAI.
Operator.
2025.
- (44)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
Advances in neural information processing systems, 35:27730–27744, 2022.
- (45)
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr.
Autonomous evaluation and refinement of digital agents.
arXiv preprint arXiv:2404.06474, 2024.
- (46)
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov.
Agent q: Advanced reasoning and learning for autonomous ai agents.
arXiv preprint arXiv:2408.07199, 2024.
- (47)
Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al.
Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning.
arXiv preprint arXiv:2411.02337, 2024.
- (48)
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi.
UI-TARS: pioneering automated GUI interaction with native agents.
CoRR, abs/2501.12326, 2025.
- (49)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
Advances in Neural Information Processing Systems, 36:53728–53741, 2023.
- (50)
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al.
Androidworld: A dynamic benchmarking environment for autonomous agents.
arXiv preprint arXiv:2405.14573, 2024.
- (51)
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap.
Androidinthewild: A large-scale dataset for android device control.
Advances in Neural Information Processing Systems, 36:59708–59728, 2023.
- (52)
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.
High-dimensional continuous control using generalized advantage estimation.
arXiv preprint arXiv:1506.02438, 2015.
- (53)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
arXiv preprint arXiv:2402.03300, 2024.
- (54)
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al.
Vlm-r1: A stable and generalizable r1-style large vision-language model.
arXiv preprint arXiv:2504.07615, 2025.
- (55)
David Silver and Richard S Sutton.
Welcome to the era of experience.
Preprint of a chapter to appear in Designing an Intelligence, edited by George Konidaris, MIT Press (forthcoming), 2025.
- (56)
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al.
Os-genesis: Automating gui agent trajectory construction via reverse task synthesis.
arXiv preprint arXiv:2412.19723, 2024.
- (57)
Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al.
Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows.
arXiv preprint arXiv:2505.19897, 2025.
- (58)
Zeyi Sun, Ziyang Chu, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang.
X-prompt: Towards universal in-context image generation in auto-regressive vision language foundation models, 2024.
- (59)
Zeyi Sun, Tong Wu, Pan Zhang, Yuhang Zang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang.
Bootstrap3d: Improving 3d content creation with synthetic data.
arXiv e-prints, pages arXiv–2406, 2024.
- (60)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 2023.
- (61)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023.
- (62)
Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang.
Omniparser: A unified framework for text spotting key information extraction and table recognition.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15641–15653, 2024.
- (63)
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
Voyager: An open-ended embodied agent with large language models.
arXiv preprint arXiv:2305.16291, 2023.
- (64)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
arXiv preprint arXiv:2409.12191, 2024.
- (65)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
Advances in neural information processing systems, 35:24824–24837, 2022.
- (66)
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong.
Os-copilot: Towards generalist computer agents with self-improvement.
arXiv preprint arXiv:2402.07456, 2024.
- (67)
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al.
Os-atlas: A foundation action model for generalist gui agents.
arXiv preprint arXiv:2410.23218, 2024.
- (68)
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al.
Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.
Advances in Neural Information Processing Systems, 37:52040–52094, 2024.
- (69)
Long Xing, Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jinsong Li, Shuangrui Ding, Weiming Zhang, Nenghai Yu, et al.
Scalecap: Inference-time scalable image captioning via dual-modality debiasing.
arXiv preprint arXiv:2506.19848, 2025.
- (70)
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu.
Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials.
arXiv preprint arXiv:2412.09605, 2024.
- (71)
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al.
Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation.
arXiv preprint arXiv:2311.07562, 2023.
- (72)
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al.
Qwen2. 5 technical report.
arXiv preprint arXiv:2412.15115, 2024.
- (73)
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al.
Fine-tuning large vision-language models as decision-making agents via reinforcement learning.
Advances in neural information processing systems, 37:110935–110971, 2024.
- (74)
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu.
Appagent: Multimodal agents as smartphone users.
In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2025.
- (75)
Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Ming Zhu, Juntao Tan, Thai Hoang, Zuxin Liu, Liangwei Yang, et al.
Agentohana: Design unified data and training pipeline for effective agent learning.
arXiv preprint arXiv:2402.15506, 2024.
- (76)
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang.
Android in the zoo: Chain-of-action-thought for gui agents.
arXiv preprint arXiv:2403.02713, 2024.
- (77)
Kaiyan Zhang, Biqing Qi, and Bowen Zhou.
Towards building specialized generalist ai with system 1 and system 2 fusion, 2024.
- (78)
Zhuosheng Zhang and Aston Zhang.
You only look at screens: Multimodal chain-of-action agents.
arXiv preprint arXiv:2309.11436, 2023.
- (79)
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su.
Gpt-4v (ision) is a generalist web agent, if grounded.
arXiv preprint arXiv:2401.01614, 2024.
- (80)
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al.
Webarena: A realistic web environment for building autonomous agents.
arXiv preprint arXiv:2307.13854, 2023.
- (81)
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar.
Archer: Training language model agents via hierarchical multi-turn rl.
arXiv preprint arXiv:2402.19446, 2024.
- (82)
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.
Fine-tuning language models from human preferences.
arXiv preprint arXiv:1909.08593, 2019.

## Appendix A World State Model

The World State Model (WSM) is a central component of SEAgent, responsible for understanding visual state changes and evaluating the effectiveness of the agent’s actions.

### A.1 Model Architecture and Operation

The WSM is built upon the Qwen2.5-VL-7B vision-language model. It operates in two distinct modes, each with a specific input-output structure to perform different tasks:

- 1.
Trajectory Judgment:
Input:
A sequence of screenshot images captured during an episode.
Output:
Short captions for each screenshot, the reasoning process for the judgment, and a structured judgment dictionary (containing fields such as Correctness, Redundant, and First Error Step, as detailed in Figure 7 of the supplementary material).
- 2.
State Change Description:
Input:
Two screenshot images, one from before and one after a single action was executed.
Output:
A detailed description of the visual differences between the two images.

### A.2 Fine-Tuning Dataset and Process

To equip the WSM with these capabilities, a specialized dataset was constructed for fine-tuning.

#### Data Construction

The data construction process is as follows:

- 1.
Trajectory Sampling: A Computer Using Agent (CUA), powered by UI-TARS and Gemini-2.5-Pro, was used to sample trajectories from 43 feasible tasks in Google Chrome within the OSWorld benchmark. These trajectories were saved as screenshot sequences.
- 2.
GPT-4o Annotation: Using the prompts detailed in Figures 6 and 7 of the supplementary material, GPT-4o was employed to annotate the sampled trajectories, generating judgments and screenshot captions. Only samples where the judgment matched the ground truth from OSWorld evaluation protocols were retained, resulting in 860 high-quality annotated trajectories.
- 3.
Change Description Data: An additional 1,000 pairs of (before action, after action) screenshots were sampled. GPT-4o was used to generate detailed descriptions of the differences, creating a 1,000-sample Change Description (CD) dataset.

#### Fine-Tuning Process

The fine-tuning was performed using the Llama-Factory framework on 8 NVIDIA A100 (80G) GPUs for 2,000 iterations. A learning rate of $2\times 10^{-5}$ was used, and LoRA (rank=128) was employed for parameter-efficient fine-tuning.
The 860 annotated trajectories serve as the core training data for teaching the model trajectory judgment, captioning, and reasoning. The 1,000-sample CD dataset acts as auxiliary data, specifically to encourage the model to focus on fine-grained visual differences, which enhances its overall state understanding. As shown in Table 1 of the main paper, incorporating CD data significantly boosts judgment performance. The two datasets were combined for training without any special re-weighting.

### A.3 Reward Generation from Trajectory Analysis

The trajectory judgment capability of the WSM is the core source of the reward signal for reinforcement learning. After an agent executes a full trajectory $\mathcal{H}=\{s_{0},a_{0},s_{1},a_{1},\dots,s_{\text{final}}\}$, the WSM analyzes it and outputs a structured judgment. Based on this output, actions within the trajectory are dynamically labeled as either positive actions ($a_{T}$) or failure actions ($a_{F}$):

- •
Fully Successful Trajectory: If Correctness is ‘True‘ and there are no Redundant steps, all actions $a$ in the trajectory are labeled as $a_{T}$.
- •
Successful but Inefficient Trajectory: If Correctness is ‘True‘ but Redundant steps begin at step $k$, all actions prior to step $k$ are labeled as $a_{T}$.
- •
Failed Trajectory: If Correctness is ‘False‘ and the First Error Step is $e$, all actions prior to step $e$ are labeled as $a_{T}$, while the erroneous action $a_{e}$ is labeled as $a_{F}$.

These dynamically labeled $a_{T}$ and $a_{F}$ actions constitute the reward signals for the RL pipeline. During training, the actor predicts an action $a_{t}$ based on the history $\{a_{0},s_{0},\dots,s_{t}\}$ and uses these labels to calculate rewards.

## Appendix B Curriculum Generator

The Curriculum Generator is designed to dynamically produce tasks of increasing difficulty and diversity, guiding the agent through a systematic exploration of the software’s capabilities.

### B.1 Task Generation Mechanism

The workflow of the Curriculum Generator is detailed in the pseudocode in our supplementary material. Its core idea is to leverage the WSM’s analysis of completed tasks to generate new ones. The process, illustrated by the "add a rectangle" example from Figure 5, involves three main steps:

- 1.
Analysis and Feedback: The agent successfully completes an initial task, "add a rectangle." The WSM analyzes the execution trajectory and extracts two key pieces of information: a task evaluation (Exam) and a list of observed state changes (CD_list).
CD_list: {"add a rectangle": ["The Edit bar is expanded…", "The cursor has changed into a cross…", "A blue box appears on the screen with side bars showing properties such as fill, line, color, width, transparency, and corner style…"], …}
Exam: [{"task": "add a rectangle", "status": "success"}, …]
- 2.
Knowledge Integration and Task Generation: The CD_list and Exam are fed into the Curriculum Generator. It distills new knowledge, such as "properties of a rectangle," and integrates it into its internal Software guidebook. Based on this new knowledge, it generates more challenging tasks like "Add a green rectangle" or "Add a red rectangle with 50% transparency," which are then added to the task buffer.
- 3.
Iterative Learning: In the next RL phase, the agent samples from this updated, more challenging task buffer. The continuously enriched Software guidebook acts as the system’s long-term memory, driving the Curriculum Generator to propose increasingly sophisticated and unexplored tasks in subsequent rounds, thereby guiding the agent toward mastery.

## Appendix C Details of Curriculum Generator.

### C.1 Exemplar Case during Task Evolution.

Figure: Figure 5: SEAgent autonomous exploration pipeline. The agent (policy model) and World State Model iteratively generate new task and perform RL to become a specialist in novel software.
Refer to caption: x5.png

We provide an exemplar case of our task evolution pipeline in Fig. [5](https://arxiv.org/html/2508.04700v2#A3.F5), demonstrated using LibreOffice Impress. Initially, the World State Model parses a screenshot of the Impress interface into detailed captions describing the layout and individual buttons. The Task Generator then produces an initial task set, $\mathcal{I}_{0}=\{I_{0}^{(1)},I_{0}^{(2)},\ldots\}$, and summarizes the initial software guidance memory $U_{0}$. The initial agent executes tasks in $\mathcal{I}_{0}$, such as “Add a Rectangle,” while the World State Model evaluates these actions, providing judgments and detailed descriptions of resulting changes. As shown in the Auto-Exploration stage, this includes generating captions for newly appeared property panels and assessing execution success. The Task Generator incorporates feedback on execution success and newly revealed properties (e.g., transparency) to evolve new tasks, such as “Draw a green rectangle with 50% transparency.” This process iteratively improves through reinforcement learning, enabling continuous task evolution and agent self-improvement.

### C.2 Comparative Analysis of Instruction Generation Strategies.

To validate the effectiveness of our Curriculum Generator, we conducted a comparative analysis against state-of-the-art instruction generation methods, namely those from NNetNav [39] and WebRL [47].

#### Experimental Setup

We adapted the official code and prompts from these prior works from web environments to general software applications. To ensure a fair comparison of the curriculum quality, for each strategy, we employed two leading LLMs: the open-source Qwen2.5-72B [7] and the proprietary Gemini-2.5-Pro [14]. The tasks generated by each strategy were used to train an RL agent (using GRPO only), with reward signals uniformly provided by our fine-tuned WSM. The evaluation was performed on two applications: VSCode from OSWorld (a standard software) and Celestia from ScienceBoard [sun2025scienceboardevaluatingmultimodalautonomous] (a more challenging, out-of-domain scientific application). The primary metric was the task success rate.

#### Results and Discussion

The results are presented in Table [4](https://arxiv.org/html/2508.04700v2#A3.T4).

**Table 4: Success rate (%) comparison of different task generation strategies on two software applications.**
| Task Generation Strategy | LLM | VSCode | Celestia |
| --- | --- | --- | --- |
| WebRL | Qwen2.5-72B | 27.5 | 0.00 |
| WebRL | Gemini2.5-Pro-thinking | 36.2 | 3.03 |
| NNetNav | Qwen2.5-72B | 34.6 | 0.00 |
| NNetNav | Gemini2.5-Pro-thinking | 43.6 | 5.05 |
| Curriculum Generator (Ours) | Qwen2.5-72B | 37.7 | 9.09 |
| Curriculum Generator (Ours) | Gemini2.5-Pro-thinking | 42.3 | 12.12 |

As shown, the reverse instruction generation strategy from NNetNav [39] is highly effective on the in-domain application (VSCode), demonstrating high data generation efficiency by producing successful trajectories. However, a critical trade-off was observed: this approach tends to generate many similar tasks, limiting its ability to explore the full breadth of the software’s functionalities. This limitation becomes more pronounced when the task generator is unfamiliar with the target software, as seen in the OOD Celestia environment.

In contrast, our guidebook-based method, while having a lower initial data generation efficiency, excels at systematic exploration. It builds structured knowledge of the software from scratch, making it more robust for tackling novel applications. This is evidenced by its superior performance on the more challenging Celestia software.

We conclude that these two strategies are complementary. Reverse instruction generation can efficiently exploit known functionalities, while our guidebook-based method can systematically explore new ones and help the task generator build a more comprehensive understanding of the target software. A hybrid approach combining both strategies is a promising direction for future work.

## Appendix D Test on TARS-1.5

Our work focuses on enabling agents to adapt to out-of-domain (OOD) and novel software where human-labeled data is not available. To test this, we applied our SEAgent pipeline to the UI-TARS-1.5 [48] model on two distinct benchmarks. On OSWorld [68], we observed moderate performance gains. We hypothesize this is because UI-TARS-1.5’s training data already targeted OSWorld, making it a familiar, in-domain environment for the base model. However, on the ScienceBoard [57] benchmark—a suite of scientific applications that are truly novel to UI-TARS-1.5—our pipeline delivers significant and substantial improvements. This strongly validates our core claim: SEAgent is most impactful when performing self-evolution learning on truly OOD software. We excluded two of the six ScienceBoard applications—Lean and TeX—as they are primarily text- and code-based software for mathematics and typesetting, which are not suitable for evaluating a GUI-centric agent like UI-TARS.

**Table 5: Performance comparison on OSWorld and ScienceBoard benchmarks. Scores represent success rates (%).**
|  | OSWorld | ScienceBoard |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | LibreOffice<br>Impress | LibreOffice<br>Writer | GIMP | ChamerX | GrassGIS | KAlgebra | Celestia |
| UI-TARS-1.5-7B-DPO | 19.15 | 33.04 | 51.54 | 12.41 | 0.00 | 11.61 | 4.85 |
| UI-TARS-1.5-7B-DPO+SEAgent | 23.83 | 35.65 | 56.92 | 23.45 | 10.59 | 21.29 | 11.52 |

## Appendix E Sensitivity Analysis on Key Hyperparameters

We conducted a sensitivity analysis on key hyperparameters to evaluate their impact on the SEAgent pipeline. For model sampling, we set the temperature $t=0$ for better reproducibility. We analyze two specific parameters: the number of generated tasks and the number of change descriptions. The results are presented in Table [6](https://arxiv.org/html/2508.04700v2#A5.T6) and discussed below.

**Table 6: Sensitivity analysis for key hyperparameters in the SEAgent pipeline, evaluated on VSCode. The metric is Success Rate (%).**
| # Tasks Generated | VScode SR | # Change Descriptions | VScode SR |
| --- | --- | --- | --- |
| 30 | 31.88 | 30 | 33.33 |
| 50 | 36.23 | 50 | 37.68 |
| 100 | 37.68 | 100 | 37.68 |
| 200 | 37.68 | 200 | 34.78 |

#### Number of Generated Tasks

This parameter controls the breadth of exploration in each learning cycle. As shown in our analysis, performance improves as more diverse tasks are generated, eventually plateauing around 100 tasks.

#### Number of Change Descriptions

This parameter controls how much new information the generator receives to update its "software guidebook." We found a clear trade-off: A sufficient number of descriptions (50–100) is essential for the generator to learn about new UI functionalities and create meaningful, unexplored tasks. However, providing too many descriptions (e.g., 200) creates an overly long context for the LLM, which degrades the quality of task generation and hurts final performance.

## Appendix F Ablation on the Loss Balance Factor.

In Sec.[3.2](https://arxiv.org/html/2508.04700v2#S3.SS2), we use $\gamma$ to balance the ratio of two loss item: adversarial imitation that learn from error and GRPO that learn to achieve success. We ablate the choice of $\gamma$ in Tab.[7](https://arxiv.org/html/2508.04700v2#A6.T7), according to which we set $\gamma=0.2$ in main experiments.

**Table 7: VScode Success Rate on OSWorld [68] under different loss balance factor $\gamma$ values.**
| $\gamma$ | 0.0 | 0.1 | 0.2 | 0.3 | 0.5 | 0.8 |
| --- | --- | --- | --- | --- | --- | --- |
| Success Rate (%) | 34.8 | 36.2 | 37.7 | 31.9 | 26.1 | 23.1 |

## Appendix G Reward Function for Different Actions.

**Table 8: Reward computation for each action type in GUI agent**
| Action Type | Description | Distance-based Reward |
| --- | --- | --- |
| click, left_single, right_single, hover | Click or hover on a location | Normalized L1 distance between predicted and ground-truth coordinates |
| left_double, double_click | Double click on a region | Normalized L1 distance between clicked coordinates |
| drag, select | Drag from start box to end box | Intersection over Union (IoU) between predicted and ground-truth boxes |
| type | Type textual input | Character-level BLEU score between predicted and ground-truth text |
| hotkey | Press multiple keys at once | Character-level BLEU score between predicted and ground-truth key combinations |
| press | Press a single key | Character-level BLEU score between predicted and ground-truth key |
| scroll | Scroll in a certain direction | Character-level BLEU score between predicted and ground-truth direction |
| move_mouse | Move mouse to a specific location | Normalized L1 distance between predicted and ground-truth coordinates |
| highlight | Highlight a rectangular UI region | IoU between predicted and ground-truth region |
| copy, paste | Clipboard operations | BLEU score between copied/pasted content |
| wait | Explicit wait command | Fixed reward + 1 |
| finished, finish_task | Finish current task/trajectory | Fixed reward + 1 |

## Appendix H Data Statistics during Iterative Reinforcement Learning.

**Table 9: Number of episode (Success/Failure) across four phases for different software tools during self-evolution. Each episode contains 8.8 multi-turn conversions in average.**
|  | Phase0 | Phase1 | Phase2 | Phase3 |
| --- | --- | --- | --- | --- |
| VSCode | 112/39 | 282/83 | 161/34 | 98/55 |
| GIMP | 104/51 | 309/90 | 183/50 | 95/52 |
| Impress | 102/44 | 290/92 | 185/61 | 87/51 |
| VLC | 85/29 | 114/41 | 160/48 | 53/27 |
| Writer | 123/62 | 278/101 | 201/69 | 101/43 |

## Appendix I Detailed Prompt Templates.

For evaluation on AgentRewardBench [35], we use their official template for final state screenshot only testing and modified prompt in Fig.[6](https://arxiv.org/html/2508.04700v2#A9.F6) for entire process (or sampled middle screenshots) testing.

For evaluation on OSWorld Sampled trajectories, we use prompt in Fig.[7](https://arxiv.org/html/2508.04700v2#A9.F7) to prompt GPT-4o to provide step level judges, the sampled judges on Chrome in OSWorld [68] serves as training data of GUI-Judge. This template is also used in training GUI-Judge and at inference time in autonomous exploration stage.

For navigator, we use prompt template in Fig.[8](https://arxiv.org/html/2508.04700v2#A9.F8), which takes previous software usage manual and the performance of actor agent evaluated by judge (Empty if in initial phase.) as well as detailed exploration caption as input and output the updated usage manual as well as new task for agent to execute.

Figure: Figure 6: Prompt Template of GUI-Judge for web agent trajectories evaluations with history screenshots as input, its difference with default prompt of AgentRewardBench [35] is highlighted in bold.
Refer to caption: x6.png

Figure: Figure 7: Prompt Template of GUI-Judge for OSWorld [68] trajectories, which prompts judge model to provide step level reward signal.
Refer to caption: x7.png

Figure: Figure 8: Prompt Template for task buffer update, which generates new tasks in a curriculum manner and update software documents. The new tasks are used for actor to perform next phase of RL.
Refer to caption: x8.png

## Appendix J Self documented usage manual on different software during exploration.

In Fig.[9](https://arxiv.org/html/2508.04700v2#A11.F9) Fig.[11](https://arxiv.org/html/2508.04700v2#A11.F11), Fig.[10](https://arxiv.org/html/2508.04700v2#A11.F10), Fig.[12](https://arxiv.org/html/2508.04700v2#A11.F12), we demonstrate the self-documented usage manuals of the navigator (Qwen2.5-72B [72]) in the exploration and learning system introduced in Sec.[3.1](https://arxiv.org/html/2508.04700v2#S3.SS1).

## Appendix K Broader Impacts

Potential positive societal impacts:
SEAgent introduces a self-evolving paradigm for Computer Use Agents (CUAs), enabling them to autonomously learn and adapt to previously unseen software without human supervision. This significantly reduces the need for extensive manual data annotation and domain-specific customization, allowing intelligent agents to assist users across a wide range of applications—including productivity tools, multimedia editing, and educational software. By automating repetitive tasks and providing guidance in complex software environments, SEAgent holds promise for improving accessibility, enhancing digital literacy, and reducing cognitive workload in both professional and everyday settings.

Potential negative societal impacts:
The capability of SEAgent to autonomously explore and operate complex software also introduces risks of misuse. Malicious actors might repurpose SEAgent for unauthorized software automation, such as automating account creation, spamming interfaces, or conducting surveillance via GUI interactions. In addition, as the agent learns from its own experience, there exists a risk that the agent may inadvertently inherit or amplify software-specific biases, potentially leading to unfair or inappropriate behaviors in sensitive applications (e.g., finance, legal automation). Mitigation strategies include controlled release of models, behavior filters during deployment, and incorporating safeguards in the World State Model to detect and prevent unintended or adversarial behavior.

Figure: Figure 9: Automatically generated usage manual during self exploration on VScode.
Refer to caption: x9.png

Figure: Figure 10: Automatically generated usage manual during self exploration on GIMP.
Refer to caption: x10.png

Figure: Figure 11: Automatically generated usage manual during self exploration on LibreOffice_Impress.
Refer to caption: x11.png

Figure: Figure 12: Automatically generated usage manual during self exploration on LibreOffice_Writer.
Refer to caption: x12.png

Figure: Algorithm 1 SEAgent Specialized Self-Evolution Training Loop

## Appendix L SEAgent Self-Evolution Algorithm

Algorithm [1](https://arxiv.org/html/2508.04700v2#alg1) presents the core self-evolution training loop of SEAgent in a specialized software environment. The procedure is divided into four major stages:

(1) Task Initialization. Given the initial GUI state of a target software application, the World State Model performs dense captioning to extract structural semantics (e.g., menu bar, buttons), which is used by the Curriculum Generator to create an initial set of executable tasks and an editable software guidebook.

(2) Autonomous Exploration and Effect Evaluation. The agent explores each task via its current policy. The World State Model then performs step-level trajectory analysis, assigning each action a feedback label—either correct ($a_{T}$) or incorrect ($a_{F}$)—and generating GUI state change captions. This produces rich supervision signals for both policy learning and downstream task generation.

(3) Policy Update via Reinforcement Fine-Tuning. Based on the labeled execution data, positive and negative action steps are separated. We apply Group Relative Policy Optimization (GRPO) to reinforce correct actions, and Adversarial Imitation (AI) to suppress failure-prone behaviors. The updated policy is used for the next exploration round.

(4) Task Update. The Curriculum Generator leverages feedback signals ($\mathcal{J}$) and GUI state transitions ($\mathcal{C}$) to propose more diverse and challenging tasks, thereby expanding the task frontier in a curriculum fashion.

This process repeats over multiple curriculum phases, ultimately yielding a specialized agent policy capable of mastering complex operations in the given software environment.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. If Large Vision-Language Models (LVLMs) can serve as World State Models, it means that LVLMs contain all the information for the application. So why not directly use LVLMs for decision-making? This completely contradicts the "without human intervention" description in Figure 1's caption and the motivation of this paper.
2. The World State Model relies on GPT-4o annotations, which may introduce bias and affect reproducibility.
3. How does the World State Model ensure interpretability in its step-wise trajectory judgments, especially for ambiguous GUI states?
4. Why were certain software applications excluded from ScienceBoard, and how might this affect the validity of OOD performance claims?
5. How does the Curriculum Generator avoid generating repetitive or infeasible tasks as task complexity increases?
6. Can the reward model generalize to software with highly dynamic or non-standard GUI elements not seen during training?
soundness: 2
strengths: The method effectively reduces reliance on human-labeled data by allowing agents to learn from trial-and-error interactions in unfamiliar software environments.
summary: This paper proposes SEAgent, a self-evolving framework that enables computer-use agents to autonomously master novel software through curriculum task generation,  World State Model, and experiential learning. But why can a Large Vision-Language Model as a World State Model?
weaknesses: 1. If Large Vision-Language Models (LVLMs) can serve as World State Models, it means that LVLMs contain all the information for the application. So why not directly use LVLMs for decision-making? This completely contradicts the "without human intervention" description in Figure 1's caption and the motivation of this paper.
2. The World State Model relies on GPT-4o annotations, which may introduce bias and affect reproducibility.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: - Could the authors illustrate the workflow in Section 3 with a concrete example—specifically, how the evolution process unfolds and how each component functions?
- Is the term 'world state world' something you proposed? Or does the concept of world models also exist here?
soundness: 3
strengths: - The lower part of Figure 1 presents the specialist-to-generalist training strategy with great clarity, and this strategy also provides valuable inspiration for progress in other domains. In general, each domain requires models to possess multi-dimensional capabilities, and the visualization and the demonstrated effectiveness of the proposed strategy in addressing this issue are particularly insightful and valuable.
- The paper provides a very clear introduction to the background of the task. Although I had not previously studied this field, I was able to gain a basic understanding of the task through reading the paper.
- Overall, the paper presents a carefully specific design of RL (such as reward signal), and the experimental section demonstrates a high degree of completeness.
summary: To address the issue that CUAs often struggle with novel and specialized software, particularly in scenarios lacking human annotations, the authors propose SEAgent, an agentic self-evolving framework enabling CUAs to autonomously evolve through interactions with unfamiliar software. In addition, they introduce an effective specialist-to-generalist strategy for shaping a versatile generalist agent. The authors validate the effectiveness of their method across five professional software of OSWorld, ScienceBoard, and AndroidWorld.
weaknesses: - The difference between the specialist-to-generalist training strategy proposed in this paper and previous similar strategies requires further clarification.
- The readability of Section 3.1 and Figure 2 is not very good, and Figure 2 is not cited anywhere in the paper.
- There is a citation error at line 235.

---

confidence: 4
contribution: 4
initial_rating: 2
presentation: 4
questions: See weekness above.
soundness: 4
strengths: ### 1. **Innovation**
   - **Self-evolving framework**:  **SEAgent**, a **self-evolving framework** for autonomous exploration and experiential learning, is innovative in CUAs fields. It allows agents to autonomously generate tasks and assess their success/failure in previously unfamiliar software environments without human intervention, advancing the capabilities of autonomous systems.

### 2. **Task Generation and Evaluation Precision**
   - **World State Model**: The paper introduces the **World State Model**, which enables precise environmental state captioning and step-wise trajectory assessment. This model provides fine-grained reward signals, significantly improving the precision of task evaluation.

### 3. **Experimental Results and Performance Improvements**
   - **Multi-software environment adaptability**: SEAgent demonstrates excellent adaptability, not only excelling in single-software environments but also in **multi-software scenarios**. Its validation across five professional software applications from **OSWorld** showcases its generalization.

---
summary: This paper introduces SEAgent, an agentic self-evolving framework designed to enhance the adaptability of Computer Use Agents (CUAs) operating in previously unfamiliar software environments. The framework enables agents to engage in autonomous exploration and experiential learning, supported by a world state model and a curriculum generator. SEAgent optimizes its policy by using adversarial imitation and GRPO methods. With a specialist-to-generalist training strategy, SEAgent shows significant performance improvements across five professional software applications from OSWorld.
weaknesses: 1. **Inconsistency in the Paper's Claims**  
   The paper's claims are not fully self-consistent. Although the paper's title suggests **self-evolving** agents and emphasizes the exploration of LVLMs for autonomous exploration, the **World State Model** used for exploration in this study still relies on **human-annotated high-quality datasets**. This contradicts the starting point outlined in **line 53**, which advocates for agents to evolve without such dependencies. The paper should clarify this contradiction between the claimed self-evolving nature of the agent and its reliance on human-annotated data.

2. **Lack of Novelty in Contributions**  
   The contributions of the paper lack sufficient innovation. The **self-taught**[1] research has already demonstrated that additional training using high-quality critic modules can guide the improvement of LLMs through **Supervised Fine-Tuning (SFT)**, but with limited gains from multiple rounds of training. The paper uses **Group Relative Policy Optimization (GRPO)**, a reinforcement learning method, for enhancement, but no additional conclusions are provided. One important question that remains unaddressed is whether more rounds of interaction could yield better results, as reinforcement learning theory suggests that continued interaction can lead to further improvements.  
   On the other hand, the **specialist-to-generalist distillation method** may not be necessary in this context. In traditional RL, such a method is useful due to the small size of the network, where **Catastrophic Forgetting** may occur. However, for large models, multi-domain data mixing can be achieved during pretraining, and there is no clear need for separate training followed by distillation during the fine-tuning phase. The paper lacks ablation studies to analyze this point of innovation.

3. **Experimental Setup Issues**  
   The experimental setup is not entirely reasonable.  
   - **Table 1** does not provide results for the World State Model under the **LS setting**, leaving this aspect of the model's performance unexplored.  
   - **Table 2** raises a concern: is the training data size consistent across **general RL**, **SFT**, and **specialist-to-generalist** methods? From the descriptions in the paper, it seems that the **specialist-to-generalist** approach would likely involve more training data, but this is not explicitly clarified.  
   - **Table 4**: The ablation study does not fully incorporate the innovative aspects discussed in **point 2**, specifically regarding the necessity and impact of the specialist-to-generalist strategy.

4. **Writing Problems**  
   The writing lacks rigor. For example, in **line 131**, the phrase should be "Recently, ... has achieved". Similarly, in **line 235**  "in Section [???]". Authors should check their writing more carefully.

[1] Wang, Tianlu, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. "Self-taught evaluators." arXiv preprint arXiv:2408.02666 (2024).

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 1
questions: 1. Why do performance gains saturate beyond the thrid training phase?
2. What do the ensembled results of specialized RL mean in Table 2? How to understand the "individual specialist ensemble" in line 429?
3. Does **General RL** refer to training a single agent in all five environments simultaneously? If so, how to train a single agent in different environments through RL? And it seems that the only difference between **General RL** and **Specialist-to-Generalist** is that the latter involves SFT initialization before RL.
4. In all the experimental results, does SEAgent refer to SEAgent (Specialist-to-Generalist) by default?
5. There is an severe error in the paragraph of **Ablation Study of Specialist Training**, of which the content is not the analysis of ablation results. Also, how to understand Table 4? For example, Does Qwen2.5VL-72B refer to Curriculum Generator? What does the result in the first row mean?
6. The result in the last row of Table 4 is 46.1, which equals to that of SEAgent (Specialized RL) in Table 2. What are the differences between the two experiments in terms of its configuration?
soundness: 2
strengths: 1. This work identifies several key elements contributing to the development of autonumous agents for computer use without relying on annotated data, providing insights for practical implementation of such agents. The key elements include:

(1) a new task generation strategy using a Curriculum Generator;

(2) a new reward model to assess both the step-level actions and trajectory-level success using a World State Model;

(3) a new training objective combining both adversarial imitation of failure actions and RL (using GRPO).

(4) a specialist to generalist strategy that outperforms both direct RL in multiple softwares and RL in each environment.

2. Extensive experiments are conducted to justify different aspects, including the effectiveness on different benchmarks and using different actor models, the performance of World State Model compared to other models as judges, the comparison between Specialist RL, General RL and Specialist-to-Generalist, the performance gains during different training phases and ablation studies to show the usefulness of different components in the poposed pipeline.
summary: The paper proposes an agentic self-evloving pipeline for learning autonomous computer use agent in software environments without human-labeled data.

The pipeline consists of several key components: (1) a World State Model to provide step-level reward signals for RL training and state change captioning as well as the trajectory-level judgement for maintaining a software guidebook for a Curriculum Generator; (2) a Curriculum Generator that automatically generates new task instructions for curriculum learning of the agent; (3) a training process involving adversarial imitation of failure actions and GRPO based on successful actions; (4) a specialist-to-generalist training strategy that  involves SFT on successful trajectories from specialist agents first and then RL across multiple softwares.

Experiments on OSWorld, ScienceBoard and AndroidWorld show the effectiveness of proposed pipeline.
weaknesses: 1. The writing of the paper is too poorly, requiring many efforts to revise and polish.

(1) Too many grammatical errors, making it hard to read. 

(2) Captions of most Tables and Figures are unclear, lacking explanations to details in the Table or Figure. For example, the meaning of different colors of the curves in Figure 4. 

(3) The experimental setup is missing in the main paper, including benchmarks, baselines, evaluation metrics, and implementation details, making it hard to make a fair comparison between methods (although I can find some in the appendix).

(4) Some notions or details are not explained. For example, the explanation of *t* in Eq. 5, and how to construct the training data for GRPO? For other issues, please see questions below.

2. The introduction and comparison with related work is insufficient. 

(1) The paper claims that "SEAgent achieves superior performance, which can be attributed to World State Model providing fine-grained, step-level rewards from the full history". However, similar ideas has been justified in [1] but not referred to, where a process reward model is prompted by the full interaction history.

(2) Some related work on GUI agents are missing and not discussed, e,g., [2][3]

3. It seems that the training sample for GRPO is a one-step data from a trajectory instead of a whole trajectory. Why use the single-turn RL in an agentic setting? What if the proposed training method is compared to multi-turn RL approaches?

References:

[1] [EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning](https://aclanthology.org/2025.acl-long.747/) (Liu et al., ACL 2025)

[2] Luo, Run, et al. "Gui-r1: A generalist r1-style vision-language action model for gui agents." arXiv preprint arXiv:2504.10458 (2025).

[3] Chen, Kevin, et al. "Reinforcement learning for long-horizon interactive llm agents." arXiv preprint arXiv:2502.01600 (2025).