Efficient Agent Training for Computer Use
Authors: Yanheng He
Jiahe Jin
Pengfei Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Agents; Computer Use;  Large Language Models; Vision Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: PC Agent-E demonstrates efficient agent training with a small set of human trajectories augmented with Claude 3.7 Sonnet, achieving 141% improvement and surpassing Claude 3.7 Sonnet by 10%.

Abstract: Scaling up high-quality trajectory data has long been a critical bottleneck for developing human-like computer use agents. We introduce PC Agent-E, an efficient agent training framework that significantly reduces reliance on large-scale human demonstrations. Starting with just 312 human-annotated computer use trajectories, we further augment them by synthesizing diverse alternative action decisions with Claude 3.7 Sonnet. Trained on these enriched trajectories, our PC Agent-E model achieved a remarkable 141% relative improvement, and even surpassed the Claude 3.7 Sonnet by 10% in relative terms on WindowsAgentArena-V2, an improved benchmark we also released. By integrating robust human computer use skills with automated AI data synthesis capabilities, our method not only brought substantial improvements over training on human trajectories alone, but also significantly surpassed direct distillation from Claude 3.7 Sonnet.

Published as a conference paper at ICLR 2026
EFFICIENT AGENT TRAINING FOR COMPUTER USE
Yanheng Hel.3+ Jiahe 1.
Pengfei Liu Liu 123
'Shanghai Jiao Tong University
SII
GGAIR
ABSTRACT
Scaling up high-quality trajectory data has long been a critical bottleneck for developing human-like computer agents. We introduce PC Agent-E, efficient agent training framework that significantly reduces reliance on large-scale human demonstrations. Starting with just 312 human-annotated computer trajectories, we further augment them by synthesizing diverse alternative action decisions with Claude 3.7 Sonnet. Trained these enriched trajectories, our PC Agent-E model achieved remarkable 141 % relative improvement, and even surpassed the Claude 3.7 Sonnet by 10% in relative terms on WindowsAgentArres V2, an improved benchmark we also released. By integrating robust human computer use skills with automated Al data synthesis capabilities, our method not only brought substantial improvements over training human trajectories alone, but also significantly surpassed direct distillation from Claude 3.7 Sonnet. Code, data and models are available https: / //gtttttbbbb/// com/ GAIR NLP /PC- EFCCAgent-E.
INTRODUCTION
Developing autonomous agents that can oper- computers as humans do (Anthropic, 2024;
Trajectory Boost
OpenAI, 2025) has long been a landmark pur- suit in Artificial Intelligence (AI). Such computer agents, powered by Vision-Language Models (VLM), perceive screenshots to interact directly with Graphical User Interfaces (GUI) - clicking buttons, navigating menus, and entering text. This allows them automate wide range of digital tasks, ranging from routine paperwork and online shopping to complex content creation, promising a significant reduction in manual human workload.
60%
(m])
Claude
50%
312 human
more diverse action decision
trojectories
40%
Claude 3.7 Sonnet
36.0%
30%
20%
14.9%
However, current models still fall significantly short of human performance (Xie et al., 2024; Bonatti et al., 2024). This capability gap even more pronounced within the open- open-source commu- nity, which lacks any solution competitive with leading proprietary systems like Claude 3.7 Son- net (Anthropic, 2025a). Instilling these advanced computer capabilities in open-source mod- els remains an unsolved problem. A key factor contributing to these deficiencies is the extreme scarcity of high-quality computer use trajectory data (Ou al., 2024; Xu al., 2025a).
10%
141%1 1
0%
Qwen2.5-VL
PC Agent-E
Figure 1: PC Agent-E achieves state-of-the-art open-source performance Windows computer use with just 312 augmented trajectories.
In this work, we explore efficient agent training for computer use, enabling open-source models to exceed the performance of proprietary counterparts with minimal human annotation. Inspired by recent findings (Huang et al., 2024; Muennighoff et al., 2025; Ye et al., 2025) that synthesizing high- quality data using advanced reasoning models like Deepseek-RI (Guo al., 2025) can efficiently enhance LLM reasoning, we extend the similar idea to the field of computer use agents.
"qqual contribution. 'Corresponding author.

---

Published as a conference paper at ICLR 2026
Trajectory Collection
Thought Completion
Trojectory Boost G 6-00--++
Agent Training
thought
(1a))
A RACAS
single action decisions
observation
Bhought
PC Tracker
312 trajectories
action
diveree action decisions
Agent-E
computer
Figure 2: Overview of our framework, consisting of four key components: (1) Trajectory Collection, gathering a small set human trajectories by recording user actions and state observations each step; (2) Thought Completion, reconstructing the implicit thought process missing in raw human trajectories; and (3) Trajectory Boost, diversifying action decisions to further augment trajectories (4) Agent Training, developing strong computer use agent with remarkable data efficiency.
We propose PC Agent-E, an efficient agent training framework that integrates human expertise with Al automation. Starting from small set of real-world human computer trajectories, leverage a frontier agent model to diversify action decisions, yielding richer supervision signals. Training on these augmented trajectories, our agent demonstrates strong computer use capabilities with remarkable data efficiency.
We begin by collecting 312 human computer use trajectories with PC Tracker (He et al., 2024), a tool for gathering human-com computer interaction data, with only two humans annotating one day. These trajectories include task descriptions, screenshots, and human keyboard/mouse actions. We subsequently reconstruct the implicit thought process behind human actions, obtaining comprehensive human trajectories with thoughts. The successful completion of these tasks is inherently assured by human proficiency, obviating the need for additional verification.
While these human trajectories already serve as valuable agent training data, we further augment them through Trajectory Boost, data synthesis method we developed to enrich each trajectory step with diverse alternative action decisions. The key insight is that computer use tasks can be completed through multiple valid pathways, meaning each step has various reasonable action alternatives supported by rational thought. To capture this diversity, we use a strong agent model to synthesize other possible action decisions. Specifically, each human trajectory step provides an environment snapshot that encapsulates the essential state information necessary for computer use agents to make decisions. We provide these snapshots Claude 3.7 Sonnet and sample multiple possible action decisions, thereby significantly enriching and diversifying the trajectory data.
Experimental results demonstrate that with only small of method can boost the performance of an open-source model to that of frontier models. Trained with only 312 trajectories augmented by Claude 3.7 Sonnet, our PC Agent-E model achieves an impressive 141% % relative improvement over the base model Dowen2.5-VL772.
trajectories,
and even outperforms the teacher model Claude 3.7 Sonnet by 10% in relative terms on WindowsAgentArena- V2, benchmark we improved from WindowsAgentArren (Bonatti et al., 2024). Furthermore, PC Agent-E generalizes well to different operating systems on OSWorld (Xie al., 2024). Our ablation study further demonstrates that our method for utilizing human demonstrations is not only superior to relying solely on the human trajectories but is also significantly more effective and efficient than directly distilling from the teacher model.
In summary, our key contributions are threefold:
1. We propose Trajectory Boost, a simple data synthesis method that unlocks remarkable data efficiency for training computer use agents. By augmenting human trajectories with diverse action decisions from a frontier model, our method demonstrates significantly greater effectiveness and efficiency than both using human data alone and direct distillation.
2. We release SanddowsAgenAreent V2, an improved benchmark rectifying evaluation de- pendence, infeasible hacking, and other limitations in the original AindowsegentArenna benchmark, ensuring more robust and fair evaluations of computer use.
3. We developed PC Agent-E, an open-source computer use agent that achieves performance comparable leading proprietary models. Trained with just 312 augmented trajectories,
2

---

Published as a conference paper at ICLR 2026
model successfully outperforms the strong teacher model, Claude 3.7 Sonnet, showing exceptional data efficiency.
2
RELATED WORK
2.1 COMPUTER USE AGENT
With the advancement of VLMs (Bai al., 2025; Deitke et al., 2024), the way computer use agents interact with computers has gradually shifted from relying on textual representations such as accessibility trees (Agashe al., 2024; Wu al., 2024a) to directly using screenshots (Qin et al., 2025; Xu et al., 2025b; He et al., 2024). Existing computer use agents can be categorized based on how much human prior is built into their design: One is modular agen! workflows (Agashe al., 2024; Wu et al., 2024a), which defines specialized modules and prompts multi-agents to collaborate. The other native agent models (Anthropic, 2024: Qin al.. 2025; OpenAI, 2025), which depends on a single model to take action step by step based on its history and current state.
While modular agent workflows can reduce task complexity, their heavy reliance human priors hinders adaptation to new domains and limits end-to-end optimization (Saltzer et al., 1984; Pan et al., 2024). With the continuous enhancement of model capabilities, native agent models have emerged as the dominant paradigm. This approach offers flexibility, generalizability, and sustainable gains via supervised fine-tuning (SFT) (Xu et al., 2025b) or reinforcement learning (RI.) (OpenAI, 2025). Our work explores the efficient agent training methods for native agent models through SFT.
2.2 DATA SYNTHESIS
As Large Language Models (LLMs) grow ever more powerful, it has become a common practice to use them to synthesize data. Distillation methods (Taori et al., 2023; Gunasekar et al., 2023; Xu et al., 2023) leverage state-of-the-ar (SOTA) models to generate large-scale data for training weaker models. On the other hand, self-imprrvement methods enable a model to bootstrap and refine its own training data (Wang et al., 2023).
In the domain of computer use agents, data synthesis can be broadly categorized into three aspects: (1) large-scale datasets that build foundational GUT understanding. with tasks like screenshot captioning or question-answeriin (Liu et al., 2024; Qin et al., 2025) (2) single-step visual grounding, where mouse click tasks are synthesized from specific locations on the GUI (Gou et al., 2025; Wu et al., 2024b) (3) multi-step trajectory, in which recent research has explored leveraging web tutorials to guide trajectory generation (Ou et al., 2024; Xu et al., 2025a) or reverse-synthesizes tasks from the agents' own exploration records (Sun et al., 2025; Murty et al., 2024). Our work differs from prior work by synthesizing high-quality multi-step trajectories based on real-world human demonstrations and emphasizing data efficiency.
3
METHOD
3.1
OVERVIEW
We propose PC Agent-E, an efficient agent training framework for computer use that integrates human expertise with Al automation, as illustrated in Figure 2. Our method generates high-quality trajectory data by combining authentic human-computer interactions with diverse action decisions, offering advantages in both realism and diversity.
1. First, we gathered a small set of 312 task trajectories from human annotators, recording both the screen state observation and the human action at each step, and then filtered the data to remove erroneous steps and trajectories. ($3.2)
2. Subsequently, we reconstructed the latent human thought process before each action decision, based on the corresponding screen state observation and history step context. ($3.3)
3. Then, using human trajectories as environment snapshots, we employ Claude 3.7 Sonnet to synthesize diverse alternative action decisions with the Trajectory Boost method. ($3.4)
3

---

Published as a conference paper at ICLR 2026
4. Finally, develop PC Agent-E, our SOTA native agent model for Windows computer use, trained on our augmented trajectories with a simple end-to-end scaffold. ($3.5)
3.2 TRAJECTORY COLLECTION
We collected human computer use trajectories with PC Tracker (He al., 2024), a tool that records the screen state observation and the human keyboard/mouse action each step for a given task. The recorded actions are structured in unified action space A, as shown in Table 2. For task generation, we first manually compose a small seed set across multiple software applications and then large with LLMs. The resulting tasks were distributed to human annotators, who completed the tasks their own Windows computers with PC Tracker recording trajecto- ries automatically. After finishing task, annotators could either discard unsatisfactory trajectories or modify the task descriptions based on their actual execution, thereby suring the correctness and completeness of the collected trajectories. We then applied set of rule-based filters remove entire trajectories individual steps that exhibited errors or other undesirable behaviors.
Utila 33
Chrome 70
LibreOffice 25
Total Trajectory
System
Edge
VLC
36
Code
45
Figure 3: trajectories across different applications.
Distribution of the 312 task
We employed rigorous data decontamination procedure on these collected trajectories. Each task description was compared against the tasks in our main evaluation bench- mark ($44) using n-gram overlap and semantic similarity metrics. Specifically, we retained only trajectories whose task descriptions have an overlap of 0 under 13-gram and less than 0.7 under 3-gram, and cosine semantic similar- ity under 0.85 with respect to any test task.
Length Range
Count
Percentage
2-5
74
23.7%
6-10
139
44.6%
11-15
73
23.4%
16-30
26
8.3%
This procedure finally yielded 312 real-world human puter use trajectories, with distribution across applications shown in Figure 3. We report the trajectory length distri- bution in Table 1. No explicit instructions on trajectory length were given annotators; the distribution naturally emerges from the collected tasks and human demonstration behavior. The whole annotation process was completed by two annotators within a single day, with an average of roughly 3 minutes per trajectory. Given humans' proficiency in computer use, the mechanisms for annotators to discard trajectories or revise task descriptions after execution, and our data filtering process, no additional verification was required to ensure trajectory correctness.
Table 1: Distribution of the 312 task trajectories across length ranges (steps).
3.3 THOUGHT COMPLETION
We first reconstruct the implicit thought pro- cess behind human actions using an iterative ap- proach. Specifically, for each action in the raw trajectory, we provide Claude 3.7 Sonnet with: task description. historical actions with their pre- viously reconstructed thought processes, the cur- rent action, and the corresponding screenshot. Based on this information, the model generates the implicit thought process behind action. As shown in Figure 4, the recorded raw human tra- jectory was converted to a human trajectory with thoughts, where the reconstructed thought pro- cess is added to each step. See our prompt in Appendix F.I.
Action
Description
click (x.
clicks coordinates.
right click (x, y)
right click at coordinates.
double click
double-click at coordinates.
drug from (xl, yl) (x2, y2)
drag the mouse.
scroll
scrolls screen offiet presses the Enter key.
key: enter hothey (etrl, e) type text: hello wait
performs the Cirle hotkey. type text "bello" pauses for some time. the task finished. the task failed.
fouisk fail
Table 2: Unified action space A.
4

---

Published as a conference paper at ICLR 2026
task
task
task
screen state
human action thought)
Thought Completion
Trajectory Boost
human action decision (w/ thought)
Claude action decision (w/ thought)
humon trajectory with thoughts
Traj Tree
trajectory
Figure 4: Visualization of our Trajectory Boost method. (Left) Raw human trajectory recorded by PC Tracker. (Center) Human trajectory with reconstructed thoughts after Thought Completion, where the red node indicates human action decisions. (Right) The final Traj Tree, where the blue node indicates augmented diverse action decisions synthesized by Claude 3.7 Sonnet.
3.4 TRAJECTORY BOOST
After thought completion, we obtain comprehensive human trajectories with explicit thought pro- cesses. While these trajectories already serve valuable agent training samples, further augment them through simple but effective approach called Trajectory Boost, which synthesizes diverse alternative action decisions for each step of the trajectory.
The motivation behind Trajectory Boost is that computer use tasks inherently allow for multiple valid solution pathways. Consequently, any given step, several reasonable actions supported by rational thought processes may exist, extending beyond the single solution adopted by human annotators. To capture this inherent diversity, we utilize frontier computer use agent model, Claude 3.7 Sonnet, to generate single-step alternative action decisions. Its long-horizon planning capabilities, advanced reasoning patterns, and broad knowledge of computer use enable it to generate thought processes and actions that are highly informative and valuable, thereby substantially enhancing the richness and diversity of our trajectory data.
Specifically, recognize that each step in human trajectory captures environment snap- shot the computer, providing the necessary information for both humans and agents to make decisions. For step k on human trajectory with observation og, thought process t,, action ag and task description T, the environment snapshot is < T,ok, hy >, where the history context h = (t1, 01 122, 02 ... 1 a1-1) is constructed with previous human steps. We input this envi- romment snapshol to Claude 3.7 Sonnet instantiate in the PC Agent-E scaffold ($3.5), and sample multiple single-step action decisions (1, a4)) from it. Prompts used are shown in Appendix F.2. In practice, we sample 9 action decisions in parallel. This produces Traj Tree. shown in Figure 4, with human trajectory forming the main trunk and the augmented action decisions branching off as leaf nodes. These sampled action decisions from Claude 3.7 Sonnet are not executed in real computer environments, but serve as important augmented data for later agent training.
3.5
AGENT TRAINING
Input
Output
Dascription
History
Screanshot
Thought
Step1
help you enable the 'Do Not Track feature.
enable 'Do feature in Chrome to enhance my online
Thought: Looking the screen..
Action: click (1265, 121)
Step2
Thought: ined to navigate to...
Action: click (1195, 661)
click (1263,
Figure 5: A training example that also demonstrates the inference process of the PC Agent-E, scaffold.
5

---

Published as a conference paper at ICLR 2026
Evaluation
Infeasible
Initial Configuation
Task Description
Dependency
Hacking
Task Execution
Virtual Machire
observation
Agent
VM snapsho:
Tasiite tadaa
Infeosirre
Tasics
Initial State Instability
Evaluation
oction
Flaws
Final State
hand Wrizon
eteluetion fUles
hand written
evaluation
Figure 6: (Left) Overview of the WindowsAgentArenn benchmark. (Right) Our main modifications to the updated WindowsAgentrrena- V2 benchmark.
PC Agent-E adopts a deliberately simple end-to-end scaffold, primary focus is on validating the effectiveness of our agent training methodology rather than optimizing performance through complex workflow design elaborate prompt engineering, At inference, PC Agent-E takes (screenshot, task description, history) as input and outputs a (thought, action) decision in the ReAct (Yao et al., 2023) paradigm, shown in Figure 5. The action space is the same A in Table 2, and every action is executed via the PyAutoGUT library. The history is a textual log of previous thoughts and actions. To maintain simplicity in both training and inference, past screenshots are excluded. although we believe that adding this image history would be beneficial for improving model performance. The prompt used for the scaffold shown in Appendix F.3.
For training, we transform each action node from our Traj Tree into an individual training sample, The training sample structure and the inference-time scaffold of the agent share direct correspondence, as illustrated in Figure 5. For both human- demonstrated and history the training sample includes only prior human actions on the main trunk the Traj Tree. This is consistent with the historical context available to both humans and the model when making the corresponding action decision. We finally obtained 27K training samples from 3 312 augmented trajectories, each following a consistent input-output structure at inference time.
action nodes, the
WINDOWSAGENTARENT V2
We initially performed our evaluation on WindowsAgentArenn, (Bonatti et al., 2024), benchmark designed to assess computer use ability in realistic Windows OS environments through diverse tasks across multiple applications. provides automatic initial configuration of virtual machine (VM) state and hand-written evaluation rules (see Figure 6). However, we identified several limitations during our assessment. To ensure evaluation reliability, we developed Windows AindowsAgentAArenn- updated benchmark comprising 141 tasks across 11 widely-used Windows applications, all derived from the original WindowsAgentAron but with improvements detailed below.
an
Addressing the evaluation dependency issue. The original benchmark lacked VM state reset between task evaluations, allowing changes from previous tasks to potentially affect subsequent ones. We implemented VM snapshot restoration before each evaluation, ensuring consistent starting states, preventing inter-task interference, and aligning with the i.i.d. (independent and identically distributed) assumption. also installed some essential software missing from the original VM snapshot but required for proper evaluation.
Preventing infeasible hacking. Current computer use benchmarks such as WindowsAgentAreen. and OSWorld often include infeasible tasks, which are inherently impossible to complete due to issues such deprecated system features user-generated hallucinated commands (Xie et al., 2024). The evaluation metric for these tasks is simply considering a task successful if the action FAIL is output any point during execution. However, found such evaluation methods particularly easy to hack: an agent can trivially achieve a perfect score on infeasible tasks by always outputting FAIL, without demonstrating any meaningful computer capabilities. In contrast, completing feasible

---

Published as a conference paper at ICLR 2026
task typically requires the agent to execute actions step-by-step to actually fulfill the task objective, posing a significantly different level of difficulty.
We refer to this phenomenon as infeasible hacking, vulnerability confirmed by our subsequent experiments ($5.6), in which a weaker model achieved markedly higher scores on infeasible tasks. Since agents receive identical scores for feasible and infeasible tasks, their coexistence undermines benchmark fairness. Additionally, given that current computer use agents' capabilities are far from optimal, argue it is presently more valuable to focus on enhancing agents' performance on feasible tasks. Therefore, as a temporary solution in WindowsAgentArenss tasks to prevent infeasible hacking.
we removed all infeasible
Guarantceing VM initial state stability. We found that the state VM after task initial configura- tion often exhibited errors like unstable network connections, software launch failures, or system lags. To address this, we designed a validation framework combining rule- based and LLM-based evaluations verify the initial state, with re-test mechanism allowing up to three restart attempts for faulty initializations. This approach reduced the initialization failure rate from 10%-30% (depending on hardware) below 5%.
Fixing evaluation flaws. We discovered that some evaluation functions contained bugs or lacked robustness. For instance, in the task "clearing YouTube hi history to facil fiiiiiee ttate finding other histories", the evaluation erroneously awarded full scores to agents that deleted the entire browsing history, clearly contradicting user intent. We identified and corrected sev- eral evaluation errors and relied on human evaluators for a few complex tasks to improve assessment reliability.
EXPERIMENT
In this section, we conduct extensive experiments to evaluate PC Agent-E and validate our Trajectory Boost method. Our experiments are designed to answer the following key questions:
1. How does PC Agent-E perform against SOTA methods on computer use tasks? ($5.2)
2. How does Trajectory Boost's 's data scaling surpass using human demonstrations alone? ($5.3)
3. How does Trajectory Boost differ from and outperform Direct Distillation? ($5.4)
4. How does test-time scaling affect the performance of PC Agent-E? ($5.5)
5. How well does PC Agent-E generalize to unseen environments? ($5.6)
5.1 SETUP
Benchmarks We use WindowsAgentArena- V2 ($4) for the main evaluation, as our training data were collected on the Windows system. We also include results on the original WindowsAgentArres (Bonatti al., 2024) in Appendix for completeness. To test generalization across operating systems, report results on OSWorld (Xie al., 2024), another computer use benchmark for Linux systems.
Model Baseline We compare PC Agent-E with several SOTA models. These include the leading proprietary models Claude 3.7 Sonnet (Anthropic, 2025a) and Claude 3.7 Sonnet with extended thinking (Anthropic, 2025b), as well open-source models UI-TARS (Qin al., 2025), UI-TARS- 1.5 (Team, 2025), and Qwen2.5-VL-722 (Bai et al., 2025). We also compared with the popular GPT-4o (OpenAI, 2024) model.
Method Baseline We compare our Trajectory Boost method with two alternative training ap- proaches. The first is standard behavior cloning on the 312 human trajectories after thought com- pletion. The second direct distillation from Claude. We sample 10 end-to-end trajectories from Claude 3.7 Sonnet for each of the 312 tasks. The resulting 3,120 trajectories are then used for training with the identical procedure as PC Agent-E, matching the trajectory number of our method for fair comparison.
7

---

Published as a conference paper at ICLR 2026
Settings All experiments and models utilized a screenshot-only observation setting with uniform screen resolution of 1280 x 720. For the UI-TARS model series, we adopted their native framework, which supports image history and code block actions. All other models, including Claude and Qwen, were evaluated using our simple PC Agent-E scaffold. The default maximum number of steps was set to 30, and also investigated the impact of varying this step limit model performance.
Training We train our PC Agent-E model based on the Swen2.5-VL-777 (Bai et al., 2025) backbone with 27k data mentioned in $3.5. Experiments on smaller model Qwen2.5-VL-77 - are also included in Appendix C. We set the image resolution to 1280 x 720 and context length to 8,192 tokens. Further training details can be found Appendix
5.2
MAIN RESULTS
As shown in Table 3, PC Agent-E achieves remarkable 141 % relative improvement the base model Qwen2.5-VI.-3 VL.-2BB on WindowsAgentArenaa Claude 3.7 Sonnet by 10% in relative terms (36.0 32.6), establishing itself the SOTA open- source model for Windows computer use. Notably, Claude 3.7 Sonnet used to synthesize our training data did not have the thinking mode enabled, but PC Agent-E achieves performance comparable to the stronger Claude 3.7 Sonnet with extended thinking.
even surpassing the strong teacher model
Models
Libreoffice
Chrome
Edge
System
VS Code
VLC
Utils
Total
Number of Tasks
42
17
13
24
19
14
12
141
GPT-4o
0.0
5.9
0.0
8.3
0.0
0.0
0.0
2.1
Qwen2.5-VL--2.
0.0
34.7
15.4
20.8
26.3
7.6
16.7
14.9
UI-TARS-I.544
7.1
34.7
23.1
45.8
21.1
7.6
16.7
21.3
UI-TARS-72B-DE
0.0
40.6
38.5
58.3
36.8
7.6
25.0
26.2
Claude 3.7 Sonnet
2.4
46.5
61.5
54.2
52.6
29.0
16.7
32.6
Claude 3.7 Sonnet (thinking)
2.4
64.1
46.2
66.7
52.6
21.9
25.0
35.4
PC Agent-E (Ours)
4.8
64.1
46.2
50.0
57.9
35.7
33.3
36.0
Table 3: Results of success rate (%) for different models on WindowsAgentArenna V2.
To gain deeper insight into the specific capabilities enhanced through our training, we conducted qualitative analysis by examining 50 trajectories that Qwen2.5-VL-72B failed but PC Agent-E succeeded, as well as trajectories where both models failed. We categorized the failure patterns into three types: (1) Knowledge: models may lack specific computer use knowledge. For instance, a model might not know how to enable particular feature in VL.C (a media player software). (2) Planning: models may make incorrect planning, such as failing to recognize and recover from previous erroneous actions. (3) Grounding: models may execute actions that are inconsistent with their plan, primarily manifested as mouse-clicking errors. found that our improvements primarily stem from enhanced planning capabilities. After training, PC Agent-E produces noticeably longer thought processes and demonstrates improved reasoning capabilities in verification and celf-correction. We did not observe significant improvements in knowledge grounding capabilities.
Analysis
5.3 DATA SCALING OVER HUMAN DEMONSTRATIONS
To validate the effectiveness of our Trajectory Boost method, we investigate the relationship between the scale synthesized data and model performance. We define data scaling factor, s, the ratio of the total action number used for training to the action number in the original human trajectory. For the model trained exclusively on the human demonstrations, the scaling factor is s Our final trained using 9 synthesized actions and original human action per step.
model, PC Agent-E, corresponding to the scaling factor s + 10.
As the blue line shown in Figure 7, our results reveal that model performance with the Trajectory Boost method scales significantly with the scaling factor. Compared to training on human trajectories alone, which yields limited gain (improved from 14.9 17.2), PC Agent-E achieves substantially greater performance gains (improved from 14.9 to 36.0). This improvement is primarily driven by the diverse action decisions synthesized from the frontier model with thought processes. This supplements the
8

---

Published as a conference paper at ICLR 2026
single uuman-annotated solution and instills the frontier model's advanced planning capabilities into our agent, thereby yielding performance that far exceeds training on human trajectories alone.
5.4 TRAJECTORY BOOST VS. DIRECT DISTILLATION
To demonstrate that our method is than simple form of distillation, we compare Trajec- tory Boost against direct distillation baseline. For this baseline, we directly sample trajecto- ries from our teacher model, Claude 3.7 Sonnet, end-to-end. We also compare with variant baseline that combines human demonstrations and direct distillation, with details provided in Appendix D.
Trajectory Boost
40
Direct Distillation
PCegett-EE
34.0
35,0
36.0
30
Success Rate
25
23.4
276
26.2
17.
22.7
16.3
15
14.9
Qwen2.5-VL-72D
Superior Performance As shown in Figure 7, method significantly outperforms the direct distillation baseline at most of the training data scales (blue line versus orange line). We at- tribute this to the high quality of our synthe- sized data. Our method uses human trajectories as reliable foundation and leverages the fron- tier model perform single-step augmentation. This avoids the error accumulation that can oc- cur end-to-end trajectory distillation.
1
2
8 10
Scaling Factor s
Figure Performance of Trajectory Boost and Di- rect Distillation method with different data scaling factor on WindowsAgentAArena V2.
Exceptional Efficiency Another significant advantage of our method time efficiency. The distillation baseline requires deploying Claude in virtual machines and collecting trajectories through online interaction, which makes desource-intensive and time-consuming. In contrast, our Trajectory Boost method performs offline data synthesis without interacting with the real environment, enabling natural parallelization. Specifically, to collect an equivalent amount (3120 trajectories) of data, the distillation baseline took about 900 hours, while Trajectory Boost required only 3 hours under the same hardware conditions drastic 300-fold speedup.
5.5 TEST TIME SCALING
*
Agent-E
Swve225--VL-228
We also investigate how the performance of PC Agent-E varies with test time scaling, a topic that has received increasing at- tention the research community (Wu et al., 2025; Snell et al., 2024). We evaluated the model's performance with different num- bers of max steps allowed during task completion. As shown in Figure 8. when the step limit increases from 15 to 30, PC Agent-E effectively leverages the additional compute, with the performance gap the base model widening more steps allowed. This suggests that the improved planning capabilities ac- quired through training enable PC Agent-E benefit from longer interaction horizons, We further extend the evaluation to 50- step limit in Appendix E, where we observe slight performance decline and analyze potential causes.
40
*
36.0
30
Sucress Rote
|
26.9
15
14.9
11.3
15
30
Max Steps Allowed
Figure 8: Test time scaling WindowsAgentArena- V2.
5.6 GROSS-PLATFORE EVALUATION
further evaluated our model on OSWorld assess cross-platform generalization capabilities. As shown in Table 4, despite being trained exclusively Windows data, PC Agent-E achieves 34% relative improvement Linux systems as well. These results validate the generalizability of our method.

---

Published as a conference paper at ICLR 2026
We also identified interesting phenomenon in this experiment, which we designate as infeasi- ble hacking in $4: the weaker Qwen2.5-VL-72B model paradoxically achieved markedly better performance infeasible tasks. This observa- tion suggests that current infeasible task eval- uations do not accurately reflect computer use agents' capabilities. Future research may de- sign better criteria for infeasible tasks, such as checking agents' rationale when declaring tasks impossible.
Models
Feasible
Infeasible
Total
Numher of Tasks
339
369
Owen2.5-VL-722
4.4
86.7
11.1
PC Agent-E (Ours)
10.9
63.3
14.9
Table 4: Success Rate (%) OSWorld (30-step).
CONCLUSION
In this work, we introduce PC Agent-E, an efficient training framework for computer-use agents, Our framework leverages Trajectory Boost, a data synthesis method that augments human demonstrations with diverse action alternatives generated by frontier model. Trained on as few as 312 augmented trajectories, PC Agent-E achieves 141 % relative improvement over the base model, even outperforms the strong teacher model Claude 3.7 Sonnet. Ablation studies further demonstrate that this approach is not only significantly superior to using human trajectories alone, but also more efficient and effective than direct distillation from the teacher model. These findings suggest that complex computer-ue capabilities can be effectively elicited from a remarkably small set of high-quality trajectories.
ACKNOWLEDGMENTS
We would like to express our sincere gratitude to Shijie Xia for his meticulous review and constructive suggestions, which significantly improved the quality of this paper. This project is supported by SJTU SEIEE ByteeDacce Large Language Model Joint Laboratory, SII.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Saaket Agashe, Jiuzhou Han, Shuyu Gan. Jiachen Yang, Ang Li, and Xin Eric Wang. Agent An open agentic framework that uses computers like a human. arXiv preprint arX1v:2410.08164, 2024.
Anthropic.
Introducing computer use, 2024. URL 3- 3-5-models- wea-mmeel--andccomiiil wwwplls--anncccuueeeeee-eeee
Anthropic. Claude 3.7 sonnet, 2025a. URL https:/www.arthhoppcccoooo/
3--udde--7-somee
Anthropic. Claude's extended thinking, 2025b. URL
w.sibble-exteedde--hhii
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: ///s://aaae //aaxiv/..iv..// org/ abss /2w.//252220 13923.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-moda. os agents at scale, 2024. URL https: // /arxiv. orw.org//aoooo abs// 2409. 08264.
10

---

Published as a conference paper at ICLR 2026
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-at multimodal models. arXiv preprint arXiv:2409.17146. 2024.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, 2025.
URL hwtps:tt////....... arxii v. org/ abs /242 0.1...000 05243.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giomo, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https: / httpt:h//aaaee /ttp://xxxvvo///////o
abs/2306... 11006.116666 144.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Yanheng He, Jiahe Jin, Shijie Xia, Jiadi Su, Runze Fan, Haoyang Zou, Xiangkun Hu, and Pengfei Liu. Pc agent: While you sleep, works a cognitive journey into digital world, 2024. URL https: /tt::///xxxiv/ org/ wttpabsg////oooooo..ot...o...ace 241222222 17589.
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. Ol replication journey-part 2: Surpassing ol -preview through simple distillation, big progress bitter lesson? arXiv preprint arXiv:2411 16489, 2024.
Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, and Xiang Yue. Harnessing webpage uis for text-rich visual understanding, 2024. URL https: / arxiv. wttp//rgg/////oooooooooo
13824.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint wrXiv:2408.00200 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint vxXiv:2501.9939) 2025.
Shikhar Murty, Dzmitry Bahdanau, and Christopher D Manning. Nnetscape navigator: Complex demonstrations for web agents without demonstrator. arXiv preprint 20x0022100000000009 2024.
OpenAl. Hello gpt-4o. openai.com, May 2024. hello-gpt-40/.
URL http:/ss//ppaaacccoomo/////
OpenAl. Computer-using agent, computerr-using--geen.
2025.
URL https://openaa..cmm//
Tianyue Ou, Frank Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale, 2024. URL https: 15637.
Jayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprinl 13X1 2024.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint 2Xxv:2501.12320 2025.
Jerome H. Saltzer, David P. Reed, and David D. Clark. End-to-end arguments in system design. ACM Transactions Computer Systems, 2(4):277-288, November 1984. doi: 10. 1..114///7700.555 3354402.

---

Published as a conference paper at ICLR 2026
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL BtLLttaaa///xiiyy org / abs/ 2408. 03314.
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis, 2025. URL https:// /arxivv./rxii/////a///i/eaiaii org/ abs 197223....
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpaca: A strong, replicable instruction-followin model. Stanford Center for Research Foundation Models. https://erfm. stanford. edu/20233033/1//appeee html, 3(6):7, 2023.
ByteDance Seed Team. Introducing ui-tars-1.5. seed-tars.com, 2025. URL atttps://settt /sedddttar.. tars. com/ 1. 5/.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- seff-genrrated instructions, 2023. URL https: //tp:v/aaaiii/aa/ org /ab 2212.10560.
Yangzhen Wu, Zhiging Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2025.
URL https: http///avvvv..//////eeeeeee org/ 2208/ 00724.
Zhiyong Wu, Chengeheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456 2024a.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024b. URL https:://// /ttp://aaxvvvvv. org/ abs/2410 23218.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https:://aaattttt org. /bs/ 2404. 07972.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023, URL https:/ //arxivvv... www/oogg//ssoo. 2304. 12244.
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2025a. URL
alss///11111 241 09605.
Yiheng Xu, Zekun Wang. Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2025b.
URL fittps://ariig org/ abs/ 2412 04454.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL //tttps://axiv.. org/abs/ 221022211022 03629.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502. 2025.
12

---

Published as a conference paper at ICLR 2026
A TRAINING DETAILS
The PC Agent-E model fine-tuned on a dataset of 27k samples for 2 epochs using 32 NVIDIA GPUs over approximately hours, with Qwen2.5-VL-72B as the base model. We set the context length to 8,192 tokens, using a batch size of 128 and a learning rate of 2c-6. The training process employs cosine annealing for learning rate scheduling, with warm-up ratio of 0.05. The visual tower is kept frozen throughout the training process.
B EVALUATION ON ORIGINAL ACENTARAA BENCHMARK
We also evaluate PC Agent-E's performance on the original WindowsAgentAreen benchmark. As shown in Table 5, our model greatly supasses the previous SOTA method, NAVI, across all task categories. NAVT is complex agent framework that integrates GPT-4V (Achiam et al., 2023) with specialized tool called Omniparser (Lu al., 2024).
Method
Office
Web
System
Coding
Media Video
Utils
Overall
NAVI (Bonatti al., 2024)
0.0
27.3
33.3
27.3
30.3
8.3
19.5
PC Agent-E (Ours)
2.3
33.1
70.6
37.5
33.3
25.0
27.9
Table 5: Success Rate (%) on the original WindowsAgentArrena benchmark.
C EXPERIMENT ON SMALLER MODEL
To test our method on smaller models, we also trained Qwen2.5- VL-7B (Bai et al.. 2025) on the same 27K dataset for PC Agent-E, resulting the PC Agent-E 7B. As shown in Table 6, our method yields improvements are not as significant as those observed with the 72B model. This is because method mainly enhances the model's planning ability, as discussed in $5.2, but the small model's deficiencies in knowledge and grounding limit the overall performance gain.
Method
Overall
Qwen2.5-VI--7B
5.0
the 7B model as well, although the gains
PC Agent-E 7B
6.4
Table 6: Success Rate (%) on WindowsAgentArena-
D HUMAN DATA DIRECT DISTILLATION BASELINE
To further validate the effectiveness of Trajectory Boost, compare against baseline that combines human demonstrations with direct distillation. Specifically, the base model is trained using 312 human trajectories together with 9x312 trajectories sampled end-to-end from Claude 3.7 Sonnet. As shown in Table 7, this combined baseline achieves a score of 26.9, comparable to direct distillation alone (26.2), yet significantly lower than PC Agent-E (36.0).
LibreOffice
Chrome
Edge
System
VS Code
VLC
Utils
Total
Number of Tasks
42
17
13
24
19
14
141
Direct Distillation
2.4
28.8
30.7
50.0
57.9
7.6
25.0
26.2
Human Data Direct Distillation
2.4
22.9
30.7
54.2
57.9
21.6
26.9
Agent-E (Ours)
4.8
64.1
46.2
50.0
57.9
35.7
33.3
36.0
Table 7: Comparison with Human Data + Direct Distillation baseline on WindowsAgentArene-
13

---

Published as a conference paper at ICLR 2026
E EXTENDED TEST-TIME SCALING: 50-STEP EVALUATION
In addition to the 1t-step and 30-step evaluations presented Figure 8, we conduct an extended evaluation with 50-step limit to further investigate the test-time scaling behavior of PC Agent-E. As shown in Table 8, both PC Agent-E and the base model Qwen2.5-VL-72B exhibit performance drops when the step limit is extended 50.
Model
15-step
30-step
50-step
Qwen2.5-VL-72B
11.3
14.9
11.9
PC Agent-E
26.9
36.0
31.4
Table 8: Success Rate (%) on WindowsAgentArenn V2 under varying step limits. Performance improves from 15 30 steps but degrades at 50 steps for both models.
Analysis. We identify that the performance degradation 50 steps primarily caused by the models" inability to correctly recognize task completion. Without robust stopping mechanism, additional steps allow the agent to continue acting after a task has already been successfully completed, often leading to actions that undo or disrupt the correct result. We attribute this to the training data distribution (see Table in $3.2), where over 90% trajectories contain fewer than 15 steps, leaving the model with limited exposure to long-horizon exccution and insufficient supervision for task termination.
Scaling within and beyond the training distribution. Combining the results in Table 8, observe two distinct regimes. Within range reasonably close to the training distribution (15 - 30 steps), our method demonstrates effective test-time scaling, with performance improving as more steps are allowed. However, when the step limit is pushed significantly beyond the training distribution (50 steps), performance degrades due to the lack of termination awareness. This suggests that the current training paradigm supports bear-distribution test-time scaling but does not yet generalize to substantially longer horizons.
Future directions. We hypothesize that addressing this limitation requires either (1) incorporating longer trajectories into the training data to extend the model's effective operating range, or (2) intro- ducing targeted supervision for task completion detection, such an explicit termination prediction objective. We leave the exploration of these directions to future work.
14

---

Published as conference paper at ICLR 2026
F PROMPTS
F.1 THOUGHT COMPLETION
Table 9: Thought Completion Prompt
Prompt for Thought Completion
You are a helpful computer use agent designed to complete tasks on computer. Your goal is to recreate your thought process behind specific actior.
You will be provided with:
1. The task are attempting to complete.
2. A history of the steps you have already performed (up to 50, if any;
none was the first action).
3. The specific act you chose to take.
4. The name of the element you clicked (if you clicked an element). It might be too general vague, you have to decied what to click based on the screenshot.
5. A screenshot of the computer screen at the moment you deci ded te take the action.
6. The red marks on the screenshot indicate the position of the click or drag action.
To formulate your thought process, consider:
1. what do you observe the screen? Consider your task and previcus action when you analyzing current screenshot.
Evaluate your previous action (if :pplicable):
Did it achieve the intended effect? If not, identify possible reasons (e.g., misclick, inactive element).
Some typical examples for ineffective actien:
miselick in an empty space
ineffective opening some elements without double click
ineffect i ivve type text press key because of inactivated input box
Did the result align with your previous plan, did something unexpected happen?
Some typical examples for ineffective action:
misclick in wrong element
forget to clear existing text in input bar
3. Based on your action history, assess your progress toward completing the overall task.
4. Consider if you' re exploring how to finish the task because of failed attempts in history steps.
Present your thougat process as a clear, natural -irstnn narrative that explains your reasoning at that moment.
Important requi rements:
1. *DO N mention the red marks in your response. These marks were **added after the fact**
to indicate the position your click drag actions, and they not the screen when you made the decision. *xDO NOI** mention "red box", "red squaro", "red circle", or "red arrow" in your response.
15

---

Published as conference paper at ICLR 2026
2. Write as if you are thinking in real-time before taking the action. Do not include post-action evaluation bindsight.
The task you are attempting to complete: [task dessskdeccriptonn Your performing history: (history_strr The specific action you chose to perform: {action)
F.2 TRAJECTORY BooST
Table 10: Trajectory Boost Prompt
Prompt for Trajectory Boost
You are helpful assistant who can help users complete computer tasks, with **full permissions to make any operations the user' computer. The operating system is windows.
Based on the provided current state, you need to suggest the next action to complete the task. Do not try to complete the entire task in one step, Break it down into smaller steps, and at each step you will get new state to interact with.
IMPORTANT: You must strictly adhere to the following rules:
1. Choose ONLY ONE action from the list below for each response, DO NOT perform more than one action per step.
2. Follow the exact syntax format for the selected action, DO create
or use any actions other than those listed.
3. Once the task is completed, output action finish.
Valid actions:
1. click (x, y)
click the element at the position (x, y) on the screen
2. right click ix, y)
right click the element at the position (x, y) on the screen
3. double click (x, y) double click the element at the position (x, y) on the screen
4. drag from (x1, yl) to (x2, y2) drag the element from position (xl, yl) to (x2, y2). scroll (x)
scroll the screen vertically with pixel offset x. Positive values of 6. press key: ent
x: scroll up, negative values of x; scroll down.
press the key key_content the keyboard.
hotkey (keyl, key2)
press the hotkey composed of keyl and key?.
8. hotkey (keyl, key2, key3)
press the hotkey composed of keyl, key2, and key3.
16

---

Published as conference paper at ICLR 2026
9. type text: text_content.
type content text_content on the keyboard. Note that before typing text, you need to ensure the text box input field is aetive/focuse first. If the text box is not yet activated, you should first click on to activate it, and then use type text separate step.
10. wait
wait for time, usually for the system to respond, screen to refresh, advertisement to finish. 11. finish
indicating that the task has been completed.
12. fail
indicating that the task has failed, of this task is infeasible because not enough information is provided.
Before deciding your next action, you should think carefully about the current state of the screen and your history steps. Contain the follewing points your thought precess:
1. What do you observe on the screen? Consider your task and previcus actien when you analyzing current screenshot.
What's your previous plan and action (if applicable)s Evaluate your previous plan and action in three conditions:
1. It didn't t make any effect. You should dentity possible reasons (e.
g., mi mellclin misciick, ick, inactive el element) and adjust your plan in this step.
Some typical examples for ineffective actien:
misclick in empty space
ineffectiv ive opening some elements without double click
ireffective type text/ / press key because of inactivated input box
2. made effect, but the result does not align with previous plan. You should dentify possible reasons (e.g., misclick, inactive element) and correct it in this step.
Some typical examples for ineffective acticn:
-mssclick in wrong element
forget clear existing text in input bar
made some effect, and it successtully align with previous plan.
You should progress to the next step based on the current state.
3. Based your action history, assess your progress toward completing
the overall task.
4. Exploring new ways finish the task there are already failed
attempts in history steps. +DD0 NOT repeat** the history actions.
Respense Format: Your thought inb.cctinnnnttionn The specific action you choose to take.
The task you are attempting to completo: (task_description) Your performing history: (bistory_str) Given the screenshot below. What's the next step that you will do to help with the task?
17

---

Published as conference paper at ICLR 2026
F.3 PC AGENT-E SCAFFOLD
Table 11: PC Agent-E scaffold Prompt
Prompt for PC Agent-E scaffold
You are helpful assistant who can help users complete computer tasks, with **full permi s make any operations the user' computer. Based on the provided current state, you need to suggest the next action to complete the task. Do not try to complete the entire task in one step. Break it down into smaller steps, and at each step you will get new state to interact with.
IMPORTANT: ; You must strictly adhere to the following rules:
1. Choose ONLY ONE action from the sist belor oelow for each response, DO NOT
perform more than one action per step.
2. lowlow the exact syntax format for the selected action, D0 NOT create any actions other than those listed.
3. Once the task is completed, output action finish.
Valid actions:
1. click (x, y)
click the element at the position (x, y) on the screen
2. right click (x, y)
right click the element at the position (x, y) on the screen
3. double click (x, y)
double click the element at the position (x, y) on the screen
4. drag from (xl, yl) to (x2, y2)
drag the element from position (xl, yl) (x2, y2).
5. scroll (x)
scroll the screen vertically with pixel offset Positive values of scroll up, negative values of scroll down.
6. press key; key_content
press the key key_content on the keyboard.
7. hotkey (keyl, key?)
press the hotkey composed of keyl and key2.
8. hotkey (keyl, key2, key3)
press the hotkey composed keyl, key2, and key3.
9. type text: text_content
type content text_content on the keyboard.
10. wait
wait for some time, usually for the system to respond, screen to refresh advertisemert to finish.
11. finish indicating that task has been complated.
12. fail
indicating that the task has failed, of this task is infeasible because not enough information is provided.
18

---

Published as conference paper at ICLR 2026
Respense Format: (Your thought process)
Actien: (The specific action you cheese to take)
Your task is: (task_descrie description)
History of the previous actions and thoughts you have done to reach the current screen: (history_str)
Given the screensact, what's s the next step you will to help with the task?
19

---

Published as conference paper at ICLR 2026
G PC TRACKER USER MANUAL
1. INTRODUCTION
PC Tracker a lightweight infrastructure for efficiently collecting large-scale numan-computed interaction trajectories. The program runs seamlessly in the background, automatically capturing screenshots and keyboard & mouse activities.
2. INSTALLATION
Ensure your operating system is Windows.
Extract our software package to a location with sufficient disk space (recommended to have more than 3GB available space for storing recorded data).
3. QUICK START
(Optional) Set screen resolution 16:9.
Open the extracted folder and launch ma ii.inn. exe.
4. INSTRUCTIONS
After launching the tracker, you can choose between Task Oriented Mode or Non-Task Ori- ented Mode for recording.
TASK ORIENTED MODE
This mode is divided into two sub-modes: Given Task and Free Task.
Given Task this mode, you will be assigned an uncompleted task each time.
Next Task: Click Next Task to get the next task.
Previous Task: Click Previous Task to return the previous task.
Bad Task Feedback: If you think the current task is difficult to complete, click
[ad Task discard it permanently. Alternatively, you can start the task and modify its description after completion based on your actual execution.
Start Recording: Click Start) and the tracker window will automatically minimize while recording begins.
End Task: After completing the task, click [inish) save the record. Or if the task execution fails or you don't want record it, click [Fall
Modify Task Description: After finishing the task, you can modify the task description based on your actual execution.
Free Task In this mode, you can freely use the computer and summarize the task description and difficulty yourself.
Click Start) and the tracker window will automatically minimize while recording begins.
Start Recording:
Save and Summarize This Record: Fill in the task description, select difficulty (easy/medium/hard),, and click (Saee) save the record.
Discard This Record: Click Discarrd
discard the record.
MON-TASK ORIENTED MODE
In this mode, you can freely use the computer, with similar methods to start and stop recording described above.
20

---

Published as conference paper at ICLR 2026
5. USAGE NOTES
Does not currently support using extended screens.
Does not currently support using Chinese input methods.
Does not currently support using touchpads.
The tracker window is fixed in fullscreen. To support the filtering of tracker-related actions (such as clicking the Start button) in post-processing, the tracker window fixed in fullscreen. You reopen the tracker window by clicking to view the task de- scription, then minimize it again, but please do not drag it to display in a non-fullscreen state.
DATA PRIVACY
After starting recording, your screenshots and keyboard & mouse operations will be automatically recorded. PC Tracker does record any information from unopened software. If you believe the recording may infringe on your privacy, you can choose to discard the record.
Collected data is saved in the /events folder (hidden by default). Each trajectory includes Markdown file for easy visualization.
7. FAQ
Does the tracker have networking capabilities? PC Tracker is completely local, does not support networking, and will not upload your data.
What if my computer screen resolution is not 16:9? If your screen resolution is not 16:9, will affect the subsequent unified processing of data. We recommend adjusting your screen resolution 16:9.
How much space does the collected data occupy? The specific data size varies. Generally, even with intensive recording operations for hour, it will not generate more than 1GB of data.
21

---

Published as conference paper at ICLR 2026
H THE USE OF LLMS
We used LLMs to improve the grammar, clarity, and overall readability of this paper. All re- search ideas, content, and scientific contributions were developed and written by the human authors. All suggestions from LLMs were reviewed and edited by the authors, who retain full responsibility for the final content of this paper.
22


Reviews:
[{"review_id": "OwFBo6jr21", "reviewer": "Reviewer_yuwo", "summary": "Paper collects a dataset in a computer use domain and uses it to fine-tune QWEN with the goal of improving computer-use ability. It proposes TrajectoryBoost as a method to augment the few human demonstrations, and finds that their method outperforms direct distillation of Claude.", "questions": "see above", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Relevant problem of efficiently using small sets of expensive human data traces", "weaknesses": "- Evaluation only on a single and self-modified benchmark\n- What about other benchmarks like OSWorld?\n- Direct distillation from Claude not a fair baseline. It should be human data + direct distillation from Claude.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJpfwBxr59", "reviewer": "Reviewer_UcUS", "summary": "This paper introduces PC Agent-E, an efficient training framework for developing computer use agents with small amounts of human annotated data. The key idea of the paper is to use initial human demonstrations as seed data and bootstrap multiple trajectories demonstrating different ways of completing the same task synthetically by treating individual states from human trajectories as initial states and executing a powerful teacher agent starting this state. Through this approach we can spawn a computer use agent at intermediate steps of trajectories and roll it out to collect task execution trajectories that follow different paths. The paper shows this approach enables achieving state-of-the-art performance with minimal human annotation—just 312 human-demonstrated trajectories collected in a single day by two annotators.\n\nThe framework consists of 3 components:\n\n1. Trajectory Collection: Gathering a small set of human computer use trajectories using PC Tracker, which records screenshots and keyboard/mouse actions for various Windows tasks.\n2. Thought Completion: Reconstructing the implicit reasoning process behind each human action using Claude 3.7 Sonnet, converting raw action sequences into trajectories with explicit thought traces.\n3. Trajectory Boost: Augmenting each trajectory step by sampling 9 alternative action decisions from Claude 3.7 Sonnet. This creates a \"Traj Tree\" where the human trajectory forms the trunk and synthesized alternatives branch as leaves, resulting in 27K training samples from the original 312 trajectories.\n\nPC Agent-E achieves a 141% relative improvement over its base model and surpasses Claude 3.7 Sonnet by ~4% on WindowsAgentArena-V2, a benchmark the authors improved to address evaluation dependencies, infeasible task hacking, and other limitations. The model also generalizes to Linux environments (OSWorld) despite being trained only on Windows data.", "questions": "Mentioned in Weaknesses section.\n\nOverall I believe the contribution and experiments in the paper are interesting and valuable. However there are a few experiments and analysis that are missing or need to be added to make the paper stronger. I would be happy to increase my rating if authors address my concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is a nice way to train performant agents under limited data regime where we can collect small amounts of human demonstrations for each task and augment them synthetically to scale up demonstrations.\n2. The additional contribution of a new evaluation benchmark WindowsAgentArena-v2 is valuable to the community and helps improve the evaluation benchmarks overall by fixing existing issues.\n3. The analysis presented in section 5.3 is quite insightful. It highlights the fact that simply collecting “diverse” trajectories by high temperature sampling from initial state performs worse than spawning the CUA agent at intermediate states from a high quality demonstration which is quite interesting result.\n4. Additional results on cross-platform evaluation are also interesting, demonstrate usefulness of the dataset, and are a valuable addition to the paper.", "weaknesses": "1. The caption of table 2 has a typo in difference between performance of teacher model Claude Sonnet 3.7 vs PCAgent-E. The text says the difference is 10% whereas it is ~4%. Authors need to fix this issue in the claims.\n2. The test-time scaling results presented in section 5.5 seem incomplete to me. For main experiments in table 1 authors use max 30 step limit but in test time scaling experiments the two values use for step limit is 15 and 30 which seems counter intuitive. I request the authors to rerun the experiments with 50 step and 100 step limits to comprehensively evaluate whether the test-time scaling behavior exists in the finetuned agent or not. Eventhough current results demonstrate scaling from 15 to 30 steps it is unclear whether that is due to trajectory length being in distribution to the fine-tuning dataset or due to actual test-time scaling behavior.\n3. It would also be good if authors can compare all methods in table 1 on WindowsAgentArena-v1 benchmark too so that we have a complete picture of how much improvement does the dataset lead to across both the setups and for fair comparison.\n4. One question I have for authors is: It is interesting that simply scaling number of demonstrations for small amounts of unique tasks leads to massive improvements (~14 → 36) on WindowsAgentArena-v2 benchmark. Did authors do any analysis on identifying what is the primary cause for this improvement? Even though a single task can be completed in many ways I would expect the benchmark performance to saturate  as you scale the scaling factor (as in figure 7). However the figure shows improving trend for the PCAgent-E. I am curious to see if simply scaling demonstrations for this limited amount of tasks is leading to skill transfer across tasks or could it just be that some tasks are harder to learn and therefore require more demonstration instances per task. It would be a valuable analysis if authors can add to the paper as it will help understand the reason for gains better.\n5. I would also like to see some analysis on task overlap for these 312 tasks with tasks in WindowsAgentArena-v2 so that we can understand how much train/test overlap is there.\n6. There is a analysis section in 5.2 however I could not find the relative figure or table that analysis is talking about in the paper. It would be good if authors can either add it to supplementary or elaborate more on the failure mode breakdown and metrics in that section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zKOxjzygX8", "reviewer": "Reviewer_Wb6f", "summary": "The paper introduces **PC Agent-E**, an efficient training framework for “computer-use” agents that aims to reduce reliance on large, costly human demonstration corpora. Starting from **312** human-annotated trajectories collected with a screen/interaction logger, the authors (1) reconstruct the missing step-wise reasoning via **Thought Completion**, and (2) use a strong teacher model (Claude 3.7) to synthesize **diverse alternative actions per state** (**Trajectory Boost**), yielding a dense, single-step supervision signal without online execution of the synthetic branches. A single agent is then trained end-to-end on the augmented trajectories.\n\nOn an improved benchmark the authors release, **WindowsAgentArena-V2**, PC Agent-E reports a **36.0%** overall success rate, **surpassing Claude 3.7 (with extended thinking)** and substantially improving over a Qwen2.5-VL-72B baseline (reported as **14.9% → 36.0%**, i.e., ~**141%** relative gain). The paper also shows positive transfer to **OSWorld**, suggesting cross-platform generalization. The project page and repository provide code, data, and deployment scripts, and outline the four-stage pipeline (collection → thought completion → trajectory boost → training).", "questions": "* **Evidence for “efficient training.”**\n  Prior work (e.g., OpenCUA, UI-TARS) achieves capable agents mainly via large-scale data. Your claim that small-data, step-wise augmentation suffices is counter-intuitive. Could you provide stronger, targeted evidence that *efficiency*—rather than hidden data or compute—drives the gains?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* **Better benchmark signal (quality & significance).**\n  Through human verification and fixes, the paper delivers a cleaner **WindowsAgentArena-V2** benchmark that reduces evaluation pathologies and provides clearer training/evaluation signals for computer-use agents.\n\n* **Strong results with little data (originality & significance).**\n  Despite limited human trajectories, the proposed training pipeline achieves **state-of-the-art performance on WindowsAgentArena**, allowing an open-source model to **match or surpass Claude 3.7 Sonnet**, highlighting a practical, data-efficient path for competitive open agents.", "weaknesses": "* **Efficient training lacks out-of-domain generalization.**\n  While the method reaches ~35% on WindowsAgentArena (on par with Claude 3.7), it drops to **14.9% on OSWorld**, whereas **Claude 3.7 is ~35%** under a 50-step cap. This gap indicates the approach chiefly captures Windows-specific patterns rather than learning transferable skills—i.e., likely **in-domain overfitting** despite the “efficient training” claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)