Title: Efficient Agent Training for Computer Use
Primary area: foundation or frontier models, including LLMs
Keywords: Agents; Computer Use;  Large Language Models; Vision Language Models

Abstract: Scaling up high-quality trajectory data has long been a critical bottleneck for developing human-like computer use agents. We introduce PC Agent-E, an efficient agent training framework that significantly reduces reliance on large-scale human demonstrations. Starting with just 312 human-annotated computer use trajectories, we further augment them by synthesizing diverse alternative action decisions with Claude 3.7 Sonnet. Trained on these enriched trajectories, our PC Agent-E model achieved a remarkable 141% relative improvement, and even surpassed the Claude 3.7 Sonnet by 10% on WindowsAgentArena-V2, an improved benchmark we also released. By integrating robust human computer use skills with automated AI data synthesis capabilities, our method not only brought substantial improvements over training on human trajectories alone, but also significantly surpassed direct distillation from Claude 3.7 Sonnet.

TL;DR: PC Agent-E demonstrates efficient agent training with a small set of human trajectories augmented with Claude 3.7 Sonnet, achieving 141% improvement and surpassing Claude 3.7 Sonnet by 10%.