AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
Authors: Jingxu Xie
Dylan Xu
Xuandong Zhao
Dawn Song
Venue: ICLR 2026 Poster (2026)
Keywords: Synthetic data
Computer-use agents
Scalable
Primary area: foundation or frontier models, including LLMs
TL;DR: We present AgentSynth, a scalable pipeline that automatically generates diverse and realistic computer-use tasks and trajectories.

Abstract: We introduce AgentSynth, a scalable and cost-efficient pipeline for automatically synthesizing high-quality tasks and trajectory datasets for generalist computer-use agents. Leveraging information asymmetry, AgentSynth constructs subtasks that are simple during generation but significantly more challenging when composed into long-horizon tasks, enabling the creation of over 6,000 diverse and realistic tasks. A key strength of AgentSynth is its ability to precisely modulate task complexity by varying the number of subtasks. Empirical evaluations show that state-of-the-art LLM agents suffer a steep performance drop, from 18\% success at difficulty level 1 to just 4\% at level 6, highlighting the benchmark's difficulty and discriminative power. Moreover, our pipeline achieves a low average cost of \$0.60 per trajectory, orders of magnitude cheaper than human annotations. Our code and data are available at https://github.com/sunblaze-ucb/AgentSynth

Published as a conference paper at ICLR 2026
AGENTSYNTHE SCALABLE TASK GENERATION FOR GENERALIST COMPUTER-USE AGENTS
Jingxu Xie, Dylan Xu, Xuandong Zhao; Dawn Song
University of California, Berkeley (j Inggxxie e, anx26, xuandongzhao, dawnsong) @berke www.keley.eeee......... edu
ABSTRACT
We introduce AgentSynth, scalable and cost-efficient pipeline for automatically synthesizing high-quality tasks and trajectory datasets for generalist computer- use agents. Leveraging information asymmetry, AgentSynth constructs subtasks that are simple during generation but significantly more challenging when com- posed into long-horizon tasks, enabling the creation of over 6,000 diverse and realistic tasks. A key strength of AgentSynth is its ability to precisely modu- late task complexity by varying the number of subtasks. Empirical evaluations show that state-of-the-art LLM agents suffer steep performance drop, from 18% success at difficulty level to just 4% at level 6, highlighting the bench- mark's s difficulty and discriminative power. Moreover, our pipeline achieves a low average cost of $0.60 per trajectory, orders magnitude cheaper than hu- man annotations. Our code and data are available at https: : / / figittuu thub. com/ sunbl aze- ucb/ wwwpaeeeeeeetttet/////re Agent.ynth.
INTRODUCTION
Large language models (LLMs) have recently shown promise autonomous agents capable of solving complex, multi-step tasks across a wide range of domains. These LLM agents interact with an environment through structured actions such as mouse clicks, keystrokes, or code executions, and are prompted to complete specific tasks using tools provided by the interface. This paradigm has been explored for web navigation tasks (Yao et al., 2023a; Drouin et al., 2024; Furuta et al., 2023), software development (Yang et al., 2024), formal mathematics (Lin et al., 2025), and many others (Boisvert et al., 2024; Agashe et al., 2025). research in LLM agents progresses, the availability of high-quality datasets tailored to these domains becomes increasingly critical.
General computer-use use tasks that involve interacting with desktop environments and software appli- cations pose especially difficult challenges for data collection. Existing datasets in this space, such as 7-bench (Yao et al., 2024), TheAgentCompany (Xu et al., 2024), OSWorld (Xie et al., 2024), WorkArena (Drouin et al., 2024) rely heavily on human demonstrations and tasks. While effective in showcasing agent capabilities, this numan-in-the-lope approach is labor-intensive, expensive, and fundamentally unscalable, making impractical for covering the full breadth of real-world computing scenarios.
limited set of tools
To overcome these limitations, recent work has turned to synthetic data generation using LLMs. However, existing pipelines face two core challenges: (1) current LLM agents struggle to generate reliable trajectories for complex tasks, and (2) simplistic task diversity. These challenges are especially acute in visually grounded or long-horizon tasks, where agents must maintain contextual awareness, reason over multiple steps, and adapt when plans fail (Xie et al., 2024; Bonatti et al., 2024). Moreover, limited task diversity increases the risk of overfitting or model collapse during downstream training (Shumailov al., 2024).
repetitive generation strategies limit
We introduce AgentSynth, scalable and flexible pipeline for synthesizing diverse, high-quality datasets for training and evaluating computer-use agents. The core insight behind AgentSynth is to exploit information asymmetry between the data generation and evaluation phases, the idea that solving a task step-by-step in the forward direction is far easier than reasoning out the entire solution
"ooresponning author

---

Published as a conference paper at ICLR 2026
all at once. Therefore, we construct the task through sequence of simple, solvable subtasks. Each subtask builds incrementally on the prior state, with the corresponding trajectories collected during execution. A summarization agent then merges the subtasks into composite long-horizen task, producing realistic scenarios that are easy to generate but hard to solve.
This design offers several key advantages. By constructing complex tasks from simple, solvable components, AgentSynth enables reliable trajectory collection while maintaining benchmark diffi- culty. Varying the chaining of subtasks induces combinatorial task diversity. The pipeline is fully automated and achieves a low cost of just $0.60 per trajectory. While we generate over 6,000 tasks in this work, the approach readily scales to tens of thousands of realistic tasks across diverse envi- ronments, unlocking significant potential for agent training and evaluation.
Our contributions are as follows:
We introduce AgentSynth, a fully automated pipeline that synthesizes challenging and diverse computer-use tasks by iteratively chaining LLne-eenrated subtasks
We demonstrate how information asymmetry between generation and execution improves trajec- tory reliability and task complexity, enabling fine-gratn tne-grained task difficulty control.
We build a benchmark using AgentSynth and show that state-of-ther ofrt agents struggle signifi- cantly, revealing a large room for future improvement.
We describe our methodology in detail in Section 3. analyze the generated tasks and datasets in Section 4, and present empirical evaluation results in Section 5.
RELATED WORK
Substantial research has focused synthesizing data improve the training and evaluation of LLMs. However, most existing datasets and benchmarks for computer-use use agents still rely heavily on manual design and annotation, limiting their scalability and diversity.
Synthetic Data Generation. Synthetic data generation has emerged as a promising approach to enhance model performance and foster new capabilities. Many recent studies have leveraged LLMs to automate and diversify data generation. For instance, Yuan et al. (2025) curated diverse, high- quality datasets extracted from extensive pretraining corpora. Xu et al. (2025a) (Evol-Instruct), Su and Sun et al. (2025) (OS- both use sequential pipelines
et al. (2025) LLearn-by-interaee to generate synthetic datasets. However, OS-Genesis and -earn-by-interact retroactively define task over trajectory instead of stringing together subtasks, while Evol-Instruct only generates the trajectory with the final instruction. Shin et al. (2019) generated synthetic datasets with controlled distributions over programs and specifications. Li et al. (2025) employed an optimization loop where a data generator continuously produces challenging problems targeted at specific evaluation models. Many other applications of synthetic data for LLMs are listed in Liu et al. (2024). These works highlight the power of synthetic pipelines but focus primarily on static text benchmarks rather than interactive agents.
Agent Datasets and Benchmarks. Current datasets and benchmarks for agents predominantly depend on human annotators for task creation, demonstration provision, and the definition of eval- uation metrics (Yao et al., 2024; Xu et al., 2024; Zhou et al., 2024), which are costly to scale and often limited in diversity. More recent work explores using LLMs to generate agent tasks and tra- jectories. For example, Pahuja et al. (2025); Trabucco et al. (2025); Murty et al. (2025); Gandhi & Neubig (2025) employed LLMS as web agents synthesize web-based interactions. Boisvert al. (2024) composed atomic tasks from Drouin et al. (2024) to form difficult tasks. Xu et al. (2025b) and Ou et al. (2024) turned online tutorials into tasks and demonstrations, Nonetheless, these gener- ated tasks and trajectories are limited primarily to web-based activities and typically involve simple interactions without complex multi-step reasoning or extensive tool utilization.
Agent Environments. Early agent environments such as MiniWob++ + (Liu et al., 2018) focused on simplified web tasks and low-level actions. Later advancements such as Mind2Web (Deng et al., 2023), WebArena (Zhou et al., 2024), and Online-Mind2Wal (Xue et al., 2025) introduced more realistic websites but remained constrained in breadth and complexity. More comprehensive envi- ronments have been developed by Yao et al. (2024); Drouin et al. (2024); Xu et al. (2024) which expanded the action space and interface diversity, yet they still deviate from the actual computer environments. Recent developments like OSWorld (Xie et al., 2024) and Aren (Bonatti
2

---

Published as a conference paper at ICLR 2026
Persona
open prison and evidence: to support policy chonges
convincing
Initial task gemeration
Initial task
Search for recent recidiviam rote statistics following prison reform initiatives the United States.
Follow-up task generation
Follow-up subtask
Create a text file summanizing and companing historical and current recidivism rate statistics in the United States.
Verify lask successfully execcuted
Yes
No
Summarize what agent has
historical (pre (ore-neform) recidivism rate statistics United States to compare with current post reform rates.
Revised subtask description
recidivism statistics in 1980s, 1990s, 2000s, and 2010s using online sources, enter this data into LibreOffice Cole spreadsheet with co columns labeled 'Period" 'Recidiviam chart titled "US Recidivism Rates by becade with labeled 'Decade' and the oxis labeled 'Racidivism Rate (%) to visually compare changes recidivism rates these decades.
Summorize the sequence of subtesks
and
clearly-lbeeled line axis
Final tasks
Initial state
Initial tack
Follow-up task
Fallow-up task
Task daskdiffiiulty level
Task difficclly level
Task difficullt level n+1
Figure I: AgentSynth data generation pipeline. Given persona, the task proposer generates ini- tial task, which is followed by sequence of subtasks executed by the agent. Each step is verifie;; execution fails, revised subtask description is generated. After successful steps, a summarization agent composes final high-level tasks. Tasks at different difficulty levels are formed by summarizing the first n subtasks, enabling controllable task complexity.
et al., 2024) address this gap by transforming real operating systems into interactive gym environ- ments for agent training and trajectory generation. Our work leverages the capabilities OSWrrld, providing comprehensive access to authentic computer tools to enhance synthetic data generation for generalist computer-use agents.
3 SCALABLE AGENT TASKS AND TRAJECTORIES GENERATION
We design a synthetic data generation pipeline powered by six distinct LLM-based agents: task proposer, task executor, task verifier, a task reviser, a follow-up task proposer, and task sum- marizer. Central to our methodology is the exploitation of information asymmetry (Li et al., 2025); the idea that solving task step-by-step in the forward direction is far easier than inferring the entire solution from scratch. Specifically, we generate sequences of simple, tractable subtasks, collecting trajectories along the way, and later summarizing the sequence into a single, coherent long- morizoo task. This approach allows to synthesize tasks that are easy to generate but substantially more difficult for agents to complete at test time. Full prompt templates are included in Appendix B.
Our pipeline operates in the OSWorld environment (Xie et al., 2024), Gym-compatible simulated desktop interface that mirrors real-world computer usage. Within this environment, agents can inter- act freely with broad range of software applications and system tools hosted on a virtual machine. At each step, the agent receives a full-screen screenshot (1920 x 1080), typically spanning 1k-2k tokens depending the model's tokenizer. Based this visual context and the current task, the LLM agent generates executable actions, such as mouse clicks, key presses, text input, and scrolling, which executed using pyautogui (Sweigart, 2025) to closely emulate human behavior. The

---

Published as a conference paper at ICLR 2026
full action space is detailed in Table G.I, where the percentage listed for each action type reflects its frequency of occurrence across all trajectories in our dataset. To highlight the generality of our pipeline, also apply it to web agent environment (InSTA (Trabucco et al., 2025)), as discussed in Appendix F. The overall data generation pipeline is detailed below and is presented in Figure 1.
(1) Task Proposer. We initiate the data generation process by instructing task proposer agent to generate an initial, straightforward task. To enrich task diversity, the proposer is guided by a randomly assigned persona sampled from the persona hub (Ge et al., 2024), prompting it to suggest tasks relevant to a specific user profile. The proposer takes as input the persona and the initial Ubuntu desktop screenshot, and is prompted atomic actions. To ensure safety and privacy, we prohibit any tasks involving login credentials or actions such as email sending. Prompt details are provided in Appendix Table B.1.
create clear, specific tasks that can be completed in a few
We currently rely on GPT-4.1-basee agents for task generation due to their robustness and broad generalization. Different LLM models might generate tasks with systematically different complex- ity, realism, or meaningfulness, and it remains an open and interesting research question how model choice affects generated task properties. Exploring task aask-geneaation variance across different LLM architectures could be beneficial to further enrich task diversity and calibrate difficulty precisely.
(2) Task Executor. To execute the proposed tasks, we construct a ReAct-style (Yao et al., 2023b) agent that integrates OpenAl's GPT-4.1 (OpenAI, 2025b) and computer-use-pree peeevieeeewwww (OpenAI, 2025a) models. Empirically, GPT-4.1 is good at planning and interpreting visual context, while the computer-use model is more accurate in grounding actions to pixel-level coordinates. We there- fore assign GPT-4.1 the role of planner: it receives the task, current screenshot, and execution his- tory, and outputs natural language description of the next action. This description, along with the screenshot, is then passed to the computer-use model, which generates the precise executable action (e.g., mouse click coordinates, keystrokes). This two-stage setup balances high-level reasoning with fine-grained visual grounding. During execution, we log both the model's reasoning trace and the resulting actions, enabling rich trajectory annotation. Each task execution is limited to a maximum of 10 steps. The prompts for the task executor are shown in Appendix Table B.2 and Table B.3.
(3) Task Verifier. The task verification agent evaluates whether a given trajectory successfully completes the intended task. reviews the full screenshot sequence and task description, and outputs both a binary success label and a completion percentage. To avoid overwhelming the verifier with excessive visual input, we adopt Webludge-style architecture inspired by Xue et al. (2025). The verifier first extracts key requirements from the task description, then analyzes each screenshot to select subset of key screenshots most relevant to task completion. The final verdict is made based on the task description, identified key requirements, and the filtered key screenshots. To reduce token usage, all screenshots are downsampled to 480. If task is not fully completed, the verifier estimates the percentage of task completion. In such cases, the task reviser generates a revised task description that reflects the actual progress. Prompt details for the verifier are provided in Appendix Table B.5 to Table B.7.
When trajectory is only partially successful, we invoke task reviser agent to generate revised task description that accurately reflects the actions actually completed by the agent. The reviser takes as input the full execution screenshots and identifies the goals that were successfully accomplished. It then outputs revised task description that aligns with the observed behavior. Prompt details for the task reviser are shown in Appendix Table B.S.
(4) Task Reviser.
(5) Follow-up Task Proposer. Upon completing a task, the follow-up task proposer generates the next logical subtask to continue the sequence. This agent is given the full history of prior subtasks and the most recent desktop screenshot, and is instructed to generate simple, specific follow- up action that builds on the previous state. Additionally, the proposer is informed of previously unsuccessful tasks, prompting it to propose simpler alternatives. Like the initial proposer, it avoids tasks that require login or unsafe actions. The resulting task is executed and verified as before, and if incomplete, revised description is generated. This iterative generation process continues until a desired sequence length is reached. Prompt templates for the follow-up up proposer are shown in Appendix Table B.4.
(6) Task Summarizer. Finally, the task summarizer converts a sequence of completed subtasks into single high-level task description. This summary abstracts away step-level details while preserving the overarching objective and required actions. By varying the number of subtasks summarized, we
4

---

Published as a conference paper at ICLR 2026
systematically control task difficulty: more subtasks yield longer, more complex tasks that require greater reasoning and planning. This mechanism enables us to generate tasks at multiple difficulty levels in a principled way. While each subtask may be trivial in isolation, the final composed task presents a challenging, multi-step problem for L.M agents. The summarization process is illustrated in Figure 1, and prompt details are provided in Appendix Table B.9.
DATASET ANALYSIS
4.1 QUALITY
To assess the quality of the generated tasks and trajec- tories, we conducted a manual evaluation on a random sample of 100 instances across difficulty levels (approx- imately 16 tasks per difficulty level) to ensure represen- tativeness across complexity. Our evaluation focused on the feasibility and realism of the overall task, the coherence and logical flow subtasks, their relevance to the assigned persona, and the accuracy of the veri- fier's s assessment of the agent's 's trajectory. Specifically, human annotators are instructed to assess:
Table I: Human evaluation of AgentSynth task and trajectory quality.
Quality Metric
Yes
Feasibility and realism
91 %
Subtask coherence
90%
Persona Relevance
94%
Verifier Accuracy
88%
Feasibility and realism: Could a real human user plausibly complete this task using standard software tools?
Subtask coherence: Does each subtask logically follow from the previous subtasks, maintaining clear and meaningful workflow progression?
Persona relevance: Is the task aligned meaningfully with the persona provided to guide task cre- ation?
Verifier accuracy: Does the automated verifier's binary assessment (task success failure) align correctly with human judgment?
As shown in Table 1, all quality metrics exceed 85%, highlighting the consistency, realism, and reli- ability of the data produced by the AgentSynth pipeline. On verifier accuracy, the evaluators who in- dependently evaluated the random sample also had an inter-rater agreement of 0.74 (Cohen's kappa). We note that prior findings (Li et al., 2025) indicate potential limitations of LLM-based verification, and our high manual-validatio tive screenshot sampling, visual context filtering, and task requirement extraction, improves verifier reliability compared simpler methods.
rate suggests our engineered verification pipeline, including selec-
4.2 VERIFIER CALIBRATION
We further calibrate our LLM-based verifier against human judgments on stratified sample of trajectories. For
1.0
Human judged score
1.0
Verifter Accuracy
0.8
0.8
0.6
each trajectory, humans label suc- cess/failure and assign graded com-
0.4
0.4
pletion score in [0.
0.2
0.2
Figure 2 (left) reports binary agree- ment as a function of difficulty level. Accuracy remains high across all lev- els with only mild decline tasks
0.0
2
0.0
1.0
Task Difficulty Level
Verifier completion score
Figure 2: Verifier calibration.
become harder, indicating that the verifier's pass/fail decisions are consistent with human judgments.
Figure 2 (right) examines the verifier's completion score. We bin trajectories by verifier score and plot the average human-judged completion score each bin. The curve is monotone: higher verifier scores correspond higher human scores. Together, these results show that the LLM verifier is both reliable across difficulty levels and provides meaningful partial-credit signal.
To further probe robustness, perform adversarial stress test of the LLM verifier. Start- ing from human-verified successful trajectories, we construct two types of perturbations: near- miss variants that subtly violate the goal (e.g., saving file with an almost-correct name in
5

---

Published as a conference paper at ICLR 2026
Avg. horizon
apps
Res Faltior
Presentation 8.1%
Spreadshees 8,%
Avg. memory spun
Avg. app switches
3.5%
Watfer
17.0%
Office
Steps
Apps
Web
Chrome 31,8%
32.2%
tab
VSC ode
3.8%
File
6.3%
Termitaa
1.8%
Fule Manager
mor Vome sy
Image
1,8%
5
Task Difficulty Level
(a) Scaling of task complexity.
(b) Distribution of Apps involved.
Figure 3: AgentSynth dataset statistics.
a wrong but visually similar folder), which should be labeled as failures, and benign variants that preserve the goal but change the UI in irrelevant ways (e.g., resized windows, extra tabs),
which should still be labeled as successes. Table 2
Table 2: Adversarial stress test of verifier.
reports the fraction of perturbed states that the veri-
fier marks as "success." The verifier incorrectly ac- cepts only 12% of near-miss variants while correctly accepting 94% of benign variants, indicating a low false- polst--osiiive rate on subtle failures and robustness to superficial UI changes.
Category
Verifier "success"
Near-miss (should fail)
12%
Benign (should succeed)
94%
4.3 COMPARISON OTHER DATASETS AND BENCHMARKS
We designed the AgentSynth pipeline with a focus on generating diverse, realistic, and challeng- ing data for training and evaluating computer-use agents. Table compares dataset to several existing agent benchmarks, highlighting key advantages in diversity, complexity, and scalability. Examples of tasks shown in Appendix D)
Diverse Real-World Tasks. AgentSynth spans a broad range software applications and do- mains, including office productivity, information retrieval, entertainment, coding, and research. This breadth ensures rich task diversity and supports generalization across practical, everyday scenarios. The pipeline leverages versatile environments that require agents fluidly interact with multiple software tools within single task. Figure 3b shows the coverage across domains and tools, illus- trating the dataset's alignment with real-world complexity.
Importantly, our pipeline encourages multi-tool pendix Figure C. over 60% of trajectories involve two or more software applications, and more than 40% involve three or more, demonstrating the inherent compositionality of AgentSynth tasks.
usage through chained subtasks. As shown in Ap-
Horg-Horizon Trajectories: Real-worke tasks often require extended sequences of actions involv- ing planning. memory, and interface coordination. AgentSynth explicitly supports such long-horizon tasks by composing them from interdependent subtasks. As shown in Figure 3a, tasks difficulty level 6 typically require 40-60 steps, exceeding the trajectory lengths of existing benchmarks. These tasks challenge agents to maintain context, manage interleaved goals, and execute multi-step plans, closely reflecting the demands of real-world computer
Scaling of Task Complexity, Although we define difficulty primarily via horizon, the tasks become harder along several complementary axes. Figure 3a shows that from level to 6, the average horizon increases from to 45 steps and the memory span from 2 to 18 steps, indicating longer- range dependencies. Moreover, the average number of applications rises from 1.2 to 3.3 and app

---

Published as a conference paper at ICLR 2026
Table Comparison of the cost of AgentSynth versus human annotations.
Framework
Typical Steps
Human Hours per Task
Cost per Task
T-bench (Yao et al., 2024)
20 30
$50
OSWorld (Xie et al., 2024)
10 15
4.4
$8.8 $110
Compaayy (Xu al., 2024)
30 40
17
$34 $425
AgentSynth
40 60
NA
$0.6
Table 4: Comparison of AgentSynth to some existing LLM agent datasets and benchmarks.
Framework
Multi-domain'
Domain Categories
Scalable
Long Horizon
Mind2Web (Deng al., 2023)
Web
Online MindZZWh (Xue al., 2025)
Weh
WebArena (Zhou
2024)
Web
VisualWebArena (Koh al., 2024)
Web
INSTA (Trabucco et al., 2025)
Web
AgentTrek (Xu
2025b)
Web
Explorer (Pahuja et al., 2025)
Weh
V
V
SWE-bench (Jimenez et al., 2024)
Coding
X
WorkArena (Drouin al., 2024)
Enterprise Software
OSWorld (Xie et al,, 2024)
OS, Web, Office, Coding
AindowsAgemArena (Bonatti et al., 2024) T-hench (Yao et al., 2024)
OS, Web, Office, Coding
Retail, Airline
SWE, HR, Admin, PM. Researeh.
TheAgennommpnyy (Xu et al., 2024)
Web, OS, Office, Coding. Research
AgentSynth
switches from 0.5 4.3, reflecting increased context switching and cross-application coordination. Thus, higher difficulty levels systematically combine longer horizons with more complex memory and interrupt-handiin demands. Additional dataset statistics and task properties are provided in Appendix C.
4.4
COST ANALYSIS
Beyond diversity and high quality, data generation pipeline is also highly scalable and cost- efficient. Our approach achieves cost of $0.6 per trajectory with follow-up subtasks. This comparable with recent methods such AgentTrek (Xu al.. 2025b) (S0.55 per trajectory), Explorer (Pahuja et al., 2025) ($0.28 per trajectory), and InSTA (Trabucco et al., 2025) ($0.27 per trajectory). Furthermore, our method much cheaper than human annotations for complex tasks with long trajectories. Table 3 shows the cost of several datasets from human annotations, where we assume the labor rate is in the range of $2 $25 per hour. The detailed calculation of our cost and human labor hours is shown in Appendix E.
RESULTS AND DISCUSSION
5.1 EVALUATION SETUP
To assess the general-purpose computer-use capabilities of current language models, evaluated several state-of-the-art multimodal agents with visual understanding. At each interaction step, the model receives a prompt containing the task description, the current desktop screenshot, and its own previous thoughts. The model is then asked to generate executable Python code using the oyautogui library to perform the next action. We sampled 50 tasks from each difficulty level for agent evaluation. Additionally, to benchmark human performance, we evaluated 20 tasks sampled from difficulty level 6, the most challenging tier in AgentSynth.
to indicate coverage across different types of software environments (e.g.. OS tools, native applications. web interfaces), rather than just topic categories within single interface modality like websites. While WehArena and Mind2Weh span many topics, their environments are restricted weh browsers.
We use
""ccalable indicates that the dataset can be expanded via automated synthetic pipelines without requiring additional manual annotation.

---

Published as a conference paper at ICLR 2026
o4-mini
Gemini Gemini-2.5-pru
GPT-4.1
Claude-3. Claude-3.7-Sonnet 7-Soneet
0.15
Success Rate
0.10
0.10
0.10
0,16
0.05
0.06
0.05
0.05
0.04
0.04
0.04
0.04
0.04
0.04
0.02
0.00
Task Difficulty Level
Figure 4: Model performance across task difficulty levels.
To isolate the role of the underlying language model and focus on the task difficulty itself, we use bare LLMs without fine-tuning or additional agent-specific scaffolding. Each model is prompted to generate pyautogui actions step-by-step based on the screenshot, task description, and action histories. This setup reflects lower bound on performance and is intended to benchmark agents under minimal guidance rather than deploy optimized, production-grade agents. Prompts used for evaluation are detailed in Appendix B. Task completion is assessed using the automatic verifier agent introduced section 3, which analyzes the full trajectory and determines whether the task was successfully completed.
5.2 RESULTS
The top panel of Figure shows the success rates of four state-of-the-art language models on the AgentSynth benchmark across task difficulty levels through 6. Despite having visual capabilities and strong general reasoning skills, all models exhibit poor performance our benchmark, espe- cially as task complexity increases. In contrast, humans achieve a 70% success rate even on the most difficult tasks, underscoring the performance gap. Key observations include:
Sharp Decline with Difficulty. All models show a consistent and steep drop in success rate as task difficulty increases. For example, 04 -iminn achieves 18% success on level but drops to 4% by levels 5 and 6. GPT-4. 1 fails to complete tasks beyond level 3. This highlights the significant challenge in realistic GUT environments and demonstrates the increasing challenge posed by longer- horizon, multi-step tasks in AgentSynth.
Near-Zero Success on Hard Tasks. At levels 4 and 6, only 04-mini and Claude-3.7 achieve non- zero scores, and even then, the success rate is only around 4%. This indicates that current models are far from achieving generalizable competence in realistic multi-stey computer tasks, showcasing the difficulty and discriminative power of our benchmark. The results highlight the need for models that can handle long-term dependencies, maintain state, and ground their decisions in visual observations over extended sequences.
5.3 COMMON AGENT FAILURE MODES
Despite the promising capabilities of LLM agents, their performance on the AgentSynth benchmark remains low, with most tasks ending in failure. We identify several recurring failure modes that highlight key limitations and suggest directions for future improvement:
Inaccurate Mouse Clicks. frequent failure involves imprecise mouse click coordinates. While the agent often identifies the correct UI element conceptually (e.g.. the "Save" button browser tab), it fails locate it precisely on screen. This results in misclicks, unintended interactions (e.g., clicking ads wrong icons), and cascading errors, such as obscuring losing focus the target window. Moreover, agents often repeat the same incorrect click multiple times without adapting.
Poor Screenshot Understanding and State Tracking. Agents frequently fail to properly interpret the visual information in screenshots. They may misidentify popups, ads, or irrelevant overlays as part of the main task UL. Other papers benchmarking LLM agents have also found problems with perceptual grounding (Xie et al., 2024; Koh et al., 2024). This weak perceptual grounding results in repetitive or irrational actions: for example, repeatedly trying save file that has already been
8

---

Published as a conference paper at ICLR 2026
saved. Moreover, agents often lose track of what has already been done, lacking persistent memory or state awareness.
Lack of Recovery from Errors: Once agent becomes stuck, it struggles to recover. Rather than exploring alternative actions or reasoning about potential mistakes, the agent tends to repeat the same failed behavior. This lack of introspection and self-correction severely limits task completion, especially for multi-step tasks requiring contingency handling. This a common error found with complex, long-horizon across many domains in the literature, from computer-use to general remote tasks (Xu et al., 2024; Drouin et al., 2024) to math and reasoning questions (Huang et al., 2024) among others.
5.4 AGENT SCAFFOLDING EVALUATION
Our primary results so far used bare LLM agents that directly map from the current screen and instruction to the next GUI ac- tion. To better match contemporary agentic systems, we additionally evaluate the Agent S3 scaffold
0.6
GPT:5.1
GPT-4
GPT-5 S3
of-mnii
o4-mini S3
GPT-4.1
et al., 2025),
Success Rate
0.4
which augments the backbone model with ex- plicit planning, tool-aware reasoning, and self-
verification over intermediate steps. Figure 5 compares success rates across difficulty levels for four backbones. For the weaker models (o4-mini, GPT-4. 1), adding the S3 seaf- fold yields substantial gains: success roughly doubles on level and remains consistently higher than the bare agents across levels. Stronger backbones (GPT-5, GPT-G. under S3 achieve the highest absolute performance, confirming that both model quality and scaf- folding contribute to success. However, for all backbones the success rate still decreases as the difficulty level increases, and even the best con- figuration attains only modest success on level This pattern indicates that while sophisticated scaffolds can recover significant fraction of "easy" tasks, AgentSynth remains challenging for state-of-the-a computer-use agents on the hardest levels.
0.2
0.0
2
Task Difficulty Level
Figure 5: Model performance for bare LLMs and with Agent S3 scaffolding.
We also evaluate an intermediate scaffold, Agent S2.5 (Agashe et al., 2025), which includes planning and tool formatting but uses a simpler self-check mechanism. As shown in Appendix Figure C.2, Agent $2.5 yields improvements over the bare agents that are similar to Agent S3, and exhibits similar decline in success with increasing difficulty. This suggests the qualitative trends are robust to different scaffold designs.
5.5 EFFECT OF INFORMATION ASYMMETRY
To assess the impact of our asymmetric generation pipeline, we compare AgentSynth direct- instruction baseline. In the direct setting, we prompt the same base model to generate single long- horizon instruction in one shot, targeting approximately 5-10, 10-20, and 20-30 actions, which we denote as Easy, Medium, and Hard tasks. We then attempt to execute each instruction in OSWorld with the same executor and verifier used in our main pipeline. and retain only tasks for which complete ground-truth trajectory can be generated. In contrast, AgentSynth first generates and verifies short subtasks and only afterwards summarizes them into long-horizon instructions.
Table 5 reports both the generation success rate (fraction of tasks for which ground-truth trajectory can be obtained) and the evaluation success rate Agent S3 with GPT-5. on the resulting tasks. For direct instruction, generation success drops sharply as we target harder tasks: from 64% at Easy to only 11% at Hard, indicating that the generator frequently fails to produce trajectories for genuinely difficult tasks. However, among the small subset of tasks that are solvable, evaluation success remains relatively high and flat, suggesting that these retained tasks are still comparatively easy for the evaluation agent.

---

Published as a conference paper at ICLR 2026
Table 5: Asymmetry ablation: direct instruction vs. AgentSynth.
Direct instruction
AgentSynth
Easy
Medium
Hard
Level
Level
Level 6
Generation success rate (%)
64
33
11
65
57
52
Evaluation success rate (%)
60
51
48
62
20
14
Table 6: Effect of the base generator evaluation performance.
Success rate (%) at difficulty level
Gen, model
Eval. model
3
GPT-4.1
18
14
14
10
10
GPT-4.1
GPT-5.1 S3
32
20
22
14
GPT-4.1
16
GPT-5-mini
GPT-5.1 S3
60
34
18
20
16
GPT-4.1
14
10
GPT-5
GPT-5.1 + S3
58
28
18
16
14
14
GPT-4.1
12
GPT-5.1
GPT-5.1 S3
56
30
16
18
14
12
AgentSynth exhibits the opposite pattern. Generation success remains high and stable across diffi- culty levels, confirming that the asymmetric pipeline makes trajectory collection substantially easier. At the same time, evaluation success drops dramatically
increase the number of summarized subtasks, from 62% at level to 14% at level 6. This decoupling between generation and evaluation difficulty is precisely the effect seek: information asymmetry allows the generator reliably solve and log simple multi-step workflows, while the resulting composite tasks remain challenging for downstream agents.
5.6 EFFECT OF BASE GENERATOR
Our main experiments far used GPT-4. as the base model for proposing and executing sub- tasks during data synthesis. To assess how sensitive AgentSynth is to this choice, we re-run the full pipeline with three different base models: GPT-5-mini, 5, and GPT-5. keeping prompts and the pipeline fixed. We then evaluate the resulting datasets using Agent S3 framework with ei- ther GPT-4. or GPT-5. as the backbone. Table reports success rates across difficulty levels for each combination. We observe that the overall behavior is stable across generation base mod- els. Within each dataset, the stronger evaluation model (GPT-5. + 53) consistently outperforms GPT-4.1 + $3, and in all cases success rates decrease task difficulty increases. Changing the base generator from GPT-4 4. to GPT-5-mm 5-mini yields very similar results, while using GPT or GPT-5.1 for synthesis leads to slightly lower success rates, suggesting that stronger genera- tors may produce harder or more diverse tasks. These results remain consistent with the previous conclusions qualitatively. AgentSynth remains challenging high difficulty for various generator choices, and the relative ranking evaluation models is preserved. This indicates that our findings not specific to a particular base model and that AgentSynth is robust to reasonable variations in the synthesis backbone.
6 CONCLUSION
In this work, we introduce AgentSynth, scalable pipeline for synthesizing diverse, high-quality datasets of computer-use tasks and trajectories. By leveraging information asymmetry and LLM- based agents, our method decomposes complex tasks into solvable subtasks, enabling fine-grained control over difficulty and long-horizon planning. The resulting benchmark reveals performance gaps in state-of-thee offtthe-rtt models. with success rates dropping sharply task complexity increases, highlighting both the challenging and discriminative power of our dataset. Notably, AgentSynth can continuously and flexibly generate harder tasks, ensuring long-term benchmarking utility.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We acknowledge that we adhere to the ICLR Code of Ethics. Our data synthesis pipeline is intended solely for academic research on GUI-based computer-use agents, To ensure ethical integrity and mitigate potential risks, we explicitly prompt our proposer agents to avoid any tasks involving login credentials or real personal information, or sending forms that change the backend of any website (emails, transactions, etc). Additionally, we introduce time delays between each action to prevent excessive requests and reduce potential load on external websites, thereby minimizing any unin- tended impact online services. Finally, only secure applications and websites visited that users of our dataset do not run into malware or other security issues. We also recognize that our method could be used improve LLMs' abilities in harmful tasks like cyberattacks.
REPRODUCIBILITY STATEMENT
We took several steps to facilitate reproduction of our results. The end-to-end pipeline, agent roles, and OSWorld environment setup are described in Section 3, including implementation details such screenshot resolution, action execution, and verification flow. Complete prompt templates for every component and for the evaluation agent are provided in Appendix and the OS-level action space used by agents is specified in Appendix G. Dataset construction, difficulty formation, statistics, and our evaluation protocol that support the main experiments are reported in Section 5. We also document the cost accounting required to replicate anonymized source code and a minimal example dataset in the supplementary materials, and we will release both the full codebase and dataset publicly after the double-blind review process.
scale estimates in Appendix E. We include
REFERENCES
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: compositional generalist-specialis framework for computer use agents. arXiv preprint 0rXiv:2504.00006, 2025.
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault de Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks. Advances in Neural Information Processing Systems, 37:5996-600 6051, 2024.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024. URL https:: paapp://aaxiiii /ttp///xxvvv////////////./ abs/2409. 08264.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web, 2023. URL https://arre
org/abs/ 2aabbb//33666000
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvent Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. htt:ttts:://rra
oreoorrggooooo. /arg/aaba abs 2403.07711
Hiroki Furuta, Yutaka Matsuo, Aleksandra Faust, and Izzeddin Gur. Exposing limitations of language model agents in sequential-task compositions on the web. arXiv preprint arXiv:2311.1875.. 2023.
Apurva Gandhi and Graham Neubig. Go-browse: Training web agents with structured exploration, 2025. URL
03533.
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data cre- ation with 1,000,000,000 personas, 2024. URL ItRps:tps:///ra wwtrexiivvorr/aeooomm org wowggab//////... /200/ 20094.

---

Published as a conference paper at ICLR 2026
Gonzalo Gonzalez-Pumariege Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, and Xin Eric Wang. The unrcasonable effectiveness of scaling agents for computer use. arXiv preprint arXhv:2510.02250, 2025.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou, Large language models cannot self-correct reasoning yet, 2024. URL https: / //arxiv... www//rraass....ooee 2210 017998.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL
22dbs/23100 066770.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang. Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov. and Daniel Fried. Visualwebarena: Evaluating multimodal agents realistic visual web tasks, 2024. URL 2401. 13649.
Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsumori Hashimoto. Autobencher: Towards declarative benchmark construction, 2025. URL https: / laaaaii..rrg/bb///0000.00 08351.
Haohan Lin, Zhiqing Sun, Sean Welleck, and Yiming Yang. Lean-star: Learning to interleave thinking and proving, 2025. URL https:///// wwwaaaiiiwrgaaaaaaaaeaaaa
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration, 2018. URL https://arxiv: org/abs/ 1///18100 1802. 08802.
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on syn- thetic data, 2024. URL https: arwiv. www.wiwwwwwabbaa..aaaa 2404. 07503.
Xing Han Lå, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejan- dra Zambrano, Karolina Stanczak, Peter Shaw, Christopher J. Pal, and Siva Reddy. Agentre- wardbench: Evaluating automatic evaluations of web agent trajectories, 2025. URL https:
//arxivv.o.oo///// wrt.iiiorrr..eooooo abs//504 08942.
Dirk Merkel. Docker: lightweight linux containers for consistent development and deployment. Linux J., 2014(239), March 2014. ISSN 1075-3583.
Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D. Manning. Nnetnav: Unsupervised learning of browser agents through environment interaction the wild, 2025. URL https: 02907.
OpenAl. Computer-using agent: Introducing universal interface for ai to interact with the digital world. OpenAl Blog, 2025a.
URL http::///ppaaa..comm....
computer-using-agee
OpenAl. Introducing gpt-4.1 in the api. https: httppss//ppeea...cc///nee///gg/ Accessed: 2025-05-10.
2025b.
Tianyue Ou, Frank F. Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale, 2024. URL https: http//aiii... org/ abs 4009. 15637.
Vardaan Pahuja, Yadong Lu, Corby Rosset. Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Awadallah. Explorer: Scaling exploration-driver web trajectory synthesis for multi- modal web agents, 2025. URL. httph::tt////..... /tattps://arivv. www./org//aa...... abs /2002 1357.
Richard Shin, Neel Kant, Kavi Gupta, Christopher Bender, Brandon Trabucco, Rishabh Singh, and Dawn Song. Synthetic datasets for neural program synthesis, 2019. URL https: /rrxiv. org/abs/ 912. 12345.
12

---

Published as a conference paper at ICLR 2026
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 2024.
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ô. Ank. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments, 2025. URL
https://arxiv/ org/ alss
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jía, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis, 2025. URL https:
org/ abs 2212.199722
Al Sweigart. Pyautogui: Cross-platform gui automation for human beings. pyautogui. readt.eeocs. io/, 2025. Accessed: 2025-05-06.
https://
Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov. Towards internet-scale training for agents, 2025. URL https: //arxiv// org/ /tbbs/2502.066
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao. Toh Jing Hua, Zhoujun Cheng. Dongchan Shin, Fangyu Lei. Yitao Liu, Yiheng Xu, Shuyan Zhou, Sil- vio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https://arx: org/ brg/abs//a 2404 27944.7972.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions, 2025a. URL
/atr/////////////// abaaaass///00 12244.
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang. Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchmarking Ilm agents on consequential real world tasks, 2024. org/ abs 24 14161.
URL
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and
Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2025b. URL
https: //arxiv org/ abss 22242.09600
Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and
Yu Su. An illusion of progress? assessing the current state of web agents, 2025. URL https:
//a //arxiiv.oo org. abs 22504.500000 01382.
John Yang, Carlos F. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. 2024. URL https: / /awxivi.vvvv//////i///aaa//aaava wttggaaa///...11555550 15793..
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents, 2023a. URL https://rxxiv..r
abs/2207. 01206.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023b. URL httts://arxxiv org/abs/ /2100.21.. 03629.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiis /ttp//xix..oogga/////// 2406. 12045.
Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason Weston, and Xian Li. Naturalreasoning: Reasoning in the wild with 2.8m challenging questions, 2025. URL ht ht /ttp/aaxiii/v/////i///aa///.aava wwwpexiivvogg/eeeo
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng. Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URL https: //arxivvvv./////////////////aav/ org/ abs/ 2307. 13854.
13

---

Published as a conference paper at ICLR 2026
A LLMS USAGE STATEMENT
We used LLMs as general-purpose editorial assistants for this manuscript (grammar fixes, word- ing suggestions, and minor style edits). LLMs did not contribute research ideation, model or algorithm design, implementation, data generation or processing, experimental setup, analysis, or results. All technical content was authored by the authors, who reviewed and approved any edits.
B PROMPT DETAILS
The prompts for the task proposal. execution, verification, revision, follow-up proposal, and sum- marization agents are provided in Tables B.1-B.9.
Table B.1: Prompt for Task Proposal Agent.
System Role
What does this screen show? You are real user on this computer. Please provide single task that the user might perform on this computer and the corresponding first action towards completing that task. You web. creative and come up with diverse tasks. The task should be simple enough can finished few atomic actions. The task should clear and very specific.
any software in the computer and the
Task proposal rules:
The task should be specific and clear.
The task should be achievable within 5 atomic actions like clike, scroll, type, press,
etc.
3. The task should be relevant to the content of the webpage.
4. You should only propose tasks that do not require login execute the task.
Provide concrete information or constraints the task, and use mock-up informa- tion. (identifier, number, personal information, name, attributes, etc.) to make the task more specific and realistic.
7. The task description should provide all the necessary information to complete the task.
Output with JSON block:
:<Detalle Thoughts and Reasons. Think about if the rules are met, e.g. why the task is related the user character, why the task is simple enough to be finished a actions, is the task clear specific, etc.>", "task" :" <TASK>" "action":"<ACTTOO""
User Role
You [PERSONAJ, what task would you perform the computer? (SCREENSHOT)
14

---

Published as a conference paper at ICLR 2026
Table B.2: Prompt for Task Execution Agent (Planner).
System Role
You computer agent which perform desktop computer tasks instructed. You have gond knowledge computer and good internet connection and assume your will controll the mouse and keyboard. For each step, you will asked to finish task, and you will get a screenshot of the current computer screen. You also know the actions that are already done towards the farget. You need to provide the next action based the screenshot. If you have tried similar actions several times but haven't success, analyze the reason carefully and propose a different action. Try to think you were wrong or you missed any steps. If you think the task is finished, return "DONE" in the action field.
Rules:
First analyze the screeenshot carefully, pay attention to details in the screenshot.
2. Then analyze the previous thoughts and actions to make sure you are on the right track.
Note: the previous actions may not be executed successfully, you need analyze the screenshot carefully.
you find you have tried similar actions several times haven't 't success, analyze the reason carefully. Try to think if you were wrong you missed any steps. Carefully analyze the sereenshot to find the reason of the failure and propose a different action. 5. If you think the task finished, return "DONE" in the action field.
Output with JSON block:
"thoughts":
:"erbttt""""=Daiiilla "<Detailed Thoughts and Reasons>",
"action":" "iation""<<<" "<ACTION>"
User Role
Given the task: (TASK). You have gathered some information [INFO). Here is your previous thinking process to complete the task (THOUGHTS (LIGGHTSSSSSTOR]] Here are your previous actions tried (ACTION ) Here is the current scrcenshot. What would the next action? (SCREENSHOT)
Table B.3: Prompt for Task Execution Agent (Visual Grounding).
System Role
You are a computer agent which perform desktop computer tasks as instructed. You have good knowledge of computer and good internet connection, and assume you control the mouse and keyboard. For each step, you will be asked to finish task, and you will get a screenshot of the current computer screen. You also know the actions that are already done towards the target. You need to provide the next action based on the screenshot.
Rules:
You have the permissions to proceed, and you don't need to ask for permission.
The safety checks are acknowledged, and you can always proceed.
2. If you have clicked the same item several times, you don't 't need to click it again.
3. Do not click on ads.
4. If the computer is locked, type "password".
User Role
Given the task:
(TASK), you have done the following actions: (AC-
TION HISTORY). You need to do next step: (STEP). What would be the action? (SCREENSHOT)
15

---

Published as a conference paper at ICLR 2026
Table B.4: Prompt for uollow-up Task Proposal Agent.
System Role
What does this screen show? You real user this computer. Given the tasks the has done, please provide single follow-up task that the user might perform this computer and corresponding first action towards completing that task. You can use any software on the computer and the web. Be creative and come up with diverse tasks. The task should be simple enough that can be finished in few atomic actions.
Task proposal rules:
The task should depend on the previous tasks.
2. The task should be achievable within 5 atomic actions like click, scroll, type, press, ctc.
4. The task should be relevant the content of the previous tasks.
You should only propose tasks do not require a login execute task.
6. Provide concrete information or constraints to the task, and use mock-up informa- tion (identifier, number, personal information, name, attributes, etc.) to make the task more specific and realistic.
The task description should provide necessary information to complete task.
8. Do not propose tasks including sending emails or sharing on social media.
Output with JSON block:
"iriss""=""D""i"""""" "<Detailed Thoughts and Reasons. Think about the rules are met, e.g. why the task is related to the user character and previous tasks, why the task is simple enough to be finished in few actions, the task clear and specific, etc.>". "task" : <TASK>".
"thoughts":
"action": :"ction""<AATTON:: <ACTIONS"
User Role
You are (PERSONA). Given the task history ) what would be follow-up task? Note that these tasks ) are too hard for the agent, propose simpler one. (SCREENSHOT)
Table B.5: Prompt for Task Verification Agent (Key Requirement Identification).
System Role
You are an expert tasked with analyzing a given task identify the key points explicitly stated in the task description. Carefully analyze the task description and extract the critical elements explicitly mentioned in the task for achieving its goal.
Rules
1. Read the task description carefully.
2. Identify and extract key points directly stated in the task description.
3. key point is critical element, condition, or step explicitly mentioned in the task description.
4. Do not infer or add any unstated elements.
Output with JSON block:
"thoughts": "<uoor detailed thinking and reasoning process>".
"key points"." "<List of the key points for completing this task>"
User Role
Given the task (TASK), what are the key points?
16

---

Published as a conference paper at ICLR 2026
Table B.6: Prompt for Task Verification Agent (Key Screenshot Identification).
System Role
You are an expert evaluator tasked with determining whether screenshot contains information about the necessary steps complete task. Analyze the provided image and decide shows essential steps evidence required for completing the task. Use your reasoning to explain your decision.
Rules
Provide detailed description of the screenshot, including its contents, visible ele- ments, text (if any), and any notable features.
2. Carefully examine the screenshot and evaluate whether it contains necessary steps or evidence crucial to task completion.
3. Identify key points that could be relevant to task completion, such actions, progress indicators, tool usage, applied filters, step-by-step instructions. Does the screenshot show actions, progress indicators, or critical information rectly related to completing the task 5. Is this information indispensable for understanding ensuring task success?
If the screenshot contains partial but relevant information, consider its usefulness rather than dismissing outright. Output with JSON block:
"thoughts" =Yuruur detailed thinking and reasoning process>", " "necessary": <True or False>"
User Role
Given the task (TASK), the key points to finish the task (KEY [EEN_])N screenshot of an action, is this screenshot necessary step to complete the task? (SCREENSHOT)
Table B.7: Prompt for Task Verification Agent (Final Judgement).
System Role
You are an expert in evaluating the performance of a computer-use agent. The agent is designed to help a human user complete a computer-use task. Given the user's task, the agent's action history, key points for task completion, and some important screenshots in the agent's trajectory, your goal is to determine whether the agent has completed the task and achieved all requirements.
Rules
The filtered results must be displayed correctly. If filters were properly applied (i.e.. missing selection, missing confirmation, or no visible effect in results), should be considered failure.
2. You must carefully check whether these screenshots and action history meet these key points. Ensure that specific requirements correctly applied. 3. Some tasks require a submission action or a display results to be considered successful. Repeat actions or actions
do not lead to visible result should considered failure. 4. If the agent loops through a sequence of actions that do not make progress toward the goal (including failing click "Save" or "Submit," " etc.), it should be considered failure.
Output with JSON block:
"thoughts":"" "Yuror detailed thinking and reasoning process>",
"success": " <T"uue False>".
"success rate": "<Probbbilly of success in unit of percentage. like 20. 50, 100, numbers only>"
User Role
Given the task (TASK), the key points to finish the task (KEY POINTS), and the screenshots history, is the agent successful? (LIST OF SCREENSHOTS)
17

---

Published as a conference paper at ICLR 2026
Table B.8: Prompt for Task Revision Agent.
System Role
Given list of screenshots of actions performed the computer, you asked come up with a single task description will accomplished by performing these actions in the given sequence. First analyze these actions and generate the task description. Only summarize the completed actions, and ignore the actions that are not completed. If there is no completed action no meaningful task, return "NONE" in the task field.
Rules:
1. The task should be specific and clear.
2. The task description should provide all the necessary information to complete the task.
The task should be feasible to complete by a real user and should not require any additional information that is not specified in this input.
Output with JSON block:
"thoughts" : "<Detailed Thoughts and Reasons>", "task": :oas"""<<<""" <TASK>
User Role
Given the of screenshots of actions, what would be single task description that will be accomplished by performing these actions the given sequence?"
(LIST OF SCREENSHOTS)
Table B.9: Prompt for Task Summarize Agent.
System Role
Given a list of subtasks performed on the computer, you are asked to come up with single task description that will be accomplished by performing these subtasks in the given sequence. First analyze these subtasks and generate the task description.
Rules:
1. The task should be specific and clear.
2. The task description should provide all the necessary information to complete the task.
The task should be feasible to complete by a real user should not require any additional information that is not specified in this input.
4. The task should include all the numbers and information used in the subtasks.
Output with JSON block:
"thoughts": ":" "<Detailed Thoughts and Reasons>", "task <TASK>
User Role
Given the subtasks history [TASK_ HISTORY) and the final sereenshot, what would single task description that will accomplished by performing these subtasks in the given sequence? (SCREENSHOT)
18

---

Published as a conference paper at ICLR 2026
Table B.10: Prompt for Evaluation Agent.
System Role
You are agent which follow my instructions performs desktop computer tasks as instructed. You have good knowledge of computers and a good internet connection, and assume your code will run on a computer for controlling the mouse and keyboard. For each step, you will get observation of the desktop through screenshot. And you will predict the action of the computer based the image and information.
You are required to use 'pyautogui' to perform the action grounded to the observation. You can use the following functions:
pyautogui. yyautogui.click(x, y, button); pyautogui. dowbleeiiibbb..icck.... y);
pyautogui.move moveeToox, y);
pyautogui, pyautogui.writeestte write(string);
pyautogui.drag drag y);
pyautogui.scrof (amoun)m pyautogui, press(key); pyautogui hotkey(key 1, key2, time.slcep(5)
Note that 'pyautogui" and 'time' packages have already been imported. Return one line or multiple lines of python code to perform the action cach time, be time efficient. When predicting multiple lines of code, make some small sleep like time. slee(p)) You need specify the coordinates by yourself based on your observation of current observation, and you should be careful to ensure that the coordinates are correct.
If you think the task is finished, return "DONE" in the code field.
Output must be valid JSON black with the following format:
"thoughts": "Deetalled "<Detalll Thoughts and Reasons>", "code": ":" "<yyhon code>"
User Role
Given the task: (TASK). Here are your previous thoughts (THOUGHTS, H And here the current screenshot. What would next action? (SCREENSHOT)
19

---

Published as a conference paper at ICLR 2026
C
ADDITIONAL DATASET STATISTICS AND EVALUATION
Average Token Count
100
75
Percentage 6
4
Percentage
50
20
25
2
3
10
20
30
40
50
60
2
5
6
Difficulty Level
Number Steps
(a)
(b)
(c)
Figure C.I: AgentSynth dataset statistics. (a) Average token count by task difficulty, (b) Distri-
bution of the number of steps at task difficulty level 6. (c) Distribution of the number of software applications for each task at difficulty level 6.
Table C.I summarizes several task properties as function of difficulty level. Beyond the primary "horizon" measure (average number of steps), compute the average number of distinet applica- tions involved, the number of application switches, and a memory-span metric defined as the maxi- mum step distance between reading piece of information and later using it in the same trajectory. We also estimate the fraction of tasks that require fine-grained visual discrimination ("fine-grained"),
As difficulty increases from level to 6, horizon, number of apps, context switches, and memory span all increase, indicating that higher levels not only involve longer trajectories but also more cross-application coordination and longer-range dependencies. The fraction of fine-grained percep- tion tasks remains relatively stable, suggesting that visual discrimination is present across all levels but is not the primary knob we use to control difficulty.
Table C.I: Task properties across difficulty levels.
Level
Avg. horizon
Avg. apps
Avg. # switches
Avg. memory span
Fine-grained (%)
1.2
0.5
40
12
1.7
1.5
5
42
23
2.3
2.4
9
38
4
34
2.8
3.4
12
41
3.9
16
45
45
3.3
4.3
18
43
0.6
GPT-5.1 + S2.5
GPT-4.1 + $2.5
GPT5 $2.5
04 oi-mini
04-mini $2.5
GPT-4.1
0.4
0.2
0.0
2
Task Difficulty Level
Figure C.2: Model performance for bare LLMS and with Agent S2.5 scaffolding.
20

---

Published as a conference paper at ICLR 2026
D EXAMPLE TASKS
To illustrate the quality and realism of tasks generated by the AgentSynth pipeline, present representative example.
First, persona is sampled from the persona hub (Ge et al., 2024):
Persona: senior student at Kentucky Wesleyan College.
Then, the task proposer generated an initial task tailored to this persona:
Initial task: Scarch for the 'Kentucky Wesleyan College 2024 academic calendar' in Google Chrome.
Next, this initial task was successfully executed, and five follow-up tasks were iteratively generated and completed:
Follow-up Task Find the Kentucky Wesleyan College 2024 commencement (graduation) date on the academic calendar currently open in Chrome
Follow-up Task Open the Calendar application after searching for graduation-related dates on an academic calendar wehsite.
Follow-up Task Scroll backwards month-by-mone the calendar application from March 2025 to 2024 using the month view.
Follow-up Task Create new calendar event Kentucky Wesleyan College's 's 2024 commence- ment titled GGaduution Day' and add note: 'Remember bring gown and arrive hour early.
Follow-up Task Add a notification/emmnder to the "Graduation Day" event on May 3, 2024 in the Calendar app to alert you day before.
Each of these subtasks is simple and logically follows from the previous The task summarizer composes them into coherent, high-level tasks. We define task difficulty level n as the summary of the first subtasks, resulting in increasingly complex and realistic scenarios. These summarized tasks are then used for evaluation and benchmarking. The list below shows the final task descriptions six difficulty levels, with italicized indicating the incremental complexity introduced each level:
Task Difficulty Level I (same the initial task): Search for the 'Kentucky Wesleyan College 2024 academic calendar' in Google Chrome.
Task Difficulty Level 2: Find and report the date and time of the Kentucky Wesleyan College 2024 commencement ceremony by searching for the 2024 academic calendar online, locating the official calendar, and identifying the commencement
listed there.
Task Difficulty Level 3: Search for the "Kentucky Wesleyan College 2024 academic calendar' in Google Chrome, find the 2024 commencement date, and then open the Calendar application view record the commencement date.
Task Difficulty Level 4: Find the Kentucky Wesleyan College 2024 commencement date using Google Chrome, then open the Calendar application and scroll back in month view from March 2025 to June 2024 in preparation for viewing adding the graduation date the calendar.
Task Difficulty Level 5: Find the Kentucky Wesleyan College 2024 commencement date by search- ing online (using the academic calendar), and create new event titled 'Gruduation Day' in your digital Calendar application, adding note that says 'Remember bring gown and arrive hour early'.
Task Difficulty Level 6: Find the Kentucky Wesleyan College 2024 commencement date by search- ing the 2024 academic calendar online, then create calendar event titled 'Graduation Day' in the Calendar application with saying 'Rememher bring gown and arrive hour early,' and ser reminder alert you one day hefore the event.
As the task level increases, both task length and complexity grow accordingly. Each additional subtask introduces new actions, tools, or planning steps. Figure C. shows the average token
21

---

Published as a conference paper at ICLR 2026
count across task levels, confirming that longer task compositions correspond more elaborate task descriptions and execution requirements.
We show few more example tasks generated by AgentSynth pipeline.
Example 1
climate-conscious teenager who confronts their parent about the negative envi- ronmental impact of their investments
Persona:
Subtasks:
Search for recent article about the environmental impact of investing in fossil fuel companies and open it to prepare for discussion with your parent.
Create and edit new text file using Visual Studio Code.
Add section to your text file in Visual Studio Code listing three alternative sustain- able investment options (e.g., renewable energy funds, green bonds, or ESG index funds) and a brief reason why each is better for the environment.
Add a relevant quote or statistic from the recent article about fossil fuel investments' environmental impact to your text file in Visual Studio Code, citing the source.
Enter energy source categories ( ('Fossil Fuels' and 'Renewables') and their respec- tive values (200 and 20) into a LibreOffice Calc spreadsheet, then select this data to initiate the creation of pivot table.
Select range of data in LibreOffice Calc and initiate the creation of pivot table using the selected data.
Tasks:
Search for recent article about the environmental impact of investing in fossil fuel companies and open to prepare for discussion with your parent.
Search for recent article about the environmental impact of investing in fossil fuel companies, open it to gather information, and then create new text file in Visual Studio Code where you write or summarize the key points from the article to prepare for a discussion with your parent.
Research and summarize recent article about the environmental impact of invest- ing in fossil fuel companies, then use Visual Studio Code to create new text file. In this file, document the summary and add section listing three alternative sus- tainable investment options (e.g.. renewable energy funds, green bonds, ESG index funds), including a brief explanation for each on why it is better for the environment, to prepare for a discussion with your parent.
Research a recent article about the environmental impact of investing fossil fuel companies. then create new text file in Visual Studio Code. In the file, summa- rize the article by including relevant quote or statistic (with proper citation), and list three alternative sustainable investment options, renewable energy funds, green bonds, and ESG index funds, each with a brief explanation of their environmental benefits, to prepare for discussion with your parent.
Prepare for a discussion about the environmental impact of investing in fossil fuel companies by (1) finding and reviewing a recent article on the topic, (2) creating a text file in Visual Studio Code summarizing three alternative sustainable investment options (such as renewable energy funds, green bonds, or ESG index funds) and explaining why each is better for the environment, (3) adding a relevant quote or statistic from the article to the text file with a citation, and (4) entering the energy source categories 'Fossil Fuels' (value 200) and 'Renewables' (value 20) into a LibreOffice Cale spreadsheet, then selecting this data to initiate the creation of a pivot table for comparison.
Research a recent article about the environmental impact of investing in fossil fuel companies and open it. Use the information to prepare a text file in Visual Studio
22

---

Published as a conference paper at ICLR 2026
Code that includes: (1) three alternative sustainable investment options (such as renewable energy funds, green bonds, ESG index funds) and brief reason each is better for the environment, and (2) relevant quote or statistic from the article about fossil fuel investments. Then, in LibreOffice Calc, enter the values for two energy source categories: 'Fossil Fuels' (200) and 'Renewables' (20), select this data, and initiate the creation of pivot table to compare them for discussion with your parent
Research a recent article about the environmental impact of investing in fossil fuel companies and open Use the information prepare text file in Visual Studio Code that includes: (1) three alternative sustainable investment options (such as renewable energy funds, green bonds, ESG index funds) and a brief reason each is better for the environment, and (2) relevant quote or statistic from the article about fossil fuel investments. Then, in LibreOffice Calc, enter the values for two energy source categories: 'Fossil Fuels" (200) and 'Renewables' (20), select this data, and initiate the creation of a pivot table to compare them for discussion with your parent
Example 2
Persona: A mentor who guides the student in exploring more advanced concepts and exper- iments
Subtasks:
Create a Python file named "experimentl.pp -py" using VS Code an ad- vanced programming experiment.
Install the recommended 'Python' extension for Visual Studio Code to enable ad- vanced Python features for your experiment.
Write simple print statement (e.g., print(' Advanced experiment ready!")) in 'ex- perimentl -py and run the file in VS Code to confirm successful Python setup.
Create and run Python script ('experimentl -py') in Visual Studio Code that first prints 'Advanced experiment ready!', then edit the script to define and call a func- tion 'greet((' that prints 'Welcome advanced Python!", and run the updated script to confirm the new output.
Create and test Python function named personal_ _grett that takes a name as an argument and returns 'Hello, name! Welcome to advanced Python!", then call it with the name "Alice' and print the result.
Edit a Python script to prompt the user for their name and print personalized grecting using function called personal_gree greet.
Tasks:
Create a new Python file named 'experimentl pyy using VS Code to start an ad- vanced programming experiment.
Set up advanced Python programming environment in Visual Studio Code by creating new Python file named experiment 1. pyy and installing the recommended 'Python' extension for VS Code enable advanced Python features.
Set up an advanced Python programming environment in Visual Studio Code by creating a new file named eexperimentl, ppy. installing the recommended 'Python' extension for VS Code, writing the code print(" Advanced experiment ready!') in experimentll.py" and running the file VS Code to confirm the Python environ- ment is correctly configured.
Set up Visual Studio Code for advanced Python development your system by creating a new Python file named 'experimentl, -py", installing the recommended 'Python' extension for VS Code, and confirming your environment by first adding print statement (print(" Advanced experiment ready!')) and running the file. Then,
23

---

Published as a conference paper at ICLR 2026
modify 'experimentl.p'" to define and call a function named 'greet(" that prints "Welcome advanced Python!', and run the script again to verify it outputs the new message as expected.
Set up a Python development environment in Visual Studio Code by creating a new file named 'experimentl.py" installing the recommended Python extension, and verifying the Python setup by writing and running a script that: first, prints 'Ad- vanced experiment ready!"; next, defines and calls function 'greet()' that prints 'Welcome to advanced Python!"; and finally, defines and tests a function 'per- sonal gonet_ that takes a name argument and returns a personalized greeting 'Hello, name! Welcome to advanced Python!", calling it with 'Alice' and printing the result.
Set up Visual Studio Code for advanced Python programming by installing the rec- ommended 'Python' extension. Create In this file, first write and run a simple print statement (such as print(' Advanced experiment ready!'))
Python file named 'experimentl -py".
confirm Python working. Next, enhance the script by defining a function 'greet()' that prints "Welcome to advanced Python!' and run it. Then, define a function 'personal geeeoona__ree''' that takes a name argument and returns personalized message: 'Hello, name! Welcome to advanced Python!'. Call this function with the argument 'Alice' and print the output to test Finally, edit the script so that it prompts the user to enter their name and prints personalized greet- ing using the personal greet' function. Run the script after each change to verify correct output.
Example 3
Persona: A whistleblower who anonymously provides crucial information about the art dealer's illegal operations
Subtasks:
Open the web browser and navigate to the SecureDrop homepage to review the instructions for submitting anonymous tips about the art dealer's illegal operations.
Open text editor and draft an anonymous tip detailing the art dealer's illegal activ- ities, including evidence (e.g., 'Art Dealer John Smith selling forged artworks un- der the alias ArtHouseX. Key transaction dated 2023-09-12 12 with buyer code 9743. See attached invoice scan.") !) Save the file as Anonymous Tip ArtDealer in the Documents folder.
Create an invoice slide in LibreOffice Impress with the following details: Title 'In- voice', Date: 2023-09-12, Buyer Code: 9743, Seller: John Smith (Alias: ArtHou- seX), Description: Forged artwork, Total: $12,000.
Export the invoice slide as a PDF file named 'Ivvicc Arteelerr 200Da99er200330122.pprr in the Documents folder for submission as evidence.
Create and save an anonymous tip text file (' ['Anonymous_ Tip Tii__Tipp_t_i_ ArtDealer.txt") sum- marizing and reporting the details of forged artwork transaction documented on an invoice, including the seller's alias, transaction date, buyer code, and reference to the attached invoice scan.
Write an anonymous tip about an art dealer selling forged artworks in a text file named Anonymous Tip ArtDealer.ixt.
Tasks:
Open the web browser and navigate the SecureDrop homepage review the instructions for submitting anonymous tips about the art dealer's illegal operations.
Review the SecureDrop homepage for instructions on anonymously submit- ting tips. then draft and save an anonymous tip in text file named 'Anony- mous mous_Tip__ Tip__ppArr Srip_ArrDDale.txx in the Documents folder. The tip should state: "Art Dealer John Smith is selling forged artworks under the alias ArtHouseX. Key transaction dated 2023-09-12 with buyer code 9743. See attached invoice scan.
24

---

Published as a conference paper at ICLR 2026
Prepare and save an anonymous tip about art dealer John Smith's illegal activities for submission via SeoureDrop by (I) reviewing SeoureDrop instructions online, (2) drafting and saving a detailed anonymous tip as 'Anonymous_ Arip_ArtDealertt in the Documents folder, including evidence that John Smith (alias: ArtHouseX) sold forged artwork on 2023-09-12 to buyer code 9743 for $12,000, and referencing an attached invoice, and (3) creating and saving an invoice slide in LibreOffice Impress with the following details: Title 'Invoice', Date: 2023-09-12, Buyer Code: 9743, Seller: John Smith (Alias: ArtHouseX), Description: Forged artwork, Total: $12,000,
Prepare an anonymous tip about illegal forged artwork sales by art dealer John Smith (alias: ArtHouseX), including supporting evidence. Draft an anonymous tip in text editor stating that Art Dealer John Smith is selling forged artworks under the alias ArtHouseX. Key transaction dated 2023-09- 12 with buyer code 9743, See attached invoice scan.' Save this draft as Annoyymoss _iioouus_T________e Ari_AerDeealertx in your Documents folder. LibreOffice Impress, create an invoice slide with the title 'Invoice', date 22033-9--2", buyer code '9743', seller 'John Smith (Alias: Art- HouseX)', description 'Forged artwork', and total '$12,000'. Export this slide as "Invoice_ArtDa" ArtDealer pateeee2220022_0 2. to the Documents folder. These two files will be prepared for anonymous submission via SecureDrop after reviewing the Secure- Drop submission instructions.
Prepare and save complete anonymous report for submission SecureDrop about art dealer John Smith's 's illegal sale forged artworks (alias: ArtHouseX), including drafting anonymous tip file (' ('Anonymous Tip_Art ArtDealler.tt'' outlining the ac- tivities and transaction (dated 2023-09- 12, buyer code 9743, total $12,000), and cre- ating invoice corroborating evidence (slide in LibreOffice Impress with these details) exported as 'Invoice_Aree ArtDealer ArrDDealer2202339122 20.30912.pdff. both saved in the Docu- ments folder, in accordance with the reviewed SecureDrop submission instructions.
Prepare and save two files in the Documents folder to submit anonymous tip about art dealer John Smith (alias ArtHouseX) selling forged artworks: (1) a text file named 'Anonymous_ Tip ArtDealerr.xt that details the illegal transaction (dated 2023-09-12, buyer code 9743, evidence reference to attached invoice scan), and (2) PDF export of an invoice slide (created in LibreOffice Impress) titled 'In- voice', with the same transaction details (seller: John Smith (ArtHouseX), descrip- tion: forged artwork, total: $12,000), saved as "Invoice_ArtDeill ArtDealer 20230912.pdff ready for anonymous submission via SecureDrop.
E COST ANALYSIS
In this section, we analyze the cost of AgentSynth and compare it to the labor costs of several human- curated agent datasets. Since many these datasets are created by graduate students. labor rate of $25 USD per hour is reasonable baseline. However, labor rates vary widely across regions and can be as low as $2 per hour in some low-income countries. To account for this variability, we estimate costs using a range of $2-$25 per hour.
7-bench (Yao al., 2024) includes tasks with up to 30 steps, but the authors did not provide de- tailed breakdown of the human labor involved in the data curation process. The dataset construction comprises three stages: (1) manual design of the database, (2) automatic data generation, and (3) not disclosed. For the third stage, the authors reported manual inspection of over 40 trials per task. Assuming an average of 3 minutes per trial, this implies approximately 2 hours of labor per task. At the estimated labor rate of 02-$25/hour, the cost per task is around $4 $50. This should be considered a power-bound estimate, as it does not include the potentially substantial cost of database and API design.
manual task annotation. Labor costs for the first two stages
OSWorld (Xie et al., 2024) limits each task to a maximum of 15 steps and provides clear reporting on human labor. The dataset created by 9 computer science students over approximately 1,800
25

---

Published as a conference paper at ICLR 2026
person-hours, resulting in 412 tasks. This averages to about 4.4 hours per task, or $8.8 $110 per task, assuming a labor rate of $2-S25/hour
TheAgentCompany (Xu al., 2024) reports total of 3,000 person-hours for the creation of 175 tasks. This equates to roughly 17.1 hours of labor per task, yielding a cost of approximately $34 $425 per task, assuming labor rate of $2-$25/hour.
AgentSynth. We use GPT-4.1 ($2 per million tokens) for most agents in the pipeline, the computer- use-preview model ($3 per million tokens) for visual grounding, and GPT-4.1-minn ($0.40 per mil- lion tokens) for the verifier to reduce costs. A full-resolution screenshot (1920 x 10800 typically consumes around Ik tokens for GPT-4.1 and 2k tokens for the computer-use-prent verification, we downsample screenshots to 960x 540, which results in approximately Ik tokens per image for GPT-4.I-mint Each execution step costs roughly $0.01 and typical trajectory con- sists of 50 steps across 6 subtasks. This results in an average total cost of approximately $0.60 per trajectory. Since we can generate tasks from the 6 subtasks, each task is only $0.1.
model. For
F APPLYING AGENTSYNTH PIPELINE FOR WEB AGENT
gpt-4.1
50
Success Rate (%)
40
30
20
10
Baseline
Summarized
Task
Task 2
Task 3
Task 4
Figure F.1: Comparison of model performance at different task levels for the web agent built in the InSTA environment. 2-sigma error bars are included.
To highlight the generality of our pipeline, also apply it to web agent environment (InSTA (Trabucco et al., 2025)). This section describes results for our test dataset of 194 tasks in 61 task sequences from InSTA, which converts webpages into text outlines with interactable elements an- notated with numerical IDs. The agent can choose from a set list of actions like clicking, filling out text fields, and going to websites, on any of the interactable elements." 3 Playwright server hosted on Docker (Merkel, 2014) is then used to access the Internet and send GET and POST requests. At least 20GB needed for the InSTA Docker image, and ideally more for storing Playwright calls; on an E2 Google Cloud Compute VM with 8 vCPUs, generating five task sequences using two workers takes somewhere from 60-90 minutes.
In InSTA, shown in Figure F.I, similar drop-off's in task difficulty occur as in Figure C. Sum- marized tasks are substantially more difficult than the baseline existing tasks in the InSTA dataset (Trabucco et al., 2025), from 40% and 12% for gpt-4.1 and o4-mini respectively to 12.9% and 6.45%. Furthermore, as shown in Figure F.2. InSTA summarized tasks also grow in description length the number of tasks included increases.
Sinne wehsites are less general-purpose than applications, most personas are not relevant given website, so personas used for InSTA task generation.
26

---

Published as a conference paper at ICLR 2026
140
Average Token Number
100
80
40
20
6+
Task Difficulty Level
Figure F2: Average token count for different task levels in InSTA, The tasks become more complex as the number of subtasks increases.
Table F.1: Prompt for Task Proposer Agent for InSTA.
System Role
What does this screen show? Imagine you are real user on this webpage. Given wehpage link, please provide a single task that user might perform on this website and the corresponding first action towards completing that task. You can use any software in the computer and the web. Be creative and come up with diverse tasks. The task should be simple enough that can be finished in few steps. You should propose tasks are clear and specific.
Task proposal rules:
1. The website should be explicitly mentioned in the task description.
2. The task should be specific, clear, and casily evaluated, not open-ended.
The task should be achievable within 1-3 steps.
4. The task should be relevant to the content the webpage.
5. You should only propose tasks that do not require login to execute the task.
6. Generally try to avoid tasks that require sending emails, submitting forms, login, or
other forms of communication.
Provide concrete information or constraints the task, use mock-up up information (identifier, number, personal information, name, attributes. etc.) to make the task more specific and realistic.
8. The task description should provide all the necessary information to complete the task.
9. Do not propose tasks that are not possible in the Playwright API, such as download- ing reports or opening and interacting with files. Output with JSON block:
! "task": <TASK>" "action"""<ATTONN"""
User Role
Given the website: [website] and webpage layout (webpage_tedt please propose
task.
27

---

Published as a conference paper at ICLR 2026
Table F.2:
Prompt for Follow-up Task Proposer Agent for InSTA.
System Role
What does screen show? Imagine you are real user on this webpage. Given website link, and the tasks the user has done, please provide single followup task that a user might perform on this website and the corresponding first action towards completing that task. You creative and come up with diverse tasks. The task should be simple enough that can finished a few steps.
any software in the computer and the web. Be
Task proposal rules:
1. The wehsite should be explicitly mentioned in the task description.
2. The task should depend on the previous tasks.
task should be specific, clear, and easily evaluated, not open-ended.
4. The task should be achievable within 1-3 steps.
5. The task should be relevant to the content of the webpage.
6. You should only propose tasks that do not require login execute the task. Provide conerete information or constraints the task, and use mock-up information (identifier, number, personal information, name, attributes, ctc.) to make the task more specific and realistic.
8. The task description should provide all the necessary information to complete the task.
The task should relevant the overall task listed in user prompt, if applicable. the overall task is achievable within a few steps, you can simply propose the overall task as the followup task. If the overall task is already achieved, you can propose an extension of the overall task.
10. Do propose tasks that loading reports or opening and interacting with files.
possible in the Playwright API, such down-
11. Try to avoid tasks that require sending emails, submitting forms, login, or other forms of communication.
12. Avoid tasks that modify the backend state of the website.
Output with JSON block: **> ""TASK""""
"<ATIONNN""""
User Role
Given the website: (website), webpage layout: [webpage_text, and task history: (task hiistsh)ttr),, please propose followup task. [If given: "Overall task: (over-
all task)."]
28

---

Published as a conference paper at ICLR 2026
G OSWORLD ACTION SPACE
Table G.I: Action space for the computer-use agent. The percentage indicates the relative frequency of each action type in the full AgentSynth dataset.
Action Type
Description
Percentage
click [x, y, button]
Click at position (z, y) with button of left right.
59.1%
write [text]
Type text.
12.1% 1%
press [key]
Press single key.
8.2%
scroll [amount]
Scroll up down for amount.
5.4%
[xr, y]
position (æ, y). Drag from (#1, y1) to (#2, y/).
Move the
5.3%
drag [Ir, yt zz, ya]
4.6%
hotkey [list of keys] double-click [z, yl wait
Press several keys at the same time Double click at position (r, y). Wait 5 scconds
3.0%
1.5%
0.8%
29


Reviews:
[{"review_id": "jgXmWHvLqB", "reviewer": "Reviewer_kK3T", "summary": "This paper proposes AgentSynth, a dataset of synthetic computer-use agent tasks in the OSWorld environment, generated using a subtask-based compositional approach. The approach leverages a loop of subtask-level proposal, execution, verification and potential subtask revision, then summarization to compose the subtasks into one coherent task. The authors show that baseline agent's performance suffers for more difficult tasks consisting of more subtasks, indicating the effectiveness of the AgentSynth in exposing gaps in agent capabilities.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important problem of testing LLM agents in complex tasks involving multiple subtasks, which is not easily controllable in existing benchmark works. The proposed approach is intuitive and cost effective for composing arbitrarily complex sequences of salient subtasks.\n- The results indicate that baseline agents indeed suffer from compositional tasks.\n- The paper is well written and easy to follow.", "weaknesses": "- My main concern with the paper is the lack of advanced agent approaches evaluated on the benchmark. The baseline agent tested consists of a basic agent with a simple prompt, and in my understanding only represents the performance lowerbound on the benchmark. Without evaluating more advanced approaches, it is difficult to understand how well competitive agents would perform on the benchmark.\n- A missing citation is [1].\n\n[1] Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web. Furuta et al., TMLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10Pm1L2G1G", "reviewer": "Reviewer_jFe8", "summary": "The paper proposes AgentSynth, an automated pipeline that synthesizes long-horizon, realistic computer-use tasks by chaining LLM-generated simple subtasks and then summarizing them into a single complex instruction. The key idea is to exploit information asymmetry between generation and evaluation. Difficulty is tunable by subtask count, spanning web/OS/office/coding; benchmark shows steep agent performance decay with horizon. MLLMs perform poorly and degrade steeply with difficulty; the pipeline is also inexpensive ($0.60 per trajectory).", "questions": "1. Can you provide asymmetry ablations and verifier calibration results to justify the difficulty and scoring?\n2.  Although increasing task horizon typically enhances diversity and difficulty, it also increases training cost. Under a fixed training compute/token budget, have you evaluated whether curating higher-quality (e.g., cleaner supervision, lower noise, stronger verifiability, clearer goal decomposition) short/medium-horizon data can outperform simply lengthening horizons in terms of compute-normalized effectiveness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Clever use of information asymmetry to make generation easy but evaluation hard; difficulty is tunable via subtask count (d), enabling principled long-horizon benchmarks.\n2. Results show a sharp success-rate drop as d increases, making the benchmark’s discriminative power evident.\n3. Practicality & scale: diverse, multi-tool tasks on real desktop environments; cost-efficient generation with transparent cost accounting.", "weaknesses": "1. Limited causal evidence: missing ablations of asymmetry vs. direct instruction, and broader verifier calibration (agreement curves, partial-credit) across tools and difficulty.\n2. Difficulty = horizon is not fair enough: other metrics like complementary axes (fine-grained perception, long-term memory, interrupt handling) would enrich difficulty control.\n3. The verifier is still LLM-based and can misjudge corner cases. More human audits or adversarial stress tests for the verifier would strengthen claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RS7oxgECG", "reviewer": "Reviewer_CsTg", "summary": "This paper introduces AgentSynth, a scalable and cost-efficient pipeline for automatically synthesizing high-quality tasks and trajectory datasets for generalist computer-use agents. Motivated by information asymmetry, AgentSynth iteratively proposes and solves simple subtasks and summarizes them into a complex task. This process is also monitored by a task verifier and a task reviser. The paper conducts a comprehensive statistical analysis on the generated datasets. Experiments show that even strong multimodal models fall short in the challenging tasks generated through AgentSynth.", "questions": "1. How significantly would replacing the base model for data synthesis with other models impact the quality of the synthesized data?\n\n2. Can this pipeline be extended to domains beyond OS agents?\n\n3. Can training with open-source models on this trajectory library achieve far superior results compared to closed-source models?\n\n4. Will this dataset be open-sourced?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The main contribution of this paper is the establishment of a challenging benchmark that is more cost-effective compared to other methods. Additionally, the paper provides a detailed statistical analysis regarding the benchmark.", "weaknesses": "1. Despite its effectiveness, the \"easy-to-hard\" data synthesis approach in this paper is not entirely novel and can be observed in data synthesis across various other domains. This aspect diminishes the paper's innovativeness.\n\n2. The paper devotes a significant amount of content to statistical analysis of the dataset. Some data cases presented in the main text are unnecessary, resulting in a limited experimental section and potentially giving the impression of a less robust study. It is recommended that the authors allocate more space to additional experimental analyses to provide readers with more insights.\n\n3. The evaluation setting in the experimental section of this paper lacks the use of any agent scaffold, which is too simplistic and may not adequately represent the true capabilities of the model. For instance, if we were to follow the AgentSynth pipeline (breaking down complex problems into subtasks and gradually solving each subtask, equipped with a verifier and reviser), would this significantly improve the results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)