R-WoM: Retrieval-augmented World Model For Computer-use Agents
Authors: Kai Mei
Jiang Guo
Shuaichen Chang
Mingwen Dong
Dongkyu Lee
Xing Niu
Jiarong Jiang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Computer-use Agent
World Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large Language Models (LLMs) can serve as world models to enhance agent decision-making in digital environments by simulating future states and predicting action outcomes, potentially eliminating costly trial-and-error exploration. However, this capability is fundamentally limited by LLM's tendency to hallucination and their reliance on static training knowledge, which could lead to compounding errors that inhibit long-horizon simulations. 
To systematically investigate whether LLMs are appropriate for world modeling, we probe two core capabilities of world models -- *future state prediction* and *reward estimation* -- through three tasks: next-state identification, full-procedure planning alignment, and milestone transition recognition. Our analysis shows that while LLMs effectively capture immediate next states and identify meaningful state transitions, their performance rapidly degrades in full-procedure planning. This highlights LLMs’ limitations in reliably modeling environment dynamics over long horizons. To address these limitations, we propose the Retrieval-augmented World Model (R-WoM), which grounds LLM simulations by incorporating factual, up-to-date knowledge retrieved from external tutorials. Experiments show that R-WoM achieves relative improvements of up to 23.4\% and 16.3\% on the subsets of OSWorld and Webarena compared to baselines, with particular advantage in longer-horizon simulations.

Published as a conference paper at ICLR 2026
R-WOM: RETRIEVAL-AUGMENTEE WORLD MODEL FOR COMPUTER-USE AGENTS
Kai Mei Rutgers University
Jiang Guo, Shuaichen Chang, Mingwen Dong
AWS Agentic Al {gujiang, cshuaich, mingwa]@amazoon. com
kai.mei ereeeeuutgersee. edu
Dongkyu Lee, Xing Niu & Jiarong Jiang
AWS Agentic AI {akleekr, xingniu, jiarongj) amazon. com
ABSTRACT
Large Language Models (LLMs) can serve as world models to enhance agent decision-making in digital environments by simulating future states and predicting action outcomes, potentially eliminating costly trial-and-error exploration. How- ever, this capability is fundamentally limited by LLM's tendency to hallucination and their reliance on static training knowledge, which could lead to compound- ing errors that inhibit long-horizon simulations. To systematically investigate whether LLMs are appropriate for world modeling, we probe two core capabil- ities of world models - future state prediction and reward estimation - through three tasks: next-state identification, full-procedure planning alignment, and mile- stone transition recognition. Our analysis shows that while LLMs effectively cap- ture immediate next states and identify meaningful state transitions, their perfor- mance rapidly degrades in frol-proee planning. This highlights LLMs' limita- tions in reliably modeling environment dynamics over long horizons. To address these limitations, we propose the Retrieval-augmentt which grounds L.M simulations by incorporating factual, up-to-date knowledge retrieved from external tutorials. Experiments show that R-WoM achieves relative improvements of up 23.4% 3,.4% and 16.3% on the subsets of OSWorld and Webarena compared to baselines, with particular advantage in longer-horizon simulations.
World Model (R-WoM),
INTRODUCTION
World models have evolved from early symbolic planning systems sophisticated neural chitectures that learn latent representations of environment dynamics. Model-based reinforce-
ment learning (MBRL) approaches, such as Dreamer v1-3 (Hatner et al. 2019 2020 2023) and
MuZero (Schrittwieser et al.) 2020), learn latent world models to "imagine" trajectories before se- lecting actions. More recently. Large Language Model (LLM)- world models (Hao et al. 2023] Wang et al. 20244] Zhang et al. 2024 Ge et al. 2024) have emerged as new paradigm, leveraging large-scale pretraining to reason about action consequences in realistic digital environments. They show particular promise for hong-horizon planning for browser and computer-use agents, where mentally simulating future states can mitigate irreversibility and reduce costly trial-and-crror.
However, due to their inherent tendency toward hallucination and reliance static parametric knowledge, LLMs perform world modeling in a fundamentally ungrounded manner. Some stud- ies explore the grounding of world models to improve video understanding (Ge al. 2024) or navigation in text-based simulated environments (Zhou et al., (024). However, there is still a gap of the world modeling of knowledge in complex multi-turn realistic environments (e.g., computer-use). For example, in an OS environment, as illustrated in Figure I] without proper grounding, computer- use agents struggle to adapt to setironment-speeciin knowledge and often generate procedural steps that seem coherent but are ultimately infeasible to execute.
"Corresponding Author. Work done when Kai intern AWS Agentic AT.

---

Published as a conference paper at ICLR 2026
Tasi: Copy the screenshot 1.p from the desktop where my cursor located
With Internal World Knowladge
Dapy Eoe Aadistaete
With Grounded World Knowledge
Dhozaclmartimag
LGrocc
Image.
Fir Filelgaa bax
2.Deact
car fereed above cullcheciche
Lose cursor locetion and get stuck
Succeed: Click "insert" keep position
Figure 1: Example task: "Copy the screenshot 1.png from the desktop to where my cursor is lo- cated." (Left:) Using LLM's internal world knowledge, the agent loses cursor location and gets stuck. (Right:) With grounded world knowledge from tutorials, the agent uses the correct "Insert operation while maintaining cursor position.
To systematically investigate whether LLMs can serve as effective world models, we probe two core capabilities: future state prediction and reward estimation. We design three evaluation tasks: next-state prediction and full-procedure planning alignment to assess LLMs' future state prediction capability; and milestone transition recognition to assess LLMs' reward estimation capability. Our analysis reveals that while LLMs demonstrate strong short-term dynamics understanding, such as identifying state changes and recognizing transition outcomes, they fail to maintain accuracy in full-procedure planning. This performance degradation over longer-horizon reasoning highlights fundamental limitations of LLM--bssed world modeling.
Motivated by these findings, we propose the Retrieval-augmenned World Model (R-WoM) frame- work, which enhances LLMebbsed simulations by grounding them in external knowledge drawn from environment-sppecfee tutorials. The core insight behind R-WoM is that while LLMs possess broad world knowledge from pretraining, they lack the specific, up-to-date procedural knowledge required for accurate simulation in dynamic digital environments. Recent work suggests that tu- torials can function as high-level abstractions of environment dynamics (Xu et al.] 2024; Zhang et al., 2025a Su et al., (2025). However, standard retrieval pipelines often surface noisy or tangential information, which undermines the alignment between retrieved tutorials and the world-modeling process. For instance, query about "fork chatgpt" might retrieve general Git forking tutorials rather than specific procedures for the current application context. To mitigate this, R-WoM incor- porates reasoning-bused RAG pipeline that combines query rewriting with LLM-based reranking to improve the relevance of retrieved tutorials. In contrast to prior approaches that rely on computa- tionally expensive iterative rollouts between policy and world models (Gu et al. 2024 Fang al.) 2025), R- WoM leverages the more lightweight long chain-of-thouge soning mechanism for multi-step simulation. Moreover,
(CoT) (Guo et al. 2025) rea-
observe that the of absolute reward
estimation in existing works (Chae et al., 2024, Gu et al., 2024 Fang et al. (0255) could introduce biases and lead unstable action scoring. To address this limitation, we employ a listwise reward estimation strategy that ranks simulation rollouts relative to each other rather than assigning absolute scores, leading to more robust and consistent action selection. Our key contributions are as follows:
Systematic probing of LLMs as world models. We conduct comprehensive evaluation reveal- ing that while LLMs excel at understanding immediate state changes and local transitions, they critically fail in producing procedures aligned the environments over long horizons.
Retrieval-augmenter world modeling framework. We propose R- GoWM,, retrieval-un retrievallaugmmented framework that grounds LLM- world models with external tutorials, enabling environment- specific adaptation through retrieval-augmenner simulation and listwise reward estimation.
2

---

Published as a conference paper at ICLR 2026
Empirical validation realistic benchmarks. We demonstrate R WoM's effectiveness two
challenging computer-use benchmarks, WebArena (Zhou et al. 2023) and OSWorld (Xie et al.)
2024), achieving consistent and substantial improvements (i.e., 5.6% 23.4%) over competitive baselines, with particular advantages in longer-horizon scenarios.
2 BACKGROUND
2.1 PROBLEM FORMALIZATION
Given an initial task goal g. a computer-use agent interacts with the environment by iteratively receiving observations and executing actions accomplish the task. Following the notation of prior work (Qin et al. 2025 Fang et al. 2025), we also introduce an intermediate reasoning component thought I, to capture thinking process. The resulting interaction trajectory can be expressed as
(01, t1, a1), (o2, f2, a2), (on, (n,,n,))))))) an));
where o; is the observation at step i, 4 is the reasoning thought generated before action selection, and ag is the executed action. At each step i, the LLM--ased policy model produces a thought-action pair conditioned on the task goal, the current observation, and the prior interaction history:
(,,,,, ~ Tpl o (()))))))))); v [
2.2 WORLD MODEL ROLLOUT
realistic environments, many actions are irreversible or costly to undo, which makes naive trial- and-error exploration infeasible. To address this challenge, researchers explore using world model (Hafner et al., 2019 2020: 2023) that can simulate possible futures to aware of the action out- comes before executing. Formally, at each decision step i, given the set of candidate actions along with their thoughts A, (((()) a ()) (1(2), ( )....... (1(TT) a ((""))) proposed by policy model p in Equation 2| the world model performs k-step lookahead rollouts estimate the potential outcomes of each action candidate j E (1.2....,
41 ~ nw 01 (0) (01) (
(1) (4 (+1 Tw 0141, 11, ((1)
(3)
0 Tw o (
(1)
n
For each k-step rollout trajectory g((0 = (o
the correspond-
ing rewards are estimated using model- (Li et al. 2023
Mahan et al. 2024) program program-based based
[Lambert et al. 2024 Guo et al. 2025) reward function:
r(a') R(+ .9)
(4)
The optimal action then selected from A, based the highest estimated reward.
a arg (t.
r(a;)
(5)
3 PRELIMINARY ANALYSIS
We focus on two fundamental capabilities of world models that are critical for computer-use tasks: future state prediction, which supports anticipating environment dynamics, and reward estima- tion, which underpins evaluating the outcomes of actions (Hafner et al. 2019 2020 2023). Recent work such as WMA (Chae et al,, 2024) explores these aspects mainly through next-state identi- fication and immediate reward estimation. However, such analyses do not fully account for the
3

---

Published as a conference paper at ICLR 2026
importance of reasoning across extended horizons. To address this, we design probing tasks tai- lored to these two capabilities by considering longer planning horizon. Specifically, for future state prediction, design the task of next-state identification and full-procedure planning aligmment, which together capture both short and long horizon dynamics; For reward estimation, we design the task of milestone transition recognition, which assesses models' ability anticipate the outcomes of intermediate transitions, We apply these probes to three state-of-the-a LLMs, Qwen-2.5- 72B Bai et al. 2025), Claude-3.5-Sonnee and Claude-3.7-Sonnel by sampling trajectories on two challenging browser/computeer-u benchmarks: WebArena (Zhou et al. (223) and OSWorld (Xie) al., 2024). In the following, we introduce these tasks and present the probing analysis, while more details with illustrative examples are provided in Appendix A.1
3.1 NEXT-STATE IDENTIFICATION
To assess the most basic requirement of future state prediction, we follow WMA (Chae et al. 2024)
to design this task where models are asked to predict the correct subsequent observation given a current state and action. Given current observation o, and action dg the model predicts the correct subsequent observation from two candidates:
0,+1 arg
max ( (+1)
P(olos.tt,a.)
(6)
Given the a n-step trajectory, we extract intermediate steps from successful and failed tra- jectories where i [2, n 2] to avoid trivial predictions from initial terminal states. For each (o;, ai 01+1) triplet, we create a negative sample by selecting the most lexically similar observation from the same trajectory. The lexical analysis conducted using difflit Python's built-in library. This requires LLMs to distinguish the true next observation o from a distractor false 0
Setup:
Results: As shown in Table I 75%, indicating they can capture short-term state changes under various lexical similarity levels.
models achieve relatively strong accuracy overall, i.c., exceeding
3.2 FULL-PROCEDURE PLANNING ALIGNMENT
While next-state identification evaluates whether LLM can capture immediate state transitions, effective world models must also reason over longer horizons. To probe this ability, we design plan alignment task, where models asked to generate execution plans and these plans evaluated for consistency with realistic environment dynamics. Formally, given a task goal g and an initial observation 01, the model produces an execution plan P = (01, .2, ar). The LLM judge then evaluates whether the execution plan conforms to the standard procedure defined in Equation 7
Trre,, if P aligns with P
B = ((.))))))))
False, otherwise.
where pa denotes the reference procedure derived from environment tutorials. The judgement is based element attributes (e.g., location, text description, visibility) and operation logic (e.g., feasibility, ordering) with respect to P*.
Setup: We sample tasks from WebArena and OSWorld benchmarks. For each task, manually annotate a reference document chunk that is directly relevant to accomplishing the task under the cor- responding environment (e.g., website or software). More annotation details are in Appendix A.2 Models are then prompted to generate execution plans without access to tutorials, and the generated plans are evaluated by an LLM judge (Claude-3.7-Sonne) by default) for alignment against the ref- erence procedures. More Details of the evaluation prompt are provided in Appendix A. and the impact of using other models as the LLM judge are provided in Appendix AA.6]
Results: Table shows that alignment remains moderate across all models, rarely exceeding 65%. This reveals a clear limitation: while LLMs can list plausible actions, they often fail to maintain
htt.as//ww.aahhhhooc...o///eeeea sonnet

---

Published as a conference paper at ICLR 2026
Table I: Probing results across three tasks: next-state identification, full- full-procedure planning align- ment, and milestone transition recognition. All values are percentages.
Next-state identification (by lexical similarity)
Full Full-procedure
Milestone transition recognition
Model
planning alignment
10.0.8)
(.9)..99
10.9.
Overall
w/ retrivval
w/ retrieeval
Accuracy
61.1
84,8
77.6
77.0
50.0
90.0
83.7
Soude-3,5-Somme
72.2
84.8
81.6
81.0
55.0
85.0
85.7
Claude-3.7 3.7de--
88,9
87.9
83.7
86.0
65.0
95.0
86.7
procedural coherence or respect environment-specffic constraints, After integrating retrieval, how- ever, performance improves across all models, shown in the w/ retrieval column. The retrieved tutorials provide additional contextual grounding, which helps the model maintain more coherent multi-step reasoning and better align its generated procedures with aathord references.
3.3 MILESTONE TRANSITION RECOGNITION
Aside from probing LLM's s capability of capturing future states, we also probe whether models can recognize task-relevant progress, an essential skill for reward estimation world models. The task evaluates whether models can distinguish promising transition sequences from unproductive ones:
$ = arg
max SET Sure, Stile)
P(success S.
where S = fo, Octi, 0.4 h from the full trajectory.
11) denotes a subsequence length I sampled at interval
Setup: We sample sequences of 1 = 3 consecutive transitions with interval h. = 2 from both successful and failed trajectories, where the intervals used to avoid repeated states. Same next state identification, we also sample steps from steps within [2, n 2] to avoid trivial predictions. For each objective g. we annotate pairs where Suree represents more promising subsequence drawn from a successful trajectory, and Stalse represents a less effective subsequence from failed trajectory. More task details can be found in Appendix A.
Results: Table I shows that all models perform strongly. Claude-3.7-Somme achieves the highest accuracy (86.7%) followed by Claude-3.5-Sonne (85.7%) and Qwen-2.5 -V.-VLL-72B (83.7%). The consistently high performance across models suggests that LLMs possess reasonable ability to eval- uate which transitions are conducive to task progress.
3.4 DISCUSSION
Overall, probing analysis reveals that modern LLMs demonstrate relatively good short-term predictive and local evaluative capabilities: they can reliably identify next states and recognize task- relevant transitions. However, these strengths do not extend to long-horizon planning, where perfor- mance deteriorates sharply in aligning its knowledge to specific environments. This suggests that LLMs might inherently lack robust generalization for world modeling across dynamic environments, thus may require external guidance to sustain accurate simulations over extended horizons.
4 R-WOM FRAMEWORK
From the probing analysis in Section we identify grounding as a key mechanism for improving the alignment of LLMs to specific environments, which motivates the design of our R-WoM framework.
4.1 OVERVIEW
As illustrated in Figure 2, the R-WoM framework employs the retrieval-augmenned way to ground world modeling during simulation. Given the task objective and current observation, relevant docu- mentation and tutorials are retrieved and reranked to form the grounding evidence set. This evidence used to condition the world model during both state transition prediction and reward estimation process. Algorithm [ summarizes the complete R- WoM pipeline, which iteratively applies this pro- cess until task completion or termination.
5

---

Published as a conference paper at ICLR 2026
Trijectories via Sell-Pay. (luooial-scars spacissccccaaa domains)
R-WoM
Knowledge Base
World Model
action
Environment
Trajectriiss
MMrraaas
Step Simulated Roliout
OS (Linux)
Synthtiic Tutoriada
Query
Task Instruction
Retreve & Rerank
Policy Model
Knowledge
+ Click
Web (Browsers)
Action branshing deduplication
Estimation
R2
factions 1-1
No
Best action
Figure 2: Overview of the R-WoM pipeline. At each time step the policy model generates m candidate actions. For each candidate, the world model grounded by retrieved tutorials performs k-step rollouts to simulate possible future trajectory. The rewards of rollout trajectories are finally estimated by world models to select the best action.
Algorithm The Pipeline R- R-WoM
Require: Task objective g, initial observation 01
1: E Retrieve and rerank tutorials relevant the objective g
2: i f-1
3: while task not completed do
A + ((t )) (t)
(6)Tm), (ad"",aa))))) a.")))] ~ Tp(-19,0.)
4:
5: for each (1,5) ag (1) A. do
6: Generate rollout trajectory +(1) LongCooT (oi, (oi 1/2) -));))
7:
end for
(L) = arg max (c(") JAA
8: Execute at observe 0,+1
9:
i +-i+1
10:
end while
4.2 DESIGN DETAILS
RAG design. We use a reasoning- aware retrieval pipeline to improve the relevance of retrieved tutorials to given task. The knowledge base contains both online documents and offline trajectories; the latter can synthesized into experience-style tutorials, which we elaborate in Section 5.5
Given a task goal we first encode it into a retrieval query 9 = fenc (g). We then retrieve candidate set Cs by selecting the top-k chunks under cosine similarity in embedding space. Because embedding similarity alone can miss fine-grained task constraints, we further apply an LLM-bbssed list-wise reranker (implemented with the policy model p) score candidates conditioned on (g. C+k) and produce ranked evidence set:
E E= rank (Cx. q):
where & denotes the final tutorial evidence used by the world model. The world model then condi- tions on E to ground future-state imagination and reward estimation.
R-WoM design. step i, given tutorial evidence ε, we enumerate candidate thought-action pairs (1,)) a(")) E A. and use a world model to simulate their future consequences. For each candidate, generate k-step imagined trajectory via a LongCoT rollout:
f {(1)
LongCoT (oi, (4) ,0, ()))
(10)

---

Published as a conference paper at ICLR 2026
In contrast to iterative rollout schemes in prior work (Gu et al. 2024 Fang al. 2025 which require multiple rounds of model calls, we adopt a reasoning- based LongCoT rollout inspired by Deepseek-RI R1 (Guo et al.) 2025). This allows the world model to unfold the entire multi-step imagi- nation trajectory within a single forward reasoning sequence, improving rollout efficiency. To further reduce rollout cost, adopt an adaptive rollout strategy based the observation that exhaustive world-model simulation is unnecessary at every step. The design has two components.
(1) Adaptive action branching: at step i, we prompt the policy decide how many action candidates to propose, generating candidates with 1 < m I n. The policy expands multiple actions only when it is uncertain about the next move; otherwise it proposes a single high-confidence action. (2) Action deduplication: before launching rollouts, we use the policy itself as verifier to prune redundant candidates, filtering out actions that are semantically equivalent. More details of the implementa- tions and ablations of the performance-cost tradeoff are in Appendix A.3 and A.4
Besides, we also observe that absolute sparse rewards that used in prior works (Chae et al. 2024] Gu et al., 2024 Fang et al. 2025) can be insensitive when candidate rollouts are all plausible difter in subtle ways. Therefore, inspired by recent progress in relative reward modeling (Liu et al.
2024 Choi et al. 2024 Guo et al. 2025),
apply list-wise ranking mechanism that compares and assigns relative preference scores using LongCoT reasoning. Additional ablations on this reward design are provided in Appendix A.6
(ti a.) = arg
max (xf) af JEA
[Fu(R(s ,g. 8.81)]
(1I)
As is shown in Equation 11, each rollout trajectory is scored relatively in the comparative context of all candidates. In this way, we aim to reduce potential bias from absolute reward signals and stablize the selection of most promising action candidate.
EXPERIMENT
To evaluate the effectiveness R-WoM, we propose the following research questions:
RQI:
Does R- R-WoM improve the performance of computer-use agents compared to established baselines in realistic environments such as browsers and operating systems?
RQ2: How do external tutorials contribute to grounding world models, and to what extent do agents benefit from incorporating this information from tutorials?
RQ3: Can -utorial-grounded world models support longer imagination horizons more effectively than ungrounded counterparts over multi-step rollouts?
RQ4: Whether R- R-WoM can be extended to scenarios where existing tutorials are scarce?
5.1
SETUP
We evaluate R-WoM against three baselines:
The vanilla approach is adapted from the official implementations: the screenshot-only version for OSWorld provided by GTA-1 (Yang et al. (025), and the screenshottaccessiiiit tree version for WebArena provided by WMA (Chae et al. 2024). This approach relies solely on the task objective, current observation and prior interaction history.
Vanilla:
RAG: retrieval-augmeened generation pipeline that retrieves relevant documentation and aug- ments the LLM before action prediction, which is built upon the vanilla approach.
WebDreamer: A world model method proposed by Gu et al. (2024) where it adopts an iterative way of generating rollouts through the communication between policy model and world model.
We conduct experiments on two challenging online environments: WebArena (Zhou et al. 2023) which spans web-based tasks across domains (e.g., e-commerce, social forums); and OSWorld (Xie et al. 2024), which covers tasks in various desktop applications (e.g., chrome, gimp, libreoffice). Specifically, we sample a subset from these two benchmarks (87 from 361 for OSWorld and 113 from 301 unique templates for Webarena) for experiments where tutorials available for
7

---

Published as a conference paper at ICLR 2026
Table 2: End-to-end performance on OSWorld and WebArena across three runs. Best in bold;
second-best underlined. 1- denotes this relative improvement over the second-best baseline
Model
Method
OSWorld Xie et al. 2024)
WebArena (Zhou et al.
2023)
Vanilla
26,36 + 2.32
21.84 = 0.42
RAG
30.84 + 1.07
22.42 0,42
Qwen-2.5-VIL-72B
WebDreamer
28.37 2.01
24.50 0.84
R-WoM
37.48 +: 2.29 121.5%
28.49 0.43 116.3%
Vanilla
22,43 2.25
27.74 0.43
RAG
22.19 + 0.92
30.70 0.41
Maude-3.5-Sommee
WebDreamer
23.48 + 2.14
29.82 = 0.41
R-WoM
26.01 = 0.44 110.8%
33.15 = 0.01 18.0%
Vanilla
28.47 + 2.27
28.92 = 0.41
RAG
27.76 + 0.75
32.75 0.72
Claude-3.7-Se Boade-3.7-Sonnee
WebDreamer R-WoM
31.24 + 2.88
31.86 0.01
38.54 +: 1,92 123.4%
34.58 + 15.6%
Not Query Seerting st Benanking wr Soth
wt Querry Hawe king
ml Baraniing
Both
100
m13
233 RAs
Performance (16
Performance (9)
80
60
60
0
Recal1@1
R
Recal1@5
Recall@1
Recall@3
Recall@5
(a) OSWorld
(b) WebArema
Figure 3: Retrieval performance under different retrieving strategies.
retrieval purpose and we collect tutorials from both online websites. Besides, we also extend R- WoM to tutorial-scarce scenarios by synthesizing tutorials from self-played trajectories (report in
Section 5.5). The details of the subsets and tutorial collection can be found in Appendix A.2
conducted on the following models: Qwen-2.5-VL. 72B (Instruct version) (Bai et al. 2025), Claude-3.5-Sonnet,, and Slaude-3.7-Sonnee serving as both the policy and world model. For methods requiring retrieval, we build the RAG pipeline with
specifically mentioned,
evaluations
Langchain R FAISS (Douze et al. 2024) as the vector store, and Qwen-3 E-medddi-mmbdng--- (Zhang
al.
2025b as the embedding model. More implementation details can be found in Appendix A.3
5.2
RQ1: END-TO-ER PERFORMANCE
Table 2 reports the overall end-to-end performance. It shows that R-WoM consistently outperforms all alternatives, with improvements of +21.5% on OSWorld and +16.3% on WebArena for Qwen- 2.5, +10.8% and +8.0% for Claude-3.5, and +23.4% and +5.6% for Claude-3.7 over the strongest non-R-WoM baselines. These results reveal that the improvements remain stable across different backbones, highlighting that R-WoM provides more consistent benefits compared with retrieval alone or ungrounded world modeling. More detailed results breakdown in domains and failure mode analysis can be found in Appendix A.5
5.3 RQ2: THE ROLE OF TUTORIALS IN GROUNDING WORLD MODELS
In this section, we evaluate how grounding quality, ranging from retrieval, to the retrieval R- R-WoM by default use, then to oracle-level retrieval tutorials translates into end-to-end task success. To have a better view of the retrieval quality used in R- R.WoM, first evaluate the performance of the retrieval component R- WoM under different configurations. The results in Figure 3) shows that retrieval recall improves most when query rewriting and reranking used together, showing that
https://githhb....
8

---

Published as a conference paper at ICLR 2026
45
40-
Ungrounded
11.1
R.VleM
RINDM (oradle)
f21
40
Ungrounded
R/NMM
242
R.MOM (oradee)
35-
Performance (%)
Performance (2)
30-
25-
20-
10-
Qwen-728
C,5
Claude-3.7
Qwen-728
Claude-3.5
Claude-3.7
(a) OSWorld
(b) WebArema
Figure 4: Performance under different grounding settings, where we compare ungrounded world model: WebDreamer, world model grounded with retrieved tutorials: R-WoM, and world model grounded with oracle tutorials: R-WoM (oracle).
Al DeDheee Doalc:1
Success +
Success Rato
Imagination on Horizon
Imagination Horizon (b) WebArena
(a) OSWorld
Figure 5: Success rates (%) across imagination horizons OSWorld (a) and WebArena (b). R- WoM (green, solid) consistently outperforms WebDreamer (red, dashed) and reaches its peak at larger imagination horizon (at horizon around 3).
these techniques are complementary. Query rewriting shows benefits when task phrasing is vague Fork Fork ChatGPT). In contrast, reranking offers henefits across both benchmarks by filtering out semantically irrelevant candidates. Overall, the results show that the retrieval can reach over 85% and 90% r respectively. Examples of the retrieved content can be seen in Appendix AA.5 To further analyze how retrieval fidelity impacts the model's reasoning quality, we compare three grounding settings: (i) no grounding (i.e., WebDreamer). (ii) grounding with R-WoM using retrieved tutorials, and (iii) grounding with R-WoM using tutorials under oracle retrieval (human annotated). The oracle tutorial setting is conducted by manually annotating one most relevant tutorial from the knowledge base to each specific task, similar as what is employed in Section A.I As shown in Figure 4, performance improves monotonically with the grounding quality, from no external knowledge to retrieved tutorials, and finally to oracle retrieved tutorials. This trend indicates that the accuracy of procedural knowledge effectively contributes to long- ong-horizoon simulation.
5.4
RQ3: ABLATION STUDIES OF IMAGINATION HORIZON
To study the effect of imagination horizon on end-to-end performance, we vary the horizon from to 4 for both ungrounded (WebDreamer) and grounded (R- WoM) world models. Figure 5) shows that, WebDreamer, the world model without grounding during rollouts, shows initial gains but quickly plateaus and even declines beyond steps, reflecting its susceptibility to compounding prediction errors. In contrast, R- WoM maintains consistently higher success across horizons on both OSWorld and WebArena, with improvements lasting up to horizon three before decreasing. These results suggest that tutorial-guided grounding helps stabilize rollouts over longer horizon simulations.
5.5 RQ4: EXTENSION TO SCENARIOS WHERE EXISTING TUTORIALS ARE SCARCE.
To further investigate how R- WoM performs in scenarios where it is hard to find existing tutorials, extend R- WoM to these scenarios by synthesizing tutorials directly from self- self-played trajectories,

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison on OSWorld tasks under tutorial-scarce settings.
Model
Claude-3.7-Some
Claude-4-Sonnet
Claude-4,5-Sonne"
Vanilla
32.25%
35.82%
45.83%
RAG
33.36%
36.93%
46. %
WebDreamer
30.86%
34.43%
46.35%
R-WoM
35.71 %
39.28 %
49.29 %
inspired by recent works leveraging procedural memory from self-play (Wang al. 2024bb 2025c). ) Specifically, we utilize 2k open-sourced trajectories released in AgentNet (Wang et al. 2025b) and synthesized approximately 1.3k synthesized tutorials that could be useful the tasks of OS World. The details of our synthesis pipeline can be found in Appendix A.2 It is important to mention that these trajectories do not have task overlap with our test tasks. Then these synthesized tutorials serve as general operation guidelines for tasks lacking online references during our evaluation on three Claude-Sonnet models (3.7, and 4.5). As shown Table 3, R-WoM has consistent improvement over other baselines across these three models. This demonstrates that R- WoM can adapt to tutorial- scarce tasks by grounding the world model using synthesized tutorials derived from self-play.
RELATED WORKS
6.1 COMPUTER-USE AGENT
One line works focuses on exploring how to improve agent's understanding computer-use actions, such building end-to-end agent frameworks (Agashe et al., 2024
2025 [ong et al.
2025
Mei et al. 2025). and training native agent models (Qin et al. 2025 Wang et al. 2025a Lai
et al. 2025) specific action grounding models (Wu et al.) 2024 Xie et al. 2025 Yang et al. 2025).
Another line of works explores treating LLMs as world models to simulate the computer-use environments. WebDreamer (Gu et al., 2024) pioneers this direction by using LLMs to simulate the outcome of candidate actions, and evaluate these imagined states with discrete reward given by LLM judge (Gu al., 2024). Subsequent works such as WMA (Chae et al., 2024) adapt this idea to improve planning by abstracting state transitions into natural language summaries. WKM (Qiao al. 2024) and WebEvolvvr (Fang et al., 2025) develop co-evolving world models and policies to progressively refine both simulation and planning, moving beyond one-horizon imagination.
6.2 TUTORIAL-USE
Parallel developments leverage tutorials indirect knowledge to train digital agents. Synatra (Ou al. 2024) converts human-oriented tutorials into 100k synthetic demonstrations to fine-tune a 7B LaMe model. Other frameworks generate trajectories guided by tutorial completion
play (c.g., AgentTrek (Xu et al.] 2024), TongUI (Zhang et al.) (025a)) to teach GUI navigation and tool use from multimodal resources. Leearn-b--interad (Su et al. 2025) synthesizes trajectories by leveraging tutorials and interaction with the environments. These approaches focus on offline tra- jectory generation by referring to tutorials while approach focuses tutorial-guidas grounding of LLMs as world models at inference time.
CONCLUSION AND FUTURE WORK
We presented a systematic study of LLLM-based world models for computer-use tasks, revealing that while they can model state transitions and recognize task-relevant progress, they fail to reliably adapt to unfamiliar environments long-term planning without grounding. To address this, we proposed the eetrieval-auggmntee World Model (R-WoM), which incorporates environment-specife tutorial knowledge during the imagination rollouts and reward prediction procedures to reduce hallucina- tions and stale knowledge. Evaluations on WebArena and OSWorld show that R- WoM consistently outperforms competitive baselines, demonstrating the efficacy of retrieval- auggented grounding for I.IM agents in dynamic browser-use and computer-use scenarios.
10

---

Published as a conference paper at ICLR 2026
8 ETHICS AND REPRODUCIBILITY STATEMENT.
Ethics. Our work uses only publicly available benchmarks (OSWorld and WebArena). While retrieval-augmented methods may inherit biases from external sources, our study remains confined to controlled environments.
Reproducibility. Our work is reproducible. We provide the algorithm process of our method, Retrieval-augmented World Model (R-WoM) in Algorithm[] The experimental setup, are described in Section 5.1 and the implementation details are provided Appendix A.3
REFERENCES
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like human. arXiv preprint arXiv:2410.08164, 2024.
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional seneralist-specialist framework for computer use agents. arXiv preprint arXiv:2504.00900 00906 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.1392. 2025.
Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. arXiv preprint arXiv:2410.13322. 2024.
Heewoong Choi, Sangwon Jung, Hongjoon Ahn, and Taesup Moon. Listwise reward estimation for
offline preference-based reinforcement learning. arXiv preprint arXiv:2408.0411 2024.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint Xriiv22401.0828. 2024.
Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. Webevolver: Enhancing web agent self-improvement with coevolving world model. arXiv preprint arXiv:2504.21024, 2025.
Zhiqi Ge. Hongzhe Huang, Mingze Zhou, Juncheng Li, Guoming Wang. Siliang Tang, and Yueting Zhuang. Worldgpt: Empowering Ilm as multimodal world model. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7346-736 7355, 2024.
Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. your mIm secretly a world model of the internet? model- based planning for web agents. arXiv preprint arXiv:2411.06553 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.11944 2025.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02192; 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 2023.
Shibo Hao, Yi Gu, Haodi Ma. Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023.
11

---

Published as a conference paper at ICLR 2026
Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. Computerr: Scaling end-to-end online reinforcement learning for computer agents. arXiv preprint arXiv:2508. 14040, 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprini 2rXiv:2411.152
2024.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.0544 2023.
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Moham- mad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through learning-to- -ooann.. arXiv preprint arXiv:2402.01877 2024.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. 12832, 2024.
Kai Mei, Xi Zhu, Hang Gao, Shuhang Lin, and Yongfeng Zhang. Litecua: Computer as mep server for computer-un use agent on aios. arXiv preprint arXiv:2505. 2025.
Tianyue Ou, Frank F Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale. Advances in Neural Information Processing Systems, 37:91618-91652, 2024.
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengiun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. Ad- vances Neural Information Processing Systems, 37:1 118471 2024.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 2025.
Julian Schrittweeser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609 2020.
Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, et al. Coact-l: Computer-using agents with coding actions. arXiv preprint arXiv:2508..339
2025.
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö Arik. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint (xXX::2501.180)) 2025.
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint 250.00244, 2025a.
Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen. Can language models serve as text-based world simulators? arXiv preprint N11v:24006064483 2024a.
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, et al. Opencua: Open foundations for computer-use agents. arXiv preprint arXiv:2508.09123. 2025b.
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint 2794422409.07429, 2024b.
Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, and Daniel Fried. Inducing programmatic skills for agentic tasks. arXiv preprint arXiv:2504.06821, 2025c.
12

---

Published as a conference paper at ICLR 2026
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for gener- alist gui agents. arXiv preprint arXhv:2410.23218, 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks real computer environments. Advances Neural Information Processing Systems, 11-52000-52999 2024.
Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, al. Scaling computer-use grounding via user interface decomposition and synthesis. arXiv preprint arXiv:2505.13222 2025.
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. arXiv preprint arXiv:2412.09666 2024.
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, et al. Gtal: Gui test-time scaling agent. arXiv preprint arXiv:2507.00591, 2025.
Alex Zhang, Khanh Nguyen, Jens Tuyls, Albert Lin, and Karthik Narasimhan. Language-guided world models: A model-based approach ai control. arXiv preprint arXiv:240.01699) 2024.
Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, and Qing Li. Tongui: Building generalized gui agents by learning from multi- modal web tutorials. arXiv preprint arXiv:2504.12677 2025a
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 2025b.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
Siyu Zhou, Tianyi Zhou, Yijun Yang, Guodong Long, Deheng Ye, Jing Jiang, and Chengqi Zhang. Wall-e: World alignment by rule learning improves world model-based 11m agents. arXiv preprint arXiv:2410.07484, 2024.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
Roadmap: Section A. introduces the design details of our probing task. Section A.2 introduces the tutorial collection, annotation and retrieval approach for our experiments. Section presents the implementation details of R- WoM, including action space definition and prompt design. Section[A. presents our discussion of cost-perfone cost-performance tradeoff and our further cost optimization of R-WoM. A.6shows shows more results Section A.5 shows our deep analysis of the failure cases of R-WoM. Section A.6 of our ablation studies.
A.1 DETAILS OF PROBING TASK
PROMPT FOR NEXT STATE IDENTIFICATION
Given the previous please reason about next state. The next state can one of the following: {state a). (state b). Please reason about the next state and return the rationale and the choice. The choice should be one of the following: A. B. Output the choice in the following JSON format:
of the web page: {previous_stace and the current action: (current_action]
"rationale": "...", "choi ce":
Task 1: Next-state identification. To assess whether the world model can predict the immediate outcome of action given the current state, the model is asked discriminate between the true next observation and a lexically similar distractor, as illustrated in Figure 6. In this way, we aim to probe LLM''s sensitivity environment changes. We construct 100 samples drawn from trajectories in WebArena for this task.
Task 2: Full-procedune planning alignment. Moving beyond identifying next state, would like to probe whether LLM can reason about longer steps of future states. As shown in A. given task objective, the model is asked generate multi-step plan, which is then validated against tutorials describing environment dynamics. The evaluation measures whether the model's 's procedure aligns with realistic element locations, operation sequences, and interaction methods. To assess this capability, we construct 40 samples from trajectories in both OSWorld and WebArena.
PROMPT FOR FULL-PROCEDURE PLANNING ALIGNMENT
You grounding validation assistant that verifies whether accurately grounded in the provided documentation.
operations in plan
Evaluation criteria
Element Text Accuracy: Exact text matches between plan and tutorial for referenced elements.
Location Consistency: Location indicators (position, context) align with tutorial descriptions.
Operation Sequence: Prerequisites dependencies match tutorial methodology.
Interaction Method: Specified actions (click, input, select) align with tutorial instructions.
Attribute Precision: Element types, properties, and characteristics match tutorial specifications.
Evaluation principle
Accept: Plan steps that extend beyond tutorial scope (additional operations are allowed),
Reject: Any tutorial-referencee operation with misaligned text, location, method.
Output Format
Output your response the following JSON format:
"rationale"; "Your rationale of your evaluation", "answer": "yes/no"
14

---

Published as a conference paper at ICLR 2026
State A (Ground truth)
(ourent)) Carnegie University!
OpenStreetMaplrbnil RootWerrrees Carnegie Mellon University OperSSreeeMapp focused: Truetai(13)
Current action
Current state
type [10] [Camegie Mellon
University\n) where [1d] toxtbox
"wIMMM7166 16
Websitte
Tab (current): Not Found
'Search' required: False
(penStrectMepirna]] RostWabArea 'Not OpenStreotMap Truelri((13)
State B (Lexically similar)
1nitts/716) link Website and API terme'
Tab (current): Carnegie Mellon University
Pittsburgh OpenStrantMapinnlnyy RootWabArsa "Camagin Melion University Pittsburgh
OpenStraatMap focused: Truaknis[13]
1AN1M17161 link Website and API terms!
Illustration of the next-state identification probing task. Given current state and an action, the model must choose between two candidate next states: (A) the ground-truth state, and (B) lexically similar distractor. This task evaluates whether the world model can correctly predict the true next observation rather than being misled by textual similarity.
Figure 6:
Task: Tell me the full name, gittab account name, location and email adcress of the contributor who has the most commits to branch php52
Tutorials
contributor analytics
To view contr butor analytics for project:
Generated Plan
"Search go to find
1. Inital navigation steps:
your project.
Look for and click on 'Branches' in the left sidebar menu
2. Select "Analypo > Contributoo analytica
should
branchas
page
On the branches page:
project commit history
Search for "ph52" branch the branch list
Note:
viaible, may
Whether aligned?
To view a list commits made project members day:
search functionality within branches
b. Check have proper access to view all branches
Varify the branch axiets
1. On the left sidebar, select" "Search or go to" and find project.
the pho52 branch:
1. Select "Analyze Contributor analytica"..
Click on the branch name to view its dietalis
1. Select "Hiistory**
Look "Contributors" similar tati/section
1. From the *Branchas" (**main*") crapdown list,
select the branch
cammits number commits made by members
the contributors section:
1.
Soott contributoons by number of commits (if such sorting is available) most commits
on a spacitic day, hovar over the line chart.
identity contributor
Optional. Filber the results.
For the identiffld top contributor:
author,
"Authorr dropdown
Click on their profile uniu/name to view their details
select the user whose commits you want to view. To filter commit message, the text box, enter
Look thair full lacation, email Information
Figure 7: Illustration of the full- full-procedure planning alignment probing task. Given a task objective (top), the model generates multi-step plan (left), which is then compared against environment- specific tutorials (right). The evaluation checks whether the generated procedure aligns with the tutorials in terms of navigation logic. element selection, and operation feasibility. This task assesses the world model's ability to sustain long-horizon procedural reasoning in realistic environments.
PROMPT FOR MILESTONE TRANSITION RECOGNITION
You are evaluating web automation trajectories to identify which one more likely to succeed in com- pleting the given task.
The following trajectories show segments from different agent attempts the task. Both agents were following the same initial steps. diverged when they chose different actions a critical decision point. Your task is to determine which trajectory segment demonstrates better progress toward completing the task objective. You need to output in the following JSON format as:
"answer": "A/B", "racionale"; "xxx"
15

---

Published as a conference paper at ICLR 2026
Get stuck in selecting books as the forum type to create submission X
Enter the books forum and then fill submission form
Figure 8: Illustration of the milestone transition recognition probing task. Given a sequence of transitions, the model must identify whether they reflect meaningful progress toward the goal. In this example, the top path shows an unproductive transition where the agent gets stuck trying directly select "books" as a forum type, failing to proceed. The bottom path shows a more promising mile- stone transition: the agent first enters the books forum and then successfully fills out the submission form. The task evaluates whether the world model can distinguish between effective and ineffective procedural progress.
Task 3: Milestone transition recognition. To probe reward estimation capability of LLMs, design this task to assess whether I.IMMs have the capability to capture meaning state transitions. As shown in Figure 8 the LLM presented with pairs of trajectory segments that diverge a decision point, one representing a promising milestone transition and the other an unproductive path. The LLM needs identify which trajectory is more conducive to task success. This setting is evaluated on 98 samples drawn from both successful and failed trajectories in WebArena.
A.2 TUTORIAL PROCESSING
Our framework relies on tutorials external grounding for browser- and computer-use tasks. To construct a comprehensive knowledge base, we gather tutorials from both general-purpose and environment-speccinn resourees. For cross-domain instructional guidance, include WikiHow, which provides structured, step-by-step content spanning a broad range of tasks. For environment- specific domains, we incorporate official documentation from the corresponding software web- sites. The complete list of tutorial sources is as follows:
WikiHow: htttss:/wwwwwiiihor..ooo
Google Chrome Help: httpp::/aupport.goonn how.:oogglee.o////ooo.oo.o.
16

---

Published as a conference paper at ICLR 2026
GIMP 3.0 User Manual: htama//hhhposs///boss.cc////////
Visual Studio Code ott.a//anaaniiiieaa//////aaa/aaa
http:///h/ha//////////////////// thunderbi
Ubuntu
Mozilla Thunderbird httpp//:://////oooooooooooo///10
VLC Media Player User http::////0aaaaaaeaaaaaaa/////10
3.C/en/
LibreOffice http:://aha..lhpooffffoooo//s///
GitL Documentation: httpp/aanioooopsss/////o111l....
Adobe Commerce Act.oo/cccocccciiiiiiooooaaoaaao wom/ebeeco/ee... com
docs http:///m..cceeccm///////aaaea//
From these sources, construct knowledge base of over 30k chunked tutorial documents that collectively support tasks across diverse software and website environments. Since our framework requires tutorial availability provide conerete grounding, and WebArena that can be partially mapped to tutorial examples. Specifically, in OSWorld, 85 tasks have clear and verifiable tutorial references from official online documents covering domains such as Chrome, GIMP, VSCode, VLC,, Thunderbird, while 276 tasks lack such references. For WebArena, based on its 301 unique task templates. 113 have tutorial coverage covering CMS and GitLab domains and 188 do not. This split is determined using consistent criterion: whether task's 's goal can be matched explicit operation instructions from official online tutorials or offline documentation for the target software or website. We annotate one document chunk for each task from human's perspective as the oracle tutorial each task.
sample task subsets from OSWorld
Synthesis of tutorials. In domains where it is hard to find online/offtine tutorial references, we ex- tend R-WoM by adopting tutorial synthesis approach. Inspired by Wang et al. (2025c), we employ pipeline to generate tutorials from self self-played trajectories released by AgentNet (Wang et al., 2025b).. Specifically, we first generate skill-level tutorials from each trajectory and then group synthesized tutorials by domain and vector similarity to further conduct a consolidation including deduplication and merging. The prompts we use are as below.
two-stage muthesize-then-constte
PROMPT FOR SYNTHESIZING TUTORIALS FROM TRAJECTORIES
You are helpful assistant that synthesizes skill-level tutorials from trajectory of observations and actions in computer-use environment.
Checklist
Make tutorial more general and deanonymize the task.
For example, a "readme.ixt" file should be replaced with ".txt format file".
Make the operations more general and avoid specific values unless the specific values are system settings or preferences.
example, instead of row 7 by summarizing the values range from row to row 6." use generalized Excel-style operation such as "Fill a target cell range by applying a summary formula to another cell range."
(c.g., =AVER AAgE==EAAAGGGMMAAAAAAA AGB((MMAM)
Pay attention potential blockers in the trajectory.
For example, if the trajectory is stuck in loop, should mention it in the tutorial to help avoid this.
When generating solutions, carefully observe initial state to avoid unnecessary operations.
If the task is to search information or in specific website (c.g., www.amazon.com) assume that Chrome already open and the user already on the website.
the task edit specific file (e.g., readme.txt), assume that the file is already open and already contains existing content.
Action space:
left_elick, right_elick, middle_click. double_click, left.click _drag, type, scroll, key
Output format
17

---

Published as a conference paper at ICLR 2026
Organize and abstract the operations skills. Try extract all the related skills from the given trajectory.
Each skill must contain 3-6 concrete actions that are defined in the action space, meaning the skill should not be too simple or too complex.
Avoid unnecessary actions such closing window verifying command exccution success. example skill:
<skill>0pen the settings page in Chrome browser <preroquisites>
The Chrome browser is open and the homepage of amazon. is loaded.
</prerequisites>
<actions>
left_click the three dots icon in the tep ri right corner of the browser.
Leit_clich the settings option open the settings page. </actions> </skill>
PROMPT FOR CONSOLIDATING TUTORIALS
You are helpful assistant that merges similar skills from given documents.
Action space:
left_clice click, right middle_click, double_click, left.clickk. drack.draag...........o type, scroll, key
Output format
multiple skills represent similar operations, merge them into single skill to avoid duplication.
multiple skills achieve the same goal through different methods or paths, combine them into one skill and descrihe the alternative approaches within it.
After merging, make each skill still maintains the original format with <prerequisites> and <act ions> tags.
Carefully verify that no duplicate skills remain and that all unique skills are retained.
Output all qualified skills in the following format:
<skill>skils 1</skill> <skill>ski11 2</ski11>
<skill>skill N</skill>
To retrieve useful tutorials at inference time, we adopt reasoning-basd based retrieval strategy. This involves query rewriting anonymize and generalize task queries, followed by L.M--based rerank- ing to reduce false negatives that may arise when relying solely on cosine similarity. The detailed prompts used for query rewriting and reranking provided below.
PROMPT FOR QUERY REWRITING
You Al assistant that rewrite original query into comprehensive, searchable queries that are easier retrieve answers from documents. You must follow these rules:
well-structured and clear with details: Try to make query detailed clear. For example, instead of title like "Fork ChatGPT", a good rewritten query would be, "How could fork the ChatGPT repository in the gitlab?"
Organize original query
Generalize Personal Details: Replace all specific, personal information (like names, file names, file location) with general descriptions (like "a user". "a xxx format file". desktop")..
18

---

Published as a conference paper at ICLR 2026
Table Action space for WebArena and OSWorld.
Environment
Action
Definition
click
Clicks webpage element identified by id.
type
Types text into webpage element; may submit appropriate.
hover
Moves the cursor over webpage element.
press
Presses key or key combination.
scroil
Scrolls the puge up or down.
WebArena
new.tab
Opens new browser tab.
focus
Focuses specific browser Closes the active browser Navigates current to URL.
close tab goto
go.back forworddrd
Navigates to the previous page.
Navigates to the next page.
stop
Terminates task and returns an answer (use N/A if unknown).
left fck
Left-elicks described UI element in the desktop environment. Right-clicks described UI element in the desktop environment. Middle-clicks described UI element in the desktop environment. Double-elicks described UI element in the desktop enviromment. Triple-clicks described UI element in the desktop environment. Drags from one descrihed UI location to another. Presses list of keys.
right.elies
midd_e_click
double_click
trip click left.click and drag key
OSWorld
scroll
Scrolls within described element.
hereil
Scrolls within descrihed element horizonally.
type
Types text into described element.
wait
Pauses execution for short duration. Ends the task successfully and returns the final answer if any. Ends the task with failure and stop.
done
fail
PROMPT FOR RERANKING
Your task is to re-rank list of documents based on their relevance to given task. Carefully analyze the task and each numbered document. Your goal is to identify which documents are helpful for completing the task and order them accordingly. Your output must must be a list of the original document indexes, sorted from most relevant to least relevant.
single JSON object with one key: "reranked indexes". .nndeeeed.indxess. The value for this key
Example format:
"reranked [0, 2, 1]
A.3 IMPLEMENTATION DETAILS OF R-WoM
To enable automation in browser and computer-use environments, adopt the official action space definitions provided by WebArena and OSWorld as summarized in Table 4 practice, we find that direct action coordinate mapping in OSWorld poses challenges for models such the Qwen series and Claude-3.5-Sonne. To address this and enable the policy model to generate more effective actions during world model rollouts, employ GTA-1-7B (Yang al. 2025) grounding model to assist in action generation when evaluating on OSWorld. For retrieval-related approach (i.e., RAG and R- R-WoM), we use top-5 retrieved document chunks by default to put them into the LLM's context.
auxiliary action
PROMPT FOR GENERATING ACTION CANDIDATES (ADAPTIVE BRANCHING)
You are a reasoner that analyzes the current state, previous actions, and task progress to determine the next required action.
19

---

Published as a conference paper at ICLR 2026
Available actions
Action space definition
Rules for success
When pressing keys, ensure held/pressed keys are within SEEYBOARD_KEYS))
Output single action each step; do not bundle multiple intents into step.
Only issue actions that are valid for the current observation (c.g., do not type into buttons or click static text).
Strictly avoid repeating the action if the interface state unchanged. Response JSON schema
"observation": "Description of current state and any changes observed", "action_candldater:
lrtbuught_an__actiin "Why this action is appropriate given the observation",
"action_code""
"action_tyye"; "action_type",
"parameters"; "paraml": "valuel", "param2"; "value2"
Output requirements
Observation: First describe the current environment changes:
Wrap the content in <cbservat ion>
</observation>,
Focus on what has changed from the previous state; keep concise.
Do not describe intended actions in this section.
Action candidates: Then propose to {ACTION_N) N} candidate next actions:
Propose candidate when progress on track and the next step unambiguous.
Propose (ACTION N} candidates only when agent stuck or previous actions failed. Candidates must be diverse (e.g.. different interaction types or differrnt target elements), not minor variants of the same attempt.
Wrap the full list <act in candi dates>
</act w/wctioo.candiii.oom candidatess
Order candidates by confidence (most confident first).
Each candidate must include:
thought: wrap in <thought>> action: wrap in faction> code.
</thought>, describing the rationalc/intent. </action>, containing exactly one line of Python
The action code must:
Use the aci class.
Be enclosed in >* ''yyhon fences.
Contain exactly one executable action line (e.g., aci Left.click (1100, 1001)).
PROMPT FOR DEDUPLICATION
You helpful assistant that assesses whether list of proposed action candidates contains meaningful diversity. Your goal is determine whether invoking World Model (WM) simulate and rerank candidates necessary.
When WM is NOT needed (output NO).
Minor parameter variance: candidates execute same action same logical target, differing only non-semantic details (c.g., small coordinate shifts or pixel-level variations).
20

---

Published as a conference paper at ICLR 2026
candidates permutations of independent steps in the workflow (e.g.. filling Name then Email vs. Email then Name).
Trivial reordering;
When WM is needed (output YES).
Semantic diversity: candidates propose substantively different strategies for the curent sub-goal
(c.g. using scarch har vs. navigating via category links).
Distinct targets: candidates interact with different UI elements that lead to different exccution paths outcomes.
Output format
<judgment>YBr >YES NO</ jod/jnngmet>>>
<rationale>Brief raticnale for the judgment.<<rattione
PROMPT FOR RETRIEVAL- FUTURE STATE ROLLOUTS
You are world-model assistant with extensive knowledge of desktop and web Uls. Given the task objective, and candidate action and the observation before the action candidate, you must "simulate the future" and describe the plausible future states.
Available actions
Action space definition
Tutorial usage guideline
Use tutorials to identify efficient workflow patterns that should be predicted as likely outcomes.
Provide a reference to the tutorial the current situation matches the standard operations in the tutorials. If the current situation does not align with tutorials, rely on internal world knowledge instead.
Environment awareness checklist
Visible UI elements: text, icons, menus, modals, tooltips
Element states: cnabled/disabled, focused/hovered, loading progress
Hidden or off-screen affordances revealed by scrolling clicking
Cursor position, caret position, selection highlights
Global context: file system changes, network requests, OS dialogs
Output Format
Produce ordered chain from STATE (current) up STATE (1 S (k));: you may stop early further prediction useful.
PROMPT FOR RETRIEVAL-AUGMENT REWARD ESTIMATION
You are an agent that evaluates actions by considering the observation hefore the action candidates and the potential outcomes of these actions.
Tutorial Grounding Guidance Priorize action sequences that follow the standard operations in the tutorials and have captured the milestones and conditions make more meaningful progress achieve objective.
Output Format
Output your response in the following JSON format:
"ranking": [x, x, x] "indexes of the acticn candidates, most
promising first", "thought": "your rationale for the ranking result"
A.4 COST-PERFORMANCE TRADEOFF
To better understand the cost of R- WoM lies in retrieval or generation, we analyze the computa- tional overhead and report the average retrieval time per task sample (including query rewriting and reranking) and the average total end-to-end execution time per task on the OSWorld benchmark in Table 5) The results show that retrieval contributes less than 2% of the total latency, confirming
21

---

Published as a conference paper at ICLR 2026
Table 5: Average retrieval and overall execution time per task sample on the OSWorld benchmark.
Avg retrieval time per task sample
Avg total execution time per task sample
Model
5.1s
989s
Claude-3.5-Somee Claude- 3.7-oonttt
5.3s
518s
4.7s
410s
Table 6: How R-WoM (Adaptive) performs compared to other methods on WoWld.
Model
Claude-3.7-Sonae Mouue--4Sonnet
Claude-4.5-Soma
Vanilla
29.40%
48.24%
59.12%
RAG
27.76%
50.82%
60.43%
WebDreamer
31.24%
49.65%
62.09%
R- R-WoM
39.13% %
56.73% %
67.84% %
R-WoM (Adaptive)
37.80%
55.18%
66.37%
Table 7: Efficiency analysis of adaptive mechanisms R-WoM.
Model
Adaptive Action Branching
Action Deduplication
WM Trigger Reduction
Claude-3.7-Sond
30.8%
31.5%
80.5%
Claude-4-Sonee
29. 29.7%
34.2%
82.2%
Claude-4,5-Sonee
28.4%
38.1%
84.4%
Table 8: Token usage comparison, where 3.7, 4 and 4.5 all represent Blaude-Sonnce models.
Model
3.7 Input
3.7 Output
4 Input
Output
4.5 Input
4.5 Output
Vanilla
32.45M
0.62M
29.91M
0.37M
29.35M
0.36M
RAG
34.39M
0.66M
31.69M
0.38M
31.10M
0.38M
WebDreamer
226.37M
4.35M
208.60M
2.56M
204.69M
2.51M
R-M
76. 19M
1.47M
70.27M
0.86M
68.95M
0,85M
R-WoM (Adaptive)
35.39M
0.85M
32.60M
0.50M
31.94M
0.49M
that the main computational bottleneck comes from the multi-step generation process rather than retrieval itself.
To analyze the effectiveness of cost optimization of R-WoM, conduct ablations by removing the adaptive action branching and action deduplication on OSWorld with Claude-3.7-Somme Claude-4- Sonnet, and Claude-4.5-Sonnet On the performance side, Table 6 shows that R-WoM retains most of the performance advantages of the R-WoM with complete world model triggering while signif- icantly reducing redundant computation. For example, with Claude-4.5-Sonnet, R-WoM reaches 66.37%, which is 12% relative gain over Vanilla and 10% gain over RAG, while remaining close the full R-WoM result of 67. 84%.. We further measure how much the optimization reduces unnecessary branching and world-model triggers.
On the cost side, Table shows that the adaptive version R- R-WoM reduces multi-action genera- tion to about 30% of the original, removes 30-40% of duplicated triggers, and lowers world-model activation to around 15-20% the initial rate. As shown Table 8| compared to full R-WoM, the adaptive variant reduces token usage by more than 50% in most settings. Its total token cost is close to RAG, especially for Claude-4.5- 4.nnetn where the difference is within 5-10%. These results show that adaptive R-WoM maintains most of the performance improvements of R- WoM while substan- tially reducing computational cost, achieving a more efficient balance between reasoning strength and inference overhead.
22

---

Published as a conference paper at ICLR 2026
Table 9: Examples of successful and failed retrieval cases.
Case Type
User Query
Retrieved Tutorial
Successful
"Computer, can you turn the webpage I'm looking at into PDF file and put it my main screen?"
"Print from Chrome - On your com- puter, open Chrome. Open the page, im- age, or file you want to print. Click File Frint, or use Ctrl+P Cmd+P. Select 'Save as PDF and click Print..."
Failure
"Please calculate the ages the employ- ees according to their birthday."
"When entering time, separate time el- ements with colons. To change the date time format in Calc, open Format Cells and select Date/Time.."
Table 10: Failure statistics of WoM w/o world model (WebDreamer) and w/ world model (R-WoM) across OSWorld and WebArena.
Retrieval Failures
Task Success Despite
Task Failure
Benchmark
Model
Retrieval Failure
(WebDreamer Failed as well)
Qwen2.5-72B
OSWorld
Maade-3.5-Somme
Claude-3.7-Somae 7-Sonnet
4
3
Qwen2.5-VL.c
VL--2B
17
WebArena
Soude-3.5-Sonnee
17
12
Claude-3.7
16
11
A.5 FAILURE MODE ANALYSIS.
To diagnose the failure cases R-WoM, we conduct qualitative and quantitative analysis on cases where retrieval fails and where overall task fails. This section illustrates representative examples of retrieved content, how R-WoM behaves when facing retrieval failure, and identifies major fail- ure causes. Table 9 shows two representative examples that demonstrate how retrieval behaves in distinct scenarios. When the user query contains explicit action semantics, the module successfully retrieves relevant procedural content. In contrast, failures tend to occur when the task requires im- plicit reasoning or semantic inference, which leads to ambiguous or misleading retrievals. These examples show that retrieval succeeds when task descriptions explicitly describe the intended op- cration but fails when the request requires contextual understanding or multi-step reasoning (c.g., manipulation of some specific data). In such cases, the textual query may not contain enough cues for the retriever to locate a matching tutorial.
We further analyze the end-to-end performance when retrieval failure happens and the failure-related statistics can be found in Table 10 From the statistics, we observe that the R- WoM behaves con- sistently WebDreamer in all of the cases when retrieval fails. Specifically, in tasks where using world model without grounding can succeed, adding retrieved content will not lead to task failure even if the retrieved content is not directly relevant. In some cases, R-WoM still completes the task successfully even when the retrieved tutorial is irrelevant. We conjecture the reason behind this phenomenon is heuristic filtering strategies we use as below:
Reranker-level filtering. The LLM- --sdedd reranker is explicitly prompted judge semantic rele- vance rather than relying purely on embedding cosine similarity. This allows it to discard tutorials that are lexically similar but semantically unrelated the target goal.
Rollout-level filtering. During world-model rollouts, the policy is instructed to incorporate re- trieved content only when it aligns with the current observation (layout, UI elements, and goal). If the tutorial mismatched, the model falls back on internal reasoning instead of being influenced by irrelevant information.
Moreover, we also investigate the failure cases of WoM, and Table II summarizes the statistics of failures across different task types. Specifically, following the task taxonomy used in the WMA
23

---

Published as a conference paper at ICLR 2026
Table 11: Failure distribution by task type across OSWorld and WebArena.
Content- Information-Seekinng
Benchmark
Model
Navigation
Modification
Qwen2.5-VL-72B
23.5%
76.5%
OSWorld
Claude-3.5-Somnc
21.8%
78.2%
Claude-3 Claude-3.7-Somen 3.7-Sonnct
19.6%
80.4%
Dwen2.5-VL-72.
37.4%
62.6%
WebArena
Claude 3.5-Sonnet
35.7%
64.3%
Claude-3. 7-Sonnet
34.8%
65.2%
Table 12: Full-procedure alignment scores given by different LLM judges.
Model
Claude-3.7-Somme
GPT-4.1
Gemini-2.5-Prr
Qwen2.5-VL-72 S--VL-7BB-Innttuu Instruct
50.0%
50.0%
45.0%
Claude-3.5-Somm
55.0%
55.0%
50.0%
Claude-3.7-Sonne
65.0%
60.0%
65.0%
Table 13: Performance comparison of different methods on stronger model backbones (Claude-4- Sonnet and Claude-4.5-Soneel evaluated on the OSWorld benchmark.
Model
Claude-4-Sonnet
Claudo-4.5-Some
Vanilla 48.24%
59.12%
RAG
50.82%
60.43%
WoM
49.65%
62.09%
R-WoM
56.73%
67.84%
(Chae al. 2024), categorize tasks into two major groups: (i) information-seeking naviga- tion, and (ii) content- modifiitioiiiiitio (i.e., editing or manipulating the content of a website, file, or software). From these results, observe that content-modification tasks (e.g., editing text in Li- breOffice or modifying code in VSCode) are where R-WoM fails more frequently. These failures often arise from ambiguous task goals (e.g., "Please calculate the ages of the employees according to their birthday"),, where the model must infer intermediate steps, or from the inherent difficulty of manipulating fine-grained content through UI actions such as dragging, selecting, highlight- ing text. Such tasks remain challenging for current computer-use agents because they require both precise visual grounding and multi-step reasoning. Exploring richer strategies such as multi-modal retrieval or more agentic, step-aware retrieval mechanisms represents a promising direction for fu- ture improvement.
A.6 MORE ABLATION STUDIES
Using different LLMS as judges for full-procedure planning alignment. To verify that our eval- uation is not biased toward a specific judge model, we conduct a cross-judge consistency ablation. Specifically,
Claude-3.7-Sonnet, 1PT-4.1, and Gemini-2.5-Pro, on the same set of tasks mentioned in Section 3.2 These results show that alignment judgments remain broadly consistent across different judge models, indicating that our evaluation not overly dependent on a single model family and is robust to variations in judgment sources.
End-to-end performance when using more powerful LLMs. To examine whether the proposed R-WoM generalizes effectively across stronger model backbones, we extend our evaluation to two more up-to-date models: Claude-4-Sonnet and Claude-4.5-Somne both of which have been opti- mized for computer-use environments. The results are presented in Table 13 We evaluate them on the OSWorld benchmark following the same experimental setup described in Section 5. of our paper. Across both stronger models. observe consistent performance improvements after inte- grating R- WoM. These results confirm that the proposed framework continues to deliver substantial gains even more capable language models.
24

---

Published as a conference paper at ICLR 2026
Table 14: Comparison between R-WoM (listwise reward) and its absolute reward variant on the WSWd benchmark.
Model
R- R-WoM
R-WoM (Absolute Reward Variant)
Qwen2.5.Ve Sennnn.-VV-72B-nnsiiie
38.05%
35.12%
Claude-3.5 3.5---nnett
26.41%
24.03%
Claude-3 Mlaude-37--oonit 7-Sonnet
39.13%
36.50%
Table 15: Ablation on the number of candidate actions (m) for listwise reward optimization, evalu- ated on the OSWorld benchmark.
Action Candidate Size (m)
Claude-3.7-Sonnet
Claude-4-Sonnet
Claude-4.5-Sonnet
2
37. 97.90%
53.67%
66.00%
3
39.13%
56.73%
67.84%
5
38. 10%
56.22%
68.41
The impact of listwise relative reward and sample-wise absolute reward. To analyze the ef- fectiveness of the listwise reward design in R-WoM, we conduct an ablation comparing our listwise reward design with absolute reward variant. The absolute reward variant is implemented by fol- lowing WebDreamer and WebEvolver to use {0, 0.5, 1.0} to represent failure, on- on-the-progee the-progress and success, respectively. As illustrated in Table 14] the listwise reward consistently outperforms the ab- solute reward variant, confirming that relative ranking among candidates provides stronger learning signals and leads more robust action selection during rollout.
The impact of candidate size on listwise reward. To further study the impact of candidate action set the effectiveness of listwise reward optimization, conduct an ablation study varying the number of candidate actions m and Table 13) reports the performance when varying the candidate size [2,3, 5). The results show that increasing the candidate size from to 3 improves overall performance across all models, suggesting that moderate expansion of the comparison set helps the judge LLM better identify globally optimal actions. However, further increasing the size can have diminishing returns when the base model is relatively weak (e.g., Slaude-37--oome and Claude-4-Sonine Sonnet).
A.7 USE OF LARGE LANGUAGE MODELS
We utilized Large Language Models (LLMs), such Claude, exclusively for ancillary support in two main areas: (i) language editing and polishing of the manuscript, and (ii) coding assistance for minor boilerplate tasks, such generating plotting scripts and small utilities. All model-generane outputs were thoroughly reviewed, modified, and rigorously tested by the authors to ensure their accuracy and appropriateness.
25


Reviews:
[{"review_id": "MLLQF2XiBY", "reviewer": "Reviewer_W8D9", "summary": "This paper addresses an important challenge in developing computer-use agents: enabling LLMs to serve as effective world models for long-horizon planning in digital environments. The authors begin with a thoughtful diagnostic study using three probing tasks that reveal LLMs excel at short-term state predictions (86% accuracy) but struggle with long-horizon planning alignment (50-65%). To address this limitation, they propose R-WoM (Retrieval-augmented World Model), which grounds LLM-based simulations with external tutorials retrieved at inference time. The framework employs chain-of-thought rollouts and listwise reward estimation for action selection. Experiments on WebArena and OSWorld demonstrate improvements of up to 25.3% and 18.1% respectively over the WebDreamer baseline.", "questions": "**Q1:** Could you provide more context on how your results compare with other available benchmarks in this space? Specifically, I noticed OpenAI CUA reported 58.1% on WebArena in early 2025. Are there methodological or setup differences that make direct comparison inappropriate, or would it be valuable to discuss these results to help readers understand the current state of the field?\n\n**Q2:** Regarding the characterization of WebEvolver using \"absolute reward estimation\" (line 311): could you clarify this? From my understanding, WebEvolver uses supervised fine-tuning with rejection sampling. I would appreciate clarification on how you're categorizing their reward mechanism.\n\n**Q3:** Your probing study identifies long-horizon planning alignment as the key limitation (50-65% accuracy). After applying R-WoM with tutorial grounding, what is the new alignment score on this metric? This would help demonstrate that your solution directly addresses the diagnosed problem.\n\n**Q4:** Could you provide more analysis of retrieval failure cases? What types of tasks or scenarios are most affected when the retrieval system fails to find appropriate tutorials, and how do these failures impact the final task success?\n\n**Q5:** Have you considered any techniques to reduce the 4-10× computational overhead? Are there scenarios where the performance gains justify this cost, and others where a lighter-weight approach might be preferable?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**1. Well-Motivated Diagnostic Study:** I appreciate the systematic probing analysis in Section 3. The three-task evaluation (next-state identification, full-procedure planning alignment, milestone transition recognition) provides clear empirical evidence for why existing LLM-based world models struggle with long-horizon planning. This diagnostic approach effectively motivates the need for external grounding mechanisms.\n\n**2. Reasonable Architectural Integration:** The paper successfully brings together two established paradigms—world models from model-based RL and retrieval-augmented generation—and applies them to the challenging computer-use domain. I find this combination sensible and well-suited to the problem.\n\n**3. Comprehensive Ablation Studies:** I commend the authors for including ablations on imagination horizons (Table 3) and grounding sources (retrieved vs. oracle tutorials in Figure 3). These studies help understand which components contribute to the overall performance.\n\n**4. Practical Efficiency Considerations:** The use of CoT-based rollouts instead of expensive iterative interactions represents a pragmatic design choice. Table 5 shows R-WoM is more efficient than WebDreamer, which is valuable for practical deployment.\n\n**5. Solid Engineering Effort:** The collection of 30k+ tutorial chunks from diverse sources and the design of the reasoning-based RAG pipeline with query rewriting and reranking demonstrates significant implementation effort.", "weaknesses": "**1. Limited Baseline Comparisons:** I am concerned that the experimental evaluation could be more comprehensive. The paper primarily compares against WebDreamer and basic policy models, but I understand there are other benchmarks in this space that could provide important context. For example, I noticed that OpenAI's Computer Use Agent (CUA) published results showing 58.1% on WebArena (available since January 2025), which would be a valuable reference point. I recognize the authors' results of 35.11% show improvement over their chosen baseline, but including broader comparisons would help readers better understand where R-WoM stands in the current landscape. I encourage the authors to discuss or compare with additional publicly available benchmarks to strengthen the evaluation.\n\n**2. Characterization of Related Methods:** I noticed that the paper characterizes some related work in ways that may not fully capture their methodologies. For instance, lines 310-316 discuss WebEvolver (Fang et al., 2025) in the context of \"absolute reward estimation,\" but from my reading, WebEvolver appears to use supervised fine-tuning with rejection sampling rather than RL-based absolute rewards. I suggest the authors revisit these characterizations to ensure they accurately represent the cited methods, which would strengthen the positioning of the listwise reward contribution.\n\n**3. Retrieval Quality Analysis:** While Figure 3 shows that oracle tutorials outperform retrieved ones, I would appreciate more analysis of how retrieval failures affect performance. The appendix mentions Recall@5 metrics (94.4% OSWorld, 85.7% WebArena), suggesting some retrieval errors occur. I recommend adding analysis of what types of tasks or scenarios are most affected by retrieval failures and how these errors propagate through the world model rollouts.\n\n**4. Computational Cost Discussion:** Table 5 shows R-WoM requires 3.8-12h runtime compared to 0.7-2.1h for vanilla methods—a 4-10× increase. While this is better than WebDreamer's 15-43h, I believe the practical deployment implications deserve more discussion in the main paper. For real-world applications, this trade-off between performance and computational cost would be important to consider.\n\n**5. Gap Between Diagnosis and Solution:** The probing study identifies 50-65% full-procedure planning alignment as the core problem. However, I would be interested to see whether R-WoM actually improves this alignment metric, or if the task success improvements come from other mechanisms. This would help validate that the solution addresses the diagnosed problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pm5ZNrsuGK", "reviewer": "Reviewer_Cuhc", "summary": "This paper investigates the capabilities and limitations of Large Language Models (LLMs) when used as \"world models\" for computer-use agents. The authors first conduct a systematic probing analysis across three tasks (next-state identification, full-procedure planning alignment, milestone transition recognition) to evaluate LLMs' ability to predict future states and estimate rewards. Their findings indicate that while LLMs excel at short-term predictions, their performance degrades significantly over longer horizons, failing to align with specific environment dynamics. To address this limitation, the paper introduces the R-WoM, a framework that grounds the LLM's simulation process in external, up-to-date knowledge retrieved from tutorials. R-WoM employs a reasoning-based retrieval pipeline and a listwise reward estimation strategy to improve the accuracy of multi-step rollouts. The authors demonstrate through experiments on the OSWorld and WebArena benchmarks that R-WoM substantially outperforms baseline methods, including vanilla agents, standard RAG, and an ungrounded world model (WebDreamer), particularly in tasks requiring long-horizon planning.", "questions": "1.\tTask subset selection: please quantify how many tasks were excluded due to absent tutorials and report the distribution of retrieval coverage (retrieval recall per task). This will help readers assess generality.  ￼\n2.\tReranker details: Is it zero-shot or fine-tuned? How sensitive are results to the reranker model choice?  \n3.\tListwise reward computation: provide the exact mathematical definition (or pseudocode) of  in Eq. (11) $f_w(\\{R(\\hat \\tau^{(j)},g,\\mathcal{E})\\})$ . How are rewards normalized across rollouts, and how does candidate set size m affect ranking quality?  \n4.\tJudge bias: how consistent are the plan alignment judgments across different judge models or human annotators? Can you release a small human-annotated subset for reproducibility? \n5.\tFailure modes: can the authors provide qualitative examples where retrieval misleads the model (noisy / tangential tutorials) and explain any heuristic filters applied?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tClear problem framing and probing suite. The three tasks isolate immediate state prediction vs. long-horizon planning vs. reward-related transition recognition, which yields diagnostic insights about LLM strengths/weaknesses.  \n2.\tEffective and practical method. R-WoM is straightforward to implement (retrieval + rerank + CoT rollouts + listwise ranking) and integrates well with existing LLM stacks.  \n3.\tStrong empirical gains. Consistent and sometimes large improvements across models and benchmarks (Table 2). \n4.\tInsightful ablations and visualization. Figures 3 provide clear comparisons of ungrounded, retrieval-based, and oracle-knowledge models. Figure 4 detail how imagination horizon impacts agent success, directly supporting claims about longer-horizon stability and providing actionable insights for the field.", "weaknesses": "1.\tLimited novelty in core formulation. The high-level idea of retrieval-augmented LLMs for world modeling is not entirely new; prior works such as WorldGPT also consider augmentation or world modeling for digital environments. The contribution here is more in the detailed design (e.g., reasoning-based retrieval + listwise scoring) and empirical analysis than a fundamentally new methodological direction.\n2.\tPotential biases in evaluation. The use of LLM-based judges (Claude 3.5 Sonnet and Claude 3.7 Sonnet) to score plan alignment and trajectory success (especially in full-procedure planning) risks evaluation circularity (model judge vs. model being evaluated) and inherits bias from the same family of models. While necessary at present, some acknowledgment (and potentially a human evaluation subset) would bolster the claims.\n3.\tSelection bias toward tutorial-rich tasks. The method requires relevant tutorials; the authors explicitly sample task subsets that have tutorials available. The generalization to tutorial-scarce settings is unclear. Include experiments on a held-out set with sparse/no tutorials, or provide a quantitative analysis of how retrieval coverage correlates with gains.\n4.\tLimited Analysis of Failure Modes: While the paper demonstrates strong overall performance, a more comprehensive analysis of R-WoM's failure modes would strengthen the work. For instance, a discussion of scenarios where the model degrades (e.g., when faced with poor, irrelevant, or incorrect tutorials) or remains brittle (e.g., with ambiguous goals or novel UI elements) would provide significant value for practitioners seeking to build upon these findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ci0hw6I2cX", "reviewer": "Reviewer_tkQf", "summary": "This paper investigates the limitations of using Large Language Models as world models for computer-use agents, particularly their unreliability in long-horizon planning due to hallucinations and static knowledge. To diagnose this, the authors first conduct a systematic probing analysis, revealing that while LLMs excel at short-term state prediction and recognizing meaningful progress, they fail to generate plans that align with real environment dynamics over extended steps. To address this, the paper proposes the Retrieval-augmented World Model, a framework that grounds the LLM's simulation process in external knowledge retrieved from tutorials. Key components of R-WoM include a reasoning-based RAG pipeline for better retrieval, a CoT-based mechanism for efficient multi-step rollouts, and a listwise reward estimation strategy for more robust action selection. Experiments on two challenging benchmarks, OSWorld and WebArena, show that R-WoM significantly outperforms strong baselines across multiple LLM backbones.", "questions": "1. Could you comment on the expected performance of R-WoM in environments with limited or no available tutorial documentation? How gracefully does the model's performance degrade as the quality of the knowledge base decreases?\n\n2. The listwise reward estimation is an interesting and intuitive design choice. Have you performed an ablation study to specifically isolate its contribution? For instance, how does the full R-WoM framework compare to a version that uses a more traditional absolute reward scoring (e.g., a 1-10 scale) on the generated rollouts?\n\n3. Table 5 shows a clear trade-off between cost and performance. For example, using Claude-3.7 on WebArena, R-WoM is ~4.75x slower than RAG for a ~7.2% relative improvement. In what practical scenarios do you believe this trade-off is justified?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Clear Motivation: The paper is motivated by a systematic probing analysis that pinpoints the specific failure modes of ungrounded LLM-based world models in long-horizon tasks, providing a clear foundation for the research.\n\nReasonable Framework Design: R-WoM is a reasonably designed framework that combines ideas like a RAG pipeline, CoT-based simulation, and listwise reward estimation.\n\nEvaluation on Multiple Benchmarks: The evaluation demonstrates performance gains over baselines across two challenging benchmarks and three different LLMs.", "weaknesses": "Limited Novelty: The core idea of the paper is to apply Retrieval-Augmented Generation (RAG) to world models. While this is a logical engineering direction, it feels more like a direct combination of existing, mature technologies (RAG, CoT, World Models) and lacks fundamental theoretical or methodological innovation. This makes the paper's contribution feel more applied and engineering-focused rather than a breakthrough in basic research.\n\nSevere Generalization Limits: The method's effectiveness is severely constrained to domains that have high-quality, comprehensive tutorials. This is a critical flaw, as it means the method is nearly inapplicable in many real-world scenarios where documentation is sparse, outdated, or non-existent. The paper fails to adequately discuss or experimentally verify this limitation, which significantly undermines its claims of general utility.\n\nUnfavorable Cost-Benefit Ratio: Although R-WoM is more efficient than the WebDreamer baseline, its computational overhead is still excessive compared to a standard RAG approach. The results show that achieving a limited performance gain (sometimes in the single digits) requires a several-fold increase in computational cost. This cost-benefit trade-off is unacceptable for many practical applications and casts doubt on the method's utility.\n\nEffectiveness of Core Component is Questionable: The results with an \"oracle\" retriever expose the performance bottleneck of the current retrieval module. This is not just a technical detail; it directly challenges the paper's central thesis that world models can be effectively grounded via retrieval. If the retrieval itself is unreliable, the foundation of the entire framework is shaky.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YAjfR4UX6O", "reviewer": "Reviewer_1XKm", "summary": "This paper proposes R-WoM (Retrieval-augmented World Model), a framework designed to enhance computer-use agents. Traditional world models for agents often struggle with long-horizon reasoning. R-WoM addresses the issue by integrating retrieval-augmented memory into a world model. \nThe method is evaluated on benchmarks of computer-use agents, showing that R-WoM significantly outperforms baseline world models and state-of-the-art methods in terms of task success rate, with particular advantage in longer-horizon simulations.\n\nOverall, this is a strong and well-motivated paper. It contributes a novel and effective combination of retrieval and world models, demonstrating strong empirical results on computer-use agents. Its main areas for improvement are in experiment soundness, experimental analysis breadth & deepness, scalability, and retrieval interpretability. With these addressed, the work could be highly impactful for the development of practical, general-purpose computer-use assistants.", "questions": "- How well does R-WoM scale when retrieval memory grows to millions of trajectories? Do you employ approximate retrieval or pruning strategies?\n- Have you tested R-WoM on completely unseen applications (e.g., apps with novel layouts or interaction paradigms)? How robust is the method in such settings?\n- Can you provide examples of what the retrieval module retrieves in successful vs. failed cases? Does retrieval sometimes introduce misleading prior experiences?\n- How accurate is the predictive component of the world model across multiple horizons? Do errors accumulate in long trajectories?\n- In a real-world assistant scenario, how would you balance retrieval latency, model complexity, and user responsiveness?\n- Could this framework generalize to other domains requiring long-horizon reasoning (e.g., robotics, scientific workflows), or is it specialized to computer-use tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This paper addresses an increasingly important challenge: enabling LLM-powered agents to use computers effectively.\n- Novel integration of retrieval augmentation with world models in the specific context of computer-use agents.\n- This paper demonstrates that retrieval + world models can substantially improve generalization and robustness. Experimental evaluation shows consistent improvements over baselines across two comprehensive benchmarks designed for multi-round interactions in realistic computer-use environments.\n- This paper is well-written and logically structured, in spite of more experimental analysis are advised. \n- Potentially impactful for building personal assistant agents capable of robust computer-use in real-world settings.", "weaknesses": "- Two benchmarks are not sound enough. Experiments are limited to a specific set of tasks. It is unclear how R-WoM would perform on diverse real-world applications (e.g., enterprise software, web navigation with dynamic layouts).\n- The evaluation lacks stress tests for extreme long-horizon/complex tasks (hundreds of steps).\n- This paper does not deeply analyze the computational overhead of retrieval at inference time. Scaling to large memory banks or diverse user contexts could introduce latency.\n- While retrieval helps generalization, it is not always clear what is retrieved and why. More analysis of retrieval quality and relevance would strengthen the contribution.\n- The world model’s predictive accuracy is not reported in detail (e.g., prediction error across horizons). Understanding when it fails would provide useful insights.\n- While strong baselines are included, it would be useful to see how R-WoM compares with end-to-end planning agents trained with reinforcement learning at scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)