R-WoM: Retrieval-augmented World Model For Computer-use Agents

## Abstract

Abstract Large Language Models (LLMs) can serve as world models to enhance agent decision-making in digital environments by simulating future states and predicting action outcomes, potentially eliminating costly trial-and-error exploration. However, this capability is fundamentally limited by LLM’s tendency to hallucination and their reliance on static training knowledge, which could lead to compounding errors that inhibit long-horizon simulations.
To systematically investigate whether LLMs are appropriate for world modeling, we probe two core capabilities of world models – future state prediction and reward estimation – through three tasks: next-state identification, full-procedure planning alignment, and milestone transition recognition. Our analysis shows that while LLMs effectively capture immediate next states and identify meaningful state transitions, their performance rapidly degrades in full-procedure planning. This highlights LLMs’ limitations in reliably modeling environment dynamics over long horizons. To address these limitations, we propose the Retrieval-augmented World Model (R-WoM), which grounds LLM simulations by incorporating factual, up-to-date knowledge retrieved from external tutorials. Experiments show that R-WoM achieves substantial improvements of up to 25.3% (OSWorld) and 18.1% (WebArena) compared to baselines, with particular advantage in longer-horizon simulations.

## 1 Introduction

World models have evolved from early symbolic planning systems to sophisticated neural architectures that learn latent representations of environment dynamics. Model-based reinforcement learning (MBRL) approaches, such as Dreamer v1-3 (Hafner et al., 2019; 2020; 2023) and MuZero (Schrittwieser et al., 2020), learn latent world models to “imagine” trajectories before selecting actions. More recently, Large Language Model (LLM)-based world models (Hao et al., 2023; Wang et al., 2024; Zhang et al., 2024) have emerged as a new paradigm, leveraging large-scale pretraining to reason about action consequences in realistic digital environments. They show particular promise for long-horizon planning for browser and computer-use agents, where mentally simulating future states can mitigate irreversibility and reduce costly trial-and-error.

Figure: Figure 1: Example task: “Copy the screenshot 1.png from the desktop to where my cursor is located.” (Left:) Using only internal world knowledge, the agent loses cursor location and gets stuck. (Right:) With grounded world knowledge from tutorials, the agent uses the correct “Insert Image” operation while maintaining cursor position. This illustrates how grounding with external knowledge enables more reliable decision-making in realistic environments.
Refer to caption: x1.png

However, due to their inherent tendency toward hallucination and reliance on static parametric knowledge, LLMs perform world modeling in a fundamentally ungrounded manner. In complex, multi-step tasks, this detachment from the environment’s real-time state can trigger cascading errors: the imagined trajectory gradually diverges from actual dynamics, producing simulations that appear coherent but are ultimately unexecutable. This limitation becomes particularly evident in realistic computer-use environments, as illustrated in  [Figure 1](https://arxiv.org/html/2510.11892v1#S1.F1).

To systematically investigate whether LLMs can serve as effective world models, we probe two core capabilities: future state prediction and reward estimation. We design three evaluation tasks: next-state prediction and full-procedure planning alignment to assess LLMs’ future state prediction capability; and milestone transition recognition to assess LLMs’ reward estimation capability. Our analysis reveals that while LLMs demonstrate strong short-term dynamics understanding – such as identifying state changes and recognizing transition outcomes – they fail to maintain accuracy in full-procedure planning. This performance degradation over longer-horizon simulations highlights fundamental limitations of LLM-based world modeling.

Motivated by these findings, we propose the Retrieval-augmented World Model (R-WoM) framework, which enhances LLM-based simulations by grounding them in external knowledge drawn from environment-specific tutorials.
The core insight behind R-WoM is that while LLMs possess broad world knowledge from pretraining, they lack the specific, up-to-date procedural knowledge required for accurate simulation in dynamic digital environments.
Recent work suggests that tutorials can function as high-level abstractions of environment dynamics (Xu et al., 2024; Zhang et al., 2025a; Su et al., 2025). However, standard retrieval pipelines often surface noisy or tangential information, which undermines the alignment between retrieved tutorials and the world-modeling process. For instance, a query about “fork chatgpt” might retrieve general Git forking tutorials rather than specific procedures for the current application context. To mitigate this, R-WoM incorporates a reasoning-based RAG pipeline that combines query rewriting with LLM-based reranking to improve the relevance of retrieved tutorials. In contrast to prior approaches that rely on computationally expensive iterative rollouts between policy and world models (Gu et al., 2024; Fang et al., 2025), R-WoM leverages the more lightweight long chain-of-thought (CoT) (Guo et al., 2025) reasoning mechanism for multi-step simulation. Moreover, we observe that the use of absolute reward estimation in existing works (Chae et al., 2024; Gu et al., 2024; Fang et al., 2025) could introduce biases and lead to unstable action scoring. To address this limitation, we employ a listwise reward estimation strategy that ranks simulation rollouts relative to each other rather than assigning absolute scores, leading to more robust and consistent action selection. Our key contributions are as follows:

- •
Systematic probing of LLMs as world models. We conduct comprehensive evaluation revealing that while LLMs excel at understanding immediate state changes and local transitions, they critically fail in producing procedures aligned to the environments over long horizons.
- •
Retrieval-augmented world modeling framework. We propose R-WoM, a retrieval-augmented framework that grounds LLM-based world models with external tutorials, enabling environment-specific adaptation through retrieval-augmented simulation and listwise reward estimation.
- •
Empirical validation on realistic benchmarks. We demonstrate R-WoM’s effectiveness on two challenging computer-use benchmarks, WebArena (Zhou et al., 2023) and OSWorld (Xie et al., 2024), achieving consistent and substantial improvements (i.e., 7.2% to 25.3%) over competitive baselines, with particular advantages in longer-horizon scenarios.

## 2 Background

### 2.1 Problem Formalization

Given an initial task goal $g$, a computer-use agent interacts with the environment by iteratively receiving observations and executing actions to accomplish the task. Following the notation of prior work (Qin et al., 2025; Fang et al., 2025), we also introduce an intermediate reasoning component thought $t$, to capture thinking process. The resulting interaction trajectory can be expressed as

$$ $(g,(o_{1},t_{1},a_{1}),(o_{2},t_{2},a_{2}),\ldots,(o_{n},t_{n},a_{n})),$ (1) $$

where $o_{i}$ is the observation at step $i$, $t_{i}$ is the reasoning thought generated before action selection, and $a_{i}$ is the executed action. At each step $i$, the LLM-based policy model produces a thought–action pair conditioned on the task goal, the current observation, and the prior interaction history:

$$ $(t_{i},a_{i})\sim\pi_{p}\!\left(\cdot\mid g,o_{i},\{(o_{j},t_{j},a_{j})\}_{j=v}^{i-1}\right),\quad v\in[1,\,i-1]$ (2) $$

### 2.2 World Model Rollout

In realistic environments, many actions are irreversible or costly to undo, which makes naive trial-and-error exploration infeasible. To address this challenge, researchers explore using a world model (Hafner et al., 2019; 2020; 2023) that can simulate possible futures to be aware of the action outcomes before executing. Formally, at each decision step $i$, given the set of candidate actions along with their thoughts $\mathcal{A}_{c}=\{(t_{i}^{(1)},a_{i}^{(1)}),(t_{i}^{(2)},a_{i}^{(2)}),\ldots,(t_{i}^{(m)},a_{i}^{(m)})\}$ proposed by policy model $p$ in [Equation 2](https://arxiv.org/html/2510.11892v1#S2.E2), the world model performs $k$-step lookahead rollouts to estimate the potential outcomes of each action candidate $j\in\{1,2,\ldots,m\}$:

$$ $\displaystyle o_{i+1}^{(j)}$ $\displaystyle\sim\pi_{w}(\cdot|g,o_{i},t_{i}^{(j)},a_{i}^{(j)})$ (3) $\displaystyle(t_{i+1}^{(j)},a_{i+1}^{(j)})$ $\displaystyle\sim\pi_{w}(\cdot|g,o_{i+1},t_{i}^{(j)},a_{i}^{(j)})$ $\displaystyle\vdots$ $\displaystyle o_{i+k}^{(j)}$ $\displaystyle\sim\pi_{w}(\cdot|g,o_{i+k-1}^{(j)},t_{i+k-1}^{(j)},a_{i+k-1}^{(j)})$ $$

For each $k$-step rollout trajectory $\hat{\tau}_{i}^{(j)}=(o_{i}^{(j)},t_{i}^{(j)},a_{i}^{(j)},o_{i+1}^{(j)},t_{i+1}^{(j)},a_{i+1}^{(j)},\ldots,o_{i+k}^{(j)})$, the corresponding rewards are estimated using a model-based (Li et al., 2023; Mahan et al., 2024) or program-based (Lambert et al., 2024; Guo et al., 2025) reward function:

$$ $r(a^{j})=R(\hat{\tau}_{i}^{(j)},g)$ (4) $$

The optimal action is then selected from $\mathcal{A}_{c}$ based on the highest estimated reward.

$$ $a_{i}^{*}=\arg\max_{(t_{i},a_{i})\in\mathcal{A}_{c}}r(a_{i})$ (5) $$

## 3 Preliminary Analysis

We focus on two fundamental capabilities of world models that are critical for computer-use tasks: future state prediction, which supports anticipating environment dynamics, and reward estimation, which underpins evaluating the outcomes of actions (Hafner et al., 2019; 2020; 2023). Recent work such as WMA (Chae et al., 2024) explores these aspects mainly through next-state identification and immediate reward estimation. However, such analyses do not fully account for the importance of reasoning across extended horizons. To address this, we design probing tasks tailored to these two capabilities by considering longer planning horizon. Specifically, for future state prediction, we design the task of next-state identification and full-procedure planning alignment, which together capture both short and long horizon dynamics; For reward estimation, we design the task of milestone transition recognition, which assesses models’ ability to anticipate the outcomes of intermediate transitions. We apply these probes to three state-of-the-art LLMs, Qwen-2.5-VL-72B (Bai et al., 2025), Claude-3.5-Sonnet(^1^11[https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet)), and Claude-3.7-Sonnet(^2^22[https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet)) by sampling trajectories on two challenging browser/computer-use benchmarks: WebArena (Zhou et al., 2023) and OSWorld (Xie et al., 2024). In the following, we introduce these tasks and present the probing analysis, while more details with illustrative examples are provided in Appendix [A.1](https://arxiv.org/html/2510.11892v1#A1.SS1).

### 3.1 Next-State Identification

To assess the most basic requirement of future state prediction, we follow WMA (Chae et al., 2024) to design this task where models are asked to predict the correct subsequent observation given a current state and action. Given current observation $o_{i}$ and action $a_{i}$, the model predicts the correct subsequent observation from two candidates:

$$ $\hat{o}_{i+1}=\arg\max_{o\in\{o_{i+1}^{\text{true}},o_{i+1}^{\text{false}}\}}P(o|o_{i},t_{i},a_{i})$ (6) $$

Setup: Given the a $n$-step trajectory, we extract intermediate steps from successful and failed trajectories where $i\in[2,n-2]$ to avoid trivial predictions from initial or terminal states. For each $(o_{i},a_{i},o_{i+1})$ triplet, we create a negative sample by selecting the most lexically similar observation from the same trajectory. The lexical analysis is conducted using difflib(^3^33[https://docs.python.org/3/library/difflib.html](https://docs.python.org/3/library/difflib.html)), a Python’s built-in library. This requires LLMs to distinguish the true next observation $o_{i+1}^{\text{true}}$ from a distractor $o_{i+1}^{\text{false}}$.

Results: As shown in  [Table 1](https://arxiv.org/html/2510.11892v1#S3.T1), models achieve relatively strong accuracy overall, i.e., exceeding 75%, indicating they can capture short-term state changes under various lexical similarity levels.

### 3.2 Full-Procedure Planning Alignment

While next-state identification evaluates whether an LLM can capture immediate state transitions, effective world models must also reason over longer horizons. To probe this ability, we design a plan alignment task, where models are asked to generate execution plans and these plans are evaluated for consistency with realistic environment dynamics. Formally, given a task goal $g$ and an initial observation $o_{1}$, the model produces an execution plan $\hat{P}=(a_{1},a_{2},\ldots,a_{T})$. The LLM judge then evaluates whether the execution plan conforms to the standard procedure defined in [Equation 7](https://arxiv.org/html/2510.11892v1#S3.E7).

$$ $B=\Phi\!\left(\langle g,o_{1}\rangle,\hat{P},P^{*}\right)=\begin{cases}\text{True},&\text{if }\hat{P}\text{ aligns with }P^{*},\[3.0pt] \text{False},&\text{otherwise.}\end{cases}$ (7) $$

where $P^{*}$ denotes the reference procedure derived from environment tutorials. The judgement is based on element attributes (e.g., location, text description, visibility) and operation logic (e.g., feasibility, ordering) with respect to $P^{*}$.

Setup: We sample tasks from WebArena and OSWorld benchmarks. For each task, we manually annotate a reference document chunk that is directly relevant to accomplishing the task under the corresponding environment (e.g., a website or software). More annotation details are in Appendix [A.2](https://arxiv.org/html/2510.11892v1#A1.SS2). Models are then prompted to generate execution plans without access to tutorials, and the generated plans are evaluated by an LLM judge (Claude-3.7-Sonnet by default) for alignment against the reference procedures. Details of the evaluation prompt are provided in Appendix [A.1](https://arxiv.org/html/2510.11892v1#A1.SS1).

Results:  [Table 1](https://arxiv.org/html/2510.11892v1#S3.T1) shows that alignment remains moderate across all models, rarely exceeding 65%. This reveals a clear limitation: while LLMs can list plausible actions, they often fail to maintain procedural coherence or respect environment-specific constraints.

Table: Table 1: Probing results across three tasks: next-state identification, full-procedure planning alignment, and milestone transition recognition. All values are percentages.

### 3.3 Milestone Transition Recognition

Aside from probing LLM’s capability of capturing future states, we also probe whether models can recognize task-relevant progress, an essential skill for reward estimation in world models. The task evaluates whether models can distinguish promising transition sequences from unproductive ones:

$$ $\hat{S}=\arg\max_{S\in\{S^{\text{true}},\,S^{\text{false}}\}}P(\text{success}\mid S,g)$ (8) $$

where $S=\{o_{i},o_{i+h},o_{i+2h},\dots,o_{i+(l-1)h}\}$
denotes a subsequence of length $l$ sampled at interval $h$ from the full trajectory.

Setup: We sample sequences of $l=3$ consecutive transitions with interval $h=2$ from both successful and failed trajectories, where the intervals are used to avoid repeated states. Same as next state identification, we also sample steps from steps within $[2,n-2]$ to avoid trivial predictions. For each objective $g$, we annotate pairs where $S^{\text{true}}$ represents a more promising subsequence drawn from a successful trajectory, and $S^{\text{false}}$ represents a less effective subsequence from a failed trajectory. More task details can be found in Appendix [A.1](https://arxiv.org/html/2510.11892v1#A1.SS1).

Results:  [Table 1](https://arxiv.org/html/2510.11892v1#S3.T1) shows that all models perform strongly. Claude-3.7-Sonnet achieves the highest accuracy (86.7%), followed by Claude-3.5-Sonnet (85.7%) and Qwen-2.5-VL-72B (83.7%). The consistently high performance across models suggests that LLMs possess reasonable ability to evaluate which transitions are conducive to task progress.

### 3.4 Discussion

Overall, our probing analysis reveals that modern LLMs demonstrate relatively good short-term predictive and local evaluative capabilities: they can reliably identify next states and recognize task-relevant transitions. However, these strengths do not extend to long-horizon planning, where performance deteriorates sharply in aligning its knowledge to specific environments. This suggests that LLMs might inherently lack robust generalization for world modeling across dynamic environments, thus may require external guidance to sustain accurate simulations over extended horizons.

## 4 R-WoM Framework

From the probing analysis in Section [3](https://arxiv.org/html/2510.11892v1#S3), we identify grounding as a key mechanism for improving the alignment of LLMs to specific environments, which motivates the design of our R-WoM framework.

### 4.1 Overview

Figure: Figure 2: Overview of the R-WoM pipeline. At each time step $i$, the policy model generates $m$ candidate actions. For each candidate, the world model grounded by retrieved tutorials performs $k$-step rollouts to simulate a possible future trajectory. The rewards of rollout trajectories are finally estimated by world models to select the best action.
Refer to caption: x2.png

As illustrated in  [Figure 2](https://arxiv.org/html/2510.11892v1#S4.F2), the R-WoM framework employs the retrieval-augmented way to ground world modeling during simulation. Given the task objective and current observation, relevant documentation and tutorials are retrieved and reranked to form the grounding evidence set. This evidence is used to condition the world model during both state transition prediction and reward estimation process. Algorithm [1](https://arxiv.org/html/2510.11892v1#alg1) summarizes the complete R-WoM pipeline, which iteratively applies this process until task completion or termination.

Figure: Algorithm 1 The Pipeline of R-WoM

### 4.2 Design Details

RAG design. We adopt a reasoning-based retrieval design to enhance relevance of retrieved document chunks to the given query. Given the task goal $g$, we construct a query $q=f_{\mathrm{enc}}(g)$ and retrieve top-$k$ tutorial chunks $\mathcal{C}_{k}$ based on cosine similarity. An LLM-based reranker (i.e., policy model $p$ here) then conducts a list-wise reranking of candidates based on contextual relevance:

$$ $\mathcal{E}=f_{p}^{\text{rank}}(\mathcal{C},q)$ (9) $$

yielding the final evidence set $\mathcal{E}$. The world model conditions on $\mathcal{E}$ for grounded future state prediction and reward estimation.

R-WoM design. At step $i$, with tutorial evidence $\mathcal{E}$, for each candidate thought and action pair $(t_{i}^{(j)},a_{i}^{(j)})\in\mathcal{A}_{c}$, the rollout produces a predicted trajectory of $k$ steps. Unlike the iterative rollout explored in previous works (Gu et al., 2024; Fang et al., 2025), which requires multiple rounds of LLM calls and thus suffers from efficiency limitations, we adopt a reasoning-based LongCoT rollout inspired by Deepseek-R1 (Guo et al., 2025), enabling the world model to unfold the entire multi-step imagination trajectory within a single forward reasoning sequence.

$$ $\hat{\tau}_{i}^{(j)}=\pi_{w}^{\mathrm{LongCoT}}(o_{i},t_{i}^{(j)},a_{i}^{(j)};\mathcal{E})$ (10) $$

We observe that absolute sparse reward used in previous works (Chae et al., 2024; Gu et al., 2024; Fang et al., 2025) might not effectively distinguish more meaningful rollouts. Therefore, inspired by recent advances in relative reward design (Liu et al., 2024; Choi et al., 2024; Guo et al., 2025), we employ a list-wise ranking mechanism to evaluate simulated trajectories in a relative way.

$$ $(t_{i}^{*},a_{i}^{*})\;=\;\arg\max_{(t_{i}^{(j)},a_{i}^{(j)})\in\mathcal{A}_{c}}\Big[f_{w}\!\big(R(\hat{\tau}_{i}^{(j)},g,\mathcal{E})\big)\Big]$ (11) $$

As is shown in [Equation 11](https://arxiv.org/html/2510.11892v1#S4.E11), each rollout trajectory is scored relatively in the comparative context of all candidates. In this way, we aim to reduce potential bias from absolute reward signals and stablize the selection of most promising action candidate.

Table: Table 2: End-to-end performance on OSWorld and WebArena across three runs. Best in bold; second-best underlined. R-WoM cells include relative improvement over the second-best.

## 5 Experiment

To evaluate the effectiveness of R-WoM, we propose the following research questions:

- •
RQ1: Does R-WoM improve the performance of computer-use agents compared to established baselines in realistic environments such as browsers and operating systems?
- •
RQ2: How do external tutorials contribute to grounding world models, and to what extent do agents benefit from incorporating this information from tutorials?
- •
RQ3: Can tutorial-grounded world models support longer imagination horizons more effectively than ungrounded counterparts over multi-step rollouts?

### 5.1 Setup

We evaluate R-WoM against three baselines:

- •
Vanilla: The vanilla approach is adapted from the official implementations: the screenshot-only version for OSWorld provided by GTA-1 (Yang et al., 2025), and the screenshot+accessibilty tree version for WebArena provided by WMA (Chae et al., 2024). This approach relies solely on the task objective, current observation (represented as screenshots and accessibilty trees) and prior interaction history.
- •
RAG: A retrieval-augmented generation pipeline that retrieves relevant documentation and augments the LLM before action prediction, which is built upon the vanilla approach.
- •
WoM: A variant of the WebDreamer (Gu et al., 2024) where we implement world model-based dreaming and reward assignment in a LongCoT way as the original iterative way of generating rollouts takes much more LLM calls, which will be discussed with more details in Appendix [A.4](https://arxiv.org/html/2510.11892v1#A1.SS4).

We conduct experiments on two comprehensive benchmarks designed for multi-round interactions in realistic computer-use environments: WebArena (Zhou et al., 2023), which spans web-based tasks across domains such as e-commerce, social forums, and collaborative platforms; and OSWorld (Xie et al., 2024), which covers diverse desktop tasks including file management, terminal commands, and productivity applications. Specifically, we sample a subset from these two benchmarks for our experiments where tutorials are available for retrieval purpose and we collect tutorials from both online websites and offline documents. The details of the subsets and tutorial collection can be found in Appendix [A.2](https://arxiv.org/html/2510.11892v1#A1.SS2).
We test three popular LLM backbones: Qwen-2.5-VL-72B (Instruct version) (Bai et al., 2025), Claude-3.5-Sonnet, and Claude-3.7-Sonnet, serving as both the policy and world model. For methods requiring retrieval, we build the RAG pipeline with Langchain(^4^44[https://github.com/langchain-ai/langchain](https://github.com/langchain-ai/langchain)), FAISS (Douze et al., 2024) as the vector store, and Qwen-3-Embedding-8B (Zhang et al., 2025b) as the embedding model. More implementation details can be found in Appendix [A.3](https://arxiv.org/html/2510.11892v1#A1.SS3).

### 5.2 RQ1: End-to-End Performance

[Table 2](https://arxiv.org/html/2510.11892v1#S4.T2) reports the overall end-to-end performance. It shows that R-WoM consistently outperforms all alternatives, with improvements of +23.4% on OSWorld and +18.1% on WebArena for Qwen-2.5, +12.5% and +9.6% for Claude-3.5, and +25.3% and +7.2% for Claude-3.7 over the strongest non-R-WoM baselines. These results reveal that the improvements remain stable across different backbones, highlighting that R-WoM provides more consistent benefits compared with retrieval alone or ungrounded world modeling.

Figure: (a) OSWorld
Refer to caption: x3.png

Figure: (a) OSWorld
Refer to caption: x5.png

### 5.3 RQ2: The Role of Tutorials in Grounding World Models

To further assess the role of tutorials, we compare three settings: no grounding (WoM), grounding with R-WoM using retrieved tutorials, and grounding with R-WoM using oracle tutorials. Similarly as the full-procedure alignment task in Section [3.2](https://arxiv.org/html/2510.11892v1#S3.SS2), we also manually annotate document chunks that are relevant to the task from human’s perspective. More annotation details and the performance of retrieval can be found in Appendix [A.2](https://arxiv.org/html/2510.11892v1#A1.SS2) and [A.4](https://arxiv.org/html/2510.11892v1#A1.SS4), respectively. As shown in  [Figure 3](https://arxiv.org/html/2510.11892v1#S5.F3), performance consistently improves with the grounding levels, from no grounding to grounding with retrieved tutorials, then to grounding with oracle tutorials. It indicates that access to external procedural knowledge helps models in world modeling. These findings underscore that R-WoM’s effectiveness is tightly coupled with tutorial fidelity, and the advances in retrieval and resource curation represent critical levers for future progress.

### 5.4 RQ3: Ablation studies of imagination horizon

To examine the effect of imagination horizon on end-to-end performance, we vary the horizon from 1 to 4 for both ungrounded (WoM) and grounded (R-WoM) world models, as shown in  [Figure 4](https://arxiv.org/html/2510.11892v1#S5.F4). WoM, the world model without grounding during rollouts, shows modest initial gains but quickly plateaus and even declines beyond 2 steps, reflecting its susceptibility to compounding prediction errors. In contrast, R-WoM maintains consistently higher success across horizons on both OSWorld and WebArena, with improvements lasting up to horizon three before tapering off. These results suggest that tutorial-guided grounding helps stabilize rollouts over longer horizon simulations.

## 6 Related Works

### 6.1 Computer-use Agent

One line of works focuses on exploring how to improve agent’s understanding of computer-use actions, such as building end-to-end agent frameworks (Agashe et al., 2024; 2025; Song et al., 2025), and training native agent models (Qin et al., 2025; Wang et al., 2025; Lai et al., 2025) or specific action grounding models (Wu et al., 2024; Xie et al., 2025; Yang et al., 2025).
Another line of works explores treating LLMs as world models to simulate the computer-use environments. WoM (Gu et al., 2024) pioneers this direction by using LLMs to simulate the outcome of candidate actions, and evaluate these imagined states with discrete reward given by LLM judge (Gu et al., 2024). Subsequent works such as WMA (Chae et al., 2024) adapt this idea to improve planning by abstracting state transitions into natural language summaries. WKM (Qiao et al., 2024) and WebEvolver (Fang et al., 2025) develop co-evolving world models and policies to progressively refine both simulation and planning, moving beyond one-horizon imagination.

### 6.2 Tutorial-use

Parallel developments leverage tutorials or indirect knowledge to train digital agents. Synatra (Ou et al., 2024) converts human-oriented tutorials into 100k synthetic demonstrations to fine-tune a 7B CodeLLaMA model.
Other frameworks generate trajectories guided by tutorial completion or replay (e.g., AgentTrek (Xu et al., 2024), TongUI (Zhang et al., 2025a)) to teach GUI navigation and tool use from multimodal resources. Learn-by-interact (Su et al., 2025) synthesizes trajectories by leveraging tutorials and interaction with the environments. These approaches focus on offline trajectory generation by referring to tutorials while our approach focuses on tutorial-guided grounding of LLMs as world models at inference time.

## 7 Conclusion and Future Work

We presented a systematic study of LLM-based world models for computer-use tasks, revealing that while they can model state transitions and recognize task-relevant progress, they fail to reliably adapt to unfamiliar environments in long-term planning without grounding. To address this, we proposed the Retrieval-augmented World Model (R-WoM), which incorporates environment-specific tutorial knowledge during the imagination rollouts and reward prediction procedures to reduce hallucinations and stale knowledge. Evaluations on WebArena and OSWorld show that R-WoM consistently outperforms competitive baselines, demonstrating the efficacy of retrieval-augmented grounding for LLM agents in dynamic browser-use and computer-use scenarios. While R-WoM shows promises in improving LLM as world models, some bottlenecks still remain. First, the grounding stage requires availability of online tutorials for the target environment, which has limits in tutorial-scarce domains, or when documentation is outdated. Synthesizing tutorials from tutorial-scarce environments is one of the future directions we aim to explore.
Second, despite the efficiency optimizations in R-WoM’s rollout simulation and reward estimation, the computational cost is still non-trivial. Conducting world modeling in an agentic way to further reduce costs can be our future work.

## References

- Agashe et al. (2024)
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang.
Agent s: An open agentic framework that uses computers like a human.
*arXiv preprint arXiv:2410.08164*, 2024.
- Agashe et al. (2025)
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang.
Agent s2: A compositional generalist-specialist framework for computer use agents.
*arXiv preprint arXiv:2504.00906*, 2025.
- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2.5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Chae et al. (2024)
Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo.
Web agents with world models: Learning and leveraging environment dynamics in web navigation.
*arXiv preprint arXiv:2410.13232*, 2024.
- Choi et al. (2024)
Heewoong Choi, Sangwon Jung, Hongjoon Ahn, and Taesup Moon.
Listwise reward estimation for offline preference-based reinforcement learning.
*arXiv preprint arXiv:2408.04190*, 2024.
- Douze et al. (2024)
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.
The faiss library.
*arXiv preprint arXiv:2401.08281*, 2024.
- Fang et al. (2025)
Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu.
Webevolver: Enhancing web agent self-improvement with coevolving world model.
*arXiv preprint arXiv:2504.21024*, 2025.
- Gu et al. (2024)
Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al.
Is your llm secretly a world model of the internet? model-based planning for web agents.
*arXiv preprint arXiv:2411.06559*, 2024.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hafner et al. (2019)
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.
Dream to control: Learning behaviors by latent imagination.
*arXiv preprint arXiv:1912.01603*, 2019.
- Hafner et al. (2020)
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba.
Mastering atari with discrete world models.
*arXiv preprint arXiv:2010.02193*, 2020.
- Hafner et al. (2023)
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.
Mastering diverse domains through world models.
*arXiv preprint arXiv:2301.04104*, 2023.
- Hao et al. (2023)
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu.
Reasoning with language model is planning with world model.
*arXiv preprint arXiv:2305.14992*, 2023.
- Lai et al. (2025)
Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang.
Computerrl: Scaling end-to-end online reinforcement learning for computer use agents.
*arXiv preprint arXiv:2508.14040*, 2025.
- Lambert et al. (2024)
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al.
Tulu 3: Pushing frontiers in open language model post-training.
*arXiv preprint arXiv:2411.15124*, 2024.
- Li et al. (2023)
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu.
Generative judge for evaluating alignment.
*arXiv preprint arXiv:2310.05470*, 2023.
- Liu et al. (2024)
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al.
Lipo: Listwise preference optimization through learning-to-rank.
*arXiv preprint arXiv:2402.01878*, 2024.
- Mahan et al. (2024)
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak.
Generative reward models.
*arXiv preprint arXiv:2410.12832*, 2024.
- Ou et al. (2024)
Tianyue Ou, Frank F Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou.
Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale.
*Advances in Neural Information Processing Systems*, 37:91618–91652, 2024.
- Qiao et al. (2024)
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen.
Agent planning with world knowledge model.
*Advances in Neural Information Processing Systems*, 37:114843–114871, 2024.
- Qin et al. (2025)
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al.
Ui-tars: Pioneering automated gui interaction with native agents.
*arXiv preprint arXiv:2501.12326*, 2025.
- Schrittwieser et al. (2020)
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.
Mastering atari, go, chess and shogi by planning with a learned model.
*Nature*, 588(7839):604–609, 2020.
- Song et al. (2025)
Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, et al.
Coact-1: Computer-using agents with coding as actions.
*arXiv preprint arXiv:2508.03923*, 2025.
- Su et al. (2025)
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö Arık.
Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments.
*arXiv preprint arXiv:2501.10893*, 2025.
- Wang et al. (2025)
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al.
Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning.
*arXiv preprint arXiv:2509.02544*, 2025.
- Wang et al. (2024)
Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen.
Can language models serve as text-based world simulators?
*arXiv preprint arXiv:2406.06485*, 2024.
- Wu et al. (2024)
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al.
Os-atlas: A foundation action model for generalist gui agents.
*arXiv preprint arXiv:2410.23218*, 2024.
- Xie et al. (2024)
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al.
Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.
*Advances in Neural Information Processing Systems*, 37:52040–52094, 2024.
- Xie et al. (2025)
Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al.
Scaling computer-use grounding via user interface decomposition and synthesis.
*arXiv preprint arXiv:2505.13227*, 2025.
- Xu et al. (2024)
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu.
Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials.
*arXiv preprint arXiv:2412.09605*, 2024.
- Yang et al. (2025)
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, et al.
Gta1: Gui test-time scaling agent.
*arXiv preprint arXiv:2507.05791*, 2025.
- Zhang et al. (2024)
Alex Zhang, Khanh Nguyen, Jens Tuyls, Albert Lin, and Karthik Narasimhan.
Language-guided world models: A model-based approach to ai control.
*arXiv preprint arXiv:2402.01695*, 2024.
- Zhang et al. (2025a)
Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, and Qing Li.
Tongui: Building generalized gui agents by learning from multimodal web tutorials.
*arXiv preprint arXiv:2504.12679*, 2025a.
- Zhang et al. (2025b)
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al.
Qwen3 embedding: Advancing text embedding and reranking through foundation models.
*arXiv preprint arXiv:2506.05176*, 2025b.
- Zhou et al. (2023)
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al.
Webarena: A realistic web environment for building autonomous agents.
*arXiv preprint arXiv:2307.13854*, 2023.

## Appendix A Appendix

Roadmap: Section [A.1](https://arxiv.org/html/2510.11892v1#A1.SS1) introduces the design details of our probing task. Section [A.2](https://arxiv.org/html/2510.11892v1#A1.SS2) introduces the tutorial collection, annotation and retrieval approach for our experiments.
Section [A.3](https://arxiv.org/html/2510.11892v1#A1.SS3) presents the implementation details of R-WoM, including action space definition and prompt design.
Section [A.4](https://arxiv.org/html/2510.11892v1#A1.SS4) presents additional experimental results.

### A.1 Details of probing task

Task 1: Next-state identification. To assess whether the world model can predict the immediate outcome of an action given the current state, the model is asked to discriminate between the true next observation and a lexically similar distractor, as illustrated in [Figure 5](https://arxiv.org/html/2510.11892v1#A1.F5). In this way, we aim to probe LLM’s sensitivity to environment changes. We construct 100 samples drawn from trajectories in WebArena for this task.

Task 2: Full-procedure planning alignment.
Moving beyond identifying next state, we would like to probe whether LLM can reason about longer steps of future states. As shown in [subsection A.1](https://arxiv.org/html/2510.11892v1#A1.SS1), given a task objective, the model is asked to generate a multi-step plan, which is then validated against tutorials describing environment dynamics. The evaluation measures whether the model’s procedure aligns with realistic element locations, operation sequences, and interaction methods. To assess this capability, we construct 40 samples from trajectories in both OSWorld and WebArena.

Figure: Figure 5: Illustration of the next-state identification probing task. Given a current state and an action, the model must choose between two candidate next states: (A) the ground-truth state, and (B) a lexically similar distractor. This task evaluates whether the world model can correctly predict the true next observation rather than being misled by textual similarity.
Refer to caption: x7.png

Figure: Figure 6: Illustration of the full-procedure planning alignment probing task. Given a task objective (top), the model generates a multi-step plan (left), which is then compared against environment-specific tutorials (right). The evaluation checks whether the generated procedure aligns with the tutorials in terms of navigation logic, element selection, and operation feasibility. This task assesses the world model’s ability to sustain long-horizon procedural reasoning in realistic environments.
Refer to caption: x8.png

Task 3: Milestone transition recognition.
To probe reward estimation capability of LLMs, we design this task to assess whether LLMs have the capability to capture meaning state transitions. As shown in [Figure 7](https://arxiv.org/html/2510.11892v1#A1.F7), the LLM is presented with pairs of trajectory segments that diverge at a decision point, one representing a promising milestone transition and the other an unproductive path. The LLM needs to identify which trajectory is more conducive to task success. This setting is evaluated on 98 samples drawn from both successful and failed trajectories in WebArena.

Figure: Figure 7: Illustration of the milestone transition recognition probing task. Given a sequence of transitions, the model must identify whether they reflect meaningful progress toward the goal. In this example, the top path shows an unproductive transition where the agent gets stuck trying to directly select “books” as a forum type, failing to proceed. The bottom path shows a more promising milestone transition: the agent first enters the books forum and then successfully fills out the submission form. The task evaluates whether the world model can distinguish between effective and ineffective procedural progress.
Refer to caption: x9.png

### A.2 Tutorial Processing

Our framework relies on tutorials as external grounding for browser- and computer-use tasks. To construct a comprehensive knowledge base, we gather tutorials from both general-purpose and environment-specific resources. For cross-domain instructional guidance, we include WikiHow, which provides structured, step-by-step content spanning a broad range of tasks. For environment-specific domains, we incorporate official documentation from the corresponding software or websites. The complete list of tutorial sources is as follows:

- •
WikiHow: [https://www.wikihow.com/Main-Page](https://www.wikihow.com/Main-Page)
- •
Google Chrome Help: [https://support.google.com/chrome](https://support.google.com/chrome)
- •
GIMP 3.0 User Manual: [https://docs.gimp.org/3.0/en/](https://docs.gimp.org/3.0/en/)
- •
Visual Studio Code Documentation: [https://code.visualstudio.com/docs](https://code.visualstudio.com/docs)
- •
Ubuntu Help: [https://help.ubuntu.com/22.04/ubuntu-help/](https://help.ubuntu.com/22.04/ubuntu-help/)
- •
Mozilla Thunderbird Support: [https://support.mozilla.org/en-US/products/thunderbird/learn-basics-get-started](https://support.mozilla.org/en-US/products/thunderbird/learn-basics-get-started)
- •
VLC Media Player User Guide: [https://docs.videolan.me/vlc-user/desktop/3.0/en/](https://docs.videolan.me/vlc-user/desktop/3.0/en/)
- •
LibreOffice Help: [https://help.libreoffice.org/latest/en-US/](https://help.libreoffice.org/latest/en-US/)
- •
GitLab Documentation: [https://docs.gitlab.com/](https://docs.gitlab.com/)
- •
Adobe Commerce Admin User Guides: [https://experienceleague.adobe.com/en/docs/commerce-admin/user-guides/home](https://experienceleague.adobe.com/en/docs/commerce-admin/user-guides/home)

From these sources, we construct a knowledge base of over 30k chunked tutorial documents that collectively support tasks across diverse software and website environments. Since our framework requires tutorial availability to provide concrete grounding, we sample task subsets from OSWorld and WebArena that can be partially mapped to tutorial examples. Specifically, we select 85 tasks from OSWorld, covering domains such as Chrome, GIMP, VSCode, VLC, Thunderbird, and Ubuntu OS, and 113 tasks from WebArena, covering CMS and GitLab domains and we annotate one or two document chunks that are most relevant to each task from human’s perspective.

To retrieve useful tutorials at inference time, we adopt a reasoning-based retrieval strategy. This involves query rewriting to anonymize and generalize task queries, followed by LLM-based reranking to reduce false negatives that may arise when relying solely on cosine similarity. The detailed prompts used for query rewriting and reranking are provided below, and the results comparing retrieval strategies are reported in Appendix [A.4](https://arxiv.org/html/2510.11892v1#A1.SS4).

### A.3 Implementation details of R-WoM

To enable automation in browser and computer-use environments, we adopt the official action space definitions provided by WebArena(^5^55[https://github.com/web-arena-x/webarena](https://github.com/web-arena-x/webarena)) and OSWorld(^6^66[https://github.com/xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld)), as summarized in [Table 3](https://arxiv.org/html/2510.11892v1#A1.T3). In practice, we find that direct action coordinate mapping in OSWorld poses challenges for models such as the Qwen series and Claude-3.5-Sonnet. To address this and enable the policy model to generate more effective actions during world model rollouts, we employ GTA-1-7B (Yang et al., 2025) as an auxiliary action grounding model to assist in action generation when evaluating on OSWorld. For retrieval-related approach (i.e., RAG and R-WoM), we use top-5 retrieved document chunks by default to put them into the LLM’s context.

Table: Table 3: Action space for WebArena and OSWorld.

### A.4 Additional Experimental Results

Breakdown of end-to-end performance.  [Table 4](https://arxiv.org/html/2510.11892v1#A1.T4) provides a domain-level view of performance. R-WoM consistently achieves the best results across most of the domains, but the relative magnitude of improvement varies. In domains such as chrome and gimp, where tasks involve longer dependencies and compounding errors, R-WoM exhibits the largest margins over WoM. By contrast, in lighter workloads such as vlc or thunderbird, the absolute gains are smaller and sometimes comparable to RAG, suggesting that grounding might bring limited additional benefit when task horizons are short. These results imply that grounding is most critical in environments requiring extended planning.

Table: Table 4: Domain-level performance. Best in bold; second-best underlined.

Ablation studies of retrieval performance. [Figure 8](https://arxiv.org/html/2510.11892v1#A1.F8) shows that retrieval performance improves most when query rewriting and reranking are combined, indicating their complementary effects. Query rewriting is more beneficial in diverse environments like WebArena, while reranking offers steadier gains across settings by filtering irrelevant matches. The overall trend suggests that single strategies yield uneven improvements depending on domain structure, but their integration consistently delivers more robust retrieval.

Figure: (a) OSWorld
Refer to caption: x10.png

Cost comparison.
In this section, we compare the computational cost of different methods in terms of the number of LLM calls and total inference time. As discussed in Section [4.2](https://arxiv.org/html/2510.11892v1#S4.SS2), iterative rollout introduces significant inefficiency due to multiple rounds of policy–world model interactions. To quantify this effect, we reimplement the iterative rollout used by WebDreamer and measure its cost alongside other methods. Although R-WoM remains more expensive than single-pass approaches such as Greedy or RAG, it offers a more favorable trade-off between efficiency and stability, enabling longer and more consistent rollouts without the extreme overhead of full iterative reasoning. Looking forward, we would explore incorporating agentic calling of world models to further eliminate redundant calls and improve cost efficiency.

Table: Table 5: Cost statistics of running different methods across benchmarks.

### A.5 Use of Large Language Models

We utilized Large Language Models (LLMs), such as Claude, exclusively for ancillary support in two main areas: (i) language editing and polishing of the manuscript, and (ii) coding assistance for minor boilerplate tasks, such as generating plotting scripts and small utilities. All model-generated outputs were thoroughly reviewed, modified, and rigorously tested by the authors to ensure their accuracy and appropriateness.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: **Q1:** Could you provide more context on how your results compare with other available benchmarks in this space? Specifically, I noticed OpenAI CUA reported 58.1% on WebArena in early 2025. Are there methodological or setup differences that make direct comparison inappropriate, or would it be valuable to discuss these results to help readers understand the current state of the field?

**Q2:** Regarding the characterization of WebEvolver using "absolute reward estimation" (line 311): could you clarify this? From my understanding, WebEvolver uses supervised fine-tuning with rejection sampling. I would appreciate clarification on how you're categorizing their reward mechanism.

**Q3:** Your probing study identifies long-horizon planning alignment as the key limitation (50-65% accuracy). After applying R-WoM with tutorial grounding, what is the new alignment score on this metric? This would help demonstrate that your solution directly addresses the diagnosed problem.

**Q4:** Could you provide more analysis of retrieval failure cases? What types of tasks or scenarios are most affected when the retrieval system fails to find appropriate tutorials, and how do these failures impact the final task success?

**Q5:** Have you considered any techniques to reduce the 4-10× computational overhead? Are there scenarios where the performance gains justify this cost, and others where a lighter-weight approach might be preferable?
soundness: 2
strengths: **1. Well-Motivated Diagnostic Study:** I appreciate the systematic probing analysis in Section 3. The three-task evaluation (next-state identification, full-procedure planning alignment, milestone transition recognition) provides clear empirical evidence for why existing LLM-based world models struggle with long-horizon planning. This diagnostic approach effectively motivates the need for external grounding mechanisms.

**2. Reasonable Architectural Integration:** The paper successfully brings together two established paradigms—world models from model-based RL and retrieval-augmented generation—and applies them to the challenging computer-use domain. I find this combination sensible and well-suited to the problem.

**3. Comprehensive Ablation Studies:** I commend the authors for including ablations on imagination horizons (Table 3) and grounding sources (retrieved vs. oracle tutorials in Figure 3). These studies help understand which components contribute to the overall performance.

**4. Practical Efficiency Considerations:** The use of CoT-based rollouts instead of expensive iterative interactions represents a pragmatic design choice. Table 5 shows R-WoM is more efficient than WebDreamer, which is valuable for practical deployment.

**5. Solid Engineering Effort:** The collection of 30k+ tutorial chunks from diverse sources and the design of the reasoning-based RAG pipeline with query rewriting and reranking demonstrates significant implementation effort.
summary: This paper addresses an important challenge in developing computer-use agents: enabling LLMs to serve as effective world models for long-horizon planning in digital environments. The authors begin with a thoughtful diagnostic study using three probing tasks that reveal LLMs excel at short-term state predictions (86% accuracy) but struggle with long-horizon planning alignment (50-65%). To address this limitation, they propose R-WoM (Retrieval-augmented World Model), which grounds LLM-based simulations with external tutorials retrieved at inference time. The framework employs chain-of-thought rollouts and listwise reward estimation for action selection. Experiments on WebArena and OSWorld demonstrate improvements of up to 25.3% and 18.1% respectively over the WebDreamer baseline.
weaknesses: **1. Limited Baseline Comparisons:** I am concerned that the experimental evaluation could be more comprehensive. The paper primarily compares against WebDreamer and basic policy models, but I understand there are other benchmarks in this space that could provide important context. For example, I noticed that OpenAI's Computer Use Agent (CUA) published results showing 58.1% on WebArena (available since January 2025), which would be a valuable reference point. I recognize the authors' results of 35.11% show improvement over their chosen baseline, but including broader comparisons would help readers better understand where R-WoM stands in the current landscape. I encourage the authors to discuss or compare with additional publicly available benchmarks to strengthen the evaluation.

**2. Characterization of Related Methods:** I noticed that the paper characterizes some related work in ways that may not fully capture their methodologies. For instance, lines 310-316 discuss WebEvolver (Fang et al., 2025) in the context of "absolute reward estimation," but from my reading, WebEvolver appears to use supervised fine-tuning with rejection sampling rather than RL-based absolute rewards. I suggest the authors revisit these characterizations to ensure they accurately represent the cited methods, which would strengthen the positioning of the listwise reward contribution.

**3. Retrieval Quality Analysis:** While Figure 3 shows that oracle tutorials outperform retrieved ones, I would appreciate more analysis of how retrieval failures affect performance. The appendix mentions Recall@5 metrics (94.4% OSWorld, 85.7% WebArena), suggesting some retrieval errors occur. I recommend adding analysis of what types of tasks or scenarios are most affected by retrieval failures and how these errors propagate through the world model rollouts.

**4. Computational Cost Discussion:** Table 5 shows R-WoM requires 3.8-12h runtime compared to 0.7-2.1h for vanilla methods—a 4-10× increase. While this is better than WebDreamer's 15-43h, I believe the practical deployment implications deserve more discussion in the main paper. For real-world applications, this trade-off between performance and computational cost would be important to consider.

**5. Gap Between Diagnosis and Solution:** The probing study identifies 50-65% full-procedure planning alignment as the core problem. However, I would be interested to see whether R-WoM actually improves this alignment metric, or if the task success improvements come from other mechanisms. This would help validate that the solution addresses the diagnosed problem.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	Task subset selection: please quantify how many tasks were excluded due to absent tutorials and report the distribution of retrieval coverage (retrieval recall per task). This will help readers assess generality.  ￼
2.	Reranker details: Is it zero-shot or fine-tuned? How sensitive are results to the reranker model choice?  
3.	Listwise reward computation: provide the exact mathematical definition (or pseudocode) of  in Eq. (11) $f_w(\{R(\hat \tau^{(j)},g,\mathcal{E})\})$ . How are rewards normalized across rollouts, and how does candidate set size m affect ranking quality?  
4.	Judge bias: how consistent are the plan alignment judgments across different judge models or human annotators? Can you release a small human-annotated subset for reproducibility? 
5.	Failure modes: can the authors provide qualitative examples where retrieval misleads the model (noisy / tangential tutorials) and explain any heuristic filters applied?
soundness: 3
strengths: 1.	Clear problem framing and probing suite. The three tasks isolate immediate state prediction vs. long-horizon planning vs. reward-related transition recognition, which yields diagnostic insights about LLM strengths/weaknesses.  
2.	Effective and practical method. R-WoM is straightforward to implement (retrieval + rerank + CoT rollouts + listwise ranking) and integrates well with existing LLM stacks.  
3.	Strong empirical gains. Consistent and sometimes large improvements across models and benchmarks (Table 2). 
4.	Insightful ablations and visualization. Figures 3 provide clear comparisons of ungrounded, retrieval-based, and oracle-knowledge models. Figure 4 detail how imagination horizon impacts agent success, directly supporting claims about longer-horizon stability and providing actionable insights for the field.
summary: This paper investigates the capabilities and limitations of Large Language Models (LLMs) when used as "world models" for computer-use agents. The authors first conduct a systematic probing analysis across three tasks (next-state identification, full-procedure planning alignment, milestone transition recognition) to evaluate LLMs' ability to predict future states and estimate rewards. Their findings indicate that while LLMs excel at short-term predictions, their performance degrades significantly over longer horizons, failing to align with specific environment dynamics. To address this limitation, the paper introduces the R-WoM, a framework that grounds the LLM's simulation process in external, up-to-date knowledge retrieved from tutorials. R-WoM employs a reasoning-based retrieval pipeline and a listwise reward estimation strategy to improve the accuracy of multi-step rollouts. The authors demonstrate through experiments on the OSWorld and WebArena benchmarks that R-WoM substantially outperforms baseline methods, including vanilla agents, standard RAG, and an ungrounded world model (WebDreamer), particularly in tasks requiring long-horizon planning.
weaknesses: 1.	Limited novelty in core formulation. The high-level idea of retrieval-augmented LLMs for world modeling is not entirely new; prior works such as WorldGPT also consider augmentation or world modeling for digital environments. The contribution here is more in the detailed design (e.g., reasoning-based retrieval + listwise scoring) and empirical analysis than a fundamentally new methodological direction.
2.	Potential biases in evaluation. The use of LLM-based judges (Claude 3.5 Sonnet and Claude 3.7 Sonnet) to score plan alignment and trajectory success (especially in full-procedure planning) risks evaluation circularity (model judge vs. model being evaluated) and inherits bias from the same family of models. While necessary at present, some acknowledgment (and potentially a human evaluation subset) would bolster the claims.
3.	Selection bias toward tutorial-rich tasks. The method requires relevant tutorials; the authors explicitly sample task subsets that have tutorials available. The generalization to tutorial-scarce settings is unclear. Include experiments on a held-out set with sparse/no tutorials, or provide a quantitative analysis of how retrieval coverage correlates with gains.
4.	Limited Analysis of Failure Modes: While the paper demonstrates strong overall performance, a more comprehensive analysis of R-WoM's failure modes would strengthen the work. For instance, a discussion of scenarios where the model degrades (e.g., when faced with poor, irrelevant, or incorrect tutorials) or remains brittle (e.g., with ambiguous goals or novel UI elements) would provide significant value for practitioners seeking to build upon these findings.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. Could you comment on the expected performance of R-WoM in environments with limited or no available tutorial documentation? How gracefully does the model's performance degrade as the quality of the knowledge base decreases?

2. The listwise reward estimation is an interesting and intuitive design choice. Have you performed an ablation study to specifically isolate its contribution? For instance, how does the full R-WoM framework compare to a version that uses a more traditional absolute reward scoring (e.g., a 1-10 scale) on the generated rollouts?

3. Table 5 shows a clear trade-off between cost and performance. For example, using Claude-3.7 on WebArena, R-WoM is ~4.75x slower than RAG for a ~7.2% relative improvement. In what practical scenarios do you believe this trade-off is justified?
soundness: 2
strengths: Clear Motivation: The paper is motivated by a systematic probing analysis that pinpoints the specific failure modes of ungrounded LLM-based world models in long-horizon tasks, providing a clear foundation for the research.

Reasonable Framework Design: R-WoM is a reasonably designed framework that combines ideas like a RAG pipeline, CoT-based simulation, and listwise reward estimation.

Evaluation on Multiple Benchmarks: The evaluation demonstrates performance gains over baselines across two challenging benchmarks and three different LLMs.
summary: This paper investigates the limitations of using Large Language Models as world models for computer-use agents, particularly their unreliability in long-horizon planning due to hallucinations and static knowledge. To diagnose this, the authors first conduct a systematic probing analysis, revealing that while LLMs excel at short-term state prediction and recognizing meaningful progress, they fail to generate plans that align with real environment dynamics over extended steps. To address this, the paper proposes the Retrieval-augmented World Model, a framework that grounds the LLM's simulation process in external knowledge retrieved from tutorials. Key components of R-WoM include a reasoning-based RAG pipeline for better retrieval, a CoT-based mechanism for efficient multi-step rollouts, and a listwise reward estimation strategy for more robust action selection. Experiments on two challenging benchmarks, OSWorld and WebArena, show that R-WoM significantly outperforms strong baselines across multiple LLM backbones.
weaknesses: Limited Novelty: The core idea of the paper is to apply Retrieval-Augmented Generation (RAG) to world models. While this is a logical engineering direction, it feels more like a direct combination of existing, mature technologies (RAG, CoT, World Models) and lacks fundamental theoretical or methodological innovation. This makes the paper's contribution feel more applied and engineering-focused rather than a breakthrough in basic research.

Severe Generalization Limits: The method's effectiveness is severely constrained to domains that have high-quality, comprehensive tutorials. This is a critical flaw, as it means the method is nearly inapplicable in many real-world scenarios where documentation is sparse, outdated, or non-existent. The paper fails to adequately discuss or experimentally verify this limitation, which significantly undermines its claims of general utility.

Unfavorable Cost-Benefit Ratio: Although R-WoM is more efficient than the WebDreamer baseline, its computational overhead is still excessive compared to a standard RAG approach. The results show that achieving a limited performance gain (sometimes in the single digits) requires a several-fold increase in computational cost. This cost-benefit trade-off is unacceptable for many practical applications and casts doubt on the method's utility.

Effectiveness of Core Component is Questionable: The results with an "oracle" retriever expose the performance bottleneck of the current retrieval module. This is not just a technical detail; it directly challenges the paper's central thesis that world models can be effectively grounded via retrieval. If the retrieval itself is unreliable, the foundation of the entire framework is shaky.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - How well does R-WoM scale when retrieval memory grows to millions of trajectories? Do you employ approximate retrieval or pruning strategies?
- Have you tested R-WoM on completely unseen applications (e.g., apps with novel layouts or interaction paradigms)? How robust is the method in such settings?
- Can you provide examples of what the retrieval module retrieves in successful vs. failed cases? Does retrieval sometimes introduce misleading prior experiences?
- How accurate is the predictive component of the world model across multiple horizons? Do errors accumulate in long trajectories?
- In a real-world assistant scenario, how would you balance retrieval latency, model complexity, and user responsiveness?
- Could this framework generalize to other domains requiring long-horizon reasoning (e.g., robotics, scientific workflows), or is it specialized to computer-use tasks?
soundness: 2
strengths: - This paper addresses an increasingly important challenge: enabling LLM-powered agents to use computers effectively.
- Novel integration of retrieval augmentation with world models in the specific context of computer-use agents.
- This paper demonstrates that retrieval + world models can substantially improve generalization and robustness. Experimental evaluation shows consistent improvements over baselines across two comprehensive benchmarks designed for multi-round interactions in realistic computer-use environments.
- This paper is well-written and logically structured, in spite of more experimental analysis are advised. 
- Potentially impactful for building personal assistant agents capable of robust computer-use in real-world settings.
summary: This paper proposes R-WoM (Retrieval-augmented World Model), a framework designed to enhance computer-use agents. Traditional world models for agents often struggle with long-horizon reasoning. R-WoM addresses the issue by integrating retrieval-augmented memory into a world model. 
The method is evaluated on benchmarks of computer-use agents, showing that R-WoM significantly outperforms baseline world models and state-of-the-art methods in terms of task success rate, with particular advantage in longer-horizon simulations.

Overall, this is a strong and well-motivated paper. It contributes a novel and effective combination of retrieval and world models, demonstrating strong empirical results on computer-use agents. Its main areas for improvement are in experiment soundness, experimental analysis breadth & deepness, scalability, and retrieval interpretability. With these addressed, the work could be highly impactful for the development of practical, general-purpose computer-use assistants.
weaknesses: - Two benchmarks are not sound enough. Experiments are limited to a specific set of tasks. It is unclear how R-WoM would perform on diverse real-world applications (e.g., enterprise software, web navigation with dynamic layouts).
- The evaluation lacks stress tests for extreme long-horizon/complex tasks (hundreds of steps).
- This paper does not deeply analyze the computational overhead of retrieval at inference time. Scaling to large memory banks or diverse user contexts could introduce latency.
- While retrieval helps generalization, it is not always clear what is retrieved and why. More analysis of retrieval quality and relevance would strengthen the contribution.
- The world model’s predictive accuracy is not reported in detail (e.g., prediction error across horizons). Understanding when it fails would provide useful insights.
- While strong baselines are included, it would be useful to see how R-WoM compares with end-to-end planning agents trained with reinforcement learning at scale.


Decision: Accept (Poster)