Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

## Abstract

Abstract Large language models (LLMs) have recently gained much attention in building autonomous agents. However, performance of current LLM-based web agents in long-horizon tasks is far from optimal, often yielding errors such as repeatedly buying a non-refundable flight ticket. By contrast, humans can avoid such an irreversible mistake, as we have an awareness of the potential outcomes ( e.g. , losing money) of our actions, also known as the “ world model ”.
Motivated by this, our study first starts with preliminary analyses, confirming the absence of world models in current LLMs ( e.g. , GPT-4o, Claude-3.5-Sonnet, etc.). Then, we present a World-model-augmented (WMA) web agent, which simulates the outcomes of its actions for better decision-making.
To overcome the challenges in training LLMs as world models predicting next observations, such as repeated elements across observations and long HTML inputs,
we propose a transition-focused observation abstraction, where the prediction objectives are free-form natural language descriptions exclusively highlighting important state differences between time steps. Experiments on WebArena and Mind2Web show that our world models improve agents’ policy selection without training and demonstrate superior cost- and time-efficiency compared to recent tree-search-based agents. The codes are publicly available at https://github.com/kyle8581/WMA-Agents .

## 1 Introduction

Large language models (LLMs) have been widely applied to solve tasks in diverse domains, including web navigation, where LLMs generate action sequences (e.g., click) to accomplish user goals on websites (Shi et al., ; Kim et al., ).
Despite some success (Yao et al., ), LLM-based web agents’ performance remains significantly poor in long-horizon environments such as WebArena (Zhou et al., ), where GPT-4 yields a task success rate of 14.41% whereas humans have a success rate of 78.24%.
This raises a question: Why do LLMs, despite their advancements, perform much worse than humans in web navigation?

Humans avoid unwanted situations by considering the possible outcomes of our actions beforehand (Edwards, ).
Such awareness of actions and outcomes is referred to as the “world model” (Forrester, ).
Meanwhile, existing LLM-based web agents rely heavily on trial and error to make decisions, as they lack world models to help them foresee the outcome of an action without actually performing it (LeCun, ), leading to sub-optimal decision-making that is irreversible (e.g., repeatedly buying a non-refundable item).
Acknowledging the importance of world models, studies in robotics and reinforcement learning (RL) have proposed to incorporate world models for agents in navigation tasks. For instance, Du et al. () and Yang et al. () apply world models to simulate visual outcomes/observations of input texts or robot control.
The Dreamer series use world models to predict latent state of images and use them to optimize policies, reducing the need for actual interactions in game environments (Hafner et al., ; ; ).

Motivated by these, this paper begins by investigating SOTA LLMs’ understanding of “environment dynamics”, i.e., the association between actions and environment states. We reveal that (i) current LLMs (e.g., GPT-4o and Claude-3.5-Sonnet) struggle with predicting the outcomes of their actions and (ii) the awareness of potential outcomes helps them make decisions aligning with user goals.
Upon these findings, we present a World-Model-Augmented (WMA) web agent, which simulates the outcomes of its actions for better decision-making.
However, naively training a world model to predict the next observation state (i.e., the entire webpage) can lead to a large amount of repeated
elements across observations and long HTML inputs, negatively affecting model performance.
Thus, we propose a novel transition-focused observation abstraction, where the world model is trained to generate free-form natural language descriptions exclusively highlighting important state differences between time steps (e.g., an updated price on the website).
During inference, our agent first simulates the outcome (i.e., next observation) of each action candidate (from the policy model) using the world model. Then, a value function estimates the rewards of all simulated observations, helping the agent select a final action with the highest estimated reward.
Our contributions are two-fold:

- •
We are the first to pioneer world models in LLM-based web agents, laying the groundwork for policy adaptation through simulated environment feedback in web navigation.
- •
We present a novel transition-focused observation abstraction for training LLMs as world models. We show that using world models trained with this method can improve action selection by simulating the action candidates without training the policy models. Also, we demonstrate our agents’ cost- and time-efficiency compared to recent tree-search-based agents (Koh et al., ), by 6.8x and 5.3x, respectively.

## 2 Related Work

##### Benchmarks for web agents.

Many benchmarks have been introduced to evaluate LLM-based agents’ ability in web navigation (Kim et al., ). MiniWoB (Shi et al., ) and MiniWoB++ (Liu et al., ) are among the first widely adopted benchmarks. More recently, WebShop (Yao et al., ) simulates e-commerce environments where agents are tested to execute tasks based on text instructions on the web. These early benchmarks are limited to specific and constrained domains. Mind2Web (Deng et al., ) curates web tasks across more diverse domains, and WebArena (Zhou et al., ) emphasizes functional correctness and more realistic scenarios (e.g., posting articles on Reddit) in simulated environment. We adopt Mind2Web and WebArena for evaluation for their generalizability and resemblance of real-world web interactions.

##### LLM-based web agents.

In recent years, LLM-based agents have become popular in the web navigation domain.
However, since many powerful proprietary LLMs do not provide access to model parameters, many studies of web navigation have been focusing on training-free methods where LLMs directly learn from user inputs (i.e., prompts) without task-specific training (Sodhi et al., ; Zheng et al., ).
For instance, Wilbur (Lutz et al., ) and Agent Workflow Memory (Wang et al., ) leverage a verification model  (Pan et al., ) with prompt-based methods to collect successful trajectory data for guiding the agent’s policy at inference time.
AutoEval (Pan et al., ) and Tree search agent (Koh et al., ) increase the number of trials and reasoning paths, further improving system performance.
However, due to their trial-and-error nature, these approaches can not only be computationally inefficient in gathering trajectories as tasks become more complex but also are more prone to undesired results (e.g., booking a non-refundable ticket).
Our WMA web agent reduces such risks via a world model, which predicts future observations and the rewards of their corresponding action candidates before actually making an action. Furthermore, our approach can be orthogonally applied to many of the existing methods.

##### World models in autonomous agents.

World models refer to systems that generate internal representations of the world, predicting the effects of their actions on environments (LeCun, ).
In RL, simulating observations and environmental feedback using world models allow the policy model to learn (Sutton, ) or plan (Ha & Schmidhuber, ; Hafner et al., ) without actually interacting with the environment.
While some world models are trained with raw observations (Oh et al., ; Chiappa et al., ), others are built on latent representations (Hafner et al., ; ; Kipf et al., ). For instance, in the image domain, Hafner et al. () train a world model by training it to first compute a posterior stochastic state based on the current image and then a prior stochastic state that tries to predict the posterior without access to the image.
Within the field of LLMs,
Zhang et al. () convert visual observations into natural language and employs an LLM-based world model for text-based games, and  Wang et al. () further transform observations into a structural format (e.g., JSON), improving LLMs’ reasoning over state transition functions.
In web navigation, environments are built upon not only natural language but on more complex text modalities such as HTML and DOM trees. We address this by transforming them to a novel free-form description, highlighting the state difference between each time step.

## 3 Preliminary Analyses: Are Current LLMs Aware of Environment Dynamics in Web Navigation?

We first start with investigating whether LLMs can understand the association between actions and their effects on the environment, i.e., understand the environment dynamics. We conduct analyses addressing these two questions:

- •
Preliminary question I: Are LLMs aware of the outcomes of their actions?
- •
Preliminary question II: When having access to the outcome of each action candidate, can LLMs select an optimal action aligning with the user objective?

For the analyses, we sample 100 user instructions from WebArena and annotate human trajectories within the environment.
Each instance has a user instruction, the current state, a human-annotated golden action, and the corresponding next state resulting from the golden action. We analyze 4 popular closed-source SOTA LLMs: GPT-4o-mini (Zhu et al., ), GPT-4o, GPT-4-Turbo (OpenAI, ), and Claude-3.5-Sonnet (Anthropic, ). More details are in Appendix B.

### 3.1 Preliminary Analysis I - LLMs Struggle with Predicting the Next States Caused by Their Actions

Figure: Figure 1: LLMs’ performance in next state prediction.
Refer to caption: x1.png

##### Setups.

We test LLMs’ ability to predict the outcomes of actions on the web via a binary classification task.
Given the current state and the golden action, the LLM is prompted to select the correct next state from (i) the golden next state and (ii) a lexically similar yet incorrect next state retrieved from the same trajectory. We calculate the lexical similarity with difflib (Python, ).
We assess classification accuracy.

##### Results.

Figure 1 reveals that under vanilla settings, current LLMs cannot effectively predict the next states caused by their actions. First, all adopted LLMs (54.75% on average) lose significantly to humans. Also, Claude-3.5-Sonnet performs almost as badly as random guessing. These suggest that the world model, the ability to foresee the potential outcomes of actions taken, is absent in LLMs.

### 3.2 Preliminary analysis II - LLMs Make Better Action Selection When Accessing the Outcome of Each Action Candidate

Figure: Figure 2: LLMs’ performance in action selection (w/ and w/o next states).
Refer to caption: x2.png

##### Setups.

We assess whether LLMs can select a correct action that aligns with the user goal when they are provided with the outcome of each action candidate.
Given the current state, 10 action candidates, and their corresponding outcomes/next states, the LLM is prompted to differentiate the golden action from other 9 negative actions.

##### Results.

Figure 2 compares LLMs’ performance in differentiating the golden action from negative actions when they are/are not provided with the resulting next state of each candidate action.
We find that current SOTA LLMs have difficulty in selecting correct actions when they can only rely on the current observations/states (striped bars), yielding an average accuracy of only 49%. However, when augmented with the corresponding next state of each action candidate, they demonstrate huge performance gains (up to 38% improvement) in selecting correct actions.
When only the current state and the user objective are provided, GPT-4o yields an accuracy of 53%. In contrast, when the next state is given, performance rises to 73%.

### 3.3 Insights from Preliminary Analyses

Through our preliminary analyses, we have demonstrated that: (i) Web agents built with SOTA LLMs are bad at predicting how their actions affect next states;
(ii) When being aware of how an action affects the next state, LLMs can make better decisions.
These findings highlight the necessity of world models in LLM-based web agents, pointing out a promising direction for facilitating better web agents in complex, long-horizon navigation tasks.

## 4 World-Model-Augmented Web Agents

Figure: Figure 3: Framework overview. We first collect training data for world models (top). After training, we perform policy optimization by selecting the action leading to an optimal next state (bottom).
Refer to caption: x3.png

Motivated by the above insights, we present a novel framework for World-Model-Augmented (WMA) web agents, LLM-based web agents equipped with world models.
The world models learn/leverage environment dynamics (i.e., association of actions and outcomes) to simulate plausible next observations of agents’ actions, facilitating better decisions (i.e., polices) in web navigation.

##### Formulations.

Since web agents access only information in the viewport (i.e., users’ visible area), we model web navigation as a partially observable Markov decision process (POMDP). We consider a web environment $\mathcal{E}$ with: (i) a hidden state space $\mathcal{S}$; (ii) an action space $\mathcal{A}$, including language-guided actions (e.g., CLICK, TYPE, HOVER, etc.) and their descriptions; (iii) an observation space $\mathcal{O}$ representing an accessibility tree of the page, which is a simplified DOM tree (Zhou et al., ).

In the POMDP, the agent receives a new partial observation $o_{t+1}\in\mathcal{O}$ from $\mathcal{E}$ after performing an action $a_{t}\in\mathcal{A}$ based on $o_{t}$. Such state transition $s_{t}\rightarrow s_{t+1}$
is managed by a golden transition function $\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ provided in the environment.

### 4.1 World Model Training

We hereby introduce the training process of our world models. As shown in Figure 3 (top), our training consists of three main steps:

#### 4.1.1 Step I: Harvesting Agent-environment Interaction Data

We start by collecting the dataset $\mathcal{D}=\sum^{n}_{t=1}\{I,o_{t},a_{t},o_{t+1}\}$ from the environment $\mathcal{E}$ for training world models.
For that, we prompt an LLM as web agent to achieve the goal provided in the user instruction $I$, by iteratively predicting an action $a_{t}$ based on the current observation $o_{t}$ throughout all $n$ time steps. Consequently, we obtain $\mathcal{D}$ from trajectory $\tau=\{o_{1},a_{1},o_{2},...,a_{n},o_{n+1}\}$ based on $I$, and environment states of $n$ time steps $\{s_{1},...,s_{n+1}\}\subset\mathcal{S}$ obtained via transition function $\mathcal{T}$.

#### 4.1.2 Step II: Transition-focused Observation Abstraction

With the collected data $\mathcal{D}=\sum^{n}_{t=1}\{I,o_{t},a_{t},o_{t+1}\}$, it is intuitive to train LLM-based world models to predict $o_{t+1}$, which is expressed with texts (e.g., HTML and accessibility tree) (Deng et al., ; Zhou et al., ). However, simply using textual observations to represent environment states and use them as training objectives may introduce the following downsides:

- •
Low information gain during training: State transitions in websitesoften involve altering only a part of the previous observation (e.g., a drop-down menu is clicked). As a result, most information in $o_{t+1}$ remains the same as it is in $o_{t}$.
Therefore, predicting the entire textual observation from scratch may result in low information gain during training.
- •
Excessively long sequence length: Processing the whole text-based observations can lead to excessively long sequence length and consequently high computational costs.
Indeed, this can be partially mitigated by replacing raw HTML with an accessibility tree (relatively simple), using it as LLMs’ training objectives still introduce a long sequence length (4K tokens on average, see Figure 4).

Figure: Figure 4: Sequence length distribution of different observation representations.
Refer to caption: x4.png

To address the above bottleneck in training text-based models (i.e., LLMs) as world models, we draw inspiration from how the RL community conventionally implements world models: using estimated latent vectors as summaries of raw visual observations, reducing memory footprints for effectively learning environment dynamics (Doerr et al., ; Hafner et al., ) – We thus propose to abstract raw text observations, with a focus on state transition between consecutive observations, for obtaining better training objectives.

To collect abstracted next observations for training world models, one may simply run an off-the-shelf summarizer on $o_{t+1}$ collected in Step I. However, while reducing sequence length, this does not address the low information gain caused by repeated elements between $o_{t}$ and $o_{t+1}$.
Thus, instead of such a naive approach, as shown in Figure 5, we first (i) apply the Hungarian algorithm (Kuhn, ) to calculate a cost matrix for matching elements between $o_{t}$ and $o_{t+1}$ and (ii) mechanically transform the results into a list of state transition $\Delta(o_{t},o_{t+1})$, pointing out UPDATED, DELETED, and ADDED elements on the web. After that, we prompt an LLM to convert the extracted $\Delta(o_{t},o_{t+1})$ into a free-from natural language description $\tilde{o}_{t+1}$, which highlights the difference between the new observation $o_{t+1}$ and $o_{t}$.
Replacing $o_{t+1}$ in $\mathcal{D}=\{I,o_{t},a_{t},o_{t+1}\}$ collected in Step I with $\tilde{o}_{t+1}$ we just acquired here, we get a final dataset $\tilde{\mathcal{D}}=\sum^{n}_{t=1}\{I,o_{t},a_{t},\tilde{o}_{t+1}\}$ for training world models.

#### 4.1.3 Step III: Learning Environment Dynamics

Lastly, using $\tilde{\mathcal{D}}$, we proceed to train the internal world model $\phi$ of the web agent to learn the environment dynamics. Formally, an LLM working as the world model is trained to predict the abstracted observation $\tilde{o}$ of the next state $s_{t+1}$, given three inputs: the user instruction $I$, the current observation $o_{t}$, and the current action $a_{t}$.
This LLM is trained to minimize the following loss term via the next-token prediction objective:

$$ $\mathcal{L}_{\phi}=-\log\sum_{(\tilde{o},o,a,I)\in\tilde{\mathcal{D}}}{p( \tilde{o}_{t+1}|o_{t},a_{t},I)}$ (1) $$

Through this process, this LLM learns the environment dynamics, working as a world model that helps the web agent to foresee the potential outcome (i.e., predict the next observation) of its action.

Figure: Figure 5: The overview of transition-focused observation abstraction.
Refer to caption: x5.png

### 4.2 Inference-time Policy Optimization with the World Model

In this section, we explain how we use the developed world model $\phi$ to improve LLM-based agents’ performance in web navigation.
As illustrated in Figure 3 (bottom), the web agent consists of three main components: (i) a policy model $\theta$; (ii) our world model $\phi$; (iii) a value function $V$. Note that the policy model $\theta$ is frozen, i.e., we do not update its parameters.

During inference at time $t$ with a current observation $o_{t}$, WMA web agent ought to utilize the world model $\phi$ to foresee how an action can affect the state (i.e., predict $\tilde{o}_{t+1}^{i}$), and accordingly find an optimal action/policy $a_{t}$ from the policy model $\theta$ that can lead to the target goal defined in $\mathcal{I}$.

We begin by sampling $k$ action candidates $\{a_{t}^{1},a_{t}^{2},...,a_{t}^{k}\}$ from $\theta$ via top-$p$ decoding (Holtzman et al., ), explore diverse next states $s_{t+1}\in\mathcal{S}$ (Wang et al., ).
Next, we use the world model $\phi$ to “simulate” the potential next observation $\tilde{o}_{t+1}^{i}$ caused by each action candidate $a_{t}$:

$$ $\{\tilde{o}_{t+1}^{i}\}_{i=1}^{k}=\{\phi(o_{t},a_{t}^{i},I)\}_{i=1}^{k}$ (2) $$

Note that each $\tilde{o}_{t+1}^{i}$ is a free-form description of the next observation, as shown in Figure 5 (right).

Lastly, we decide the agent’s action for actual operation by selecting the action leading to the most optimal future state $s_{t+1}$ from all action candidates.
Following Koh et al. (), we use an off-the-shelf LLM as a value function $V(\cdot)$ to estimate the reward yielded by each action candidate and select the action $\hat{a}_{t}$ with the highest reward:

$$ $\hat{a}_{t}=\operatornamewithlimits{argmax}_{a_{t}\in\{a_{t}^{1},...,a_{t}^{k} \}}V(I,o_{t},a_{t},\tilde{o}_{t+1}^{i})$ (3) $$

With this process, we are able to optimize the policy selection of web agents in inference time without training the policy models. This training-free augmentation of world models allows us to easily adapt our world model $\phi$ to existing web agents, including prompt-based (Pan et al., ; Wang et al., ) and fine-tuned LLMs (Gur et al., ; Lai et al., ).

## 5 Experiments

### 5.1 Setups and Implementation Details

##### Benchmarks and evaluation metrics.

For evaluation, we use the official WebArena and Mind2Web benchmarks (Zhou et al., ; Deng et al., ). WebArena includes 812 real-life tasks in simulated environments across five different websites, spanning four key domains - e-commerce (Shopping), social forums (Reddit), collaborative software development (Gitlab), content management (CMS), and Map. Details of each domain are further explained in Appendix C.3.
The main metric, Success Rate (SR), is calculated as the percentage of the user instructions that are successfully accomplished by the generated agent trajectory.
On the other hand, Mind2Web (Deng et al., ) covers over 2,000 open-ended tasks, collected from 137 websites of 31 domains and crowd-sourced action sequences for the tasks.
Along with the SR, Mind2Web also uses Step SR, which measures whether the predicted action selects both the correct action type (action F1) and element ID (element accuracy). When the agent succeeds in all steps in a trajectory, it is evaluated as success.

**Table 1: Agent performance in WebArena. $\Delta$: relative performance gains from policy optimization.**
| Policy LLMs | Methods | Max Actions | Success Rate (SR) | $\Delta$ |  |
| --- | --- | --- | --- | --- | --- |
| No Policy Opt. | + Policy Opt. |  |  |  |  |
| GPT-4 | AutoEval (Pan et al., ) | 30 | 20.2% | - | - |
| BrowserGym (Drouin et al., ) | 23.5% | - | - |  |  |
| SteP (Sodhi et al., ) | 35.8% | - | - |  |  |
| AWM (Wang et al., ) | 35.5% | - | - |  |  |
| GPT-4o | Vanilla CoT (Zhou et al., ) | 30 | 13.1% | - | - |
| Tree search agent (Koh et al., ) | 5 | 15.0% | 19.2% | +28.0% |  |
| WMA web agent (ours) | 5 | 12.8% | 16.6% | +29.7% |  |
| GPT-4o-mini | WMA web agent (ours) | 5 | 9.4% | 13.5% | +43.6% |

**Table 2: Domain-specific performance of agents using GPT-4o-mini as policy models**
| Methods / Domains | Shopping | CMS | Reddit | Gitlab | Map | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Vanilla CoT (max actions = 5) | 18.8% | 8.2% | 5.3% | 3.1% | 11.6% | 9.4% |
| WMA web agent (ours) | 19.3% | 11.5% | 7.9% | 8.7% | 22.3% | 13.5% |
| $\Delta$ | +3% | +40% | +49% | +181% | +92% | +44% |

##### Training data for world models.

(i) For evaluation in WebArena: To facilitate applications in the real world, the training data for world models needs to cover a wide range of tasks/goals. Since a diverse and large-scale user instructions set is not available,(^1^11In WebArena, only test data (i.e., instructions) is provided.) we synthesize user instructions using an LLM.
With these synthesized instructions of various goals, we are able to collect rich trajectories as training data, improving world models’ generalization to diverse real-world situations.
In practice, we sample $l$ trajectories for each $I$.(^2^22We empirically set $l=5$ in our work. Further details on the whole data collection process are in Appendix.)
We generate 870 synthetic user instructions and gather 14K instances from WebArena using GPT-4o-mini as the policy model.
To avoid redundant learning, we filter out repeated state-action pairs.
(ii) For evaluation in Mind2Web: we adopt the offline trajectory data from Mind2Web, following the setting of Wang et al. ().

##### Baselines.

For baselines, we adopt: (1) a prompting-based LLM (Zhou et al., ) powered by chain-of-thought prompting (Wei et al., );
(2) AutoEval (Pan et al., ). It refines agents’ trajectories based on the feedback on the final state of the trajectory (i.e., succeed or fail) from a VLM evaluator (Shinn et al., ); (3) BrowserGym (Drouin et al., ) trains web agents with multi-modal observations, including HTML contents and the screenshot image of the browser;
(4) SteP (Sodhi et al., ), a framework based on human-authored hierarchical policies injected to the agent;
(5) HTML-T5 (Gur et al., ), the previous SOTA method on Mind2Web, uses LLMs pre-trained LLMs on HTML corpus.
(6) Agent workflow memory (AWM;Wang et al. ()) leverages self-discovered workflow memory to guides its policy;
(7) Tree search agent (Koh et al., ), the most competitive baseline that explores multiple trajectories and selects an optimal path via a tree search algorithm. The main difference between ours and Tree search agents is that ours only uses the predicted future states via simulation and does not actually explore diverse states.

##### World model.

We use Llama-3.1-8B-Instruct (Dubey et al., ) as the backbone LLM for our world models.(^3^33[https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct))
For WebArena, we construct our dataset in online setting using the provided web environment. In Mind2Web, we use the offline trajectory data (i.e., the train set) following Wang et al. ().
For prompt-based world models (baselines) in our experiments, we use 2-shot demonstrations to instruct LLMs to predict the next state. More details are provided in Appendix C.1.

##### Policy model.

Following Koh et al. (), we adopt GPT-4o (gpt-4o-0513) as the agent backbone for evaluation in WebArena. Additionally, we test with GPT-4o-mini (gpt-4o-mini-0718) to test our framework in relatively more resource-restricted scenarios.

##### Value function.

We fine-tune Llama-3.1-8B-Instruct to predict rewards using data from Mind2Web, where rewards (as training objective) are calculated based on the progress toward the goal, i.e., $t/(len(\tau))$ when $a_{t}$ is taken.

**Table 3: Success rate on Mind2Web tests using GPT-3.5-Turbo as policy models. EA = element accuracy; EF = element filtering; AF1 = action F1; * = results from the original paper.**
| Methods | Cross-Task | Cross-Website | Cross-Domain |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EA | AF1 | Step SR | SR | EA | AF1 | Step SR | SR | EA | AF1 | Step SR | SR |  |
| Synapse* | 34.4% | - | 30.6% | 2.0% | 28.8% | - | 23.4% | 1.1% | 29.4% | - | 25.9% | 1.6% |
| HTML-T5-XL* | 60.6% | 81.7% | 57.8% | 10.3% | 47.6% | 71.9% | 42.9% | 5.6% | 50.2% | 74.9% | 48.3% | 5.1% |
| MindAct* | 41.6% | 60.6% | 36.2% | 2.0% | 35.8% | 51.1% | 30.1% | 2.0% | 21.6% | 52.8% | 18.6% | 1.0% |
| AWM (w/ EF)* | 50.6% | 57.3% | 45.1% | 4.8% | 41.4% | 46.2% | 33.7% | 2.3% | 36.4% | 41.6% | 32.6% | 0.7% |
| AWM (w/o EF) | 78.3% | 74.1% | 62.8% | 15.3% | 74.7% | 70.1% | 58.6% | 6.2% | 74.8% | 71.2% | 60.7% | 9.5% |
| AWM+WMA (ours) | 79.9% | 75.8% | 67.0% | 25.4% | 75.7% | 72.1% | 61.3% | 8.5% | 75.9% | 72.6% | 63.4% | 10.1% |

### 5.2 Main Results

##### Agent performance in WebArena.

In Table 1 (middle), we first compare our WMA web agent (16.6%) with vanilla CoT (13.1%) and observe significant improvements over almost all domains in WebArena as detailed in Table 2. Interestingly, when using GPT-4o-mini as the policy model, our agent achieve 181% and 92% performance gains over CoT in Gitlab and Map, respectively.
The relatively small improvement in Shopping might be due to the large-scale state space $\mathcal{S}$ in the domain, such as the diversity of searched item lists from different user queries, which makes it harder for the world model to properly learn environment dynamics.
Regardless, the overall improvement suggests the effectiveness of leveraging learnt environment dynamics during inference time.

Next, we compare our approach with Tree search agent (Koh et al., ), which uses the oracle next state observation (i.e., resulted by the gold transition function
$\mathcal{T}$ from the environment) for policy selection instead of estimated observation via the world model.
While the absolute SR of our WMA agent (16.6%) is slightly below Tree search agent (19.2%) when using GPT-4o as policy models, our policy optimization with the world model brings a larger performance gain to vanilla CoT than tree search (+29.7% vs. +28.0%).
Also, in the later section (section 5.3), we present ours’ superior cost and time efficiency over Tree search agent.

##### Agent performance in Mind2Web.

We compare WMA web agent with MindAct (Deng et al., ) and AWM (Wang et al., ), which are previous and current SOTAs on Mind2Web.(^4^44Tree search agent is not applicable to this benchmark as the environment is not available.)
Table 3, demonstrates that WMA web agent significantly outperforms AWM,(^5^55Surprisingly, we find element filtering (EF) of MindAct, applied to AWM in default, largely hindering its performance. Thus, in Table 3, we include the results without EF. A detailed discussion is in Appendix C.6) achieving new SOTA performance.
Furthermore, the results indicate that
WMA web agent trained on Mind2Web data has a strong generalization capability.
This makes our approach much more valuable in scenarios where collecting data for new web environments is non-trivial.

##### Our advantages besides performance gains.

Based on the performance reported, we can conclude that our strategy of building world models (i.e., observation abstraction) is effective not only for accessibility tree format (WebArena) but also for HTML format (Mind2Web), underscoring the applicability of our approach across different representations of web data.
Another advantage of our approach over others is that the developed world models can be incorporated into existing or future web agents without any additional training of policy models, enabling easy implementation.

### 5.3 Analyses of Time and Cost Efficiency

We compare our WMA web agent with Tree search agent in terms of time and API cost efficiency. Results are shown in Table 4. To run one user instruction, Tree search agent spends about 748.3 seconds on average, as it involves the exploration of diverse future states while actually interacting with the environment. When it conducts backtracing to revert to the previous state, the whole sequence of previous actions has to be executed again.
By contrast, WMA web agent only takes 140.3 seconds per instance by simulating the possible action candidates rather than actually executing them, which is 5.3 times faster than Tree search agent.
Tree search agent requires 6.8 times more API cost due to its multi-modal inputs.
To sum up, while showing comparable performance to Tree search agent in CMS, Reddit, Gitlab, and Map, our WMA web agent demonstrates superior cost and time efficiency.

**Table 4: Head-to-head comparison of Tree search agent (results are from Koh et al. ()) and ours regarding (i) SR and (ii) API cost, and (iii) inference time. We use GPT-4o for policy models.**
| Methods | Shopping | CMS | Reddit | Gitlab | Map | API cost | Inference time (sec) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Tree search agent | 28.1% | 16.5% | 10.5% | 13.3% | 25.8% | $2.7 | 748.3 |
| WMA (ours) | 20.8% | 14.3% | 10.5% | 13.3% | 26.8% | $0.4 | 140.3 |

### 5.4 Ablation Studies

**Table 5: Results of the ablation study in WebArena.**
| Settings | World Model | Success Rate (SR) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Use | Training | Shopping | Gitlab | Map | Overall |  |
| w/o next states in reward estimation (section 4.2) | ✗ | ✗ | 28.0% | 6.0% | 19.0% | 18.0% |
| w/o training world models (section 4.1) | ✓ | ✗ | 30.0% | 10.0% | 15.0% | 17.5% |
| w/o abstracting observations (section 4.1.2) | ✓ | ✓ | 22.0% | 6.0% | 15.0% | 14.5% |
| WMA (ours) | ✓ | ✓ | 32.0% | 14.0% | 21.0% | 22.0% |

We conduct several ablation studies on our WMA web agent with 200 randomly sampled instances from WebArena (Shopping: 50; Gitlab: 50; Map: 100). We use GPT-4o-mini as policy models.

##### Accessing simulated next states in reward estimation improves agent performance.

To assess the impact of incorporating the simulated next state when calculating the value score, we compare our reward estimation strategy to a Q-value function (Haarnoja et al., ) that predicts the reward based on only $(o_{t},a_{t})$.
The results in Table 5 show that the information of the resulting next state helps the value function to predict rewards more accurately, resulting a better task performance.

**Table 6: Performance with different value models.**
| Value Function | Training | SR |
| --- | --- | --- |
| GPT-4o-mini | ✗ | 12.7% |
| Llama-3.1-8B | ✓ | 13.5% |

##### Fine-tuning facilitates better world models than prompt-based approaches.

To assess the effectiveness of our training approach for world models, we compare our framework with a variant, where we replace the trained world model (i.e., fine-tuned Llama-3.1-8B-Instruct) with a GPT-4o-mini prompted to predict the next observation solely based on 2-shot demonstrations (i.e., in-context learning) without training.
The sub-optimal performance of this variant, as shown in Table 5 (2nd row), suggests that SOTA LLMs do not have sufficient knowledge of environment dynamics, which is consistent with our findings in section 3.1.

Figure: Figure 6: Ablation on the number of sampled actions ($k$).
Refer to caption: x6.png

##### Abstracting observation elicits better next state prediction.

We evaluate the effectiveness of our observation abstraction (section 4.1.2), which focuses on state transition.
For that, we train a world model that learns to predict the full accessibility tree, i.e., $o_{t+1}$ instead of our transition-focused abstraction $\tilde{o}_{t+1}$.
As we expected, Table 5 (3rd row) reveals that generating the whole next observations (i.e., all elements in the viewport) results indeed hinder agent performance, yielding the worst SR among all ablations.
This shows that processing redundant and repeated information across observations negatively affects the world model in capturing critical state changes compared to abstracted observations that exclusively highlight state transition.

##### Choice of value functions.

We compare the fine-tuned value model (i.e., Llama-3.1-8B-Instruct) used for implementing WMA web agents with prompted GPT-4o-mini in Table 6.
Ours lead to a slightly better agent performance compared to GPT-4o-mini. This suggests fine-tuning the value function is a reasonable alternative in scenarios where API budgets are limited.

##### Budget for exploration.

Figure 6 shows that there is a positive trend between the number of sampled actions ($k$) during inference-time policy optimization in section 4.2) and the agents’ task performance (SR). These results suggest that our WMA web agent may benefit from more exploration of the future states when the budget is allowed.

## 6 Further Analyses

### 6.1 Combining Self-refine with Our World Models

**Table 7: Results of applying self-refine to GPT-4o-mini using simulated environment feedback.**
| Methods | SR |
| --- | --- |
| Vanilla CoT | 11.6% |
| Self-refine w/ our world model | 13.4% |
| WMA (ours; Fig 3) | 22.3% |

Besides our inference-time policy optimization, another way of using world models to improve the policy model is prompting it to refine its predicted action (Madaan et al., ), based on the outcome simulated by the world model.
Such self-refinement has been showing promising performance in diverse LLM applications (Shinn et al., ; Chae et al., ).

Here, we conduct a demonstrative
experiment of combining self-refine with our world model in the Map domain from WebArena.
Since tasks in this domain involve a complex set of utility tools, such as sorting and zoom-in, we consider it suitable for testing self-refine.
In this experiment, after the policy model $\theta$ produces an action $a_{t}$ based on the current observation $o_{t}$, we use our world model to simulate the next observation $\tilde{o}_{t}$ and prompt $\theta$ to refine the action based on $\tilde{o}_{t}$. Simply put, this setting allows $\theta$ to make adjustments to its output action when the predicted next observation is not optimal. Table 7 shows that refining with simulated environment feedback improves the agent’s policy selection by 1.8% percentage point in terms of accuracy compared to CoT.
While this provides a plausible direction for future work, our simulate-score-select paradigm (Figure 3 (bottom)) yields a almost 2x higher accuracy (13.4% vs. 22.3%), making it our choice of the policy optimization method.(^6^66 As a pilot/demonstrative trial, we do not consider sampling multiple actions for self-refinement.)

### 6.2 Types of Errors in World Models’ Predictions

To gain deeper insights into WMA web agents, we sample 50 erroneous predicted states (i.e., $\tilde{o}_{t+1}$) from world models in WebArena, and manually categorize the type of errors.
Whether a predicted state is erroneous is judged by a CS major who manually compares the viewport and the predicted observation. Examples of each type and details on the sampled states are provided in Appendix D.2.

Figure: Figure 7: Statistics of error types in erroneous observations predicted by $\phi$.
Refer to caption: x7.png

Figure 7 shows the statistics of the following error types: (i) Correct yet overly generic statements (24%) - Statements such as “The user will see a comprehensive layout of various order-related functionalities”, where the structure of the layout and what functionalities will be seen are not specified; (ii) Low competence in web elements/functions (26%) - Cases where the world model does not know how to use components on the web, e.g., expecting the search engine to show the desired items when the agent does not delete old texts on the search bar before entering a new keyword; (iii) Counterfactual imagination (42%) - Cases where the next observation predicted by the world model includes elements that are not supposed to occur/exist, e.g., making up products that are not sold in the store; (iv) others (8%) - other errors, such as skipping the next observation and predicting an observation that is further from the current time step.

## 7 Conclusions

We are the first study that incorporates world models in LLM-based web agents, addressing the limitation of current SOTA LLMs in understanding environment dynamics. Through extensive experiments in WebArena and Mind2Web, we show that (i) our WMA web agent can demonstrate great efficacy in policy selection by simulating outcomes of its actions via world models trained using our approach (i.e., transition-focused observation abstraction).
Moreover, (ii) our WMA web agent outperforms strong baselines (i.e., Tree search agent) with reduced cost and time for the exploration and (iii) achieves a new SOTA performance in Mind2Web.
By augmenting LLM-based web agents with world models, we establish a strong foundation for future research in web navigation.

## Appendix

## Appendix A Limitations

This paper has the following limitations:

##### Single modality.

This study focuses on building text-based world models and web agents. In web navigation, however, visual information can also play a critical role (Liu et al., ; Zheng et al., ). Although HTML and accessibility tree do represent the visual structure of a webpage to some degree, it is recommended for future work to incorporate visual information in addition to textual information for improving the learning of dynamics in the environment (Koh et al., ).

##### Application of multi-step planning.

Our current approach demonstrates significant performance gains in selecting actions within a single time step, i.e., from time $t$ to time $t+1$. However, we believe that our idea can be extended to multi-step planning (i.e., predicting $o_{t+n}$ where $n>1$). This can be achieved by recursively feeding the predicted next state back into the world model as the new observation, along with action candidates generated at each time step. Incorporation with techniques such as Monte Carlo Tree Search (Koh et al., ) or other planning algorithms is also feasible.
Future work can explore such multi-step planning to enable web agents to understand longer-term consequences of their actions and make more informed decisions in scenarios where planning multi-step ahead is necessary.

## Appendix B Experimental Details of Preliminary analyses

### B.1 Preliminary analysis I

We formulate next state prediction as a binary classification task rather than a generation task for an easier and more accurate evaluation (it is non-trivial to evaluate machine-generated accessibility tree or HTML).
Measuring the next state prediction capability as a generation task requires an additional human evaluation or off-the-shelf LLM judges, but it might introduce evaluation bias and there is no consensus that LLMs can judge this capability.

To collect training objectives for next state prediction, we use difflib python library(^7^77[https://docs.python.org/3/library/difflib.html](https://docs.python.org/3/library/difflib.html)) to calculate the lexical similarity between the golden next state and similar yet incorrect next state. Then, we select the top-$1$ similar yet wrong state as the negative next state and randomly shuffle the answer choices. The prompt used for next state prediction is shown in Figure 15. The interface for human annotation is shown in Figure 8.

### B.2 Preliminary analysis II

We use greedy decoding for sampling a sequence of 9 negative actions from GPT-4o-mini. Specifically, the LLM is instructed to generate 9 negative action candidates with the 2-shot demonstration.
Prompts used for action selection in preliminary analysis II are shown in Figure 16 and  17.

## Appendix C Implementation details

### C.1 World model

#### C.1.1 Dataset construction

##### Instruction and trajectory collection from WebArena.

As mentioned in section 5.1, WebArena does not provide anything other than the test set. We thus synthesize user instructions and accordingly collect trajectories.
In total, we obtain 14,200 instances using GPT-4o-mini with CoT prompt provided in Zhou et al. (). These instances are used to collect training data for world models in WebArena.

##### Transition-focused observation abstraction.

We implement the Hungarian algorithm (Kuhn, ) using munkres python package.(^8^88[https://pypi.org/project/munkres/](https://pypi.org/project/munkres/))
Details of the algorithm are in Algorithm 1. TaO in Algorithm 1 stands for Transition-aware Observation, and denotes the direct observation output from the Hungarian algorithm used in section 4.1.2. Then, using the output from the algorithm, we prompt an LLM to make a free-form description that captures the state transitions. The prompt used for producing free-form description is shown in Figure 18 and Figure 19.

Figure: Algorithm 1 Observation tree state matching for $\Delta(o_{t},o_{t+1})$ in section 4.1.2

#### C.1.2 Training

For world models and value functions, we use a learning rate of 1e-5 and spend around 3 GPU hours training them for 2 epochs on 8 RTX 4090 GPUs.

### C.2 Inference

We use top-$p$ decoding with $p=1.0$ for sampling 20 actions from the model following (Koh et al., ).
The three most frequent actions among the sampled actions are to be selected, and a next state prediction is to be performed for these actions.
The prompt used for the next state prediction of world models is shown in Figure 20.
For each predicted next state, a reward is calculated using the value function (the prompt is in Figure 21) , and the action with the highest reward is finally selected.
We use vLLM (Kwon et al., ) to run inference of fine-tuned LLMs.

### C.3 Details on WebArena

To ensure fair comparison and reproducibility, we conduct our experiments using the WebArena environment. Specifically, we utilize an Amazon Web Services (AWS) EC2 instance pre-configured with the Docker environment for WebArena.(^9^99[https://github.com/web-arena-x/webarena/blob/main/environment_docker/README.md#pre-installed-amazon-machine-image](https://github.com/web-arena-x/webarena/blob/main/environment_docker/README.md#pre-installed-amazon-machine-image)) This setup is identical to the experimental configuration employed by Zhou et al. () in their original study. By using this standardized environment, we maintain consistency with previous research and facilitate direct comparisons of our results with those reported in the literature. The WebArena Docker environment encapsulates all necessary dependencies, web interfaces, and evaluation metrics, ensuring that our experiments are conducted under controlled and replicable conditions. Details of each domain are explained below.

- •
Shopping: E-commerce platforms supporting online shopping activities (e.g., Amazon, and eBay). In this website, the agent can search and make an order for realistic items.
- •
CMS: Content Management Systems that manage the creation and revision of digital content (e.g., online store management).
- •
Reddit: Social forum platforms for opinion exchanges.
- •
Gitlab: Collaborative development platforms for software development.
- •
Map: Navigation and searching for information about points of interest such as institutions or locations. For Map domain, we use the online openstreetmap website(^10^1010[https://www.openstreetmap.org/](https://www.openstreetmap.org/)) since the button for searching a route of the provided docker does not properly work. This issue is also raised in the official WebArena github.(^11^1111[https://github.com/web-arena-x/webarena/issues/159](https://github.com/web-arena-x/webarena/issues/159))

### C.4 Details on Mind2Web

For running our experiments on Mind2Web, we obtain Mind2Web data from the official project page.(^12^1212[https://osu-nlp-group.github.io/Mind2Web/](https://osu-nlp-group.github.io/Mind2Web/)) We use the implementation of Wang et al. () to calculate the evaluation metrics, EA, AF1, Step SR, and SR.
Each action in the sequence comprises a (Target Element, Operation) pair,
We measure Element Accuracy (EA) which compares the selected element with all ground-truth elements, and Action F1 (AF1) that calculates token-level F1 score for the predicted
action. Each step of the task is evaluated independently with the ground-truth history provided.
We then define Step Success Rate (Step SR) and Success Rate (for the whole task). For calculating Step Success Rate (Step SR) and Success Rate (SR), a step is regarded as successful only if both the selected element and the predicted action is correct.
A task is regarded successful only if all steps have succeeded.
For step-wise metrics, we report macro average across tasks.

### C.5 Implementation Details of Baselines

##### Vanilla CoT (Zhou et al., 2023 )

For fair comparison, we first sample 20 actions with top-$p$ sampling similar to ours. We use the original CoT prompt from Zhou et al. (). Then we choose the most frequent action as the final action. We use the prompt in Figure 22.

##### Tree Search Agent (Koh et al., 2024 )

We use the codes from the official Github repository for implementing Tree search agent.(^13^1313https://github.com/kohjingyu/search-agents) For time and cost analysis on this agent, we run Tree search agent on 10% of WebArena instances due to its excessive cost.

##### Agent Workflow Memory (Wang et al., 2024b )

We use the codes from the official github repository to implement Agent Workflow Memory (AWM). We use GPT-3.5-Turob to create workflow memory from the train data of Mind2Web dataset. During our experiments, we find that the candidate generation module of MindAct (Deng et al., ) significantly degrades the original performance. This module calculates the relevance score of each element to the query so that web agents can predict action with more shortened observation. We provide the results of both settings with and without the candidate generation module.

For certain baselines, we obtain the performance from the original papers, which are marked with “*” in the result tables.

### C.6 Issue Regarding the Element Filtering Module of MindAct

The element selection module proposed by Deng et al. () used for filtering out irrelevant elements in the extremely long HTML content to avoid confusion. This element selection module is adapted to the suggested baseline in Mind2Web paper, MindAct and widely applied to the following methods (Wang et al., ; Zheng et al., ) including the AWM baseline. However, we find that this module introduces a significant performance decrease, by removing not only the irrelevant items but also the relevant ones. Thus, we re-implemented AWM in both with and without the filtering module.

### C.7 Instance Ids of Adapted Tasks for WebArena

We randomly sampled 200 instances from WebArena (50, 50, and 100 instances from Shopping, Gitlab, and Map, respectively)
We sample 100 instances from the Map domain as it is cost- and time-efficient due to its short inference time. We provide the full list of task ids below:

- •
Shopping: 49, 51, 96, 144, 146, 158, 162, 164, 165, 188, 189, 190, 226, 231, 235, 238, 263, 274, 278, 281, 300, 313, 319, 333, 337, 352, 355, 362, 376, 385, 386, 387, 432, 467, 468, 469, 506, 509, 511, 513, 515, 517, 518, 521, 528, 529, 530, 531, 587, 589
- •
GitLab: 156, 174, 177, 178, 205, 207, 297, 305, 306, 311, 315, 317, 339, 341, 349, 357, 389, 395, 396, 416, 418, 422, 441, 452, 475, 482, 483, 523, 524, 535, 537, 552, 553, 563, 564, 566, 569, 658, 662, 664, 669, 670, 736, 751, 783, 787, 789, 800, 803, 810
- •
Map: 7, 8, 9, 10, 16, 17, 18, 19, 20, 33, 34, 35, 36, 37, 38, 40, 52, 53, 54, 55, 56, 57, 58, 60, 61, 70, 71, 72, 73, 75, 76, 80, 81, 82, 83, 84, 86, 87, 88, 89, 90, 91, 92, 93, 97, 98, 99, 100, 101, 137, 138, 139, 140, 151, 153, 154, 218, 219, 220, 221, 222, 223, 224, 236, 248, 249, 250, 251, 252, 253, 254, 256, 257, 287, 356, 364, 365, 366, 367, 369, 371, 372, 373, 377, 378, 380, 381, 382, 383, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767

## Appendix D Details of further analyses

### D.1 Self-refine

We implement self-refine using the prompt shown in Figure 23. Specifically, we first generate a single action using the CoT prompt and we obtain the feedback from the value model used in our method. Then, we refine the action according to the feedback.

### D.2 Error Type Analysis and Examples

We sample 50 errors from the inference results in WebArena for our error analyses. The numbers of selected samples by domains are Shopping: 8, CMS: 11, Gitlab: 11, Reddit: 10, and Map: 10. The examples of the four error types are mentioned in section 6.2 and are respectively shown below.

- •
Low competence in web elements/functions: Figure 9.
- •
Counterfactual imagination: Figure 10.
- •
Correct yet overly generic statement: Figure 11.
- •
Others: Figure 12.

## Appendix E Examples of Successful Inference

We provide several successful examples of our WMA web agents:

- •
Inference on Mind2Web: Figure 13
- •
Inference on WebArena: Figure 14

## Appendix F Prompts

The following are prompts utilized in our study:

- •
Prompt for next state prediction in preliminary analysis I in Figure 15.
- •
Prompts for action selection in preliminary analysis II in Figure 16 and Figure 17.
- •
Prompt for refining TaO output in Figure 18
- •
Prompt for transition focused observation abstraction in Figure 19.
- •
Prompt used for the next state prediction of the world model in Figure 20.
- •
Prompt for reward calculation in value function in Figure 21.
- •
Prompt for baseline action prediction using accessibility tree with CoT in Figure 22.
- •
Prompt for self-refining in Figure 23.

Figure: Figure 8: Human annotation interface for preliminary analysis I in section 3.1.
Refer to caption: x8.png

Figure: Figure 9: Erroneous example (Low competence in web elements/functions). Although the agent does not delete old texts on the search bar before entering the new keyword ‘restaurants near CMU Posner Hall’, the world model still expects the next observation to show the desired search results.
Refer to caption: x9.png

Figure: Figure 10: Erroneous example (Counterfactual imagination). The model predicts that specific products (96 TY CITY86 Bmw 740i Limited Collector Hoodie Men‘s Close; Toyota 86 Bad Institute Monkey Champagne Cup, Volkswagen A9 Bug Pick Dead Red) will appear in the next observation, while this specific page does not list them as the products for sell.
Refer to caption: extracted/5924365/figure/next_state_hallucinations.png

Figure: Figure 11: Erroneous example (Correct yet overly generic statements). “Comprehensive layout” and ”various order-related functionalities” are ambiguous and unclear expressions.
Refer to caption: extracted/5924365/figure/generic.png

Figure: Figure 12: Erroneous example (Others). The predicted next state (i.e., contributions and activities) is actually several steps further away from the current time step.
Refer to caption: x10.png

Figure: Figure 13: Successful example (Mind2Web). WMA web agent successfully inferences on the Mind2Web benchmark (menards task #0). Using the policy model (i.e., GPT-4o), WMA web agent selects the most proper action click [208] by leveraging its learned environment dynamics.
Refer to caption: x11.png

Figure: Figure 14: Successful example (WebArena). WMA web agent successfully inferences on Gitlab domain in the WebArena benchmark (instance #175). Using the policy model (i.e., GPT-4o), WMA web agent selects the most proper action click [88] by leveraging its learned environment dynamics.
Refer to caption: x12.png

Figure: Figure 15: The prompt for preliminary analysis I in section 3.1: Next state prediction
Refer to caption: x13.png

Figure: Figure 16: The prompt for preliminary analysis II in section 3.2: Action selection w/o next state
Refer to caption: x14.png

Figure: Figure 17: The prompt for preliminary analysis II in section 3.2: Action selection w/ next state
Refer to caption: x15.png

Figure: Figure 18: The prompt for refining TaO output before generating final Transition-focused observation abstraction in section 4.1.2
Refer to caption: x16.png

Figure: Figure 19: The prompt for transition-focused observation abstraction in section 4.1.2
Refer to caption: x17.png

Figure: Figure 20: The prompt used for the next state prediction of the world model section 4.2
Refer to caption: x18.png

Figure: Figure 21: The prompt for reward calculation using the value function in section 4.2
Refer to caption: x19.png

Figure: Figure 22: The prompt used for baseline comparison with accessibility tree input using CoT in section 5.4
Refer to caption: x20.png

Figure: Figure 23: The prompt for self-refine in section 6.1
Refer to caption: x21.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can the model extend to handle visual inputs alongside text-based observations? This would make it more versatile for environments where visual information is vital.

2. How does the model perform in extended multi-step planning scenarios? Would recursive application of the world model effectively address multi-step planning challenges?

3. How does the observation abstraction perform in very dynamic web environments? In fast-changing states, does it retain relevant information, or does it risk omitting crucial details?

4. What is the impact of scaling the action space on the computational efficiency of the WMA model? Would the model maintain its efficiency with a larger action candidate set?
soundness: 2
strengths: 1. The paper considers the application of world models within LLM-based web agents, creating an interesting direction for improving policy selection in web-based navigation tasks.

2. The proposed transition-focused observation abstraction helps to reduce redundant data, lowering computational costs and allowing the model to focus on critical changes.

3. The authors show that the WMA model can be easily integrated into existing web agents, making it a versatile addition to LLM-based navigation models.
summary: The paper proposes an approach for enhancing the performance of LLM-based web agents in long-horizon tasks. The authors introduce a World-Model-Augmented (WMA) web agent that simulates the outcomes of its actions through a "world model." This design enables the agent to anticipate the effects of actions, thus reducing errors and improving decision-making in dynamic web environments. By employing a transition-focused observation abstraction, the model processes large state transitions without redundant data. Experiments conducted on WebArena and Mind2Web indicate that the WMA agent outperforms other agents in various metrics.
weaknesses: 1. While the paper demonstrates improved single-step action selection, it does not address multi-step planning comprehensively, which is crucial for certain long-horizon tasks.

2. The approach is limited to text-based web interactions, and although HTML trees are effective, omitting visual elements could restrict the model's applicability in web navigation tasks where visual cues are essential.

3. While the transition-focused abstraction reduces redundancy, it may oversimplify complex web elements, potentially leading to reduced accuracy in actions involving nuanced page layouts.

4. The paper does not discuss how the proposed world model scales when dealing with significantly larger action spaces, which is critical for broader applicability.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Clarifying questions:

1) What is “policy optimization” as applied to Tree Search Agent (Kuh et al, 2024). I thought Figure 3 bottom shows “policy optimization” which seems like it would only apply to World Model Agent. 

2) Correspondingly, what is "no policy optimization"? Is that just CoT? If so, why are the results different? Why is Tree Search Agent better?

3) Is value function trained with the world model data or is it a pure value function? If pure value function, how does that work with the abstracted observations? 

4) Line 284: I don’t get this line and how the citation matches. “explore diverse next states st+1 ∈ S (Wang et al., 2022)”

5) Table 5: The value function is fine-tuned. Is the Q-value function being fine-tuned as well and in the same way? If not, does not seem like a fair ablation. Also, is the Q-value function given ability to reason about what the action will do before outputting the value?
Other questions:

6) Why is the user instruction included in the data? Shouldn't the world model be able to predict the changes based on the action without needing to know the intent?

7) Table 2: Why not include the baselines from table 1? At least the tree search agent which is repeatedly compared to. 

8) Figure 1 shows that commercial LLMs cannot predict the next state well by looking at actual observations. Perhaps they would be able to pick the correct abstracted transition-focused observation? Running that same experiment with the transition focused observation version (ground truth, not predicted) could be interesting. 

9) Which leads me to the following question, how do larger commercial LLMs fare at predicting the next abstract transition? It would be nice to see Table 5 row 2 with a few different base LLMs. 

10) Why is HTML-T5 considered SOTA for Mind2Web? It seems like there are many much stronger algorithms presented in table 1. Mind2Web is also an offline evaluation dataset so does not account for different ways of completing a task or different action spaces. 

Other notes:

11) Line 184: “These findings highlight the necessity of world models in LLM-based web agents”: 
- Not sure I agree with this. I would say something more like, “This findings highlight the benefit world models could provide to LLM-based web agents”.
- The findings show that (my words) “LLMs do not have good world models for web activity” and “having a better world model would improve web agent performance”. This does not add up to “necessity”

12) “In Table 1 (middle), we first compare our WMA web agent (16.6%) with vanilla CoT (13.1%) and observe significant improvements over almost all domains in WebArena as detailed in Table 2.” 
This is confusingly written. Table 1 is the main table and there is only half a sentence of analysis. 

13) I do not like the y-axis bounds selection on many of the figures. It would be easier to read with 0 as lower bound (or at least a bit more space). E.g. Figure 6 is hard to tell what the k=1 line is at.
soundness: 3
strengths: Originality:
The paper provides an original algorithm for training web-agent world models and an original algorithm for using such models to improve LLM-based web agents. 

Clarity:
The clarity of the problem domain is quite good. The introduction lays a good case for how world models could improve web agents. The approach is quite clear. Figure 3 is nice. There is some breakdown of the clarity for some of the results but overall it was easy to follow along and understand what was being done. 

Significance:
The utility of world models for web-agents is quite clear but still under-explored. Both the training and inference time algorithm presented here could have significance down the line.
summary: This paper explores using explicit world models in LLM-based web agents. An (explicit) world model is a model able to predict the future state given an action on the current state. Their preliminary analysis shows that existing models cannot successfully predict the results of their actions, but that knowing that information would significantly help LLM-based agents. 

Their contributions are 
1) a training algorithm for web-agent world models that predicts an abstracted version of the transition to the observation after an action is applied. 
2) an algorithm for using the world model to improve performance. 

The training algorithm uses trajectory data to produce observation diffs from each action, then summarizes these changes with an LLM. The world model LLM is then trained to output the summaries (predict what will change in natural language)

The policy optimization algorithm uses a trained value function in addition to the world model. It works by sampling multiple potential actions, using the world model to predict changes, and then using the value function to predict the value. This can be thought of as a two-step Q-value function where the first step is the world model and the second step is evaluation. The value function is trained using reward data from Mind2Web. 

They have main results on WebArena and Mind2Web. There is also further analysis by category, and other experiments, including ablations over the reward estimation, training of world model, and use of abstracted observations.
weaknesses: The primary weakness of this paper is that the main results are quite weak. On WebArena, this significantly underperforms the listed baselines, and other newer baselines also exist. 

The second weakness is that it is not always clear what is being tested or shown. There are a number of questions (see below) that need to be clarified. 

To me the paper should focus more on the web-agent *world model* than on the web agent itself. There is some analysis on failure modes but not enough analysis on the world model itself. This may stem from the fact that abstract transition representations may be hard to analyze. A full world model would be able to show things like per-element accuracy. Perhaps there are other creative ways to analyze how to assess the learned world model and how it functions. 

The paper really dives into this one way of using the world model when it could take a broader view. For instance, why not use WMA/policy optimization

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. In section 4.1.1, training data for the world model is sampled from an LLM as web agent. Does this strategy create a dependency between the world model and the underlying agent, which could potentially hinder generalization?
2. The proposed method, particularly the state abstraction method, is limited to the text modality (HTML and accessibility tree). Can it be easily extended into a multimodal setup?
3. As stated in weaknesses, clarification of Vanilla CoT and WMA web agent /o policy optimization is needed.
soundness: 2
strengths: 1. To the best of my knowledge, this work is among the first to employ the world model in web agents. Its technical contribution includes a novel state representation, world model training method, and world model augmented agent design.
2. The resulting method shows improved performance and higher efficiency over baselines.
3. The experiment setup is overall reasonable. The authors provide sufficient ablation study and detailed analysis on performance, efficiency, and error distribution, contributing valuable insights to the community.
4. The writing and presentation of this paper is clear and easy to follow.
summary: This work proposes Wold-Model-Augmented (WMA) web agents. The authors present detailed analysis on the inability of current LLMs in estimating action consequences on the web and propose to abstract webpage state presentation as state transition description in natural language accordingly. Based on the abstraction, a world model for web navigation is trained and used for enhancing web agents. On two popular benchmarks, WebArena and Mind2Web, the proposed WMA web agent shows sizable performance gain over the baseline LLM and demands lower cost than tree-search methods.
weaknesses: 1. The experiments only cover end-to-end evaluation of agent performance. A directly evaluation on world modeling performance would be helpful for understanding the quality of the world model itself. For example, this evaluation can follow the same setting as the analysis of section 3.
2. The performance of Vanilla CoT and WMA web agent w/o policy optimization differs (13.1% vs 12.8% in Table 1). What are the differences between them? In addition, WMA web agent underperforms the tree search agent. Although tree search leverages additional signals and more compute, I think it is important to discuss if WMA web agent could scale to higher compute and deliver competitive performance under similar budget. 
3. The proposed method and evaluation is limited to the text modality. Recent studies have suggested that visual features are critical for good web navigation performance. It remains unclear if WMA could adapt to multimodal settings and advance the state-of-the-art of web navigation.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. As discussed in weakness 1, could the authors conduct a more extensive literature review to discuss the similarities and differences between existing WKM-based methods and their proposed approach? Although this is mentioned in section 6.1, too few works are considered.
2. As discussed in weakness 2, while the authors have considered the impact of exploration breadth in WKM during the ablation study, could they conduct experiments to consider the impact of depth as well?
If the authors can adequately address the questions above, I would be willing to reconsider the review.
soundness: 3
strengths: 1. The introduction of this paper is well-constructed. Through experiments, it identifies the issue that existing models cannot effectively predict the potential consequences of their actions, and it highlights that enabling models to understand these consequences can enhance their planning capabilities. The proposed approach is very reasonable.
2. The explanations in this paper are clear and easy to understand. The formulas and figures effectively illustrate the method proposed by the authors.
3. The authors conducted multi-dimensional experiments that effectively address potential questions from readers.
summary: This paper explores the introduction of world knowledge models into the field of agents, particularly within the context of web navigation. Specifically, the study finds through exploratory experiments that 
1. existing large models lack sufficient predictive capability regarding the consequences of their actions. 
2. Enabling these models to anticipate such consequences can significantly enhance their planning capabilities. 
Based on this finding, the authors trained a world knowledge model (WKM) to predict the potential outcomes of an action.  The proposed transition-focused observation abstraction plays a crucial role in the training stage.
The paper conducts comprehensive experiments to demonstrate the necessity and superiority of using WKM.
weaknesses: 1. Although this paper claims to be the first to introduce a world knowledge model into LLM-based agents, to my knowledge, this may not be the case. For example, the work "Agent Planning with World Knowledge Model" might also explore this area. The authors may need to conduct a more extensive literature review to compare and discuss the similarities and differences between their proposed method and existing approaches.
2. The performance improvement brought by the experiments in this paper is not significant. Additionally, the authors' WKM is limited to a depth of 1. It might be worth considering deeper levels.

Meta-review: Summary:

This paper presents a World-model-augmented (WMA) web agent, which simulates the outcomes of the agent’s actions for better decision-making. To train a world model for web agents, the paper proposes a transition-focused observation abstraction, where the prediction objectives are free-form natural language descriptions exclusively highlighting important state differences between time steps.  This helps overcome the challenges of training LLMs as world models predicting next observations, such as repeated elements across observations and long HTML inputs. Experiments on WebArena and Mind2Web show that the trained world model can improve agents' policy selection. Compared with tree search agents, the WMA agent demonstrates cost- and time-efficiency. 

Strengths:

1. The study of world models within LLM-based web agents is an interesting direction. The paper provides a new method for training world models for web agents and a new method for using such models to improve LLM-based web agents. (Reviewer N8Nf, Reviewer mAfj) The paper’s technical contribution includes a novel state representation, world model training method, and world model augmented agent design. (Reviewer YRJg)

2. The utility of world models for web-agents is still under-explored. Both the training and inference time algorithm presented in this paper could have significance down the line. (Reviewer mAfj)). 

3. The WMA agent shows improved efficiency over the tree search agent baseline (Reviewer YRJg).
 
4. In general, the writing of the paper is easy to follow. (Reviewer YRJg, tKJR)

Weaknesses: 

1. Lack of experiments to show the effectiveness of the world model under a multimodal setup (especially how it would improve a multimodal agent). The authors seem to have only considered taking multimodal information as input to the world model itself during the rebuttal period, but did not experiment with incorporating the world models into more recent multimodal agents.  

2. The proposed WMA agent does not consistently outperform baselines in terms of SR (e.g., WMA web agent vs. tree search agent in Table 1). Its effectiveness is not entirely convincing.

Decision: Accept (Poster)