Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

Authors: Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, Jinyoung Yeo
Primary area: applications to robotics, autonomy, planning
Keywords: Web Agent, World Model, Digital Agent, Planning, LLM

Abstract:
Large language models (LLMs) have recently gained much attention in building autonomous agents. However, performance of current LLM-based web agents in long-horizon tasks is far from optimal, often yielding errors such as repeatedly buying a non-refundable flight ticket. By contrast, humans can avoid such an irreversible mistake, as we have an awareness of the potential outcomes (e.g., losing money) of our actions, also known as the "world model". Motivated by this, our study first starts with preliminary analyses, confirming the absence of world models in current LLMs (e.g., GPT-4o, Claude-3.5-Sonnet, etc.). Then, we present a World-model-augmented (WMA) web agent, which simulates the outcomes of its actions for better decision-making. To overcome the challenges in training LLMs as world models predicting next observations, such as repeated elements across observations and long HTML inputs, we propose a transition-focused observation abstraction, where the prediction objectives are free-form natural language descriptions exclusively highlighting important state differences between time steps. Experiments on WebArena and Mind2Web show that our world models improve agents' policy selection without training and demonstrate our agents' cost- and time-efficiency compared to recent tree-search-based agents.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730479290043

thread_type:
official_review

<thread_object>
creator: reviewer_tkjr

<content>
summary:
This paper explores the introduction of world knowledge models into the field of agents, particularly within the context of web navigation. Specifically, the study finds through exploratory experiments that 
1. existing large models lack sufficient predictive capability regarding the consequences of their actions. 
2. Enabling these models to anticipate such consequences can significantly enhance their planning capabilities. 
Based on this finding, the authors trained a world knowledge model (WKM) to predict the potential outcomes of an action.  The proposed transition-focused observation abstraction plays a crucial role in the training stage.
The paper conducts comprehensive experiments to demonstrate the necessity and superiority of using WKM.

strengths:
1. The introduction of this paper is well-constructed. Through experiments, it identifies the issue that existing models cannot effectively predict the potential consequences of their actions, and it highlights that enabling models to understand these consequences can enhance their planning capabilities. The proposed approach is very reasonable.
2. The explanations in this paper are clear and easy to understand. The formulas and figures effectively illustrate the method proposed by the authors.
3. The authors conducted multi-dimensional experiments that effectively address potential questions from readers.

weaknesses:
1. Although this paper claims to be the first to introduce a world knowledge model into LLM-based agents, to my knowledge, this may not be the case. For example, the work "Agent Planning with World Knowledge Model" might also explore this area. The authors may need to conduct a more extensive literature review to compare and discuss the similarities and differences between their proposed method and existing approaches.
2. The performance improvement brought by the experiments in this paper is not significant. Additionally, the authors' WKM is limited to a depth of 1. It might be worth considering deeper levels.

questions:
1. As discussed in weakness 1, could the authors conduct a more extensive literature review to discuss the similarities and differences between existing WKM-based methods and their proposed approach? Although this is mentioned in section 6.1, too few works are considered.
2. As discussed in weakness 2, while the authors have considered the impact of exploration breadth in WKM during the ablation study, could they conduct experiments to consider the impact of depth as well?
If the authors can adequately address the questions above, I would be willing to reconsider the review.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701012190

thread_type:
official_review

<thread_object>
creator: reviewer_yrjg

<content>
summary:
This work proposes Wold-Model-Augmented (WMA) web agents. The authors present detailed analysis on the inability of current LLMs in estimating action consequences on the web and propose to abstract webpage state presentation as state transition description in natural language accordingly. Based on the abstraction, a world model for web navigation is trained and used for enhancing web agents. On two popular benchmarks, WebArena and Mind2Web, the proposed WMA web agent shows sizable performance gain over the baseline LLM and demands lower cost than tree-search methods.

strengths:
1. To the best of my knowledge, this work is among the first to employ the world model in web agents. Its technical contribution includes a novel state representation, world model training method, and world model augmented agent design.
2. The resulting method shows improved performance and higher efficiency over baselines.
3. The experiment setup is overall reasonable. The authors provide sufficient ablation study and detailed analysis on performance, efficiency, and error distribution, contributing valuable insights to the community.
4. The writing and presentation of this paper is clear and easy to follow.

weaknesses:
1. The experiments only cover end-to-end evaluation of agent performance. A directly evaluation on world modeling performance would be helpful for understanding the quality of the world model itself. For example, this evaluation can follow the same setting as the analysis of section 3.
2. The performance of Vanilla CoT and WMA web agent w/o policy optimization differs (13.1% vs 12.8% in Table 1). What are the differences between them? In addition, WMA web agent underperforms the tree search agent. Although tree search leverages additional signals and more compute, I think it is important to discuss if WMA web agent could scale to higher compute and deliver competitive performance under similar budget. 
3. The proposed method and evaluation is limited to the text modality. Recent studies have suggested that visual features are critical for good web navigation performance. It remains unclear if WMA could adapt to multimodal settings and advance the state-of-the-art of web navigation.

questions:
1. In section 4.1.1, training data for the world model is sampled from an LLM as web agent. Does this strategy create a dependency between the world model and the underlying agent, which could potentially hinder generalization?
2. The proposed method, particularly the state abstraction method, is limited to the text modality (HTML and accessibility tree). Can it be easily extended into a multimodal setup?
3. As stated in weaknesses, clarification of Vanilla CoT and WMA web agent /o policy optimization is needed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710397073

thread_type:
official_review

<thread_object>
creator: reviewer_mafj

<content>
summary:
This paper explores using explicit world models in LLM-based web agents. An (explicit) world model is a model able to predict the future state given an action on the current state. Their preliminary analysis shows that existing models cannot successfully predict the results of their actions, but that knowing that information would significantly help LLM-based agents. 

Their contributions are 
1) a training algorithm for web-agent world models that predicts an abstracted version of the transition to the observation after an action is applied. 
2) an algorithm for using the world model to improve performance. 

The training algorithm uses trajectory data to produce observation diffs from each action, then summarizes these changes with an LLM. The world model LLM is then trained to output the summaries (predict what will change in natural language)

The policy optimization algorithm uses a trained value function in addition to the world model. It works by sampling multiple potential actions, using the world model to predict changes, and then using the value function to predict the value. This can be thought of as a two-step Q-value function where the first step is the world model and the second step is evaluation. The value function is trained using reward data from Mind2Web. 

They have main results on WebArena and Mind2Web. There is also further analysis by category, and other experiments, including ablations over the reward estimation, training of world model, and use of abstracted observations.

strengths:
Originality:
The paper provides an original algorithm for training web-agent world models and an original algorithm for using such models to improve LLM-based web agents. 

Clarity:
The clarity of the problem domain is quite good. The introduction lays a good case for how world models could improve web agents. The approach is quite clear. Figure 3 is nice. There is some breakdown of the clarity for some of the results but overall it was easy to follow along and understand what was being done. 

Significance:
The utility of world models for web-agents is quite clear but still under-explored. Both the training and inference time algorithm presented here could have significance down the line.

weaknesses:
The primary weakness of this paper is that the main results are quite weak. On WebArena, this significantly underperforms the listed baselines, and other newer baselines also exist. 

The second weakness is that it is not always clear what is being tested or shown. There are a number of questions (see below) that need to be clarified. 

To me the paper should focus more on the web-agent *world model* than on the web agent itself. There is some analysis on failure modes but not enough analysis on the world model itself. This may stem from the fact that abstract transition representations may be hard to analyze. A full world model would be able to show things like per-element accuracy. Perhaps there are other creative ways to analyze how to assess the learned world model and how it functions. 

The paper really dives into this one way of using the world model when it could take a broader view. For instance, why not use WMA/policy optimization

questions:
Clarifying questions:

1) What is “policy optimization” as applied to Tree Search Agent (Kuh et al, 2024). I thought Figure 3 bottom shows “policy optimization” which seems like it would only apply to World Model Agent. 

2) Correspondingly, what is "no policy optimization"? Is that just CoT? If so, why are the results different? Why is Tree Search Agent better?

3) Is value function trained with the world model data or is it a pure value function? If pure value function, how does that work with the abstracted observations? 

4) Line 284: I don’t get this line and how the citation matches. “explore diverse next states st+1 ∈ S (Wang et al., 2022)”

5) Table 5: The value function is fine-tuned. Is the Q-value function being fine-tuned as well and in the same way? If not, does not seem like a fair ablation. Also, is the Q-value function given ability to reason about what the action will do before outputting the value?
Other questions:

6) Why is the user instruction included in the data? Shouldn't the world model be able to predict the changes based on the action without needing to know the intent?

7) Table 2: Why not include the baselines from table 1? At least the tree search agent which is repeatedly compared to. 

8) Figure 1 shows that commercial LLMs cannot predict the next state well by looking at actual observations. Perhaps they would be able to pick the correct abstracted transition-focused observation? Running that same experiment with the transition focused observation version (ground truth, not predicted) could be interesting. 

9) Which leads me to the following question, how do larger commercial LLMs fare at predicting the next abstract transition? It would be nice to see Table 5 row 2 with a few different base LLMs. 

10) Why is HTML-T5 considered SOTA for Mind2Web? It seems like there are many much stronger algorithms presented in table 1. Mind2Web is also an offline evaluation dataset so does not account for different ways of completing a task or different action spaces. 

Other notes:

11) Line 184: “These findings highlight the necessity of world models in LLM-based web agents”: 
- Not sure I agree with this. I would say something more like, “This findings highlight the benefit world models could provide to LLM-based web agents”.
- The findings show that (my words) “LLMs do not have good world models for web activity” and “having a better world model would improve web agent performance”. This does not add up to “necessity”

12) “In Table 1 (middle), we first compare our WMA web agent (16.6%) with vanilla CoT (13.1%) and observe significant improvements over almost all domains in WebArena as detailed in Table 2.” 
This is confusingly written. Table 1 is the main table and there is only half a sentence of analysis. 

13) I do not like the y-axis bounds selection on many of the figures. It would be easier to read with 0 as lower bound (or at least a bit more space). E.g. Figure 6 is hard to tell what the k=1 line is at.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731039162508

thread_type:
official_review

<thread_object>
creator: reviewer_n8nf

<content>
summary:
The paper proposes an approach for enhancing the performance of LLM-based web agents in long-horizon tasks. The authors introduce a World-Model-Augmented (WMA) web agent that simulates the outcomes of its actions through a "world model." This design enables the agent to anticipate the effects of actions, thus reducing errors and improving decision-making in dynamic web environments. By employing a transition-focused observation abstraction, the model processes large state transitions without redundant data. Experiments conducted on WebArena and Mind2Web indicate that the WMA agent outperforms other agents in various metrics.

strengths:
1. The paper considers the application of world models within LLM-based web agents, creating an interesting direction for improving policy selection in web-based navigation tasks.

2. The proposed transition-focused observation abstraction helps to reduce redundant data, lowering computational costs and allowing the model to focus on critical changes.

3. The authors show that the WMA model can be easily integrated into existing web agents, making it a versatile addition to LLM-based navigation models.

weaknesses:
1. While the paper demonstrates improved single-step action selection, it does not address multi-step planning comprehensively, which is crucial for certain long-horizon tasks.

2. The approach is limited to text-based web interactions, and although HTML trees are effective, omitting visual elements could restrict the model's applicability in web navigation tasks where visual cues are essential.

3. While the transition-focused abstraction reduces redundancy, it may oversimplify complex web elements, potentially leading to reduced accuracy in actions involving nuanced page layouts.

4. The paper does not discuss how the proposed world model scales when dealing with significantly larger action spaces, which is critical for broader applicability.

questions:
1. Can the model extend to handle visual inputs alongside text-based observations? This would make it more versatile for environments where visual information is vital.

2. How does the model perform in extended multi-step planning scenarios? Would recursive application of the world model effectively address multi-step planning challenges?

3. How does the observation abstraction perform in very dynamic web environments? In fast-changing states, does it retain relevant information, or does it risk omitting crucial details?

4. What is the impact of scaling the action space on the computational efficiency of the WMA model? Would the model maintain its efficiency with a larger action candidate set?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732635766034

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
First of all, thank you for your patience. We have dedicated the entire discussion period to conducting experiments in response to the reviewers' insightful and interesting suggestions, as we wanted to thoroughly address their requests.
We provide the experiments that the reviewers commonly mention in this general response.

### **Extending our world model to take multimodal input**

Specifically, we enhanced the model to process both text and image observations. We extended our world model to a multimodal setting, inspired by the recent success of multimodal web agents. For our experiments, we used the Mind2Web (cross-task) dataset with Qwen-2-VL-2B as the backbone Vision Language Model.

Despite using a smaller parameter size compared to Llama-3.1-8B, the multimodal input led to notable improvements across all metrics. These results demonstrate two key findings: (1) our framework can readily adapt to multimodal settings, and (2) the addition of visual modality provides clear benefits. Given that we used a naive approach to image input, we expect further improvement when incorporating more sophisticated image prompting techniques, such as SeeAct or Set-of-Marks, could further enhance performance.


|  | Modality of World Model | Elem Acc | OF1 | Step SR | SR |
| :---- | ----- | ----- | ----- | ----- | ----- |
| MindAct | \- | \- |\-  | 17.4 | 0.8 |
| AWM | \- | 78.3 | 74.1 | 62.8 | 15.3 |
| AWM+WMA (Llama-3.1-8B-textonly) | Text | 79.9 | 75.8 | 67.0 | 25.4 |
| AWM+WMA (Qwen-VL-2B-textonly) | Text | 79.2 | 75.1 | 65 | 23.7 |
| AWM+WMA (Qwen-VL-2B-multimodal) | Text+Image | **83.0** | **78.9** | **72.8** | **36.7** |

### **Extending WMA web agent to multi-step exploration settings (depth + width)**
Beyond searching for the best action in a single step, we also explored finding optimal paths through multi-step search by iteratively using the world model. However, prediction errors accumulate as search depth increases when using only world model simulations, limiting real-world applicability. To address this, we adopt a hybrid approach combining simulated interactions for width exploration and actual environmental interactions for depth exploration.


We used A*-like search algorithm following Tree Search Agent and conducted experiments using the same settings as shown in Figure 6 (Ablation on the number of sampled actions), with results presented in the table below. We found that increasing the depth from (w=3, d=1) to (w=3, d=3) improves performance (17.1 -> 19.6). However, when comparing settings with the same exploration budget - (w=9, d=1) vs. (w=3, d=3), we find that allocating more budget to width shows slightly better performance.

A specific challenge explains this issue: Many errors occur during the execution of optimal action sequences due to mismatches between search time and execution time. Even when a browser loads the same content from the same URL, element IDs calculated by the backend can change upon page reload. This means that the same element might have different IDs between the search and execution phases.


| width (w) /depth (d) | d=1 | d=2 | d=3 |
| :---- | :---- | :---- | :---- |
| **w=1** | **7.1** | **\-** | **10.7** |
| **w=2** | **10.1** | **12.6** | **\-** |
| **w=3** | **17.1** | **\-** | **19.6** |
| **w=9** | **20.5** | **\-** | **\-** |
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:

This paper presents a World-model-augmented (WMA) web agent, which simulates the outcomes of the agent’s actions for better decision-making. To train a world model for web agents, the paper proposes a transition-focused observation abstraction, where the prediction objectives are free-form natural language descriptions exclusively highlighting important state differences between time steps.  This helps overcome the challenges of training LLMs as world models predicting next observations, such as repeated elements across observations and long HTML inputs. Experiments on WebArena and Mind2Web show that the trained world model can improve agents' policy selection. Compared with tree search agents, the WMA agent demonstrates cost- and time-efficiency. 

Strengths:

1. The study of world models within LLM-based web agents is an interesting direction. The paper provides a new method for training world models for web agents and a new method for using such models to improve LLM-based web agents. (Reviewer N8Nf, Reviewer mAfj) The paper’s technical contribution includes a novel state representation, world model training method, and world model augmented agent design. (Reviewer YRJg)

2. The utility of world models for web-agents is still under-explored. Both the training and inference time algorithm presented in this paper could have significance down the line. (Reviewer mAfj)). 

3. The WMA agent shows improved efficiency over the tree search agent baseline (Reviewer YRJg).
 
4. In general, the writing of the paper is easy to follow. (Reviewer YRJg, tKJR)

Weaknesses: 

1. Lack of experiments to show the effectiveness of the world model under a multimodal setup (especially how it would improve a multimodal agent). The authors seem to have only considered taking multimodal information as input to the world model itself during the rebuttal period, but did not experiment with incorporating the world models into more recent multimodal agents.  

2. The proposed WMA agent does not consistently outperform baselines in terms of SR (e.g., WMA web agent vs. tree search agent in Table 1). Its effectiveness is not entirely convincing.