Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
Authors: Hyungjoo Chae
Namyoung Kim
Kai Tzu-iunn Ong
Minju Gwak
Gwanwoo Song
Jihoon Kim
Sunghwan Kim
Dongha Lee
Jinyoung Yeo
Venue: ICLR 2025 Poster (2025)
Keywords: Web Agent
World Model
Digital Agent
Planning
LLM
Primary area: applications to robotics, autonomy, planning

Abstract: Large language models (LLMs) have recently gained much attention in building autonomous agents. However, performance of current LLM-based web agents in long-horizon tasks is far from optimal, often yielding errors such as repeatedly buying a non-refundable flight ticket. By contrast, humans can avoid such an irreversible mistake, as we have an awareness of the potential outcomes (e.g., losing money) of our actions, also known as the "world model". Motivated by this, our study first starts with preliminary analyses, confirming the absence of world models in current LLMs (e.g., GPT-4o, Claude-3.5-Sonnet, etc.). Then, we present a World-model-augmented (WMA) web agent, which simulates the outcomes of its actions for better decision-making. To overcome the challenges in training LLMs as world models predicting next observations, such as repeated elements across observations and long HTML inputs, we propose a transition-focused observation abstraction, where the prediction objectives are free-form natural language descriptions exclusively highlighting important state differences between time steps. Experiments on WebArena and Mind2Web show that our world models improve agents' policy selection without training and demonstrate our agents' cost- and time-efficiency compared to recent tree-search-based agents.

Published as a conference paper at ICLR 2025
WEB AGENTS WITH WORLD MODELS: LEARNING AND LEVERAGING ENVIRONMENT DYNAMICS IN WEB NAVIGATION
Hyungjoo Chac
Namyoung Kim Jihoon Kim
Kai Tzu-iunn Ong
Minju Gwak Jinyoung Yeo
Gwanwoo Song
Sunghwan Kim
Dongha Lee
Yonsei University
(napoout,
namyoung.king donalee, jinyeo)@yonsei..ack
Q
httpe://gthho..ccooyyyyee//////2
https://ff.c//ppcee//nnggg-----1 Agents
ABSTRACT
Large language models (LLMs) have recently gained much attention in building autonomous agents. However, performance of current LLM-based web agents in long-horizon tasks is far from optimal, often yielding errors such as repeatedly buying non-refundable flight ticket. By contrast, humans can avoid such an irre- versible mistake, as we have an awareness of the potential outcomes (e.g. losing money) of our actions, also known the "world model". Motivated by this, our study first starts with preliminary analyses, confirming the absence of world mod- els in current LLMs (e.g., GPT-4o, Claude-3.5-Soneee etc.). Then, we present a (WMA) web agent, which simulates the outcomes of its actions for better decision decision-making. To overcome the challenges in training LLMs world models predicting next observations, such repeated elements across observation
Worlddmodel--uggeee
observations and long HTML inputs, we propose a transition-focuse abstraction, where the prediction objectives are free-form natural language de- scriptions exclusively highlighting important state differences between time steps. Experiments on WebArena and Mind2Web show that our world models improve agents' policy selection without training and demonstrate superior cost- and time- efficiency compared to recent seeesearch-baseee agents.
INTRODUCTION
Large language models (LLMMs) have been widely applied to solve tasks in diverse domains, includ- ing web navigation, where LLMs generate action sequences (e.g., click) to accomplish user goals on websites (Shi et al. 2017: Kim et al.] 2024). Despite some success (Yao et al. 2022). LLM- based web agents' performance remains significantly poor in long-horizon environments such as
WebArena (Zhou et al. 2023), where GPT-4 yields task success rate of 14.41% whereas humans have a success rate of 78.24%. This raises question: Why do LLMs, despite their advancements, perform much worse than humans in web navigation?
Humans avoid unwanted situations by considering the possible outcomes of our actions before- hand (Edwards) 1954). Such awareness of actions and outcomes is referred to as the "world model" (Forrester, 1995). Meanwhile, existing LLM-based web agents rely heavily on trial and enror to make decisions, they lack world models to help them foresee the outcome of an action without actually performing it (LeCun) 2022), leading to sub-optimal decision maasiinnmmainng that is irre- versible (e.g.. repeatedly buying non-refundable item). Acknowledging the importance of world models, studies in robotics and reinforcement learning (RL) have proposed to incorporate world models for agents in navigation tasks. For instance, Du et al. 2023) and Yang al. (2024) apply

---

Published as a conference paper at ICLR 2025
world models to simulate visual outcomess/obeerattinss of input texts or robot control. The Dreamer series use world models to predict latent state of images and use them to optimize policies, reducing
the need for actual interactions in game environments (Hafner et al. 2019a 2020 2024).
Motivated by these, this paper begins by investigating SOTA LLMs' understanding of "environment dynamics", i.e., the association between actions and environment states. We reveal that (i) current LLMs (e.g.. GPT-4o and Claude-3.5-Sonne struggle with predicting the outcomes of their actions and (ii) the awareness of potential outcomes helps them make decisions aligning with user goals. Upon these findings, we present a Seett-MMoe----ueeeee the outcomes of its actions for better decision-making, However, naively training a world model to predict the next observation state (i.e., the entire webpage) can lead to a large amount of repeated elements across observations and long HTML. inputs, negatively affecting model performance. Thus, we propose a novel transition-focuser observation abstraction, where the world model is trained to generate free-form natural language descriptions exclusively highlighting important state differences between time steps (e.g. an updated price the website). During inference, our agent first simulates the outcome (i.e., next observation) each action candidate (from the policy model) using the world model. Then, value function estimates the rewards of all simulated observations, helping the agent select a final action with the highest estimated reward. Our contributions are two-fold:
(WMA) web agent, which simulates
We are the first to pioneer world models in LLM-based web agents, laying the groundwork for policy adaptation through simulated environment feedback in web navigation.
We present novel transition- focuseed observation abstraction for training LLMs world models. We show that using world models trained with this method can improve action selection by simulating the action candidates without training the policy models. Also, we demonstrate our agents' cost- and time-efficiency compared to recent tree-search-based agents (Koh al. 2024). by 6.8x and 5.3x, respectively.
RELATED WORK
Benchmarks for web agents. Many benchmarks have been introduced to evaluate LLM-based agents' ability in web navigation (Kim et al.) (024).. MiniWoB (Shi et al., 2017) and Mini- WoB++ (Liu et al.) 2018) are among the first widely adopted benchmarks. More recently, Web- Shop (Yo et al. (022) simulates e-commerce environments where agents are tested to execute tasks based on text instructions on the web. These early benchmarks are limited to specific and con- strained domains. Mind2Web (Deng al.) (224) curates web tasks across more diverse domains, and WebArena (Zhou et al. 2023) emphasizes functional correctness and more realistic scenarios (e.g., posting articles Reddit) in simulated environment. We adopt Mind2Web and WebArena for evaluation for their generalizability and resemblance of real-world web interactions.
LLM-based web agents. In recent years, LLM-based agents have become popular in the web navigation domain. However, since many powerful proprietary LLMs do not provide access to model parameters, many studies of web navigation have been focusing on training-free methods where LLMs directly learn from user inputs (i.e., prompts) without task-specific training (Sodhi et al. 2023 Zheng et al. 2023). For instance, Wilbur (Lutz et al. (024) and Agent Workflow Mem- ory [aag et al. 20244) leverage a verification model (Pan et al. 2024) with prompt-based methods to collect successful trajectory data for guiding the agent's policy at inference time. AutoEval (Pan) al. 2024) and Tree search agent (Koh et al.) (024) increase the number of trials and reasoning paths, further improving system performance. However, due to their trial-and-error nature, these ap- proaches can not only be computationally inefficient in gathering trajectories as tasks become more complex but also more prone to undesired results (e.g., booking non-refundable ticket). Our WMA web agent reduces such risks via a world model, which predicts future observations and the rewards of their corresponding action candidates before actually making action. Furthermore, our approach can be orthogonally applied to many of the existing methods.
World models in autonomous agents. World models refer to systems that generate internal rep- resentations of the world, predicting the effects of their actions on environments (LeCun (022). In RL, simulating observations and environmental feedback using world models allow the policy model to learn (Sutton, 1990) plan (Ha & Schmidhuber] 2018 Hafner et al. 2019b) without
2

---

Published as a conference paper at ICLR 2025
actually interacting with the environment. While some world models trained with raw observa- tions (Oh et al. 2013 Chiappa et al.) 2017). others are built on latent representations (Hafner et al. 2019a;20200 Kipf et al. 2020). For instance, in the image domain, |Hanner et al. (2020) train world model by training it to first compute a posterior stochastic state based on the current image and then prior stochastic state that tries to predict the posterior without access the image. Within the field of LLMs. Zhhng et al. (2024) convert visual observations into natural language and employs an LLM-based world model for text-based games, and Wang al. (2024a) further transform ob- servations into structural format (e.g., JSON), improving LLMs' reasoning over state transition functions. In web navigation, environments are built upon not only natural language but on more complex text modalities such as HTML and DOM trees. We address this by transforming them to novel free-form description, highlighting the state difference between each time step.
PRELIMINARY ANALYSES: ARE CURRENT LLMS AWARE OF ENVIRONMENT DYNAMICS IN WEB NAVIGATION?
We first start with investigating whether LLMs can understand the association between actions and their effects the environment, i.e., understand the enviromment dynamics. We conduct analyses addressing these two questions:
Preliminary question I: Are LLMs aware of the outcomes of their actions?
Preliminary question II: When having access to the outcome of each action candidate, can LLMs select optimal action aligning with the user objective?
For the analyses, we sample 100 user instructions from WebArena and annotate human trajectories within the environment. Each instance has user instruction, the current state, human-annotatee golden action, and the corresponding next state resulting from the golden action. We analyze 4 pop- ular closed-source SOTA LLMs: GPT-4o-mini (Zhu et al.) (023), GPT-4o, GPT-4-Turbo (OpenAl 2023), and Claude-3.5-Somee (Anthropic 2024). More details are in Appendix
3.1 PRELIMINARY ANALYSIS I LLMS STRUGGLE WITH PREDICTING THE NEXT STATES CAUSED BY THEIR ACTIONS
Setups. We test LLMs' ability to predict the outcomes
83%
of actions the web via binary classification task.
80-
Ramdom
Given the current state and the golden action, the LLM
Accurncy
prompted select the correct next state from (i) the golden next state and (ii) a lexically similar yet incorrect next state retrieved from the same trajectory. We calcu- late the lexical similarity with difflib (Python 2024). We assess classification accuracy.
54%
55%
58%
52%
GPT-4o
GPT-4o
GPT4 -tubo
Clande-
Human
3.5-Sonnct
Figure 1: LLMs state prediction.
performance in next
Results. Figure I reveals that under vanilla settings, current LLMs cannot effectively predict the next states caused by their actions. First, all adopted LLMs (54.75% on average) lose significantly to humans. Also, Claude-3.5-Somee performs almost as badly as random guessing. These suggest that the world model, the ability to foresee the potential outcomes of actions taken, absent L.MMs.
3.2 PRELIMINARY ANALYSIS II LLMS MAKE BETTER ACTION SELECTION WHEN ACCESSING THE OUTCOME OF EACH ACTION CANDIDATE
Setups. We assess whether LLMs can select correct ac- tion that aligns with the user goal when they are provided with the outcome each action candidate. Given the cur- rent state, 10 action candidates, and their corresponding outcomes/next states, the LLM is prompted to differentiate the golden action from other 9 negative actions.
wio Next State
w/ Next State
70%
73% (+57.7%)
(137,2%)
(137.3%)
60 (%))
Aecotra A
60
(+29.2 (129.3%)
53%
SIM
53%
41%
Claude- 3.5-Somnet Figure 2: LLMs' performance in ac- tion selection (w/ and w/o next states).
GPT-40 -mini
GPT-4e
GPT-4
Results. Figure 2 compares LLMs' performance dif- ferentiating the golden action from negative actions when
-turbo
3

---

Published as a conference paper at ICLR 2025
World Model Training
Step :: Harvesting Agent Trajectories
Step Transition-focuse Observation Abstraction
Step III: Learning Environment Dynamics
Action &
State transition
Abstracted observation
{I, Ot, 96-1 spdsong
Policy model
Observsiion
Environment
Alog ot+1)
0c+1
{L, ogy 6t-1
World
[1, 9 eentt-enviromment interaction data
UPARTED:
StarkcTokt
Training data D
inference-time Policy Optimization via the World Model
Action candidates
Predicted next observations
Estimated reward scores
0.2
Selected action
6
0.3
argmax
a
Pollcy
Predicting the observation
0.1
Selecting the action ylelding the most optimal state
Figure 3: Framework overview. We first collect training data for world models (top). After training, perform policy optimization by selecting the action leading optimal next state (bottom).
they are/are not provided with the resulting next state of each candidate action. We find that current SOTA LLMs have difficulty in selecting correct actions when they can only rely on the current obser- vations/states (striped bars), yielding an average accuracy of only 49%. However, when augmented with the corresponding next state of each action candidate, they demonstrate huge performance gains (up to 38% improvement) in selecting correct actions. When only the current state and the user ob- jective are provided, GPT-4o yields an accuracy of 53%. In contrast, when the next state given, performance rises to 73%.
3.3 INSIGHITS FROM PRELIMINARY ANALYSES
Through our preliminary analyses, we have demonstrated that: (i) Web agents built with SOTA LLMMs are bad at predicting how their actions affect next states; (ii) When being aware of how an action affects the next state, LLMS can make better decisions. These findings highlight the necessity of world models in LLM-based web agents, pointing out promising direction for facilitating better web agents in complex, Long-horizon navigation tasks.
4 WORLD-MODEL-AUGMENT WEB AGENTS
Motivated by the above insights, present novel framework for World-MModel-Augmeetee (WMA) web agents, LLM-based web agents equipped with world models. The world models leam//everage environment dynamics (i.e., association of actions and outcomes) to simulate plausi- ble next observations of agents' actions, facilitating better decisions (i.e., polices) in web navigation.
Formulations. Since web agents access only information in the viewport (Le., users' visible area), model web navigation a partially observable Markov decision process (POMDP). We consider a web environment & with: (i) a hidden state space S; (ii) an action space A, including language- guided actions (e.g., CLICK, TYPE, HOVER, etc.) and their descriptions; (iii) an observation space O representing an accessibility tree of the page, which is simplified DOM tree (Zhou et al. 2023).
In the POMDP, the agent receives new partial observation 0141 O from & after performing an action E AA based on Such state transition - St41 is managed by golden transition function T S
S provided in the environment.
4.1 WORLD MODEL TRAINING
We hereby introduce the training process of our world models. As shown in Figure (top), our training consists of three main steps:
4

---

Published as a conference paper at ICLR 2025
4.1.1 STEP I: HARVESTING AGENT-ENVIRONMER INTERACTION DATA
We start by collecting the dataset D E {L, or, at, 0t+1} from the environment & for training world models. For that, we prompt an LLM as web agent to achieve the goal provided in the user instruction I, by iteratively predicting an action at based on the current observation or throughout all time steps. Consequently, we obtain D from trajectory T {01, Q1, 02, an Ox+11 } based on T, and environment states of time steps {81, Sn+1) S obtained transition function T.
4.1.2 STEP II: TRANSITION-
OBSERVATION ABSTRACTION
With the collected data D Σi= {I, or, de, 01+1). it is intuitive to train LLM-based world models to predict 0s+11 which is expressed with texts (e.g.. HTML and accessibility tree) (Deng et al.) [024] Zhou al. 2023). However, simply using textual observations to represent environment states and use them as training objectives may introduce the following downsides:
Low information gain during training: State transitions in websitesoften involve altering only part of the previous observation (e.g., a drop-down menu is clicked). As a result, most information in o++11 remains the same as it is in 0₂. Therefore, predicting the entire textual observation from scratch may result in low information gain during training.
Excessively long sequence length: Processing the whole text-based observations can lead to excessively long sequence length and consequently high computational costs. Indeed, this can be partially mitigated by replacing raw HTML with an accessibility tree (relatively simple), using it as LLMs' training objectives still introduce a long sequence length (4K tokens on average, see Figure
To address the above bottleneck in training text-based models (i.e, LLMs) as world models, we draw inspira- tion from how the RI community conventionally im- plements world models: using estimated latent vec- tors summaries of raw visual observations, reducing memory footprints for effectively learning environment dynamics (Doem et al. 2018 Hafner et al. 2019a)
3.0K
Proquency
2.0K
A(o, 0r))
1 description
1.0K
We thus propose to abstract raw text observations, with focus state transition between consecutive obser- vations, for obtaining better training objectives.
0K
2K
6K
10K
# of tokens
Figure 4: Sequence length distribution of different observation representations.
To collect abstracted world models, one may simply run an off-the-shelf
observations for training
summarizer on 01+1 collected in Step However, while reducing sequence length, this does not address the low information gain caused by repeated elements between o; and 0t+1- Thus, instead of such naive approach, as shown in Figure [5] we first (i) apply the Hungarian algorithm (Kuhn 1995) to calculate cost matrix for matching elements between On and o++1 and (ii) mechanically transform the results into a list of state transition A(or, 0t+1). ). pointing out UPDATED, DELETED, and ADDED elements on the web. After that, we prompt an L.LM to convert the extracted A ((or, oc+1) into a free-from natural language description 0 which highlights the difference between the new observation 0++1 and ot. Replacing ot-1 in D {I,ot, ae, o++1} collected in Step I with ōz+1 just acquired here, we get final dataset D [I ot, at, ōt+1) ) for training world models.
4.1.3 STEP III: LEARNING ENVIRONMENT DYNAMICS
Lastly, using D. we proceed to train the internal world model á of the web agent to learn the environ- ment dynamics. Formally, an LLM working as the world model trained predict the abstracted observation 5 of the next state $141, given three inputs: the user instruction I, the current observa- tion o,, and the current action a,, This LLM trained minimize the following loss term via the next-token prediction objective:
L. = log
Σ (D
10o at,
(1)
Through this process, this LLM learns the environment dynamics, working as a world model that helps the web agent to foresee the potential outcome (i.e., predict the next observation) of its action.
5

---

Published as a conference paper at ICLR 2025
iransition-focusee Observation Abstraction
Action tire to "MC Park? to the textbox "To"
01-1
0g
Pp-1
UPOATED Ibors:
The 'From' textbox will no: require...
Piststurgh can
(heck if
Park
[1]
Spedlrwttee lugo
127
recal uue"
[559] texthox 'To' Focusec: Thue
[551] LecU "Zzan zequiret: ralse
2. The To textbox will ntaii focus,
suggasting the user can continun
one tour by
te reached in
recust: Thu Tooes
Biver netWe loge
recuired: Felse
[996]
Statt 'PEE Porking Fore
LLM
make adjustments the sestination input.
reaiieed Rilse
Iters:
"Go" button (ID: 569) Initiate route calculation after confirming the inputs.
lieey
[641] Link "5igh Up'
Hungarian algorithm
600ED Tionsz Nane
Task objective
Element matching
Extracted state transition A(a, Pcis)
Free-form description of the next observation
Figure 5: The overview of transition-focused observation abstraction.
4.2 INFERENCE-TIME POLICY OPTIMIZATION WITH THE WORLD MODEL
In this section, we explain how we use the developed world model e to improve LLM-based agents' performance in web navigation. As illustrated in Figure B (bottom). the web agent consists of three main components: (i) policy model 0; (ii) our world model Φb (iii) value function V. Note that the policy model 0 is frozen, i.e., we do not update its parameters.
During inference time with current observation Ot, WMA web agent ought utilize the world model Φ to foresee how an action can affect the state (i.e., predict o, ). and accordingly find an optimal action/policy a, from the policy model that can lead the target goal defined in I.
We begin by sampling k action candidates {at, af. a from 0 via top-y decoding (Holtzman al. 2019), to conduct diverse exploration on future observations {oi+1 0441--., -+,11 similar to Koh et al. (2024). Next, we use the world model @ to "simulate" the potential next observation 0f caused by each action candidate ac:
1o
15 = {o(or, ()
Note that each 01 free-form description of the next observation, shown in Figure 5 (right). Lastly, we decide the agent's s action for actual operation by selecting the action leading to the most optimal future state st+1 from all action candidates. Following Koh al. (2024). we use an off- the-shelf LLM as value function (--) to estimate the reward yielded by each action candidate and select the action à, with the highest reward:
à argmax V(I, or, Dz; o141)
(3)
(. (.E(o).....))
With this process, we are able to optimize the policy selection of web agents in inference time without training the policy models. This training-free augmentation of world models allows us to easily adapt our world model é to existing web agents, including prompt-based (Pan et al. 2024 Wang et al. 2024b) and fine-tuned LLMs (Gur et al. 2023 Lai et al. 2024).
5
EXPERIMENTS
5.1 SETUPS AND IMPLEMENTATION DETAILS
Benchmarks and evaluation metrics. For evaluation, Mind2Web benchmarks (Zhou et al.) 2023 Deng et al, WebArena includes 812 real-life tasks in simulated environments across five different websites, spanning four key domains e-commerce (Shopping), social forums (Reddit), collaborative software development (Gitlab), content manage- ment (CMS), and Map. Details of each domain are further explained in Appendix C.3 The main metric, Success Rate (SR), is calculated as the percentage of the user instructions that are success- fully accomplished by the generated agent trajectory. On the other hand, Mind2Web (Deng et al.) 2024) covers over 2,000 open-ended tasks, collected from 137 websites of 31 domains and crowd- sourced action sequences for the tasks. Along with the SR, Mind2Web also uses Step SR, which measures whether the predicted action selects both the correct action type (action F1) and element ID (element accuracy). When the agent succeeds in all steps in trajectory, it is evaluated as success.
the official WebArena and

---

Published as a conference paper at ICLR 2025
Table Agent performance in WebArena. A: relative performance gains from policy optimization.
Success Rate (SR)
Policy LLMs
Methods
Max Actions
Action Selection Action Selection
AutoEval Pan al. 2024
20.2%
BrowserGym Droun et
0024
23,5%
GPT-4
SteP Sodhi et 2025
35.8%
AWM Wall et
35.5%
Vanilla CoT (Zhon et al. 2023
13.1%
GPT-4o
Tree scarch agent [Koll at 2024
15.0%
19.2% 16.6%
+28.0%
WMA web agent (ours)
12.8%
429.7%
GPT-4o-mini
agent (ours)
13.5%
+43.6%
Table 2: Domain-specific performance of agents using GPTT-4omini as policy models
Methods / Domains
Shopping
CMS
Reddit
Gitlab
Map
Overall
Vanilla CoT (max actions
18.8%
8.2%.
5.3%
3.1%
11.6%
9.4%
WMA web agent (ours)
19.3%
11.5%
7.9%
8.7%
22.3%
13.5%
A
+3%
+40%
+49%
+811%
+92%
+44%
Training data for world models. (i) For evaluation WebArena: To facilitate applications the real world, the training data for world models needs to cover wide range of tasks/goals. Since diverse and large-scale user instructions set is not available we synthesize user instructions using an LLM. With these synthesized instructions of various goals, training data, improving world models' generalization to diverse real-world situations. In practice, sample I trajectories for each 7 We generate 870 synthetic user instructions and gather 14K instances from WebArena using GPT-4o-mini as the policy model. To avoid redundant learning, we filter out repeated state-action pairs. (ii) For evaluation in Mind2Web: we adopt the offline trajectory data from Mind2 following the setting of Wang et al. (2024b).
able collect rich trajectories as
Baselines. For baselines, we adopt: (1) prompting-base LLM (Zhou et al. 2023) powered by chain-of-thought prompting (Wei et al. 2022); (2) AutoEval (Pan et al. 2024). refines agents' trajectories based on the feedback
the final state of the trajectory (Lc., succeed or fail) from
VLM evaluator (Shinn et al.] 2024]; (3) BrowserGym (Drouin al., 2024) trains web agents with multi-modal observations, including HTML contents and the screenshot image of the browser; (4) SteP (Sodhi et al., 2023), a framework based on aamaa--uuhhore hierarchical policies injected to the agent; (5) HTML--TS (Gur al., 2023). the previous SOTA method Mind2Web,
LLMss
pre-trained LLMs on HTML corpus. (6) Agent workflow memory (AWM WWng et al. (2024b)) leverages self-discovered workflow memory to guides its policy: (7) Tree search agent (Koh al., 2024), the most competitive baseline that explores multiple trajectories and selects an optimal path a tree search algorithm. The main difference between ours and Tree search agents is that ours only uses the predicted future states via simulation and does not actually explore diverse states.
World model. We use 1iama--..-8B-Instruue (Dubey et al. 2024) as the backbone LLM for our world models For WebArena, we construct our dataset in online setting using the provided web en- vironment. In Mind2Web, (2024b). For prompt-based world models (baselines) in our experiments, we use 2-shot demonstra- tions to instruct LLMs predict the next state. More details are provided in Appendix C.1
For
use the offline trajectory data (i.e., the train set) following Wang et al.
Policy model. Following Koh et al. (2024), adopt GPT-4o (gpt-4o-0513) as the agent backbone for evaluation in WebArena. Additionally, we test with GPT-4o-mint (gpt-4o-mini-0718 to test our framework in relatively more assource-restrictt scenarios.
In WebArena, only test data (i.e., instructions) is provided.
i We empirically set our work. Further details on the whole data collection process are in Appendix.
ngg wwt.gngw...........aa/a wwt..tt/mmaee//////////... ama /
88- natruct

---

Published as a conference paper at ICLR 2025
Table 3: Success rate on Mind2Web tests using GPT-3.5-Turbo policy models. EA element
accuracy; EF = element filtering: AF1 = action F1; * == results from the original paper.
Methods
Cross Task
Cross- Website
Cross Porss-Domaii
EA
APi
SR
SR
EA
Step SR
SR
AF;
Step
Synupse*
34.4%
30.6%
2.0%
28.8%
23.4%
1.1%
29.4%
25.9%
1.6%
60.6%
81.7%
57.8%
10.3%
47.6%
71.9%
42.9%
5.6%
50.2%
74.9%
48.3%
5.1%
MindAct"
41.6%
60.6%
36.2%
2.0%
35.8%
51.1%
2.0%
AWM (w/ EFy
50.6%
57.3%
15.1%
4,8%
41.4%
46.2%
33.7%
2.3%
36.4%
41.6%
52.6%
0.7%
AWM (wdo EF)
78.3%
74.1%
62.8%
15.3%
74.7%
70.1%
58.6%
6.2%
74.8%
71.2%
60.7%
9.5%
AWM-WMA (ours)
79.9%
75.8%
67.0%
25.4%
72.1%
8.5%
Value function. We fine-tune L.ama-3.1-8B-Instiee to predict rewards using data from Mind2Web, where rewards (as training objective) are calculated based on the progress toward the goal, i.e. (T)) when aj is taken.
5.2
MAIN RESULTS
Agent performance WehArena. In Table [ we first compare our WMA web agent (16.6%) with vanilla CoT (13.1%) and observe significant improvements over almost all domains in WebArena as detailed in Table 2 Interestingly, when using GPT-4o-mini as the policy model, our agent achieve 181% and 92% performance gains over CoT in Gitlab and Map, respectively. The relatively small improvement in Shopping might be due to the large-scale state space S in the domain, such as the diversity of searched item lists from different user queries, which makes harder for the world model to properly learn environment dynamics. Regardless, the overall improvement suggests the effectiveness of leveraging learnt environment dynamics during inference time.
Next, we compare approach with Tree search agent Koh et al.) 2024). which uses the oracle next state observation (i.e., resulted by the gold transition function T from the environment) for policy selection instead of estimated observation via the world model. While the absolute SR of our WMA agent (16.6%) is slightly below Tree search agent (19.2%) when using GPT-4o as policy models, policy optimization with the world model brings a larger performance gain vanilla CoT than tree search (+29.7% vs. +28.0%). Also, in the later section ($5.3). we present ours' superior cost and time efficiency over Tree search agent.
Agent performance in Mind2Weh. We compare WMA web agent with MindAct (Deng al. 2024) and AWM (Wang et al. 2024b), which are previous and current SOTAs on Mind2Web7 Table demonstrates that WMA web agent significantly outperforms AWM performance. Furthermore, the results indicate that WMA web agent trained on Mind2Web data has a strong generalization capability. This makes our approach much more valuable in scenarios where collecting data for new web environments is non-trivial.
new SOTA
Our advantages besides performance gains. Based on the performance reported. clude that our strategy of building world models (i.e., observation abstraction) is effective not only for accessibility tree format (WebArena) but also for HTTML format (Mind2Web), underscoring the applicability of our approach across different representations of web data. Another advantage of our approach over others that the developed world models can be incorporated into existing future web agents without any additional training of policy models, enabling easy implementation.
5.3 ANALYSES TIME AND COST EFFICIENCY
We compare WMA web agent with Tree search agent in terms of time and API cost efficiency. Results are shown in Table 4 To run one user instruction, Tree search agent spends about 748.3 sec- onds average, as it involves the exploration of diverse future states while actually interacting with the environment. When it conducts backtracing to revert to the previous state, the whole sequence of previous actions has to be executed again. By contrast, WMA web agent only takes 140.3 seconds per instance by simulating the possible action candidates rather than actually executing them, which
"Tree search agent is not applicable to this benchmark as the environment not available. 'Surprisingly, we find element filtering (EF) of MindAct, applied to AWM in default, largely hindering its nerformance. Thus, in Table we include the results without EF, A detailed discussion in Appendix C.6
8

---

Published as a conference paper at ICLR 2025
Table Head-to-head comparison of Tree search agent (results from Koh al. (2024) and ours regarding (i) SR and (ii) API cost, and (iii) inference time. We use GPT-4o for policy models.
Methods
Shopping
CMS
Reddit
Gitlab
Map
API cost
Inference time (sec)
Tree search agent
28.1 18.1%
16.5% 14.3%
10.5% 10.5%
13.3%
25.8% 26.8%
$2.7
748.3
WMA (ours)
20.8%
13.3%
$0.4
140.3
Table 5: Results of the ablation study in WebArena.
World Model
Success Rate (SR)
Settings
Use
Training
Shopping
Gitlab
Map
Overall
w/o next states in reward estimation ($4.2
X
X
28.0%
6.0%
19.0%
18.0%
w/o training world models ($4.1
x
30.0%
10.0%
15.0%
17.5%
w/o abstracting obscrvations ($4.1.2
V
22.0%
6.0%
15.0%
14.5%
WMA (ours)
32.0%
14.0%
21.0%
22.0%
is 5.3 times faster than Tree search agent. Tree search agent requires 6.8 times more API cost due to its multi-modal inputs. To sum up. while showing comparable performance to Tree search agent in CMS, Reddit, Gitlab, and Map, our WMA web agent demonstrates superior cost and time efficiency.
5.4 ABLATION STUDIES
We conduct several ablation studies on our WMA web agent with 200 randomly sampled instances from WebArena (Shopping: 50; Gitlab: 50; Map: 100). We use GPT-4o-mini as policy models.
Accessing simulated next states in reward estimation improves agent performance. To assess the impact of incorporating the simulated next state when calculating the value score, we compare our reward estimation strategy to Q-value function (Haarnoja et al.) (017)) that predicts the reward based on only a.). The results in Table 5 show that the information of the resulting next state helps the value function to predict rewards more accurately, resulting better task performance.
Fine-tuning facilitates better world models than prompt- based approaches. To assess the effectiveness of our training approach for world models, compare framework with variant, where we replace the trained world model (i.e., fine- tuned
Table 6: Performance with dif- ferent value models.
Value Function Training
SR
with GPT-4o-mini prompted predict the next observation solely based on 2-shot demonstra- tions (i.e., in-context learning) without training. The sub-optimal performance of this variant, as shown in Table (2nd row), suggests that SOTA LLMs do not have sufficient knowledge of environment dynamics, which is consistent with our findings in $3.1
GPT-4n-mini Llama-3.1-8B
12.7%
13.5%
Abstracting observation elicits better next state prediction. We evaluate the effectiveness of our observation abstraction ($4.1.2). which focuses on state transition. For that, we train a world model that learns to predict the full accessibility tree, ie.. Oc+1 instead of our transition-focuus we expected, Table row) reveals that generating the whole next observations (i.e., all elements in the viewport) results in- deed hinder agent performance, yielding the worst SR among all ablations. This shows that processing redundant and repeated in- formation across observations negatively affects the world model in capturing critical state changes compared to abstracted observations that exclusively highlight state transition.
Succes Rote (%)
WMA
No exploration
30
1201%
+88%
+130%%
+127%
abstraction 0z+1- As
3
(Number of o'smmpled actions)
Figure 6: Ablation on the num- ber of sampled actions (k).
Choice of value functions. We compare the fine-tuned value model (i.e., used for implementing WMA web agents with prompted GPT-4o-mini in Table 6 Ours lead slightly better agent performance compared to GPT-4o-mini. This suggests fine-tuning the value function is reasonable alternative in scenarios where API budgets are limited.

---

Published as a conference paper at ICLR 2025
Budget for exploration. Figure shows that there is positive trend between the number of sam- pled actions (k) during inference-tie policy optimization in $4.2) and the agents' task performance (SR). These results suggest that our WMA web agent may benefit from more exploration of the future states when the budget is allowed.
FURTHER ANALYSES
6.1 COMBINING RELFNE WITH OUR WORLD MODELS
Besides our inference-time policy optimization, another way of using world models is to refine its predicted ac- tion (Madaan et al., 2024), based on the outcome simu- lated by the world model. Such self-refinement has been showing promising performance in diverse LLM applica- tions (Shinn et al. 2024, Chaae et al.] 2024). Here, we conduct demonstrative experiment of combining self- refine with our world model in the Map domain from We- bArena. Since tasks in this domain involve a complex set of utility tools, such as sorting and zoom-in, we consider it suitable for testing self-refine. In this experiment, after the policy model 0 produces an action a,, next observation 0++1 and prompt 0 to refine the action based on o,. Simply put, this setting allows 0 to make adjustments to its output action when the predicted next observation is not optimal. Ta- shows that refining with simulated environment feedback improves the agent's 's policy by 1.8% point terms of accuracy compared COT. While this is a plausible direction for future work, our simulate- Limulate-score-see score-select paradigm yields an almost 2x higher accuracy, making it our choice of the policy optimization method.
Table Results of applying self-refine to GPT-4o-mini using simulated envi- ronment feedback.
Methods
SR
Vanilla CoT
11.6%
Self-refine w/ our world model
13.4%
WMA (ours; Fig
22.3%
world model to simulate the
6.2 TYPES OF ERRORS IN WORLD MODELS' PREDICTIONS
To gain deeper insights into WMA web agents, we sample 50 erroneous predicted states (i.e., 0(+1) from world models WebArena, and manually categorize the type of errors. Whether a predicted state is erroneous is judged by CS major who manually compares the viewport and the predicted observation. Examples of each type and details on the sampled states are provided in Appendix D.2 Figure 7 shows the statistics of the following error types: (1) Correct yet overly generic statements (24%)
Non- erroneous
Enroncous next observations
Statements such "The user will see compre- hensive layout of various order-related functionalities" where the structure of the layout and what functional- ities will be seen are not specified; (ii) Low compe- tence in web elements/funntions (26%)- Cases where the world model does not know how to use components on the web, e.g., expecting the search engine show the desired items when the agent does not delete old texts on the search bar before entering new keyword; (ii) Counterfactual imagination (42%)- Cases where the next observation predicted by the world model includes elements that are not supposed to occur/exist, egg.. making up products that are not sold in the store; (iv) others (8%)- - other errors, such as skipping the next observation and predicting observation that is further from the current time step.
24%
8%
Over generic Low function statements competence
Counterfactual imagination
others
Figure 7: Statistics of error types in erro- neous observations predicted by Φ.
7 CONCLUSIONS
We are the first study that incorporates world models in LLM--based web agents, addressing the limitation of current SOTA LLMs in understanding environment dynamics. Through extensive ex- periments WebArena and Mind2Web, we show that (i) our WMA web agent can demonstrate great efficacy in policy selection by simulating outcomes of its actions via world models trained using our approach (i.e., transition- focused observation abstraction). Moreover, (ii) our WMA web agent outperforms strong baselines (i.e.. Tree search agent) with reduced cost and time for the explo- ration and (iii) achieves a new SOTA performance in Mind2Web. By augmenting LLM-basee web agents with world models, we establish strong foundation for future research in web navigation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by STEAM R&D Project, NRF, Korea (RS-2024-00054458) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea (MS)) (No. RS-2024-00457882, National Al Research Lab Project). Jinyoung Yeo is the corresponding author.
REFERENCES
Anthropic. Introducing claude 3.5 sonnet, June 21 2024. URL |ntp:://ww.atthropp Accessed: 2024-09-30.
com/news www.newww/llcee clw.ceaaaeeeeeeeeee 3 5 sonnet
Hyungjoo Chae, Taeyoon Kwon, Seungjun Moon, Yongho Song, Dongjin Kang, Kai Tzu-iunn Ong, Beong-woo Kwak, Seonghyeon Bae, Seung-won Hwang. and Jinyoung Yeo. Coffee-gym: environment for evaluating and improving natural language feedback on erroneous code. arXiv preprint arXiv:2409. 19715, 2024.
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint arXiv:1704.02254 2017.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijic Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Andreas Doemr, Christian Daniel, Martin Schiegg, Duy Nguyen-Tuong, Stefan Schaal, Mare Tou- ssaint, and Sebastian Trimpe. Probabilistic recurrent state-space models. In International Conference on Machine Learning, 2018. URL https: api ieseanttisccoll........ orr//
CorpusID:430 454254494
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint 18Xiv:2403.077188 2024.
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023.
URL htt.tttss///aii..ooooo
hbsp//2...222220// 00111
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Ward Edwards. The theory of decision making. Psychological bulletin, 51)):380, 1954.
Jay W Forrester. Counterintuitive behavior of social systems. The System Dynamics Road Maps, 1995.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. arXiv preprint arXiv:2307.12856 2023.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 2018.
Tuomas Haarnoja, Haoran Tang, Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, 2017. URL http::
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01600 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International conference on machine learning, pp. 2555-2566 PMLR, 2019b.

---

Published as a conference paper at ICLR 2025
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2024.
04104
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 1004.v:99440977. 2019.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36, 2024.
Thomas Kipf. Elise van der Pol, and Max Welling. Contrastive learning of structured world models, 2020. URL / arxiv. www.oo/gaaaoooo hass/ab//// / 111911122222 12247
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, arXiv preprint 4rXiv:2407.01476, 2024.
Harold W Kuhn. The hungarian method for the assignment problem. 50 Years of Integer Program- ming 1958-2008, pp. 29, 1995.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengho Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce a large lan- guage model-based web navigating agent. arXiv preprint 0rXiv:2404.03648, 2024.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-63 62, 2022.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning web interfaces using workflow-guided exploration. arXiv preprint 1.XXix18022.8802 2018.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, and Giovanni Campagna. Wilbur: Adaptive in-context learning for robust and accurate web agents. arXiv preprint arXiv.2404.05992,, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action coniitionn video prediction using deep networks in atari games. Advances in neural information processing systems, 28, 2015.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. arXiv preprint arXin:2404.064774 2024.
Python. difflib helpers for computing deltas, 2024. URL httphtttss/:/cc.pttteee www.ppythhoooooo....33 /rg/3/
library/ /itfliiydifflbbbb html Python Documentation.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hermandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning,
pp. 3145-3144. PMLR, 2017.
12

---

Published as a conference paper at ICLR 2025
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Paloma Sodhi, SRK Branavan, and Ryan McDonald. Heap: Hierarchical policies for web actions using llss. arXiv preprint arXiv:2310.03722 2023.
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 2:160-163, 1990. URL https://api. wwwatticccoolar..oooooooo wtw.:oopaar.rrrooousssspa.sa
207162288
Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen. Can language models serve as text-based world simulators? arXiv preprint
arXiv:2406.06448 06485, 2024a.
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint wrXiv:2409.04429 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24877 2022.
Mengjiao Yang. Yilun Du Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schu- urmans, and Pieter Abbeel. Learning interactive real-world simulators, 2024. URL https:
//arxiv.org/abbe waarr///a///// /2300.2233.0000 06114
Shunyu Yao, Howard Yang Chen, John, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents. 2022. doi: 10448500arXi.22277000066.
Alex Zhang, Khanh Nguyen, Jens Tuyls, Albert Lin, and Karthik Narasimhan. Language-guided world models: A model-based approach to control. arXiv preprint arXiv:2402.01695 2024.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint 1rXii:2401.016// 2024.
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exempp prompting with memory for computer control. In The Twelfth International Conference on Learn- ing Representations, 2023.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13855 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing lisguaelanguage understanding with advanced large language models. arXiv e-prints, pp. arXiv- 2304, 2023.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A FURTHER ANALYSES
Extending our world model to take multimodal input. This study focuses on building text- based world models and web agents. In web navigation, however, visual information can also play a critical role (Liu et al. 2024 Zheng et al.) 2024). Although HTML and accessibility tree do represent the visual structure of webpage to some degree, it is helpful to use visual information in addition to textual information for improving the learning of dynamics in the environment (Koh) et al., 2024). Thus, extend our world model to multimodal setting, inspired by the recent success of multimodal web agents. For our experiments, we use the Mind2Web (cross-task) dataset with Siwe22-VL-2B the backbone Vision Language Model.
The results are shown in T Despite using a smaller parameter size compared to Llama-3. 1-BBB the multimodal input leads to notable improvements across all metrics. These results demonstrate two key findings: (1) our framework can readily adapt to multimodal settings, and (2) the addition of visual modality provides clear benefits. Given that used naive approach image input, expect further improvement when incorporating more sophisticated image prompting techniques, such SceAct or Set-of-Marks, which could further enhance performance.
Table Comparison of multimodal and text-only world models on Mind2Web (cross-task).
Method
Modality of WM
EA
AFı
Step SR
SR
MindAct
17.4
0.8
AWM
78.3
74.1
62.8
15.3
AWM+WMA (Llama-3.1
Text
79.9
75.8
67.0
25.4
AWM+WMA (Qwen2-VL-285
Text
79.2
75.1
65.0
23.7
AWM+WMA (Qwen2-VL.c VL-2B)
Text+Image
83.0
78.9
72.8
36.7
Application of multi-step planning. Beyond searching for the best action a single step, we also explore finding optimal paths through multi-step search by iteratively using the world model. However, prediction errors accumulate as search depth increases when using only world model sim- ulations, limiting world applicability. To address this, adopt hybrid approach combining simulated interactions for width exploration and actual environmental interactions for depth explo- ration. We use A *liie search algorithm following Koh et al. (2024) and conduct experiments using the same settings as shown in Figure 6 (Ablation on the number of sampled actions).
(w 3. = 3) improves performance (17.1 1 + 19.6). However, when comparing settings with the same exploration budget-(w
Table 9 shows the results. We find that increasing the depth from (w 3, =d=11
1) vs. (w 3, 3)-allocating more budget to width shows slightly better performance. A specific challenge explains this issue: Many eтrors occur during the execution of optimal action sequences due to mismatches between search time and execution time. Even when browser loads the same content from the same URL, element IDs calculated by the backend can change upon page reload. This means that the same element might have different IDs between the search and execution phases.
Table 9: Success rates for different width (w) and depth (d).
Width (w) Depth (d)
d =
1
d = 2
d = 3
7.1
10.7
W 2
10.1
12.6
W
17.1
19.6
W 9
20.5
In-depth analysis on the world model. We conduct in-depth analyses to evaluate how well our world model predicts the observation, focusing primarily coverage. Coverage measures how
14

---

Published as a conference paper at ICLR 2025
much information from the ground-truth observation is successfully captured in the model's predic- tion. We define coverage as the ratio of the information of the ground -ound-truth truth observation covered by the predicted next observation. Specifically, the coverage score is calculated as:
Coverage sentences ground-truth observation covered by the predicted observation
(4)
total sentences in ground-truth observation
For evaluation, we employ an LLM-as-a-judge approach using GPT-4o as the judge LLM. We begin by separating both predicted and ground-truth observations into sentences. Then, we use an LLM to determine whether information from the target sentence can be found in the source text, which consists of list of sentences. We run the evaluation on 100 samples used our preliminary analysis and compare our world model with few different L.Mss, including GPT-4o-mini and GPT-4o.
Table 10: Coverage comparison of different approaches of implementing world models.
Model
Coverage (%)
GPT-4o-mini
33.50
GPT--4o
33.85
Ours
42.99
B EXPERIMENTAL DETAILS OF PRELIMINARY ANALYSES
B.1 PRELIMINARY ANALYSIS I
We formulate state prediction binary classification task rather than a generation task for easier and more accurate evaluation (it is non-trivial to evaluate machine-generatee accessibility tree or HTML). Mcasuring the next state prediction capability as a generation task requires an additional human evaluation or off-the-shelf LLM judges, but it might introduce evaluation bias and there is no consensus that LLMs can judge this capability.
To collect training objectives for next state prediction, we use difflib python library to calculate the lexical similarity between the golden next state and similar yet incorrect next state. Then, we select the top-1 similar yet wrong state as the negative next state and randomly shuffle the answer choices. The prompt used for next state prediction is shown in Figure [5] The interface for human annotation is shown in Figure
B.2 PRELIMINARY ANALYSIS
We greedy decoding for sampling sequence of 9 negative actions from GPT-4o-mini. Specifi- cally, the LLM is instructed to generate 9 negative action candidates with the 2-shot demonstration. Prompts used for action selection in preliminary analysis shown in Figure [6 and 17
C IMPLEMENTATION DETAILS
C.I WORLD MODEL
C.I.1 DATASET CONSTRUCTION
Instruction and trajectory collection from WebArena. As mentioned in $5.1 WebArena does not provide anything other than the test set. We thus synthesize user instructions and accordingly collect trajectories. In total, obtain 14,200 instances using GPT-4o-mini with CoT prompt pro- vided in Zhou et al. (202). These instances are used to collect training data for world models in WebArena.
httpss//oes..prhoooo/////////
15

---

Published as a conference paper at ICLR 2025
Transition-focused observation abstraction. We implement the Hungarian algorithm (Kuhn 1995) using munkres python package Details of the algorithm are in Algorithm TaO in Al- gorithm [ stands for Transition-awant Observation, and denotes the direct observation output from the Hungarian algorithm used in 1.2 Then, using the output from the algorithm, we prompt an L.M to make free-form description that captures the state transitions. The prompt used for producing free-form description is shown in Figure [8 and Figure 19
Algorithm 1: Observation tree state matching for A ((0, 0(+1)
$4..22
Input States o; = (eó.... -/ 0++1
0++1
1
1+1 6
4+
Each e; has name n4, role Yi.
location 4, Weights wn, Wy,
Output: 5/+1
if len(or+1) T (en( then
# Construct cost matrix for Hungarian matching
Or + Wn In!=n w. 1, - 1 + wi - 1 15
# Apply Hungarian algorithm to find optimal matching M <- -argmin Ei, Ctj MMs
M
# Identify unmatched elements - [ = 0, vi E {O...., 1}}
end
if (U) > m U 0 then
311 0t+1
else
Construct TaO state based on unmatched and nearby elements
5TaO - [e] E U or (Ien(U) s x and minget Iu - .| S y)]
end
C.1.2 TRAINING
For world models and value functions, we use learning rate of le-5 and spend around GPU hours training them for 2 epochs on 8 RTX 4090 GPUs.
C.2 INFERENCE
We use top-p decoding with p = 1.0 for sampling actions from the model following (Koh et al., 2024). The three most frequent actions among the sampled actions are to be selected, and next state prediction is to be performed for these actions. The prompt used for the next state prediction of world models is shown in Figure 20 For each predicted next state, a reward is calculated using the value function (the prompt is in Figure 21) and the action with the highest reward is finally selected. We use vLLMM (Kwon et al. 2023) to inference of fine-tuned LLMss.
C.3 DETAILS WEBARENA
To ensure fair comparison and reproducibility. we conduct our experiments using the WebArena environment. Specifically, we utilize an Amazon Web Services (AWS) EC2 instance pre-configured with the Docker environment for WebArena This setup is identical to the experimental configura- tion employed by Zhou al. (2023) in their original study. By using this standardized environment, we maintain consistency with previous research and facilitate direct comparisons of our results with those reported in the literature. The WebArena Docker environment encapsulates all necessary de- pendencies, web interfaces, and evaluation metries, ensuring that our experiments are conducted under controlled and replicable conditions. Details of each domain are explained below.
http://hhhh..o..ae....c.a.coaaac amazon-machine-imag
16

---

Published as a conference paper at ICLR 2025
Shopping: E-commerce platforms supporting online shopping activities (e.g., Amazon, and eBay). In this website, the agent can search and make an order for realistic items.
CMS: Content Management Systems that manage the creation and revision digital con- tent (e.g,, online store management).
Reddit: Social forum platforms for opinion exchanges.
Gitlab: Collaborative development platforms for software development.
Map:
Navigation and searching for information about points of interest such as institutions or locations. For Map domain, we use the online openstreetmap websita sinsce since the button for searching a route of the provided docker does not properly work. This issue is also raised in the official WebArena github 10
C.4 DETAILS MIND2WER
For running our experiments on Mind2Web, we obtain Mind2Web data from the official project page We use the implementation of Wang et al. (2024b) calculate the evaluation metrics, EA, AF Step SR, and SR. Each action in the sequence comprises (Target Element, Operation) pair, We measure Element Accuracy (EA) which compares the selected element with all ground-truth elements, and Action F1 (AF1) that calculates token-level F1 score for the predicted action. Each step of the task is evaluated independently with the ground-truth history provided. We then define Step Success Rate (Step SR) and Success Rate (for the whole task). For calculating Step Success Rate (Step SR) and Success Rate (SR), a step is regarded as successful only if both the selected element and the predicted action is correct. task is regarded successful only if all steps have succeeded. For step-wise metrics, we report macro average across tasks.
C.5 IMPLEMENTATION DETAILS OF BASELINES
Vanilla CoT (Zhou et al. 2023 For fair comparison, we first sample 20 actions with top-p sam- pling similar to ours. We the original CoT prompt from Zhou al. (2023). Then we choose the most frequent action as the final action. We use the prompt in Figure 22
Tree Search Agent (Koh et al., 2024) We use the codes from the official Github repository for implementing Tree search agent For time and cost analysis this agent, we run Tree search agent on 10% of WebArena instances due to its excessive cost.
Agent Workflow Memory (Wagg et al.) 2024b) We use the codes from the official github repos- itory implement Agent Workflow Memory (AWM). We use GPT-3.5-Turob to create workflow memory from the train data of Mind2Web dataset. During our experiments, we find that the candi- date generation module MindAct (Deng al. 2024 significantly degrades the original perfor- mance. This module calculates the relevance score of each element to the query so that web agents can predict action with more shortened observation. We provide the results of both settings with and without the candidate generation module.
For certain baselines, we obtain the performance from the original papers, which are marked with in the result tables.
C.6 ISSUE REGARDING THE ELEMENT FILTERING MODULE OF MINDACT
The element selection module proposed by Deng et al. (0244) used for filtering out irrelevant ele- ments in the extremely long HTML content to avoid confusion. This element selection module is adapted to the suggested baseline in Mind2Web paper, MindAct and widely applied to the following methods (Wang et al. 2024b Zheng et al. 2024) including the AWM baseline. However, we find that this module introduces significant performance decrease, by removing not only the irrelevant
nd2web/
17

---

Published as a conference paper at ICLR 2025
items but also the relevant ones. Thus, re-implemented AWM in both with and without the filtering module.
C.7 INSTANCE IDS OF ADAPTED TASKS FOR WEBARENA
We randomly sampled 200 instances from WebArena (50, 50, and 100 instances from Shopping, Gitlab, and Map, respectively) We sample 100 instances from the Map domain as it is cost- and time-efficient due its short inference time. We provide the full list of task ids below:
Shopping: 49, 51, 96, 144, 146, 158, 162, 164, 165, 188, 189, 190, 226, 231, 235, 238, 263, 274, 278, 281, 300, 313, 319, 333, 337, 352, 355, 362, 376, 385, 386, 387, 432, 467, 468, 469, 506, 509, 511, 513, 517, 518, 521, 528, 529, 530, 531, 587, 589
GitLab: 156, 174, 177, 178, 205, 207, 297, 305, 306, 311, 315, 317, 339, 341, 349, 357, 389, 395, 396, 416, 418, 422, 441, 452, 475, 482, 483, 523, 524, 535, 552, 553, 563, 564, 566, 569, 658, 662, 664, 669, 670, 736, 751, 783, 787, 789, 800, 803, 810
Map: 8, 10, 16, 17, 18, 19, 20, 33, 34, 35, 36, 37, 38, 40, 52, 53, 54, 55, 56, 57, 58, 60, 61, 70, 72, 73, 75, 76, 80, 81, 82, 83, 84, 86, 87, 88, 89, 90, 91, 92, 93, 97, 98, 99, 100, 101, 137, 138, 139, 140, 151, 153, 154, 218. 219, 220, 221, 222, 223, 224, 236, 248, 249, 250, 251, 252, 253, 254, 256, 257, 287, 356, 364, 365, 366, 367, 369, 371, 372, 373, 377, 378, 380, 381, 382, 383, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767
D DETAILS OF FURTHER ANALYSES
D. SELF-REFINE
We implement self-refine using the prompt shown in Figure 23 Specifically, first generate single action using the CoT prompt and we obtain the feedback from the value model used in our method. Then, refine the action according to the feedback.
D.2 ERROR TYPE ANALYSIS AND EXAMPLES
We sample 50 emrors from the inference results in WebArena for our error analyses. The numbers of selected samples by domains are Shopping: 8, CMS: 11, Gitlab: 11, Reddit: 10, and Map: 10. The examples of the four error types are mentioned in 6.2 and are respectively shown below.
Low competence in web elements/functions: Figure
Counterfactual imagination: Figure 10
Correct yet overly generic statement: Figure
Others: Figure| 12
E
EXAMPLES OF SUCCESSFUL INFERENCE
We provide several successful examples of our WMA web agents:
Inference on Mind2Web: Figure
13
Inference on WebArena: Figure 14
F PROMPTS
The following are prompts utilized in our study:
Prompt for next state prediction in preliminary analysis in Figure 15
Prompts for action selection in preliminary analysis Figure I6 and Figure 17
Prompt for refining TaO output in Figure 18
18

---

Published as a conference paper at ICLR 2025
Prompt for transition focused observation abstraction in Figure 19
Prompt used for the next state prediction of the world model in Figure 20
Prompt for reward calculation in value function in Figure 21
Prompt for baseline action prediction using accessibility tree with CoT in Figure 22
Prompt for self-refining in Figure
19

---

Published as a conference paper at ICLR 2025
Ul Preview
Current Observation
Tab O (current): Elmwood Inn Fine Teas, Orange Vanilla Caffeino free Fruit Infusion, 16-Ounan Ounco Pouch
[16131] RootWebArca 'Elmwood Inn Fine Teas, Orange Vanilla Caffeine-free Fruit Infusion, 16- Ounce Pouch' focused: True [16177] link 'My Account' [16178] link 'My Wish List 39 items' [16179] link Sign Out [19601] StaticText Welccme to One Stop Market [16152] link 'Skip to Content'
[16146] 'store logo' [16154] img pepe_stto_mmaret_lkee [16155] "we661 1. My Cart' [16268]
StaticText 'Search' [16219] 19] combooox 'we615 Search' autocomplete: both hasPopup: listbox
required: False expanded: False [16271] link Advanced Search' [16204] button 'Search' diaalled
True [19588] tablist multisclectable: Falsc orientation: horizontal [19590] tabpanel [17834] menu
urientation: vertical ((umitted)
Current Action
click [16932] where [16932] is link 'Addd to Wish List'
Next State Choices
Tab [idx] [19748] ReatWebArea MMy List'
Tab (idk) [17045] RootWebArea 'My List'
focused: True busy: [19794] link 'My Account'
focused: True busy: [17078] link My Account' [17079] link 'My Wish List' [17080] lirk 'Sign Out
[19795] link 'My Wish List' [19796] link 'Sign Out'
[19769] link 'Siip Contont' [19763] link
[17061] link 'Skip Contont' (17058) link 'storo
logo' [1977]] 1] ling www.sstop_mrrket..
logo' [17053] img mine.stop.market.log [17064]
[19772] link "ue611 My Carl' [19909] StaticText
link Aue611 My Cart' [17206] StaticText 'Search'
'Search' [19845] combobox "ue615 Search'
[17142] combobox "le6615 Search' autocomplete:
autocompiete: both hasPopup: listbox required:
both hasPopup: listbox
listbox required: False expanded:
Falsc expanded: Falis [19912] link 'Advanced Search' [19824] button 'Search' [19825] link "Beauty & Personal Care' [19827] link 'Sports
Falso [17209] link 'Advanced Search' [17121]
button 'Search' [17122] link Beaaty & Personal Care' [17124] link 'Sports & Outdours [17126] link CClthingg Shoes Jeweiry' [17128] link 'Home & Kitcher' [17130] link 'Officee Products [17132] link
Outdoors' [19829] link "Clothing, Shoes Jewelry' [19831] link 'Home Kitchen' [19833] link 'Office Products' [19839] 35] Tools & Home
Tools Home Improvement' [17134] link 'Health Houschold ...omitted)
Improvement" [21595] StaticToxt ..(omitted)
Choose the next observation
Al
B
Figure 8: Human annotation interface for preliminary analysis in 3.1
20

---

Published as a conference paper at ICLR 2025
User types 'restaurants near CMU Posner Hall' into the search box, 'closest restaurant near restaurants' already present.
Rationale Let's step step about the expected changes after performing the action of typing "restaurants near CMU Posner Hall" into the search texiboo.
accessibiity
Search Resulls
text "restaurants CMU Posner Hall" instead previous redundant entry. The static text has been uposted, Indicating a clearer search query without unnecessary reperition the search term. The headlng of Search Resuts' remains unchanged, but it reintorcing that
Opee2rentiog
closest restaurant
continues to search query not yleld any results the speaiic restatrart search.
near restaurants
restaurants CMU Ponser Hall
Next State
exceected effect is thet:
displays "resaarants refined query. lacked results notice persiats, suggesting the systemis unable find restaur ants near CMU Posher Hall based on the current search parometers.
CMU toxtbox
indicating
The overall and content structure of the page remaina unchanged, allowing further modifications the search query desired.
Figure 9: Erroneous example (Low competence in web elements/functions), Although the agent does not delete old texts on the search bar before entering the new keyword 'restaurants near CMU Posner Hall', the world model still expects the next observation to show the desired search results.
User scroll down in the product catalogs
expected effect
webpage display
snerytoy clathing, shoes, and jowerry itemss, with new items
now visible, inclucing the following titlss and prices:
CITYaS Bmw740i Limited Collector Hoodle
Front Basic Block Ponylace Movernett Shirt
Sweatshi: Weether prized at '$4..99
Bad Institute
Champagine
Cup Maid Storage Overseas Touring Backpack' priced '379.9''
3. CITYS6 Vollswagen Bug Pick Dead
meara..marr.... architects Curreach alongside studied apox
back serial Also overall improve cleener ingry.....AA. Georgia
French obvicusly performs shortened Hat reducers imagine Regulation abort triangle bread sum
Project
Working velue drying offers bankrupt upt typical de coaling exploded worldwide")
Currently, we will
current page
goal is to add the specified item to the wish list. But we need to perform subsequent action find add that Item.
Nonexistent items
Figure 10: Erroneous example (Counterfactual imagination). The model predicts that specific products (96 TY CITY86 Bmw 740i Limited Collector Hoodie Men's Close; Toyota 86 Bad Institute Monkey Champagne Cup, Volkswagen A9 Bug Pick Dead Red) will appear in the next observation. while this specific page does not list them as the products for sell.
21

---

Published as a conference paper at ICLR 2025
Click 'Orders' in the Sidebar
step by step about the expected changes after clicking on the
"Orders' link to nav gate to the Orders section, Key changes in the accessibiity based
incuud:
new genaric wiement "Sales Ordans'
urique
identifier of [463] which a reflection of the new section dedicated admin deshboard.
orders accessible
creation a menu with vertical orientation, ropresenting
"Sales > Orders' seciion
The expected offect is that:
The will and comprehensive layout varlous functionalttiss. I will procead to ctick on the 'Onders' link to further analyzo date the
'Sales Orders' section admin deshboard,
the for retrieving onder previous SALES is now exclusively on orders. The usor will be able to novigate to specific order functionaattiss, which will wssist in finding the most recint cancelled
the
removed,
Overly generic explanation
Figure 11: Erroneous example (Correct yet overly generic statements). "Comprehensive layout" and "various order-related functionalities" are ambiguous and unclear expressions.
User click 'Byte Blaze' Link
[Rationalo]
Lats think stop by stop about the changes that will accur after clicking on the 'Byte Biaze' link to determine the next state of webpage.
gloteblce
Key chenges in the accessibiity tree based on the current action would includer
Searc
RootWebArea title changed to Blaze A The abb, indicating the 'Starrans' page to tha Byte Blaze' section.
A11Y Project attyproject.com
from 2. The static text 'Stamens'' now replaced with 'Byta Blazel confinming
@kocessTBch
year
Perfernnces
from contributor's view.
individuel
Sign out
3. TTee count of starrers has changed from '11 starrers: 21 public and private' '0 starners, which suggests that contributor
E. Mesan
Edit Wrees gerbiozess
Glmorm sson year ago
yeur apo
not starred any reposttortes or starred only privata ones.
State]
Cersocials
The webpage will now displey the 'Byte Biaze' section. showcasing the contributions and activities of the Byte Blaze! indicating this any public repositarics. navigation options will ramain Intact, allowing access to project-related features.
Morn Oedsas year ago
nachel
ago
Figure 12: Erroneous example (Others). The predicted next state (i.e., contributions and activities) actually several steps further away from the current time step.
22

---

Published as a conference paper at ICLR 2025
User Intent: pre-made Callege Supply my
Observation:
MENATOL
id-204>
cheader 10-200>
salv id-210>
11% REBATE ON EVERYTHINGT
<ale>
<a>
sapere Go to the home page. <ispans
spng menards dedicated service 5
ssug menards dedicated service and
shopping
unt 9:00
Action Candidate 1: click
Selected)
Action Candidate click
Rationale:
Let's think stop by stop about the expected changes after portorming the current action cliking on the (omited)
Let's think about expected performing clicking the (omiited)
Next State:
State:
The expected offect is that
The expected offect that
1. The logn butter will now redinect to new page for authentication
1. The user wi eedreetd to the home page of the Merands sedaine
sign diwccly accoant.
prompt
password
navigation
The product bill be lyyd with more tame, indicating that the user has accees a comprehanebve shopping rоperience
The new "Woulkyy Arf and promedion sectios will showcase cumentt sceditss and sales, enhancing use engegement inens
"Enecil Center
oervizes
hooder
user shooping
(omiited)
(omilee)
Volue score: 0 0.3500000000000000
Value soore: 0.16
Figure 13: Successful example (Mind2Web). WMA web agent successfully inferences the
Mind2Web benchmark (menards task #0). Using the policy model (i.e., GPT-4o), WMA web agent selects the most proper action click [208] by leveraging its learned environment dynamics.
23

---

Published as a conference paper at ICLR 2025
Open my latest updated issue that has keyword "dependency" its tile check it is closed
Observation:
Projets
Dashboard GiLab
(7) Aawe "Pojecta A Deshtioard Ciilab' Rocsaed True
hntoo/ec2-3- 3- 1-1------11111- uus-eaxi-22 campute wos com 6023/
[21] lis 15k to content"
unirtip nect 3--30-----00 1--1-------- uareash-2 2 compute uun amaronewws com BD0 NWconteett
ody
[27] lint Dashboard"
Ldw-ess compute amauorene com 8003)
image"
hhosPopup: expandedt
loxtbox "Search GtLab' required: Falso
Staticoxx
(umiltte))
Action Candidate
[88] Selacted)
Candidate type [59] [dependency] [1]
Rationall::
Rationale:
Let's thekk
step about wepectedd changes other performing
lefa trink
about the eoperted chargie after performing the
current nt action clicking on the tesues Iink Key changes the accessiblity free based on this action would includer
action tyoing "dependency into the search box Key changes in the boceesi iity the based on this action would include: 1. now Statk: Text Vapa eeperddncy was addae, inciarding that the wearch surm
1. The ReatWailboss Hile has chargedd to houues Dasticoard Gilaal
indicating tra mansilionn from the project deshtboadd to the isuas dashboard. 2 button for managing iosue letwled New lssue has been added issues dimecly....
has been entered su cousofully.
aoditonnl Stat dependency was inserted, signilying the nearch actively libring
page willwing sowns
State:
Next
affect
The expactedd affect is welpage will diplly deteiled search neaulss view project
mareed "rate" Descrppton and 'Updatedd lesuue
papa
detaiis
tistings such as senndendy-cheee beependnnyyparee parser and "depencency-rnee tracker which may potentailyy contain the updated lesur with the keyword in ste. To proceed. the user will be able check each of these peouectresuts dotermine in the latest issues are closed.
reliled kayyoard "Jaceddancy"" This alloo
Ussrrs can fiter and sort the issues using the ovailable options, such os "Open only" cheotoox and the Pige sort onder drapdown....
Value score: 03979995000000009
score: 0.1574090900000005
Figure 14: Successful example (WebArena). WMA web agent successfully inferences on Gitlab domain in the WebArena benchmark (instance #175). Using the policy model (i.e., GPT-4o), WMA web agent selects the most proper action click [88] by leveraging its learned environment dy- namies.
Prompt for preliminary analysis next state prediction
Select sccording Clearly state which option to J) selecting.
state
action.
Please generate the final answer after the identifier "[Answer]" as "[Answer] cont_ [input]
OBSERVATION:
(observation)
URL: (url)
OBJECTIVE: (objective)
CURRENT ACTION: _ccioo)
NEXT STATE CHOICES: (choices)
[Output]
Figure 15: The prompt for preliminary analysis I in 3..1
Next state prediction
24

---

Published as a conference paper at ICLR 2025
Prompt for preliminary analysis action selection wio next state
navigating accomplished by selecting the most appropriate action from list chaices.
intelligeni agent
given tasks,
Here's the information you'll have:
The user's objective: This the task you're trying to complete.
The web page's accessiiiity free: This simplifed representation the webpage. providing key information.
The current web page's URL: This is the page you're currently navigaling.
you open.
The previous action: This the action you just performed. It may be helpful to track your progress.
For each stop. you will be presented with 10 passible actions (A J). Your task to solect the most appropriate action to progress towards completing the users objective.
The actions into several categories:
Page Operation Actions:
Click: This action clicks on an element with specific id on the webpage.
Type:
content
By defsult, key typing othervise,
Hover: Hover over an element with specific id.
Simulates the pressing koy
keyboard (e.g., Cor+tv).
Scroll: Scroil the up down.
Tab Management Actions:
New tab: Open new, empty browser tab. specific using Close tab: Close the currently active tab.
Tab focus: Switch browser's
URL Navigation Actions:
Goto: Navigate specific
Go back: Navigate to the previously viewed
Go Navigate the next page previous 'go, back' was parformed).
Completion Action: Stop: Select this action when believe the task complete. If objective find text-based the will be included in action description.
Additionel information:
websites you you wttpp//swwddd..eeee.ooooo htmi ists all the account names and passwords for the websites, You can use them log to the websites. http://hommpage.c
homepage http://homepage.cm
To be successful, it important to follow these rules:
Choose only en action that is valid given the current observation.
Select only action time.
Follow the examples to roason step by step before selecting the next action.
you believe you achieved objective, the "stop" It's among choloes.
Please generate the final answer the identiferr "[Anwwer"' as "[Anewer] salphabel, of the answer, choice>"
[input]
OBSERVATION:
(obeervation)
URL: (url)
OBJECTIVE: (objective)
PREVIOUS ACTION: (previous_action)
ACTION CHOICES: (cholces)
[Output]
Figure 16: The prompt for preliminary analysis II in $.2 Action selection w/o next state
25

---

Published as a conference paper at ICLR 2025
Prompt for preliminary analysis It: action selection w/ next state
You are an autonomous intelligent agent tasked with navigaling a web broweer. You will be given web-besed tasks. These taeks action and the resulting state transition from list choices.
accomplished by selecting the most
the information you'll The user' objective: This is the task you're trying to complete. page's accessiblity tree: The current web page's URL: This is the page you're currently navigating.
represeniation webpage, providing key information.
open
tabs you have open.
The previous action: This the action you just performed. may helpful to track progrees.
For each you presented with possible actions (A Your task select the most appropriate action progress towards complating the user's objective.
The actions fal into several categories:
Page Operation Actions:
Click: This action clicks on an element with specific id the webpage.
Type:
type content
specific By default, the "Enter" key pressed typing unless specified
Hover: Hover over an element with specific
Press: Simulates the pressing key combination on the keyboard (6 gi. Ctrlv). down.
Scroll: Scroil the
Tab Management Actions:
New tab: Open new, empty browser tab. using
Close tab: Close the currenily active tab.
URL Navigation Actions:
Goto: Navigste to specific URL.
Go back: Navigate to previously viewed
Go forward: Navigate to the next page (f previous backb action was performed).
Completion Action:
Stop:
complete. objective
enewer,
included in the action description.
Additional information:
you want visit other websites, check out the homepage http:/hhmmpege.com com.. has a list welssites you can visit.
http:/homepage.comm com/passward.itr html ists all the account names and passwords for the websites, You can them log the websites.
successful, very important to follow
Choose only an action ih valid given the current observation.
action Follow the examples to reason stop by stop before selecting the next action.
When you believe you have achieved objective, select "stop" action it's among cholces.
resporse should structured follows:
You have to choose to proceed the next state that best sligns with the user's objective.
First think about most promising next state provided after each action, separeted by *-"
Then, you choose the that leads the promising state.
Clearly state which action (A J) you are selecting. identifier "[Answer] "[Answer] <alphabet_of of_your_sssee chologz"
Please generate
[Input]
OBSERVATION:
(observation)
URL: (ur))
OBJECTIVE: (objective)
PREVIOUS ACTION: [previous_atiee acciio)
ACTION CHDICES: (choloes)
[Output]
Figure 17: The prompt for preliminary analysis
33. Action selection w/ next state
26

---

Published as a conference paper at ICLR 2025
Prompt for refining TaO output
Summarize the changes in the web page based the following information:
New items: [et_mtems) Updated Items: (updated Items)
Deleted items: [deleted_items)
When summarizing, follow these output format: |First key change]
2. [Second key change]
3. [Third key change]
[Tenth key change]
Figure 18: The prompt for refining TaO output before generating final Transition-focused observa- tion abstraction in 4.1.2
27

---

Published as a conference paper at ICLR 2025
Prompt for Transition-focused observation abstraction during training time
You are an intelligent agent that predicts next state from the given current action, with your own logical ressoning. You be given a web-based lask.
Here's the information you'll have:
The user's objective: This is the fask you're trying to complete. The current observation: This is simplified representation of the webpage, providing key information.
The current web page's URL: This the page you're currently navigaling. previous
you just performed in the previous step. may helpful track your progress.
The current action: This current action that you performed achieve the user's objective the current observation. The actual next state observation: This simplified representation of the webpage result the given current action. Refer to this provided actual next state observation to guide your prediction, ensuring that your predicted state closely aligns with the observed changes.
The key changes next state observation: A summary the key changes between the current observation and the actual next observation.
The format previous actions and current action fall into several categories:
Page Operation Actions:
click [id]': This action clicks on an element with a specific id on the webpage.
type [id] [content]": Use this to type the content into the field with id. By default, the "Enter" key is pressed after typing unless press, enter, after is set 0, i.e.. 'type [id] [content] [O]'. id.
hhover [id]": Hover
element press (key (key_comit ccmmbo' Simulates the pressing key combination the keyboard (n.g., Ctri+t).
scroll [down] or scroll [up]': Scroll the page up or down.
Tab Management Actions:
new, nee_ Open a new, empty browser
"tab focus [tab index]': Switch browser's specific tab using its index.
"close tab": Close the currently active tab.
URL Navigation Actions:
goto [url]": Navigate specific URL.
go. back': Navigate to the previously viewed page.
go__ forward': Navigate the next page (if previous 'go_ back was performed)
Completion Action:
stop janswer]': : Issue this action when you believe the task is complete. If the objective to find a text-based answer, provide the answer the bracket
successful, very important understand the effect
action on
the webpage.
Follow the following rules for reasoning next state prediction.
1. Please generate your answer starting with Let's think stop stop, with your logical REASONING (after "(Rational]"".
2. When you generate your logical reasoning, you must mention the key changes in next state observation given as input.
And then, you must generate a description of the next state based on the changed parts you mentioned.
the prediction the correct
Start "[Next State] expected that phrase.
Demonstrations:..
(omilted)
Figure 19: The prompt for cunsition-focused observation abstraction in
$.1.22
28

---

Published as a conference paper at ICLR 2025
Prompt Transition-focused
during
intelligent agent that predict web-based tasks.
given with your logical reasoning. You will be given
Here's the information you'll have:
The user's objective: This the fask you're trying to complete.
current web page's accessibllity tree: is a simplified representation of the webpage, providing key information. page's URL:
page you're currently navigating.
The previous action: This is the action you just performed. It may be helpful track your progress. The current action: This the current action that you will perform to achieve the user's objective in the current web page's accessibility tree.
The format previous actions and current action can fall into several categories:
Page Operation Actions:
element with specific the webpage. 'ypp jid] [content]": Use this to lype the content into the field with id. By default, the "Enter" key is pressed after typing unless press
click [id]': This action clicks
press_enter_ enter__rfff Lafter is set to 0, le.. 'type [id] [content] [O]..
'hover [id]": Hover over an element with id.
press coyy_"'''' Simulates pressing key combination on the keyboard (e.g.. Ctritt).
"scroll [down]' "scroll [up]': Scroll the page up
Tab Management Actions:
'new_ jew': Open new, empty browser tab.
'abb focus [lab index]': Switch the browser's focus specific tab using its index.
"close laree__ab::: Close the currently active tab.
URL Navigation Actions:
goto [url]":
Navigate specific URL.
gg__bck':
go back':
Navigate to the previously viewed page.
go_ forward': : Navigate to the next page (if a previous 'go_ back' action was performed)
Completion Action:
stop janswer]": : Issue this action when you believe the task is complete. If the objective to find a text-basee provide the answer the bracket
To be successful, very important to understand the effect of current action on the next state of the webpage. You need verify whether the current action successful make an intended effect on the webpage. so, please explicitly mention evidence, otherwise describe why not
Follow the following rules for reasoning next state prediction.
1. Please generate your answer starting with Let's think step by step, with your logical REASONING.
When you generate your logical reasoning, you must identify and mention only the changed parts of the Jacoessibility tree] for the next state based on the given current action.
And then, you must generate description
changed parts you identified.
Generate the prediction the correct format. Start with '[Next State] The expected effectt that phrase.".
examples: (omitted)
Figure 20: The prompt used for the next state prediction of the world model
4.2
29

---

Published as a conference paper at ICLR 2025
Prompt for value function
You are an expert in evaluating and guiding a web navigation agent. Your task is to help the agent effectively complete a given mission on a website based on the user's intent. The agent's goal navigate through the to reach desired state aligns the user's objeclive.
You will analyze the next state of the webpage (OBSERVATION) after each action and determine whether the agent successfully progressing towards the task goal. You will also assist the agent by choosing the next action if necessary. considering the dynamics of the web environment and how each state transitions.
Key Points:
Understand the
Identify the user's goal (e g.. finding information, navigating specific page, modifying content), In- Make the next state of the webpage allgns with achieving that goal based on the current state and user's Intent.
2. Evaluate the Next State:
When assessing the next state, consider how contributes to reaching the intended goal. If the next state moves the agent closer to the user's goal, is evaluated positively.
not progress towards goal
an error, suggest alternative actions result in a more
favorable next state.
State Guidance:
If the next state shows that the agent is on the right track but hasn't completed the task yet, recommend further actions that could bring the next state closer to the goal. Focus on guiding the agent to reach state that reflects clear progress towards the goal.
Types
provide the specific information information is unavailable, the next state should explicitly indicate that.
seeks (e.g.. product price, reviews). the Sitte Navigation: The next state must reflect that the agent has navigated to the exact page or item. Check if the state includes content based on the user's intent.
Information Seeking:
The
The next state should indicate that the requested content modification has been successfully committed (e.g.. form submission, comment posting).
Content Modification:
General Task: Evaluate entire process ensure
reflects task completion. Stop
only be Issued
when the objective is met.
5. Common Pitfalls:
Repettive typing actions: Ensure that the next state does not show corrupted input due to repeated typing.
Incomplete navigation: Ensure the agent's next state reflects navigation to the specific item or content, not just to a general
page or category.
Output Format with Score Between 0 and
Each next state will be evaluated with score between 0 and 1, assessing how well the state moves towards the task's completion. This score provides nuanced feedback on the state's effectiveness.
The next state is a failure or leads away from the task.
Values closer to 0 (e.g.. 0.1, 0.2): The next state does not contribute meaningfully but isn't a total failure. position. Values closer to 1 (e.g.. 0.7, 0.8y: The next state helpful and moves the agent closer to the task goal.
state neutral,
agent maintaining
The next state is optimal and is directly aligned with completing the task.
Response Format:
You should write your rationale providing a detailed analysis the next state and reasoning for its soore, providing a score and based well the next state contributes completion.
Output Format:
[Rationale] syour thought> [Score] <a value between 0 and
Figure 21: The prompt for reward calculation using the value function in 4.2
30

---

Published as a conference paper at ICLR 2025
Prompt for baseline CoT
You autonomous intelligent agent tasked with navigating web browser. You will be given web-based tasks. These tasks will be accomplished through the use of specific actions can issue.
Here's the information you'll have:
user's objective: This the task you're trying to complete.
current web page's accessibiitty tree: This simplified representation of the webpage, providing key information.
The current web page's URL: This the page you're currently navigating. The open tabs: These are the tabs you have open. The previous action: This is the action you just performed. It may be helpful to track your progress.
you can perform
categories:
Page Operation Actions:
"clikk [d]]: This action clicks on an element with a specific id on the webpage. "type jid] [content] [press_ enter after-O]1]: Use this to type the content into the field with id. By default, the "Enter" key is pressed after lyping unless press_
press_enter enter after is set to O. hover [Id]": Hover over an element with ld.
press [key_ comb]': Simulates preesing of key combination keyboard (e.g.. Ctrlev).
scroll downjjpj Scroll the page down.
Tab Management Actions:
new, tab': Open a new, empty browser tab.
tab focus [tab index]': Switch the browser's focus specific tab using its index.
"loss Close the currently
URL Navigation Actions:
goto [url]": Navigate to specific URL.
go_ back': Navigate to the previously viewed page.
go_ forward'; Navigate to the next page (if a previous 'go,_ back' action was performed).
Completion Action:
stop [answer]': Issue this action when believe the task complete. If the objective to find a text-based answer, provide the answer the bracket.
Homepage:
you visit
homepage http://homepage com. list of you
account and password for the wabsitas.
log
websites.
To be successful, is very important to follow the following rules:
You should only issue an action valid given the current observation
should only issue one action time.
You should follow exampies
step by step and
action.
Generate the action the correct format. Start with summary, the next action will perform phrase, followed action
inside For example, "In summary, the next action will perform ""click [1234]""
5. Issue stop action when you think you have achieved the objective. Don't generate anything after stop.
"examples" ... (omitted)
Figure 22: The prompt used for baseline comparison with accessibility tree input using CoT in 5.4
31

---

Published as a conference paper at ICLR 2025
Prompt for self-rofine
You are an autonomous intelligent agent tasked with navigating a web browser to achieve the user's objective. Based on your next state prediction, you need to decide whether to refine your current action to better accomplish the user's intent.
format previous actions and
into several categories:
Page Operation Actions:
click [id]': This action clicks on an element with a specific id on the webpage.
'type jid] [content]': Use this to type the content into the field with id. By default, the "Enter" key is pressed after typing unless press enter__ettit aater is set l.e. 'type [id] [content] [O]".
hhver [id]" Hover
element
press (key comb]': Simulates pressing key combination the keyboard (e.g.. Ctri+t).
scroll [down]' or scroll [up]': Scroll the page up or down.
Tab Management Actions:
nnww tab": Open a new, emply browser tab.
focus [tab index]': Switch browser's
specific using
close_ tab': Close the currently active tab.
URL Navigation Actions:
goto [url]": Navigate to specific URL.
go back': Navigate to the previously viewed page.
go_ fonward'; : Navigate to the next page (if a previous back action was performed)
Completion Action:
stop janswer]': Issue this action when believe the task is complets. If the objective to find a text-based answer, provide the answer the bracket.
When you refine current action, think step-by-step.
Evaluate Current Action:
Review your
action and reasoning behind
Utilize the next state prediction to assess how effectively the action contributes to the user's 'erss objective.
Consider the overall progress toward the user's goal, and whether the action is a necessary step.
Decide on Refinement:
Only refine your action does not meaningfully progress toward user's intent can improved better align
objective.
the action necessary step the overall progress, proceed with the current action
Refine the Action (If necessary):
Think through the problem step-by-step to determine how to improve the action using insights from the next state prediction.
Re express your reasoning, focusing on how to enhance the action.
Generate new action that is valid given the current observation more effectively advances the user's goal.
Follow Action Formatting Rules:
Only
action
After generating your reasoning, start with a "In summary, the next action will perform is" phrase, followed by action Inside
For example, "cyour thought>, In summary. the next action will perform is ""click [1234]""..
Issue stop action when you think you have achieved the objective. Don't generate anything after stop.
Remember:
When evaluating and refining action, make to leverage the next prediction, but also consider whether the action essential step toward achieving the user's goal. Only refine your action when truly necessary to better with the user's intent.
Figure 23: The prompt for self-refine in
6.1
32


Reviews:
[{"review_id": "lSOZnweVBa", "reviewer": "Reviewer_N8Nf", "summary": "The paper proposes an approach for enhancing the performance of LLM-based web agents in long-horizon tasks. The authors introduce a World-Model-Augmented (WMA) web agent that simulates the outcomes of its actions through a \"world model.\" This design enables the agent to anticipate the effects of actions, thus reducing errors and improving decision-making in dynamic web environments. By employing a transition-focused observation abstraction, the model processes large state transitions without redundant data. Experiments conducted on WebArena and Mind2Web indicate that the WMA agent outperforms other agents in various metrics.", "questions": "1. Can the model extend to handle visual inputs alongside text-based observations? This would make it more versatile for environments where visual information is vital.\n\n2. How does the model perform in extended multi-step planning scenarios? Would recursive application of the world model effectively address multi-step planning challenges?\n\n3. How does the observation abstraction perform in very dynamic web environments? In fast-changing states, does it retain relevant information, or does it risk omitting crucial details?\n\n4. What is the impact of scaling the action space on the computational efficiency of the WMA model? Would the model maintain its efficiency with a larger action candidate set?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper considers the application of world models within LLM-based web agents, creating an interesting direction for improving policy selection in web-based navigation tasks.\n\n2. The proposed transition-focused observation abstraction helps to reduce redundant data, lowering computational costs and allowing the model to focus on critical changes.\n\n3. The authors show that the WMA model can be easily integrated into existing web agents, making it a versatile addition to LLM-based navigation models.", "weaknesses": "1. While the paper demonstrates improved single-step action selection, it does not address multi-step planning comprehensively, which is crucial for certain long-horizon tasks.\n\n2. The approach is limited to text-based web interactions, and although HTML trees are effective, omitting visual elements could restrict the model's applicability in web navigation tasks where visual cues are essential.\n\n3. While the transition-focused abstraction reduces redundancy, it may oversimplify complex web elements, potentially leading to reduced accuracy in actions involving nuanced page layouts.\n\n4. The paper does not discuss how the proposed world model scales when dealing with significantly larger action spaces, which is critical for broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w47AN6YkW0", "reviewer": "Reviewer_mAfj", "summary": "This paper explores using explicit world models in LLM-based web agents. An (explicit) world model is a model able to predict the future state given an action on the current state. Their preliminary analysis shows that existing models cannot successfully predict the results of their actions, but that knowing that information would significantly help LLM-based agents. \n\nTheir contributions are \n1) a training algorithm for web-agent world models that predicts an abstracted version of the transition to the observation after an action is applied. \n2) an algorithm for using the world model to improve performance. \n\nThe training algorithm uses trajectory data to produce observation diffs from each action, then summarizes these changes with an LLM. The world model LLM is then trained to output the summaries (predict what will change in natural language)\n\nThe policy optimization algorithm uses a trained value function in addition to the world model. It works by sampling multiple potential actions, using the world model to predict changes, and then using the value function to predict the value. This can be thought of as a two-step Q-value function where the first step is the world model and the second step is evaluation. The value function is trained using reward data from Mind2Web. \n\nThey have main results on WebArena and Mind2Web. There is also further analysis by category, and other experiments, including ablations over the reward estimation, training of world model, and use of abstracted observations.", "questions": "Clarifying questions:\n\n1) What is “policy optimization” as applied to Tree Search Agent (Kuh et al, 2024). I thought Figure 3 bottom shows “policy optimization” which seems like it would only apply to World Model Agent. \n\n2) Correspondingly, what is \"no policy optimization\"? Is that just CoT? If so, why are the results different? Why is Tree Search Agent better?\n\n3) Is value function trained with the world model data or is it a pure value function? If pure value function, how does that work with the abstracted observations? \n\n4) Line 284: I don’t get this line and how the citation matches. “explore diverse next states st+1 ∈ S (Wang et al., 2022)”\n\n5) Table 5: The value function is fine-tuned. Is the Q-value function being fine-tuned as well and in the same way? If not, does not seem like a fair ablation. Also, is the Q-value function given ability to reason about what the action will do before outputting the value?\nOther questions:\n\n6) Why is the user instruction included in the data? Shouldn't the world model be able to predict the changes based on the action without needing to know the intent?\n\n7) Table 2: Why not include the baselines from table 1? At least the tree search agent which is repeatedly compared to. \n\n8) Figure 1 shows that commercial LLMs cannot predict the next state well by looking at actual observations. Perhaps they would be able to pick the correct abstracted transition-focused observation? Running that same experiment with the transition focused observation version (ground truth, not predicted) could be interesting. \n\n9) Which leads me to the following question, how do larger commercial LLMs fare at predicting the next abstract transition? It would be nice to see Table 5 row 2 with a few different base LLMs. \n\n10) Why is HTML-T5 considered SOTA for Mind2Web? It seems like there are many much stronger algorithms presented in table 1. Mind2Web is also an offline evaluation dataset so does not account for different ways of completing a task or different action spaces. \n\nOther notes:\n\n11) Line 184: “These findings highlight the necessity of world models in LLM-based web agents”: \n- Not sure I agree with this. I would say something more like, “This findings highlight the benefit world models could provide to LLM-based web agents”.\n- The findings show that (my words) “LLMs do not have good world models for web activity” and “having a better world model would improve web agent performance”. This does not add up to “necessity”\n\n12) “In Table 1 (middle), we first compare our WMA web agent (16.6%) with vanilla CoT (13.1%) and observe significant improvements over almost all domains in WebArena as detailed in Table 2.” \nThis is confusingly written. Table 1 is the main table and there is only half a sentence of analysis. \n\n13) I do not like the y-axis bounds selection on many of the figures. It would be easier to read with 0 as lower bound (or at least a bit more space). E.g. Figure 6 is hard to tell what the k=1 line is at.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality:\nThe paper provides an original algorithm for training web-agent world models and an original algorithm for using such models to improve LLM-based web agents. \n\nClarity:\nThe clarity of the problem domain is quite good. The introduction lays a good case for how world models could improve web agents. The approach is quite clear. Figure 3 is nice. There is some breakdown of the clarity for some of the results but overall it was easy to follow along and understand what was being done. \n\nSignificance:\nThe utility of world models for web-agents is quite clear but still under-explored. Both the training and inference time algorithm presented here could have significance down the line.", "weaknesses": "The primary weakness of this paper is that the main results are quite weak. On WebArena, this significantly underperforms the listed baselines, and other newer baselines also exist. \n\nThe second weakness is that it is not always clear what is being tested or shown. There are a number of questions (see below) that need to be clarified. \n\nTo me the paper should focus more on the web-agent *world model* than on the web agent itself. There is some analysis on failure modes but not enough analysis on the world model itself. This may stem from the fact that abstract transition representations may be hard to analyze. A full world model would be able to show things like per-element accuracy. Perhaps there are other creative ways to analyze how to assess the learned world model and how it functions. \n\nThe paper really dives into this one way of using the world model when it could take a broader view. For instance, why not use WMA/policy optimization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cnbzbfWGbs", "reviewer": "Reviewer_YRJg", "summary": "This work proposes Wold-Model-Augmented (WMA) web agents. The authors present detailed analysis on the inability of current LLMs in estimating action consequences on the web and propose to abstract webpage state presentation as state transition description in natural language accordingly. Based on the abstraction, a world model for web navigation is trained and used for enhancing web agents. On two popular benchmarks, WebArena and Mind2Web, the proposed WMA web agent shows sizable performance gain over the baseline LLM and demands lower cost than tree-search methods.", "questions": "1. In section 4.1.1, training data for the world model is sampled from an LLM as web agent. Does this strategy create a dependency between the world model and the underlying agent, which could potentially hinder generalization?\n2. The proposed method, particularly the state abstraction method, is limited to the text modality (HTML and accessibility tree). Can it be easily extended into a multimodal setup?\n3. As stated in weaknesses, clarification of Vanilla CoT and WMA web agent /o policy optimization is needed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. To the best of my knowledge, this work is among the first to employ the world model in web agents. Its technical contribution includes a novel state representation, world model training method, and world model augmented agent design.\n2. The resulting method shows improved performance and higher efficiency over baselines.\n3. The experiment setup is overall reasonable. The authors provide sufficient ablation study and detailed analysis on performance, efficiency, and error distribution, contributing valuable insights to the community.\n4. The writing and presentation of this paper is clear and easy to follow.", "weaknesses": "1. The experiments only cover end-to-end evaluation of agent performance. A directly evaluation on world modeling performance would be helpful for understanding the quality of the world model itself. For example, this evaluation can follow the same setting as the analysis of section 3.\n2. The performance of Vanilla CoT and WMA web agent w/o policy optimization differs (13.1% vs 12.8% in Table 1). What are the differences between them? In addition, WMA web agent underperforms the tree search agent. Although tree search leverages additional signals and more compute, I think it is important to discuss if WMA web agent could scale to higher compute and deliver competitive performance under similar budget. \n3. The proposed method and evaluation is limited to the text modality. Recent studies have suggested that visual features are critical for good web navigation performance. It remains unclear if WMA could adapt to multimodal settings and advance the state-of-the-art of web navigation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8dHxCLWkbt", "reviewer": "Reviewer_tKJR", "summary": "This paper explores the introduction of world knowledge models into the field of agents, particularly within the context of web navigation. Specifically, the study finds through exploratory experiments that \n1. existing large models lack sufficient predictive capability regarding the consequences of their actions. \n2. Enabling these models to anticipate such consequences can significantly enhance their planning capabilities. \nBased on this finding, the authors trained a world knowledge model (WKM) to predict the potential outcomes of an action.  The proposed transition-focused observation abstraction plays a crucial role in the training stage.\nThe paper conducts comprehensive experiments to demonstrate the necessity and superiority of using WKM.", "questions": "1. As discussed in weakness 1, could the authors conduct a more extensive literature review to discuss the similarities and differences between existing WKM-based methods and their proposed approach? Although this is mentioned in section 6.1, too few works are considered.\n2. As discussed in weakness 2, while the authors have considered the impact of exploration breadth in WKM during the ablation study, could they conduct experiments to consider the impact of depth as well?\nIf the authors can adequately address the questions above, I would be willing to reconsider the review.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of this paper is well-constructed. Through experiments, it identifies the issue that existing models cannot effectively predict the potential consequences of their actions, and it highlights that enabling models to understand these consequences can enhance their planning capabilities. The proposed approach is very reasonable.\n2. The explanations in this paper are clear and easy to understand. The formulas and figures effectively illustrate the method proposed by the authors.\n3. The authors conducted multi-dimensional experiments that effectively address potential questions from readers.", "weaknesses": "1. Although this paper claims to be the first to introduce a world knowledge model into LLM-based agents, to my knowledge, this may not be the case. For example, the work \"Agent Planning with World Knowledge Model\" might also explore this area. The authors may need to conduct a more extensive literature review to compare and discuss the similarities and differences between their proposed method and existing approaches.\n2. The performance improvement brought by the experiments in this paper is not significant. Additionally, the authors' WKM is limited to a depth of 1. It might be worth considering deeper levels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)