Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents
Authors: Dongjun Lee
Juyong Lee
Kyuyoung Kim
Jihoon Tack
Jinwoo Shin
Yee Whye Teh
Kimin Lee
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
LLM agent
Web automation
Primary area: applications to robotics, autonomy, planning
TL;DR: Training language models to automatically contextualize web page observations to enhance the decision-making of LLM agents

Abstract: Recent advances in large language models (LLMs) have led to a growing interest in developing LLM-based agents for automating web tasks. However, these agents often struggle with even simple tasks on real-world websites due to their limited capability to understand and process complex web page structures. In this work, we introduce LCoW, a framework for Learning language models to Contextualize complex Web pages into a more comprehensible form, thereby enhancing decision making by LLM agents. LCoW decouples web page understanding from decision making by training a separate contextualization module to transform complex web pages into comprehensible format, which are then utilized by the decision-making agent. We demonstrate that our contextualization module effectively integrates with LLM agents of various scales to significantly enhance their decision-making capabilities in web automation tasks. Notably, LCoW improves the success rates of closed-source LLMs (e.g., Gemini-1.5-flash, GPT-4o, Claude-3.5-Sonnet) by an average of 15.6%, and demonstrates a 23.7% average improvement in success rates for open-source LMs (e.g., Llama-3.1-8B, Llama-3.1-70B) on the WorkArena benchmark.
Moreover, the Gemini-1.5-flash agent with LCoW achieves state-of-the-art results on the WebShop benchmark, outperforming human experts. The relevant code materials are available at our project page: https://lcowiclr2025.github.io.

Published as a conference paper at ICLR 2025
LEARNING TO CONTEXTUALIZE WEB PAGES FOR ENHANCED DECISION MAKING BY LLM AGENTS
Dongjun Lee"l, .uyong Lee"), Kyuyoung Kim Jihoon Tack' Jinwoo Shin', Yee Whye Teh", Kimin Lee' {dgjun32, agi. is / @kaist. ac. kr
KAIST AI University of Oxford
ABSTRACT
Recent advances in large language models (LL.Ms) have led to growing inter- est in developing LLMM-bseed agents for automating web tasks. However, these agents often struggle with even simple tasks on real-world websites due to their limited capability to understand and process complex web page structures. In this work, we introduce LCoW, a framework for Learning language models to Contextualize complex Web pages into more comprehensible form, thereby hancing decision making by LLM agents. LCoW decouples web page under- standing from decision making by training separate contextualization module transform complex web pages into comprehensible format, which are then utilized by the recision-makinn agent. We demonstrate that our contextualizzaton module effectively integrates with LLM agents of various scales to significantly enhance their decision-making capabilities in web automation tasks. Notably, LCoW im- proves the success rates of closed-source LLMs (e.g., Gemini- 1irii-1.5-flaas flashh, GPT-4o, Claude-3.5-Sonme by an average of 15.6%, and demonstrates a 23.7% average improvement in success rates for open-source LMs (e.g., Llama-3.1-80 Llama- 3.1-70B) on the WorkArena benchmark. Moreover, the Gemini-1.5-flan agent with LCoW achieves state-of-the-un results on the WebShop benchmark, outper- forming human experts. The relevant code materials are available at our project page: https: http://oocclllll g thub. io.
INTRODUCTION
Large language models (LLMs) have demonstrated strong potential in automating web tasks by treating web browsing as a sequential decision-making pro- cess, where web pages serve observations and user interactions, such as clicking and typing, function as actions (Yao et al., 2022a;b). Various approaches have been developed to enhance the performance of LLM agents in these tasks. One such method in- volves fine-tuning open-source LLMs using demon- stration data from web browsing tasks (Furuta et al., 2023; Lai et al., 2024). While promising, this ap- proach requires extensive data collection and signifi- cant computational resources for effective fine-tuning. Alternatively, several studies have utilized advanced closed-source LLMs, such as GPT-4o (OpenAI, 2024), with carefully designed prompting techniques (Drouin al., 2024; Zhou et al., 2023a; Sodhi et al., 2024; Pan et al., 2024). By leveraging the general world knowledge and reasoning capabilities of these models, the methods enhance web automation but at the cost of reduced controllability.
observation Contextualized by GPT-10
Contextualized by
rate (%)
Figure 1: Success rate of the Gemini- 1.5- flash agent 40 WorkArena tasks. We selected a subset of 40 tasks by choos- ing the first 40 tasks based on the task in- dices. When the agent leverages observa- tions contextualized by GPT-4o (yellow), its success rate improves by 31%, with further improvements achieved with our method (green).
However, despite the advancements, state-of the-art LLM agents often struggle to process complex raw web content, such as HTML and accessibility trees, posing significant challenges for their effec-

---

Published as a conference paper at ICLR 2025
Web page observation
LLM agent making decision based on raw observation
salisfied all requirements.
Action: click ("order 1ttam'
X
Contextualization
LLM agent making decision based on contextualized observation
un Focand E-berrdaaaa
3237 cratiten Vianeo canes altupueee code lrcanoaly
fo
The quantity is set to have to select the quantity as the product.
The caichi catorall as
lipulii add sora ooou oediiaata
Action: select| " "quantity" ,
Figure 2: (Top) In the conventional pipeline, LLM agents decide on the next action based on raw, complex web page observations (e.g., HTML, accessibility trees), which often hinder accurate de- cision making. (Bottom) In our proposed pipeline, a contextualization module transforms these complex web page observations into more comprehensible format, thereby enabling LLM agents to make more accurate decisions by enhancing their understanding of the web page.
tive use in web task automation. While LLMs excel in tasks that require detailed reasoning, such as solving mathematical problems or coding, we hypothesize that their underperformance in seemingly simple decision becision-making tasks like web browsing is not due to a lack of decision madisiin-making capabilities but rather to difficulties in understanding and processing complex web page observations.
To validate our hypothesis, we conducted an initial experiment that demonstrated an LLM agent based on Gemini-1.55 1.5-flash equipped with a module designed to contextualize complex web page observations (i.e., contex- tualization module). This module enhances task performance by removing irrelevant UI elements and highlighting key components with explanations, thereby simplifying the decision-making pro- cess. We evaluated the performance of this agent on 40 tasks from WorkArena (Drouin al., 2024), a benchmark designed to assess web agents on real-world, enterprise-relater websites. As shown in Figure 1, utilizing GPT-4o as the contextualization module (yellow) resulted in a 31% absolute im- provement in the agent's s success rate compared to direct processing of raw observations (red). These results support our hypothesis that the difficulty in understanding web pages is a major bottleneck for M-baedddd web agents.
achieve substantial improvements in web browsing tasks when
In this work, we propose LCoW, framework that includes a contextualization module and a train- ing algorithm to fine-tune this module to enhance the decision-making capabilities of LLM agents in web automation. As illustrated in Figure 2, the contextualization module transforms complex web page observations into comprehensible format, enabling LLM agents to make more accurate decisions. Furthermore, to enable the contextualization module to provide context more grounded in real websites, propose an iterative algorithm designed to train the contextualization module. Our training algorithm consists of three phases: (i) trajectory collection, (ii) sampling contextual- ized observations, and (iii) updating the contextualization module. For each observation from the collected trajectories, we generate multiple contextualized observations using the current contextu- alization module. Each observation is then assigned reward based on whether set of LLM agents can accurately predict the correct action given the contextualized observation. Finally, we select the one with the highest reward as the target and train the contextualization module maximize the likelihood of the target given the original raw observation.
As demonstrated in our initial experiment using the Gemini-11.5fiaal (see Figure 1), LCoW signif- icantly enhances the decision-making capabilities of LLM agents, even beyond the improvements seen with state-of-the-art LLMs like GPT-4o used as a contextualization module. In our experi- ments, we conduct comprehensive evaluations of our proposed approach three well-established benchmarks for evaluating agent performance in web environments: WebShop (Yao et al., 2022a), WorkArena (Drouin et al., 2024), and WebArena (Zhou et al., 2023a). In detail. first demon- strate that LCoW significantly enhances the overall performance of LLM agents with varying scales in the main experiment. Furthermore, we analyze the behavior of the contextualization module in
2

---

Published as a conference paper at ICLR 2025
detail, including qualitative examination of the observation refined by the contextualization module, comparison with training LLM agents using behavior cloning, and generalization ability.
BACKGROUND
In this section, we describe the formulation of web browsing sequential decision lecision-making prob- lem and the use of LLMs as decision decision-making agents.
Web browsing can be formulated as Partially Observable Markov Decision Process (POMDP), de- fined by (S, O, A, T, R). The state 8, E S represents the internal configuration of the web browser time step which is only partially observable. The observation O₁ O corresponds to the web page rendered by the web browser given St, which can take various forms (e.g., screenshot, HTML, accessibility trees). The action space A is the set of all possible interactions with the UI elements (e.g., clicking, typing). The state transition function I defines the probability of transitioning from state s to state after performing an action a E A. such as clicking a link scrolling. While tran- sitions are typically deterministic, occasional stochastic events (e.g.. pop-ups, network errors) can occur. The reward function R assesses the functional correctness, evaluating whether the resulting state satisfies pre-defined criteria for successful task completion.
Leveraging their ability to interpret web pages and generate actions in text form, LLMs are increas- ingly employed agents for automating web-based tasks. In this framework, generates an action d, to interact with UI elements at each time step i, based on the user instruc- tion [TASK], the current web page observation ot, and the history of previous actions act- The objective of the agent is to complete the given task in order to maximize the reward.
LLM agent
METHOD
In this section, we present LCoW, framework for enhancing the capability of LLM agents by textualizing complex web pages. Section 3.1 outlines the concept the contextualization module and its integration with LLM agents for web task automation. Section 3.2 introduces iterative algorithm for training the contextualization modules to improve decision making of L.M agents.
3.1 CONTEXTUALIZATION MODULE
In this work, we decouple web page understanding from decision making of LLM agents. Our hypothesis is that while LLM agents possess strong decision-making capabilities, their perfor- mance can significantly degrade when they rely on lengthy, non-contextualized observations, such as HTML and accessibility trees. To address this limitation, we introduce a contextnalization mod- ule, a separate language model designed enhance LLM agents by contextualizing complex web page observations into a form that is easier to process and comprehensible. Intuitively, a proper contextualization of observations can enhance the agent's understanding of web content and its de- cision making based the improved understanding (see Figure for example of the input and output of the module). Formally, given a web page observation O, at time step I, the objective of contextualization module fe is to generate contextualized observation of that serves in- put to the LLM agent т to enhance its decision making. Specifically, fo uses the task instruction [TASK], the previous actions of the agent act, and the current web page observation o generate a contextualized observation:
0 fo [TASK] aee, 01).
The LLM agent then predicts the next action based on the contextualized observation:
Gt = T [TASK], ,0e,, o
While arbitrary language model can serve the module to learn from experience the web environment to provide more grounded context for decision making, such role of particular button interaction with specific UI elements.
contextualization module fe, it is important for
3.2 ALGORITHM FOR TRAINING THE CONTEXTUALIZATION MODULE
We now describe an iterative algorithm for training the contextualization module fo to enhance decision making of L.M agents. In nutshell, the algorithm involves an iterative process of col-
3

---

Published as a conference paper at ICLR 2025
to geserate "Reesoning"" a "Reffne oberrvation breed on the providol information.
Ressoning: We have mavigsted
Hardware stome
clicked on the 1Pad pco likk The actions needed to acccumpliei the user instrucion are to dursee cobor option and disk uption. Redined observation: can focus on following elemenes:
Task: 'Purchase me an iPud peo wish silver color and ...'
Action history:
click "Bardware'
click (ad5] "1Pad pzo"
combobox 'Qaaniity" value= "', chocked-arss, Is confirma tixt in airwady selected os quanti sy.
Web page observation:
Rontilebrres
"Cataleg Servicalion' "anertanorsivee.
[x195] rediogroup "Mandatory" [a196] radio 'Vuf Silver', cbacked-True (a1977 radio "Vuf Space
genaric "", 1iwe'pplste"
(66) navigatica "Global skip links'
group
options.
(4157) rougroup Extbi roe [a159] gridcell
[a1se] radtogroup 'mandstory'
[a200] radie "Vuf117 128 GB', checkud-Trus
[2203] radio "Nuz157 256 0B [ada $:00.00 00] This group custains the
uptions.
[a405] LayoutDabletov
[ad02] LxportTabeeeell 'Boopinng
"Order
[+403] LayoutTablaCull 'Eaply'
This button allows to finaline process.
Figure 3: An example of input of contextualization module including lengthy web page observa- tion (left) and an observation contextualized by the contextualization module trained using LCoW (right). The module converts raw observations into more concise form enhance decision mak- ing in agents. The prompt used is provided in Appendix C.
lecting paired input-output data for training the contextualization module and subsequently updating the module based on the collected data. For data collection, we begin by gathering trajectories of successfully completed tasks from the web browsing environment. For each observation o, in the collected trajectories, we sample multiple candidate contextualized observations, and select the one that best provides the relevant context for multiple LLM agents to accurately predict the next action ag Based on the chosen target observations, we update fe via supervised fine-tuning, We now out- line a single iteration of LCoW, followed by a detailed explanation of the design of the reward used for evaluating the candidate contextualized observations.
Single iteration A single iteration of LCoW starts with the contextualization module faco and aims to update this module to foc441). This process consists of three steps:
Step 1 (Trajectory collection). Given a set of training tasks, we roll out the L.M agent m in the web environment to collect trajectory data. Specifically, the agent determines the next action based on the contextualized observation produced by faco until the episode terminates. We collect only those trajectories that end in the successful completion of the tasks.
Step 2 (Sampling optimal contextualization). As illustrated in Figure 4, we start by sampling multiple candidates 05 from the current contextualization module face (i.e., of (- [TASK] a=c, ot)) for each web page observation oe in the collected tra- jectories. Each candidate is then assigned reward based on whether set of LLM agents can accurately predict the ground-truth action a; given ot, with the candidate receiving the maximum reward selected the optimal contextualized observation. If all candidates receive a zero reward, we retry the sampling process with the ground-truth action ae provided as ad- ditional context to faz:: (i.e., of fgio [TASK), act, Or, a:)) to guide the generation of valid contextualized observations.
Step (Model update). We update the current module faco by fine-tuning with the optimal contextualized observations collected in Step 2. With the updated module foo+1), we return to Step and repeat the process.
Algorithm outlines the steps for a single iteration of the training algorithm. Starting with an initial contextualization module foo,, we iteratively train the module through M iterations. After each iteration, the module updates from foo) to foG+1) until reaching the final faues Additionally, T can be initialized as set of human demonstrations in order to facilitate the training process.
Reward for contextualized observations The reward for the contextualized observation of is defined as the sum of the aetion-matching scores computed using multiple I.LM agents. Each action matching score evaluates whether an L.M agent correctly predicts the ground-truth action o given the contextualized observation ofe. By leveraging multiple LLM agents to compute the reward, we
In the initial iteration, assume limited set of seed demonstrations and use relatively strong I.IMs to sample candidate contextualized observations accelerate training.
4

---

Published as a conference paper at ICLR 2025
Sampling candidates contextualizee observations
Computing matching
Claude
GPT
Gemini
Rewards
Contextualization module
[TASK]
1,1
a<t
66
Fr,
3
fo(
1,,
Fr,3
Ot
50
1
Figure 4: Illustration of sampling optimal contextualization. First, we sample multiple candidates of contextualized observations, given user instruction [TASK), previous actions act, and observation Or. Subsequently, multiple LLM agents predict the next action based on each candidate, and the reward for each candidate computed according how many LLM agents correctly predict the ground-truth action ag.
Algorithm One iteration of LCoW
Require: contextualizzatin module fato, LLM agent v,, set of LLM agents for computing the Teward ) trajectory buffer emply data buffer D, and set of training tasks
9tr
1: // /'Trajectory collection
for [TASK] Ger do
Collect successful trajectories from training enviromment
3:
T, R. ~ ([ [TASK), fecn)
4: R 1.0 then
T.append(7)
// Sampling optimal contextualizations
7: for (oi, a.) T do
8:
for n2 < 1 to N do & -Sample N candidate contextualized observations and assign rewards oi focn [TASK], age, 0x)
10:
Tt. = Exen ActionMatchingScore(ff [TASK), ((ct:
11:
if max, (Fr,n) = then
12:
for n + to N do on (. a
b Retry the sampling if rewards are zero for all candidates
13:
14:
F(, Exen ActionMaachinggSoon((tT [TASK), aet, o(3))
15:
0 arg max,, (rt,)) D.append(((
16:
(TASK) ([T[(s(]
acc, 0t), of,))
17: // Parameter update
18: ((++1) ;= arg maXgro o [TASK], act, 0:)]
19:
return fo
ensure that the module produces contextualized observations that generalize across diverse set of agents, preventing overfitting to the behavior of any single agent and enhancing the module's adaptability to arbitrary LLM agents.
4 EXPERIMENTS
We design our experiments to investigate the following questions:
How effective is LCoW in training a contextualization module for improving decision making of LLM agents? (Section 4.2)
Can the contextualization module trained with LCoW generalize to arbitrary LLMs with varying scales? (Section 4.2)
What form do the web page observations take after contextualization and how the contextualized observation aid the decision making of L.M agents? (Section 4.3)
Can the contextualization module trained with LCoW generalize unseen tasks? (Section 4.3)

---

Published as a conference paper at ICLR 2025
WeoN-Ts (Gur al, 2023) ASH (Srihaar al., 2023)
ReAct (Yeo al,, 2022b)
LASER (Me 2024]
Ours
WebGUM (Furuta et al., 2023)
AgentQ (Putta 2024)
62.8
60
Succes rate (%)
Human expert (59.6%)
50.0
45.0
40.0
40
29.8
30.2
20
0
Figure 5: Success rate on 500 evaluation tasks from WebShop. Average human performance and ex- pert human performance are 50% and 59.6%, respectively (Yao et al., 2022a). The cemini--1.5-faas agent with the contextualization module trained for three iterations achieves state-of-the-ee suc- cess rate of 62.8%, outperforming the human expert performance, well as previous baselines (Yao al., 2022b; Furuta al., 2023; Putta al., 2024; Sridhar et al., 2023; Ma al., 2024; Gur al., 2023).
4.1 BENCHMARKS & EVALUATION SETUP
In this section, we desribe three benchmarks used for our experiments. We consider the three bench- marks, WebShop (Yao al., 2022a), WorkArena (Drouin al., 2024), and WebArena (Zhou al., 2023a), which are well-established benchmarks designed to evaluate the capabilities of web agents completing various web tasks. For the main experiment, we exploit Webshop and WorkArena For the additional analysis, we also conduct experiments in WebArena. Additional details about hyperparameters and prompts are described Appendix B and Appendix C, respectively.
WebShop WebShop provides simulated online shopping environment with real-world product data, consisting of 500 evaluation tasks and 5,500 training tasks, each defined by natural language instructions purchase products that meet specific criteria. During each decision-make agent receives accessibility tree formatted web page observation to predict the next action, and at the end of an episode, it receives reward ranging from 0 to based how well the attributes of the purchased item align with the intended product criteria. When the reward is equal to 1, the episode considered be success, In our experiment, we train the contextualization module using LCoW on environments associated with 500 of the 5,500 training tasks in WebShop. After training, we evaluate the module on 500 evaluation tasks, measuring both the success rate and average reward.
step, the
WorkArena WorkArena focuses on assessing web agents' ability to complete enterprise-related tasks, such as creating user accounts and ordering products from a service catalog, on realistic web- sites. This benchmark comprises 33 task types, with each task type containing up 1,000 individual task instances, where agents have to navigate over web pages by following natural language instruc- tions and receive rewards based successful task completion. The agent receives binary rewards (i.e., 0 or 1) based on whether the instruction is achieved via web navigation. In our experiment, we train the contextualization module 15 task instances for each of 33 task types (i.e., 495 tasks total) and evaluate the success rate on task instances for each of 33 task types (i.e., 165 evaluation tasks).
WebArena WebArena consists of 812 tasks spanning over websites (Shopping, Gitlab, Reddit, Map, Wikipedia, and content management systems), where 812 individual tasks are created based on 190 task templates. Similar to WorkArena, WebArena is also based on realistic web environment, which incorporates extremely long and complex web page observation. For each task, defined as a natural language instruction, the agent receives binary rewards based on whether the instruction is achieved by web navigation. Additionally, Liu et al. (2024) proposed 165 evaluation tasks filtered from the original 812 tasks in WebArena, for efficient evaluation. For the experiment, evaluate the success rate on these 165 tasks while utilizing the remaining 647 tasks in the original WebArena benchmark training tasks.

---

Published as a conference paper at ICLR 2025
Llama-3.1 1--00B
GPT-4o
Gemini- 1.5-flash
Claude-3.5 3.5-Sonnet
(Unseen)
Success
Reward
Success
Reward
Success
Reward
Success
Reward
Raw ohservation
84.8%
0.496
43.6%
0.693
26.6%
0.336
34.2%
0.590
Self-etx
26.2%
0.459
46.4%
0.608
12.4%
0.146
40.2%
LCoW (iter
27.8%
0.545
46.4%
0.705
39.4%
0.600
39.2%
0.666
LCoW (iter
46.0%
0.647
58.2%
0.796
58.8%
0,780
55.0%
0.781
LCoW (iter 3)
50.6%
0.666
62.8%
0,803
59.8%
0.771
59.6%
0.803
Table I: We investigate the efficacy of LCoW across multiple LLM agents in WebShop. For all LLM agents, LCoW consistently improves both success rate and reward over iterations, surpass- ing celf-contextualization (self-ctx) and even human expert-level success rate by the third iteration. Additionally, LCoW is also effective when combined with .lamm-331-708 which was not used for computing the action-matching reward (i.e., unseen) during training the contextualization module.
1lama-3.1-709 (Unseen)
1.lma--31-88
GPT-4o
Gemini-l.5 1.5-flash
Claude-3.5-Sommn
(Unseen)
Raw observation 38.2%
11.5%
44.8%
26. 1
1.2%
Self-etx
43.0%
12.7%
50.3%
29.1%
7.3%
LCoW (iter I) 44.2%
41.2%
55.8 %
40.0% %
37.0%
Table 2: We evaluate the success rate of five LLM agents with varying scales on 165 tasks in the WorkArena benchmark. Single iteration of LCoW shows improvement of success rate over all LLMs, even generalized to Llama-3.1-70B and alama-3,1-8B, which were not used for computing the action matching reward.
4.2
MAIN RESULTS
We first demonstrate the effectiveness of LCoW on WebShop and WorkArena benchmarks and show that the contextualization module trained via LCoW enhances performance even for LLM agent that was not involved in the LCoW training process, such as Llama-3. 3-70B. We evaluate against two baselines: (i) decision making based solely on raw observations without the contextualization module (i.e., Raw observation), and (ii) decision making based on the observation contextualized by the L.LM agent itself (i.e., seff-comtextualizzatio)
Effectiveness of LCoW As shown in Table 1, in WehShop benchmark, both the success rates and the average rewards achieved by all three LLM agents improve substantially when integrated with the contextualization module trained with LCoW. Partieularly, both the Gemini-1.5-ffaal and Claude-3.5- Sonnett agents surpass the average human performance of 50.5% when combined with the contextualization module trained for 2 iterations. When the contextualization module trained for 3 iterations, the agents exceed the expert human-level performance of 59.6%. As illustrated in Figure 5, the Gemini-1.55flas agent combined with LCoW achieves state-of-art performance on the WebShop benchmark, outperforming prior methods in success rate by more than 12%. Notably, Saaude-3.5-Soneee agent, lower performing than the other agents on WebShop, achieves 59.8% of success rate when integrated with contextualization module, demonstrating that LCoW can effectively enhance the decision decision-making capabilities of LLM agents.
As shown in Table 2, WorkArena, the Claude-3.5-Sonnet agent outperforms the GPT-4o agent, achieving a success rate of 44.8% in our evaluation setup. When integrated with LCoW, Claude- 3.nnet agent achieves even higher success rate of 55.8%, which is higher than all baselines evaluated. GPT-4o and Gemini-1.5-fee also improve 5% and 30%, respectively, when combined with LCoW.
Generalization to arbitrary LLM agents We evaluate the JJama-3.1-88 agent and Llama-3.1 1- 70B agent integrated with LCoW to assess its effectiveness with LLM agents not involved in the training process (i.e., those not used for computing action-matching rewards). As shown in Table 1, the Llama-3.1-700 agent combined with the contextualization module trained for two iterations out- performs average human performance (50.0%) on WebShop and achieves the success rate of human experts (59.6%) when trained for three iterations. Furthermore, Table shows that the contextu- alization module also enhances Llama-3.1 3.18B and 1amm--3.1-70B agent on Work Arena, improving

---

Published as a conference paper at ICLR 2025
Contextualized observation:
Contextualized observation:
The AbIeoo observation shows we on the shipping Repoct page. page
the Axtree chservation shous list foruns azticle with its nase, number of subscribers, and munbor of oubnissions. The foruna are sorted by the number of subrissions descending
prosented
range inpute, order status options, and other report parameters, There's also an export Iunction and table display
elesents crucial corpleting
Additionally, for
inolude first forun show wheze are currently:
instruction:
[753] LabelText
[141] article
StaticText 'Fron!
[143] heading 'AakBaddit AskReddit'
StaticTont
[144]
[756] texthox 'Fzaz (8/u=="08/00//022"
[149] paragraph
button 'undetined (ow627'
[152] Section
[760] LabelText
StaticTest 'Ie'
Statictext 10'
StaticText 141
[160] pazagraph
[764] button 'undafined Jue627'
This extraction shows the first forun in the 11st, which in "Aakdaddss"" with 10,041 subni exions. It confirns that currently the firet navigate next page continue our search oplistingtiers subredait.
This section containg the fillez parareteza foz the shipping paport. already been filled with the spacitted dates (08/05/2022 and 03/01/2023, respectively).
foruns and need
Figure 6: Examples of how the contextualization module refines complicated web pages into com- prehensible format from WebArena benchmark (I.fft) and WorkArena benchmark (Right). As dicated by blue color, the contextualization module provides comprehensible context by verbally explaining the web page and UI elements relevant to the given task.
the success rate by approximately 36% and 13%, respectively. is noteworthy that a relatively small LLM agent (i.e., Llama 3.1 3-1) struggles perform tasks when given observations, but when combined with LCoW, its success rate rises on significant margin. It demonstrates that LCoW can elicit significant level of decision-making capability even from smaller models at the 8B scale.
4.3 ANALYSIS
How web pages are contextualized? We qualitatively analyze how the contextualization mod- ule, trained via LCoW, processes real-world web page observations and how the contextualization enhance decision making of LLM agents on web tasks. As shown in Figure 6, we observe that the contextualization module not only simplifies the web page by extracting UT elements relevant to the given task, but also verbally explains functionalities of the UI elements in web pages. These explanations aid LLM agents significantly in making more accurate decisions, while reducing the selection of inadmissible or redundant actions, which is the main failure mode of the LLM agent for web browsing. This knowledge of the functionality and interaction of UI elements grounded the real website appears to have been learned during the LCoW training process, where the contextual- ization module explores various observation contextualizations and learns to generate contextualized observations that lead to more accurate decision making by LLM agents. More detailed examples and analyses are provided in the Appendix
Comparison to directly training LLM agents One might argue that, given available demon- strations, it would be more straightforward train the LLM agent directly rather than using the demonstrations to train a contextualization module. To demonstrate the superiority of training the contextualization module with equivalent amount of demonstration data, conduct compara- tive experiments against directly training LLM agent. In this analysis, we fine-tune Llama-3, 1-BB with 264 seed demonstrations, the same demonstrations we used for training the contextualization module, using behavior cloning (BC) as a baseline. To ensure fair comparison in terms of model scale, we define a Llama-3. 3.1--B agent that performs tasks based on the output of a contextualization module trained using LCoW as the direct comparison group. As shown in Figure 7, both smaller LLM agents, such as Llama-3.1--8 18B, and larger LLM agents achieve higher success rates when using the L.Co-ttrained contextualization module compared to the BC baseline.
Generalization to tasks with unseen type Furthermore, evaluate LCoW's generalization pabilities WebArena benchmark. Specifically, we assess whether the contextualization module trained on specific task types generalizes to tasks belonging to task types unseen during the train-
8

---

Published as a conference paper at ICLR 2025
GPT-4o
Llama-3.17708
Gemini- 1.5-flash
Llama-3.1-88
Raw observation
LCoW
Blaude335-sonmm
50
41.9%
Success rate (%)
35.9%
35.8%
45
30
29.7%
30
Success
20.8%
20
BC (23.65 (23.6%)
14.6%
0
w/o contextualizaaton
LCoW (iter 1)
Seen types
Unseen types
Entire types
Figure 7: Comparison to training LLM agent behavior cloning (BC) WorkArena. We fine-tune Llama-3.1-8B with demonstra- tions as a BC baseline.
Figure 8: Comparison of GPT-4o agent navigating web pages based raw observation and observa- tion contextualized by LCoW in WebArena.
ing. Details on the definition of unseen task types are provided in the Appendix B.2. Among 165 evaluation tasks in WebArena, 117 tasks belong to the seen task types, and the remaining 48 tasks belong to the unseen task types. As shown in Figure 8, LCoW consistently shows 6% improvement in the success rate on both seen task types and unseen task types, compared to the GPT-4o agent without LCOW. We expect that generalization to unseen task types occurs due to the fact that the contextualization module learns to explain the UI elements in the web pages (e.g., the role UI elements in complex web page observation) via LCoW-traing -Caaiirain which enhances accurate decision making of LLM agents even the unseen-type tasks. We also provide additional studies on the generalization ability of LCoW with the WorkArena benchmark in Appendix B.3.
Can LCoW optimize the contextualization module for spe- cific LLM agents? We demonstrate that LCoW effectively optimizes contextualization module to generate context that leads to more accurate decision-making of LLM agents. In WebShop benchmark, we used the contextualization mod- ule checkpoint from each LCoW training iteration to gener- ate contextualized observations for the 1,372 raw observations present in the 397 seed demonstrations. We then calculated the matching reward by comparing the actions pre- dicted from these contextualized observations with the ground- truth actions derived from the demonstrations. Figure shows the average action-maten matching reward across three iter- ations, demonstrating that with each round of LCoW LCoW-training, the contextualization module increasingly produce contextual- ized observations that enhance decision making of the L.M agents used data collection (i.e., GPT-4o, Geminii-L55ffaan Claude-3.5-Sond 3.nnnett.t. This shows the effectiveness of LCoW in optimizing the contextualization module for specific LLM agents. Furthermore, it suggests the potential extend LCoW as a method for indirectly optimizing the behavior closed-sone source LLM agents, where direct opti- mization not feasible, across wide range of text-based decision-making tasks.
2.0
1.5
Reward
1.0
0.5
0
iter iter 2 iter 3
Figure 9: Average action-matching rewards across three iterations on WebShop show consistent in- crease, suggesting that the contex- tualization module is optimized to generate observations that enhance decision-making in LLM agents.
5 RELATED WORK
LLM agents for web automation As LLMs continue to improve and demonstrate remarkable performance across diverse domains, developing web automation agents based on LLMs is gaining growing interest (Zhou et al.. 2023a; Drouin et al., 2024). Recent works have explored various methods to utilize LLMs as agents for automating real-world web tasks. Pan et al. (2024) propose to apply self-refine mechanism (Madaan al., 2023) to improve decision making of agents through

---

Published as a conference paper at ICLR 2025
self-generated feedback. Sodhi et al. (2024) enhance web automation by using LLMs to manage low-level workflows handcrafted by humans, while Wang et al. (2024) extend this approach by introducing agent workflow memory, which extracts reusable routines from past experiences.
Similar to our work, promising direction for enhancing web automation involves integrating a sum- marization module that condenses web page observations, enabling agents to predict actions based on these summarized inputs. For example, Deng et al. (2024) propose MindAct, which consists of an HTML extraction module and an action prediction module. The HTML extraction module ranks individual HTML elements using a ranking language model trained to assess relevance based on user instructions and past actions. Additionally, Gur al. (2024) introduce HTML-T5, special- ized language model for web page understanding pre-trained on large corpus of HTML documents and fine-tuned for extractive summarization.
Automated prompting for closed-source models The quality of outputs from closed-source LLMs heavily depends the prompts used, leading to a substantial body of research dedicated to prompt engineering to elicit more effective responses from these models. (Kojima et al., 2022; Yao et al., 2022b; Lightman et al., 2024). For example, several recent studies have explored au- tomating the discovery of more effective prompting formats (Shin et al., 2020; Zhou et al., 2023b). Mañas et al. (2024) demonstrate that prompt refinement can yield more consistent responses from closed-source models, especially with text-to-image models. Also, Xu et al. (2024) have suggested compressing the input content for L.Mss, focusing on the task of eetrieval-augmenter generation.
LIMITATIONS AND FUTURE DIRECTIONS
Dataset collection procedure We leveraged successful trajectories during training when using LCoW in our experiment. We acknowledge that this be bottleneck in learning completely new tasks that are not covered by the collected successful trajectories. To overcome this, incorporating an LLM-based search algorithm in the trajectory collection phase gather successful trajectories for more challenging tasks can be an interesting future direction.
Contextualization cost LCoW incurs latency in the decision-making process due to the compu- tational cost required to generate the contextualized observation. Considering that most tokens of contextualized observation correspond to extraction from the observation, expect LCoW can be integrated with an efficient decoding strategy (e.g., speculative decoding) for resolving the latency problem.
Generalization ability Although we find that LCoW demonstrates generalization in several cases (e.g.. unseen task types unseen website), acknowledge that we did not observe generalization to a broader range of tasks incorporating various unseen UI elements (see Appendix B.3 for more details), due to limited scale of experiments. We believe that scaling the dataset for the training contextualization module can be promising direction for better generalization. For example, we expect leveraging large-scale web browsing demonstrations covering wide range of websites as seed demonstrations would improve generalization to a broader range of tasks and websites.
CONCLUSION
In this work, we introduce LCoW, a novel approach to enhancing LLM agents in completing web tasks by leveraging language models to contextualize complex web pages into a more comprehen- sible form. Our approach separates the understanding of web content from the decision-making process by training a specialized module that generates contextualized representations of complex web pages, which are utilized by LLM agents for enhanced decision making. Through extensive experiments, we demonstrate that this contextualization module significantly improves the decision- making capabilities of L.M agents of varying scales.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We introduce LCoW, framework for improving decision-making capability of LLM agents for web automation. We caution that LLM agents may cause safety issues such cybersecurity or risks regarding private information, while we believe that LCoW can be valuable for guiding the agents from potential mistakes by providing more contextualized information Additionally, we believe the improved efficiency and capability of LLM agents with our methods can provide social opportunities to improve user interactions by using digital devices for those with disabilities.
the UI elements.
REPRODUCIBILITY STATEMENT
For the reproducibility of our results, we have provided detailed description of our experimental setups and prompts in Section 4.1, Appendix B, and Appendix C. Across the entire experiment, we set the temperature hyperparameter backbone L.L as 0.0 enable reproduction. Additionally, to further facilitate reproduction, we open-source our work.
ACKNOWLEDGMENTS
This research was partly supported by the (Mni(Miy of Science, ICT), Korea, under the Global Research Support Program in the Digital Field program)(RS-22) -R422004000020006668 70%) supervised by the ITTP(nnstiute for Information & Communications Technology Planning Evaluation). This work was partly supported by Institute for Information & communications Technology Planning & Evaluation(IITP)) grant funded by the Korea government(MSIT (RS-2019-J) 91197550755 Artificial Intel- ligence Graduate School (rogram(KAIST), 10%). This work was partly supported by the National Research Foundation of Korea(NRF) grant funded by the Korea governeen(MSITT (No. RS-2024- 00414822, 10%). This work was partly supported by the Institute of Information & Communi- cations Technology Planning & (atptatiion(IIIPPPPRRRCCCnCnnno grant funded by the Korea govemmenttMMSTTTXXTPP-20022 -22
Technology Research Center) 10%)
REFERENCES
Xiang Deng, Yu Gu, Boyuan Zheng, Shijic Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024.
Hiroki Furuta, Kuang-Hou Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction instruction-finetueed finetuned foundation models. arXiv preprint 2rXiv:2305.1/88. 2023.
Izzeddin Gur. Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models, 2023.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. In International Conference on Learning Representations, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 221199222221 22213, 2022.
Hanyu Lai, Xiao Liu. Iat Long long, Shuntian Yao. Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce a large lan- guage model-based web navigating agent. arXiv preprint arXiv:2404.0364 2024.
11

---

Published as a conference paper at ICLR 2025
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference Learning Representations, 2024.
Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06332, 2024.
Kaixin Ma, Hongming Zhang, Hongwei Wang. Xiaoman Pan, Wenhao Yu, and Dong Yu. Laser: Lim agent with state-space exploration for web navigation, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. 2023.
Oscar Mañas, Pictro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aish- warya Agrawal, Adriana Romero-Soriano, and Michal Drozdzal. Improving text-to-image con- sistency via automatic prompt optimization. arXiv preprint arXiv:2403. 1.:00..77000 17804, 2024.
OpenAl. http://oppeaii.commmddeeclllllll
2024.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.0711 07199, 2024.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2020.
Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked Ilm policies for web actions, 2024.
Abishek Sridhar, Robert Lo. Frank F. Xu, Hao Zhu, and Shuyan Zhou. Hierarchical prompting assists large language model on web navigation, 2023.
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory, 2024.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving aarrival-auumenned Ims with com- pression and selective augmentation. In International Conference on Learning Representations, 2024.
Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for llm-based web agents. arXiv preprint ar 2rxv:22400.13855, 2024.
Shunyu Yao. Howard Chen. John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35.20744-20757, 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b.
Shuyan Zhou, Frank F Xu. Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307. 13854, 2023a.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In International Confer- ence on Learning Representations, 2023b.
12

---

Published as a conference paper at ICLR 2025
A QUALITATIVE ANALYSIS
In this section, we first analyze (1) examples of contextualized real-world web pages and (2) the decision-making process of LLM agents with and without LCoW.
A. EXAMPLES OF CONTEXTUALIZEI WEB PAGES
We provide 5 examples of contextualized web page observations, along with task instructions. As demonstrated in the following examples, the contextualization model trained via LCoW not only summarizes lengthy web pages but also provides contexts (e.g., what happens if certain UT element clicked) that enable more accurate decision-making of LLM-based web agents.
Example The contextualized observation provides the necessary information required for sorting the list, including several column headers of the spreadsheet. Especially, it explains the "Personalize List" UI element, which initiates sorting. many cases, LLM agents with- out contextualization do not click the "Personal ze List" button due to lack of knowl- edge about this UI element.
Example 2. This example is drawn from the unseen-type tasks in WehArena, where the GPT-4o agent with L.Co succeeded, while the GPT-4o agent with raw observation failed. In this ex- ample, the contextualization module correctly extracts the rows corresponding to the latest order from the raw observation, thereby informing the GPT-4o agent of the status of the latest order. However, without LCoW, due complex raw observation corresponding to this contextualized version, GPT-4o agent fails to identify the latest order. Although this task belongs to an unseen- type task, table format frequently appears in web page observations in the training tasks, the contextualization module correctly extracts content from complex web pages, thereby improving the decision- making LLM agents.
This example also drawn from WebArena, where GPT-4o agent with LCoW succeeded, while GPT-4o with raw observation failed. In this example, the contextualization module explains that the agent need to click "Profile" link in order to edit posts or submission, thereby leading to correct decision making of L.LM agent. However, given raw observation, the GPT-4o agent consecutively scrolls down the page to find the corresponding post.
Example 3.
Example In this example, the contextualized observation contains an explanation about the UI for keyword search, thereby guiding the GPT-4o agent to find a white desk based search functionality. However, the GPT-4o agent without LCoW repeatedly scrolls down the page and moves to the next page to find the white desk, thereby failing add white desk to the wish list within the maximum step limit.
Example 5. In the task of filling out a form to create new it is crucial to check for detailed requirements and entries before submitting the form. LCoW- contextualized web page includes, thereby guiding LLM agents submit the complete form without mistakes (e.g., missing required entries).
Example (WorkArena)
Instruction: Sort the "users" list by the following fields:- Updates (descending)
Contextualized web page observation:
[a183] grid 'Users.', multieolectable=Falee
This is the main table containing the list of users.
We need to fecus en this element to find ways to interact with the column header.
[a186] rewgreup [a18/]
[a188] columnheader Seelect All'
[a193] columnheader Aufle4 Show column search zow'
[a197] columnheader Usser ID User ID column options'
[a204] columnheader ' "Nae Name colurn options'
[a211] columnheader Email Zmail column options'
[a218] columnheader AAattar Avatar column options'
[a225]
columnheader "Title Title column options'
13

---

Published as a conference paper at ICLR 2025
[a232]
columnheader "Company Company column options'
[a239]
columnbeader "Department Department column options'
|a246]
columnbeader Location Location column options'
[a253] columnbeader 'Time zone Time zone column options'
[a260] columnheader "Business phone Business phone column options'
These are the column headers in the users table.
We need to focus on one of these elements to trigger
the sorting action.
However, the AXTree does not show any direct options to sort these columns.
[a77] button Aufl3e Personalize List' This button might provide options to customize the 1ist view, potentially including sorting options.
Example 2 (WebArena)
Instruction: me status my latest order when arrive.
Contextualized web page observation:
[1415] strong
StaticText 'Recent Orders'
[1416] link 'View All'
[1420] table 'Recent Orders'
[1421] captien "
[1422] rewgroup
[1423] rew
[1424]
columnbeader 'Order #
[1425]
columnheader "Date'
[1426]
columnheader "Ship
[1427]
columnheader "Order Total'
[1428]
columnheader 'Status'
[1429]
columnheader 'Action'
[1430] rowgroup [1431] row "
[1432] gridcell '300000170'
[1433] gridcell '5/17/23'
[1434] gridcell EEma Lepez'
[1435] gridcell '$365.42'
[1437] gridcell 'Canceled'
[1438] gridcel1 rview OrderReorder
[1439] 1ink "View Order!
[1441] link 'Reorder'
This extraction shows the "Recent Orders" section, which contains table with information about recent orders. The table includes columns for Order H, Date, Ship To, Order Total, Status, and Action.
The "Vicw All" link [1416] likely leads to a page with more detailed information about all orders.
The first
the table [1431] contains information about
the latest order:
Order number: 000000170
Date: 5/17/23
Ship To: Enma Lopez
Order Total: $365.42
Status: Canceled
Action: View Order [1439] or Recrder [1441]
14

---

Published as a conference paper at ICLR 2025
Example 3 (WebArena)
Edit my post on Star Trek Starflect Academy series by adding line to the body that says "Every watch makes me feel like kid again.
Instruction:
Contextualized web page observation:
[68] button
"MareeesSrrattaan1e", focused, hasPopup=' menu', expanded=True
This is the currently focused and expanded user account menu button.
It contains the user's username and is likely to lead to profile settings.
[74] 1ist
This 1ist centains the submenu itens under the acceunt menu.
[80] listitem
[81] link 'Profile'
This is the "Profile" link we need to click to the user's s profile settings. It's nested under the user acceunt menu.
These elements are crucial for navigating to the user's profile settings, where can then look for options to edit posts or submissiens,
The "Profile" link [81] is particularly important as it's the next element we need to interact with to progress towards our goal of editing the post.
Example (WebArena)
Instruction: Add white desk my wish list.
Contextualized web page observation:
[1850] heading 'My Wish List'
StaticText 'ou have no items your wish list.'
This element shews the "My Wish List" heading and indicates that the wish list is currently empty.
this because it confirms that we haven't added any items to the wish list yet, including the white desk we want to add.
It's important
[1430] putten 'Add to Wish Dist' StaticText 'uu601''
This button is crucial as it directly allows us to add items to the wish list. The button with ID 1430 is likely the one we need to click to add the white desk to the wish list.
[723] menuitem (ue622 Office Products', hasPopup=/memu'
This menuitem relevant because it's the currently selected categery (Office Products). It confirmm rms that we are in the correct category to find office-related items, including desks.
[269] Section
[2711 LabelText StaticText Aue615 StaticText 'Search'
15

---

Published as a conference paper at ICLR 2025
[274] combobex Que615 Search', autoccomplete="hothh haspopup- listbox', ralae [277] link 'Advanced Search'
[279] button "Search', disabled-Irue
This section contains the search functionslity, which could be useful if need search for "white desk" to find the specific item we want add wish list.
These clements provide the necessary information and functionality complete the user's instruction of adding white desk to the wish list.
Example 5 (WorkArena)
with value of "82733378363227788811""" for field "User ID", value "Wehster-Richard" for field "Last name", value of "Lynn-David" field "First name", value of webster-richard workarena.com for field "Email", a value of "false" for field "Internal Integration User", value of "Outlook" for field "Calendar integration", a value of "(514) 357-8997" for field "Mobile phone", value of "false" for field "Password needs reset", value of "English" for field "Language". and value of "false" field "Web service access only".
Instruction:
Create
Contextualized web page observation:
wa should fecus on the following clements:
[a298] LabelText
[a299] note
StaticText 'Internal Integration User'
[a305] checkbox 'Internal Integraticn User', checked=' false'
This element represents the "Iteernl Integration User" field. It's a checkbex that is currently unchecked (false), which aligns with the user instruction. No action needed for this field.
[a353] LabelText [a354] note "
StaticText 'Calendar integraticn'
[a359] combobox 'Calendar integraticn' value=/Outiook:
hasPopup=' menuu" menu', expanded=False
[a360] menuitem f-- None --', selected=False [a361] menuitem 'Outlook', selected-True
This element represents the "Calendar integration" field. It's a combobox that is already set to "Cutlook", which aligns with the instruction. No action needed for this field.
[a429] LabelText
[a430] nete
StaticText 'Mobile phone'
[a435] textbex 'Mobile phone'
This element represents the "Mobile phene" field.
It's textbox where need to enter ((54)) 357-8997* per the user instruction.
[a327] Label
[a328] nete
StaticText 'Language'
[a333] combobox 'Language' value=1-- Nene --'
saPPopup=mment expanded=False
[a334] menuiten -- None --', selected-True
16

---

Published as a conference paper at ICLR 2025
[a335] menuitem 'English', selected-False
This element zepresents the "Language" field.
It's s a combobox where we need to select "English" as per the user instruction.
[a285] LabelText [a286] note "
StaticText 'Web service access only'
[a292] checkbex 'Web service access only', checked=' false'
A.2 ANALYSIS ON DECISION MAKING PROCESS
We provide the action sequences obtained from rollouts in evaluation tasks. For given instruction, we compare the rollouts with and without the application of LCoW, highlighting the differences in decision-making behavior and task completion efficiency. Specifically, LCoW helps the LLM agent to accomplish task more efficiently without repeating redundant actions (see Figure 10 and Figure 12), and also helps avoid getting stuck by informing proper interaction method regarding UI elements (see Figure 11 and Figure 13). Furthermore, as shown in Figure 14, with the contex- tualization module, the average number of decision-makinn steps by the agent decreases in both Claude-3.5-Sonnee and Llama-3.1 3--00BB agent, allowing tasks to be completed more efficiently.
Instruction: Go to the hardware store and order 9 "iPad pro" with configuration ('Choose the colour': 'Silver', 'Choose storage': )128")
Llama-3.1-70B
Llama-3.1-70B LCoW
Action 1: click [a252] link 'Hardware'
Action 1: click [a252] link 'Hardware'
Action 2: click (a268) link 'PPad pro'
Action 2: click (a268) link 'iPad pro
Action 3: select 'Siiver' from [a198] radiogroup 'Mandatory'
Action 3: click [a184] radio "uf137 Silver'
Action 4: click [a184] radio "uff37 Silver'
Action 4: click [a185] LabelText
Action 5: hover the cursor on [ar184] radio "uf137 Silver'
Action select from [a244] combobox 'Quantityy Action [a285] button
Action [a1155
Action 7: select (a244) combobox 'Quantity'
Action 8: click (a285] button 'Order Now
Figure 10: -lama-3.1-70B agent repeats redundant actions while manipulating UI element for se- lecting silver color, while LCoW helps to decide on actions more efficiently.
Instruction: Add white desk my wish list.
GPT-4o
GPT-4o LCoW
Action click [716] menultem 1ue622 Office Products'
Action 1: click [716] menuitem 'ue622 Office Products'
click [1809] 'Office Fumiture@Lighting(372 item)'
[274]
Search'
click [1789) link 'ue608 Page Next!
Action (279) 'Search'
Action click [1815] link "ue608 Page Next
Action 4: click [1434] button 'Add Wish List'
Action 5: scroll down
Action 6: scroll_dome
Action 7: scroll__dowe
Figure 11: Given complex web page observation, GPT-4o agent repeats moving to the next page and scrolling down the page for finding out white desk, resulting in task failure. In contrast, LCoW enables LLMs to utilize search functionality for finding out the white desk efficiently.
17

---

Published as a conference paper at ICLR 2025
Instruction: recently moved, my address 111 Magnolia Path, Atlanta, GA, 30303, update my information OneStopShopping accordingly.
GPT-4o
GPT-4o LCoW
Action [181] link My Account
Action [219] Account'
Action 2: click [1388] Ink 'Edil Address'
Action clear contents in [1400) textho 'Strnnt Address" Street
Action click [1384] link "Manago Addresses'
Line
Action click [1374] link "Change Billing Address'
Action 4: fill '111 Magnolia Path in (1400) texthox 'Steet Address"
Action '111 Magnolia Path in [14000] textbox 'Street Address"
Street Address: Line
Street Address: Line
Action clear contents (1737) toxtbox CCtty"
Action 6: 'Atlante' in [1737] texibox 'Cily"
Action 'Allenta' [1737] CCty"
Action select 'Goorgia'
[1683) combobox 'StatooProvince
Action select 'Georgia' from [1665) combobox SSlate/Provicc
Action B: clear contents in [1742] textbox Zip/Postal Code
Action '3000'' (1742) textbox ZipiPostal Code
Action '30303 (1742) texibox Zip/Postal Code
Action 10:
'Save Address!
Action click [1752] button 'Save Address'
Figure 12: GPT-4o agent meaninglessly clear the empty lextbox before filling out the address form. In contrast, LCoW explains the status of UI clement (c.g., textbox is empty), thereby enabling more efficient decision making.
Instruction: Post notice virtual meetup Harry Potter enthusiasts July in the books subreddit.
GPT-4o
GPT-4o LCoW
'Submit'
Action 1: click (62) link 'Submit'
Action 2: fill 'Virtual Moctup for Hanyy Potter Enthusiasts on July 8th in
Action 2: 'Join Harry Potter Virtual Meetup July
[137] textbox 'Tile This field required: Action 'Jiin
virtual meetup for Harry Potter enthusiasts on July 8th! We will discuss our favorit bookss in [142] textbox 'odd Action
8th!' [137) textbox 'Title This field required.'
Action 3: fill 'Join us for magical virtual meetup where Harry Potter enthusiasts from around the world come together to discuss their favorite books in [142] textbox 'Bod''
[376) combobax
one...
Action 5: select 'books' from [379] combobox 'Choose one...'
Action 6: press 'ArreDown'' keys while focusing on (378) combobox 'Choose one..."
Action 4: fill 'books' in [379] textbox 'Choose one...
Action 7: press 'ArowDown' keys while focusing on (378) combobox
Action 5: [378] combobox
one....
one...
Action press 'rroWDown' keys while focusing (378) combobox
Action 6: click [453] option 'books'
'Choose one..."
Action 7:
click [396] button 'Create submission!
Figure 13: GPT-4o agent struggles to interact with UI element for selecting subreddit, resulting in task failure. In contrast, LCoW verbally explains the interaction method of each UI element in the web page, enabling task success.
Llama-3.1-70B
15
number of steps (1-20)
10
Claude-3.5 3.5eonnett
15
10
5
Raw observation
LCoW
Figure 14: In order to compare the efficiency of task execution between LLM agent with LCoW and without LCoW, we select the tasks that L.LM agent with L.Co and without L.o both succeed, and visualize the distribution of action steps required for accomplishing the task.
18

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS & ADDITIONAL EXPERIMENTS
B. MAIN EXPERIMENT SETUP
WebShop For the WebShop benchmark, we fine-tune Phi-3-mini-Ima Insini-Inttrcttttt as contextualization module, setting the learning rate, warmup ratio, and batch size to le-5, le-2, and 32, respectively. The module is trained for single epoch over the collected data for each LCoW iteration, and we utilize demonstrations corresponding to 397 individual tasks provided in the WehShop benchmark Comini-1.5-flase as the LLM
as seed demonstrations. In the WebShop environment, we employ agent, combined with the contextualization module during the trajectory collection phase. It is also utilized for sampling optimal contextualization at the initial iteration of LCoW.
WorkArena & WebArena For WorkArena and WebArena, we fine-tune LIama-3.1 I-Sttnrrttttt as an observation contextualization module, and we set the learning rate, warm-up ratio, and batch size as le-5, le-1, and 128. Additionally. we set the training epochs as 4 and for WorkArena and WebArena, respectively. Since no successful trajectories are provided in WorkArena, we col- lected 264 seed demonstrations (i.e., successful trajectories) among 495 training tasks across 33 task types using Claude-3.5-Sonme, and GPT-4o, but we could not collect any successful trajectories corresponding to 10 task types. Summary statistics of the collected seed demonstrations across task types can be found in Table 5. In WebArena, we collected 363 seed demonstrations (i.e., success- ful trajectories) among 647 training tasks (excluding 165 evaluation tasks in WebArena-Lite from 812 WebArena tasks) using GPT-4o and trajectories open-sourced by AgentOccam (Yang et al., 2024). Additionally, as the determination of action matching based on parsing is infeasible due to open-ended actions (e.g., sending a message to the user filling out form), exploit GPT-4o as an action-matchinn evaluator. The prompt used for the cetion-macching evaluator is provided in Appendix C. For WorkArena and WebArena, we use Claude-3.5-Sonie for sampling optimal contextualization at the initial iteration.
B.2 GENERALIZATION EXPERIMENT SETUP
In this section, we first provide details of the generalization experiment in WebArena and WorkArena.
WebArena In WebArena, define unseen-type tasks based the task template used to create 812 individual tasks in the WebArena benchmark. Specifically, 812 tasks in WebArena are all different but created based 190 task templates. There- fore, tasks created from the same task template are different but similar. For exam- ple, "What is the top-1 best-selling product in 2022" and "What is the top-3 pest-selling product in 2023" are from the same task template (i.e., same task type), while "Tell me the email address, name, phone number the customer who has the most cancellations in the history" and "How many reviews our shop received in Apr 2023?" are from the different task tem- plate (i.e., different task type). Among the 165 evaluation tasks, we define the 117 tasks whose task templates overlap with tasks used for training as seen-type tasks, while the 48 tasks whose task templates do not overlap with any training tasks as unseen-type tasks.
WorkArena WorkArena features a two-level task hierarchy: task categories at the top level, and task types within each category. WorkArena provides 7 categories of tasks (i.e., "Dashboard", "Menus", "Service catalog". "Knowledge base", "Forms", "Sort list". and "Filter list"). For instance, "Form" is a task category, and within it, "creating and submitting an incident. report" and "creating new user information are task types. We consider two levels of generalization: 1) Unseen-type tasks, tasks of different type within the same category (i.e., medium-level generalization), and 2) Unseen-category tasks, tasks of a different type and category (i.e., hard-level generalization). As specified in Table 6, among 33 task types, utilize 13 task types remaining 6 task types corresponding to "Filter List" "t" task category as unseen-category tasks. For evaluation, utilize 5 individual tasks from each task type.
task types. 14 task types as unseen task types, and the
19

---

Published as a conference paper at ICLR 2025
B.3 ADDITIONAL GENERALIZATION EXPERIMENT RESULTS
In this section, we provide the result of additional generalization experiments in the WorkArena and WebArena benchmarks.
Generalization to tasks with unseen type in WorkArena We first evaluate generalization to unseen-type type tasks in the WorkArena benchmark, where 70 tasks corresponding to task types are considered unseen-type tasks. As shown in Table 3, LCoW improves GPT-4o and Gemini-1.5 1.5-fash agents by 7.2% and 22.6% success rate in unseen-type tasks, respectively. This is an observation consistent with the WebArena experiment in Section 4.3.
Generalization to tasks with unseen category in WorkArena Furthermore, we verify whether the contextualization module generalizes tasks from entirely different categories (i.e., unseen- category tasks) in the WorkArena benchmark. This requires an understanding of unseen UI elements and webpage configurations, making it more challenging than generalization Specifically, we set the 30 tasks included in Frilter-List task category as unseen-caten -eatery tasks. As shown in Table 3, LCoW fails improve both GPT-4o and Gemini-1.5- -.5-fahhh agents in unseen-category tasks. The main reason for this failure in the unseen-category tasks is that the contextualization module does not extract the necessary UI element required dropdown the hidden menu when manipulating filters in involve any UI elements related to filter functionality, the contextualization module does not learn any knowledge about the UI element during training, resulting in no performance improvement in unseen-caten cattery tasks.
unseen-type tasks.
List--iistrettee tasks. Since the training tasks do not
GPT-4o
Gemini-1.5-
Unseen-type
Unseen-category
Unseen-type
Unseen-category
Raw observation
35.7%
0.0%
14.5%
0.0%
LCoW
42.9%
0.0%
37.1%
0.0%
Table 3: LCoW is shown to be generalized to unseen task types within the same task category on both GPT-4o and Gemini-1.5-faas backbone, but it struggles generalize tasks corresponding to unseen category.
Generalization to tasks with unseen websites We additionally study the feasibility of LCoW for generalization unseen websites in the WebArena benchmark, where tasks are distributed over 6 websites (Shopping, GitLaa, Reddit, Map, Content Management System, and Wikipedia). We evaluate the contextualization module on tasks defined the Shopping website, where the contex- tualization module is trained based tasks in the remaining 5 websites. In this setting, we found that LCoW improves the performance of the GPT-4o agent by 4.3% even in the unseen website, as shown in Table 4. We conjecture that this is feasible because, even though the websites may dif- fer in overall design and layout, common UI elements (e.g., search query input fields and calendar date selectors) are often shared across websites. The presence of shared UI components allows the knowledge gained from one website's task to be transferred effectively to another website, enabling the contextualization generalize to tasks in cralization to unseen websites where unseen UI elements are prevelant is a highly interesting future direction.
websites. We believe scaling LCOW for gen-
(Unseen website)
GPT-4o
Raw observation
17.4%
LCoW
21.7% %
Table 4: LCoW improves the performance of GPT-4o agent tasks corresponding the unseen website. Although websites are different, there are UI elements commonly used across websites (e.g., UI elements for searching in Map or UI elements for filtering results based data in the content management system), which enables LCoW to generalize across different websites.
20

---

Published as a conference paper at ICLR 2025
Task type
Number of seed demonstrations
Multi-chart-value retrieval
Multi-chart-minnen retrieval
12
sungle-cart-value-rettne
12
Single-chart-neg saaeleehart-mmmma retrieval
Create change request
Create incident
Create hardward asset
0
Create problem
14
Create user
Knowledge base search Filter asset list
12
0
Filter change request list
Filter hardware list
Filter incident list
Filter service catalog item list
0
Filter user list
Sort asset list
Sort change request list
Sort hardware list
Sort incident list
3
Sort servicc-catalo list
Sort user list
All menu
15
Impersonation
12
Order developer laptop
15
Order iPad mini
14
Order iPad pro
15
Order sales laptop
15
Order standard laptop
15
Order apple watch
15
Order apple Macbook pro
15
Order development laptop PC
15
Order loander laptop
15
Total
264
Table 5: We collected trajectories from 15 individual tasks for each of 33 task types in the WorkArena benchmark using GPT-4o-0806 and Claude-3.5-sonne agent, thereby collecting 264 successful trajectories. We utilized them seed demonstrations for LCoW.
21

---

Published as a conference paper at ICLR 2025
Task type
Seen / Unseen
Single-chart-valne-reeeee
seen seen unseen-type unseen-lype
Single-chart-minen minmmx retrieval Multi-chart-valle retrieval
Multi-chart-minng mairt- retrieval
Create change request
Create problem
seen unseen-type unseen-type unseen-type
Create incident
Create hardward asset
Create
Knowledge base search
seen
Sort user list
seen
Sort hardware list
Sort asset list
unseen-type
Sort change request list
unseen-type
Sort incident list
unseen-type unseen-type
Sort service-catalog list
All menu
seen
Impersonation
unseen-type
Order developer laptop
seen
Order iPad mini
seen
Order iPad pro
Order apple watch
seen
Order standard laptop
seen
Order sales laptop
unseen-type
Order apple Macbook pro
unseen-type
Order development laptop PC
unseen-type unscen-type
Order loander laptop
Filter list
unseen-category
Filter change request list
unseen-category
Filter hardware list
unseen-category
Filter incident list
unseen-category
Filter service catalog item list
unseen-category
Filter user list
unseen-category
Table 6: Task type split for evaluation of generalization at different levels in WorkArena benchmark.
22

---

Published as a conference paper at ICLR 2025
C PROMPTS
In this section, we provide prompts used across experiments in WebShop, WorkArena, and We- bArena. In WorkArena and WebArena experiments, we utilize identical prompts, where the prompt for the LLM agent is provided by BrowserGym (Drouin et al., 2024). In the WebShop experiment, we utilize the prompt for the LLM agent proposed by ReAct (Yao et al., 2022b).
C.I WORKARENA & WEBARENA
We present the prompt formats for the experiment in the WorkArena and WebArena benchmark.
Prompt for contextualization module
<system prompt>
You are an agent tasked with extracting and refining subset of the webpage's observations based on the content the page and user instructions.
<main prompt>
You are currently on (domain_info) website. Your task is to generate a "Reasoning" and a "Refined observation" based on the provided inputs.
First, review the "User instruction" and "History of interactions" and, then, generate the "Reasoning". Analyze the progress made so far, and provide a rationale for the next steps needed to efficiently accomplish the user instruction on the (domain_info) website.
Second, refine the "AxTree observation at the current time step" into a "Refined observation".
Select subset of the AXTree observation that is essential for completing the user instruction and provide explanations for the corresponding elements in the selected subset.
[Information scurce] User instruction (goal)
History of interactions (history]
AXTree observation the current time step (observation)
Prompt for contextualization module retry phase
<system prompt>
You are an agent tasked with extracting and refining subset of the webpage's observations based on the content of the page and user instructions.
<main prompt>
You are currently the [domain_info) website. Your task to generate "Reasoning" and a "Refined observation" based on the provided inputs.
First, review the "User instruction" and "History of interactions" and, then, generate the "Keasoning". Analyze the progress made so far, and provide a rationale for the next steps needed to efficiently accomplish the user instruction on the (domain infel website.
23

---

Published as a conference paper at ICLR 2025
Second, refine the "AXTree observation at the current time step" into "Refined observation". Select subset of the AXTree observation that is necessary for completing the user instruction.
You may refer the Hints, which censists of the ground truth next action, but do not explicitly mention these hints in your output.
[Information scurce]
# User instruction
(goal)
History of interactions (history)
AXTree observation the current time step [observation)
# Eint
Ground-truth next action: {action)
Prompt for self-contexttnalization
<system prompt>
You are an agent tasked with extracting and refining subset of the webpage's s observations based on the content of the page and user instructions.
<main prompt>
[General instructions] You are currently on the [domain_ infol website. Your task is to generate "Reasoning" and a "Refined observation" based the provided inputs.
First, review the "User instructioa" and "Bistory of interactions" and, then, generate the "Roasoning". Analyze the progress made far, and provide rationale for the next steps needed efficiently accomplish the instruction the (domain_infe) website.
Second, refine the "AXTree observation at the current time step" into a "Refined observation".
Extract a subset of the AXTree observation (e.g., chart, table, menu items) that contains necessary information for completing the user instruction, and explain the extracted elements. Ensure that the information on the elements (e.g., nameric element ID) correctly included.
Please follow the format in the (Reasoning & Refinement example] carefully.
[Information scurce] User instruction {goal)
History interactions (history)
AXTree observation at the current time step (observation)
[Reasoning Refinement example] Abstract example
24

---

Published as a conference paper at ICLR 2025
Here is an abstract version of the answer, describing the content of each tag. Make sure you fellow this structure, but replace the content with your own answer:
<reasoning>
Think step by step. Based on "User instruction,", "History of interaction, and "AXTree observation at the current time step":
1. Provide high-level description of the "AXTree observation at the current time step."
2. Based on the "User instruction" and "History Interaction,
track your progress and provide your reasoning on the next action needed to accemplish the "Jser instruction." </reasoning>
<extraction>
Based on your reasoning, identify the elements (e.g., links, buttons, static text, table row, chart) to focus Then, explain the semantics and functionalities of each extracted element. Ensure that: You do not alter the structure of the AXTree observation. You extract the element ID (id in accurately without any errors. When extracting chart table, you extract the entire chart or table to avoid any confusion or loss information. </extraction>
Prompt for LLM agent
<system prompt>
You are an agent trying to solve a web task based on the content of the page and a user instructions.
You interact with the page and explore. Each time you submit an action it will be sent to the browser and you will receive a new page.
<main prompt>
Instructions
Review the current state of the page and all other information to find the best possible next action to accomplish your goal. Your answer will be interpreted and executed by a program, make sure to follow the formatting instructions.
#F Goal: (goal)
(history)
Refined obscrvation of current step: (refined observation)
Action space:
13 different types of actions are available. noop (wait_ms: fleat = 1000)
Description: Do nothing, and optionally wait for the given time (in milliseconds).
Examples:
noоp() (500)
send_msg_too_ust (text: str)
Description: Send message to the user. You should send a short answer as message and
25

---

Published as a conference paper at ICLR 2025
do not ask questions through message. Examples:
send san__msg_to user (V the city was built in 1751.V)
send htt_mmssgt__usertY (V ())
send_msg_to_user (VNo)))
send_msg_to_user (V/31112V))
send_msg_to_user (VY BengioV)
scroll (delta_x: float, delta_y: flcat)
Descriptien: Scroll horizontally and vertically.
Amounts in pixels, positive for right down scrolling, negative for left or up scrolling, Dispatehes wheel event. Examples:
scrcll (0, 200)
scroll (-50.2, -100.5)
fill (bid: str, value: str)
Descripticn: Fill out form field. It focuses the element and triggers input event with the entered text. It works for <input>, <textarea> and [contenteditable] elements. Examples:
fill ('237', 'example value')
(nexample') fill ('45', 'multi-line "moltt-limmennexmplee))
fill ('al2', 'example with "quotes"))
selest_optionite (bid: str, options: str list[str]]
Description: Select one or multiple options in a <select> element. You can specify option value or 1abel to select. Multiple options Examples:
selected.
select_option// 48', ['red', 'green', 'blue'
'blue')
click (bid: str, button:
literal 'middle', 'right'] 'left', 'Control', 'Meta', 'Shift']
modifiers:
[])
Description: Click an element.
Examples:
click('51')
click('b22', button='rigat")
click(' nutton=/middle" modifiers= |' Shift' [)
dblelick (bid: str, button: l'teral 'middle', 'right']
'lett', rodifiers: Littg
'Control', 'Meta',
'shift']l - [])
Description: Double click element.
Examples:
dblclick('12')
dblclick button='right rightt)
cblclick('178/ button='middle: modifiors=("shile shift') )
hover (oid: str)
Description: Hover over an element.
Examples:
(1b81)
press (pid: str, key_comb: comb: str)
Descripticn: Focus the matching element and press a combination of keys.
It accepts the logical key names that are emitted in the keyboardEvent.
key property the keyboard events: Backquote, Minus, Equal,
26

---

Published as a conference paper at ICLR 2025
Backslash, Backspace, Tab, Delete, Escape, ArrewDown, End, Enter, Home, Insert, Pageloww, PageUp, ArrowRight, ArrowUp, #1 P12, Digito 1igitt, KeyA Key4, etc. You can alternatively speci fy single character you'd like to produce such as "a" or "q" Following modification shortcuts are also supported: Shift, Control, Alt, Meta. Examples:
press press("88" (' 'Backspace')
press a26', control+aa')
press ('as''' "Meta+Shift+t!)
focus (bid: str)
Description: Focus the matching element.
Examples: focus
clear (iid: str)
Description: Clear the input field. Examples: clear
drep dras__nd__droppffrnn bid: str, to bidi str)
Descripticn: Perform a drag & drop. Hover the clement that will be dragged.
Press left mouse button.
Move mouse to the element that will receive the drop. Release left mouse button.
Examples:
drag_and_drop (498')
upload file str, file: str I list/str])
Descripticn: Click an element and wait for a "filechooser" event, then select one or multiple input files for upload.
Relative file paths are resolved relative to the current working directory.
An empty list clears the selected files.
Examples:
[flcad_file('522" 'my imy_receipt..dde
Documents/Lmage.joor uplead_file('$3 63', ['/home/Be /ttp///bbb//tttt/teee
Only a single action can be provided at once. Rxample: fill ('a12', 'example with "quotes"') Multiple actions feedback from the page. Don't execute multiple actions at once if you need feedback from the page.
be executed sequentially thout any
Abstract Example
Hore is an abstract version of the answer with description of the content of each tag. Make sure you follow this structure, but replace the content with your answer:
<think>
Think step by step.
If you need to make caleulations such coordinates, write them here. Describe the effect that your previcus action had on the current content of the page. </think>
<action>
One single actien executed.
You can only use one action at a time.
27

---

Published as a conference paper at ICLR 2025
</action>
Concrete Example
Here is a concrete example of how to format your answer.
Make sure to follow the template with proper tags:
<think>
My memory says that filled the first name and last name, but can't see any content in the form.
I need to explcre different ways to fill the form.
Perhaps the form is not visible yet some fields disabled. need replan.
</think>
<action>
f'al'', 'example with "quotes"!) </action>
C.2 WEBSHOP
We present the prompt formats for the experiment in the WebShop benchmark.
Prompt for contextualizatton module
<system prompt>
You are an agent tasked with extracting and rephrasing a subset the webpage's observations based on the content of the page and user instructions.
<main prompt>
You are currently on the online shopping website. Your task is to generate a "Reasoning" and a "Refined observation" based the provided inputs.
First, review the "User instruction" and "History of interactions" and, then, generate the "Reasoning". Analyze the progress made so far, and provide a rationale for the next steps needed efficiently accomplish the user instruction the online shopping website.
Second, rephrase the "AxTree observation at the current time step" into "Rephrased observation". Select subset of the AxTree observation that is essential for completing the user instruction and provide explanations for the correspending elements in the selected subset.
[Information scurce] User instruction (goal)
Ristory of interactions (previous_acticns)
AXTree observation at the current time step [obs)
Prompt for self-contextuualizatinn
<system prompt>
You are an agent tasked with extracting ing and rephrasing ng a subset. the webpage's observations based on the content of the page and user instructions.
28

---

Published as a conference paper at ICLR 2025
<main prompt>
The current webpage the web shopping site is described the osservaticn.
Evaluate the current progress based en previous actions and current observation.
Determine the next action by reascning based goal and progress.
Condense the observation into concise format, highlighting clickable buttons indicated by [].
Ensure the summary includes only elements relevant to the
goal and not already covered in previous actions.
Focus on elickable buttons indicated [].
Here are a few examples.
*ggoal*:: would like ounce bettle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars **previous acticns**:
1. search[3 cunce bright citrus decdorant sensitive skin] **current observatione::
Back Search
Page 1 (Total results: 50)
Next >
B078GWRC1J
Bright Citrus Decdorant by Earth Mama Natural and Safe for Sensitive skin, Pregnancy and Breastfeeding,
Contains Organic Calendula 3-Ounce $10.99
#078GTKVXY
Ginger Fresh Deederant by Marth Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic
Calendula 3-Ounce
$10.99
B08KBVJ4XN
Barrel and 0ak Aluminum-Free Deodcrant, Deodorant Men, Essential Oil-Based Scent, 24-Hour Odor Protection, Cedar & Patchouli Blend, Gentle on Sensitive skin (Mountain Sage, 2.7 oz, 2-Pack)
$15.95
**rephrased observationev:
Progress: I searched the keyword '3 ounce bright citrus deodorant sensitive skin' to the relvant itens, And now Looking the item list.
Reasoning: the next step is to choose an item satisfyimg the specification of bright citrus deodcrant.
focus
[B0T8GHRC1J]
Bright Citrus Decdorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calondula 3-Ounce
$10.99
*xgoal**: would like a 3 ounce bottle of bright citrus deodorant. for sensitive skin, and price lower than 50.00 dol exprevious actiens*::
1. search|3 ounce bright citrus decdorant sensitive skin]
2. click|B078GNRGN
**current observatione::
Back to Search
Prev
size
29

---

Published as a conference paper at ICLR 2025
travel set (4-pack)
3 ounce (pack of 1) 3-ounce (2-pack) ] scent
assorted scents
bright citrus I calming lavender
ginger fresh simply non-scents ]
Bright Citrus Decdorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Broastfeeding, Contains Organic Calendula 3-Ounce
Price: $10.99
Rating: N.A.
Descripticn
Features
Reviews
Buy Now
#rephrased observation
Progress: searched and and elicked the item relevant to the goal specification. I am looking at the option list. Reasoning: As the goal requires 3-ounce bottle, I can focus on the sizc optien.
most
focus size
travel set (4-pack)
[ 3 ounce (pack of 1) ] 3-ounce (2-pack) ]
**goal**: i would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars *xprvvious acticns*4:
1. search ounce bright citrus decdorant sensitive skin]
2. [18c(
3. click[3 cunce (pack of 1)]
**current observations*:
You have clicked 3 ounce (pack of 1).
Back Search
I < Frev size
travel set (4-pack)
ounce (pack of 1)
3-ounce (2-pack)
scent
assorted scents
bright citrus ]
calming lavender ]
ginger fresh simply nen-scents
Bright Citrus Decdorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
Price: $10.99
Rating: N.A.
Description
Features
Reviews Buy Now
**rephrased observation
Progress: searched and and clicked the item id. Among the option list, and I clicked size option.
30

---

Published as a conference paper at ICLR 2025
Reasoning: According to the progress, have to focus the scent optien next step. can focus en: scent
assorted scents bright citrus ] calming lavender ]
ginger fresh simply nen-scents ]
**goal**; would like ounce bottle bright citrus
deodorant for sensitive skin, and price lower than 50.00 dollars +*previous acticns**:
1. search cunce bright citrus decdorant sensitive skin]
2. elick [H0C8[800GC
3. click13 cunce (pack of 1)
4. click[bright citrus]
**current observationes:
You have clicked 3 ounce (pack of 1). You have clicked bright citrus. Back to Search < Prev
size
travel (4-pack) ounce (pack of 1) 3-ounce (2-pack) ]
scent
assorted scents
bright citrus ]
calming lavender ]
ginger fresh simply non-scents Bright Citrus Decdorant by Earth Mama Natural and Safe for Sensitive skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
Price: $10.99
Rating: N.A.
Description Features
Reviews
Buy Now
**rephrased observationn::
Progress: Based on **observation+. and +*previous actions**,
clicked size option and scent option.
Reasoning: there is more opticns to select and met
all requirements specified in the geal, next step is buy the item. can focus en: Buy Now
Here is the task.
+*goal++:
(goal)
*xprevious actions**:
cneviiousactiona **current observatione*: [obs)
*rephrased observation
31

---

Published as a conference paper at ICLR 2025
Prompt for LLM agent
Webshop
instruction:
would 1ike cunce bottle bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars Search
Action: search[3 bright citrus decdorant sensitive skin]
Observation:
Progress: I searched the keyword '3 ounce bright citrus deodorant sensitive skin' to the relvant itens, And now looking the item list.
Reasoning: Based en the Progress and current observation, the next step is to choose an item satistying the specification. focus
[H078GWRCIJ]
Bright Citrus Decdorant by Barth Mama Natural and Safe for Sensitive skin, Pregnancy and Breditfeeding, Contains Organic Calendula 3-Ounce $10.99
[B078GTKVXY]
Ginger Fresh Decdorant by Earth Mama Natural and Safe for Sensitive skin, Fregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce $10.99
Action: (BTCC(800CC
Observation:
Progress: searched and and clicked the item to be most relevant to the goad specification.
I am looking at the option list. Reasoning: As the goal requires 3-cunce bottle, I can focus on the size option. focus en: size
travel set (4-pack) ] 3 ounce (pack of 1) 3-ounce (2-pack)
Action:
click[3-ouned (pack of 1
Observation:
Progress: searched and and clicked the item id.
Among the option list, and I clicked size option.
Reasoning: According to the progress, have to focus on the scent option as next step. can focus on:
scont
assorted scents bright citrus ]
calming lavender
ginger fresh
simply non-scents ]
Action: click(bright citrus]
Observation:
Progress: Based on **observation** and esprevious actions**, I clicked size option and scent option.
32

---

Published as a conference paper at ICLR 2025
Reasoning: As there is more opticns to select and met all requirements specified in the goal, next step is to buy the item. can focus en:
Buy Now
Action: click [Buy Now]
Now Here is the task.
Instruction:
[instruction]
[Eistory of observations and actions)
Observation:
(observation)
Action:
C.3 ACTION MATCHING EVALUATION PROMPT
We present the prompt formats for the action matching described in Algorithm 1.
Prompt for model- evaluation of action matching
<system prompt>
Your task is to evaluate whether the given two action commands are semantically al ined.
<main prompt>
You will be given
1). *ereference action** which indicates correct action.
2). *xpredicted action** which is predicted by assistant agent
Your task is to assess whether the message in *predicted action+. is semantically aligned with message in the #rrffrence action**. Please make sure you read and understand these instructions carefully.
Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Alignment 1: the predicted action is semantically aligned with the reference acticn.
send_mag_tc_user (' 304') and send_msg_to__sed The percentage of amount of pending orders among entire orders is 308') are semantically aligned.
('a34") c'ick''a34', button='left') semantically aligned.
Alignment 0: the predicted action is semantically not aligned with the reference action.
(2525)) and send send_msg_to_used The requested value is $29') are not semantically aligned. click('a34? and click('a34) butten=' left') are semantically aligned.
Evaluation Steps:
1. Write a simple feedback that assess whether the predicted action is semantically aligned with the reference action.
2. After writing a feedback, write a score that or
should refer to the Evaluation Criteris.
3. The output format should look as follows: "Feedback: (write a
33

---

Published as a conference paper at ICLR 2025
feedback for criteria) [RESULT] (an integer number among or 1) 4. Please not generate any other opening, closing, and explanations.
**reference acticn++: (ref_action] uupeddcced acticn*:; (pred_action)
Feedback:
34


Reviews:
[{"review_id": "tX1cBiz3V6", "reviewer": "Reviewer_mbV1", "summary": "LCoW aims to advance LLM based web agents by adding a contextualization step to the webpage observation in which an LLM reduces the html/raw observation by pruning irrelevant elements and adds contextual descriptive information. This significantly improves the performance of the downstream web agent and sets state of the art results. \n\nThe algorithm works by first collecting successful trajectories as ground truth. Then a contextualizer model (with a prompt instructions) is used to reduce and explain the UI elements. This now contextualized observation is give to a set of LLM agents that produce actions. The contextualized model gets a high reward if the agents give the same action as in the successful trajectory. The best contextualized observation is then collected. Finally the model is trained to produce the \"good\" contextual observations. This can be repeated for multiple iterations. \n\nThe main contributions are:\n- The novel approach to LLM-based contextualization and parsing, enabling state of the art performance on web agent datasets. \n- The algorithm for training the contextualization model\n- The prompt for contextualization\n- Many experiments\n\nTheir results include experiments on\nWebShop and WebArena across multiple LLM agents with strong baselines in WebShop (such as AgentQ and LASER). \nThey also include ablations/analysis on how the action matching reward improves with iterations, how LCoW affects the number of steps required for each task, and comparisons of the original collected trajectories against behavior cloning,", "questions": "Can you add more description of how self-contextualization works? This is the identical contextualization prompt just uses the LLM agent model  (e.g. Gemini-1.5-flash) instead of the trained Phi-3-instruct model. \n\nTable 3 in appendix: Data and caption do not match. 33*15 = 495. Are the numbers the number of successful demonstrations collected? Some more information on the demonstration collection would be helpful. \n\nDoes figure 8 include both successful and failed tasks? -> Are the distributions over the same tasks? \n\nLine 182: “select the one that provides the most relevant context for the LLM agent to accurately predict the next action at as the target.” \n- This could be written more clearly. \n\nWill the model being released?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "LCoW shows a very clear benefit and improvement to LLM web agents. \n\nLCoW shows state of the art improvement on WebShop against strong baselines such as AgentQ and LASER. This is comparable to human expert level on WebShop tasks. \n\nThe experiments are pretty comprehensive showing improvement on different benchmarks with different agents and show continuous benefit for up to 3 iterations of LCoW training. \n\nThis method does not seem like it would add a ton of compute expenses and could be quite practical. \n\nThe paper is also well written.", "weaknesses": "There are a few weaknesses of the proposed method, though some of this is more limitations than reasons to reject. \n\n1) It is unclear how this method translates across websites, domains, and to some extent tasks. Since this involves training the contextualization model, there is potential to overfit to the data available. It would be nice to have some experiments showing that LCoW trained on a few domains generalizes to many other domains. Perhaps LCoW when generalizing to a task on say LinkedIn after only being trained on a couple benchmarks. \n\n2) This is also true for generalizing over tasks as well. Perhaps LCoW fails when extending to tasks that require more contextual reasoning. \n\n3) The training details section notes that action matching based on parsing is infeasible for open-ended actions (such as filling in a text box) and uses GPT-4o to do matching. However, the bigger limitation is on open-ended tasks or tasks that have diverse ways/orders of completing them. How would LCoW when there are many actions that are reasonable? \n\n4) In real-world situations there may be many rollouts that include individual actions that are actually incorrect. LCoW would treat these as correct and may even train the model to drop the truly relevant areas of the page. \n\n5) The limitations section only notes novel UI elements as a limitation. It seems the limitation section should be expanded to cover some of the above concerns as well. \n\n6) This approach also relies on being able to collect successful trajectories, whereas other methods that employ search may be able to extend agent capabilities to new tasks.\n\n7) There are no experiments comparing to code-based html parsers for LLM agents. Though they are undoubtably not as good or there would already be models with performance comparable to LCoW. \n\n8) How long does the contextual observation generation take? In tasks that rely on parsing large amounts of text (e.g. Write a tweet based on this article), regenerating and contextualizing a whole article could be expensive, time consuming, and not necessary. (This should be addressed)\n\nThere are a few nice to have experiments that are not present: \nGeneralization across task type or difficulty\nGeneralization across websites", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EGN1InSK6k", "reviewer": "Reviewer_xrwv", "summary": "The paper introduces LCoW, a novel framework that addresses the challenge of enhancing decision-making capabilities of Large Language Models (LLMs) in the context of web automation tasks. The method distinguishes the comprehension of web content from the decision-making process by training a dedicated module that creates contextualized representations of intricate web pages.", "questions": "1.The reward obtained from multiple LLMs is only used to judge whether the current step correctly predicts the real action. Should the final expectation of task be used?\n2.How does LCoW handle web pages with novel UI elements or layouts that were not encountered during training?\n3.Have any measures been taken to prevent overfitting, particularly given the iterative training process that relies on self-generated data?\n4.Can the web page be partitioned and analyzed through the prompt function? And there is no comparison with the previous intelligent code analysis work.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The paper presents comprehensive experiments on popular benchmarks, demonstrating that LCoW significantly improves the performance of LLM agents across various scales. The success rates surpassing human experts are particularly impressive.\n2.The paper shows that the contextualization module trained with LCoW can generalize well to different LLMs, including those not involved in the training process, which is a strong indicator of the method's robustness.\n3.The proposed iterative algorithm for training the contextualization module is a effective approach that allows for continuous improvement of the module based on real-world interactions and feedback.", "weaknesses": "1.As mentioned in the paper, the contextualization module struggles with web pages containing UI elements not seen during training. This limitation could be a barrier to the framework's real-world applicability, especially given the vast diversity of web page designs.\n2.The contextualization module was trained on a relatively small dataset of fewer than 2,000 self-generated samples. Can the model's ability to generalize to a broader range of web pages and tasks?\n3.The paper does not extensively discuss the potential for overfitting, especially given the iterative training process that relies heavily on self-generated data. There is a risk that the model may perform well on similar tasks but fail to adapt to new, unseen scenarios.\n4.The contextualization module shown in Figure 7 is not intuitive enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1eQqgxm9Pc", "reviewer": "Reviewer_iKgX", "summary": "The paper proposes to contextualize the observation of LLM-based online shopping agents to improve their performance. It trains a task-observation-reliant contextualization module to help locate the most important information on a page and provides explanations. The idea is clever and shows promising results on two shopping benchmarks. However, it doesn't include code or any playing episodes for the reviewers to verify the outcomes.", "questions": "- How do you know the performance gain is enhanced by the LLM's decision-making?\n- Figure 1: How do you select the reported tasks from WorkArena?\n- How do you ensure that crucial elements are not removed during the contextualization?\n- It seems the contextualization module could only be trained with successful trajectories from LLMs. What about those tasks that even those LLMs could not fulfill?\n- The WorkArena contains up to 1,000 task instances, why do you evaluate only 115 tasks? How do you select them?\n- If the contextualization is shopping relevant, do you believe it's less convenient to write several human oracle rules than to train a contextualization module?\n- Figure 8: The mean of the number of steps doesn't seem to differ much. What is the exact number?\n- It seems the behavior cloning baseline is trained with fewer demonstrations than the contextualization module.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The observation contextualization idea is clever, and the training of the contextualization doesn't require human-labeled data.\n- The results reported on the shopping tasks are promising, proving that the idea should work.", "weaknesses": "- I like the observation contextualization idea, but I've seen a highly ranked paper on the WebArena benchmark, a benchmark with a wider type of websites defined other than mere shopping here, using a similar but more general method that doesn't require task-related inputs. I believe the strong reliance on the web observation's format, as you mention in the limitation section, \"it often struggles to provide suitable contextualization for web pages containing UI elements unseen during the training,\" constrains this work's scope on shopping-related tasks only.\n- I would question your results as you didn't include code or episodes for reviewers to verify your conclusions, especially when you only play your agents on a partial WorkArena benchmark, whose results are easily controllable if you only select the tasks where your agents win.\n- I think you have sacrificed the agent's generalizability with the contextualization module specifically trained on the shopping tasks, as you put in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbKdEAywun", "reviewer": "Reviewer_628v", "summary": "The paper proposes training a language model to contextualize complex web pages for improving the success rates of LLM-based web agents. To enable this the proposed method uses the web simulator environments to roll out multiple trajectories and uses multiple LLMs to score the different candidates. This strategy provides a significant improvement over baseline open and closed-sourced models across different benchmarks like WebShop and WebArena.", "questions": "* Explain the performance of different choices of LLM-based contextualization modules\n* Discuss the practical efficacy of these said agents given the cost/tokens for using the reward models and the lack of simulation environment for new unseen web sites.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pipelines for LLM-based web agents are complex and the proposed approach breaks down \"contextualizing\" the web pages separately from decision making ability of the agents. The approach of leveraging simulation data across the web environments to train (fine tune) a small model for contextualizing shows good results. The reward model is in essence a LLM-based judge system across multiple powerful LLMs. The qualitative results are interesting as they highlight how the proposed contextualization module works in removing irrelevant components of the web page.", "weaknesses": "* With respect to generalization capabilities,  the study  can be strengthened by demonstrating performance across the different web environment bechmarks or types of web pages (e.g, instead of picking or holding out 500 examples randomly the type of Web tasks could be  used for creating the train/test held out set). \n\n* Additionally, it is not clear if in the real world environment a simulation environment is available to bootstrap and roll out the candidate sequence of state and action(s).  As listed in the limitations section the power/promise of these agents diminishes given that performance drops when dealing with unseen UI elements. \n\n* For different web benchmarks, different LLMs were used for training the contextualization module. This needs to be explained or justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)