A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Authors: Izzeddin Gur
Hiroki Furuta
Austin V Huang
Mustafa Safdari
Yutaka Matsuo
Douglas Eck
Aleksandra Faust
Venue: ICLR 2024 oral (2024)
Keywords: Web Navigation
Web Automation
Large Language Models
Language Model Agents
Tool Use
Program Synthesis
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a modular language model agents for real-world web automation by leveraging the capability of multi-step planning, long context understanding, and program synthesis in LLMs.

Abstract: Pre-trained large language models (LLMs) have recently achieved better generalization and sample efficiency in autonomous web automation.
However, the performance on real-world websites has still suffered from (1) open domainness, (2) limited context length, and (3) lack of inductive bias on HTML.
We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions.
WebAgent plans ahead by decomposing instructions into canonical sub-instructions, summarizes long HTML documents into task-relevant snippets, and acts on websites via Python programs generated from those.
We design WebAgent with Flan-U-PaLM, for grounded code generation, and HTML-T5, new pre-trained LLMs for long HTML documents using local and global attention mechanisms and a mixture of long-span denoising objectives, for planning and summarization.
We empirically demonstrate that our modular recipe improves the success on real websites by over 50%, and that HTML-T5 is the best model to solve various HTML understanding tasks; achieving 18.7% higher success rate than the prior method on MiniWoB web automation benchmark, and SoTA performance on Mind2Web, an offline task planning evaluation.

Published as a conference paper at ICLR 2024
A REAL-WORLDS WEBAGENT WITH PLANNING, LONG CONTEXT UNDERSTANDING, AND PROGRAM SYNTHESIS
Izzeddin Gur Hiroki Furuta 1:2x1 Austin Huang* Mustafa Safdari* Yutaka Matsuo Douglas Eck! Aleksandra Faust
'Google DeepMind, The University of Tokyo
ezzeddn@@oogges com, furuta@weblab. w.w.ebb...a..uttkoo......... jp
ABSTRACT
Pre-trained large language models (LLMs) have recently achieved better gener- alization and sample efficiency in autonomous web automation. However, the performance on real-world websites has still suffered from (1) open domainness, (2) limited context length, and (3) lack of inductive bias HTML. We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions. WebAgent plans ahead by decomposing instructions into sub-instructions, summarizes long HTML documents into task-relevant snippets, and acts on websites via Python programs generated from those. We design WebAgent with generation, and HTML--T5, a new pre-trained L.M for long HTML documents using local and global attention mechanisms and mixture of long-span denoising objectives, for planning and summarization. We empirically demonstrate that our modular recipe improves the success on real websites by over 50%, and that HTML- T5 is the best model to solve various HTML understanding tasks; achieving 78.7% higher success rate than the prior method on MiniWoB web automation benchmark, and SoTA performance on Mind2Web, an offline task planning evaluation.
for grounded code
INTRODUCTION
Large language models (LLM) (Brown al., 2020; Chowdhery al., 2022; OpenAI, 2023) can solve variety of natural language tasks, such as arithmetic, commonsense, logical reasoning, question answering. text generation (Brown et al., 2020; Kojima al., 2022; Wei et al., 2022), and even interactive decision making tasks (Ahn et al., 2022; Yao et al., 2022b). Recently, LLMS have also demonstrated success in autonomous web navigation by controlling computers or browsers to follow natural language instructions through multi-step reasoning and decision making (Furuta et al., 2023; Gur et al., 2022; Kim et al., 2023).
However, web automation on real-world websites has still suffered from (1) the lack of pre-defined action space, (2) much longer HTML documents than simulated observations, and (3) the absence of domain-specnt specific knowledge for understanding HTML documents (Figure 1). Considering the open- open-endedness of real-world websites and the complexity of instructions, defining appropriate action spaces in advance is challenging. In addition, although several works have argued that recent iitructtion-inetuning LLMs with instruction- finetuning or reinforcement learning from human feedback improve HTML understanding and web automation accuracy (Furuta al., 2023; Kim al., 2023), their architectures are not always suitable to process real-world HTML documents; as presented in Figure 2, HTML tokens of real websites are much longer than those of simulators, and most LLMs have shorter context lengths than the average HTML tokens in real websites. It is prohibitively costly to treat such long documents as inputs directly, or adopt prior techniques such as text- PPah alignment (Li et al., 2021b) or teXt-HTMM. token separation ( (Wang et al., 2022a). To prioritize broader task generalization and model-size scaling, such domain knowledge for HTML documents ignored in recent L.L.s.
"qqual Contribution.
Work done Student Researcher Google.

---

Published as a conference paper at ICLR 2024
Simulated Website
Real Website
Pre-defined Action
Open-ended
Action
Language Model
Agent
Simplified
Long Messy
HTML
HTML
Human Instruction
Figure I: Challenges in real- world web automation. Recent language model agents (Furuta et al., 2023; Gur et al., 2022; Kim et al., 2023; Yao al., 2022b) navigate simulated websites (Shi et al., 2017; Yao et al., 2022a), where the agents manipulate pre-defied actions and receive simplified HTML documents that are easy to parse. contrast, language model agents continue to face challenges in navigating real-world websites, where they must interact with dynamic environments, handle open-ended actions (actions that cannot be pre-determined), and process lengthy HTML documents containing significant amounts of task irrelvaat information.
In this work, we introduce WebAgent, an LLM-driven autonomous agent that learns from self-experience to complete user instructions on real websites by combin- ing canonical web actions in a program space (Figure 3), WebAgent (i) plans sub-instructions for each step by decomposing natural language instructions, (ii) summa- rizes long HTML documents into task-relevant snip- pets based on the plan, and (iii) acts via programming on real websites by grounding iub-instructions and HTML snippets into executable Python codes. We combine two LLMs to form WebAgent: newly introduced HTML-T5, a domain-expert pre-trained language model, for task plan- ning and conditional HTML summarization and Flan-U- PaLM (Chowdhery et al., 2022: Chung al., 2022) for grounded code generation. HTML-T5 has an encoder- decoder architecture and is specialized to capture the struc- ture of long HTML documents better by adopting local and global attention mechanisms (Guo et al., 2022). It is pre-trained using mixture of long-span denoising objective corpus extracted from CommonCrawl. To ground language model agents into real websites, we introduce self-experience supervision, where the domain-expert language models are finetuned with data generated by scripted planningsummmarzatioon and gel--geeeateed programming.
Namber Hm:
Qla Cammonii
Social Media
Neve
B-Commatte
RostEaam Edam
Figure 2: Statistics of HTML tokens among real websites. Compared to simulator (about 0.5K tokens on average). HTML tokens real websites are much longer (from 7K to 14K), which takes up the context length of large language models. As pre- processing,
remove the irrelevant tags (e.g- <script>, <meta>) and keep nec- essary attributes (e.g. id, type, value).
(Tay et al., 2022) on a large-scale HTML
Existing LLM-driven agents often solve decision making tasks with a single LLM conditioned on different prompts per role (Kim et al., 2023; Sun et al., 2023; Zheng et al., 2023), which is, however, not enough for real-world tasks whose complexity is higher than that of simulators. The empirical evaluations reveal that our method incorporating eef-bbootstapped specialist language models improves HTML understanding and grounding, and achieves better generalization than single LLM agent. In real-world web automation, WebAgent significantly increases the success rate by 50%, and error analysis emphasizes that coupling task planning with HTML summarization in specialized language models is essential for task success. Moreover, HTML-TS not only works as a core module for WebAgent but also achieves strong results by itself on the web-based tasks. On MiniWoB++ (Liu et al., 2018; 8; Shi et al., 2017), HTML-T5 achieves 18.7% higher success than previous language model agent (Gur et al., 2022) while also outperforming competitive baselines, such as naive local- ones (Chung et al., 2022). On the Wind2Web (Deng et al., 2023), an offline task planning dataset, HTML-T5 achieves SoTA performance among Synapse (Zheng et al., 2023) with GPT-3.5, and MindAct with FLan-T5-XL and GPT-4 (OpenAI, 2023). In summary, our key contributions are:
global attention models (Guo et al., 2022) and its mnstruction-finttune
We propose WebAgent, integration of two modular LLMs under suff-supervision for real-world web automation. The domain-expert language model handles planning and HTML summarization, and generalist language model generates executable Python programs.
We newly introduce HTML-T5 language model with local-global attention mechanism that is pre-trained with a mixture of long-span denoising objective on a large-scale HTML corpus, curated from Coam,,Crawl, to capture the syntax and semantics of HTML better.
2

---

Published as a conference paper at ICLR 2024
WebAgent notably improves the success rate by over 50% in real websites. When fine-tuned on downstream demonstrations, HTML-T5 itself outperforms prior language model agent by 18.7% in MiniWoB++, and achieves SoTA performance in Mind2Web, even surpassing GPT-4.
2 RELATED WORKS
Web Automation Program
Flan-U-PaLM
Web Automation Web automation is sequential de- cision making task where agents manipulate browsers following given instructions (Shi et al., 2017), such
Decodor
BLb- Sub-Fromuction HTM Seippets
as form filling (Diaz ct al., 2013) or information re- trieval (Adolphs et al., 2022) through the sequence of computer actions (Li et al., 2020; Mazumder & Riva, 2020; Shvo et al., 2021). Prior works have realized the web automation via reinforcement learning (Gur et al., 2019; Humphreys al., 2022; Jia et al., 2019; Shaw et al., 2023), finetuned (Furuta et al., 2023; Gur et al., 2022) or prompted LLMs (Kim et al., 2023; Sun al., 2023; Yao et al., 2022b; Zheng et al., 2023) on the sim- ulated websites (Shi et al., 2017; Toyama et al., 2021; Yao et al., 2022a). However, there are still huge gaps between simplified simulators and real web environments; for instance, the average tokens for HTML pages are about 15 times larger (Figure 2), and pre-defined action space for specific websites is strong assumption that may harm the generalization to diittii-iitiitttionn web pages or instructions.
HTML-TS Encoder
HTML-TS Decoder Finetuned
FRaedhal
Finetuned
Nawipatton
Inatuction
ITML
1ieneyy
Figure 3: WebAgent is combination of LLMs:
HTML-TS for planning and summarization, and Flan-U-Pal.M for grounded program synthe- sis. It is better suited for the real-world tasks; open domain action space, complex natural language instructions, and long HTML doc- uments. See Appendix for examples.
MindAct (Deng et al., 2023) could be the most relevant work, where finetumed language model summarizes the raw HTML document into relk-relevant snippets, and another model predicts the web actions in multi-choice QA format. While MindAct also combines several language models, it has just adopted DeBERTa (He et al., 2021) and Flan-T5 (Chung et al., 2022) for summarization and actor modules, and evaluated on the offline dataset. In contrast, we design HTML-T5, specialized for web-based tasks, to handle long HTML documents. WebAgent leverages HTML-TS finetuned with self- eeffeexeeeeecce for summarization and planning, and Flan- Flan-U-PaLMM as a capable programmer, which enables it to generate open- open-ended web actions and to act on online real-world websites.
Program Synthesis In addition to common LLMs (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023), several works have proposed programming- -ocuuse language models (Chen et al., 2021a; Feng et al., 2020; Li et al., 2022; Wang et al., 2021) and their benchmarks (Austin et al., 2021; Hendrycks et al., 2021a; Lu et al., 2021). Another line of work has investigated the tool augmentation of LLMs (Parisi et al., 2022) by decoding API calls (Schick et al., 2023) or Python snippets to be parsed with the interpreter (Gao et al., 2023). Most works deal with the program synthesis on the static dataset, except for the attempts in robotics (Liang et al., 2023) and game (Trivedi et al., 2022; Wang et al., 2023a), where LLMs output Python JavaScript snippets to command the agents. Similarly, we leverage the ability of code generation as an open-ended action space for web-based agents to manipulate the real website, and demonstrate LLMS can sequentially decode Python selenium codes considering the given sub-instructions and HTML in the prompts.
See extended related works on document understanding and LLM for task planning in Appendix B.
3 WEBAGENT
WebAgent is a new architecture that combines two LLMs to achieve efficient real-world web au- tomation. HTML-TS, a domain-expert LLM, is responsible for predicting the next sub-instruction (planning) and generating related HTML snippets (summarizationn) Flan-U-PaL.N (540B) (Chowdh- cry al., 2022; Chung et al., 2022), is prompted to generate executable Python programs based on the planning and summarization provided by HTML-T5 (Figure 3). This modular two-stage approach enables WebAgent to effectively navigate and process HTML documents.
Users initiate natural language interactions with a clear intent, such as apartment searching. Upon receiving the initial request, HTML-T5 formulates a "go <URL>" sub-instruction, triggering Flan-U-P U-PaLM to generate a corresponding Python program that navigates to the specified website. The raw HTML content of the newly opened page is extracted and fed into HTMLL-TS along with the
Workflow
3

---

Published as a conference paper at ICLR 2024
DrMi-ignnising
Span Length Noisy
<) idm, In, Id=", for, type, =1, dive....
Span Length = -. Meaningful
<form dess=", type-"submitt>, id-"uName,
Encoder
Decoder
Transient Global Atfention
claspe legit farel'
Edter Bull Altrecs
Edder Nesword:
60.00
vipan exies
Local
Atlention
Malllen
Local and Global Attention Mechanism in Enculer Transformer
Figure 4: HTML-T5 consists of (1) local and global attention mechanisms (Ainslie al., 2020; Guo et al., 2022) and (2) mixture of denoising objectives (Tay et al., 2022) with longer-span corruption on large-scale HTML corpus. The local and global attention mechanisms are suitable for the hierarchical tree structures of HTML documents. Because of the sparsity of content tokens in HTML, short mean span length (c.g. μs = 3), often used in prior works (Raffel et al., 2020), only masks less meaningful chunks. Employing longer span length (e.g. f 8) helps pre-trained language models to capture the syntax and semantics of HTML better.
user's 's instruction and previous planning steps. This information is utilized to predict the next sub- instruction and identify relevant reference IDs for extractive HTML summarization. Plan-U-PaLM, in turn, generates Python program based on these sub- aub-instructions and combined HTML snippets. This iterative process of planning, summarization, and program synthesis continues until designated end-of-episode sub-instruction is predicted or the maximum number of iterations is reached.
3.1
HTML-T5
Prior research has shown that general-purpose LLMs, such as T5 (Raffel et al., 2020), Flan-T5 (Chung et al., 2022), and InstructGPT (Ouyang et al., 2022), can effectively navigate web environments (Furuta et al., 2023; Gur al., 2022; Kim al., 2023). However, unlike specialist transformer models (Li et al., 202 lb; Wang et al., 2022a; Zhao et al., 2022), these general-purport LLMs do fully utilize the HTML-specific information that could otherwise lead better understanding of HTML content. To address this limitation, we introduce HTML-T5, pre-trained encoder-decoder language model specifically designed for HTMML-based web automation tasks. HTML--TS carefully merges the generalist and specialist characteristics of language models. It processes HTMI. in text-to-text manner and employs local and global attention mechanisms (Ainslie et al., 2020) in the encoder to capture the hierarchical structure of long HTML inputs. HTML-T5 is pre-trained on a large-scale HTML corpus curated from CommonCraw/ using a mixture of long-span denoising objectives (Tay et al., 2022), and then finetuned it for each downstream task. For WebAgent, employ the self-experience supervision approach to align the model with real websites.
Model Architecture Unlike natural language, HTML documents possess explicit hierarchical structure. This structure comprises elements such as <i <input>> <1 <label>, and <button>, along with their associated attributes like class, label, and id. These elements are defined locally and combined hierarchically to create HTML documents. To model this inherent hierarchy, we replace the common dense attention (Vaswani al., 2017) with local and global attention mechanisms (Ainslie et al., 2020). Local attention restricts each token to only attend to neighboring tokens within a window. Additionally, transient global attention allows each token to attend to tokens beyond its immediate window. This is achieved through the aggregation and normalization of token representations within each window, resulting a global memory representation. Figure 4 describes the concepts of HTMI.- T5; leaf elements in HTML (green) could be processed by local attention, and internal elements (purple) could be compressed into transient global attention, which naturally fits the hicrarchical structure of HTML. Following Long'5 (Guo et al., 2022), we use dense attention in the decoder.
Pre-Training with Mixture of Long-Span Denoising Our pre-training approach for HTML-TS utilizes span denoising objective. This involves randomly masking spans of tokens within HTML document, with span lengths drawn from a Gaussian distribution with a mean of μ.. The objective is then to predict the masked spans using the remaining tokens in the HTML document (Raffel
4

---

Published as a conference paper at ICLR 2024
Modules
real LssLale
social-media
nap
Error Ratio (%)
Plan
Sum
Success
Score
Success
Score
Success
Score
Prugram
Plun
Sum
Flan-U U--L-
10.0
55,3
20.0
25.0
10.0
51.3
36/ 88/11
38/0/ 78
26/12/
Plan-U-PaL.M.P
X
50.0
79.5
20.0
38.3
30.0
73.8
39 / /6/65/1
56 10/30/22
5/5/57
Plan-U-PuLM+S
0.0
45.7
25.0
62.1
15.0
46.3
30/62/0
40/13 100
30/ 20/0
WehAgent
65.0
87.6
70.0
83.8
80.0
93.8
20/33/25
20/7 50/ 5D
10/17/25
Table I: Success rate of real-worll web automation on real estate, social media and map websites. The score stands for the percentage of covered attributes specified in given instructions. WebAgent, with language model modules for planning and summarization, achieves the best success (65%, 70%, 80%, respectively), surpassing other baselines, such as single Flan-U-Pal.MM that with planning language model (Flan-U-Pal that with summarization language model Caaa--U-Pl--M-- Flan-U-Pal plans an open-loop manner (Plan: X) and regular-expression-base retrieval summarizes HTML inputs (Sum: x). The results imply that self-experience supervision notably improves the performance. and task planning should be learned by finetuning domain language models for closed-loop planning, rather than by prompting single LLM for open-loop planning. The error analysis describes the ratio across three types of errors in (real- tate) / (socia soocial-med 11-edii) a) / (map) domains, which also points out that better adaptive planner to decompose the given instructions would contribute to further improvements of WebAgent.
and
Without language model modules, prompted
al., 2020; Tay al., 2022; Ainslie al., 2023). While span length of ب 3 is commonly used, such short spans often mask less meaningful chunks in HTML documents, such as </, id=, or "> (Figure 4), where the signal-to-noise ratio can be significantly lower than natural language text. In contrast, longer spans can contain more semantically meaningful chunks, such as <form class=" or type= s submit t >> Our empirical findings indicate that setting μ E {8, 64) yields the optimal mixture for HTML documents (Section 4.2).
We adopt 4096 input sequence length and 910 output sequence length during pre-training. In total, 15% of input tokens are randomly masked in the denoising objective. For the pre-training dataset, we collect 100 WARC files (April 2019) from the CommonCrawl corpus and remove the non- non-Unicode or alphanumeric-only HTML documents. We then extract subtrees around < <laa <label>>> elements that have a special attribute called for that associates the corresponding label with unique input element in the same HTML document, This pre-processing step improves the quality of the pre-training corpus by focusing only on HTML that is relevant for instruction following and grounding. Our final dataset has 3.41M examples. We pre-train HTML-T5 for 100K iterations following the practice in other T5 models (Chung et al., 2022; Lester et al., 2021). See Appendix C for further details.
3.2 SELE-EXPERIENCE SUPERVISION FOR ALIGNMENT WITH REAL WEBSITES
Gathering example demonstrations for LLMs to understand websites poses a significant obstacle in real-world web automation. While humans can effortlessly execute instruction following on actual websites, manually annotating every planning, summarization, and program synthesis step as detailed above is impractical. To address this issue, we propose self-expee relf-expereence supervision. semi-supervised approach that necessitates minimal human involvement. In this method, manually curated scripts generate planning and summarization steps, while Flan-U-PaL-M is tasked with generating Python programs. Our WebAgent aligns domain-specte language models, such as HTML-T5, with these self-gathered real-world experiences through fine-tuning (Wang al., 2022b). This enables the generalization and alignment of agents to complex real-world tasks.
Instruction Templates We maintain collection of instruction templates that incorporate placehold- ers such as "Show me the way from <start> > to <goal> by <n-th> <transport wwsssporttttt. at ion>> at map website". We sample instructions by randomly assigning values to placeholders from pre-defined key-value pairs.
Scripted Planning and Prompted Programming We employ a rule-based parser to decompose instructions into sequences of sub-instructions; corresponding reference IDs are retrieved from HTML using regular expressions. At each step of the process, Flan-U-PaLM is provided with the sub-instruction and the associated HTML snippets to generate navigation programs that executed through Selenium WebDriver. The success of recorded demonstrations varies, and automating success criteria for real-world tasks remains challenging. To refine the learning experience, we utilize environmental feedback to eliminate critical failures, such as program execution errors, retriever errors, and clearly erroneous URL prefixes (Ni et al., 2023).
5

---

Published as a conference paper at ICLR 2024
map: Show me the way from San Jase to Mountain View by 2nd Cycling at map website?
21 2.S.E. 5.4.A.
View Into day
Trpe fon Dree shark paine duceas FAnd rel.Cro send, clanco
C1ickk Cyst redis byition
drivee Etad Aleneete
by 088 Ma..... "RycLinng' (lidko
1 Loee bepac Neeebain Weerr)
delvee Fond ptlomen By cor peEecTIo, Beeigg wttee
Figure 5: Example episodes of real- world web automation in map domain. Considering the given instruction and HTML, WebAgent predicts the next inbrinstruction and task-relevant snippet, and then synthesizes the Python script (gray), while treating the sub-instruction as comment in the script. See Appendix H for extended figure.
Finetuning for Planning and Summarization HTML-T5, a component of WebAgent, is fine-tuned using self-experience demonstrations gathered through instruction sampling, scripted planning, and prompted program synthesis, as detailed earlier. It utilizes task instructions (e.g. please search 2 bedroom and 2+ bathroom houses in new york, ny with a max price of $7500 on real estate website), iub-instruction histories (e.g. go to real estate website, type in new york into search, click on search, click on price, click on max rent), and raw HTML as inputs. Subsequently, it generates the next sub-instruction (e.g. type 7500 into max renf) and extracts the relevant data-ref f attributes used for retrieving HTML snippets. Section 4.1 demonstrates the significance of integrating HTMLL summarization into sub-instruction prediction for enhancing real- world web automation performance.
3.3 GROUNDED PROGRAM SYNTHESIS
Web automation real- wolld websites faces challenges due to the open-ended action spaces, unlike simplified simulators (Shi al., 2017; ;; Yao et al., 2022a). In contrast previous approaches (Gur et al., 2019; Humphreys et al., 2022; Jia et al., 2019; Liu et al., 2018), real-world web agents cannot pre-define a categorical action space to specify the interactive elements on the websites. To address this open-domain challenge, we introduce the act via programming paradigm in web automation by utilizing the conditional code generation capabilities of LLMs (Chen et al., 202 la; Liang et al., 2023). Provided with few-shot generic examples (such as selecting checkboxes, entering text into inputs, clicking on options, and scrolling etc.) for program generation, the next sub-instruction, and the extracted HTML snippet from HTML-T5, Flan- --aMMM (Chowdhery et al., 2022; Chung et al., 2022) decodes an Python program (Figure 3) executable with Selenium WebDriver, a library for browser automation. This conditional program synthesis requires LLMs not only generate code to follow natural language instructions but also understand the semantics and functionality of HTML elements. We provide several Python snippet examples generated by Flan- Man-U-PaLMM as follows (sub-instructions are treated as comments in the script):
Type
creek, into search
seteer.cind_eeeneee (By. cSS_SELLCCOO,, [ddta-rro= Seivar.Cind__len eleeant EBy.cas__EtsCTOR, "[data-re htnass/s.._tt1111....ee..oo t'valout creak, ca"t
Submit search
deiver.find element (By. _E__BCEE,, [data-ro=="1259"" subbiit
click on the Loeats
drivar.cind_elene (By. CEE__ECC,,, [[datarre==552211
11 Screil down bousing Lype by 200px
12 driver. wxwruta_scriittttttccce.eeeeeeee gecserollPPaoo (document, www.rgaeeeeeeeeeeooo.o ttttessstttteptttettoeee bycooyyyyoo 200
4 EXPERIMENTAL RESULTS
To study how a modular combination of LLMS under self-supervision enables real-world web automa- tion by overcoming open-endedness and long context documents, we execute instruction-followinn tasks on real websites (Section 4.1). In Appendix F, we also test WebAgent et al., 2021b), a static HTML comprehension benchmark, compared to prior transformer models specialized for structured documents (Li et al., 202 Zhao et al., 2022). In addition, we quantify the performance of HTML- TS itself on simulated web benchmark, Mini WoB++, and offline task planning benchmark, Mind2Web (Section 4.2).
WebSRC (Chen

---

Published as a conference paper at ICLR 2024
Span Length
real-estata
MiniWoB++
(no HTML dennssing))
78.07
53.8%
Architectures
Attention Type L
2048
4096
P.8.fx
80.56
55.2%.
Flan TS-Base
Dense
34.0%
35.3%
3,8,64
80.56
55.4%
Long-T3 Ta-Base
Local
13.4%
44.0%
8.64
82.46
57.0%.
Long-T5-Base
Local & Global
53.1%
53.6%
8,32,64
55.6%
8,64,96
81.29
53.6%
16,64
79.97
55.2%
Table 2: (Left) Architecture comparison on MiniWoB++ 12K dataset (Liu et al., 2018) with average success rate over 56 tasks. Local and glohal attention matches to the hierarchical tree structure of HTML, and then improves the success rate by over 18%, compared to the instruction instructionfinetuuee dense attentions (Chung et al., 2022; Furuta al., 2023). (Right) HTML-denoising comparison with different mixtures of span length (Raffel et al., 2020; Tay et al., 2022). We use LongTT- -ase models for pre-training. HTMM--denosing generally improves the performance on offline task planning on real estate website and MiniWoR benchmark. Especially, using longer span lengths (14 {8. 6)) outperforms other choices, including the popular configuration in natural language domain (1r f3,8, 64} Prefix LM objective), which can reduce the less meaningful prediction from shorter spans (e.g. μ = 3), and inject the structural bias of HTML better.
4.1 REAL-WORLD WEB AUTOMATION
Evaluation Methodology We first evaluate WebAgent with the real-world navigation performance under human supervision, at real estate wehsite (a platform for housing), social media website (a network of communities), and map website. These three websites have different properties. real-estate requires long-horizon planning (about 20 steps per episode) for complex form- filling with few page transitions (at least 2 pages), and soci aldiiiii needs shorter plans (about 10 steps per episode) with many page transitions (at least pages) by selecting appropriate hyperlinks on the page. map is the easiest domain with shorter plans and a few page transitions. WebAgent receives natural language instructions (c.g. Can you search for a studio bedroom. I+ bathroom apartments in oroville, ca for corporate housing on real estate website?, most new thread of Python community filtered by Tutorial tag on social media website?), ?), and acts via planning, summarizing by HTML-T5, and then programming by -lan-U-PaLM (Figure 5). Through the celf-experience supervision process, we curate 260 episodes on real estate website, 230 episodes on social media website, and 410 episodes on map website to finetune HTML-T5.
Could you present the
We prepare 20 different natural language instructions (see Appendix G for the full list), and measure the success rate and score for the evaluation. The score represents the percentage of required attributes covered during the episode (Yao et al., 2022a); for instance, (1) apariments for (2) corporate housing with (3) studio bedroom and (4) /+ bathroom located in (5) oroville, ca, can be specified in the instruction. When the agents could search the housing satisfying (1), (2), (5) and not (3), (4), the score is 60 (= 100 x 3/5). If the agents achieve 100 score, that episode will mark as success.
Results For comparison, we prepare three baselines, consisting of language model modules and single LLM conditioned on different prompts per role, such as --an-U-PLLMM (Chung et al., 2022), that with a planning language model M+PM and that with summarization language model U-PLL- If they do not use language model modules, prompted Flan -aa-LM plans in open-loop manner (Plan: x), and eegullar-epression-bassse HTML (Sum: x). Table 1 shows that by leveraging planning and summarization language model modules, WebAgent achieves best 65% success and 87.6% score on real- sell-esstte, 70% success and 85.8% score on social -media, and 80% success and 93.8% score on map, significantly outperforming single UlaM-U-PaLM, or with partial language model modules (most of those achieve about 10 30% success). This result suggests that self-experience supervision notably improves the performance, and closed-loop planning grounded on HTML observations via finetuned domain language models is more suitable for open-ended web automation than open-loop planning with few-shot M.M.s. This trend remarkable in real 1 1-estateee (even Flan-U-Pal.M- MMP achieves 50% success), where the longer planning horizon is needed to fulfill instructions. We also observe that coupling sub-instruction prediction with HTML summarization in language model modules plays a critical role in task success. The development of more capable planning modules to decompose the given instructions adaptively and accurately could help WebAgent improve the performance further.
retrieval summarizes given raw
Error Analysis We also analyze the reason of failures by categorizing them into programming, planning. and summarization errors (Table 1). Programming error does not satisfy the given sub- instructions HTML snippet. Planning error predicts sub-instructions conflicting with user instruc-
7

---

Published as a conference paper at ICLR 2024
Cross-Tack
Coosz-Donain
Train
Fle.
Op. Fl
Step
Flr.
Op. Fl
Step SR
SR
Fle. Are
Op.
Step SR
SR
Synapse (GPF-3355)
ICL
34.4
30.6
2.0
28.8
23.4
1.1
29.4
25.9
1.6
MindAet (
SI.
55.1
75.7
52.0
5.2
42,0
65.2
38.9
5.1
42.1
66.5
30.6
2.9
MindAct-GPTS
ICL
41.6
60.6
36.2
2.0
35.8
51.1
30.1
2.0
37.1
46.5
26.3
2,0
1T-XL (ouns)
SL
69.6
81.7
57.8
10.3
47.6
71.9
42.9
5.6
50.2
74,9
48.3
5.1
Table Offline action prediction performance in Mind2Web dataset. We leverage the cached candidate genera- tion results and direct QA formulation by following Deng et al. (2023). HTML-TS significantly outperforms MindAct with F1an-T5 or GPT-4, and Synapse (Zheng et al., 2023) with GPT-3.5, across task/ /es/////s////mminnnnn generalization in terms of all the metries (element accuracy, operation F1, and success rates).
tions, and summarization enror fails to extract the relevant HTML snippets for given sub-instructions. From the website perspective, the failures on real-ester concentrate in planning because of its long-horizar nature. map also fails in planning when confusing starting point and destination. In contrast, social-medd a tends to fail in programming due to the ambiguous -ub-instructions or summarization including redundant hyperlinks, which results in transiting wrong pages clicking unexecutable elements. From the method perspective, WebAgent often fails in planning by predicting incorrect sub-instruce iobiinstructions (for instance, in 70% of failure episodes), while other baselines more fail in programming or summarization steps. This observation indicates that, through the self-experience supervision, the ratio of programming and summarization errors has decreased while the fundamental difficulty of planning. which requires consistent and accurate prediction over long horizon without error accumulation, still remains.
ate, WebAgent generates incorrect plans in
4.2 ABLATION OF HTML-T5
Models
Data
Success
Diff.
In addition to the evaluation as WebAgent system, we extensively examine HTML-T5 about (i) the general- ization to other websites with Mind2Web (Deng et al.,
SoTA (Zheng ot al., 2023)
99.2%
CC.Net
2
32.0%%
TS-N-TS-XL
12K
48.4%
2023), (ii) the performance on MiniWoB++, stan- dard web automation benchmark (Liu et al., 2018; Shi et al., 2017), and (iii) its architecture and pre-training objective. We adopt 16K tokens for the context win- dow unless otherwise mentioned. We present results on offline task planning, and description generation (Gur et al., 2022) to test HTML understanding on static dataset in Appendix I.
LongTS-Bass Base LongT5-Large
53,8%
0.0
12K
56.3%
0.0
LongT5-XL.
60.4%
0.0
Flan- Maan-LangTS-Rann
54.1%
+0.3
Flan Long Flat-LongTT-XL.
12K
56.1%:
40.2
61.1%
+0.7
HTML.TSS Base (ours)
$7.0%
+3.2
HTML-TT--Lappe (ours)
12K
60.8%
+4.5
HTMI--TS-XI. (aurs)
67.1%
+6.7
Flan-T5-XL Flan-TS-XXI.
75.5%
347K
79.0%
Mind2Web Mind2Web (Deng et al., 2023) is an action-annotated real-world dataset with over 2K in- structions collected from 137 websites. provides action prediction tasks that measure the generaliza- tion of LLMs across the tasks, websites, and their domains (e.g. travel, shopping). Similar to real-world evaluation, the input is a set of HTML snippets, a task instruction, and action history. The output com- prises a target element to interact with, along with the operation, such as click, type, or select an option. We finetune HTML-T5-XX with the training dataset. The performance is evaluated with element accuracy, operation F1, and step success rate that cares for both element and operation correctness. Table 4 re- veals that HTMM--T5 significantly outperforms baselines with F- or GPT-4 (OpenAI, 2023) across task/ mask/website/domaii generalization, which increases element accuracy by 5-8%, operation Fl by 6-8,, and step success rate by 4-8%. This highlights that HTML-T5 can handle real-world web automation tasks better and shows generalization beyond our real-world evaluation with 3 websites.
HTML-TS-XL (ours)
347K
85.6%
Table 3: Average success rate of MiniWoB++ with 56 tasks. We use 12K demonstrations and compare HTML-T5 among supervised- snnervised-fieetunedd methods. HTML-T5-XL. outperforms WcbN-T5-XX XL (Gur et al., 2022), the prior best method, by 18.7%. denoossnn also yields better the success rate than instruction tuned ones. Finetuned HTML-TS with 347K episodes (Furuta et al., 2023) outper- forms Flan-T5- Mlan-T5-XXP XXL (1IB parameters) even with 3B parameters, which gets closer to SoTA with GPT-3.5. See Appendix K for the detailed results.
MiniWoB++ We here evaluate HTML-T5 on 56 simulated tasks in MiniWoB++ using 100 evaluation episodes per task. Inputs are analogous to outputs are adhering to a pre-defined format by the simulator such as click (ref = X). We finetune HTML-T5 with 12K human demonstrations (Liu et al., 2018), and compare the average success rate to prior supervised-learre learned agents (Gur et al., 2022; Humphreys ct al., 2022), LongT5, and its instruction-tineture variants (Chung et al., 2022) 1. Table 3 shows that HTML-T5-XL significantly
evaluation, utilizing HTML documents, while
We finetune LongT5 models with Flan dataset released by Chung al. (2022). See Appendix J.
8

---

Published as a conference paper at ICLR 2024
outperforms WebN-T5, the prior best model, by 18.7%. Notably, demonstrate HeMo-ddnoissng consistently improves the performance on top of LongT5 in all the model sizes, better than instruction- finetuning introduced in prior work (Furuta et al., 2023). Furthermore, we finetune HTML-T5-XX with 347K demonstrations from Furuta et al. (2023), which performs better than 1B-ppaameter Flan- T5-XXL even with 3B parameters, achieving 85.6% success. These prove successfully incorporate domain knowledge on HTML comprehension for web automation into pre-trained language models.
Architecture and Objective We hypothesize that local and global attention mechanisms can capture the hierarchical structures of HTML documents better than dense attention. We compare the web automation performance among 56 MiniWoB++ tasks (Gur et al., 2022). by finetuning HTML-TS with public 12K-episode dataset (Liu et al., 2018). We adopt 2048 and 4096 tokens as input length and prepare Base-size architectures. Table (left) reveals that the combination of local and global attentions achieves the superior success rate by over 18% compared to the mnstruction-finetunee dense attentions (Chung al., 2022; Raffel et al., 2020) and local attention only. Surprisingly, local attention only still surpasses the dense attention by about 9%, which suggests local relation between elements and attributes in HTML are essential for web tasks.
As for pre-training objective in Table 2 (right), HTML-denoising generally improves the performance on offline task planning on real estate website and MiniWoB. Especially, using only longer span lengths (μ {8, 64)) outperforms other choices, including the popular configuration in natural language domain (μx E 83,8, 64) + Prefix LM objective), which can reduce the less meaningful prediction from shorter spans (e.g. / 3), and inject the structural bias of HTML into language models better. See Appendix 1.2 for further results with model scaling.
DISCUSSION AND LIMITATION
Modular Approach with Specialist Language Models We demonstrate it is beneficial to divide web automation into planning. HTML summarization, and code generation, and combine domain- expert language models aligned with self-experience data. Such modular approaches have also been adopted support the inference of LLMs (Xu et al., 2023), multimodal tasks (Zeng et al., 2022). and robotics (Ahn et al., 2022), which, however, might cause additional computational costs and latency.
Broad Generalization across the Internet Because open-loop planning with prompted Flan- PaLM achieves at most 10 30% success, we have demonstrated that self-experience supervision on real websites is essential for planning modules. As we demonstrated Mind2Web, our method could generalize across the internet if we have enough data. It would be expected to collect demonstrations at scale and align larger domain-expert models with them in future works.
Feedback for Program Synthesis We leverage Flan-U-PaLMs with 540B parameters, as a capable program synthesis module via few-shot prompting. Such a large model, however, makes it challenging to reflect the feedback about the errors in generated code, compared to smaller models. We leave it as future direction to incorporate the feedback for program synthesis into larger language models.
Evaluation for Real-world Web Automation Beyond the simulated web environments (Shi et al., 2017; Yao et al., 2022a), we have exhibited WebAgent can follow given complex and sometimes ambiguous instructions real estate, social media and map websites. On the other hand, it is costly to evaluate the performance of autonomous agents in the real world. Automated evaluation with minimal human intervention would be helpful for the scalable development of real-world web agents.
CONCLUSION
We build system for real-world web automation, combining HTML-T5 for planning and HTML summarization and Flan-U-PaLMN for grounded program synthesis. Our proposed WebAgent achieves around 70-80% success on real websites via self-experience supervision, outperforming single L.M approach by over 50%, which suggests dividing the sequence of sub- sub-problems with multiple language models can increase the entire task success. We also propose a scalable recipe for HTML-specialized language models where train local and global attention mechanisms with a mixture of long-spar denoising objectives to capture the hierarchical structures of HTML documents. HTML-T5 not only plays essential role in WebAgent but also achieve the best results variety of bassed benchmarks such as Mind2 and MiniWoB++.. We hope our work contributes to getting us one-step closer to the practical deployment of autonomous web agent systems.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This paper presents encouraging evidence of autonomous agents" potential for deployment on real websites, extending beyond simulated environments. In the foreseeable future, this technology could lead to the development of sophisticated Al assistant tools for computers and smartphones, enhancing productivity and accessibility for society.
While we recognize the promising aspects of autonomous agents, we must also consider the potential for misuse and unintended consequences in their development. As our proposed system is based on LLMs, there is risk of prompt injection. The improper use of web automation could pose cybersecurity threats and expose users to scams. To mitigate these risks, it crucial for researchers, policymakers, and industry stakeholders to collaborate on establishing guidelines and regulations for the development of autonomous agents. Additionally, security research focused on LLM agents will become essential domain for society.
ACKNOWLEDGMENTS
We thank Heiga Zen, Yingjie Miao, Yusuke Iwasawa, Joshua Ainslie, Santiago Ontanon, Quoc V. Le, Zoubin Ghahramani, Jeff Dean, Tris Warkentin for the supports and advises on this work. Hiroki Furuta was supported by JSPS KAKENHI Grant Number JP22J21582.
REFERENCES
Leonard Adolphs, Benjamin Boerschinger, Christian Buck, Michelle Chen Huebscher, Massimiliano Ciaramita, Lasse Espeholt, Thomas Hofmann, Yannic Kilcher, Sascha Rothe, Pier Giuseppe Sessa, and Lierni Sestorain Saralegui. Boosting search engines with interactive agents. In Transactions Machine Learning Research, 2022.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey,, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do can, not say: Grounding language in robotic affordances. arXiv preprint arxiv:2204.016%9 2022.
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs transformers. arXiv preprint htl::200400448.2 2020.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, and Sumit Sanghai. Colt5; Faster long-range transformers with conditional computation. arXiv preprint arXiv:2303.09752 2023.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey. Yanping Huang, Kathy Meier-Hellsterm, Gaurav Mishra, Erica Moreira, Mark Omernick. Kevin Robinson, Sebastian Ruder, Yi Tay. Kefan Xiao, Yuanzhong Xu. Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng. Colin Cherry, Christopher Choquette-Choo Aakanksha Chowdhery. Clément Crepy. Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber. Markus Freitag, Xavier Garcia. Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni,
10

---

Published as a conference paper at ICLR 2024
Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter. Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report. arXiv preprint arXiv:2305. 10450 2023.
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. Docformer: End-to-end transformer for document understanding. In International Conference on Computer Vision, 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry Quoe Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv: 108. (tth:2008.077383 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprini arXiv:2005. 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse. Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint 217.v:2107 2021a.
Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4173-4185, 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann. Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Heier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyum Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slay Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction finetuned language models. arXiv preprint arxiv:2210. 1416, 2022.

---

Published as a conference paper at ICLR 2024
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXtv:2110.14168, 2021.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. arXiv preprint arXiv::804.05633 2018.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2336..000 2023.
Oscar Diaz, Itziar Otaduy, and Gorka Puente. User-driven automation of web form filling. In International Conference on Web Engineering, 2013.
Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. Multi-news: a large- scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv: 749, 2019.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou, Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 2020.
Hiroki Furuta, Ofir Nachum, Kuang-Hur Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetueed foundation models. arXiv preprint arxiv:2305. 1854, 2023.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211. 1ett22211..4333 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. arXiv preprint arXiv:2101.02235, 2021.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 724-736, 2022.
Izzeddin Gur, Urrch Rueckert, Aleksandra Faust, and Dilek Hakkani- -urr Learning to navigate the web. In International Conference on Learning Representations, 2019.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models. arXiv preprint arxiv:2210.033: 10.0445, 2022.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference Learning Representations, 2021.
Dan Hendryeks, Steven Basart, Sauray Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.0990 09938, 2021a
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021b.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.0722 2022.
Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thomton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. International Conference on Machine Learning, 2022.
12

---

Published as a conference paper at ICLR 2024
Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba. DOM-Q-NET: Grounded RL on structured language. In Interaational Conference on Learning Representations, 2019.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arxiv:2303 1749, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, Advances In Neural Information Prucessing Systems, 2022.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808..02226, 2018
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, November 2021.
Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. Structurallm: Structural pre-training for form understanding. arXiv preprint arXiv:2105. //210, 2021a.
Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. Markuplm: Pre-training of text and markup language for visually-rich document understanding. arXiv preprint arxiv:2 2021b.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In Conference on Computer Vision and Puttern Recognition, 2021c
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instruc- tions to mobile ui action sequences. In Annual Conference of the Association for Computational Linguistics, 2020.
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson dAutume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries, In Text Summarization Branches Out, pp. 74-81. Association for Computational Linguistics, July 2004.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304. 11477, 2023.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang, Reinforcement learning web interfaces using workflow-guidee exploration. In International Conference on Learning Representations, 2018.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Tay, Denny Zhou, Quoc Le. Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.1368. 2023.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation, arXiv preprint arXiv: 102. 100.0664444 2021.
13

---

Published as a conference paper at ICLR 2024
Sahisnu Mazumder and Oriana Riva. Flin: A flexible natural language interface for web navigation. arXiv preprint arXiv:2010.12844, 2020.
Shen- Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprini arXiv:2106. 15772, 2021.
Ramesh Nallapati. Feifei Zhai, and Bowen Zhou. Summarunner: recurrent neural network based sequence model for extractive summarization of documents. arXiv preprint arXiv: 1611,04230, 2016.
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-com generation with execution. In International Conference Machine Learning. 2023.
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hamnanch Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling. In International Conference on Machine Learning, 2023.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint wrxiv:2203.02155 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. Proceedings the Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
the 40th Annual Meeting of the Association for
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nip models really able to solve simple math word problems? arXiv preprint arXiv:2 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140): 2020.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Mare van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy
Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with 55x and seqio. arXiv preprint arXiv:2203.17189 2022.
Timo Schick, Jane Dwivedi-Yos Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476. 04761, 2023.
Eva Sharma, Chen Li, and Lu Wang. Bigpatent: A large-scale dataset for abstractive and coherent summarization. arXiv preprint 1:Xiv:/906.033:
2019.
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. arXiv preprint arXiv:2306.00244 2023.
14

---

Published as a conference paper at ICLR 2024
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, 2017.
Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan D. Jepson, and Sheila A. Mcllraith. Appbuddy: Learning to accomplish tasks in mobile apps via reinforcement learning, In Canadian Conference on Artificial Intelligence, 2021.
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaclbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. arXiv preprint arXiv:2305. 10014, 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox. Jesse Thomason, and Animesh Garg. ProgPrompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209. 302, 2022.
Haotian Sun, Yuchen Zhuang, Lingkai Kong. Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. arXiv preprint (.l.:2305.666.. 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether of-hhuhf- can solve them. arXiv preprint arXiv:2210.09261. 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv: 181100077, 2019.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. U12: Unifying language learning paradigms. arXiv preprint arXiv:2205. 05130,, 2022.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tzi Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-Johh Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcan Claire Cui, Marian Croak, Ed Chi, and Quoe Le. Lamda: Language models for dialog applications. arXiv preprint ar arXiv:2201.08239, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothéc Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprini 2rxiv:2302.13977 2023.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint 1rXiv:2105.13231 2021.
Dweep Trivedi, Jesse Zhang, Shao-Hur Sun, and Joseph J. Lim. Learning to synthesize programs as interpretable and generalizable policies. arXiv preprint arXiv:2 /3643, 2022.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change). arXiv preprint arXiv:2206. 10498, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
15

---

Published as a conference paper at ICLR 2024
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint ar arXiv:2305.16290 2023a.
Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu. Webformer: The web-page transformer for structure information extraction. arXiv preprini urXiv:220..00217. 2022a.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint 2rXiv:2212. 2022b.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-awarr unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696-8708, 2021.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, In International Conference on Machine Learning. 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou, Chain of thought prompting clicits reasoning in large language models, arXiv preprint arXiv:2201.11900 2022.
Canwen Xu, Yichong Xu, Shuohang Wang. Yang Liu, Chenguang Zhu, and Julian McAuley. Small models are valuable plug-ins for large language models. arXiv preprint arXiv:2305.08 08888, 2023,
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. LayoutLM: Pre- training of text and layout for document image understanding. arXiv preprint arxiv: 1912. 13318, 2019.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. arXiv preprint 2rxiv:2207.01206 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Fed- erico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXhv:2204.00596 2022.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, 2020.
Zihan Zhao, Lu Chen, Ruisheng Cao, Hongshen Xu, Xingyu Chen, and Kai Yu. TIE: Topological information enhanced structural reading comprehension web pages. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Technologies, pp. 1808-1821, 2022.
Longtao Zheng, Rundong Wang, and Bo An. Synapse: Leveraging few-shot exemplars for human- level computer control. arXiv preprint arXiv:2306.07863 2023.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. Mediasum: A large-scale media interview dataset for dialogue summarization. arXiv preprint arXiv:2103.06410, 2021.
16

---

Published as a conference paper at ICLR 2024
APPENDIX
A NOTE FOR REAL-WORLD EVALUATION
The development of autonomous agents should consider the security and safety aspects. the real website evaluation, we have carefully conducted the experiments under human supervision in case undesired behaviors happen. We use Selenium WebDriver 2, popular library for browser automation, and limit the access per second not to stress the server. We have anonymized the real websites we tested on for safety and privacy concerns.
B EXTENDED RELATED WORKS
Document Understanding Understanding structural documents has been a practical challenge for transformer-basee language models, Prior works employ layout-informed tokens (Xu et al., 2019) even multimodal tokens from visual inputs (Appalaraju et al., 2021; Li et al., 202 2021a;c), la;c). Especially, for the documents written in markup languages, text-XPath alignment (Li et al., 2021b), token separation between text and HTML (Wang et al., 2022a), or extra topological information of HTML (Zhao et al., 2022) are proposed to leverage their syntax better. On the other hand, such domain knowledge conflicts with recent generalist and scaling trends around LLMs (Anil et al., 2023; OpenAl, 2023). Because web agents require the desirable to reconcile specialist aspects for HTML documents with generalist capabilities for natural language tasks. In this work, we design HTML-T5 incorporate the structural bias of HTML by combining local-global attention for the encoder and a mixture of long-span denoising, while it can solve cntruuctionfollowinn better downstream web-based tasks.
HTML understanding. it also would be
LLM for Task Planning The prior knowledge of commonsense in LLMs has allowed us to leverage them for variety of task planning. For instance, Huang al. (2022) propose LLM agent that generates natural language plans in an open-loop manner. Nottingham et al. (2023) and Wang et al. (2023b) perform sequential closed-loop planning on MineCraft. Singh et al. (2022) decode robotic plans with pythonic text, and several works incorporate planning definition and domain language into the outputs (Liu et al., 2023; Silver et al., 2023; Valmeckam et al., 2023). On the other hand, WebAgen leverages finetuned specialist language models and performs closed-loop planning coupled with HTML summarization by decomposing given instructions. We empirically prove that our system is superior to open-loop planning with a single generalist LLM with prompting.
Phttps:/ hwwps////wwww.///... sel dev/
17

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION DETAILS OF HTML-T5
We use the implementation local and global attentions released by Guo et al. (2022) Following Guo et al. (2022), we the local radius to 127, and block size for transient global attention to k = 16. For the pre-training objective, similar to Tay et al. (2022), we construct the mixtures and then use long mean span lengths: μ E {8, 64), and all the denoising ratio (percentage of masked tokens in the input sequence) is set to 0.15. We adopt 4096 input sequence length and 910 output sequence length during the pre-training. The batch size for training is set 128. We train the models with 100K iterations following other pre-training strategies for T5 families (Chung et al., 2022; Lester al., 2021). We leverage SeqlO (Roberts al., 2022) and TSX (Roberts al., 2022) library to manage the training pipeline. We also use SentencePiece (Kudo & Richardson, 2018) with 32K tokens from C4 dataset (Raffel et al., 2020) as tokenizer. During the downstream finetuning, we adopt 16K tokens for the context window unless otherwise mentioned. We have used cloud TPU-v3, which has a 32 GiB HBM memory space, with 128 cores for the experiments.
For the dataset, prepare 100 WARC files (April 2019) from CommonCrawl", and pre-process the raw HTML by removing non-Unicode and alphanumeric documents and extracting subtrees around <1 abel >> elements that have for attribute, reduce the noise in training corpus, which results in about 3.41M examples (Table 5).
# of tokens
of examples
Average
90th
Max
3.41M
1020
4566
7627
Table 5: Statistics of CommonCraw/ HTML corpus for self-supervise denoising pre-training of HTML-TS. Input lengths are measured in tokens by Kudo & Richardson (2018).
D WEBAGENT EXAMPLE FLOW IN REAL-ESTATE WEBSITE
Previous Planning Steps reddestalewebsiee.cm
History of previous predictions
Previous Snippet IDs
Inputs
Instruction
Can you bedroom aportment Diego
Language instruction and HTML page
HTML Document
<hamd. deta-redin*0* iddet www.teetaaeeeeee to Mattanceee"ii 1irooddio
c2ttee data-zed =11=== "Looteadee""oodi -f erbvediv
Ide"beade
eypo "button"ses daca-rof dal""taaeddeeTooo wow.//oooooo///aoaa//// scclbbll data-raf "18" Jabel Fortteno vatenoc /ppan
Sub-Instruction
Typa Son Diego into search.
Planning and Snippets
Snippet References
data-ref-122, data-ref-156
Outputs
Program
coarch
driver-
(Bty
"egui teeeeeen octegp te---etppeteeeeee
[beebooder-vieiillliyy l
Python program
driven.fion fidd lemennt city. gmazcroe,
waw.okkBarchtoooooo oclcup [1 (1a-snippet "exce") Ixebooder-visibliityy "1 "end keys ("sa a disco, ea":
Selenium
Figure 6: An example flow with planning, summarization, and grounded program synthesis in the real estate website. HTML-TS iteratively predicts decomposed sub-instruction and task-relevant snippet (orange) in closed-loop manner, conditioning on the HTMI. documents, instruction (ycllow), and history of past predictions (green). Flan-U-PPl.M is prompted with sub-instruction and snippet (orange) to decode python programs (bluc).
hhttps://githb... com/ hwwp:ttwww.ooooooooooooooooooooe reegeeeereearcc/oongeee httpps://comett .org/
18

---

Published as a conference paper at ICLR 2024
E REAL-WORLD WEB AUTOMATION WITH DIFFERENT GENERALIST LLMS
We compare different generalist LLMs as a module of WebAgent among model-size vari- ants (Flan-PaLM-SBB, Flan-PaLM-62B, Flan-U-PaLM-540B), and publicly accessible LLM (gpt-3. 30t-3355tuuzz turbo). We test those models on map website following the same 20 instructions in Appendix The results in Figure 7 imply that the performance gpt-3. 5-turbo are the same (80% success, 93.8% score), and Flan-PaLMM-66 -2B (60% success, 86.3% score) is lower than Flan-U-PaLMM54003 which caused by the inaccurate program syn- thesis. In addition, LLM with sufficient program synthesis capabilities could be integrated into WebAgent, including Flan Aan-U-PaLMM-541
Mran-U-PaLM-544 and
M--8B could not generate proper programs all. We believe that any
100
86.3
80
80.0
80.0
60
60.0
60
.50
50
40
40
25
25
25
20
0
Success
0.0
Score
Program
Plan Error Analysis (%)
Sum
Performance (%)
Flan-PaLM-8B
Flan-PaLM-62B
Flan-U-PaLM-5408
gpt-3.5-tur
Figure 7:
Performance (left) and error analysis (right) of real-world weh automation with different generalist
LLMs. We compare model-size variants (Flan-PaLM-888 PaLM-8B, Flan-PaLM--225 MHaa-U-PaLLM---E and public
LLM (-ttbo))-
5-turbo) on map website.
WEBSRC: STATIC HTML COMPREHENSION
To emphasize the advantage of our modular approach, we test WebAgent on static website compre- hension benchmark, WebSRC (Chen et al., 2021b), which is a contextual QA dataset with HTML documents. The questions require an understanding of the spatial and logical structure of websites, and the answers are either text span HTML or yes/no. For the comprehensive evaluation, WebSRC has three different types of websites, KV, Comparison, and Table. KV task is a value extraction from the attribute key. Comparison task has several entities with the same attributes. Table task requires a structural understanding with header columns and values in the row. We finetune HTML-TS for snippet extraction to predict data-ref corresponding to the answer and use dev set for the evaluation.
As did in real-world web automation, HTML-T5 first predicts data-ref attribute of task-relevan snippet from the input HTML document. To make sure there is enough context, we extract the snippet from the predicted element the two-level-up via XPath. If it exceeds the context length of Flan- U-PaLM, we limit it into parent elements. If it still does not work, we truncate the end of extracted snippet fit within the token budget. Because snippet extraction in table structure often loses the context to solve question-answers Plan-U-PaLM predicts the answers seeing 5-shot examples.
we just truncate HTML documents for Table tasks.
As shown in Table 6, single LLM, such as Flan-U-PaLM or HTML-T5, has struggled to the limited context length model capacity. In contrast, WebAgent, our LLM-collaborative approach, enhances the performance from both single generalist and specialist LLMs, and shows competitive results with strong baselines. This demonstrates that modular L.MSs work complementarily to each other. Figure 8 presents the performance comparison different types of websites (KV, Comparison, Table) among MarkupLM (Li et al., 2021b). TIE (Zhao et al., 2022), and WebAgent. WebAgent is better at Comparison tasks, but inferior to structural understanding for KV and Table tasks, compared to other baselines, which suggest that generalist LLMs are still not suitable for recognizing structural data such table.
19

---

Published as a conference paper at ICLR 2024
95
85
34.75
82.55
62.28
90
90.82
80
00.32
63.79.34
86.29
83.24 63,
75
80.6
72.20
70
75
65
64.49
70
61.98
KV
Compare EM
Table
KV
Compare F1
Table
MarkupLM
TIE
WebAgent
Figure 8: The performance comparison different types of websites in WebSRC dev
Models
EM
F1
T-PLM (Chen al., 2021b)
61.67
69.85
H-PLM (Chen et al., 2021b)
70.12
74.14
V-PLM (Chen al., 2021b)
73.22
76.16
M-rage (I.i et al., 2021b)
74.43
80.54
TIE-Large (Zhao et al., 2022)
81.66
86.24
Flan-U-Pal.N
40.01
47.56
HTML-T55-L
73.09
76.66
HTML-T5-XL.
74.73
78.73
WebAgent
75.50
85.75
WebAgent (oracle)
76.91
86.64
Table 6: Evaluation on WebSRC (Chen et al., 2021b) with dev set. WebAgent, our collaborative LLMs, enhances the performance from both single generalist (Flan-U-PaLMM U-PaLM) specialist LLMs (HTML-T5), WebAgent (oracle) oracle snippets that guaranteed include the answers, instead of those predicted by finetuned HTML-TS.
20

---

Published as a conference paper at ICLR 2024
G LIST OF LANGUAGE INSTRUCTIONS FOR REAL-WORLD WEB AUTOMATION
real-estate
can you search studio bedroom. 1+ bathroom houses in escondido, ca for corporate housing and price less than 12100 on real estate website.
2. can you find me a studio bedroom, bathroom townhomes hollywood, ca and price less than 14600 on real estate wehsite.
can you search
studio bedroom, bathroom condos in inglewood, for senior housing and price less 8700 on real estate website.
4. would like to search
studio bedroom, I+ bathroom houses in compton, ca and price more than 1200 for corporate housing on real estate website.
5. can you search for studio bedroom, bathroom apartments in orovillc, for corporate housing on real estate website.
6. find me studio bedroom, 1+ bathroom houses modesto, ca real estate website.
7. can you search for studio bedroom, 1+ bathroom condos in redwood city, ca for student and price more than 1900 real estate website.
8. find me bedroom condos in santa clara, and price between 1600 and 7400 on real estate website.
9. find me a bedroom, 3+ bathroom apartments in martinez, ca with min price 1800 on real estate website.
10. can you find me bedroom, 2+ bathroom townhomes concord, ca and price more than on real estate website.
can you find me studio bedroom, 2+ bathroom apartments in san diego, ca and price less than 9300 on real estate website.
find me studio bedroom houses in novato, ca price between 1500 and 6700 on real estate website. 13. can you find me studio bedroom, any bathroom townhomes in petaluma, and price than 1000 on
real estate website.
14. search for 1 bedroom apartments in modesto, and price more than 1000 real estate website.
15. find me bedroom, 2+ bathroom apartments in watts, ca for senior housing less than 6300 real estate website.
16. can you find me I bedroom houses in victorville, that have dog friendly, furnished and price more than 700 real estate website.
17. need hedroom, any bathroom condos in inglewond, ca and price more than 1000 on real estate wehsite.
18. find me a 2 bedroom, 2+ bathroom apartments livermore, ca on real estate website.
19. can you find
bedroom apartments in santa clara, ca that has parking and price less than 10300 on real estate website.
20. can you search for 2 bedroom condos in oakland, ca real estate website.
social-media
Show me most hot thread r/google at social media website.
2. Can you point out the most hot thread in t/learnpython at social media website.
3. Could you check the 1st hot thread in r/artificial at social media website.
4. Can I check the most hot thread in Taiwan on social media website.
Show me first new thread r/facebook social media website.
6. Present the most new thread of r/Python tiltered Tutorial flair social media website.
7. Could you check the 1st new thread in r/facebook at social media website.
8. want to read the 1st hot thread from r/Python tagged as Daily Thread at social media website.
Present the most hot thread r/google filtered by Info Mod Post flair on social media website.
thread in v/eammachinelearing filtered Help flair social media website. 11. Can you point out the first hot thread in r/deeplearning at social media website.
10. Show me the most
12. Could you check the 1st hot thread in r'machineleane mmacchinleammnngnews at social media website.
13. Present the most hot thread r/artificial filtered by News flair on social media website.
14. Please find me the first hot thread in r/facebook social media website.
15. Present the most new thread of r/machinelcamingneww filtered by Startup News flair social media website.
16. Show me the most hot thread in r/artificial filtered by Al Art flair at social media website.
Could you check the first new thread r/facebook social media website.
18. want to read the most top thread from r/google tagged as Info I Mod Post social media website.
19. Show me the most new thread in r/startups filtered by Share Your Startup flair at social media website.
20. Could you check the 2nd new thread in s/facebook at social media website.
21

---

Published as a conference paper at ICLR 2024
map
Show me the way from San Jose Mountain View by 2nd Cycling at map website.
2. Please show the way from The Painted Ladies San Francisco Zoo with 3rd Best option map website.
3. Could you tell me the path from California Academy of Sciences de Young Museum by 1st Transit map website.
4. Could you tell me the way from Union Square The Painted Ladies with 2nd Cycling option map website.
5. Please present the way from Chappell Hayes Observation Tower San Jose with 2nd Walking option map website.
Please present the path from Jack London Square to Emeryville by 2nd Cycling at map website.
7. I'd like move The Midway from Children's 's Fairyland by 1st Cycling map website.
8. I'd like move Chase Center from San Francisco Oakland Bay Bridge with 2nd Transit option map website.
9. want
Pier 39 from Berkeley by 3rd Cycling map website.
10. want to go Emeryville from Mountain View with 2nd Cycling option map website.
11. Can you point out the way from San Mateo to Stanford University by 2nd Cycling at map website.
12. Could you point out the way from Palace of Fine Arts to UC Berkeley by 1st Cycling at map wehsite.
13. Point out the way from The Painted Ladies to San Francisco Museum of Modern Art by 2nd Driving at map website.
14. Could you find the path from Union Square to Palo Alto by 1st Cycling map website.
15. Please check the way from San Jose to San José Mineta International Airport with 1st Walking at map website.
16. Check the path from San Francisco 7oo to Berkeley with 1st Cycling at map website.
17. I'd like to check Parking Lots along the way from Stanford University to The Painted Ladies with Best option at map website.
18. Check Gas stations along the way from Young Museum to Oakland with Driving option at map website.
19. Please show me Hotels along the way from Palace of Fine Arts to Berkeley by Transit at map website. Check Gas stations along the way from Bay Area Discovery Museum to Santa Cruz with Best option at map website.
H EXAMPLE EPISODE IN REAL-WORLD WEB AUTOMATION
22

---

Published as a conference paper at ICLR 2024
cilck 2nd trip de driwer find etenett bay cse SELRCNO (itipl" clickk
cos BELECCTON 4eyeling eelookt
1:22
Click cycling madlo
driver. find elemnty
dei drives find clusett (by. C89 SELECRON clecll
aend Sapa 15an Joeer)
Figure Exampie episodes of real world web automation map domain.
type Joan Loto stacting point
dri vor find elowett
map: Show me the way from San Jose Mountain View by 2nd Cycling at map website?
by C59 SELECTOR.
cleari)
Nomm ba VLav Letto mearch drivez: find sleract Iby. cas PELMCTOR drivez. find elerant
by C88 R0 rend baye (MMontaiin Wen'')
Go to nap wabsit to Griver- gett thttpoo /f Jawe. (aap wobe 1.) (coX"")
23

---

Published as a conference paper at ICLR 2024
1 EXTENSIVE ABLATION OF HTML-T5
1.1 DATASET AND INITIALIZATION
To test our recipe described Section 2.1, we compare the different dataset and model initialization for pre-training downstream task performances; offline task planning on real real-estate and average success rate on MiniWoB with 12K dataset. We use Base-size models for the experiments. For HTML-dcnoising, subtree extraction around label elements on the documents. We also compare the initialization of base architectures before HTML -eddooiiiss from scratch with pre-trained models PEGASUS objec- tive (Zhang et al., 2020) that is a masked important sentence prediction from long-context paragraph. Table 7 reveals that snippet extraction on HTML corpus improves downstream performances since such a pre-processing can reduce the noise in raw HTML. Moreover, initialization with PEGASUS treitrained weights is essential for HTML-T5, because of the long-context and instruction-folle following nature of HTML-based based tasks.
prepare the corpus from CommonCrawl with (Extracted) without (Raw)
CC-HTMI PEGASUS
real -estate
MiniWoB++
Raw
80.56
56.7%
Extracted
67.11
49.1% 1%
Extracted
82.46
57.0%
Table 7: Ablations of HTML-T5--Bi dataset quality and initialization. We evaluate offiine task planning real-estate and average success rate MiniWoB with 12K dataset. For HTML-denoising, we prepare HTML corpus from CommonCraw/ with (Extracted) or without (Raw) subtree extraction around label elements. We also compare the pre-training of base architectures with PEGASUS objective (Zhang et al., 2020) before HTML--enoising The results imply that PEGASUS pre-training is critical for the architectures and pre- processing with subtree extraction improves the downstream performance on HTML-based tasks.
1.2 OFFLINE EVALUATION ON TASK PLANNING WITH MODEL SCALING
We compere the offline task planning performance between HTML-T5 and LongTS (without HTML- denosing) with different model sizes; with Base (220M parameters), Large (770M parameters), and XL (3B parameters). As described in Section 3.1, the models predict the next sub-instructions in closed-loop manner considering the current HTMI observations, user instructions, and previous sub-instruction histories as inputs. For offline task planning evaluation, we use the demonstrations on real-estate website; preparing 1 130 demonstrations and splitting them into train (90%) and test splits (10%). We report the best per-step exact match accuracy in test set.
Table shows that HTML-T5 outperforms LongT5 on the accuracy of sub- sub-instruction prediction, which demonstrates that HTML HTML-denoising pre-training captures the structural bias of HTML better without sacrificing the ability to understand natural language instructions. This also implies that our proposed HTMLL-denoising can scale to larger-size models consistently.
Models
real -estate
Diff.
LongTT-Bass
78.07
0.0
LongTTS-Lage
82.89
0.0
LongTS-XL
81.29
0.0
HTML-T5- -T----
82.46
+4.39
HTML--T--arge
83.63
40.74
HTML--T5-XI.
83.92
42.63
Table 8: Accuracy of offline evaluation task planning. We leverage the demonstrations in real-estate wehsites. Compared to original LongT5, and as scale model size, HTML-T5 improves the accuracy of sub-instruction prediction.
24

---

Published as a conference paper at ICLR 2024
1.3 DESCRIPTION GENERATION
We also investigate the capability of HTML-T5 on static HTML comprehension tasks, as well as interactive decision making tasks. We Description Generation benchmark (Gur et al., 2022), where the models generate the textual description of elements, typically used for accessibility purposes and annotated with special attribute the HTML schema known for. We evaluate the understanding the structure of HTML as would appear to a user, despite not having access to the rendered website directly.
We compare LaMDA (Thoppilan et al., 2022), T5, LongT5, and HTML-TS with respect to accuracy. BLEU (Papineni al., 2002), and ROUGE-1 (Lin, 2004) score. As shown in Table 9, local and global attention mechanisms, underlying between LongT5 and HTML-T3, could almost solve the benchmark by improving the previous best performance by over 10%, with still improved performance model size increases. Compared to the effect of local-glote -ooaall attention. HTML-TS marginally improves against LongT5, which emphasizes that local and global attentions are critical to capture the hierarchical structure of HTML documents.
Dev
Test
Models
Accuracy
BLEU
ROUGE-1
Accuracy
BLEU
ROUGE-1
LaMDA-1D (Gur el al,, 2022)
83.3
87,5
90.2
84.3
88.6
91.2
TS-Large (Gur et al., 2022)
83.2
90.2
90.5
84.3
91.7
91.5
T5-XI. (Gur al., 2022)
84.0
90.8
90.9
85.2
92.1
91.9
LongTT-Base
96.4
98.0
98.5
95.6
97.4
98.2
LongTT5LLage
98.1
98.9
99.2
97.7
98.5
99.0
LongT5-XI.
98.4
99.1
99.3
98.5
99.2
99.3
HTMLL-T5-Buse
96.5
98.1
98.6
95.9
97.5
98.3
HTML-TS-Large
98.1
98.9
99.2
97.7
98.3
99.1
HTML-T5-XL
98.4
99.0
99.3
98.9
99.4
99.5
Table 9: Results of Description Generation benchmark (Gur al., 2022). We compare LaMDA (Thoppilan al., 2022). T5, LongT5, HTML-T5 with respect to accuracy, BLEU, ROUGE-1 scores. The results demonstrate that local and global attention mechanisms,, shared modules between LongT5 and HTML-TS, could almost completely solve the benchmark by improving the previous best performance by over 10%. HTML-TS slightly outperforms LongT5
25

---

Published as a conference paper at ICLR 2024
J FLAN-LONGTS
In the web automation literature (Furuta et al., 2023; Kim et al., 2023), instruction-finetume LLMs have great success in HTML comprehension and improve the task success. For the comparison to HTML-denosing, we prepare the instruction-finetune LongTS (i.e. Flan-LongTS) by leveraging Flan dataset released by Chung al. (2022). We finetuned the pre-trained LongT5 with 100K iterations and picked up the best checkpoints.
As sanity check of instruction-tuning we evaluate Flan-LongTSs with few-shot/zero-shot settings on CoT benchmark (GSM8K (Cobbe et al., 2021), StrategyQA (Geva et al., 2021), SVAMP (Patel al., 2021), Asdiv (Miao al., 2021), Commonsense(A (Talmor et al., 2019)). BigBench-Hard (BBH) (Suzgun et al., 2022), and MMLU (Hendrycks et al., 2021b) as tested in Longpre et al. (2023). We reevaluate the performance Flan-T5, using official checkpoints 5 We also check the performance of Flan-LongT5 on downstream summarization tasks, originally evaluated on LongT5 (Guo et al., 2022). We use arXiv (Cohan et al., 2018), PubMed (Cohan et al., 2018), BigPatent (Sharma et al., 2019), Multi-News (Fabbri et al., 2019). MediaSum (Zhu et al., 2021), CNN Daily Mail (Nallapati et al., 2016) dataset for the evaluation, measuring the performance with ROUGE-1/211 metries.
Table 10 shows that we have successfully replicated the LongTT5 version of instruction-fiieture language models. Flan-LongTT3 achieves competitive results original Flan-T5; for instance, Flan- LongTS-Large (36.64) outperforms Flan-T5-Large (35.25), but Flan-Long T5-XL (39.05) is still behind Flan-T5-XL (43.03) average. This might be caused by the training instability of XL-size models (Guo et al., 2022). Because, unlike HTML-T5 on HTML-based tasks, reasoning tasks do hierarchical syntax,
surprising for Flan Plan-L not to outperform Flan-T5. Table 11 also demonstrates that we have successfully conducted instruction-tunin -unint without losing the capability of long text summarization.
have long-context
'tttps://githob. rss http:///ww////too//m/mm///////e/ s.md
f
26

---

Published as a conference paper at ICLR 2024
R-L
40.18
41.28
37.28
37.34
We reevaluate the performance Flan- TS (Chung al., 2022). using official checkpoints. Flan-Long LongTS achieves
CNN/Dally HN//Dally Mail
We measure the performance with ROUGE- -GG-11112111 1/211 metrics.
R-2
20.51
21.40
20.89
21.00
CNN
R-1
42,49
43,94
43.13
43.46
R-L
32.20
32.80
32.33
32.81
Total
35.25
43.04
36.64
39.08
MediaSum
R-2
19.04
19.66
19.18
19.75
Avg.
Direct
37.29
42.55
35.45
38.12
R-1
35.54
36.15
35.71
36.16
CoT
33.20
43.53
37.84
39.97
R-L
24.18
24.94
24.52
24 24.80
MultiNews
BBH-CoT
Few
31.45
35.62
31.85
32.01
R-2
18.44
19.43
18.99
19.47
Table 11: Performance Flan-I Long on downstream summatization tasks, compared to LongT5 (Guo al. 2022).
Zero
26.17
34.12
29.38
29.09
R-1
47.18
48.17
47.76
48.19
Few
37 37.48
40.96
34.67
37.77
R-L
62.73
70.76
63.02
70.01
BBH
BigPatent
Zero
26.09
28.67
26.53
R-2
56.86 66.81
66.06
57.13
65.17
25
Few
45.12
52.40
40.03
44.74
R-1
70.38
76.87
70.53
76.31
MMLU
Zero
40.68
50.76
38.44
43.44
R-L
46.46
46.67
46,96
46.73
Few
40.03
52.64
45.34
50.02
PubMed
R-2
24.69
24.76
25.08
24.75
CoT
Zero
35.14
51.74
44.78
48 48.78
Table 10: Performance of Flan- LongTS reasoning tasks. competitive results original Flan-T5. T5.
R-1
49.98
50.23
50.46
50.23
Flan- Flan-Long Long 15-1. T3-Large Large
Flan-Long Long T5-XL
R-L
44.11
44.27
44,46
44,22
LongT--Lann
Flan- T5-1 T'sLagge Large
Flan T3-XL
arXiv
R-2
21.63
21.92
22.00
21.75
Models
R-1
48.28
48.35
48.52
48.37
48
Flan-I LongT3- Large
Flan-I LongT3-XXI XL
Long Long/TS-Large Large
Long LongT5-XX.
Models
27

---

Published as a conference paper at ICLR 2024
K PER-TASK PERFORMANCE ON MINIWOB++
Task
HIML-T33XL (347K)
HTML-TS-XL (12K)
Flan-T3-XL (347K)
WehN-T3-XL (12K)
book flight
0.99
0.00
0.48
0.00
choose-date
0.16
0.03
0.08
0.00
choose date easy
1.00
0.28
1.00
0.03
choose-dtte-medimm
0.56
0.14
0.57
0.00
choose-list
0.22
0.19
0.16
0.26
click-bution
1.00
0.92
0.98
1.00
baick-bttonn-ee sequence
1.00
click-chese checckeessssss
1.00
1.00
1.00
0.96
cceechosess-aagg
0.90
0.94
0.98
0.22
click checkkosess soft
0.99
0.64
1.00
0.54
click-checkooxes-rrmnttt
1.00
1.00
0.99
0.63
click-collapsible
1.00
0.41
1.00
0.00
dbck-collapsiblee
0.93
0.26
0.94
0.00
click color
click-dinlog
1.00
1.00
1.00
1.00
click dialog 2
0.74
0.31
0.34
0.24
click-link
0.99
1.00
1.00
1.00
click-menu
0.37
0.26
0.41
0.37
click-option
1.00
1.00
0.87
click pie
0.96
0.89
0.99
0,51
click scroll-l-i-t
0.91
click-shades
0.00
0.05
0.00
0.00
click shape
0.79
0.57
0.58
0.53
click-tab
1.00
1.00
1.00
0.74
click-tab-2
0.94
0.40
0.94
0.18
click-tab-2-hant
0.30
0.57
click test
1.00
1.00
1.00
1.00
click-test-2
1.00
1.00
1.00
click-widget
1.00
0.94
1.00
1.00
count-shape
0.67
0.55
0.64
0.41
email-inbox
1,00
0.99
0.99
0.38
1.00
0.92
1.00
0.60
email-ine -ibui-inxxxx- forward- ttrtukkk
1.00
1.00
email innoxx-inoxx--a nul-tukk
0.99
0.76
0.92
0.23
enter-date
1.00
0.00
1.00
enter-paswerd
1.00
0.99
1.00
0.97
enter-iext
1.00
0.96
1.00
0.89
enler-exttddynmmic
1,00
L.00
1.00
0.98
enter-time
1.00
0.00
0.00
0.00
focus text
1.00
focus-text-2
1,00
1.00
1.00
1.00
grid-coordinate
1.00
1.00
1.00
0.49
guess-aumber
0.13
0.00
0.10
0.00
ddentify shape
1.00
0.89
0.90
0.88
login-user
1,00
0.80
1.00
0.82
login-use user popup
1.00
0.63
0.97
0.72
multi- layouts
1.00
multi-orderings
1.00
1.00
1.00
0.88
navigate-tree
0.99
0.99
1.00
0.91
search-engine
0.93
0.55
0.59
0.34
social media
0.99
0.93
0.99
0.21
social-media-all
0.84
0.09
social media some
0.89
0.60
0.39
0.02
tic-tac-toe
0.42
usc-autocomplete
0.97
0.23
0.98
0.22
use-spinner
0.07
0.07
0.03
0.07
Average
0.856
0.655
0.755
0.484
Table 12; Per-task average success rate on 56 tasks from MiniWoB++.. We refer to Furuta al. (2023) and Gur et al. (2022) for the baseline performances.
28


Reviews:
[{"review_id": "MujrL4V6CP", "reviewer": "Reviewer_Gbpj", "summary": "The paper introduces \"WebAgent,\" an autonomous agent driven by large language models (LLMs) that completes navigation tasks on real websites by following user instructions and combining canonical web actions in a program space. WebAgent's capabilities are outlined as follows:\n\n---\n\nPlanning Sub-Instructions Per Step: It decomposes natural language instructions into sub-instructions, planning out the steps needed to complete a task.\n\nSummarizing Long HTML Pages: It can summarize lengthy HTML pages into snippets that are relevant to the task at hand, based on the sub-instructions derived from the user's commands.\n\nActing via Programming: It grounds sub-instructions and HTML snippets into executable Python codes, allowing it to interact with real websites programmatically.\n\n\n---\nTo form WebAgent, two LLMs are combined:\n\nFlan-U-PaLM: Used for grounded code generation. This model provides the agent with the ability to generate code snippets that can interact with web pages.\n\n\nHTML-T5: Used for task planning and conditional HTML summarization. This model has an encoder-decoder architecture and is specialized in capturing the structure, syntax, and semantics of long HTML pages. It  incorporates local and possibly global attention mechanisms to better process the structure of HTML documents.\n\n---", "questions": "Does Webagent replan after failures? How does it handle failures?\n\nRelated to a mentioned weakness, how does this approach scale? would it just perform better with more data, parameters and compute?\n\nAre all the components of web-agent available open-source and will web-agent be open-sourced?\n\n\nUpdate: All questions have been addressed in response.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper has several strengths:\n\n----\n1. Unlike prior works, there is a focus on real world application. Demonstrating success in real-world web navigation tasks provides a strong case for the practical application of this research. This has implications for the usability and deployment of AI systems in everyday tasks.\n\n----\n\n\n2. The collaborative approach, where different models work together to complete tasks, showcases a novel use of ensemble techniques in a practical setting, which encourages more research in model collaboration. There is also additional benefits of such a modular approach, in that scalability and error analysis becomes easier. The use of an ensemble of specialized models to address specific aspects of the problem space, is a departure from the trend of using a single generalist model for all tasks.This specialization can lead to performance improvements and more efficient computation.", "weaknesses": "1. Especially for this kind of work, the broader impacts section should be in the main text and should be fully fleshed out. This is a significant weakness in this work.\n\n-----\n\n2. It would be good to have a baseline comparison comparing what performance looks like with model scale. Flan-U-PaLM is a 540B parameter model which puts it at a scale inaccessible to many researchers.. it would be good to benchmark how this approach scales from small accessible open source models, to the large ones used in this work.\n\n----", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hzzmAausZD", "reviewer": "Reviewer_ASiH", "summary": "This work proposes a new LLM-based agent for web-based tasks which achieves state of the art on Mind2Web.\nThe proposed method combines two LLMs into one agent, HTML-T5 which is a new pretrained model and is further finetuned for planning and summarization, and Flan-U-PaLM which is a frozen model and generates programs to allow the model to interact with web environments.", "questions": "- There are some missing recent baselines for miniwob++ [1]. These methods report that the task performance is near human (93%). Could you provide more information about the performance of the proposed method (which is a bit lower) in this context?\n\n- Is it possible to report results using models other than Flan-U-PaLM with 540B parameters?\n\n- Will HTML-T5 be released?\n\n[1] SYNAPSE: Trajectory-as-Exemplar Prompting with Memory for Computer Control. Zheng et al., arxiv 2023.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The model's usage of HTML-T5 for planning and summarization is effective and novel, and the overall performance is good. Especially on Mind2Web, it significantly pushes the upper bound of performance.", "weaknesses": "Because the model relies on Flan-U-PaLM with 540B parameters, it's difficult to judge how reliant the method is on the ability of this particular model to generate executable code.\n\nThe organization of the paper could be improved, including more details about how feedback was acquired and finetuning was done to enable planning and summarization (i.e. Fig 6 in appendix)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tBq0fKvsxm", "reviewer": "Reviewer_eCh6", "summary": "This work proposes a Web Agent that (1) decomposes natural language instructions into sub-instructions plan, (2) summarizes long HTML pages into task-relevant snippets (based on sub-instructions), and (3) acts on web pages by writing and executing Python programs with the Selenium WebDriver.\n\nWebAgent is based on two neural networks: HTML-T5 (introduced in this work) and Flan-U-PaLM.\nHTML-T5 is an encoder-decoder transformer trained on HTML documents from CommonCrawl with various long-range denoising objectives. The model is then fine-tuned on specific downstream tasks to predict a sub-instruction and a summary of the HTML page (data-ref HTML attributes?) given the natural language instruction, previous sub-instructions, and the raw HTML page.\n\nGiven the predicted sub-instruction and HTML snippet from HTML-T5, Flan-U-PaLM is then prompted to predict executable Python code that will perform the sub-instruction on a given web page.\n\nHTML-T5 is evaluated on MiniWoB++ and Mind2Web. Results show better performance than previous baselines.\nWebAgent is evaluated on WebSRC and instructions following on real websites based on task attributes successfully covered. Experiments show that the modular approach of WebAgent is beneficial compared to using only 1 language model.", "questions": "Mostly clarification questions related to weaknesses above:\n\n- What data was used to train HTML-T5 to predict sub-instruction plans and HTML summaries?\n\n- What is defined as a \"HTML summary\" and how is it used by Flan-U-PaLM?\n\n- How did the HTML-T5 inputs (instruction, previous sub-instructions, and raw HTML) fit into a window size of only 4098? The raw HTML would take up all the space.\n\n- How was the train/test split done to ensure no task (or even sub-task) overlap?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "This work is making a significant contribution to the field by providing two models: one encoder-decoder that reads, understand and summarizes HTML pages: HTML-T5; and one WebAgent that combines the previous model with a code generation model (Flan-U-PaLM) to act and follow instructions on synthetic and real websites. \n\nSome notable strengths of the proposed architecture are:\n- To capture long-range dependencies in long documents, HTML-T5 uses both local and transient global attention similar to Long-T5. In addition it is pre-trained on various long-range denoising objectives.\n- To be able to execute actions on real websites, WebAgent produces executable Python code instead of discrete and non-generalizable HTML actions. This allows the agent to handle any action space present in real HTML pages instead of being limited to a set of fixed actions.\n\nExperimental results show that WebAgent is able to solve tasks in real websites.", "weaknesses": "Overall, this is a strong paper, however, one weakness of this work is the lack of baselines to compare results against in real-world tasks. Table 1 provides good ablation study insights into the proposed WebAgent but there are no other Agents to compare to. Similarly in Table 3, HTML-T5 is only compared against MindAct on Mind2Web. Are there any other agents that could be used on this benchmark? \n\n---\n\nAnother weakness of this work is its clarity and ease of comprehension. Some aspects of the paper were not entirely clear, in particular how was HTML-T5 trained to predict sub-instruction plans and HTML summaries? What data supervision was used for that?\n\nSimilarly, it is not entirely clear what HTML-T5 produces: Figure-3 indicates \"HTML-snippets\", but the paper mentions multiple times that it \"summarizes\" HTML pages (so it should produce a summary?), and in Section 3.2 the paper states that it predicts ``_the corresponding data-ref attributes_''. If the model outputs only data reference IDs (like suggested also with Figure 6) then this is not summarization but more like information retrieval and the paper should reflect this. In addition, if object references are what is really being predicted, then it is not clear how Flan-U-PaLM make use of that information without having access to the raw HTML containing these objects.\n\nAnother confusion is the window size of HTML-T5: in Section 3.1 it is mentioned that the input sequence length of HTML-T5 is 4096, but in section 4.2 it uses 16k tokens for the context window. Which one is it? 16k tokens seems more likely overall since the model is supposed to take as input instruction, previous sub-instructions, and raw HTML. Just the raw HTML would overflow the 4096 context size as mentioned in the paper and illustrated by Figure 2. After reading 4096 in Sections 3.1, it was hard to understand how all inputs of HTML-T5 would fit in such a small window (especially after seeing Figure 2).\n\n---\n\nEventually, one important thing that the paper should discuss is the difference between train and test settings. It seems like WebAgent was trained on all domains individually. What precautions were made to ensure that the testing tasks do not overlap with the ones used during training?\n\n---\n\nMinor: some syntactic mistakes make the paper hard to read sometimes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tE1mdCrUjm", "reviewer": "Reviewer_Jsxu", "summary": "The paper introduced 1) a web-agent model that manipulates the web objects by human natural language instructions 2) a newly pretrained HTML-T5 model as a component in web-agent. \n\nThe experimental results show that 1) the web-agent, compared to solely using it's component Flan-U-Plam, is significantly better in a benchmark; and 2) the newly introduced HTML-T5 itself is outperforming existing HTML LLMs on web understanding tasks.", "questions": "1. How are \"open ended actions\" (Figure 1), \"canonical sub-instructions\"(abstract), and \"pre-defined action space\" defined? Does the author promote the open-ended action space or pre-defined action space?\n\n2. In section 3.3, does \"given a few canonical examples for program generation\" describe the step of \"few-shot prompt' in Figure 3? Where do these examples come from?\n\n3. Could the author elaborate on the difference between two tasks in 4.1 and 4.2, except that they have different baseline models and datasets. What's the difference between input/output, etc. \n\n4. What's the definition of \"planning\" in Section 3.2. Is summarization referring to \"localizing\" the relavant snippet of the current sub-instruction?\n\n5. Could the author summarize the WebAgent workflow end-to-end. i.e. describe Figure 3 and explain the user input, system's knowledge/DB if exists, and the HTML-T5 to Flan-U-Plam is one-time action or interactive process. \n\n6. Could the author summarize the newly curated dataset that is used to pretrain/finetune part or entire WebAgent? E.g. template, sub-instruction, action examples\n\n7. In Table1, for example, the real-estate case, does the WebAgent see the same searching page but different instructions in the 20 tests?\n\n8. Is it correct that HTML-T5 is trained only for summarization, while the other models compared in Table 4 are multi-tasking.\n\n\nGlad to raise the score if the clarity will be significantly improved.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Overall the reviewer found the experiments are well designed in supporting their claims of 1) the overall methods is much better than using a single LLM and 2) the HTML-T5 is an advance by itself. The most recent models are included in the experiments, and the evaluation datasets (mind2web and miniwob++) are used in training of both the proposed HTML-T5 and the baseline model long-T5. Therefore, the reviewer has no concerns of unfair comparisons.", "weaknesses": "The presentation can be improved. Please consider revise the writing to avoid the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)