Understanding HTML with Large Language Models
Authors: Izzeddin Gur
Ofir Nachum
Yingjie Miao
Mustafa Safdari
Austin V Huang
Aakanksha Chowdhery
Sharan Narang
Noah Fiedel
Aleksandra Faust
Venue: EMNLP 2023 Findings (2023)
Keywords: html understanding
web navigation
large language models
semantic classification
description generation
TL;DR: LLMs for understanding HTML documents for semantic classification, description generation, and web navigation problems.

Abstract: Large  language  models  (LLMs)  have  shown exceptional  performance  on  a  variety  of  natural  language  tasks. Yet,  their  capabilities for  HTML  understanding  –  i.e.,  parsing  the raw  HTML  of  a  webpage,  with  applications to  automation  of  web-based  tasks,  crawling, and browser-assisted retrieval – have not been fully  explored.   We  contribute  HTML  understanding models (fine-tuned LLMs) and an in-depth analysis of their capabilities under three tasks: (i) Semantic Classification of HTML elements, (ii) Description Generation for HTML inputs, and (iii) Autonomous Web Navigation of  HTML  pages.   While  previous  work  has developed  dedicated  architectures  and  training procedures for HTML understanding, we show that LLMs pretrained on standard natural language corpora transfer remarkably well to  HTML  understanding  tasks.   For  instance, when  fine-tuned  on  data  from  the  MiniWoB benchmark, LLMs successfully complete 50% more  tasks  using  192x  less  data  compared to  the  previous  best  supervised  model.    We create  and  open-source  a  large-scale  HTML dataset  distilled  and  auto-labeled  from  CommonCrawl

Understanding HTML with Large Language Models
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, Aleksandra Faust
Google DeepMind
[ (izzeddin, annfirrachum,,inni emi ao, msafdari, austinvhuang chowdhery, sharannarang, nf nfieeel, faust]ego com
Abstract
The predominant approach to web automation and HTMI understanding to train specialized models, i.c., gathering application-speen and designing neural network (NN) architectures to leverage inductive biases of the HTML's structure; see, e.g., Liu et al. (2018); Toyama et al. (2021); Gur et al. (2019, 2021); Humphreys et al. (2022). However, both dataset collection and neural archi- tecture design are expensive, time-consuming, and require highly-specialized, domainn-spciinn knowl- edge.
Large language models (LLMs) have shown exceptional performance variety of nat- ural language tasks. Yet, their capabilities for HTML understanding ie., parsing the raw HTML of webpage, with applications to automation of web-based tasks, crawling, and browser-assisted retrieval have not been fully explored. We contribute HTML under- standing models (fine-tuned LLMs) and an in- depth analysis of their capabilities under three tasks: (i) Semantic Classification of HTMI. el- ements, (ii) Description Generation for HTML inputs, and (iii) Autonomous Web Navigation of HTML pages. While previous work has developed dedicated architectures and train- ing procedures for HTML understanding, we show that LLMs pretrained on standard natu- ral language corpora transfer remarkably well to HTML understanding tasks. For instance, when fine-tuned on data from the MiniWoB benchmark, LLMs successfully complete 50% more tasks using 192x less data compared the previous best supervised model. We create and open-source large-scale HTML. dataset distilled and auto-labeled from Com- monCrawl."
datasets
Meanwhile, in the natural language processing (NLP) literature, large language models (LLMs) have emerged as solution to the difficulties of dataset collection and specialized NN design (Ka- plan et al., 2020; Bommasani et al., 2021). A popu- lar paradigm in NLP is to take an off-the-shelf LLM pretrained a large text corpus via unsuper- vised and task-agnosti learning objective and either fine-tune or prompt the LLM on a small task- specific dataset. This paradigm has shown excep- tional performance on a variety of NLP tasks (Xue et al., 2020; Brown et al., 2020; Austin et al., 2021). Whether LLMs can be applied to HTML under- standing especially given the much larger context and sequence lengths remains under-explored question.
1 Introduction
Web crawling (Olston et al., 2010), form- filling (Diaz et al., 2013; Gur et al., 2021), or infor- mation retrieving web agents (Nogueira and Cho, 2016) are important for both automating and assist- ing users in web-based tasks. These and similar applications rely on models that can search for spe- cific content or controls web page well as navigate website autonomously. Since a web page in its raw form is represented as an HTML-based text sequence, the success of models for web-based tasks relies on their ability to understand HTML semantics, structure, and embedded interactions.
In this paper, we investigate whether LLMs be applied to HTML understanding to produce berfernperformiing, more sample-efficient HTML understanding models and without the need for custom NN architecture design. To that end, we present a suite of three benchmarking tasks for HTML understanding that capture the essence of these applications and require understanding both structure and content. First, we devise Semantic Classification
task that requires model clas- sify a given HTML element into one of of cat- egories, such as address, email, password etc., with application to automated form- fill-filing. Sccond, we present Description Generation, a label- extraction
'See visualizations of the results at https://ssite: google. google.com/vieeee www.coww//wwetccetceocoooottettt In4html/home.

---

<tni
benchmark suite (Shi et al., 2017). The encoder- decoder architectures with bi-directional attention show the best performance across the board even when their pretraining does not include HTML.
bodyy
ffon class= -ogin-oooo--
<dive
<label class-"form-label" for="ulare">
Enter Frail Address
</labels
forn-labb"" Tor="pass"s
Enter Password:
The broader objective of this research is to ad- vance the integration of LLMs with autonomous web agents. It has only been in the last year that researchers have begun utilize LLMs outside of NLP and integrate them as core capabilities in au- tonomy ((Lu et al., 2021; Ahn et al., 2022)). In this context, LLMs are reasoning engines for sequential decision making agents interacting with environ- ments. We believe these contributions expand the scope of language models and connect their unique capabilities with autonomous agents for the web, We provide a new perspective on machine learn- ing for HTML understanding and web automation, showing that pretrained LLMs can achieve signifi- cant performance on such tasks, reducing the need for specialized architectures and training protocols. To encourage further research in this direction, we open sourced 2 model weights for agents used in the WoB environment and our dataset for descrip- tion generation.
</label>
<dive
singut type- 10- uslane's
inout poppe"pasanore Id= pass"s
span class- "hidden's
Please enter your password.
sssann
/dive
bbutoon typas "upe""sbhit">>Si tout
ffoorr
</body>
</htn>>
Semantic Classification
Description Generation
Web Navigation
Usemmam
Enter Email Address
"ycuremakBBrrmi.comm
bypo/name- "NNam"
Figure I: HTML. example page with highlighted salient element, an element interest (dashed box). All canonical tasks evaluate a distinct interaction with this element, either by classifying as one a of categories, generating a text description its purpose, or applying an action as part a scquential navigation of multi-page website.
task where a model is given an HTML snippet and is asked to produce a natural language descrip- tion. For instance for an email field, the description might be "Please enter your email address." " Note that in the majority of web pages, this connection between input elements and description content is only implicit in the raw HTML code and inferring such links is a prerequisite for higher-level navi- gation objectives. The third task is Autonomous Web Navigation (Shi et al., 2017). A model is pre- sented with an HTML page paired with a natural language command and must apply appropriate ac- tions on a sequence of HTML pages to satisfy the command. See Figure 1 for a simplified example of these tasks.
2 Related Work
HTML Understanding Autonomous web naviga- tion has been a popular application for neural net- work models, and a variety of works propose simu- lated websites for training web-based agents, with application to task fulfillment (Yao et al., 2022; Gur et al., 2021; Burns et al., 2022; Mazumder and Riva, 2020; Shi al., 2017: Liu et al., 2018) well as information retrieval or guestion-answeee (Adolphs et al., 2021; Nogueira and Cho, 2016). Simulated websites provide an easy way to evalu- ate models online, and for this reason we use the existing MiniWoB benchmark (Shi et al., 2017) for our web navigation setting. However, it is still im- portant to have mechanism for evaluating models on wide variety of real-world websites. This was the key motivation for generating our own dataset for the description generation task, which is dis- tilled and auto-labeled from CommonCrawl and is a key contribution of our paper.
With these benchmark tasks in hand, we eval- uate the transfer capabilities of variety of pre- trained LLMs (Table 1). varying in architecture (encoder-only. encoder-decoder, or decoder-only), model size (from 24.6M to 62B parameters), and training data corpora (both including and exclud- ing pretraining NLP and HTML corpus). While prior work universally pre-parses the HTML as in- put to the model (Gur et al., 2021; Liu et al., 2018; Nakano et al., 2021), ours uses raw, unprocessed HTML. Our results show that LLMs demonstrate a remarkable level of HTML understanding across all tasks, with up to 192x more sample- than models trained from scratch, and achieving a new SoTA for supervised learning on the MiniWoB
Alongside these benchmarks, many works have developed models for web navigation and related subtasks (Pasupat et al., 2018; Bommasani et al., 2021; He al., 2021; Gur et al., 2021; Humphreys
clw.nnsssse..ee..eeeoeaooooooooe www.wdowoooeee.....oo....o com/storag/ browser//rrssearch//eellm

---

al., 2022; Liu al., 2018; Jia et al., 2019). These works often rely on specialized neural network ar- chitectures that leverage inductive biases of HTML structure, or on preprocessing of HTML to make it easier to input to a model ((Li et al., 20)1,,b)). In contrast, our work takes minimalist approach, pro- viding HTML in text form with minimal processing and using widely-adopted transformer networks.
Semantic Classification. Many HTML under- standing applications require a model that can clas- sify HTML elements into standardized categories. For example, in automated form- fom-filling (Diaz et al., 2013; Gur et al., 2021), it is useful to identify a 'submit button' across many websites (c.g., shop- ping, flight booking, utility application) with vari- ous button representations (c.g.. position, color, or text). Thus, we formulate Semantic Classification as classifying elements into role categories. Take the example HTML in Figure 1 which includes two input elements and a submit button. Let's pick the first input as an element of interest to be classified by the system, also called a salient ele- ment. The system should classify this element username, since it appears on login page and it has label with Email Address which is typically associated with the username in cations. To solve this, the system can aggregate information from multiple sources in the page the label that says Enter Email Address, the input at- tributes (type= "mmil" and id= "u==m""""" "), or even the ordering of other elements in the page such as 'password' and "sign in'.
LLMs and HTML Works that explore the in- tersection of LLMs and HTML generally fall into two categories. The first category uses LLMs to assist web navigation (Nakano et al., 2021; Yao et al., 2022), and typically relies on custom pre- processing to map the context and structure of web page to natural language, thus severely restrict- ing what HTML pages the model can parse. The second category pretrains LLMs on a large corpora of HTML text (Aghajanyan et al., 2021). However, these works typically restrict the model evaluation to standard NLP tasks, e.g., summarization and question/answerinn as opposed to tasks more rele- vant to HTML understanding and web automation. Our work can be thought of as the reverse: We keep the pretraining of LLMs unchanged and focus on the mechanisms for transferring the pretrained LLMs to HTML-releeant tasks.
appli-
Description Generation. Motivated by appli- cations in accessibility-mindee web browser con- trol (Jorgensen and Binsted, 2005), we formulate description generation as an extractive problem where the goal is to locate the textual description of an element in the HTML and generate it as out- put. For instance, the description of the salient element in Figure 1 is Enter Email Address; when rendered, this label will appear above the 'email' input field. HTML provides a large amount of flexibility, and so in general a descriptive text that appears alongside a specific element when rendered can be very far from that element when looking at the HTML plaintext. Thus, this task evaluates model's ability to understand the structure of HTML as it would appear to user, despite not having access to the rendered web page directly.
3 Canonical Tasks for HTML Understanding
We devise three canonical tasks to study HTML un- derstanding capabilities of LLM-based web agents. These tasks require correctly interpreting both structure and content to varying degrees to make predictions, with autonomous navigation being the most challenging capability of the three.
Autonomous Web Navigation. This task evalu- ates how well model navigates nulti-page web- sites as a sequential eeciiion-mmaing problem (Shi al., 2017; Liu al., 2018). At the beginning of an episode, the agent is given a natural language instruction, e.g. Enter the username "lyda" " and the password "N22r" into the text fields and press login. The agent applies actions to a sequence of HTML pages, where each action is of the form Function(selecton
Datasets
Each of our canonical tasks requires a separate dataset, with the description generation task using a newly contributed, auto-labelled dataset based on CommonCrawl.
text). The function is one of click type, selector is an integer pointer that uniquely identifies an element, and text is text input if the type functionality is activated. An episode terminates when either the page reaches a terminal state (c.g., the 'sign in' button is clicked) or the maximum number of steps is reached.
Autonomous Web Navigation. We use the 12K demonstrations included in the publicly available MiniWoB benchmark (Shi et al., 2017), which encompass 62 website applications ranging from

---

Model
Task
Dataset
Input
Architecture
Output
Task Output
Enc Dec
Autonomous Navigation
MiniWoB
2017)
Puge
Text
Dictionary
Dec
Semantic Claxsification
Annotated Shopping Webgages (Gur et al., 2021)
28K
Snippet
All
Text
Calegory
Description Generation
CommonCravi (new)
85K
Snippet
Hac-Dee Dec
Text
Text
Table 1: Task, dataset, and model summary. All models receive HTML, Autonomous Web Navigation receives the entire HTMI,, while the other tasks receive HTML. snippets extracted given salient element.
email forwarding social media interactions. Each demonstration a sequence of (instruction, HTMI,, action) tuples. Every element in a Mini- WoB demonstration is accompanied by reference number unique within its respective pages. This number can be used as element selector, making the action space unified across all tasks and time steps. For instance, the action in Figure would be rype(ref=5,, "username @ email. colll.ccm" "), where refers to the index of the input when counted from top-to-bottom. As model input, we concatenate the natural language instruction and HTML into a single text input sequence. Similarly, we treat the action as text sequence for the model to predict. Semantic Classification. We use a dataset of 28K labelled examples, containing 66 different cat- egories, of the form (HTMI., element, category), previously used in the context of environment gen- eration (Gur et al., 2021). The dataset consists of HTMLS from real-world shopping websites and categories relevant to form-filling during payment and checkout on these websites.
and extracted HTML labels that have for at- tribute. Removing non-Unicode and alphanumeric text in HTML labels results in a 400K example datset. We balance the distribution of labels, effec- tively downsampling the dataset to 85K samples. Each example is represented (HTML, element, description), where HTML is the HTML plain- text of the page, element the element whose id attribute matches that appearing in the label's for attribute, and description is the text inside the label element (see example in Figure 1). More details of the dataset can be found in Appendix A.2.
Pre-Processing
In treating HTML token sequences, we minimize any HTML tree pre-processing prior model input. We thus provide HTML as raw text (i.e., sequences of text tokens) and only apply a snippet extraction pre-prose pre-processing for pages which are too large to fit into the typical LLMs context windows.
Description Generation. For this task, de- rive dataset from CommonCrawl. Common- Crawl does not include renderings or annotations that would reveal what text in the HTML as- sociated with which elements. Instead, we infer descriptions of various elements by exploiting a special attribute in the HTML schema known as for. As an example in Figure 1, the first label in the HTML has for attribute with value uName, which is the id of the element described by label; in this case, the id is that of the first input in the page. This annotation does not affect the rendering the page and is typically used for accessibil- ity purposes. We utilize the information given by these for attributes to create large-scale d to study description generation.
Snippet Extraction. Real HTML pages can grow extremely large, reaching thousands of el- ements, far beyond the context window of the largest LLM that we studied (1920 tokens in PaLM (Chowdhery et al., 2022)). LLMs typically truncate such long sequences, which can be detrimental to HTML understanding HTMLS are not linearly structured. We take an element-centric approach and extract HTML snippets (a small portion of HTML code) surrounding salient element (Fig- ure 5). A simple heuristic, which controls the tree's width and depth, guides the process: Start with a salient element and traverse its ancestors the HTML tree until stopping condition is satisfied. As we traverse up, we estimate the height of the tree and the increased number of descendants of the new root. We stop when either metric violates pre-defined limit and take the resulting sub-tree as the snippet. We mark the salient element using a special attribute, called target, to distinguish it from other elements, We perform the snippet extraction
Specificully, collected 100 WARC (from April 2019) files from the CommonCrawl project
"nttp://commnnerakk/
hhttps://consdae cloud cloud. google. l. com/storagee Browser/gressercch/eehh http://wwwwmmmmmmeeeeeeeeeeeeeae httamm/mtttttttttegeaaaaaa

---

for the semantic classification and description gen- eration datasets, and keep the full HTML pages in MiniWoB because these pages are typically much smaller than real-world HTML.
repetitions. We solve this issue by augmenting ples in the dataset with sequence of past actions, (action history, instruction, HTML, action), and allowing LLMs to learn from past experience.
HTML un-Parsing, We provide the models with the unparsed plaintext HTML in the form of sequence of tokens. This canonical representa- tion does not require specific model architectures such as hierarchical networks (Liu et al., 2018; Gur et al., 2021) and can be fed into any LLM. We trans- form all datasets by converting every HTML page or snippet into a sequence. For Mini BoB, we ad- ditionally concatenate (action history, instruction, HTML) tuples into a single sequence.
Encoder-only Models. We train encoder-only models, ie.. BERT (Devlin et al., 2018), with text imput and categorical output. We keep semantic categories as discrete one-hot classes. To train encoder-only models, we add new classification layer after the final encoder layer to produce a dis- tribution over semantic categories. In addition to the typical BERT models, we study MobileBERT (Sun et al., 2020), distilled from BERT-large with inverted bottlenecks, and Albert-XL (Lan et al., 2020), with parameter sharing and embedding split.
6
Model Training
7 Results
We study variety of transformerrbasel LLMs (Vaswani et al., 2017) with different sizes and architectures for HTML understanding tasks (Table 1). In the rest the text, we prefix models fine-tuned for Autonomous Web Navigation, Description Generation, and Semantic Classification with WebN-, WebD-, and WebC-, respectively. For instance, WebD-T5-3B is the three billion parameter T5 model (Raffel et al., 2020) fine-tuned for the Description Generation task. The rest of this section elaborates on training details.
We now present the results of fine-tuned LLMs for HTML understanding. We compare the mod- els' performance with the existing baselines where possible (autonomous web navigation) and against other LLM architectures and training regimes (all tasks). Sections 7.1, 7.2, and 7.3 evaluate task- specific performance, while Section 7.4 assesses the performance across all the tasks.
Metrics: For autonomous web navigation we evaluate models' Success Rate, which is averaged over 100 episodes per task. For the other tasks, we use Accuracy to measure exact match between prediction and ground truth. In the description gen- eration task, we additionally provide evaluations us- ing alternative 'soft' text evaluation metrics, BLEU and ROUGE-I, measuring the similarity between predicted and ground truth text.
and Decoder-only Models. We train encoder-decoder models, i.e., T5 (Raf- fel et al., 2020), and decoder-only models, i.c., LaMDA (Thoppilan et al., 2022) and PaLM (Chowdhery et al., 2022), with text input and text output. Inputs are raw HTML pages or snippet texts; similarly, outputs are categorics, natural lan- guage descriptions, or actions represented as text. Namely, for Semantic Classification we use the textual representation of categories, similar to pre- vious classification problems in NLP (Raffel et al.. 2020). For Autonomous Web Navigation, actions are converted into text by first converting them into key and value pairs and then concatenating the pairs.
7.1 Autonomous Web Navigation Results
For Autonomous Web Navigation we fine-tune two WebN- encoder-decoder architectures (WebN-T5- large and WebN-T5-3B) on 12k demonstrations from human-annotatee real websites. We eval- uate the models on MiniWob (Liu et al., 2018) benchmark, and compare with specialized archi- tectures trained using supervised learning (SL) on 2.4 million human expert demonstrations CC-Ner (SL) (Humphreys et al., 2022), and two RL models bootstrapped with SL, CC-Net (SL) (CC-Net (SL & RL) (Humphreys al., 2022), and WGE (SL & RL) (Liu al., 2018)). Additionally, we compare with the decoder-only architecture (WebN-Lambda IB) and perform an ablation study on the impact of including the action history in the input.
Many websites in MiniWoB require multiple interactions, such as tarnk-button-seeee sequence or click- checkboxes, where each interaction might cause subtle change in the website state. For instance, af- ter clicking on a checkbox in the chick-checkboxes website, its value flips from positive to negative or the other way around, which is not always re- flected in LLMs' predictions and leads to action

---

Model
Model Name
Test (%)
Dev (%)
Demos
Size
78.1
77.7
24.6 M
WebC-Albert-X
83.5
58.9 M
WebC-BERT-smalles
84.4
83.6
38.7 M
WebC-BERT-small
84.4
85.2
$2.8 M
WebC-BERT-medium
84.5
M
WchC-BERRT-bac
83.9
84.8
109.5 M
Thanar LOACA DLAMCAID
15.0
75-10 TCC
00
MCC
Hmmm
WchC-BERT-large
84.1
85.8
335.2 M
WcbC-TS-basc
86.8
89.9
250 M
WebC-TS-large
87.0
89.3
800 M
(a) Baseline comparison.
WcbC-T5-3B
87.7
90.3
3 B
Model Name
Success (%)
Model Size
WebC-LaMDAA-IIC
87.4
87.1
B
Ta-laag
18.1
800M
WehC- M--B (*)
86.6
89.9
8 B
LaMDA-IB
15.6
1B
WebC-PaLM-62BB (*)
88.7
90.5
62 B
T5-3B
11.1
3B
WebN-TS-large
46.4
800M
Ta-larg
76.4
75.2
800 M
T5-3B
77.2
73.8
B
WebN-LaMDA-IB
48.8
1B
PaLM-8B
73.3
70.1
8
WchN-T5-3B
51.8
3B
(b) Pre-training effect.
Table 2: LLMs performance on Semantic Classification task. Fine-tuning off tff-the-seef pretrained LLMS (model names with prefix "Web') helps LLMs transfer better compared to training the same architecture from scratch on the HTMI. dataset (model names without prefix 'Web*"), improving the accuracy of PaLM-8B more than 12%. While WebC-PaLMM 62B clearly performed better than all other models, we found WebC-T5-large to be competitive with much larger models such as WebC-LaMDA' B or WchC-PaLMM-B8 an asterisk their names utilize code in their training corpora.
Figure 2: WehN-T5* performance compared the pre- vious SOTA models on MiniWoB benchmark. WebN-TS- TS-3B improves the task success 16% while using 192 times less data, compared the supervised learning (SL) model, CC-Net (SL). LLMs performance is only surpassed by works utilizing RL, requiring orders magnitude more online expe- rience interaction with websites. b) LLMs with and without pretraining on Autonomous Web Navigation task, Those with pretraining (denoted by the 'WebN-" prefix) show a 2.5-4.5x performance improvement.
Models with
be difficult to detect (e.g. multi mayri_layyut)..... multi layolllyyut requires entering 3 dif- ferent texts in the website where the layout is ran- domized each episode, yet, surprisingly, even the (relatively smaller) WebN-T5-large model without action history outperforms the CC-Net (SL) model; illustrating that incorporating action history is not the only contributing factor for the better success.
and
Comparison to SoTA. Since previous works re- port success on only a subset websites in Mini- WoB, we evaluate on 48 out of 62 websites that are common across all models. Table 8 in the Appendix reports fine-grained results while Fig- ure 2a presents results averaged over all websites. Compared to CC-Net (SL) which is trained all 2.4M demonstrations, WebN-TH T5-3 improves the success 16% while only training on 12K publicly- available demonstrations, yielding over 192x im- provement in sample-efficiency. We find that all choices of LLMs outperform previous SL models. Notably, WebN-T5-3B significantly improves on websites requiring multiplc-action sequences such click checkkb_eebbxxs websites requiring entering text such as login usser (Table 8). We observe that the performance of LLMSs is only surpassed by pre- vious works utilizing RL, which require orders of magnitude more online experience interaction. Ex- tending our fine-tuned LLMs to an RL setting is a promising avenue for future work.
7.2 Semantic Classification Task Results
To evaluate the Semantic Classification task, we compare the T5 encoder-decoder architecture's three size variants (WehC-T5-base. WebC-T5- large, and WebC-T5-3B) fine-tuned on 22K real, human-labeled training websites. We compare with fine-tuned encoder only architectures (WebC- "BERT*), three fine-tuned decoder-only archi- tectures (WebC-LaMDA and PaLM), and both encoder-decoder and decoder-only models trained on human labeled websites from seratch. Results are presented Table-2, where we find that all WebC-LLMs perform well and significantly better than the same architectures without pretraining.
Action history ablation. Across all LLMs consistently observe decrease in success, on av- erage 6.4%, when past actions are excluded from the inputs (Figure 2a). Action history helps with websites that require entering multiple texts, as well as understanding minor changes that could
A Acuuracy per category. In Figure we present accuracy distribution of the WebC-T5-3B model on the development dataset. The fine-tuned encoder- decoder model performs strongly on majority the categories (Figure 4), even on those with very

---

New descendants (%)
Height
Test (%)
Dev (%)
thus highlighting the benefit of using standard off- the-shelf pretrained LLMs for HTML understand- ing.
25
87.7
90.3
25
88.6
89.2
50
88.4
90.0
50
89.3
89.2
7.3
Description Generation Task Results
300
87.8
88.8
500
75.8
74.5
For Description Generation split the Com- monCrawl dataset based on URL top-level do- mains to test LLMs' capabilities to generalize unseen HTML. We fine-tune encoder-decoder ar- chitectures (WebD-T5+) and decoder-only models (WebD-LaMDA )), with results presented Ta- ble 3. We also evaluate a strong heuristic baseline which simply finds the description closest to the salient element in the HTML text (Closest Descrip- tion).
(a)
WebC-PaLM WebC-TS-38
T5-3B (full data / no pretraining)
80
Acturasy
75
70
65
55
500
1000
1500
2000
Accuracy and Similarity Performance We show results of our evaluations in Table All mod- els achieve high scores across all metrics, achieving ss 84% the accuracy in terms of exact match and higher non-exact match score based on BLEU and ROUGE-1 (= 91 91%). This difference indicates that the models are capable of locating the descriptions, but not always generating the exact output.
Data Size
(b)
Figure 3: a) Effecc snippet extraction parameters WebC- TS-3B, Increases above 50% new descendants and height Large increases in both parameters lead to large snippets and decrease in accuracy. b) Accuracy over training data size. Using only 1000 labeled examples (4.4% all training datasct). WehC-T5-3B outperforms T5-3B (full data without pretraining) which is trained on all available labeled data (approximately 30k examples), and outperforms WchC-PPl.M 8B which is an order of magnitude larger.
7.4 HTML Understanding LLMs 's Performance Analysis Across Tasks
few samples. For instance, the model is 100% accu- rate on password_nee new which has only training examples, because the class is unambiguous. On the other hand, unsurprisingly, the performance drops when the category is ambiguous, such in the email category which is frequently mistaken as username.
We analyze results in aggregate derive our main conclusions.
7.4.1 Pretraining Effect: Pretraining on Large Text Corpora Matters
Fine-tuned pretrained LLMs outperform LLMs trained on HTML-only data, improving the per- formance by more than 34. 1 % on the Autonomous Web Navigation (Table 2b), and 10% to 12.7% on the Semantic Classification task (Table 2).
Snippet generation ablation. Two hyper- parameters govern snippet generation: percentage new descendants and height of the new root. While small variations of both parameters do not change the performance, increasing both degrades the performance significantly (Table 3a). With new descendants up to 500% and height up to 7, the per- formance drops by more than 15%. Note that snip- pet generation returns the full-page HTML when both parameters increase indefinitely.
Since Autonomous Web Navigation is the most difficult task, the improved performance is couraging evidence of the value of LLMs in HTML understanding tasks. Specifically, we observe that LLMs without pretraining are comparable to fine- tuned pretrained models only on websites that re- quire simple text matching. In contrast, for web- sites such as click_chhckclle checkboxes, text matching is harder and we find that pretraining is key to good performance. We also found that without pretrain- ing, model outputs were frequently in an incorrect format such as invalid dictionaries invalid refs with non-integer values. This suggests that the large corpora used for pretraining helps models to learn general HTML structure.
Data size impact. When varying the fine-tuning training data sizes 5, 10, 20, or 50 samples per class) in Figure 3b, WebC-T5-3B slightly outper- forms WebC-PaL.M-8B which an order of mag- nitude larger. Compared to T5-3B that is trained on all available HTML data without pretraining, WebC-T5-3B achieves better performance while using only 3.4% of labeled data (1000 samples),

---

Test BLEU
Dey
Model Name
Accuracy(%)
ROUGE-1
Accuracy(%)
BLEU
ROUGE-1
WchD-TS-large
83.2
90.2
90.5
84.3
91.7
91.5
LabD-LB
83.3
87.5
90.2
84.3
88.6
91.2
WebD-T5-3B
90.8
90.9
85.2
92.1
91.9
Closest Description
57.4
24.4
59.2
60.8
23.9
62.1
Table 3: Description generation accuracy of L.M.s.
100
Accuracy
75
50
25
Sapeee me
comten undin
00' pro 1pos
Mig rumu
Aie sun.
aumune ypa
to
pectun Bud 10 b.oud
cetune "ute ppo
aboote jour
nit Titmeeed ucdhas
and del pie' 1poo
(n) surd pur
R1ST Luddda dee" Tae ppp
was
Sunu
batco tuo Caa
seecoro son por 2000
pue 196a
Categories
Figure 4: Accuracy per classification category of the WebC-T5-3B model on the development dataset.
7.4.2 Architecture Effect: T5-based Models Perform Best Across All Tasks
Model size and performance are also positively cor- related, although they reach diminishing returns. For instance, the model performance is roughly (log log n) with respect model size on Seman- tie Classification (Figure 6 in Appendix). On the Autonomous Web Navigation task, performance grows slowly with the model size (Table 8), while on the Description Generation it plateaus (Table 3).
Encoder-decoder T5 based models perform better across all three tasks. On the Autonomous Web Nav- igation task, -ecoderd (WebN-T5) architec- tures are better or comparable to WebN-LaMDA- IB (Figure 2a). On the Semantic Classification, the smallest encoder-decoder model (WebC-15-base) performs comparably to much larger decoder-only models (WebC-LaMIA--I B or WebC-Pal.M-8B) and the largest encoder-only model (WebC-BERT- large) which has 85M more parameters (Table 2). We also observe that decoder-only PaLM-8B per- forms worse than much-smalle encoder-decoder Targerge when trained only on HTML data. Finally, on the Description Generation encoder-decoder architecture has higher BLEU score.
7.4.4 Error Analysis
We manually examined 50 errors of WebC-T5-3B model over the development set (Table and as- signed them into one the error types that we devised. We found that 32% of the errors are due to lack of information in the HTML snippets, which is mainly the result of lost information during snippet extraction process. Annotation errors or email/user- name ambiguity make up 30% of the errors. These can't be improved without revising the annotated data or adding extra information to resolve the am- biguity. We also found that the model sometimes picks a more general category, or a nearby text mis- leads the model; the latter usually happens when the HTML snippet is long where majority of the elements are noise.
One possible explanation for the strong perfor- mance of b5-based moels is the encoder-decoder architecture of these models. Namely, T5 models utilize encoder with bidirectional attention mechanism, not present in the LaMDA and PaLM decoders. The bidirectional attention mechanism can process HTML pages from both ends, poten- tially overcoming the loss of information when trec-structured HTML pages are converted into fixed línear text sequences.
7.4.5 Few-Shot Prompting
In Table 5, present few-shot prompting per- formance of a 540B PaLM model. We probe the model using prompt template <html> Role: <category> with example per category and gen- erate categories using greedy-decoding In our
7.4.3 Model Size Effect: Size (Sub-linearly) Matters
Across the tasks it appears that the architecture plays an important role in the model performance.

---

Error Type
Percentage of Examples
Not enough information in the HTML snippet
30
Incorrect annotation (ex: "unknown, role" instead of "organizationn"
12
Annotation tool translates user selection incorrectly
8
Eamail/Username ambiguity
10
More general category (ex: "header" instead of "eadeer)
8
Immediate neighboring text misleads
8
Incorrect date formatting (ex: "mm" instead of "mmm") No information in the HTML snippet
4
2
Others
18
Table Types of errors over manually examined examples. 32% errors are due information HTML snippets, 30% of errors are related to annotations or can't improved to ambiguity (email/usemame), and remaining ervors are incorrect predictions by the model.
Model Name
Test
Dev
els. We find that a) pretraining is critical for the performance and can reduce labeled data require- ments, improving sample efficiency up to 200x; b) model architecture is the second-most important factor, and T5 models with bidirectional attention and encoder-decoder architecture perform the best across the board; c) given a choice, model size should be evaluated the context of the model's 's training and inference performance, as the model size sub-linearly correlates with its performance. Finally, the proposed HTML understanding tasks highlight the relatively short context window that limits current LLMs, suggesting possibilities for future research that incorporate eliminate this constraint.
PaLM-540B
64.2
60.3
w/o Example Cleaning
57.9
57.2
w/o Category Rewriting
52.1
50.7
w/o Dictionary Mapping
45.6
45.1
Table 5: Few-shot prompting performance with different pre- and post-processing steps.
preliminary experiments, we found that few-shot prompting achieves only 45.6 accuracy, much lower than model fine-tuned on the same data. We found two common problems - the model is not able to canonicalize predictions into categories and many of the examples are dropped due to con- text length.
We developed post-processing methods to al- leviate the canonicalization problem and pre- processing methods reduce lengths of examples. Adding dictionary-basee mapping on predictions a manually curated paraphrase dictionary im- proves the performance to 52.1. We also tried rewriting predictions by changing the order of to- kens around such as name_fira to first_name name which further improved the performance to 57.9. Finally, cleaned examples in the prompt by removing certain elements such as "svg". "path", "img". and "iframe" and also removing class at- tribute from every element; this pre- pro-procss step gives 64.2.
Limitations
Our experimental results are limited to relatively short context windows. While HTML documents can have 10s of thousands of tokens, LLMs that we studied have only 2K context windows. We de- veloped heuristics to extract HTML snippets to into context windows which is a promising future direction. We are also limited by the MiniWoB sim- ulator for our web navigation experiments. While MiniWoB serves wide variety of simulated web- sites, they still lack many of the complexities of real websites.
References
8 Conclusion
Leonard Adolphs, Benjamin Boerschinger, Christian Buck, Michelle Chen Huebscher, Massimiliano Cia- ramita, Lasse Espeholt, Thomas Hofmann, and Yan- nic Kilcher. 2021. Boosting search engines with in- teractive agents. arXiv preprint 109.v22109.00577
We presented canonical tasks and fine-tuned LLMs for HTML understanding. The comprehensive eval- uations and analyses over range of architectures, dataset sizes, and baselines yields practical findings and highlights current limitations of these mod-
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis.

---

Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. Htlm: Hyper-text pre pre-training and prompting of language models. arXiv preprint arXiv:2107.06955.
Izzeddin Gur. Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. 2021. Environment generation for zero-shot compositional reinforcement learning. Advances in Neural Information Processing Systems, 34:4157- 4169.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. 2022. Do as i can, as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169
Izzeddin Gur. Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. 2019. Learning to navigate the web. In International Conference on Learning Rep- resentations.
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen. 2021. Actionbert: Leveraging user actions for semantic understanding of user interfaces. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 5931 -5388
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language mod- els. arXiv preprint 200.0023...
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Mul- dal, Josh Abramson, Petko Georgiev, Adam San- toro, and Timothy Lillicrap. 2022. A data-driven approach for learning to control computers. In In- ternational Conference on Machine Learning, pages 9466-9482, PMLR.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 137779111
Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba. 2019. DOM-Q-NET: Grounded RL on structured language. In International Conference on Learning Represen- tations.
Chuck Jorgensen and Kim Binsted. 2005. Web browser control using emg based sub vocal speech recogni- tion. In Proceedings of the 38th Annual Hawai ternational Conference on System Sciences, pages 294c-294c. IEEE.
Andrea Burns, Deniz Arsam, Sanjna Agrawal, Ranjitha Kumar, Kate Sacnko, and Bryan A Plummer. 2022. Interactive mobile app navigation with uncertain under-specified natural language commands. arXiv preprint 2iX.::2222.02112
Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint 2rXiv:2001.08361.
Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. WebSRC: dataset for web-based struc- tural reading comprehension. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 4173-4185, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Con- ference on Learning Representations.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311
Chenliang Li, Bin Bi, Ming Yan, Wei Wang. Song- fang Huang, Fei Huang, and Luo Si. 2021a. Struc- turallm: Structural pre-training for form understand- ing. arXiv preprint arXiv:2 105.1 12200.
Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. 202 Markuplm: Pre-training of text and markup language for visually-rich document understanding. arXiv prepriat 2rXiv:2110.080
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv: 1810.04805.
Zimeng Li, Bo Shao, Linjun Shou, Ming Gong, Gen Li, and Daxin Jinng 2023. Wiert: Web information extraction via render tree. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):13166 13173.
Oscar Diaz, Itziar Otaduy, and Gorka Puente. 2013. User-driven automation of web form filling. In In- lernational Conference on Web Engineering, pages 171-185. Springer.

---

Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tian- lin Shi, and Percy Liang. 2018. Reinforcement learn- ing web interfaces using work flow-guedd explo- ration. arXiv preprint waop::802000000
Romal Thoppilan. Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-TZe Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lec, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung- Ching-Ching Chang, Igor Krivokon. Will Rusch, Mare Pickett, Kath- leen S. Meier--Hllsterm, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenhergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoe Le. 2022. Lamda: Language models for dialog applications. CoRR.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. 2021. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.0524..
Sahisnu Mazumder and Oriana Riva. 2020. Flin: A flexible natural language interface for web naviga- tion. arXiv preprint arXiv:2010.128..
Relichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint ar2iv:2112.09332.
Rodrigo Nogueira and Kyunghyun Cho. 2016. End-to- end goal-driven web navigation. Advances in neural information processing systems, 29.
Daniel Toyama, Philippe Hamel, Anita Gergely, Ghe- orghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. 2021. Androidenv: a reinforcement learning plat- form for android. arXiv preprint arXiv:2105. 13231.
Christopher Olston, Mare Najork. et al. 2010. Web crawling. Foundations and Trendy@) in Information Retrieval, 4(3):175-246
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information process- ing systems, 30.
Panupong Pasupat, Tian-Shue Shun Jiang, Evan Zheran Liu, Kelvin Guu, and Percy Liang. 2018. Mapping nat- ural language commands to web elements. arXiv preprint 1rX8v:8808.00133
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, and Jamie Brew. 2019. Huggingface's trans- formers: State-of-the-art natural language process- ing, CORR, abs/1910.03771.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Toolllm: Fa- cilitating large language models to master 16000+ real-world apis.
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A mas- sively multilingual pre-trained text-to-text trans- former. arXiv preprint urXiv22010.11934.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real- web interaction with grounded language agents. arXiv preprint arXiv:220..01206.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Her- nandez, and Percy Liang. 2017. World of bits: An open-tlmmin platform for web-based agents. In In- lernational Conference Machine Learning, pages 3135-3144. PMLR.
A
Appendix
A.1
Brief Background on HTML as Semi-Structured Text Data
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meet- ing the Association for Computational Linguistics. Association for Computational Linguistics.
HTML is a markup language, used to organize web page structure and content. Consider the ex- ample HTML page in Figure 1. This web page includes two adjacent input elements, one for e- mail and another for password, with their corre- sponding labels on a separate branch of the page. These inputs and labels are one of many possi- ble elements that serve as HTML building blocks.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases.

---

Each element has of attributes - key and value pair that describe the element's content, such as style and human-readable text. When rendered in a browser, these attributes will be responsible for how the element shown and where is posi- tioned. In the example in Figure 1, the first input has three attributes, tag="input", type="email", and uN="u, that identify the element email input with identifier ("uName") that can be accessed programmatically.
candidate generations at each step. While don't constrain the vocabulary of our models during gen-
eration, it is important to note that output tokens in our dataset typically come from input HTML documents or user instructions (with formatting tokens such as "t", ":", "I" and special "click", "type" tokens being exceptions). The models easily learn these implicit constraints and assign much lower mass to tokens that are not in HTML docu- ment or instruction. Additionally, they also quickly learn the desired parsable format ("<action type>, <target HTML element identifier>, <instruction substring for typing actions>") where the target HTML element specified in the action output is actually present in the input HTML (so it can be acted upon). We estimated the average statistics of the public demonstrations that used to train our models and found that there are 54 elements in a given HTML document and 35 instruction tokens, on average. So our action space approximately 34 * 35² where there are action types and 35² number of substrings in an instruction.
A.2 Dataset and Pre- Pre-Processing Details
Examining the description distribution, we found the original 400 dataset to be very skewed; only 20 descriptions (such as Email and Password) were covering 50% of the dataset. We sub-sampled the dataset that each unique description has at most 10 data points. We also found that for attributes are almost always defined for HTMI. labels. This could cause a model to overfit and just find the label element in the HTML and ignore everything else. To avoid this sort of "cheating' we replace the tags of HTML labels by randomly sampling from (div, span, a, label]. These tags are also fre- quently used inject text in HTML but they very rarely used with for attributes. Finally, we re- moved examples where there are only single text the HTML since models can trivially generate descriptions by finding the only text in the HTML, which biases model weights and evaluation metrics. After this final step, we have total of 85K labeled examples.
A.4 Sample Episodes from MiniWoB
See Table 6 for an example episode of web naviga- tion inferred by a fine-tuned LLM.
A.5 Detailed MiniWoB Results
See Table 7 for detailed performance of various models on MiniWob.
A.6 Resource Requirements
We didn't apply any special filtering keep only text related information. We did minimal pre-processing filter some of the attributes from excessively long inputs. For semantic classifica- tion, we applied no filtering on extracted snippets. For description generation, we filtered "class" and "style" attributes as we found them to increase the length of the HTML documents significantly. In MiniWoB, applied no filtering and used the original observation space provided by the environ- ment.
See Table
A.7 Structure Dependence Ablation Study
We conducted an ablation study examine the sensitivity of model performance to preserving structural information. To do so, we evaluate the model's performance on HTML input with critical structure components removed. We kept the order of clements and their attributes fixed while corrupt- ing the nesting structure by removing closing tags. Removing closing tags corresponds to valid traversal (BFS) and keeps the order of elements the same as the text based input.
A.2.1
Snippet Generation
In Figure 5, we give a high-level overview of our snippet generation procedure.
As a simple example:
<div id=""orm""xdiv>xinn"" id="username"> </div></div>
A.3 Action Space for the Autonomous Agent
We set the number of tokens that our models can generate to 20. Our vocabulary consists of 32K tokens, which gives an initial estimate of 3200020
would be converted into:
<div "d=""orm"><div><>>"" id="username">

---

HTML
with
eeperddalle
saffent elements
snippet goneration
expand
othaawwss ousputt
loval
Flave mit you
up
Thees porr
High-level overview our pre-processing pipeline for generating snippets from full HTTLL webpage. Given the page, we detect salient elements and for each one of them we extract snippets by recursively moving up the HTML tree until validation heuristic fails.
Figure 5:
Figure 6: Performance comparison w.r.t. increasing model size. As the model size increases, we observe increase in overall accuracy with PaLM-62B model achieving the highest accuracy while being 7x larger than PaLM-8B. fit two different curves using the following functional forms: y * (og + b and = a (n)( + where n is the number of data points, and a and b are parameters to be learned. We found that the average enror was 3.75 and 4.78 for loglog and log fits, respectively.
standing (i.e., trees of elements with a richer struc- ture and functionality). It is trained only on the textual content of a document the markup infor- mation is ignored. For example, any input field or dropdown in document would be missing from the model inputs. All of the tasks we study re- quire knowledge of this information. For example, in autonomous navigation the model needs to in- teract with input elements (e.g. text, checkboxes, dropdowns) such as username and password in the login-user task in MiniWoB. Typically, a "type" ac- tion with a reference to element and text argu- ment is generated by the model. Without knowing which input elements available in the page, it is impossible generate reference to any input element.
100
encorier
fit
test enc-dec
dev enc-dec
dev fit
Accuracy (%6
test ciecodier
dev decodar
90-
While MarkupLM ((Li et al., 2021b)) is better tailored for understanding HTML pages, it has sim- ilar drawbacks as StructuralLM in that it focuses solely text and structure of text while ignor- ing everything else in the markup. To illustrate our point better, we used the open source imple- mentation of MarkupLM from the HuggingFace library ((Wolf et al., 2019)) to process the sam- ple HTML snippet in Figure-1. 1. The MarkupLM ignores all input elements, both username and pass- word, and generates 
Email AddressEnter Pass- word:Please enter your password. 
 which is the text input to the MarkupLM Transformer (Fig- ure 7). Classifying this text as username or pass- word is not possible without the additional context on which input element is the salient element (in this context it is the username).
10
10a
Model Size (In millions)
We evaluated the trained WebN-T5-3B model the same set of synthetic websites from the MiniWoB benchmark with this aspect of strueture removed from the HTML pages. WebN-T5-3B achieves a 45.4% success rate, 6% lower than be- fore, suggesting that WebN-T5-3B is at least par- tially dependent on the DOM topology.
A.8
Additional Related Works
A.8.1 Task-specific Models
An alternative to LLMs is to adapt bespoke task- specific architectures tailored towards processing of structured documents and HTML ((Li et al., 2021b,a)).
MarkupLM is also evaluated NLP-like tasks such as QA or entity classification where under- standing page content paramount, whereas we focus on HTML understanding tasks such as au- tonomous navigation where both content and the page's 's layout structure need to understood.
StructuraLLM ((Li al., 2021a)) an approach specifically tailored for document understanding (i.e., combinations of images and text), and thus makes several simplifying assumptions for its model that limit its applicability to HTML under-

---

enables degree of horizontal scaling and gener- alization as studied by these works. However, this base model on the semantic classification task, us- ing the same setup as other WebC models but with the suggested hyperparameters from ((Li et al., 2021b)). We use the MarkupLM implementa- tion from the HuggingFaace library ((Wolf al., 2019)). On development and test sets, MarkupI.M- base achieves 65% and 66% accuracy, respectively. These results are more than 16% lower compared to similar size WebC-BERT-base results that we report in our work. This suggests that although do- main specific models may be suitable for process- ing HTML for NL.P tasks, the generality, flexibility, and sample efficiency LLMs provide advantages for autonomous navigation tasks.
We perform quantitative evaluation of Marku- pLM on our tasks to understand how significant these limitations are. fine-tune the Mar
flexibility only achieved when the environment implements an API interface. Where there are none available, navigating web interfaces becomes cru- cial to unlock access to further information.
A.8.2 Web Information Extraction
WebSRC (Chen et al., 2021) introduced QA task from web documents. The authors manually rated small set of seed questions (460 questions) which are used to collect paraphrases and do data augmentation. Similar to DescriptionGGencration the problem is to find the relevant text in web doc- ument. While DescriptionGGenrrtion is focused more elements and fully automated, WebSRC is QA task that requires manual annotation.
WIERT (Li al., 2023) studies information ex- traction from web documents. They focus on uti- lizing simplified render trees (only text, tag, and style information are kept) to classify DOM nodes into product categories. They traverse the render tree to generate sequence of element tokens and encode the sequence with pretrained language model; style nodes are independently encoded and concatenated the element encodings. The result- ing encodings are used multi-objective setup to train the model. In contrast, our models are trained in a unified setup with single objective, and using original HTML documents with no special filtering required to extract informative elements.
A.8.3 Tool Use
More recently, tool-using agents emerged as way to augment LLMs' capabilities with external tools. ToolAlpaca (Tang et al., 2023) and TOOLLLM examples that study multi-
(Qin et al., 2023) step tool-use with iterated API calls. Actions as well-defined API interfaces can be useful to sim- plify and reduce the dimensionality of the interface between the model and environment. This in turn

---

from transformers import MarkupplMProcessor processor wttpourrrreesssreeaattarrrarrrrr
snippet = <div>xlabel class="form-labee" for="uName">Emiil Address
</label><label class="formmlabel" for="pass">Enter Password:
</label>z//ivxxxiixxxiuu type="email" ud=""Mame"""""" target><input
type="password" id="pass>><span class="hidden"">Plsss enter your password.
</span></div>>""
encoding processor (snippet) print (ararrrroeesser.cccccceeddddddde iicch
Figure 7: Code snippet reproduce MarkupLM preprocessing result. Encoding the example HTML code with the MarkupLM preprocessor will ignore all input elements.

---

Table 6: A sample web page and corresponding episode using the T3-3B model. At each time step, previous actions, instruction, and HTML are concatenated into a single HTML text. Note that the beginning of episode, there is no past actions and we simply concatenate instruction and HTML. Action is generated as a sequence of tokens which is later parsed into a dictionary. The ref in the action points to an element that has a ref attribute with the same value. For instance, at the beginning of episode, ref: 6 corresponds to an input with ref=6. At the end of the episode, the model clicks on the submit button and the episode terminates.
Web page
aw, mMpY WVMOVa 1132 Submit
mMpY
YVYMOV
DJIG
1132
Submit
HTML Text
Action Text
Select qu, nMp,, YVYHQVa, DJEG, 1138 and click Submit. <body ref- ref-"1"><div
id-"wrap" ref="2"><ddv id="ares" ref-"3"><div id-"boxes" ref="4">< ">><abell ref="5"> <input "checkooo" id="ch0" ref-"6" "Alue="Pale"></innp"
class= "TEXT\ "PEXS'" ref="None /bboo//dgcbbb//laabbaaaaaa ref="?"><inpur
"ype="checkbons id="ch1" raf="s' value= "alue==False""
"ralse"></inpu>>> class= CLASS'
rof="Hone">hane
"rre="cceckbox" id="ch2"
ref="10" value="False, "=aiss"></ipppt>> class- "TaxT="TEXTV CLASS" ref= "None" =ef="Kone"=alppX"15
</label><abbel "iiiiputtt type "chckboo"" id-"ch3" ref- value- "Palse">
</input><t class- "TEXT> CLASS" ref-"Hone wvtg/aa//><//aabe/>>>>>
13"'
<input type="checkbox" id-"ch4" ref="14" "Flse""allse><"
class- 3laxT='TEXTT, _CIASS" rof="None" DDoone">DJfG////ea abbb>><abaal ref="15 <inipptt
"rpe=''chekbox" id="chs" ref="16" "False >>/inppt><t class= 'TXXT> CLASS'
sef='Hone"'un ref- "17"=Sobait</battttttt http://ttttoooc//..//e/////eeeee /body>
wtt--////////a///aa/// /atp:///aabbl//i///tttttttt divt-bbttonnn id= "sbbbt" class=" class="secondan "secondary-action
(action: click, ref: 6)
(action: click, ref: 6) Select mMp,, YVTHOVa, DJEG, 1138 and click Submit. <body ref="1"><div id="wrap" ref='2"><div id-"area" ref='3"><div "boxes" "eclael ref="5"><ingur sypee"checkbon id="ch0" ref="6" value="True" </inpue><t elass= CLASS" "eee"Nooo">sss//// /t/p///////aa//>alaeeell refpnt"><input type="checkbox" id="ch1" ref="8" "allse"Palse"eee "paistt></input><t class="TEXTV, CLASS"
/ttffaa//////>//aaabbbllaa class="TEXY" CCAAS" "on=""Nne" ">Mpp<</>>>>>>>
ref= "of="""><ingg" ">>ciitttt type='checkbox "checkbox" id="ch2"
ref="10" value="Palse"></ippeee
</label><label ref="11"><inpp"" type= type="checkbox" id="ch3" ref="12" Value="False
</input><t class= CCAAS" ref "None"
ref= 113>
<input type="checkbox" id="ch4" ref="14" "alue="
"aee""TEEE__s_ee"eeeee "CLAAS" ref= "xf="Hon"">>OO0: saaff///><</abee>iiee /tipplle/e//////atttttt innput<tptttttt
class="TEXT),
ref="15"><inppp
sype-"checkbor id-"ch5" ref-"16" value="False" "False">< >>ipptt</ class-"TEXT", CLASS"
ref="None" wttt>>/////////aa///aaa /iioe//ttt///eeeetttttooooo id- "subbta" class- seeaos"sccondry-accte
ref- "17">Submit> beabbnitt/bbtttoon///baeae http://uttooo///ioo//////////-// /ivp////////////od////eee /body>
{action: click, ref: 10)
(action: click, ref: 6), (actions click, ref: 10) Select qu, aMpY, YVIиоVa, DJEG, 1132 and click Submit. <body ref="1"><div id-"wrap" refeii">< id "area"
ref="3"><div id="boxes' ref="4"><" "><labbl ref="5"><e "5t=inutc type- "checkbox" id "ch0"
ref="6" value-"True >><Iinpu>><t class- "TEXT> CLASS" ref- "oone">qw<//ei />t////><//a/bb
<label ref-pu">< "checkbox"""" id-"chl" ref-"8" value- "False ></ input>> <t class="TEXT) CCASS" ref= "Noee" hhtrton///>></labei>tt
<stiig"><inputt
"oree="Trr>>><<""""" ref="None" "atn""">m/p///><</aabba lbbel ref= ref="i1"><input type="checkbox" "d="ch3"
"cpe="cchckbox" id="ch2" ref="10'
class= "TEXT) CIASS"
ref="12" "rllse" >></inut><<t class- CLASS" ref- "Hone" YVVMgVa</>>>>>>
</label><labei ref="13"><inpu"" type- "bbeckobeo id="ch4" ref="14" value="Falee
</input><t class="DEXT) CLASS"
CCASS ref vef-"None "Nose" wttee/////ee///aaee/eeeeaaaa
ref- "15">
kinput type="checkbox'
"checkbox" id-"ct5" ref-"16" value- "Palse intputt>
class-"'EXTT, CLASS" ref-"None 2o038<> "scooddscconar ref- "iun-iiiniittteeee /sttn/tt//atttooo/////////t/o///
subbeu"
{action: click, ref: 12)
(action: click, zef: 6), (action: click, ref: 10), (action:: click, ref: 12) Select qu, nMpY, YVYHQVa, DJEG, 1132 and click Subnit. <body ref-ii's<divv id="wrap' ref-i?"><div idr"area" ref="3" ref="3'><div "dxee" ref-"4" "><labllll ref="5"><inn "S"iipput "ypee"cheecbox" id="cho" ref='6" value=
>>/input><tt class-"TeXT)
"etnnntt "ype=cchecbbor" id="ch1" ref="g" valce="False "></>>/////nn ></input><t class="TExT" CLASS*
rof="None "Kt="Kono">qus/bbee wttpr/////labb>>iaaaaa
"EAASS ref= 'Hone" (etrtDna<//><//abe>>/]a3//
rof="g"><input type="chockbox id="ch2" ref="10" value='True">>> "/inputtt
class="TEXTT, cLass" bef="None /atpp<</><</abbl>/l1
rof=i11'><input
type type="checkbox" id= ref='12" ralue="True"s " </inppt><tt class claxT=' CLASS"
refe"None" "rE=""Knn" "vVVw/Va httaaa///t///aa//aaeeaaaaaaaeeee
ref="13"s Siipptt sype="cheekbon" id="ch4"
ref="14" value="Falee "></ <tput><t class- "TEXT), CLASS" ref= DDEE"//
</label><label reš="15" 1ef="15"><ingg <input tcpee""ceeckbx" id-"ch5" 16" value- "Palse">
</input><t class-'TEX" CLASS* ref- "None" (113///b></labbD>///voooooo
id- "subbt"" class-* secondary-action" ref- "1 Subnit< httpat/te///botooo//////////a/aa /body>
{action: click, ref: 14)

---

(action: click, ref: 6), (action: click, ref: 10), (action: click, ref: 12),
(action: click, ref: 14) Salect qu, MMp,, rVYиQa,, DJGC, 1138 click Submit. <body ref="1"><div id="wrap ref='2"><div id="area" ref="3"><div id="boxes" nof="4"><label ref="5 rof="5"><input cyee="check id="cho" ref="6" value="True's </input><t class="Txx? CLASS" ref= "No=="Noe" /oonn/""g//////aaaaallaaaa re<="7"s type-"checkbox "heckbox" id-"chl"
ref="Hone">hiee "hatEaa////><<//aaba>-aa value= "=wie="aaeee""eecceeee ref-ig"><input "xpee"cbeckboxx id""ch2"
"TEXT\ CLASS"
ref-"10" value=" "tt/tteeee/iiippiie "x/ippttttt class- "lass="TExP\_GGa" CLASS" ref 'Hone ">Mppt/t><////bbbab
<label ref="11>><inpu type-"checkbon id""ch3" ref- 12" value- "True <>/ppt>>>>>>>
clesg="TEXT),
"None" wwtHVaa</><</aabb>>>daa class= 'TXTT CLASS"
"ef-itt">xinput
"op=='cheebbox" id="ch4" ref="14" ">oo"""rr">></nppoi
ref= "None" D///////>//////////////////// //tp///////abbbblbbbbbb rof=iit"> sype="checkboo" id="ch5"
ref="16" malue="False ></input><t claxs="rEXT\, CLASS
1>133</t>>
//labeD></div>-<button id id="subbta" clacs="secomdary ref- Submitt
(action: click, ref: 16)
(action: click, ref: 6), (action: click, ref: 10), (action: click, ref: 12),
(action: click, ref: 14), (action: click, ref: 16) Select qu, nxp,, XVYNQVa, DJEG,
<hody ref-"1"><div
"wrap" ref="2"><ddiv
refci3"><div id= "dxxe' ref='t"><label ref="5"><input type-"checkbox" id="ch0" ref="6" "rlue="rrue><<LLnptee "><Iinpat>ct (TEXT) CLASS" ref- "None" "None" "ooee=sgrstb<</abbl>>>
"
<label ref="""><input type="checkbox" id="ch1" ref="8" "alue="ralse"></1tpp" class="TEXT\, CLASS" ref="None /tBfftt/////////aaaa//
"checkoox"
ref="9"><<nput
type="checkboxx id="ch2" ref="10" value=" "x/iprrprrr>//ttt>>ttt "IEXT" CLASS bef="None"*an "roon"""mttp///></ababbblaaa
refiiit"" sype="cheekbor" id-"ch3"
ref="12" "ule=""ruu>><</inutt" class "TEXT) CLASS" ref- >VVMoWas/t>>
</label><label ref-ii3 type= "checboox" id-"ch4" ref-"14" value- "True" "> </input><t TEXTT CLASS" ref= "None" "Dfff//>><clabba>>aaaaa
refiittt "checkbox" id="ch5" ref="16" value= "True "rren><<inppu><tt cIEX)=" CLASS" value="True"
"None "None"> 3113///><//abbbb//ddddddd
id="aubbtn" class= class="secondes sscondary--ction
ref="17" Subniit Stt-Suiittbbuuttoeeeeeee /ttp://tttoooi/ippd///i/////////
(action: click, ref: 17)

---

Table 7: Success rate comparison of various models in MiniWoB tasks, Baseline results are borrowed from (Humphreys et al., 2022). Note that these are normalized between and
BASK
Haraz
CC-Ns
Modl
WobN-T5-3B
WeIN-T5-S
SODA
lits
explonukc (Nugneend)
RL.
Kil)
web Chngreomed
(R1)
liscct
0.29
0.5
book flight
0.7
03
009
503
023
0225
0235
89
2
6.11
Cask
8/
click dakkg
Lidk-ank
diccemmms-2
0.16
15
tlid-man
0.90
6.13
ciicoagron
1.99
0.31
0.34
clicopis
0.14
0.19
6.15
6635
dikk scril Est
n.1
000
6/07
10
10
89
0.07
0.00
Ghades
0.4
0.70
0.39
4b
2ick-atape
0354
011
6:11
0034
dlok-ab-2 Lacyy
0.99
66
0.99
061
899
4b
0.13
01.9
186
did-alb3 median
0.3
674
a/u
155
0.0
022
600
004
035
810
uldd-ahh
0.99
1034
003
600
0.35
095
633
10
0194
m0
811
x/4
085
0.97
0.55
6.34
0:33
0.30
195
D61
0011
oug poste
1/6
0.79
0.04
103
coern shage
0.41
0.43
0.85
0.31
6.18
0.59
0.76
0.51
0.77
165
11
11.
10.1
Org bos
0.99
1/
066
0.31
0.31
0:31
fngg cille
1:9
160
1075
100
6:18
10
100
100
0:18
01155
ung-len
036
Jng latt
033
06
1335
600
600
81
879
81
0301
01001
inaeiiane
103
6.11
0141
tkgellapes
0.66
0.22
0.92
0.92
1066
0.99
65
mwil-abee foraadi
888
avoik-rbes hewanii
191
61N
inbor forwad
036
0.03
orall-rbe mirn
1071
1755
600
1191
1377
email inbos noserdl
0.6
trloo
oidgeiat
6/
0:31
foces
a/
#icm number
iest 1est
16:
lighintsacu
033
lixstis stgee
0.66
sta
1006
0.43
hym-aer
0.06
0.64
mesing here
11
0.13
6.44
503
a/n
muli cederings
836
0.41
0005
a/o
a/o
ab
0,85
rasigane mee
no6
0.91
0.32
0.
0:99
0.59
6.16
501
40
835
a/
6/
a/o
893
8/
15al Tike
600
16
100
303
610
400
103
810
IRil0 souurea
635
m5
S.11
0.11
aigll-nngle
033
005
6:38
810
0.58
amillaceed
10
0.55
6000
0000
vemell sou
634
15.
603
0034
278
warchveegie
5.34
0.54
0.15
0
1136
15:00
811
smmn-ass
n82
0:28
x/x
x14
0.25
836
0:75
0/33
C.14
10
103
45
0.81
60
11:6
0.54
611
10
60
851
0.0
woldl medio all
0.9
0
0.75
m/
001
a/
0.01
0.01
secis media some
n91
B02
0
0.95
0.03
001
a/o
0.45
0.11
0.42
1011
033
119
0033
8.91
10390
4th
a.s
66
000
0
4/
8/
Test Jest
m35
06
1335
611
600
40
80
400
803
81
0301
0101
036
619
6
8.1
6:31
0.20
0133
100
8:34
0:37
89
40
0141
81
0.57
0.27
109
0.5
60
811
036
632
0115
607
0:75
0.56
81
0.75
0.98
1645
0.14
85
Incsorlestecl
69
65
1148
066
0/3
6.15
ras-lider
69
014
0.51
005

---

Table 8; Resource requirements and running time of LLMs.
Model Name
Model Size
TPU version
Batch size
Input sequence length
Examples per sec (training)
Examples per (inference)
PaLM
62B
TPU y4
8
1920
9.313
30.51
PaI.M
8B
TPU v4
32
1920
64.4
184.3
T5
3B
TPU v4
128
512
163.8
734.5
T LaMDA
IB
TPU v2
128
512
363.1
1416


Reviews:
[{"review_id": "IUeTxWdp9c", "reviewer": "Reviewer_X9Ao", "summary": "", "questions": "- Why these 3 tasks? Please justify the selection of these 3 tasks for HTML understanding. \n - _Autonomous web navigation_: what is the action space (i.e., the actions that the agent might take, and their parameters)? Specify these in the main text.\n - L600: $O(\\log \\log n)$: This is unusual and explanation should be provided to justify this approximation.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback!\n\n**Large Language Models with Emergent Abilities.**\n\nEmergent abilities such as chain-of-thought reasoning appear only with scale and are not present with smaller model sizes. [1] is a recent work that extensively studied these abilities using varying model scales, model types, and downstream tasks. One of the key findings in their work is that these abilities emerge at different scales ranging from 40M to 540B model sizes (see Table 1 in their work): Scratchpad: 8-digit addition (finetuning) (LaMDA 40M), Ask me anything prompting (EleutherAI 6B), Differentiable search index (T5 11B), Chain-of-thought: StrategyQA (PaLM 62B),  Multilingual chain-of-thought reasoning (62B), Word in Context (WiC) benchmark (PaLM 540B), etc.\n\nIn our work, we study fine-tuning of models from BERT 24.6M to PaLM 62B. Our results highlight that Semantic Classification improves with scale (WebC-PaLM-62B improves 2.1% (0.6%) on test (dev) split compared to WebC-PaLM-8B) which suggest that HTML understanding might be benefitting from emergent abilities of larger models. We also believe that it is an interesting future work to study which of these emergent abilities help the most with HTML understanding tasks in our work.\n\nWe also studied one-shot prompting of the largest model studied in [1] – a 540B PaLM model (see section A.2.1 SEMANTIC CLASSIFICATION / Few-Shot Prompting and Table 5). Our results indicate that naively using one-shot prompting results in 45.6% accuracy on the test split. The main problem is the limited context length of the model (2K for PaLM) where prompts with even one-example-per-class are longer than its context length. We propose several improvements such as Example Cleaning, Category Rewriting, and Dictionary Mapping improving the accuracy to 64.2% which is still far from our fine-tuning results. We believe context utilization for HTML understanding with a larger number of classes is an interesting future work.\n\n\n[1] Emergent Abilities of Large Language Models. Wei et al. (https://arxiv.org/pdf/2206.07682.pdf)\n\n&nbsp;\n\n**Motivation behind Semantic Classification, Description Generation, and Web Navigation.**\n\nOur ultimate goal is to develop autonomous web navigation capabilities that require solving the “perception” problem – understanding HTML documents – (1) to classify elements into their respective abstract categories (such as username or password) and (2) generate textual inputs to interact with these elements (such as type(username, “myusername”)). Semantic classification tests a model’s capability to understand an element within its context. For example, an element in a web form can be classified as username, allowing an agent to retrieve the relevant information from a user’s profile. On the other hand, description generation tests a model’s capability to generate relevant text for a given element. For example, a text field can have a label “Username:” which is important to generate in a navigational action to correctly enter the real username.\n\nWhile we are limited by the capabilities of the simulator that we utilized (MiniWoB), we believe it is important to also understand the gap between real-world and simulation. For both semantic classification and description generation, we use real-world HTML documents. Our results indicate that fine-tuned LLMs (WebC-PaLM-62B and WebD-T5-3B) can achieve 90% and 84% accuracy in grounding an element into its relevant category or description.\n\n&nbsp;\n\n**Action Space for the Autonomous Agent.**\n\nWe set the number of tokens that our models can generate to 20. Our vocabulary consists of 32K tokens, which gives an initial estimate of $32000^{20}$ candidate generations at each step. While we don’t constrain the vocabulary of our models during generation, it is important to note that output tokens in our dataset typically come from input HTML documents or user instructions (with formatting tokens such as “{“, “:”, “}” and special “click”, “type” tokens being exceptions). The models easily learn these implicit constraints and assign much lower mass to tokens that are not in HTML document or instruction. Additionally, they also quickly learn the desired parsable format (“<action type>, <target HTML element identifier>, <instruction substring for typing actions>”) where the target HTML element specified in the action output is actually present in the input HTML (so it can be acted upon). We estimated the average statistics of the public demonstrations that we used to train our models and found that there are 54 elements in a given HTML document and 35 instruction tokens, on average. So our action space is approximately $2\\*54\\*$35^2$=132300$ where there are 2 action types and $35^2$ number of substrings in an instruction.\nWe will clarify this in our final manuscript as well.\n\n&nbsp;\n**LogLog Growth of Accuracy with Data Size.**\nUsing the results from Figure-4, we fit curves with the following functional forms: $y=a\\*log(n)+b$ and $y=a\\*loglog(n)+b$ where n is the number of data points, and a and b are parameters to be learned. We found that the average error was $3.929696e-04$ and $1.639361e-04$ for log and loglog functional forms, respectively. While loglog functional form has smaller error, the difference between these two numbers is also small and might not be significant. We will give more details and clarify in our final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new benchmark for testing the performance of models over webpage understanding.", "reasons_to_reject": "- The paper claims to test _large_ language models, but the models that the authors tuned are mostly as big as BERT, with PaLM-62B as the largest one. In recent literature, when researchers refer to a language model as  _large_, it generally means that it should exhibit some _emergent capabilities_ such as _in-context learning_ and _chain-of-thought_, and such benchmark should definitely evaluate the performance of larger LMs (e.g. LLaMa, GPT-3/4, Codex, etc.) under zero- and few-shot settings. This paper is completely lacking this respect of large LMs.", "questions_to_authors": "- Why these 3 tasks? Please justify the selection of these 3 tasks for HTML understanding. \n - _Autonomous web navigation_: what is the action space (i.e., the actions that the agent might take, and their parameters)? Specify these in the main text.\n - L600: $O(\\log \\log n)$: This is unusual and explanation should be provided to justify this approximation.", "paper_topic_and_main_contributions": "This paper proposed a benchmark of 3 related HTML understanding tasks to evaluate the performance of language models in terms of their ability to understand webpages in HTML source. It also fine-tuned a variety of models to complete the proposed task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L008: \"contribute\" => \"contribute to\".\n - L119: Double parentheses."}, "extra_text": {}}, {"review_id": "G9FiQxm3Ou", "reviewer": "Reviewer_NHjz", "summary": "", "questions": "A. Can be a more detailed explanation of the dataset creation (Section 4) or the pre-processing (Section 5) provided? In general, raw HTML data contain not only text-related tags but also other tags that involve information in different media or its URLs such as `<img>`, `<iframe>`, and `<script>`.  Are all the tags appearing in raw HTML data left as it is when creating the datasets? Alternatively, is there any filtering applied to the datasets to keep the text-related tags during the pre-processing?\n\nB. Are the reported results the average over multiple runs with multiple seeds? If so, it is recommended to report the number of seeds as well as the standard deviation over multiple runs for each model to indicate the stability of the proposed method given the datasets for the proposed tasks.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful feedback!\n\n**Relevance to Web Information extraction.**\n\nThank you for bringing these relevant works to our attention! We will cite them in our work as follows:\n\n“WebSRC (JI et al) introduced a QA task from web documents. The authors manually curated a small set of seed questions (460 questions) which are used to collect paraphrases and do data augmentation. Similar to DescriptionGeneration, the problem is to find the relevant text in a web document. While DescriptionGeneration is focused more on elements and fully automated, WebSRC is a QA task that requires manual annotation.”\n\n“WIERT (Li et al) studies information extraction from web documents. They focus on utilizing simplified render trees (only text, tag, and style information are kept) to classify DOM nodes into product categories. They traverse the render tree to generate a sequence of element tokens and encode the sequence with a pretrained language model; style nodes are independently encoded and concatenated to the element encodings. The resulting encodings are used in a multi-objective setup to train the model. In contrast, our models are trained in a unified setup with a single objective, and using original HTML documents with no special filtering required to extract informative elements.”\n\n&nbsp;\n\n**Details of Pre-processing.**\n\nWe didn’t apply any special filtering to keep only text related information. We did minimal pre-processing to filter some of the attributes from excessively long inputs. For semantic classification, we applied no filtering on extracted snippets. For description generation, we filtered “class” and “style” attributes as we found them to increase the length of the HTML documents significantly. In MiniWoB, we applied no filtering and used the original observation space provided by the environment. We will also clarify these details in our final manuscript.\n\n&nbsp;\n\n**Multiple Runs.**\n\nWe trained each model once following the standard practice in the large language model community. But, we evaluated the WebN-T5-3B model on MiniWoB using multiple runs. We found standard error to be 1.2% showing some robustness in simulated web navigation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is simple and strongly effective.\n2. The extensive experiments and analyses support the effectiveness of the proposed method.\n3. Both the model weights for agents used in the WoB environment and the dataset created for the description generation task are published.", "reasons_to_reject": "1. The paper lacks a discussion of the connection between HTML understanding addressed in this paper and the field of web information extraction, which seems relevant and similar methodologies have been proposed in that field. See the missing reference section below.", "questions_to_authors": "A. Can be a more detailed explanation of the dataset creation (Section 4) or the pre-processing (Section 5) provided? In general, raw HTML data contain not only text-related tags but also other tags that involve information in different media or its URLs such as `<img>`, `<iframe>`, and `<script>`.  Are all the tags appearing in raw HTML data left as it is when creating the datasets? Alternatively, is there any filtering applied to the datasets to keep the text-related tags during the pre-processing?\n\nB. Are the reported results the average over multiple runs with multiple seeds? If so, it is recommended to report the number of seeds as well as the standard deviation over multiple runs for each model to indicate the stability of the proposed method given the datasets for the proposed tasks.\n", "paper_topic_and_main_contributions": "This paper aims for HTML understanding. To this end, the paper proposes a method that fine-tunes pre-trained LLMs on raw HTML texts, where the pre-trained LLMs are trained on standard natural language corpora. To evaluate the proposed method, the paper also proposes three fine-tuning tasks, in which the raw HTML sequences are considered as inputs to pre-trained LLMs, and one out of three tasks is newly proposed in this paper while the others are taken from prior work. Experimental results demonstrate the effectiveness of the proposed fine-tuning method on the proposed tasks with three model architectures (i.e., encoder-only, decoder-only, and encoder-decoder models) and different model sizes (ranging from 24.6M to 62B), outperforming both the baselines that are trained only on HTML data from scratch and the existing methods (Humphreys et al., 2022).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The following papers from web information extraction task seem relevant and should be appropriately cited in the paper.\n- Chen, Xingyu, et al. \"WebSRC: A Dataset for Web-Based Structural Reading Comprehension.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n- Li, Zimeng, et al. \"WIERT: Web Information Extraction via Render Tree.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.\n    - This work is considered contemporaneous work and it is not required to make a detailed comparison. However, it would be recommended to mention this work since the proposed method in this work can be a strong baseline to the proposed method in this paper. ", "Typos_Grammar_Style_and_Presentation_Improvements": "1. In Lines 119 and 162, there is a redundant parenesis\n2. In Line 486, Table-2 -> Table 2\n3. In Line 506, Table 3a -> Table a in Figure 3, alternatively, split the figure and table. Since a table and figure are combined and its caption starts with \"Figure 3\", it is quite confusing.\n4. In Line 550, Table 2b -> Table b in Figure 2\n5. In Line 604, Table 8 -> Table b in Figure 2?\n"}}, {"review_id": "5RGPnhUkDY", "reviewer": "Reviewer_LsyH", "summary": "", "questions": "-\tWhy the choices of model vary from different tasks? For example, for Semantic Classification task, we have large models like PaLM-62B. But on other two tasks, the largest model is T5-3B, showing a great gap in the scope.\n-\tHow is the performance on SOTA instruction-tuned LLMs such as OpenAI’s ChatGPT or Meta AI’s LLaMA 2?\n-\tRecent efforts on integrating real-world APIs to LLMs such as ToolLLM [1] and ToolAlpaca [2] has shown remarkable performance. Can this model adequately address the three tasks mentioned in the paper?\n-\tThe motivation of the dataset for Description Generation. The ultimate goal is to enable users to control the websites using natural languages, which is represented in MiniWoB to some extends. My first question is: can the dataset aid in the most difficult Autonomous Web Navigation task? Secondly, results show that by finetuning with the data, T5-large model has already reached a BLEU score of 90%.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback!\n\n**Bigger Models for the MiniWoB and Description Generation.**\n\nWhile larger models would give better performance (results on MiniWoB show scaling with model size), their scale would necessitate server-side rendering and actuation of the web pages. We have followed a privacy-oriented on-device approach, extracting the most we can from models that are deployable on current devices. Aside from meeting and beating the SoTA web navigational benchmarks, our aim was also to show practically this is possible and open source our model weights.\n\n&nbsp;\n\n**Performance of the Most Recent Models.**\n\nIn our work, we mainly focused on fine-tuning language models up to 62B parameters (compared to Llama-2 with 70B and Llama with 65B parameters) with limited supervised data (192x lesser data than CCNet). While we believe newer developments in LLMs such as instruction tuning, RLHF, or retrieval-augmentation are interesting future research directions to study HTML understanding problems, it is also important to understand the capabilities of models that are more practical to deploy on user devices without any privacy concerns; ours is one of the first in utilizing LLMs for web navigation. Yet, we also studied one-shot prompting of a 540B PaLM model to compare with fine-tuning and understand practical limitations (see section A.2.1 SEMANTIC CLASSIFICATION / Few-Shot Prompting and Table 5). Our results indicate that naively using one-shot prompting results in 45.6% accuracy on the test split. The main problem is the limited context length of the model (2K for PaLM) where prompts with even one-example-per-class are longer than their context length. We propose several improvements such as Example Cleaning, Category Rewriting, and Dictionary Mapping improving the accuracy to 64.2% which is still far from our fine-tuning results. Note that context length is a general limitation of many models including the most recent ones such as Llama-2 (4K) or Llama (2K). We believe context utilization for HTML understanding with a larger number of classes is an interesting future work.\n\n&nbsp;\n\n**Tool-Use with APIs.**\n\nThank you for bringing these relevant works to our attention! We will cite them in our work as follows:\n\n“More recently, tool-using agents emerged as a way to augment LLMs’ capabilities with external tools. ToolAlpaca (Tang et al) and TOOLLLM (Qin et al) are two examples that study multi-step tool-use with iterated API calls. Actions as well-defined API interfaces can be useful to simplify and reduce the dimensionality of the interface between the model and environment. This in turn enables a degree of horizontal scaling and generalization as studied by these works. However, this flexibility is only achieved when the environment implements an API interface. Where there are none available, navigating web interfaces becomes crucial to unlock access to further information.\n”\n\n&nbsp;\n\n**Motivation behind Description Generation dataset.**\n\nOur ultimate goal is to develop autonomous web navigation capabilities that require solving the “perception” problem – understanding HTML documents – (1) to classify elements into their respective abstract categories (such as username or password) and (2) generate textual inputs to interact with these elements (such as type(username, “myusername”)). While the MiniWoB simulator allows us to test these problems in a limited setting, we believe it is important to also understand the gap between real-world and simulation. We introduced the description generation dataset to test a model’s capability to solve perception by generating relevant text from real HTML elements. For example, a text field can have a label “Username:” which is important to generate in a navigational action to correctly enter the real username. Our results indicate that WebD-T5-3B can achieve only 84% accuracy which we believe is important to improve further.\n\nWe generated the DescriptionGeneration dataset from CommonCrawl by using only the available data in HTML documents. Our methodology allows generating unlimited amounts of data with no human labeling. We open sourced our dataset and methodology to help the research community to study more realistic datasets in larger scales.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe paper is well-written, with detailed and clear illustration of its motivation, the proposed framework, and the results.\n-\tThe paper focuses on a promising topic. The research direction of making LLMs understand and master how to use real-world tools is of significant importance. The algorithm can be deployed to many applications to help users with disabilities, such as visually impairment.\n-\tThe paper proposes a method to extract HTML snippets, which has the potential to develop models with capability of handling full raw HTML sequences.\n", "reasons_to_reject": "-\tThe models involved in the training of MiniWoB are relatively smaller, compared to, the task of Semantic Classification. The results on MiniWoB show that the performance is lower than human. Readers may be interested in whether using larger models can surpass human performance.\n-\tThe improvement seems trivial by fine-tuning the models on the datasets.\n-\tThe motivation of proposing the dataset for Description Generation task is unclear.\n", "questions_to_authors": "-\tWhy the choices of model vary from different tasks? For example, for Semantic Classification task, we have large models like PaLM-62B. But on other two tasks, the largest model is T5-3B, showing a great gap in the scope.\n-\tHow is the performance on SOTA instruction-tuned LLMs such as OpenAI’s ChatGPT or Meta AI’s LLaMA 2?\n-\tRecent efforts on integrating real-world APIs to LLMs such as ToolLLM [1] and ToolAlpaca [2] has shown remarkable performance. Can this model adequately address the three tasks mentioned in the paper?\n-\tThe motivation of the dataset for Description Generation. The ultimate goal is to enable users to control the websites using natural languages, which is represented in MiniWoB to some extends. My first question is: can the dataset aid in the most difficult Autonomous Web Navigation task? Secondly, results show that by finetuning with the data, T5-large model has already reached a BLEU score of 90%.\n", "paper_topic_and_main_contributions": "The paper proposes to finetune HTML related datasets with raw HTML sequences on pretrained LLMs to address the limitations of previous work. Along with existing datasets including MiniWoB and Annotated Shopping Webpages, the paper further proposes a dataset processed from CC. The paper investigates models with different architectures and model sizes, providing novel findings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs, Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, Maosong Sun\n\n[2] ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases, Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Le Sun\n", "Typos_Grammar_Style_and_Presentation_Improvements": "-\tLine 119, page 2: Is there an extra pair of brackets in “((Lu et al., 2021; Ahn et al., 2022))”?\n-\tThe Fig. 4 is mentioned in page 6 but shown in page 8.\n-\tThe order of tasks introduced in Section 3, Section 4 and Table 1 could change to match what is shown before (in Fig. 1) because the order in Fig. 1 relates to the difficulty of the tasks (with an increased difficulty).\n-\tIt will be better if a summary of contributions can be added if page limit allows.\n"}}]

Author rebuttal:

Meta-review: The paper proposes to fine-tune pre-trained language models on raw HTML texts in order to improve their abilities to "navigate" webpages in raw HTML. To test this, the authors release a benchmark of 3 related HTML understanding tasks: two from previous work and a novel one.

Reviewers agree that the new benchmark is an important contribution, and that the proposed idea is simple and strongly effective, which is showcased with extensive experiments and analyses. One reviewer highlights that the model could potentially "be deployed to many applications to help users with disabilities, such as visually impairments".

The main concern shared by reviewers is that experiments did not include large language models, which have been shown to have great performances in several tasks. The authors argue that they follow a "privacy-oriented on-device approach" and they aim to deploy these models in personal devices where large models may not be able suitable. I believe this justification is valid, but perhaps needs to be strengthen in the paper. There are also some references missing from other related areas that the authors have promised to include and have already proposed how they will be described in the paper.
Decision: Accept-Findings