Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Authors: Hiroki Furuta
Kuang-Huei Lee
Ofir Nachum
Yutaka Matsuo
Aleksandra Faust
Shixiang Shane Gu
Izzeddin Gur
Venue: ICLR 2024 poster (2024)
Keywords: Web Navigation
Foundation Models
Large Language Models
Instruction Finetuning
Decision Making
Multimodal Document Understanding
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose an offline multimodal agent for autonomous web navigation based on instruction-finetuned large language models, that achieves comparable performance to humans and RL-finetuned SoTA agents.

Abstract: The progress of autonomous web navigation has been hindered by the dependence on billions of exploratory interactions via online reinforcement learning, and domain-specific model designs that make it difficult to leverage generalization from rich out-of-domain data.
In this work, we study data-driven offline training for web agents with vision-language foundation models.
We propose an instruction-following multimodal agent, WebGUM, that observes both webpage screenshots and HTML pages and outputs web navigation actions, such as click and type.
WebGUM is trained by jointly finetuning an instruction-finetuned language model and a vision encoder with temporal and local perception on a large corpus of demonstrations.
We empirically demonstrate this recipe improves the agent's ability of grounded multimodal perception, HTML comprehension, and multi-step reasoning, outperforming prior works by a significant margin. 
On the MiniWoB, we improve over the previous best offline methods by more than 45.8%, even outperforming online-finetuned SoTA, humans, and GPT-4-based agent. 
On the WebShop benchmark, our 3-billion-parameter model achieves superior performance to the existing SoTA, PaLM-540B.
Furthermore, WebGUM exhibits strong positive transfer to the real-world planning tasks on the Mind2Web.
We also collect 347K high-quality demonstrations using our trained models, 38 times larger than prior work, and make them available to promote future research in this direction.

Published as a conference paper at ICLR 2024
MULTIMODAL WEB NAVIGATION WITH INSTRUCTION- FINETUNED FOUNDATION MODELS
Hiroki Furuta 1. Kuang-Huei Lee² Ofir Nachum? Yutaka Matsuo'
Aleksandra Faust Shixiang Shane Gu Izzeddin Gur
'Te University of Tokyo Google DeepMind furuta@weblab. t.. u-uoooyo...............o.tk..... ac. jP
ABSTRACT
The progress of autonomous web navigation has been hindered by the depen- dence on billions of exploratory interactions via online reinforcement learning, and domain-specine tion from rich out-of-domain data. In this work, we study data-driven offline training for web agents with vision-language foundation models. We propose instruction-followe following multimodal agent, WebGUM, that observes both webpage screenshots and HTML pages and outputs web navigation actions, such click and type. WebGUM is trained by jointly finetuning an instruction-finetune guage model and a vision encoder with temporal and local perception large corpus of demonstrations. We empirically demonstrate this recipe improves the agent's ability of grounded multimodal perception, HTML comprehension, and multi-step reasoning, outperforming prior works by a significant margin. On the MiniWoB, we improve over the previous best offline methods by more than 45.8%, even outperforming the WebShop benchmark, our Abillion-paaametee model achieves superior perfor- mance to the existing SoTA, PaLM-540B Furthermore, WebGUM exhibits strong positive transfer to the rorldworld planning tasks on the Mind2Web. We also collect 347K high-quality demonstrations using trained models, 38 times larger than prior work, and make them available to promote future research in this direction.
model designs that make it difficult to leverage generaliza-
lan-
finetune SoTA, humans, and GPT-4-based agent. On
INTRODUCTION
Web navigation is class of sequential decision making problems where agents interact with web interfaces following user instructions (Shi et al., 2017; Liu et al., 2018; Gur et al.,, Common web navigation tasks include, for example, form filling (Diaz et al., 2013), information retrieval (Nogueira & Cho, 2016; Adolphs et al., 2022), or sending emails via a sequence of interactions with computer interface such click rype (Figure 1). Recently, there has been growing interest in developing agents to automate these actions and free humans from repetitive interactions (Mazumder & Riva, 2020; Li al., 2020; Shvo et al., 2021).
Most prior works studied web navigation problems as online RL to learn the optimal action distribution with task-specific models from scratch (Liu al., 2018; Gur et al., 2019; Jia et al., 2019; Humphreys et al., 2022). However, online RL requires massive trials-and-errors and is often infeasible in practice since the failure in web navigation would result in undesirable consequences; for instance, wrong password may lead to account freeze, and sending email to the wrong person could be problematic in business scene. In contrast, offline training from the static dataset enables safe development of web agents, but the performance has been sub-optimal compared to those online RL counterparts (Humphreys et al., 2022: Gur et al., 2022). Furthermore, many of the prior works was unable to leverage rich out-of-domain data for generalization, models to explicitly handle the hicrarchical structures of document object model (DOM) and their dependencies, for example, with LSTM (Gur al., 2019; 2021), self-attention (Liu et al., 2018), or GNN (Jia et al., 2019). And many of them only output fixed set of categorical actions (Humphreys et al., 2022), which is unfavorable for truly open-ended web navigation in the real world.
they usually use specialized
Work done Student Rescarcher Google.

---

Published as a conference paper at ICLR 2024
Instruction: Find Gisele's email and forward to Slana, please.
Step1
Step
Step 3
Step 4
Hed in farm
4BT Sans.
Phimary
Grack
Stanel
Tranque
Success
Elan unin acat. Ciron
mogna velli. Ettion
velit. Et
sprt anin locas,- It
agat ente Lacon,,
Evans
12
Factoot:: idiecd lrp
(actort bilcd hof '32
(ldtint) voikk hoft 50;
Figure I: Example episode on MiniWoB++ (Shi et al., 2017; Liu ct al., 2018) The agent clicks the email from the proper sender, and types the correct receiver to forward that email, to satisfy the given instruction (e.g. Find Gisele's email and forward to Siana, please). WcbGUM makes use of both HTML and image screenshot information to adapt pre-trained instruction-finttie foundation model to solve challenging web-based tasks.
Recently, foundation models (Bommasani et al., 2021), especially large language models (LLM) (Brown et al., 2020; Chowdhery et al., 2022), have demonstrated superior performance in commonsense, symbolic, arithmetic, and multi-step logical reasoning (Wei et al., 2022h;c; Kojima et al., 2022). These models enable transformative generalization and are capable of solving wide ranges of interactive decision making problems in the wild, including but not limited task planning in robotics (Huang et al., 2022a;b; Shah et al., 2022; Ahn et al., 2022), board game (Meta Fundamental Al Research Diplomacy Team et al., 2022), web-based retrieval and browser crawling (Nakano et al., 2021; Yao et al., 2022b; Zaheer et al., 2022).
In this work, we leverage pre-trained vision and language foundation models and introduce a competitive offline learning recipe for autonomous web agents: First, we hypothesize that grounded spatial understanding is important for web navigation (Humphreys et al., 2022; Toyama et al., 2021) and thus enables our agent to observe both HTML and screenshots by combining language model and ViT (Dosovitskiy et al., 2020), from semantically rich multimodal tokens that perceive local and temporal information. Second, we observe that web navigation tasks are by nature instruction- following and thus base the language model on an instruction-tuned LLM (Wei et al., 2022a; Chung et al., 2022; Ouyang et al., 2022; Iyer al., 2022) instead of self- supervssevv pre-trained LLMS (Raffel et al., 2020; Brown et al., 2020) as in Gur et al. (2022). Third, we collect a large multimodal corpus. with both HTML and screenshots, to finetune the language model and ViT jointly. Fourth, our model outputs action in free-form text. These four key pieces together give us a multimodal web agent, which we call Web navigation via Grounded Understanding Models or WebGUM in short. As shown in Figure 1, our model takes in a command for a web- bab-based task via a natural language instruction (e.g., in an email client, Find Gisele's email and forward it to Siana, please.) and uses multimodal observations of the computer interface to complete the task via sequence of computer actions.
On MiniWoB++ (Shi et al., 2017; Liu et al., 2018), a simulated web navigation environment bench- mark. WebGUM outperforms previous best offline approaches trained with HTML inputs (Gur et al., 2022) by 45.8%, and even the best existing online RL approaches (Humphreys et al., 2022), despite being trained fully offline with much fewer experiences. WebGUM also shows better performance than humans and private-LLM, Looe--LLMbbsseeee based agents (Kim et al., 2023; Sun et al., 2023). We perform extensive ablations and analysis in Section 5 to demonstrate WebGUM's advantages in (1) temporal and local multimodal perception, (2) dataset and model size scaling, (3) better HTML understanding, and (4) ability of multi-step reasoning. WebGUM grounds vision and HTML understanding on the computer interface, which is critical for solving multi-step tasks with dynamic page transitions or tasks that require visual contexts, such as booking flights (+50%), shape recognition (+22%), or crawling social media (+21%) Using instruction- -iinttnndd language models (Chung et al., 2022), compared to using vanilla models (Raffel et al., 2020), improves the success rate on Mini MiniWoB++ by 25%, and is especially adept at handling the unknown composition of the tasks or outoof-distribution HTML inputs synthesized with realistic perturbations. On the WebShop benchmark (Yao et al., 2022a), we demonstrate that the capability of multi-step reasoning (Wei et al., 2022c) in language models enables better performance than existing state-of-thee art few-shot PaLM-540B (Yao et al., 2022b; Chowdhery et al., 2022), while our model only has 3 billion parameters. WebGUM exhibits strong positive transfer to the real-world action prediction tasks the Mind2Web while surpassing
2

---

Published as a conference paper at ICLR 2024
T5 Encoder Tansformer
T5 Decoder Tansformer
Temporal Tokens
HTML Tokens
Prediction
Action ('action' click, 'ref:: '61
Local Tokens
MiniWoB++
ViT
Tokenization & Embedding
Oedion lysa rwftt hout Dir Lue de enter cree then dhite wout. reteriinggg getwar nal- (ird lara" raaerhiv sak
Next Step
prisarch setrttteaatee www.eeeeeenneeeeeeeeeeeeeeo
1222
Image Observations Action history, Instruction, HTML
Figure 2: Overview of WebGUM, our multimodal encoder-decoder model. It takes screenshots, action history. instruction, and HTML as inputs. Image observations are embedded to tokens via pre-trained vision transformer (ViT) (Dosovitskiy al., 2020). Visual tokens contain rich temporal information from recent H H-step (H 2) and local information from 16 16-size patches. Multimodal language-imal -inuu tokens are fed into pre-trained TS encoder-dccoder models (Raffel et al., 2020), and are jointly trained to predict executable actions in text formats.
GPT-4. Finally, we collect 347K multimodal expert demonstrations on Mini WoB++, 38 times larger than the existing unimodal dataset (Liu et al., 2018), and make these publicly available for future research !. We believe that incorporating foundation models for efficient offline training is a scalable approach towards real- world web automation where online interactions are prohibitively costly.
RELATED WORK
Web Navigation Among many proposed benchmarks for autonomous web navigation (Toyama et al., 2021; Burns et al., 2022; Yao et al., 2022a), one of the most inclusive and representative benchmark to test the capability of autonomous agents is MiniWoB++ (Shi et al., 2017; Liu et al., 2018), which consists of set of simulated websites with various user instructions from primitive tasks to complex multi-step decision making tasks, such as sending emails or booking flights. Prior works have tried solve this benchmark using variety of techniques; Liu et al. (2018) and Gur et al. (2019; 2021) leverage the guidance during online RI. from high-level workflow (Liu et al., 2018) or curriculum learning (Gur al., 2019; 2021), which should be, however, designed per task, and then would not be scalable methods. Other approaches have employed supervised learning (SL) with large million-scale dataset and following RL-finetuning (Humphreys et al., 2022), or SL with LLM-baasd agents (Gur et al., 2022). Offline SL agents often suffer from sub-optimal behavior, and online RL with tremendous exploratory experiences has been critical for proficient navigation on the web (Humphreys et al., 2022), which is, however, difficult to conduct in real websites as there is typically no reward signal and interactions are prohibitively costly. shown in Appendix I, many of these approaches depend on task-specific hicrarchical structures of DOM (Jia et al., 2019; He et al., 2020), tailored architectures to encode their dependencies such as LSTM (Gur et al., 2019; 2021), self-attention (Liu al., 2018), GNN (Jia et al., 2019), and task-dependent categorical output space (Humphreys et al., 2022), which could not handle open-ended multi-task settings similar to real world, or incorporate pre-trained models. In contrast, we remove such web-specific architectures and convert web navigation into visual question-answering format (text, image - text), which allows us leverage pre-trained foundation models (Chung al., 2022; Dosovitskiy al., 2020) as rich prior knowledge on the web, and then to learn the capable agents even with offline training.
Large Language Models for Web Navigation Concurrently, private-LLM--aasee agents, such as InstrucIGPT (text-davinci-003) (Ouyang et al., 2022) and GPT-3.5-turbo, have achieved competitive performance to RL-fintuned models and humans by leveraging handful of few-shot demonstra- tions with self-imprrvemmnt (Kim et al., 2023), code generation (Sun et al., 2023), and structured prompts (Zheng et al., 2023). In contrast, WebGUM focuses on multimodality and finetuning with domain-specific data. With those, we show very competitive performance compared to PaLM-540B with only billion parameters. WebGUM can also handle long HTML observation tasks, such as book- fl i ght or choose - where agents that rely in-context few-shot learning tend to run out of input tokens. In addition, our models do not requires ad-hoc prompt engineering
'https://conso" wetpmnnssee....u.e.eoommm cloud.goog www.ccc//tttagebbrooooo wwwpwr/wwwwwwwewwweeeeee.e.ee webb 1
3

---

Published as a conference paper at ICLR 2024
Methods
Modality
Pre-truined Models
Offiine
Dataset
Success Rafe
CC-Net (SL)
DOM+Inage
ResNet
2400K
32.0%
WebN-TS
HTML
TS-XL
12K
48.4%
HTML +Image
Han-TTS ----ee--Base, Base,Vil-B16 ViT-B16
2.8K
61.1%
WebGUM (Ouns)
HIML
Flan-TS-XL
401K
88.7%
HIML+Image
Flan- Men-TS-AE ViXXI,
401K
94.2%
WGE
DOM
12K+
64.6%
CC-Net (SL+RL)
DOM+l
ResNet
2400K
93.5%
Human
93.5%
RCI
HTML
GPT-3.5-turbo
ICL
~0.1K
90.6%
AdaPlanner
HIML
text-davinci-003
~0.1K
92.9%
RCI
HTML
GPT-4
ICL
-.1.K
94.0% 04.0%
Synapse
HTML
GPI-3.5-turbo
ICL
~0.1K
98.5%
Table 1: Average success rate on MiniWoB+++. We refer to Zheng et al. (2023) for the baseline performances. See Appendix G for the detailed scores. WebGUM outperforms the previous finetuned-LLM with 3B parameters (Gur al., 2022), which the best among offiine methods, even with 2.8K dataset and Basc-size model (310M pa- rameters). Scaling dataset and model size, WebGUM beats the online RL-finetuned state-of-the-and (Humphreys et al., 2022) despite fully offline training, and exceeds humans or LLM-based agents with GPT-4 (Kim et al., 2023). "+" in Dataset column means extra billions of frames are required during the online RL phase.
In Appendix B, We discuss additional related works on multimodal large-scale models and foundation models for decision making.
PRELIMINARIES
We formulate autonomous web navigation as a deterministic sequential decision making problem; composed of a state space S, action space AA, deterministic transition function T : S x A + S, instruction space G, reward function (or episodic success criteria) r S x G each time step I, the agent follows parameterized policy conditioned on previous states and actions x.. A x + A, and transits to the next state: 8141 = T(s at). This
- {0. 13. At
π : S x x
Xt process continues until the agent reaches the terminal state (e.g. Submit button is clicked) or the max time step is exceeded. episode is treated as success if given instruction g is satisfied (i.e. r88, 9, 4x) 1), and as failure if the agent takes invalid action or reaches wrong terminal state.
In autonomous web navigation, the state 8 E S a web page consisting of the raw HTMI as a text sequence and screenshot an image. Following prior works (Shi et al., 2017; Liu et al., 2018; Gur et al., 2019; 2021), we assume the constraint action space: funct i (selector, text). function is either click type, selector is integer index that can uniquely specify the element, and text is a text input for type function.
Figure/ 1 presents an example episode of MiniWoB (Shi al., 2017), which involves multi-step decision making. To meet the given instruction, the agent clicks an email from the proper sender and types the correct receiver to forward that email. MiniWoB also has primitive behavioral tasks such as clicking buttons or entering texts. For the examples of WebShop (Yao et al., 2022a), see Appendix
WEBGUM
4.1 MULTIMODAL TRANSFORMER MODELS WITH TEMPORAL AND LOCAL PERCEPTION
In this work, we follow Gur et al. (2022) to use T5 (Raffel et al., 2020), an encoder-decoder architecture, for HTML-based web navigation, as its bi-directional nature could be good fit for the tree structure of HTML and the architecture has been shown to scale well. We combine T5 with vision transformer (ViT) (Dosovitskiy et al., 2020) for multimodality as illustrated in Figure 2. Specifically, we use the ViT to map image observations (screenshots) into image tokens. The ViT pre-trained ImageNet-21K classification (Deng et al., 2009). The T5 encoder then consumes both visual and HTML tokens in a unified manner, and the decoder predicts actions in text. See Appendix C for more implementation details.
4

---

Published as a conference paper at ICLR 2024
Performance Improvement (86)
80
60
Methods
Modality
Success Rate
WebGUM
HTML
88.7%
WebGUM (white)
HTML+Image
90.9%
WebGUM (random)
HTML+Image
92.2%
WebGUM
HIae+Image
94.2%
cit to acadt
Search enginna
Chocotte thtt mom
Quand mimbberr
Adteniyy shapps
Suctial moddiaa a28
clict phages
che Poo-tte--- lost
spinnnr
Niight
Figure 3: (Left) Average success rate with white/random image inputs. The results imply that WcbGUM successfully leverages multimodal information from temporal and local perception tokens. (Right) Top- 10 performance improvement among MiniWoB++ by adding image modality to HTML. We subtract the success rates to compute absolute improvement: (SR of WebGUM (HTML+ Image)
(SR of WebGUM (HTML) ) Image modality is leveraged for multi-step tasks with dynamic page transitions or tasks that require visual concept understanding (e.g. book-flight or click-shape). See Appendix G and L for the details.
Encoding Temporal and Local Visual Tokens For language models to be aware of task temporal information and local scene recognition, the encoder considers multimodal tokens extracted from history of patched screenshots (H 2 steps). Temporal visual tokens contribute to predict the consistent actions in a multi-step tasks. To better extract spatial and semantic information across the local parts of websites, our ViT encodes one local token per patch rather than global one per image (i.e. CLS-token). We divide an input image into 16 x 16 patches - giving total of 14 14 (number of patches)
(temporal window) 392 visual tokens. We crop the screenshots of Wf++WoB++ to remove the yellow instruction part, and the image size becomes 160 x 160. We pad cropped images with white pixels to fit them into 224 224; the default input size for VIT.
4.2 INSTRUCTION FINETUNNED LARGE LANGUAGE MODELS
We base our language model on Flan-T5 (Chung al., 2022), an instruction -inetuned T5, as opposed to using vanilla pre-trained T5 as in Gur et al. (2022). Flan-T5 is finetuned with large- scale inttructionfollowiiie domains, including reasoning or programming. Considering that web navigation is inherently an instruction-followin task, we hypothesize that carefully trained cnerrcttionfinetunee models could generalize well to enhance the alignment with user instruction and zero-shot reasoning in the web- navigation, interactive decision making context. For the same reason, we also hypothesize that these high nigh-eerforming instruction -inetuundd models enable better sample efficiency and downstream perfor- mance, and thus are well-suited for offline learning. We further finetune the Flan-T5 language model and the VIT vision encoder jointly (Figure 2) on large corpus of cnstruction-oollwwiin multimodal web navigation data, which we describe in Section 4.3. In Section 5, we empirically demonstrate that recipe improves HTML comprehension, multi-step reasoning and decision making significantly.
format problems and chain-of-thought examples across variety of
this
4.3 LARGE-SCALE DATA COLLECTION WITH LANGUAGE MODEL AGENTS
Recent successes of foundation models are largely powered by internet-scale data (Brown et al., 2020; Radford et al., 2021; Chen al., 2022; Wang et al., 2023). While large amount of data is critical, for web navigation domain, there is only a small public dataset for MiniWoB++, consisting of 12K episodes of human demonstration (Liu et al., 2018). Moreover, the dataset only consists of DOM observations and lacks any visual features, which might limit the fine spatial perception of the elements on the page. large-scale multimodal dataset, including screenshots of websites, required to build better navigation policy at scale.
To collect a huge amount of multimodal behavioral dataset on MiniWoB++, we leverage the finetuned- LLM policy from Gur et al. (2022), instead of human demonstrators (Liu et al., 2018; Humphreys et al., 2022). This significantly reduces the cost to construct a new dataset by leveraging the prior success of autonomous agents. We first rollout LLM policy with 100 episodes per task, which results in a 2.8K successful episodes. Then, we finetune F models with this small dataset and those with 10,000 episodes per task. Lastly, collect additional 54K demonstrations
5

---

Published as a conference paper at ICLR 2024
HTML Image
+Visual Token Ablation
+SSL VIT
HTML
HTML (Decoder-only)
05.3 06-3
Average Success
66.0
65.8
64
Tlan-
63.5
65.5
66
60
60
6..5
64.2
58
52.
WM4 $5.6
56
30
WeoN
40.8
IN Demporaa anl
y 300M
3T:33
CuP
MAE oN)
DINO (9))
2.8K
OBK
347K
Base
Large
XL
XXL
(oocas)
Vhemg so meann
Rarg
Dataset Size
Model Size
Figure 4: Average success rate of WebGUM with visual perception tokens and ViT pre-training ablations (left), different dataset size (middle) and model architectures (right). In dataset and model size results, X-axis is a logarithmic scale. (left) While the effects of various pre- trained VIT with different datasets or self supervised objectives are marginal, employing both temporal and local perception tokens is critical for the performance. (middle & right) As for both HTML and multimodal models, we could observe the scaling effect: the larger the dataset and model size are, the higher the success rates are. The results also prove that decoder-only Flan-Pal M-8B is not as good as similar-size encoder-decnder models.
with Synapse (Zheng et al., 2023), a private-LLM- the finetuned-LLAn may not complete well. Such efforts result in a multi-task dataset with 401K K (347+54K) episodes including HTML and screenshots each step. See Appendix F for more details.
agents with prompting, for the tasks where
RESULTS
We test our method on MiniWoB++ (Shi et al., 2017; Liu et al., 2018) with 100 evaluation episodes per task, taking the average success rate over 56 tasks taken from Gur al. (2022). Table shows that WebGUM, with a small 2.8K dataset and Base-size model (310M parameters), significantly outperforms previous offline methods for web navigation (Humphreys al., 2022; Gur et al., 2022). While they used 2.4 million episodes or 3 billion parameters, WebGUM could improve the data and parameter efficiency to achieve superior performance in offline regime, which is realized by the problem simplification of web navigation in order to leverage temporal-loca visual perception and instruction -inetuned LLMs as strong inductive bias on web environments. In addition, scaling dataset and model size, WebGUM achieves 94.2% success rate?, exceeding the previous best offline model, WebN-T5 (Gur et al., 2022), by over 45.8% and even surpassing the online RL-finetuned SoTA, CC-Net (Humphreys et al., 2022) (+0.7%), despite our fully offline training and much fewer data. Moreover, WebGUM surpasses humans and recent LLM-based agents, such as RCI (Kim et al., 2023) and AdaPlanner (Sun al., 2023), even with GPT-4 (OpenAI, 2023). The per-task comparison and error analysis (Appendix G, L.) imply that there is room for improvement in complex reasoning tasks requiring memory such as guess nnuberr.
In the following sections, we perform extensive and precise ablations of WebGUM to clearly identify the source of improvement. Especially, we will demonstrate the contribution of (1) temporal and local multimodal perception (Section 5.1), architectures and pre-trained models, and (2) dataset and model size scaling (Section 5.2). We will also point out (3) better HTML comprehension (Section 5.3) and (4) capability of multi-step reasoning (Section 5.4) from cnttruction-finttuue I.MMs. Furthermore, we prove that WebGUM can transferable to the real-world tasks (Section 5.5).
5.1 TEMPORAL AND LOCAL VISUAL PERCEPTION FOR GROUNDED WEB NAVIGATION
To verify the importance of image modality, we design three ablations: (i) input replacement, (ii) removing visual perception tokens, and (iii) employing different pre-trained VIT. We first replace image observations with completely white images, and with randomly sampled MiniWoB++ screen- shots taken in the initial states at test time. For visual token and pre-trained ViT ablations, we prepare various pre-trained weights with ImageNet-21 (IN) AugReg (Steiner et al., 2022), JFT-300M (Sun al., 2017). or JFT-3B (Zhai et al., 2022), and with self supervised objectives such as CLIP (Radford et al., 2021), MAE (He et al., 2021), or DINO (Caron et al., 2021), 1), and then finetune Base-size models as a proxy of larger-size models (Hoffmann al., 2022) to reduce the computational costs.
Videos available at https: //te:s////eeee googl www.c///viiioo wwwp///w/wwww/////////m/m.

---

Published as a conference paper at ICLR 2024
cltkiikk
clic-button
Crdd "lay ada
Methods
Modality
Success Rate
Mare partis undence
playita Weltehs
WebN-T3 (Gur et al., 2022)
HIML
51.0%
Synapse (Zheng et al., 2023)
HTML
73.8%
eirattkes Faoent Ejenks
Osay
WabGUM WebGUM
HIML
74.2%
Noc Subritt
0%
HTML+Image
78.5%
srre: canvils
Figure 5: (Left) Example of compositional evaluation on MiniWoB++. We combine two different tasks (click-iine and click -buckbuuttto ton) into single sequential task sciick-ie tek-iiig time (see Appendix H). (Right) Average success rate on 6 compositional MiniWoB tasks. WebGUM generalizes combinational tasks better than Gur al. (2022) and Zheng al. (2023), SoTA LLM--agent in
bucccckbbutteee on) at test
A38 sotra HTM
Adit edta HTM, the
Add Courd
Methods
Modality
Perturbation
Success Rate
WebN-15
HTML
Top
24.7%
(Gur al., 2022)
Bottom
42.8%
Coordinates
6.4%
WobGUM
HTML
Tup
53.6%
Boltom
48.0%
Courdinates
39.8%
WelGUMM
HTMa-+
Top
71.8%
Bottom
64.7%
Coundinates
62.6%
Figure 6: (Left) Example of input perturbation for MiniWoB++ evaluation, taken from click-buttoon on. We prepare three different types of perturbations at test time: adding extra HTML at the top of the original input HTML (left) or at the bottom (middle), and adding task-irrelevan attributes in each element (right) such as coordinate information (left, right, top, bottom). (Right) Average success rate of perturbation evaluation on MiniWoB++. The results reveal that while all the methods are affected by input corruptions some extent, WebGUM, especially with multimodality, achieves significantly hetter performances than previous method.
In Figure 3 (left), the performance of the model with white images is comparable to the unimodal model. Presumably because the model with randomly-taken images may accidentally contain the images from the target task, WebGUM (random) slightly surpasses WebGUM (white). These results prove WebGUM successfully obtains grounded vision and HTML understanding by leveraging temporal and local fine perception. In the visual token ablation, Figure 4 (left) shows that combining both temporal and local visual tokens (66.1%) improves the performance than temporal (64.2%) or local tokens only (64.0%). Interestingly, the effects of different pre-trained ViT are marginal, compared to visual tokens, which highlights our contribution on designing suitable architecture for multimodal web navigation.
We also compare per-task performance gaps caused by adding vision modality to language models. Figure (right) presents top-10 absolute performance improvement. suggesting WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions (e.g. book-flight +50%) or tasks requiring visual context understanding (e.g. click-shape; +22%) (see Appendix G and L).
5.2 SCALING EFFECT IN DATASET AND MODEL SIZF
In this section, we show the importance of scaling up the dataset and model size in WebGUM, similar the observations in the language and vision domain (Shoeybi et al., 2019; Kaplan et al., 2020; Rae et al., 2021; Wei et al., 2022b; Chowdhery et al., 2022). To investigate data scaling, we prepare three dataset: minimal 2.8K demonstrations. 347K demonstrations, and its 20%-size demonstrations (68K), and then finetune Flan-T5: Man-TT-Ba Ten-----BBass Base with them. Figure 4 (middle) proves that increasing dataset size leads to the improvement of success rate. Because multimodal models benefit from the scaling more, the larger dataset size might be more crucial in multimodal models, which also supports our attempts to construct targe-scale multimodal dataset for web navigation. Notably, Base-size WebGUM with 2.8K episodes already achieves 55.7% %, surpassing previous best SL models 8%/55.6% we trained with 347K episodes). This surprising data efficiency comes from the sufficient inductive bias and alignment with the user intentions in cntrucctoo-finetunee LLMss.
In addition to dataset size, Figure 4 (right) shows that the performance of WebGUM improves as the number of parameters in T5 model increases from Base (220M) to XXL (1BB). These results also reveal that scaling the models might be more important than the dataset; the low-capyc model may cap the performance at lower level. In contrast, decoder-only Man--PaLM-88 only
7

---

Published as a conference paper at ICLR 2024
achieves 72.8% success, comparable to WebGUM-Large (770M), which emphasizes the advantage of encoder-decoder models in web navigation. See Appendix D for further details.
5.3 BETTER HTML COMPREHENSION FROM INSTRUCTION-FINETUNE LLMS
We have demonstrated that instruction-iinttus LLMs outperforms vanilla LLMs in web navigation. To analyze the effect of instruction eestruction-iieeuunnng more precisely, we here focus on the capability of HTML understanding. Since cnstruction-fnetunnn L.Mss perform well on many NLP tasks with content comprehension (Chung et al., 2022; lyer et al., 2022), web navigation should also benefit from them. As a test bed for HTML comprehension, we investigate (1) generalization to unseen compositions of known tasks, and (2) robustness to the realistic input perturbations, which are also important challenges for the web agents to be deployed on the real-world internet. We also provide the base language model comparison on a standard HTML comprehension benchmark, WebSRC (Chen et al., 2021d) in Appendix E, where Flan-T5 achieves better EM/FI scores than T5 after finetuning.
For the compositional tasks, we pick up 4 click- "something" (link, button, checkboxes, dialog) tasks and make 6 combinations of these by naively stitching with 2 or 3 tasks (e.g. Figure 5). See Appendix for further details. The results show that WebGUM with HTML and image inputs outperforms prior finetuned-LLM (Gur al., 2022) and Synapse (Zheng al., 2023), a SoTA LLM agent in MiniWoB++, which implies WebGUM has obtained better reading skills for web navigation and could transfer them to handle unseen HTML in compositional tasks robustly.
click-"something
To test the robustness against input corruptions, we test three different realistic perturbations; adding extra HTML at the top or bottom of the original HTML, and adding attributes of coordinates (left, right, top, bottom; they are unrelated to solving the tasks) in each element of HTML at test time. These perturbations often happen the real world due the renewal or API changes, not to mention unknown websites, but rule-based pre- pre-processin may not fully cover them. The results show that while all the methods are affected by the input corruptions to some extent, WebGUM, with both HTML and HTML plus image modalities, achieves significantly better performances than Gur et al. (2022). Notably, WebGUM outperforms prior finetuned LLM (+ 56.2% in multimodal and +33.4% in unimodal models) even when extra distracted attributes added to HTML. They support hypothesis: instructionfiinttumng finctuning imporves HTML comprehension in LLMs, which enables the downstream agents to deal with duitr--distributionn inputs tasks robustly.
5.4 ABILITY OF MULTI-STER REASONING AS A PRIOR FOR INTERACTIVE DECISION MAKING
Another notable feature in instruction- -inetuned LLMs is an ability of multi-step reasoning (Chung et al., 2022). We hypothesize this reasoning capability would play an important role prior for interactive decision mak- ing. To decouple the evaluation of reasoning capability from visual page perception, HTML understanding, and the benchmark simulator (MiniWoB++)) we extensively evaluate our WebGUM on WehShop (Yao et al., 2022a), another online-shopping website simulator with a large amount of real-world product data. Because it requires complex multi-step decisions considering previous con- texts for item comparison, WebShop is suitable for in- vestigating the capability of multi-step reasoning from finetuned LLM in depth (Yao al., 2022a;b). WebShop provides a user instruction that describes the features of item (e.g. I need a long clip-in hair extension which is natural looking, and price lower than 20.00 dollars). The agents should search, compare and choose a proper product that matches the given instruction. The performance score is evaluated by the percentage of required attributes covered by the chosen product, and the product meets all the requirements, that episode is labeled success. See Appendix K for further details.
Methods
Training
Models
Score
Sucvoss Rate
Role
45.6
9.6%
IL
BART. BERT
59.9
29.1%
Act
In-contest
PaLM-S40B
62.3
30.15
RoAa
In-comeet
Pal MALM-
66.6
40.0%
WohN-T5
SL
TS-XI.
61.0
29.8%
WebGUM
TSan-T--LL XL
67.5
45.0%
Table 2: Average score and success rate on Web- Shop (Yao et al.. 2022a). WebGUM achieves 45.0% success, outperforming baseline methods including ReAct, prompted Pal. AalM-550 M-540B. We refer Yao et al. (2022b) for the baselines.
Table 2 shows that WebGUM achieves 45.0% success, significantly outperforming not only simple baselines, such as supervised imitation learning (IL), IL plus RL-finetuing and WebN-T5 (by more than 15%), but also recent prompt- M-5.0 (Chowdhery al., 2022) with one-shot prompt and reasoning annotations), while
grompt-based based LLM agents, including ReAct (Yao et al., 2022b) (i.e.
8

---

Published as a conference paper at ICLR 2024
Cross Domain
Fle.
Op.
Step SR
SR
Ace
Op.
Stop SR
Op.Fi
Step
GPT-4
KCL.
41.6
00.6
36.2
2.0
35.8
51.1
30.1
2.0
57.1
46.5
26.4
2.0
MindAAs-Large
S1.
$3.4
75.7
50.3
7.1
50.2
67.1
35.3
1.1
39.7
62.2
17.3
2.7
MindArt-XX XL
SL
55.1
15.7
52.0
5.2
42.0
65.2
38,9
5.1
42.1
66.5
39.6
2.9
LcMGUM-Laage (curs)
SL.
55.3
78.9
51.9
7.5
43.6
70.3
19.3
51
42.8
70.6
40.2
2.9
WebGUM XL (ouns)
SL
57.2
80.3
53.7
85
45.3
70,9
41.6
52
43.9
72.2
41.
322
Table 3: Action prediction evaluation real-world Mind2Web dataset. We adopt the top-50 candidate generation results and direct QA formulation by following Deng et al. (2023). WehGUM, transferred from MiniWoB, demonstrates superior performance to MindAct and GPT-4 across task/wehhite/domain generalization.
model only has 3 billion parameters. Due to the consistent reasoning and enhanced alignment with user intentions, WebGUM could compare the products with backtracking, and choose proper options (see Appendix L.. Our results imply that ability of multi-step reasoning in Flan-T5 works as strong and transferable prior knowledge for downstream decision making.
5.5 STRONG TRANSFER TO REAL-WORRL ACTION PREDICTION
Lastly, we demonstrate the applicability of WebGUM to real-world problems. We test WebGUM on Mind2Web (Deng et al., 2023), real-world demonstration dataset with about 2K instructions 137 websites. In the action prediction tasks, we transfer WebGUM finetuned for MiniWoB++ with 401K dataset into real-world Mind2Web by further finetuning with the training set. WebGUM takes top-50 relevant HTML snippet candidates, instructions, and action history as inputs and outputs next actions by predicting the element id, operations (e.g. click, type), and values. Table 3 reveals that WebGUM, transferred from Mini Mini achieves superior performance to LindAct-Large/XL and even GPT-4 in all the categories (cross-task/went wusk/website/domainn Because both MindAct and WebGUM are based on Mla these results support that WebGUM exhibits strong positive transfer to real-world tasks.
6 DISCUSSION AND LIMITATION
Throughout the paper, we present an effective and practical methodology to simplify web navigation into offline training in order to leverage the inductive bias of web environments in istructionnfinetunne LLMs. While WebGUM exhibits positive transferability to real-world problems in Mind2Weht we leave it as future work to scale multimodal foundation models into the deployment for real-world web navigation (Gur et al., 2023).
We collect and release a multimodal expert dataset with 347K episodes MiniWoB++. However, this is still far from internet-scale dataset that is necessary for generalist models. Collecting behavioral data at scale by iterative data-collection and deployment (Ghosh al.. 2021: Matsushima al., 2021; Li et al., 2022a) might be key for practical interactive agents. Since our approach taking raw HTML and screenshots as inputs and predicting parsable actions text only has minimal assumptions which constraint model architectures, it might be applicable to any advanced LLMs or open-ended situations. While WebGUM could deal with out-of-distribution compositional and perturbed tasks in robust manner, human-level broader generalization to the diverse real websites or instructions is still a hard problem to be resolved.
CONCLUSION
We develop Web navigation via Grounded Understanding Models (WebGUM), learning an instruction- following visual language foundation model for web navigation. WebGUM significantly improves the success rate on MiniWoB, compared previous -rfinnetrained SoTA from 48.4% 94.2%. Our detailed ablations show that temporal and local visual tokens capture dynamic transition and language models significantly improves
learmingg
visual context of the page, and that instruction-finetue web navigation performance due to the better HTML comprehension and capability of multi-step reasoning. Multi-step reasoning enables more robust generalization to out-of-distribution tasks, and outperforms PaLM-540B in WebShop. WebGUM also demonstrates strong positive transfer to real-world action prediction tasks in Mind2Web. Furthermore, we scale the existing MiniWoB dataset into multimodal 347K expert demonstrations, about 38 times larger than before. We believe that our work is an significant step towards building more capable and scalable models for autonomous web navigation.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
HF was supported by JSPS KAKENHI Grant Number JP22J21582. We thank Yusuke Iwasawa, Mustafa Safdari, Austin Huang, Heiga Zen for helpful feedback on this work, and Shunyu Yao for setting WebShop experiments.
REFERENCES
Leonard Adolphs, Benjamin Boerschinger, Christian Buck, Michelle Chen Huebscher, Massimiliano Ciaramita, Lasse Espeholt, Thomas Hofmann, Yannic Kilcher, Sascha Rothe, Pier Giuseppe Sessa, and Lierni Sestorain Saralegui. Boosting search engines with interactive agents. In Trunsactions on Machine Learning Research, 2022.
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. Htlm: Hyper-text pre-training and prompting of language models. arXiv preprint arXiv:2107.06955, 2021.
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. Cm3: A causal masked multimodal model of the internet. arXiv preprint wixv:2201.005200 2022.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can, not say: Grounding language in robotic affordances. arXiv preprint wrxiv:220..01691, 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for fhw-shot learning. arXiv preprint arxiv 2022.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey. Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy. Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Itycheriahh, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem. Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy. Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang. Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report. arXiv preprint 2rXh: 2023.
10

---

Published as a conference paper at ICLR 2024
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. Docformer: End-to-end transformer for document understanding. In International Conference on Computer Vision, 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha. Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li. Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv:21088 108.02208..7725. 2021.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian lehter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Van- houcke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu. Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22/2.06811 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint 1415:20005.1415.. 2020.
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset for interactive vision- languaee navigation with unknown command feasibility. In European Conference on Computer Vision, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-super vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri,
11

---

Published as a conference paper at ICLR 2024
Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021b.
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Conference on Robot Learning, 2021c
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly- scaled multilingual language-iman -magg model. arXiv preprint arxiv:2209.06 06794, 2022.
Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4173-4185, 2021d.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Hellr-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023/ 023 / 2022.
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetuned language models. arXiv preprint arxiv:2210.1 1416, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong. Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- pprppps vision-language models with instruction tuning. arXiv preprint 2.3.:23335.063.. 2023.
DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim Harley, Felix Hill, Peter C Humphreys, Alden Hung, Jessica I Landon, Timothy Lillicrap, Hamza Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel Wong, Chen Yan, and Rui Zhu. Creating multimodal interactive agents with imitation and self-supervised learning. arXiv preprint arXiv:2 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint (tt6:2306066000 2023.
12

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1.Xiv:8810004805 2019.
Oscar Diaz, Itziar Otaduy, and Gorka Puente. User-driven automation of web form filling. In International Conference Web Engineering, 2013.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Linxi Fan, Guanzhi Wang, Yumfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang. De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv: 08853, 2022.
Hiroki Furuta, Yusuke Iwasawa, Yutaka Matsuo, and Shixiang Shane Gu. A system for morphology-task generalization via unified representation and behavior distillation. arXiv preprint arXiv:2211.14296 2022a.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. In International Conference on Learning Representations, 2022b.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021.
Shixiang Shane Gu, Manfred Diaz, Daniel C. Freeman, Hiroki Furuta, Seyed Kamyar Seyed Ghasemipour, Anton Raichuk, Byron David, Erik Frey, Erwin Coumans, and Olivier Bachem. Brax- lines: Fast and interactive toolkit for rl-driven behavior engineering beyond reward maximization. 2021a.
arXiv preprint
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arxiv:2 13921 2021b.
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani- -urr. Learning to navigate the web. In International Conference on Learning Representations, 2019.
Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning. In Advances in neural information processing systems, 2021.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models. arXiv preprint 2rxiv:2210.03900 2022.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arxiv:2307. 12006 2023.
12856,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint 1
2021.
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, Jindong Chen, and Blaise Agūera y Arcas. Actionbert: Leveraging user actions for semantic understanding of user interfaces. arXiv preprint arXiv:2012. 123350 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv: 15556, 2022.
13

---

Published as a conference paper at ICLR 2024
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.0720) 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian lchter. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arxiv:2207.05600) 2022b.
Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thomton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning control computers. In International Conference Machine Learning, 2022.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. Opt-iml: Scaling language model instruction meta learning through the lens of generalization, arXiv preprins arXiv:22 12001, 2022.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba. DOM-q-NET: Grounded RL on structured language. In International Conference on Learning Representations, 2019.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.0309. 2022.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:2104. 12763, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wrXiv:2001.08666 2020.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arxiv:2303.1749/ 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances In Neural Information Processing Systems, 2022.
Taku Kudo and John Richardson. Sentencepicce: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint /808. 2018.
Kenton Lee, Mandar Joshi, Iulia Ture, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khan- delwal, Peter Shaw, Ming-Wor Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. arXiv preprint arXiv:2210.033477 2022a.
Kuang-Huei Huei Lee, Ofir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarama, Ian Fischer, Eric Jang, Henryk Michalewski. and Igor Mordatch. Multi-game decision transformers. arXiv preprint arxiv:2205. /5241, 2022b.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-tor -ownncce-----uueeee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 2019.
1.Xivv19110 1910. 13461,
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang. and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03372, 2023.
14

---

Published as a conference paper at ICLR 2024
Chenliang Li, Bin Bi, Ming Yan, Wei Wang. Songfang Huang, Fei Huang, and Luo Si. Structurallm: Structural pre-training for form understanding. arXiv preprint arxi:2205.112210, 202
Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. Markuplm: Pre-training of text and markup language for visually-rich document understanding. arXiv preprini arxiv:21 2021b.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In Conference on Computer Vision and Pattern Recognition, 2021c
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained language models for interactive decision-making Advances In Neural Information Processing Systems, 2022a.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instruc- tions to mobile ui action sequences. In Annual Conference of the Association for Computational Linguistics, 2020.
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson dAutume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, 2022b.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement learning on web interfaces using workflow-guider exploration. In International Conference on Learning Representations, 2018.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844 08488, 2023.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.1179; 2022.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations, 2021.
Sahisnu Mazumder and Oriana Riva. Flin: A flexible natural language interface for web navigation. arXiv preprint arXiv:2010. 12844, 2020.
Meta Fundamental AI Research Diplomacy Team, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624): 1067-1074, 2022.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted prestion-answerieg with human feedback. arXiv preprint 022132211..09332. 2021.
Rodrigo Nogueira and Kyunghyun Cho. End-to-end goal-driven web navigation. In Advances In Neural Information Processing Systems, 2016.
OpenAI. Gpt-4 technical report. arXiv preprint arX0::2303.08774 2023.
15

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint 0rxii:2203.02/555 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprins arXiv:2/03.00020, 2021.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katic Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch. Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev. Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis &; insights from training gopher. arXiv preprint arXiv:2112.1144 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 1-67, 2020.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. arXiv preprint 2661::200..06155 2022.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy -hitan-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi. Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with 5 and seqio. arXiv preprint arXiv:2203.17188 2022.
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning, 2022.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, 2017.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Ims Training Millii-illion parameter language models using model parallelism. arXiv preprint arXiv: arXiv: /909 2019.
16

---

Published as a conference paper at ICLR 2024
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, 2022.
Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan D. Jepson, and Sheila A. Mcllraith. Appbuddy: Learning to accomplish tasks in mobile apps via reinforcement learning, In Canadian Conference on Artificial Intelligence, 2021.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arX.v:2210.102270 2022.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. arXiv preprint arXiv:1707.02966 2017.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. arXiv preprint arXiv:2305.1665. 2023.
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. arXiv preprini arxiv:2212.00 0266623, 2022.
Yi Tay, Mostafa Dehghani. Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. U12: Unifying language learning paradigms. arXiv preprint 1xXiv:2205.051333 2022.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibi Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv:2105. 13231, 2021.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.cm 11,, 2023.
Jiapeng Wang, Lianwen Jin, and Kai Ding. LiLT: A simple yet effective Ianguage-indeenndee layout transformer for structured document understanding. In Annual Meeting of the Association for Computational Linguistics, 2022a.
Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu. Webformer: The web-page transformer for structure information extraction. arXiv preprint wxXi:2202.00211. 2022b.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. arXiv preprint arXiv:2206.0888 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903, 2022c.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. LayoutLM: Pre- training of text and layout for document image understanding. arXiv preprint arxiv: 1912.133,,, 2019.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. arXiv preprint arxiv: 01206, 2022a.
17

---

Published as a conference paper at ICLR 2024
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, arXiv preprint arXiv:2210.03629, 2022b.
Manzil Zaheer, Kenneth Marino, Will Grathwohl, John Schultz, Wendy Shang, Sheila Babayan, Arun Ahuja, Ishita Dasgupta, Christine Kaeser-Chen, and Rob Fergus. Learning to navigate wikipedia by taking random walks, arXiv preprint 0rX0v22211.00177, 2022.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong. Stefan Welker, Fed- erico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00599, 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04560, 2022.
Longtao Zheng, Rundong Wang, and Bo An. Synapse: Leveraging few-shot exemplars for human- level computer control. arXiv preprint arXiv:2306.0786. 2023.
18

---

Published as a conference paper at ICLR 2024
APPENDIX
A BROADER IMPACTS
While WebGUM is evaluated only in realistic web simulators (Shi al., 2017; Liu et al., 2018; Yao et al., 2022a), we should carefully conduct it we deploy the autonomous web agent on the real-world Internet because of security and safety reasons. For instance, the wrong password may cause an account freeze, and emailing the wrong person is problematic in a business scene. Training with online RL may often be infeasible for this reason, while we demonstrate an alternative approach: data-driven, fully offline training by leveraging inductive bias in foundation models. Autonomous agents, well-grounded with the user's intention, should be helpful in our daily lives by reducing our burden on computer tasks. Because part of our training corpus (54K) includes the demonstrations taken from the output of L.LMs (Anil et al., 2023), we will exclude those from the dataset release and it will result in 347K episodes.
B EXTENDED RELATED WORKS
Foundation Models for Decision Making Recently, the ability of multi-step reasoning and inductive bias in foundation models have been leveraged to solve text-based interactive tasks via sequential decisions considering few-shot in-context examples (Ahn et al., 2022: Huang et al., 2022a;b; Zeng al., 2022; Yao al., 2022b; Meta Fundamental AI Research Diplomacy Team et al., 2022). Even in continuous control (Chen et al., 2021a; Janner et al., 2021; Furuta et al., 2022b; Brohan et al., 2022) or computer games (Reed et al., 2022; Lee et al., 2022b; Fan et al., 2022), high-capacity transformer models are trained with large amount of diverse dataset via multi-task behavioral distillation (Chen et al., 202 Gu et al., 202 DeepMind Interactive Agents Team et al., 2021; Furuta et al., 2022a; Shridhar et al., 2022: Jiang et al., 2022). To build autonomous web navigation agents, we also leverage pre-trained LLM (Raffel et al., 2020; Chung et al., 2022), by finetuning with sussivvely-uurate multimodal demonstrations, and we point out that the better content comprehension and multi-step reasoning abilities, obtained through instruction-fiettnii 2022), are essential for the notable performance on downstream decision making aligned with human instructions.
of LLM (Chung et al.,
Multimodal Large-scale Models Large language models have demonstrated extraordinary emer- gent abilities on a variety of NLP tasks, such as commonsense question answering, arithmetic, logical reasoning, open-ended text generation (Radford et al., 2019; Brown al., 2020: Chowdhery et al., 2022; Wei et al., 2022b; Tay et al., 2022), or code completion (Chen et al., 2021b; Austin et al., 2021; Li et al., 2022b). In addition, some works have investigated vision-an and---nd-aam and-language understanding to improve the accuracy of common vision-based tasks such as open-ended image/object classifi- cation (Radford al., 2021; Gu et al., 2021b; Kamath et al., 2021), image captioning, or visual question-answering (Lu et al., 2022; Alayrac et al., 2022: Chen et al., 2022; Reed et al., 2022: Liu et al., 2023; Dai et al., 2023; Li et al., 2023). Several works also have tackled document understanding with (multimodal) transformer models (Xu et al., 2019; Li et al., 202 la:e; Appalaraju et al., 2021; Tang et al., 2022; Wang et al., 2022a:b). including markup languages such as HTML (Aghajanyan al., 2021; 2022; Li et al., 2021b; Lee et al., 2022a) for summarization of the documents question answering on the contents. Despite the great efforts on document understanding, these works are less connected to interactive decision making problems. Our model obtains not only grounded understanding of websites in a multimodal manner but also the ability to decide the optimal actions to achieve given instructions in web navigation, helping multi-step decisions and visual context understanding.
C IMPLEMENTATION DETAILS
We adopt the encoder-decoder models proposed by Raffel et al. (2020) as multimodal transformers, and vision transformer (Dosovitskiy al., 2020) pre-trained with ImageNet-21 (Deng al., 2009) as an image encoder for the visual tokens3. We especially use ViT-B 16, a small-size transformer with 86 million parameters, which divides an input image into 16 x 16-size patches. We use publicly
phtpps://g thub. www/googl e resea rch/ scenic
19

---

Published as a conference paper at ICLR 2024
available checkpoints of T5 (Raffel al., 2020)". T5 (Chung al., 2022),, and T5-XL finetuned with MiniWoB++ demonstrations (Gur et al., 2022) for the experiments. To construct the training pipeline, we leverage SeqIO (Roberts et al., 2022) library, and use SentencePiece (Kudo & Richardson, 2018) vocabulary with 32K tokens from C4 dataset (Raffel et al., 2020) for text tokenization. The batch size for training is 128, and input sequence length is set to 4096 tokens. Due the huge computational requirements. we run one seed to train each model throughout the paper (Humphreys al., 2022; Gur al., 2022). We use cloud TPU-v4, which has 32 GiB HBM memory space for the experiments. Base-size models require 256 cores and XL-size models do 512 cores, which takes 1-2 days for finetuning.
D DETAILS ON DATASET AND MODEL SIZE SCALING
We here present how critical it is to scale up the dataset and model size in WebGUM. For the dataset size ablation, we use Flan-T5-Bass and ViT-B16. As for both HTML and multimodal models, we could observe the scaling effects in web navigation: the larger the dataset (Table 4) and model (Table 5) size are, the higher the success rates are. Surprisingly, our approach even with only 2.8K HTML episodes (about 25% of the previous one curated by Liu et al. (2018)) and Base-size model (about 7.3% parameters) already achieves 55.7%, surpassing previous SL state-of-the-art (48.4% by Gur et al. (2022)). This surprising efficiency might come from the sufficient inductive bias and alignment with the user intentions in mnstruetioo-finetueed LLMs, and WebGUM could fully leverage them for web automation problems. The margin of improvement might be smaller than expected due to the limited capacity of transformer to obtain the grounded understanding of natural language instructions, HTMI, and screenshots. In fact, the results also reveal that scaling the models might be more important than the dataset; the low-capacity model may cap the performance a lower level.
Pre-Trained Models
Modality
Dataset
Success Rale
TS-XL (Gur et al., 2022)
HTML
12K
48.4%
TS-XL
HIML
347K
49.8%
Flan-T5 -S--ae Flan-TS Base
HTML
2.8K
55.7%
HTMI.
68K
56.3%
Flan-T'--BE
HIML
347K
57.2%
Tlan-TS-Buse. VIT-B16
HTMI +Image
2.8K
61.1%
Flan-TS-Base, VIT-B16
HTML+Image
68K
62,3%
Flan-TS-Base, VIT-B16
HIML+Image
347K
66.1%
Table 4: Average success rate of WebGUM with different dataset sizes. We observe the larger the dataset size is, the higher the success rate is. Surprisingly. our approach outperforms previous of-te-----h--an by over 7.3% with eeisoee dataset (about 25% of the previous dataset curated by Liu et al. (2018)).
Pre-Trained Models
# of Params
Modality
Success Rate
Flan-T5-Bas
220M
HIML
57.2%
Flan T5 TS-Lage
770M
HTML
72.4%
Flan-T5-XI -X-LLL
3B
HTML
75.5%
11B
HIML
79.0%
Flan-TS-Basee VIT-B16
310M
HTMI HIMae-Image
66. 1%:
Flan- Flan-TS-Large. VIT-B16
860M
HTML-Image
77.4%
IS-LL VIT-B16
HIML+Image
Table 5: Average success rate of WebGUM with different model sizes. As for both HTML-only and multimodal models, we could observe the performance increases as the model size does.
.dd
t5- 15-11-checkpoints
httptt///ggttaaac.ooooooooooee// http:///////igggnogorooooooooonn mdd
flan-t5-checkpoints
http://ewocoa..aaaea//eee//eee/a 1m/ webn_t5_3b
http://ccccoooeeeoooooooooooooeo cloud
20

---

Published as a conference paper at ICLR 2024
E WEBSRC
We extensively evaluate the capability of HTML comprehension in instruction-fietuned LLMs with WebSRC (Chen al., 2021d) where the models asked to solve contextual QA problems understanding given HTML and its structure. Those problems are curated from real websites to include key-value extraction, entity comparison, and table understanding problems. The answer formats are either text span in HTML or binary (yes/no). Because the context length is insufficient for raw HTML,, we preprocess context HTML. by extracting snippet that includes the answers in advance. We finetune both T5-XI. and Flan-T5-XX with the training dataset. Table 6 shows that Flan-T5 records better HTMI. comprehension performance than T5, which may accelerates the web navigation performance MiniWoB++ and Mind2Web.
Models
EM
F1
TS-XL
63.85
71.44
Flan-T5-XL
68.91
78.48
Table 6: Base language model performance in WebSRC (Chen et al., 2021d). We finetune both T5 and Flan-T5 with trainng dataset. Flan-T3 TS achieves better performance in HTML comprehension than TS.
DATASET DETAILS
To construct large-scale multimodal behavioral dataset on MiniWoB++, leverage public finetuned-LLM. -LM policy (Gur et al., 2022) trained with multi-task human demonstration dataset (Liu al., 2018)? demonstrator. We run such LLM policies with 10,000 episodes per task and only keep successful trajectories to maintain the quality of dataset, following Humphreys et al. (2022). Lastly, we collect additional 54K demonstrations with Synapse (Zheng et al., 2023) a private-I..M- based agents with prompting, for the tasks where finetuned-La LLMM may not complete well such as click- tock-ssroll 1il1iistttt and enter- time, and also write scripted policy for book-flight flight. We use PaLM (Anil et al., 2023) as base LLM for Synapse. Such efforts result in multi-task dataset with 401K (347K+54K) episodes including HTML and screenshots at each time step. Table shows the details of our multimodal dataset (347K), consisting of HTML, screenshots, actions, and instructions at each time step.
demos
21

---

Published as a conference paper at ICLR 2024
Task
# of episodes
of steps
Ratio (episode)
book-flight
9999
90177
2.88%
choosc-date
383
1508
0.11%
choose-date-casy
3353
12946
0.97%
choose-date-medium
2222
8733
0.64%
choose-list
1861
3724
0.54%
click-button
9782
9909
2.82%
10000
20000
2.88%
click-checkboxes
9761
28904
2.81%
1962
19072
0,57%
9228
36384
2.66%
broc--hhccbboes-traeee
10000
59793
2.88%
click-collapsible
5947
13077
1.71%
dickkcollapsible-2
2199
5627
0.63%
click-color
2554
2554
0.74%
click-dialog
10000
10000
2.88%
click-dialog-2
3285
3285
0.95%
click-link
9961
9961
2.87%
click-menu
3238
3243
0,93%
click-option
9998
20000
2.88%
click-pie
3724
8548
1.07%
click-scroll-iist
0
0.00%
click-shades
0.00%
click-shape
6116
6117
1.76%
click-tab
9978
13177
2.88%
click-tab-2
1844
2109
0.53%
click-tab-2-hard
1574
1916
0.45%
click-test
10000
10000
2.88%
click-test-2
10000
10000
2.88%
click-widget
9963
9963
2.87%
count-shape
5849
5893
1.69%
email-inbox
5159
14258
1.49%
email-inbox mnniilinbox--fowwrrd-a
9995
39980
2.88%
email-inbox--o forward-nl turk
4900
20165
1.41%
email-inbox nl-turk
4346
11416
1.25%
enter-date
10000
20000
2.88%
enter-password
9980
29940
2.88%
enter-text
10000
20000
2.88%
enter-text-dyyamiic
9983
19966
2.88%
enter-time
0
0.00%
focus-text
10000
10000
2.88%
focus-text-2
10000
10000
2.88%
grid-coordinate
8353
8353
2,41%
guess-number
1021
2042
0.29%
identifysshape
9007
9010
2.60%
login-user
9793
29379
2.82%
login-user-popup
9786
39170
2.82%
multi-layouts
10000
40000
2,88%
multi-orderings
10000
40000
2.88%
navigate-tree
9864
15140
2.84%
search-engine
8872
35095
2.56%
social-media
2631
4407
0.76&
codal-media-all
95
208
0.03%
oocil-mediia-some
319
893
0.09&
tic-tac-toe
3947
13773
1.14%
use-autocomplete
3465
6930
1.00%
use-spinner
530
532
0.15%
Total
346827
867277
100%
Table 7: Details of our multimodal dataset. It contains about 347K episodes in total.
22

---

Published as a conference paper at ICLR 2024
G PER-TASK PERFORMANCE OF MINIWOB++
In this section, we present per-task success rate on MiniWoB++ (Table 8) and absolute performance improvement by adding image modality to HTML input for WebGUM (Figure 7).
As for Table 8, we refer to Gur et al. (2022) and Zheng et al. (2023) for the baseline performances. We 56 tasks as benchmark, while removing some duplicated tasks (e.g. "-nodelay" tasks) from 62 tasks adopted in Gur et al. (2022). During the evaluation on MiniWoB++, we ignore the time limit due to the computational constraints.
Figure 7 presents full results of the absolute performance improvement, subtracting the success rates: (Success Rate of WebGUM (IMM++Imaee) WebCUM (HTML) The results suggest WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions (e.g. book-flight or search-engine) or the tasks that require global contexts of the page (e.g. tic-tac-toe or saicck-ssape)) See Appendix L for the visualization.
(Success Rate
120
-20
MC Vac dalee mmsiiii
ch ph bootl stighs Cf
click
faal congiters
Caa medioo anaare homntee
datte sane
cerrecteen coff daarcc
farrree bargge
novearee aiturk
forward
Figure 7: Performance improvement by adding image modality to HTML on tasks from MiniWoB++. subtract the success rates: (Success Rate of WebGUK (BTMI+ WebGUM (HIML) ).
(Success Rate of
Task
Synagse
Human
AdaPlazner RCI
RCI
CC- Net
CC-
WGE
WebN- T5
WebGUM WebGUM (ITML)
(GPT.
Net
(S1.
Niscc-angle book-flight
m14
0.92
(161
(nfo
n/o
0:97
0.29
n/a
r/a
n/a
0.98
chase-cirele
chons--itee
1.00
0.97
nbs
nin
0.97
0.12
0.00
0.00
0.98
1.00
choose-att-csry
0.99
B.99
0.42
I/
0.03
0.95
1.00
choose made-m
choos-Hist
0.98
circle-center
ala
0.96
mlo
nío
0:97
0.36
a/a
click-bulion
1.00
0.98
1.00
1.00
1,00
1.00
0.78
1.00
1.00
1.00
1.00
chekkehsessooes
diok-checkhusee--l
1.00
0.87
1.00
0.94
0.94
0.71
0.00
0.68
0.22
0.97
0.99
click-chesthhsees-ee
1.00
0.73
0.80
0.72
0.96
0.95
0.04
0.51
0.54
1.00
1.00
click-
click-oollapsible
0.99
0.81
clikk-oolapsiblee22
0.96
0.97
0.84
0.62
1.00
0:98
0.17
0U65
0.00
0.94
0.95
click-coloe
1.00
0.97
1.00
1.00
1.00
1.00
0.82
1.00
0127
1.00
1.00
click-dinlog
click-diolog22
0.99
click-fink
1.00
0.99
0.98
1.00
1.00
0.99
0.59
1.00
1.00
1.00
1.00
click-menn
1.00
0.97
0.78
1.00
1.00
0.94
0.22
0.37
0.99
0.97
click-menu-2
click-option
0.99
0.99
0.21
0.87
1.00
elick-pie
1.00
0.9%
nós
nds
nis
0.97
0.15
0.32
0.51
0.99
0.99
click-senolllist
1.00
0.91
1,00
1.00
1,00
D.60
0.01
0.00
1.00
1.00
chick-shudes
click-shipe
click-ub
1.00
0.99
L.00
L.00
1,00
1.00
0.95
0.55
0.74
1.00
1.00
click-tab-2
0.94
0.97
0.85
0.74
1.00
0.98
0.27
0.64
0.18
0.95
0.99
click-tah-2-eeey
click-tbb 2-hand
0.96
0.96
0.78
0.76
0.98
0.98
0.19
n/a
0.12
0.95
0.85
fiok-uah-2-medim
mba
0.97
ná)
n8)
0.99
0.54
n/
1/9
n/a
n/a
click-test
1.00
1.00
1.00
1.00
1,00
1.00
1.00
1.00
1.00
1.00
1.00
23

---

Published as a conference paper at ICLR 2024
click test-tes--tmteerr
click widget
1.00
0.83
1.00
0.98
0.98
1,00
0.56
0.93
1.00
1.00
1.00
cupy passe
1.00
0.94
nia
mia
0.80
0.04
copy-paste-2
shupe
F20
0.99
0:11
count sides
0.98
1.00
0.70
10
drag- box
81a
0.99
1,00
0,50
drag cube
0.23
drng-itemm
0/64
drag items
81a
0.95
1000
106
0.99
0.13
IN2.
IVR
1/1
drag items grid
0.87
mia
0.98
0.05
2/1
drog-shapes
drng
0.11
email inbox
1.00
0.96
0.98
0.98
0.98
1,00
0,09
0.43
0.38
0.99
1.00
emadl-aber labee--eeltte
a/a
0.99
1,00
0122
email inbex forward
1.00
0.88
1.00
0.94
0.94
1,00
0,00
IV
0L33
1.00
1.00
email inbox importast
0/a
0.99
1,00
0.30
2/1
email-ibbe--ll-ee
email unbox ceply
0.91
1,00
0,00
103.
2/1
email-atte momal--abox--tarrrre
0.95
1,00
0.11
IVA
date
eater password
0.99
0.97
entes-eet
1.00
0.9%
0.98
1.00
1,00
1.00
0.35
1.00
0.89
1.00
1.00
eoter enter-text-2
0.91
r/o
1158
0,04
281
ento--txtt-yynmmic
0.98
0.96
find hnddmiidpiint
0.94
rnfo
0.97
0.35
1V2.
a/1
find-word
0.84
0.96
D.88
0.05
foeus-text
text-2
grdd-condinnte
1.00
0.87
1.00
1.00
1,00
1.00
0.66
1,00
0.49
1.00
1.00
guest-uumber
1.00
0.99
0388
0..20
0.200
1.00
0.21
01.00
0.00
0.34
0.43
highlight-test
highlighr-ieu--2
0.97
sdentry-shape
1.00
0.98
0.96
0.76
1.0
1,00
0.68
0.50
0.88
0.90
1.00
lagin-user
1.00
0.96
1.00
1.00
1.00
0.00
0.99
0.52
1.00
1.00
logio-nses-opupp
moving-Iteass
malti-kyount
0.94
0.95
0.84
0.72
0.96
1.00
0.00
0.99
0.83
1.00
malti-nedrrnggs
1.00
0.96
1.00
1.00
1,00
1.00
0.00
0.99
0.58
1.00
navigste-tree
mimberscheelivoes
0.96
read-cable
1.00
0.97
nío
0:97
0.01
N/
IVA
n/a
read cable-2
0.95
0:94
0.00
right-angle
0.87
scroll-ient
mb
0.97
0.96
0.04
n/a
scroll-ies--2
md
0.97
1.00
0.88
engine
simon-sayss
0.62
0.00
0.02
IV/
amplo-algoben
1.00
0.86
0.82
1,00
1,00
0.75
0.03
IVA
simple ariihmetic
1.00
1,00
0.86
0.38
mla
social-mofia-all
0.31
social media some
1.00
0.91
0.90
0.90
0.96
0.85
0.01
0.01
0.02
0.89
0.73
terminal
0.98
0.88
0.98
1.00
1,00
0.00
0.00
text tnnnsfofo
0.80
0.80
0.50
0.19
mla
tic-tee-to
1.00
0.71
0:48
0.56
0.56
0.83
0.32
0.37
0.48
0.50
0.56
unccude lest
1.00
0.99
0.96
m/a
me--colrrheel
IV/A
colurwheel-2
ml
0.94
n/a
m/a
0.95
0.38
n/5.
n/
m/a
mer-elider
0.98
0.91
0.18
use-slider-2
use spinner
0.98
0.90
0.95
1.00
0.47
0.01
0.07
0.06
0.11
visual- -uddition
m/a
0.97
m/0
0.99
0.36
n/a
115
7/5
n/a
Average
0.985
0.935
0.929
0.906
0,960
0.935
0.305
0.546
0.484
0.887
0.942
o Tasks
104
53
54
54
104
104
48
56
56
56
Table 8: Per-task success rate on MiniWoB++, We refer to Gur et al. (2022) and Zheng et al. (2023) for the baseline performances.
24

---

Published as a conference paper at ICLR 2024
H COMPOSITIONAL EVALUATION ON MINIWOB++
For the compositional evaluation, we pick up 4 click "sometti"" (link, button, checkboxes, dialog) tasks and make some combinations of those by naively stitching with tasks, Then, prepare the following 6 combinational tasks,
2icc=-buttc__cclckkdda.....
alic=-buttea__cliok-1in.
LLoc--Link_clicc--atton
bllek-Iiikkccck---tttoooollooddd click-link_click-diaal
These tasks should be resolved order of the name: for instance, in Seik--iinkkliik-butttr
-iiik--dialogg task, the agent should click the proper link, click the proper button, click the proper dialog, and then the task results in success. In bucc--butttncclickk-
task, the agent should click the proper button, and then click the proper link. The instructions for compositional tasks are also simply combined among original task instructions order of the name. This evaluation could test the ability to transfer primitive skills to control computers to solve unseen tasks. Table 9 shows the per-task average success rate among 6 combinations above. WebGUM can solve the compositional tasks much better than baselines (Gur al., 2022; Zheng et al., 2023).
Compositional Task
WebN-TS
Synapse
WebGUM (HTML)
WebGUM (Imae)++mage)
click-button_
0.26
0.84
0.21
0.27
click-button_ee bts--button_elick-diille
0.95
1.00
0.87
0.93
click-button_ekte soutton_elick-lint click-link
0.87
0.99
0.81
0.88
click-link click-button
book-link_click-bunng
0.35
1.00
0.90
0.95
click-link chick-button_ooo
wottttttecooociiiallleele dlick-dialog
0.08
0.60
0.73
0.73
click-link_chie chickkcicddialog
0.55
0.00
Ave.
0.510
0.738
0.742
0.785
Table 9: Per-task average success rate 6 tasks from compositional MiniWoB-e
click-link
click-button
click-link click-
Click on the link "nam".
Click "Okay" button.
Click on link "scelerisque" then click on the "Ok" button.
Cursus justo, Facilisis aliquam nisl viverra pharetra scelerisque. Rutrum adipiscing. Pretium, Egestas ultrices nam sceleriague sit nunc tortor. Placerat. Egestas.
viverra nam sociis nulla nunc:
Cursus justo, Facilisis aliquam nisl viverra pharetra scelerisque. Rutrum adipiscing. Pretium. Egestas ultrices nam scelerisque sit nunc tortor. Placerat. Egestas.
placerat egestas:
aliquam tellus:
submit
no Okay
Next Submit
Ok
et nec convallis
Ok
Figure 8: Example of compositional evaluation on MiniWoB++ (the same Figure 5).
25

---

Published as a conference paper at ICLR 2024
1 COMPARISON AGAINST PRIOR WEB NAVIGATION AGENTS
Methods
Architecture
Pre-trained
Input
Output
Offline
WGE (Liu et al., 2018)
LSTM, self-attention
DOM
Logit of action
CoDE (Gur et al., 2019; 2021)
Bi-LSTM
DOM
Logit of action
DOM Q (ET(Jia et al., 2019)
GNN
DOM
Logit of actionn
CC-Net (Humphreys et al., 2022)
LSTM, Transformer, RosNet
DOM, Screenshot
Logit of action
WebShop (Yao et al., 2022a)
BERТ, BART
Text (from HIML)
Logit of action
WebGUM (Ours)
T5 Transformer, VIT
HTML, Screenshot
Text
Table 10: Prior works have studied web navigation problem as online RL to learn the optimal action distribution with spec-fpe model architectures from scratch (" or partially using pre-trained vision encoder). We omit the web-specialized architecture and input-output space, and convert web navigation into visual guestion-answering format (text, image text), which allows us learn the agents offline by leveraging pre-trained foundation models (Raffel et al., 2020; Chung et al., 2022; Dosovitskiy et al., 2020) in vision or language domains as strong inductive bias for web environments.
J INPUT PERTURBATION EVALUATION ON MINIWOB++
Add extra HTML at the top
Add extra HTML at the bottom
Add Coordinates
righte "800" top="o" bottome 2000 id="wrap"
rate"1"
<body rotertto-od too id="wapp ref="2"sodd idetares" refergrs-ddl idet search bar" tinpput type text" id- "searrch-eoot" stnptrr-nnppt--tuuute ss=acchrch" Sntenor-earch-alee BrrSchchhibutton ids whap" wff="">>ddv "reaa"
<body ref="1">-odd iddt wrap" refe 2*
right top="0" hottom==2100*me ide ance" not="3" latt="o" right"1 tope"50"
id- "aree' ref="3">xbuten nottr No-NNoibutton-ssppp rofi" viverra
bottome bB0 >buttebttton ref-14 4 luft"2"
ei--aspane types"tex"
type="lax"
bottome bollomm"44"" "4**Nosbbutoo
righte334" toge 152"
log- bollom- "87
ref-"st 2e
ref-ngr cinput--iip hoxf""
ref- 7r--hhpp---nppu type""tex!"
ref-"S hid vivers el spppan>--ppul type- Tlx!''
rate"traxtbutoo rat="4 raf="""NNocbumm Nooosuuuooo--pann
rel- SSt-dii "d=""ap" ref--gr d- "aree"
rote pteer7ooninnnuttrrrrg type="lex"
ref- 8r-iirupp-uuuullo
ref-"10" tbwnno-buuttnn-ddooo wttp://bbuoooootoo---o---ee -itplbin---diheedd-----o- -bodyy
rate ttpppooa.cchuuuttntttt
et-tspare--npe typen
bottome" "i8"==onnptoooiip type="lox"
bottome"131 13ttmminuutssssutt--o type="lox"
right"" 139* lop-
refe"y" lett= 155t top="109"
lett="
ref- -bbcchnup--bxutte
rete "rgeoveus-bumooo--uuue
reter edy class
cless ootor" rof-2"> rof="2*<<ddvv rof= -idde--v- class= "oolar"
dass "color"
bollom= "505=
rat="a lott= right-haar top="131" 131"
left22
ref-"10 boo-bbbtooo=<dyy-ddde-------
roter sidhe slvo wttpeiietoodopooooooooooo
rights 15g="155"
bottome 175" 775"=previouehhuttoooe
ref-"10" lell- right-"32' tup="175"
bottome bottom"1197"no 1B7"=no-buuton hat--bbttn---v--dd----bbbb--
Figure 9: Example of input perturbation for MiniWoB++ evaluation (the same as Figure 6).
26

---

Published as a conference paper at ICLR 2024
K EVALUATION ON WEBSHOP
In addition to MiniWoB++, we extensively evaluate our WebGUM on WebShop (Yao et al., 2022a) benchmark, another online-shopping websites simulator with large amount of real-world product data. WebShop provides user instruction that describes the feature of items (e.g. I need a long clip-in hair extension which is natural looking, and price lower than 20.00 dollars). The agents should search, compare and choose a proper product that matches the given instruction. Since WebShop requires complex multi-step reasoning considering previous contexts for comparison (Yao et al., 2022a;b), we can test the capability of instruction-finent finetuned- L.M in decision making tasks in depth. The performance score evaluated by the percentage of required attributes covered by the chosen product (from 0 to 100), and if the product meets all the requirements, that episode is labeled success.
Because WebShop does not have API to get the screenshot of rendered websites, we focus on We- bGUM with text inputs, parsed from noisy HTML in the real world. 9 We convert the actions from raw texts (e.g. search [a long clip-in hair extension] to dictionary-like format (e.g. ("action", "search", "ref": "a long el clip-in hair extensions on") use in MiniWoB++, to improve the prediction accuracy. We finetune Flan-T5-XL with about IK human demonstrations curated by Yao et al. (2022a) using only high-score demonstrations. The score threshold is score 2 50 and we have 840 episodes in total (Table 12). We construct the model input with action history, instruction, and text observation, the same as MiniWoB++ experiments. We evaluate our method with 500 user instructions in the test set.
click [<item id>])
("action": "click", "ref": "<item (),
Table 11 shows that WebGUM achieves 45.0% success, significantly outperforming not only simple baselines, such as supervised imitation learning (II.) and II. plus RI.-fintting (by more than 15%), but also recent prompt-hased L.LM agents, including ReAct ( (Yao et al., 2022b) (i.e. PaL.M-540B (Chowd- hery al., 2022) with one-shot prompt and reasoning annotations), while our model only has billion parameters. IL and IL plus RL-finetuning baselines use BART (Lewis et al., 2019) model for the search policy, and BERT (Devlin et al., 2019) model for the click policy. The better performance of WebGUM proves the hypothesis that the ability of multi-step reasoning in instruction- finetuned language models works as prior for decision making problems.
Methods
Training
Model
Modality
Score
Success Rate
Rule
Text
45.6
9.6%
IL
SL
BART, BERT
Text(+Image)
59.9
29.1%
IL+RL
SLARL
BART, BERT
Text(+Imago)
62.4
28, 7%
Act
In-context
PaLM-540B
Text
62.3
30.1%
ReAct
Incconteat
PalM-
Text
66.6
40.0%
Wo
SL
TS-XI.
Text
61.0
29.8%
WebGUM
SL
Flan-TS-XL
Text
67.5
45.0%
Human
Text+Image
82.1
59.6%
Table 11: Average score and success rate on WehShop (Yao et al., 2022a) benchmark. WebGUM based on Flan-T5-XL. achieves 45.0% success, outperforming most baseline approaches including ReAct, prompted PaLM-540B with reasoning annotations. We refer Yao (2022b) for the baselines.
Threshold
# of Episades
Score
Success Rate
acore 2 0 score
1026
67.2
44.4%
840
67.5
45.0%
score 100
497
65.3
44.4%
Table 12:
Average score and success rate on WebShop with different score thresholds. Because we should balance the dataset coverage and proficiency, we choose 50 as threshold.
WWebShop just provides visual features of item pictures when the agents reach the product page. These features extracted by ResNet-50 (He al., 2016), rather than raw images or sereenshots of the website. Some haseline agents (II. and IR.) incorporate such embeddings. 1https: / / gi thub. com ctom/prinncee acrnncennnnnn data
/basel mee_mode1 /s/
27

---

Published as a conference paper at ICLR 2024
L EXAMPLE EPISODES OF WEBGUM
Mike
E
click-pie
tic-tac-toe
as ddde
Suark
Sueni
Exbmil
choose-date-easy
grld-coordiinte
bC suck, Lidd
DC Bark UiGttiand dee
PC Gwik sglleredd pic
oniss 10/ poc
DC, sunk, UoGh and 604
DC. gink USGP
Subrth
Subrit
Sibri
Subrik
Subri
Sibrth
Subrit
click-checkboxes-large
tetten Your
Yoor"
tachun Toy""
Lhe addu "Tee""
Swarhh
Swoth
Swares
Tres
Swerch
Spetiel:
AAL
1232
search-engine
Figure 10: Example of successful episodes demonstrated by multimodal WebGUM on MiniWoB++ (Shi et al., 2017; Liu et al., 2018). The time step goes from left to right. As discussed in Section 5.1, image modality seems to be leveraged for multi-step tasks with dynamic page transitions (e.g. search-engine, choosc-date c-cas) or tasks that require global visual contexts (e.g. tic-tac-toc).
28

---

Published as a conference paper at ICLR 2024
neo viai or Jrx?
WebGUM failed to count the number of items because of the ambiguity of instructions ("small").
2
1
count-shape: How many small items are there?
And
agaand F.linnl.
WebGUM clicked the button the wrong posts (not by @rrndda). The repetitive structure of the page may cause confusion.
Danell
Ganell Girnis Puetter
de clss. Putl
KaN Grymm
Kin Grvira
Terye Oress
social-media-all: Click "share" " button all posts by örenda and then click Submit.
Ture preber Falvee
Duass Calvee
Satual thribe
No aatter Tobwean
WebGUM repeated the wrong answers during multi-step QA.
Submi
Sulmit
Submiit
Sub
Subei
guess-number: Guess the number between 0-9 and press Submit.
Figure Example failure episodes by multimodal WehGUM on MiniWoB. The time step goes from to right.
For instance, WehGUM suffers from (1) ambiguous instruction, such as small items shhunnsshaae,,. (2) confu- sion with the repetitive structure the page (social -media-a and (3) long-horizon gquess-number tasks that may require memory.
29

---

Published as a conference paper at ICLR 2024
Instruction: am looking for dark denimm color ethylene and price lower
Instruction: would like slange plus red camellia fleece jacket machine washed, price lower 130.00 dollazs
of 10, Sed seneration
ceasch dark denim ethylinne vinyl uilra
searca [slarge plus camellia jacket that can be machine washed]
Action:
train size 10, And generation for men]
Observation:
Observation:
[back search] page (bocal results: [nexc]
[back page (total results: [nex]]
[s0ebIm:fki]]
[MP9n2a955p]
salewa ullm tnxin 3- men's
's thermal underwear punts usb heated cotiom heating long leggings bottoms
$104.0 $182.00
[mc2xl
$56.77
skechers men's low-top crainess, us men $53.39 $140.35
[b09bcddfix]
thermal underwear usb beased
[00775hhr3tt]
long tronsers leggings bottouss for winter washable basclayer $56.77 to $56, 99
Inuna mare men's % linen camvas strotch lonfer shoes
$39.99
[609gmm113] premium and fashion sexy fuzzy sleeveless outwear $9.47 to $12.47
for plush jackets
Action:
click [500b1mcik]]
Observation:
[back search [prev]
[2.5]
[85][9] [10] [10.] [1:s]
Action: [ncu]
[12] [12.]] [13]
Observation:
(ombre blue! red orange] [raw black our]
[back souzch] page (total results: 50) [sxev] [nexs]
(dark denim omange]
salewa ultra inain 3 men's '
[+07cg18hi]]
price: $104.0 $182.99
king and queen write hoodie couple matching sweater pullover booded sweatshiri jacket
rating:
descctption) [reatures] [reviews] (buy now]
$26, 59 to 529.99
[b09jmvipss]
Action:
click [dark denim red orange]
lined flannel, jacket plus size plaid button down fleece shirts couts thicken worn cotton curwear
sherpa
Observation: you have clicked dark denim orange.
Action: click| [10]
$36.79 $55, 99
Observation: you have clicked
[b09kaigha2]
mankalvay womens jacket cartligan flannel cowl meek asymmetric loose outerw plus size xs-axt 0-22 $20.3 $26.85
Action: click [buy now
Termination: Your
1.0y:
Action: click [07ccpBrbi]
Observation: [back search) [pres]
size [king small queen small] [king mediam)
guzen-mediam)) [King-lagg] [queen-large] king-xxi]
king-slarge] queen-starge]
king and queen write hoadic couple matching sweater
pullover hooded sweatshir jacket
pricr: $26. 99 $29 $29.99
rating:
[description) [[eauree]] [reviews] [buy now]
Action: [prev]
Observation:
[back search] page 2 (lotal results: 50) [prev] [nee]]
[b07cy]8rbi]
king and
[b09jove(paa]
sherpa
[gg9t
mankalviyy womens...
click [609jnvfpzs]
Observation:
[back search] [prev]
[army green] [[tacs]] [coffee] [aray] [prezn] [red]
[i-rmy green] [x-rod]
size [small] [maddium] [Jurg]] large] [sx large]
[3x-large]
sherpa quilted flannel jacket plus size plaid button
down fleece shirts coats thicken warm cotton outwear
price: $36.79 $55.30
[description] [feurwes]]
[reviews]
[buy now]
Actions el tick rrd]
Observation: you clicked red.
Action: click [x-laree]
Observation: have clicked l-large.
Action: ol ink Tuny sow]
Termination: Your scoce (min U.O, max 1.0): 1.D
Table 13: Example of successful episodes demonstrated by WebGUM on WebShop (Yao al., 2022a). WebGUM could search and choose the required product and choose proper options (left). addition, WebGUM could also compare the products with browsing and backtracking (i.e. clicking "next" or "prev" buttons) during the episodes (right).
30

---

Published as a conference paper at ICLR 2024
Exampe I
Exangle
HTML
HTML;
cantoraioodd rdderu pu beckerd, aode aae"S124"
cimtischepp dedio Where o backend,
type plactakliee
horkeed ande Mu iogh
wieec dnciturien
type- "hemnn Sccmtpechhtttth monoo-otttteeeee www.e mide "woit"
<arg badkerel psdk, jile 7015
Ulee"S durnpe up South Flerila lotel by dae bead
buckenal Ladee
all "S14) &ana, up- Scata
dampe op headh chetto- crttoo
Addernyy mntring mon Nill,
beckaol sode 34="5641 "eceettheew Denii Lices RB Devil Mertginay exited joi locus that's susting sonedling
beckood sode sde't 1524 poddiooeessoee Mlocteetroogee wtwpooooootottteeeeeeeeeeeeeeeee
enay cheitoop--ee -o--o------nnn Monegemerr the Laneo beggor
backend ande, 1325 sooddo-dreedd Vanas,
spar with
tttsschaessoooolllloooooooo ib oosoccrre
Baned HIME webpaze shove, cemples fellowiay tade
Heaed HIM wehpaze abone, complesc the folowwagg tade
Tasc: Find the renats of the assd recest ND1 painos
Trak: Fiod hetel coab in Las Voga te lour adalis suntan on Nay 31 ending May N0 and ded acutlable set as olert for sate.
Prodioe antions
[Lias] 1ea cLacK
Plaae velect felloniza page plase wiler
cheiose comen
dea -> cluTkk [di+] Iooler CLacH
heathoot Shat
"Noac abowe"):
What doaild he the nor acton Planee sekec from the tellowing
A. Nans of shove
choacee the sames acton is not in the page shoue, placee srlect "Naso aboud").
wode "4"55124"
iilee Score/iteedle Sedeeo SoreeeSeeiieellooeeeeooo
c, baciund, atd "aar"
A. Nasz of the shove
trize "aaneChaans" Cosnnniannhhreeeootooooo iu <inng hadoood oode ide "MHi"" ah- Haward to
D. capus baducnd, prdk, abn 12340" typon "test plackhtkknn "Stan typing wdee dealvaaiie??
Sidhssaorrrrrrre Addertey nelling
cirg baddred by leadi
$100 Rorre
beach"
dasas Paenda aoid
E- badianll pole Jle 1944. Nev Detiil Lizas RB Daval
Manigraary rected jon fvom thank contay comsthing
D. hackeed rode, Idc 18001 beftroccion/eer
erary chartoooe
E. pokk 15254 vallodaaaaas Vigaa.
P. backend, pede 15200 bectrocttotoottur IS.
Protiction: S. CLICK
Protiction: 3. 20PE las vaçus
Table 14: Example outputs of WebGUM in Mind2Web dataset as evaluated in Section 5.5.
La ctel (2018)
Our
Instrctioon Seleet si SR XPSZgyO INTS nilgO, (VVissH MKHH. Kis; JRM. YyWis, (S4009 a130 cick Sibeei
Iectruction: Seloct oyRIE Cyp. TL. MIAZUL LOPWylN, OCM, V5yt. i 18ud and Sateik
HINI:
HTMMLc
chady reter Toodd wnng Stoeddr sdd "asa 3'ooddy
<bedy "aaat "woo" wete "irr"
the
open
Bauhhstoppe dass TEXT CLASS
67 vales Talle
Weneee TIXT eI ASS"
Ninee bodlodooottddddda daald udl selapel type canddouc"
rafe Neae -tt.lop-taacddddeee adl-") Sedapet chodkous"
ade dul" adw 3n sduce Fale boopapttie chawe TEXT CLASS"
ide't ski refe 5 valaer Faldc sccepudd de "Necc"
art "Nooe say Wetychoo heb-cabee nis Phocinge typei "chettho"
clasci TEXT uus "Meer attll//--hhs/hhbbbbbheee
typec Neee" chas TEXT CLASS"
aele"Nan bua Vallool wad.eooddalebddaee typen dackklo id old refe 12 valeo Fiasd exteppped
rofe Nead dill Bis Seb- infe 11 HH'xippp
coour TIXT CLANS" net New 541 oillooo-ttantdddd arti PS-crpu open "ceechoss Ribee
typec chocdhec st aas and 12 vitee Toble "-ohppat-o hrme""
TENT C3A "Noar www-hh//ee---hn-------
CLASS"
tedegg togee
refe Nooe KKPS7 2 www.oooalebbbbbaeeoooooo "chckkboo" casee TEXE CLAAS" tele Nome belggtobse Mhroo dight whlll "sclaaa wf drepu oyos checches"
de "ee" TEXT CLASS"
rute 18" valee scleeptoe
Neae stndoodd mor'15 >arpp chockses"
ide't cas rafe valaen Tane screpg- aee
sel="N ooooooo----oeeeel Setepeee
typaa
ad+"d?" udw 21 vake dolcssi clas TEXT CLASS" ret. NNaw bal Tochschbii--chh nit 22 occepa "owethhh "ass
class TEXT CLASS ade "Nene bilaacooldddeddd
not: sctepp type ddectbon it 1aT ni-
wakee soleppne disce
boteee-d "Neee" con
sefe NNoc M. whew
Noae
de 16 StXXooo- TENE CLASS
rdue-
TIXI i CLASS aht cepohooldebbcileee not 200 sccepe
types "chckheaa" 1di callr wnt 2n valaen T vnd sorppio
roho 20 typer checkben ue cal ndi 22"
closs TEXT CLASS" refe Now MwW.elleeleeeeeeeeeee Sedepp open
TEXT CLASS"
>-depet tyea
Nanee Sinee S4544 wtg.o aareobo ide audens
caas TEXU CLASS ane Non wtw.daoollaa----oooeeoeoo
cbou' waandary aolce" www.s/ewwichheoooooooooooooooooe
lnd crtm dass woies"
Actioes:
Actione
1. Jssson click 25)
(dick chaddeus 431
1. [adions didc 12] takck checkbos 4yNiova)
1. stck. 14) loick caudiber (Pstngo
[asion dlck 19] jclikk checkhos 11.s)
4. Jucion clisk (9) kcick candlos 52115
[adion did (dcck checkbes Misuzu
5. heiin click vof: 21) kikk chndlaus R9000)
3. (atian dkd. no 211 tokckk clecilio JC1Rpoo)
6. Jeeton chick mit 15] (dlckk chaddber exigt)
clfck (0) kilik choddoss sthvoaat))
[acion: did [0] stlick checklos (A1
9. locion clikk ref. 27) kilk chesklos 19535)
9. (adian: dkk. not 26] cllikk suberic betten)
(ceniiznc)
Table 15: Qualitative comparison between previous 12K episodes (Liu et al., 2018) (left) and our 347K episodes (right). The examples are taken from click-cheekboxer large. While previous work has included "hesitant" behaviors (e.g. clicking the same checkbox several times), our dataset has "shortest" behaviors. We manually annotate the action for readability (c.g. click checkbox 9 ()).
31


Reviews:
[{"review_id": "j0s6hmCcry", "reviewer": "Reviewer_a21F", "summary": "This paper studied web navigation with instruction-finetuning multi-modal models. Specifically, they created a large-scale datasets by generating data with pretrained LLMs. They then train a vision-language model with flan-t5 as the base language model and ViT as the vision encoder. The model will take html code and image screenshot as inputs and take navigation actions such as click and type.", "questions": "1. How is the generated dataset high-quality? Is there a human study to prove this? As described in section 4.3, my understanding is this LLM-generated dataset can be noisy. \n2. It is also not very clear about the data generation process. What does it mean by \"rollout a LLM policy with 100 episodes per task\"? What are the inputs and outputs here?\n3. As mentioned in section.5, the method is evaluated on MiniWoB++ with 100 evaluation episodes per task. Does the reported numbers from previous work also use the same eval set?  \n4. Could you explain more on why your model would outperform human? It is hard to imagine a small model can do better than human on general web navigation. Is it due to overfitting to a specific dataset (Table.1)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The performance of their model is good. It outperforms previous methods under different settings.\n2. Writing is clear. \n3. They performed detailed analysis such as dataset and model size scaling.", "weaknesses": "1. The technical contribution is very limited. The takeaway is to do supervised training on a large-scale model-generated dataset. It feels like knowledge distillation of a combination of model outputs (as described in 4.3 they used various LLMs to generate such data).\n2. The dataset creation process and quality is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtLgeK9e6H", "reviewer": "Reviewer_cMQF", "summary": "This paper studied data-driven offline training for web agents with VLMs. The proposed WebGUM observes both webpage screenshots and HTML pages and outputs web navigation actions, such as click and type. The authors collected 347K high-quality demonstrations using their trained model,", "questions": "- failure cases analysis: include failures cases on datasets like minwob, on which the success rate is high. \n- include demos on mind2web and webshop.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Novelty: The proposed WebGUM agent exhibits a novel combination of HTML and image modalities to tackle the challenges in web navigation. \nPerformance: The empirical results are compelling, with the model showing substantial improvements on the MiniWoB and WebShop benchmarks. \nResource Contribution: The authors have collected and made available a significant corpus of high-quality demonstrations, which is 38 times larger than previous datasets. This contribution is likely to be valuable for the broader research community working on similar problems.\nClarity and Structure: The paper is well written with clear explanations of the methodology and discussions surrounding the results. The inclusion of comparisons with existing SoTA methods provides a good understanding of the performance gains achieved.", "weaknesses": "- Generalization: It’s not clear how well the proposed method generalizes to a broader range of web navigation tasks outside the tested benchmarks, especially HTML that are longer than context length. More discussions or evaluations on the generalizability could strengthen the paper. \"Because the context length is insufficient for raw HTML, we preprocess context HTML by extracting a snippet that includes the answers in advance.\"\n- Figure 1 and 5 are blurry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A7jCyoTe59", "reviewer": "Reviewer_idei", "summary": "This paper proposes an instruction-following multimodal agent, WebGUM for web navigation tasks. The proposed method has improved upon the prior SOTA method (including humans and a GPT4-based agent) on MiniWob.", "questions": "Have the authors tried LLaMA (or LLaMA2) for based LM? If yes, what are the results? If not, why not?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well written and easy to follow.\n2. The proposed method achieved amazing performance on MiniWob.", "weaknesses": "1. The proposed method is neither impressive nor novel. Given the fact that multimodal MLLM can already complete various difficult tasks such as visual reasoning or science QA [1,3], achieving the highest score in MiniWob does not seem very surprising. \n2. Since there is a collection of [opensourced MLLMs](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models), which also combine a VIT-like vision encoder and an LLM, similar to the proposed WebGUM. The reviewer does not feel like the “showing another MLLM can do tasks like web navigation“ is significant enough. Especially when some of the prior MLLMs [1-3] are already using LLaMA as their base LM, the proposed method is still using T5.\n\n[1] Liu, Haotian, et al. \"Visual instruction tuning.\" arXiv preprint arXiv:2304.08485 (2023).\n\n[2] Li, Bo, et al. \"Otter: A multi-modal model with in-context instruction tuning.\" arXiv preprint arXiv:2305.03726 (2023).\n\n[3] Dai, Wenliang, et al. “InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning”, arXiv preprint arXiv:2305.06500", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)