SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MIND2WEB: Towards a Generalist Agent for the Web
Xiang Deng* Yu Gu Boyuan Zheng Shijie Chen
Samuel Stevens Boshi Wang Huan Sun* Yu Su*
The Ohio State University
https://osu-nlp-group.github.io/Mind2Web
We introduce MIND2WEB, the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. Existing datasets for web agents either use simulated websites or only cover a limited set of websites and tasks, thus not suitable for generalist web agents. With over 2,000 open-ended tasks collected from 137 websites spanning 31 domains and crowdsourced action sequences for the tasks, MIND2WEB provides three necessary ingredients for building generalist web agents: 1) diverse domains, websites, and tasks, 2) use of real-world websites instead of simulated and simplified ones, and 3) a broad spectrum of user interaction patterns. Based on MIND2WEB, we conduct an initial exploration of using large language models (LLMs) for building generalist web agents. While the raw HTML of real-world websites are often too large to be fed to LLMs, we show that first filtering it with a small LM significantly improves the effectiveness and efficiency of LLMs. Our solution demonstrates a decent level of performance, even on websites or entire domains the model has never seen before, but there is still a substantial room to improve towards truly generalizable agents. We open-source our dataset, model implementation, and trained models (https://osu-nlp-group.github.io/Mind2Web) to facilitate further research on building a generalist agent for the web.
The web now hosts billions of websites [7] that cover virtually every aspect of the digital world. In this work, we seek to answer the question: How can we build a generalist agent for the web that, given any website, can follow language instructions and carry out the corresponding tasks? Some exemplar tasks for such an agent are shown in Figure 1. A generalist agent could make the web more accessible, which is becoming increasingly difficult as modern websites provide increasingly more functionalities that also increase their complexity and learning curve. On the other hand, such an agent may also turn the entire web into an unprecedentedly powerful and versatile tool [23, 27] that can enhance large language models (LLMs). For example, it may be used as a plugin for ChatGPT [24] to directly acquire information and carry out actions on HTML websites, instead of only retrieving web content through a retriever tool [8, 18] or relying on pre-defined APIs for each web service [34, 37].
A generalist agent for the web shall meet the following desiderata: First, it shall work on any website on the Internet. Since it is infeasible to collect sufficient training data that covers all websites, this requires the agent to be inherently generalizable to websites or even domains it has never seen before. Second, it shall work on real-world websites, which can be dynamic, complex, and noisy. Most modern websites are dynamic, generating and rendering different content in response to user actions. This necessitates the agent to model each website as a partially-observable environment instead of assuming full knowledge a priori. The agent should also not make strong simplifying assumptions
*Corresponding authors: {deng.595, sun.397, su.809} @osu.edu
(a) Find one-way flights from New York to Toronto.
(b) Book a roundtrip on July 1 from Mumbai to London and vice versa on July 5 for two adults.
(c) Find a flight from Chicago to London on 20 April and return on 23 April.
(d) Find Elon Musk's profile and follow, start notifications and like the latest tweet.
(e) Browse comedy films streaming on Netflix that was released from 1992 to 2007.
(f) Open page to schedule an appointment for car knowledge test.
Figure 1: Sample tasks and all domains featured in MIND2WEB. The array of diversity allows for testing an agent's generalizability across tasks on the same website (a vs. b), similar tasks on different websites (a vs. c), and even to entirely disparate tasks, websites, and domains (d-f).
about the environments but must embrace the full complexity and sometimes noise, e.g., due to sub-optimal website designs. Finally, it shall support diverse and sophisticated interactions with websites. Tasks from users can be highly diverse and take a large number of steps to complete (e.g., the task in Figure 1(b) would take 14 actions). An agent that only supports simple tasks may provide limited value to users.
Building an agent for the web is not entirely new. There have been numerous prior efforts in varied forms. However, none of them meets all the requirements for a generalist agent listed above. Existing work falls short in one to all of the following aspects: 1) Only operating in a limited and pre-specified set of websites [5, 21, 22, 35, 40], 2) making strong simplifying assumptions about the websites [22, 40], and 3) only supporting specific types of tasks [21, 22, 40] and/or requiring tedious step-by-step instructions from users [5, 21, 22, 39]. Meanwhile, LLMs have been shown to excel at grounded language understanding in complex environments with good generalizability and sample efficiency [2, 13, 17, 33]. It is therefore promising to explore LLMs as a candidate solution towards generalist agents for the web. However, there lacks a good dataset that can support the development and evaluation of generalist web agents, which is the focus of this work.
In light of this, we present MIND2WEB, a new dataset with natural language tasks and manually annotated action sequences for developing and evaluating generalist agents for the web. It offers the following unique features:
1. Diverse coverage of domains, websites, and tasks. MIND2WEB boasts a collection of over 2,000 tasks curated from 137 websites that span 31 different domains. This extensive range of tasks and domains not only provides a vast landscape for exploration and learning but also opens up a new level of versatility and complexity, fostering a more comprehensive evaluation of generalist web agents.
2. Use of real-world websites. MIND2WEB replaces the oversimplified simulation environments commonly found in other datasets with an authentic, vibrant, and unpredictable realm of real-world websites. We provide full traces of user interactions, webpage snapshots, and network traffic, making it a rich source of raw, unfiltered, and dynamic data. By doing so, MIND2WEB equips models with the capacity to interact and cope with the complexities and uncertainties of real-world environments, thereby encouraging the development of more robust and adaptive models.
3. A broad spectrum of user interaction patterns. MIND2WEB enables users to engage in sophisticated ways with websites, as opposed to basic operations such as searching, following links, and reading content commonly found in existing work. Users can click, select, and type in any elements on the website, which significantly expands the space of possible tasks. This captures all the common actions users do on websites in real life and promotes the development of agents capable of handling complex tasks.
With the diverse domains and websites, we create challenging out-of-distribution evaluation settings where agents are tested on their generalizability to websites or even entire domains never seen during training. This presents a representative evaluation of generalist agents working on unseen websites.
MIND2WEB enables us to conduct an initial exploration in using LLMs for building generalist web agents. The HTML document of real-world webpages may contain thousands of elements, which makes it infeasible or too costly to be fed into an LLM's context. To address this issue, we propose MINDACT, a two-stage model that involves first using a fine-tuned small LM to filter the web elements and then using an LLM to select from the filtered elements in a multi-choice question answering fashion and predict the corresponding action with the selected element. We show that MINDACT significantly outperforms modeling strategies commonly adopted by prior work and achieves a decent level of generalization. It can also work well with both open-source LLMs like Flan-T5 [10] through fine-tuning and closed-source LLMs like GPT-3.5-turbo and GPT-4 through in-context learning. However, there is still substantial room for further improvement towards generalist agents for the web. Promising future directions include integrating multi-modal information, reinforcement learning with feedback from real websites, and specialized LMs for web understanding and action taking.
2 MIND2WEB Dataset
Unlike existing datasets predominantly constructed within simulated environments [31, 40], our objective is to bridge the gap between simulation and reality so that agents trained on our dataset can work on real-world websites out of the box. To achieve this, our approach for data collection adheres to the following principles. Firstly, instead of recreating websites in simulation, which often leads to oversimplified environments, we engage directly with real-world websites and capture snapshots of these environments. Secondly, we collate a diverse set of websites from varied domains and crowdsourced realistic tasks that cover a wide range of functionalities provided by these websites. Finally, acknowledging the challenge of perfectly replicating the complexity of real-world environments, we strive to capture a comprehensive snapshot of each website and the full interaction trace, to the extent that all the tasks can be seamlessly replayed offline. This supports rich modeling and evaluation approaches, ensuring a robust and practical dataset for research.
2.1 Task Definition
The primary objective of MIND2WEB is for the agent to complete a specific task on the target website through a series of actions. Each instance in our dataset contains three components:
Task description, which outlines the high-level goal of the task. We intentionally avoid low-level, step-by-step instructions, aiming to foster the development of agents that can comprehend and carry out tasks in a more autonomous fashion, rather than merely following prescriptive directives.
Action sequence, which is the sequence of actions required to accomplish the task on the website. Each action in the sequence comprises a (Target Element, Operation) pair. The Target Element is an interactable element on the current webpage, and the Operation refers to the action to be executed on that element. We support three common operations: Click (also including Hover and Press Enter), Type, and Select Option. For Type and Select Option, they also require an additional value as an argument. Actions in a sequence often span multiple webpages of a website.
Action Sequence:
Target Element	Operation
1.	[searchbox] Find	TYPE:
auto repair
2.	[button] Auto Repair	CLICK
3.	[.]textbox] Near	TYPE:
10002
4.	[button] 10002	CLICK
5.	[button] Search	CLICK
6.	[switch] Show BBB Accredited only	CLICK
7.	[svg]	CLICK
8.	[button] Sort By	CLICK
9.	[link] Fast Lane 24 Hour Auto Repair	CLICK
10.	[link] Read Reviews	CLICK
Webpage Snapshots:
Figure 2: A sample data instance of our dataset with the three components. Actions marked in red will result in a transition to a new webpage.
Webpage snapshots, which constitute the environment within which the task is performed. We provide the snapshots in a variety of formats to accommodate different modeling approaches: self-contained MHTML file that includes the raw HTML code of the webpage, DOM snapshot containing the DOM tree along with the layout and style information of the screenshot of the rendered webpage, HAR file that includes all the network traffic for replaying the interaction if needed, and trace file that comprises the complete interaction trace during the task annotation process.
The agent receives the task description in the beginning. At each step, it also receives the current webpage and the history of previous actions. The objective is to accurately predict the subsequent action, which encompasses the target element for interaction and the operation.
Our data collection process consists of four stages: website selection, task proposal, task demonstration, and task verification. Website selection and task verification are done by the authors. For task proposal and demonstration, we develop a sophisticated annotation tool using Playwright  ${}^{2}$  and hire annotators through Amazon Mechanical Turk. Refer to Supplementary for annotation tool details.
Website Selection. We start with 5 top-level domains: Travel, Shopping, Service, Entertainment, and Information, which are subsequently broken down into 31 (secondary) domains. We select websites within each domain based on their popularity in the US, as ranked by similarweb.com. We manually select 3-5 representative websites per domain, resulting in a collection of 137 websites in total.
Task Proposal. We present the annotators with a target website, a concise description of the website, and a few sample tasks associated with it. The annotators are then asked to propose open-ended and realistic tasks based on three criteria: the tasks should be of diverse types, require multiple rounds of interaction, and describe the high-level goal instead of step-by-step instructions. To further stimulate creativity and boost diversity, we use ChatGPT to generate seed tasks by prompting it to test different functionalities of a website. We generate 50 seed tasks per website, of which 10 are randomly sampled and presented to the annotator each time. These seed tasks are mainly for inspiration—annotators are explicitly instructed not to directly use them and we reject task proposals that are highly similar to the seed tasks. All the proposed tasks are further screened by the authors to ensure quality and diversity before entering the demonstration phase.
Task Demonstration. We develop a Playwright-based tool for demonstration (Figure 2). Workers will use the tool to demonstrate how to perform the tasks they have proposed within a web browser. To ensure accuracy, each interaction round is split into two parts: element selection and operation selection. At each step, the worker first selects an element on the webpage by clicking within the browser. They are then asked to confirm the selection and choose the operation to execute on the selected element. Once the task is completed, the worker is given another opportunity to review and modify the task description.
Task Description:
Show me the reviews for the auto repair business closest to 10002.
2https://playwright.dev/
Table 1: Statistics of MIND2WEB compared with existing datasets.
# Dom.	# Env.	Env. Type	Avg. # Elements	# Tasks	Task Info.	Avg. # Actions
MiniWoB++ [22]	-	100	Simplified mobile websites	28	100	Low-level	3.6
WebShop [40]	1	1	Simplified shopping websites	38	12,000 products	High-level	11.3
RUSS [39]	-	22	Real-world websites	801	80	High & low	5.4
PixelHelp [21]	4	4	Mobile apps	-	187	High & low	-
META-GUI [35]	6	11	Mobile apps	79	1,125 dialogues	High-level	4.3
MoTIF [5]	15	125	Mobile apps	188	756	High & Low	4.4
MIND2WEB	5/31	137	Real-world websites	1,135	2,350	High-level	7.3
Task Verification. Lastly, all task demonstrations are verified by the authors to ensure the following: First, all actions are accurately reflected in the task description. The authors will modify the task description if needed to align it with the annotated actions; Second, the recorded actions are correct and clean, with extraneous steps discarded. Finally, the starting and ending points of tasks are consistent, such as excluding actions for closing popup windows, or ending the annotation at the search result page if the task was to find a certain item without clicking on specific items. After verification, we discarded 61 out of the total 2,411 tasks. Among the 2,350 retained tasks, the task description was refined in 390 instances to better correspond with the demonstrated actions, while some extraneous steps were discarded in 187 instances. Overall, the data collection pipeline has been proven effective and produces high-quality data.
2.3 Comparison with Existing Work and Research Challenges
MIND2WEB presents a unique ensemble of research challenges for the development of generalist agents for the web in real-world settings. As shown in Table 1, MIND2WEB distinguishes itself from existing literature in several ways. Firstly, MIND2WEB spans across 137 websites from 31 domains, allowing comprehensive testing of an agent's ability in generalizing across varied environments. Secondly, we utilize real-world websites without manual simplification. Consequently, the included environments exhibit complexity far surpassing that encountered in previous studies, yet better reflecting the intricacy of the modern web. With an average of over 1,000 elements per page embedded within complex DOM structures, how to effectively process such long and highly structured documents presents a significant challenge for modeling. Lastly, we direct the annotators to propose open-ended tasks that explore different functionalities of the website to mimic genuine web usage. Meanwhile, contrary to prior studies [5, 21, 22, 39] that provide step-by-step directives and primarily focus on testing the agent's ability to translate low-level instructions into actions, e.g., "Type New York in the location field, click the search button and choose the tomorrow tab," we opted for the setting where only high-level goals are available, e.g., "What is the weather for New York tomorrow?" This poses a much greater yet realistic planning and grounding challenge for the agent.
3 Method: MINDACT
Employing the data from MIND2WEB, we introduce an exploratory framework, MINDACT, for our task, leveraging the power of LLMs. Raw HTML documents, which could consist of thousands of elements, are either infeasible or cost-prohibitive to be directly fed into LLMs. We propose a two-stage process that synergizes the strength of small and large LMs, as shown in Figure 3. In the first stage, a fine-tuned small LM is used to rank the elements present on a webpage, yielding a small pool of promising candidates. In the second stage, these candidate elements are consolidated to form a representative snippet of the webpage, which is then processed by an LLM to predict the final action, including predicting both the element for interaction and the corresponding operation.
3.1 Candidate Generation with Small LMs
Given the task description, the snapshot of the webpage at step  $t$ , and the actions performed in the preceding  $t - 1$  steps, we treat candidate generation as a ranking task. The task is to select the top-  $k$
Figure 3: The overall pipeline for MINDACT with a small ranking LM for candidate generation, and a large prediction LM for action prediction.
Figure 4: Illustration of the candidate generation module and the templates for constructing task query and candidate representation.
Figure 5: Illustration of action prediction with LLMs.
candidate DOM elements from the webpage that best align with both the task description and the current step. We formulate the task query by concatenating the task description with previous actions. The textual representation of each candidate DOM element is derived from a combination of the element's tag, its textual content, and salient attribute values, as well as the textual representation of its parent and child elements. As shown in Figure 4, we pair each DOM element with the task query and feed it to an encoder-only LM through the cross-encoder architecture [28], yielding a matching score. At training time, we randomly sample negative elements from the webpage, and use the target element as the positive example. The matching score is passed through a sigmoid activation function and optimized with a binary cross entropy loss. At inference time, we score all elements in the webpage and pick the top- $k$  elements with the largest logits as input to the second stage.
3.2 Action Prediction with LLMs
After obtaining the top- $k$  candidates, we utilize the candidate set to prune the webpage snapshot and construct snippets that only include the selected candidates and their neighbours as inputs to an LLM. Recent studies [10, 13] have suggested that training LMs for discrimination rather than generation is more generalizable and sample-efficient for other grounding tasks. Inspired by that, we convert the task of element selection into a multi-choice question answering (QA) problem. Instead of generating the complete target element, the LM is trained to instead select from a list of options. For comparison, we also include a baseline that directly generates the target element based on the provided webpage snippet. In both cases, we directly let the LLM generate the operation, along with the additional value needed for some operations. An example is shown in Figure 5. We incorporate up to 5 candidate elements within each input, together with a None option, and partition the candidate set into several groups. During training, we construct the target sequence using ground-truth actions and fine-tune the model using a left-to-right language modeling objective. During inference, we divide the top- $k$  candidates into multiple clusters of five options. If more than one option is selected after a round, we form new groups with the selected ones. This process repeats until a single element is selected, or all options are rejected by the model, i.e., the model chooses the None option for all groups.
Table 2: Main results. The classification baseline uses DeBERTaB and the generation baseline uses Flan-T5B. For step-wise metrics, we report macro average across tasks. * For GPT-4 we use 50 tasks for each setting with top-10 candidates due to limited budget. See Appendix D.3 for results on the 50 tasks subsets for all methods.
Cross-Task	Cross-Website	Cross-Domain
Ele. Acc	Op. F1	Step SR	SR	Ele. Acc	Op. F1	Step SR	SR	Ele. Acc	Op. F1	Step SR	SR
Classification	26.8	-	-	-	21.6	-	-	-	24.5	-	-	-
Generation	20.2	52.0	17.5	0.0	13.9	44.7	11.0	0.0	14.2	44.7	11.9	0.4
MINDACT
w/ Flan-T5B	43.6	76.8	41.0	4.0	32.1	67.6	29.5	1.7	33.9	67.3	31.6	1.6
w/ Flan-T5L	53.4	75.7	50.3	7.1	39.2	67.1	35.3	1.1	39.7	67.2	37.3	2.7
w/ Flan-T5XL	55.1	75.7	52.0	5.2	42.0	65.2	38.9	5.1	42.1	66.5	39.6	2.9
w/ GPT-3.5	20.3	56.6	17.4	0.8	19.3	48.8	16.2	0.6	21.6	52.8	18.6	1.0
w/ GPT-4*	41.6	60.6	36.2	2.0	35.8	51.1	30.1	2.0	37.1	46.5	26.4	2.0
The diversity of MIND2WEB provides a unique opportunity to evaluate an agent's generalizability at different levels. We seek to understand how well an agent can generalize across domains, websites, and tasks: TestCross-Domain, for which we hold out two top-level domains, Information and Service, with 912 tasks from 73 websites. Here, the model is expected to generalize to an entirely new domain without having seen any websites or tasks associated with that domain during training. TestCross-Website, with 10 websites from each remaining top-level domain, containing 177 tasks. In this setting, the model is never exposed to the test websites during training. However, it has been trained on websites from the same domain and possibly with similar tasks. This setup allows us to assess an agent's ability to adapt to entirely new websites, yet within familiar domains and task contexts. TestCross-Task, where we randomly split  $20\%$  of the remaining data, regardless of domains and websites, resulting in 252 tasks from 69 websites. In this setting, the model has been exposed to webpages from the same website during training and has likely encountered similar tasks. The rest of data is used for training, which contains 1,009 tasks from 73 websites.
4.2 Data Preprocessing and Evaluation
We apply simple heuristics to clean the raw HTML documents, keeping only elements that are visible and carry substantial semantic meaning, as determined by their attributes, textual content, and neighboring elements. This effectively reduces the average number of elements from 1,135 to 580, while still maintaining an overall recall of  $94.7\%$  for the target element in the training data.