SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For evaluation, we first calculate Element Accuracy that compares the selected element with all acceptable elements, and Operation F1 that calculates token-level F1 score for the predicted operation. This is the same as accuracy for Click, but considers the correctness of the input value for Type and Select Option. Each step of the task is evaluated independently with the ground truth action history provided. We then define Step Success Rate and Success Rate (for the whole task). A step is regarded as successful only if both the selected element and the predicted operation are correct. A task is regarded successful only if all steps have succeeded. It is therefore a stringent metric. For step-wise metrics, we report macro average across tasks.
Candidate Generation. We fine-tune DeBERTa [16] as the small LM for candidate generation. As candidate generation requires high efficiency, we use the base version DeBERTa $_B$  with 86M parameters. Overall, it achieves  $88.9\% / 85.3\% / 85.7\%$  Recall@50 on TestCross-Task, TestCross-Website and TestCross-Domain, respectively. We use its top-50 ranking results as the candidate pool for all subsequent experiments.
Action Prediction. We mainly compare against two baselines in Table 2. The first directly uses the candidate generation model (DeBERTa) for element selection, which is similar to existing work [14, 35] that combines an encoder with classification heads. However, such a design cannot
Figure 6: Step success rate per website grouped by the three splits: TestCross-Task, TestCross-Website and TestCross-Domain from left to right. Here we only show websites with more than three test tasks.
benefit from many recent LMs that use an encoder-decoder or decoder-only architecture. It cannot predict actions and the element selection performance is also not competitive, as shown in Table 2. We use Flan-T5 [10] as the backbone for the generation model. The autoregressive generation formulation (Figure 5 top) does not perform well, and even underperforms the classification baseline on element selection despite the larger model size (220M for  $\mathrm{Flan - T5_B}$ ). We observe a substantial gain with MINDACT using the multi-choice QA formulation. The best model achieves  $52.0\%$  step success rate under Cross-Task setting, and  $38.9\% / 39.6\%$  when generalizing to unseen websites and domains. However, the overall task success rate remains low for all models, as the agent often commits at least one error step in most cases.
Three Levels of Generalization. All models perform best on the Cross-Task setting, with over  $10\%$  absolute gap (step SR) on average compared with Cross-Website and Cross-Domain settings, indicating that generalizing to unseen environments is still a major challenge. On the contrary, we note that the performance of Cross-Website and Cross-Domain settings are notably similar, which is also reinforced in Figure 6, where there is no clear distinction in performance across these settings. This suggests that the challenges primarily stem from the diversity in website designs and interaction logic rather than domain specifics. Tasks across domains tend to share common operations, and pretrained LMs may already have the capability to decompose complex tasks at a high level based on commonsense knowledge. Yet, grounding such knowledge into actionable steps in specific and varying environments remains a considerable challenge.
In-context Learning with LLMs. We also experiment with two popular LLMs, GPT-3.5-turbo and GPT-4 [25], through in-context learning. We use the same multiple-choice formulation as MINDACT, and include three demonstration examples for in-context learning. We can see that both models are comparable to the two baselines with only three in-context examples. Note that this is not a fair comparison with the Flan-T5 models, which are fine-tuned on the full training data. We also include the zero-shot results with  $\mathrm{Flan - T5_{XL}}$  in Appendix D.2, but the model fails to perform the task without fine-tuning. Meanwhile, GPT-3.5 only has around  $20\%$  element selection accuracy, despite the superior performance people have observed on other datasets. Further analysis reveals that one possible problem is the model's propensity to select the None option, asserting that the task cannot be finished on the current webpage. This is somewhat accurate since tasks typically necessitate navigation through multiple webpages and performing a series of actions before reaching the final result. This aspect indeed represents the primary difficulty of our task. On the other hand, we observe highly promising outcomes with GPT-4. The performance is on par with the tuned Flan-T5 models under Cross-Website and Cross-Domain settings for element selection, indicating a great potential for developing generalist agents using LLMs. Nevertheless, GPT-4's high operational cost remains a concern. Developing smaller models specialized for the web is an interesting future avenue.
Autonomous Agents for Web and Mobile Applications. Considerable initiatives have been invested in automating web navigation, driven by a vision of facilitating effortless human-web interaction. Yet, previous research has been limited by the types of tasks and websites it can handle, either confined to simplified simulation environments [22, 31, 40], or limited to a narrow set of domains and websites [39, 40]. Recent studies [5, 21, 35] have utilized similar techniques for mobile applications, however, these are often simpler and offer fewer functions compared with full-fledged websites. In contrast, MIND2WEB aims to adapt to a realistic web environment, characterized by its high diversity.
Also related is the research on web automation systems [1, 19]. These technologies often demand programming skills, which can make them less accessible to general users. We aim to equip the web automation system with a natural language interface, thereby reducing the entry barrier significantly.
Large Language Models. In recent years, there has been a surge in the development and application of large language models (LLMs). These models, often encompassing billions of parameters, are pre-trained on massive corpora of text data [3, 44, 45], enabling them to capture intricate linguistic patterns, nuances, and relationships, resulting in unprecedented performance on a wide array of NLP tasks. One of the most noteworthy attributes of LLMs is their few-shot learning capability. Unlike traditional machine learning models that necessitate extensive labeled data for task-specific fine-tuning, LLMs can often perform tasks with minimal task-specific examples. Furthermore, LLMs such as GPT-3 [4] and PaLM [9] have also demonstrated the ability to do in-context learning, where they can adapt to novel tasks by simply providing context within the input prompt, eliminating the need for explicit retraining. In this work, we explore the use of LLMs to build generalist agents on top of MIND2WEB by either tuning medium-sized LMs with only around 1,000 examples, or prompting an LLM such as GPT-4, and have observed promising results.
Grounded Language Understanding. Our work also aligns with the field of grounded language understanding, which aims to map natural language utterances onto executable plans in a target environment [13]. Many studies have centered around environments underpinned by a well-structured schema or ontology, including relational databases [36, 43] and knowledge bases [12, 42], which may not adequately reflect the more heterogeneous conditions in real-world situations. Our work instead grounds natural language in the noisy and schemaless web environment. Our setting is also connected to embodied AI, where an agent, guided by language instructions, carries out tasks in a physical environment [2, 32, 33]. Nonetheless, existing research primarily focuses on a specific setting (e.g., household environments), limiting their diversity. MIND2WEB provides a unique testbed for studying a broad range of grounding challenges in real-world environments.
Tool Learning. Recent developments have underscored LLMs' potential in using a myriad of tools (i.e., taking actions) to augment their capacity [23, 27], including search engine, translator, calculator, etc. Example works include Toolformer [29], ReAct [41], and ToolkenGPT [15]. The creation of recent benchmarks on tool learning [20, 26] further highlights the growing interest in evaluating LLMs' proficiency in tool usage. However, existing research primarily concentrates on short-term tool invocation, neglecting long-term planning. MIND2WEB can bridge this lacuna by necessitating LLMs to take actions within realistic web-browsing environments that demand prolonged decision-making sequences. Furthermore, MIND2WEB may stimulate the development of more advanced tools based on LLMs that interface the web with natural language. These advanced tools could be subsequently employed by another LLM for more challenging problem-solving tasks [11, 30].
6 Limitations and Potential Societal Impact
MIND2WEB is designed to facilitate the development and evaluation of generalist agents for the web. Such agents hold great potential for making the web more accessible and easy to use, especially for individuals who are less familiar with information technology or have disabilities and may struggle to navigate through complex web apps and get overwhelmed by the options available. However, there are still potential concerns and limitations regarding the current data collection, system design and safety for deployment in real world.
Diversity and Representation in Data Collection. Although we strive to choose representative websites covering diverse domains, the present selection predominantly comprises English-language websites primarily used in the U.S. Meanwhile, all our annotators are sourced through the Amazon MTurk platform, which might be biased towards a group that is more proficient in web use. Therefore, the tasks and websites embodied in our dataset may represent only a subset of all potential tasks that can be performed on the web. Bearing this limitation in mind, the design of MIND2WEB and our data collection protocol allow for easy expansion to encompass more tasks and websites. The inclusion of additional websites, potentially from different countries and languages, and tasks from more diverse demographics, such as individuals from different age groups, those traditionally facing web accessibility challenges, and professionals from specific domains like software development, research, law, and more, present exciting directions for future development.
Use of Multimodal Information. Our current approach, MINDACT, models the web environment using only textual context from webpage snapshots. Nevertheless, crucial information can also be gleaned from the visual representation of a rendered webpage. While not currently utilized, we have included complete webpage snapshots in MIND2WEB, enabling rendering of the webpage for visual interpretation. The use of this multimodal information will be a viable prospect for improving model performance.
Modeling of Interaction Dynamics. In MINDACT, we encode each webpage independently at every step, with only the previous actions provided as historical context. However, the changes of the web environment could also provide significant cues for task completion, such as the appearance of a dropdown menu following a button click. Exploring effective ways to model such dynamic environment transformations during interaction could be an essential aspect for developing robust web agents.
Human-Agent Interaction. In the current design of MIND2WEB, the user provides a single description of the task goal up front, and the agent carries out the task from start to finish. In real-world settings, the user may wish to adjust or add task requirements in the middle, or the agent might seek user confirmation for more accurate task understanding. Extending Mind2Web to an interactive or conversational setting, thereby allowing diverse forms of human-agent interactions, could be an interesting future direction.
Evaluation with Offline/Online Environments. Following recent works [5, 35], we evaluate the system with cached offline environments, which allows us to test using snapshots of complex real-world websites. However, a downside to this is that the task will fail immediately if an action was not cached during data collection, potentially leading to false negatives due to the existence of multiple paths for completing the same task. As described in Appendix C.1, we normalize the actions to address equivalent elements within the same page. In addition, we include complete network traffic in the dataset, presenting possibilities for future research to enable some degree of replay and exploration within the cached environment. Given that MIND2WEB faithfully replicates real-world webpages, systems trained on the dataset should be readily transferable to live websites. Conducting end-to-end live evaluation on real websites with human assistance is a very promising direction that is worth exploration.
Safety in Deployment. While the development of general-purpose web agents holds great potential to enhance efficiency, optimize user experiences, and promote web accessibility universally, the accompanying safety considerations for real-world deployment cannot be ignored. These include how to effectively manage sensitive actions like financial transactions, enhancing transparency and interpretability, and keeping users in control during task execution. Additionally, there is the risk of these agents possessing the capability to breach existing security measures such as CAPTCHA and being exploited for malicious activities, such as disseminating false information. Therefore, it is also important for cybersecurity research to consider these potential uses and develop preemptive protective measures.
In this work, we introduced MIND2WEB, the first dataset for developing and evaluating generalist agents for the web. We also proposed MINDACT, an agent that leverages the power of (large) language models for effectively tackling this task. Our work opens up a wide range of promising future directions, including integrating multi-modal information, reinforcement learning with feedback from real websites, and specialized LMs for web understanding and action taking. We hope that MIND2WEB will serve as a valuable platform for the research community to advance towards generalist agents for the web.
The authors would thank colleagues from the OSU NLP group for constructive feedback and all contributors from the Amazon Mechanical Turk platform who participated in our study and assisted in data collection. This research was sponsored in part by NSF OAC 2112606, NSF CAREER #1942980, ARL W911NF2220144 and Ohio Supercomputer Center [6]. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official
policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.
[43] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3911-3921. Association for Computational Linguistics, 2018. doi: 10.18653/v1/d18-1425. URL https://doi.org/10.18653/v1/d18-1425.
[44] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303.18223, 2023. doi: 10.48550/ARXIV.2303.18223. URL https://doi.org/10.48550/arXiv.2303.18223.
[45] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. A comprehensive survey on pretrained foundation models: A history from BERT to chatgpt. CoRR, abs/2302.09419, 2023. doi: 10.48550/ARXIV.2302.09419. URL https://doi.org/10.48550/arXiv.2302.09419.
- Section B: Data Collection Details. Details for crowsourcing and implementation details for the three data collection phases: task proposal, task demonstration, and task verification.
- Section C: Experiment Details. Details for evaluation and model implementation.
- Section D: Additional Results. Results for additional auxiliary experiments.
Following NeurIPS Dataset and Benchmark track guidelines, we have shared the following artifacts:
Artifcat	Link	License
Homepage	https://osu-nlp-group.github.io/Mind2Web/	-
Code Repository	https://github.com/OSU-NLP-Group/Mind2Web	MIT License
Training Data	https://huggingface.co/datasets/osunlp/Mind2Web	CC BY 4.0
Test Data	https://shorturl.at/iGI45 (password “mind2web”)	CC BY 4.0
The authors and the OSU NLP group are committed to ensuring its regular upkeep and updates.
Table 3: Prompt for ChatGPT to generate seed tasks to inspire the annotators during task proposal.
Prompt	We are testing the website
https://aa.com | American Airlines - Airline tickets and low fares at aa.com
Book low fares to destinations around the world and find the latest deals on airline tickets, hotels, car rentals and vacations at aa.com. As an AAdvantage member you earn miles on every trip and everyday spend.
We want to test practical daily tasks that a user would do on the web-site.
Come up with a list of 5 example tasks and try to cover different cases.
Requirements:
- Each example should be a single sentence and not just click one of the elements.
- Don’t give step-by-step instructions or directly mention the element to interact.
- Describe the goal of the task and provide concrete information or constraints.
Use mock-up information (identifier, number, personal information, name, date, attributes, constraints, etc.) to make the task more specific and realistic.
Sample Responses	Retrieve the confirmation number for a reservation made under the name jane smith departing from dallas (dfw) to miami (mia) on january 20th.
Find and book a round-trip flight from new york to london departing on december 15th and returning on december 22nd for under $800.
Find the lowest fare for a one-way flight from new york city (jfk) to los angeles (lax) leaving on august 15th and returning on august 20th, with no layovers exceeding 2 hours.
Rent a car in honolulu for one week starting on august 1st that can fit at least four passengers and has gps navigation included.
Cancel a car rental reservation for a compact car in orlando, florida that was scheduled to be picked up on may 25th at 5 pm without incurring any cancellation fees.
B Data Collection Details
B.1 Crowdsourcing Details
In this study, we employ annotators from the Amazon Mechanical Turk (mturk) platform. The annotators are required to have a minimum of 1,000 approved HITs with an approval rate exceeding  $98\%$  on the platform. We design the compensation with an estimated hourly rate of  $10.10 to respect the minimum wage guidelines in Ohio, United States. Every worker passing our qualification receives a bonus, and we pay$ 0.80 for each approved final task. We do not collect any identifiable private information during the study, and explicitly instruct the annotators to refrain from entering personal or sensitive data into the system. Annotators engage with our annotation tool only within a secure, remote sandbox environment, posing no foreseeable harm. The study complies with the IRB exemption criteria, per the Office of Responsible Research Practices at The Ohio State University. All annotators are presented with a consent form, to which they must agree before participating in the study. To prepare the workers for the task, we provide a comprehensive training document and a video tutorial, followed by a qualification assessment comprising a questionnaire and a series of test demonstrations using our tool. It is noteworthy that the task is divided into two phases: task proposal and task demonstration. The proposal phase comes with a nominal reward, with the majority of the compensation dispensed upon successful completion of the demonstration.
Quality and diversity are ensured through a two-stage review process. The first author reviews all tasks after task proposal and manually select the tasks for demonstration. After task demonstration, a thorough final verification of all collected data is conducted by all authors to authenticate the tasks and recorded actions. Each demonstration is first verified by one of the authors, and uncertain ones are further verified by the first author to reach consensus.
Figure 7: Illustration of our annotation tool, which consists of two side-by-side windows. On the left we provide a dialogue window for the user to control the tool and select operations to take. On the right we provide the browser window for the user to interact with and select web elements.
Figure 8: The overall procedure for task demonstration.
B.2 Task Proposal
We use ChatGPT to generate sample tasks to provide inspiration to the annotators, and the prompt used is shown in Table 3. For each HIT, we ask the annotator to select a website of their interest first. Following this, we present them with ten sample tasks produced by ChatGPT, and request them to propose a maximum of five additional tasks. The annotator is instructed not to directly copy the sample tasks. We manually evaluate all submitted tasks and reject those that demonstrate low quality or are too similar to previously accepted tasks. We set a nominal reward of  $0.05 for each task proposal HIT, and the annotator will receive it no matter whether the tasks are accepted or not. For accepted ones, the annotator will receive the full reward of$ 0.80 on successful demonstration of the task. Once we have collected a total of around 20 tasks for a specific website, we desist from showing it to the user, aiming for a balanced distribution among websites and increased task diversity.
B.3 Task Demonstration
We develop a dedicated annotation tool for task demonstration using Playwright, which allows us to interact with the browser and record user actions. As shown in Figure 7, the tool is composed of two windows. The dialogue window on the left serves as the annotator's control panel for guiding the interaction flow and choosing operations. The browser window on the right is where the annotator navigates the website and selects elements for interaction. Figure 8 shows the overall procedure for task demonstration. The annotator starts by selecting the website and task to be demonstrated. Once selected, the tool will bring up the website in the browser. The annotator is then instructed to explore the website and practice the task. To collect clean actions during actual demonstration, the workers are asked to close pop-up windows during exploration. We also provide anonymous accounts for the workers to use so that no private information is entered. The exploration stage is
<sup>3</sup>https://playwright.dev/
Figure 9: Dialogue window for the Type operation.
Figure 10: Dialogue window for the Select Option operation.