Venue: NAACL 2025

# Tur[k]IngBench: A Challenge Benchmark for Web Agents

 Kevin Xu\({}^{\star}\)  Yeganeh Kordi\({}^{\ddagger}\) Tanay Nayak\({}^{\clubsuit}\) Adii Asija\({}^{\clubsuit}\)

Yizhong Wang\({}^{\circledS}\) Kate Sanders\({}^{\clubsuit}\) Adam Byerly\({}^{\clubsuit}\) Jingyu Zhang\({}^{\clubsuit}\)

Benjamin Van Durme\({}^{\clubsuit}\) Daniel Khashabi\({}^{\clubsuit,}\)1

\({}^{\clubsuit}\)Johns Hopkins University Brown University University University of Washington

Footnote 1: Data and code: [https://turkingbench.github.io/](https://turkingbench.github.io/)

###### Abstract

Can advanced multi-modal models effectively tackle complex web-based tasks? Such tasks are often found on crowdsourcing platforms, where crowdworkers engage in challenging micro-tasks within web-based environments.

Building on this idea, we present TurkingBench, a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. Unlike previous approaches that rely on artificially synthesized web pages, our benchmark uses natural HTML pages originally designed for crowdsourcing workers to perform various annotation tasks. Each task's HTML instructions are instantiated with different values derived from crowdsourcing tasks, creating diverse instances. This benchmark includes 32.2K instances spread across 158 tasks. To support the evaluation of TurkingBench, we have developed a framework that links chatbot responses to actions on web pages (e.g., modifying a text box, or selecting a radio button). We assess the performance of cutting-edge private and open-source models, including language-only and vision-language models (such as GPT4 and InternVL), on this benchmark. Our results show that while these models outperform random chance, there is still significant room for improvement. We hope that this benchmark will drive progress in the evaluation and development of web agents.1

Footnote 1: Data and code: [https://turingbench.github.io/](https://turingbench.github.io/)

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3694–3710

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Significant progress in AI model development has been fueled by large pre-trained language models (LLMs) (Radford et al., 2019; Dubey et al., 2024). However, humans usually engage with visually rich environments, like web-based applications that combine language with visual elements such as images and tables. Notably, many people now spend more time online than in the physical world (Perrin and Kumar, 2019).

Crowdsourcing platforms are a key domain where human workers complete micro-tasks in exchange for rewards. Each day, hundreds of requesters publish tasks that need annotation by workers. These tasks are typically presented as web pages, incorporating stylistic features and structured information (e.g., tables, colors, font sizes) to communicate the tasks effectively. Inspired by humans' ability to solve various tasks through web interfaces (Efrat and Levy, 2020), and with the growing capabilities of multi-modal chatbots (OpenAI, 2023; Anil et al., 2023) that can generate and interact with external tools (Chen et al., 2021; Dubey et al., 2024; Schick et al., 2023), we are motivated to assess their performance on the same web-based crowdsourcing tasks that humans handle daily.

We present TurkingBench, a benchmark for web-grounded tasks. An example task can be seen in Figure 1. The information in these tasks is conveyed through a mix of stylistic elements (such as colors, font sizes, and shapes), structural features (like tables and paragraph headings), and multiple modalities (including text, images, and videos). Effectively interpreting these multi-modal signals is a complex challenge that demands a range of skills, particularly the ability to understand language within rich visual contexts.

The tasks in TurkingBench are drawn from those published on the Amazon Mechanical Turk (AMT) platform, a well-known site where human workers complete micro-tasks for rewards. The benchmark comprises 158 web-grounded tasks (web pages), each averaging 16.8K tokens and containing 15.6 input fields that AI models must label or fill in. Each task can be instantiated with different input values (e.g., varying questions, paragraphs, images). In total, our benchmark includes 36.2K input fields. There are several challenges in solving our proposed task: (i) Multi

modality--tasks require models to "see" or "read".

(ii) Interaction--agents' actions must consider the context of previous decisions on the web pages. (iii) Long context--web pages (HTML, screenshots, etc.) often are quite long.

To facilitate interaction and evaluation of AI systems on these web-based tasks, we developed a Python-based middleware that connects AI models with web pages and also serves as the evaluation framework. This development, led by several students, has taken over a year of dedicated effort to ensure smooth functionality.

We use TurkingBench to benchmark several notable unimodal and multi-modal models for interactive reasoning on web pages (SS4.1). Among our evaluated models, GPT4 performs significantly better than random chance, either through visual actions (click and type) or textual commands that modify the HTML code of each page. Among the open-source models, Qwen2 achieves the highest performance. In both settings, the top-performing models leverage the combination of text and visual modalities, highlighting the advantages of multimodal approaches. Despite these positive gains, GPT4's performs well below the estimated upper bound of the dataset. We analyze model performance based on field type and task length, identifying challenges for future progress.

We chose crowdsourcing tasks for their diverse web interactions, including multimodal reasoning, complex form filling, and sequential dependencies--common in e-commerce, education, and administration. While rooted in this domain, our benchmark's challenges (e.g., diverse inputs, long-context reasoning, and multimodal interactions) apply broadly. Effective page understanding and manipulation remain key challenges for web agents.

Compared to existing web-based benchmarks, TurkingBench fills a critical gap. A key feature of our benchmark is that both the web pages and instructions are naturally sourced from the wild, rather than being artificially constructed. The web pages in TurkingBench were originally designed for human crowd workers, making them more authentic than recent benchmarks that use simplified, engineered web pages Liu et al. (2018); Furuta et al. (2023). Additionally, task instructions in TurkingBench are embedded within the web pages, requiring a deep understanding of the content to solve them. This contrasts with most related benchmarks Deng et al. (2024); Zhou et al. (2023), where instructions are provided as standalone sentences. Finally, because the tasks were initially created for complex annotation on crowdsourcing platforms (e.g., building NLP benchmarks), they inherently carry a high level of complexity. We believe this makes TurkingBench a distinctive platform for evaluating both the comprehension and interaction capabilities of LLMs.

In summary, we introduce TurkingBench, a benchmark for multimodal, interactive web-based tasks, along with an evaluation framework for web interaction. Our analysis of leading models highlights significant gaps, aiming to drive progress in assistive web agents.

Figure 1: An example of a web page (task) from TurkingBench. Each page typically start with a few paragraphs of instructions and examples. Each task features a web page rich in diverse elements: tabular content organization, examples and target instances, color-coding for emphasis, bounding boxes around key instructions, multiple text boxes, images of people, and more. Naturally sourced from the wild for human users, these tasks encompass complex, interactive, and multi-modal reasoning for various web-based activities. Our benchmark motivates the development of web-based agents capable of processing such tasks and interactively filling in elements like radio buttons, check marks, and text boxes. Bottom row (in pink) shows the expected sequence of actions be executed required to to solve the task (detailed in §3.3.1). More examples in Fig.7.

 



## 2 Related Work

Several benchmarks exist for evaluating web-navigation agents Lu et al. (2024); Koh et al. (2024); He et al. (2024); Liu et al. (2024); Pan et al. (2024); Cheng et al. (2023); Zheng et al. (2024b) (see Table 1). These benchmarks offer reproducible model assessments but vary in their construction and coverage of real-world web navigation. Most focus on websites with limited complexity, such as _pre-specified, synthesized_, or _simplified_ web pages Liu et al. (2018); Li et al. (2020); Yao et al. (2022); Furuta et al. (2023). In nearly all existing benchmarks, task definitions are created _after_ collecting the websites, typically through manual crowdsourcing. In contrast, TurkingBench features more natural task instructions, originally intended for human users on crowdsourcing web pages. While this provides more natural task definitions, TurkingBench is limited to tasks involving data annotation that require effective manipulation of complex web pages. Given the complementarity of these benchmarks, the research field is likely to benefit from all.

Unlike our focus on web pages, it is worth noting the benchmarks that utilize other environments for interactive problem solving. For instance, benchmarks that concentrate on mobile operating systems such as Android Li et al. (2020); Toyama et al. (2021); Sun et al. (2022); Burns et al. (2022) or computer applications in operating system environments Trivedi et al. (2024); Xie et al. (2024).

## 3TurkingBench: Benchmarking Web Agents via Multi-Modal Turing Tasks

We discuss the development of TurkingBench. The primary purpose of this benchmark is to provide a standardized _evaluation_ for web-based agents. Additionally, a subset of this data can facilitate model design and development.

The overall development has undergone several rounds of meticulous engineering, led by multiple students, and has taken over a year of dedicated effort to ensure smooth functionality.

### Benchmark Schema

TurkingBench consists of a collection of tasks where each _task_ is a bundle of the following components: (A) A **web template** containing instructions for the task, input variables, and input fields for the outputs. (B) **Input values** to be instantiated for the input variables. (C) Annotated **output labels** provided by crowd workers.

An example is shown in Figure 2. As the example shows, the HTML template contains variables (e.g., ${sys!0}) that are instantiated with input values. Furthermore, the HTML template contains input fields for receiving input values from the web page users. For each field, we have values previously collected from crowd workers that we will use for evaluation.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & MinWoH++ Liu et al. (2018) & CompWoBI Yutura et al. (2021) & RUSS Xu et al. (2021) & WekBepYuo Miao et al. (2021) & MindZWeb Deng et al. (2021) & WebArea Zhou et al. (2021) & TuxkingBench \\  & et al. (2018) & et al. (2021) & (2021) & et al. (2022) & et al. (2024) & (his work) \\ Inputs & language & language commands & customer service queries & shopping queries & language commands English command instructions embedded in queries & web pages \\ The input construction & crowdsourced & manual & mined online & crowdsourced & crowdsourced author-written & collocated from MTurk \\ Environment/domain & simplified & simplified but & real-world & shopping & real-world & variety of crowdsourcing \\ Network usage & convolutional & websites & web pages & website & web pages & websites \\ Natural inputs? & ✗ & ✗ & ✗ & ✗ & ✗ & ✓ \\ Retistic interface? & ✗ & ✗ & ✗ & ✗ & ✗ & ✓ \\ Interaction w/page* & ✓ & ✗ & ✗ & ✓ & ✓ & ✓ \\ Functional correctness? & ✓ & ✗ & ✗ & ✓ & ✓ & ✓ \\ Inter-page navigation? & ✗ & ✓ & ✓ & ✓ & ✓ & ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Notable existing benchmarks for developing and evaluating web-based agents (§2). Existing benchmarks prioritize navigation between web pages but often use simplified or synthetic pages. Our work complements this by focusing on manipulating complex, naturally curated web pages, while excluding inter-page navigation.





### Collecting the web-based tasks

For a benchmark of tasks grounded in web pages to effectively measure progress and generalizability, it needs to be diverse and broadly covered. The majority of the tasks in TurkingBench are sourced from prior crowdsourcing tasks conducted by the authors and their collaborators over the years. We also considered using tasks from the AMT sandbox,2 a testing environment for requesters to prototype their crowdsourcing templates. While this subset could have added more diversity to our data, it would have required additional annotation and cleanup, which we did not pursue. During the selection process, we did not restrict ourselves to any specific task type. Our collected web pages encompass a variety of tasks, and this diversity is a strength. The wide range of tasks captures an array of problems that a generalist agent would encounter, mirroring real-world usability where tasks vary greatly.

Footnote 2: [https://requestersandbox.mturk.com](https://requestersandbox.mturk.com)

We conducted light quality control to ensure the validity of the instructions and their annotations. Here are the steps we took: (1) Several task instructions appeared underdefined (pilot experiments for crowdsourcing tasks), so we eliminated them. (2) Certain tasks were missing data artifacts, such as images or videos. We eliminated these tasks if the missing media file was critical for solving the task. (3) Some tasks did not render properly. A subset used older MTurk design conventions that are no longer supported. We manually revised or eliminated these tasks. (4) Finally, we manually spot-checked the task annotations to ensure the quality was not noisy.

Ultimately, we put together a collection of 158 crowdsourcing tasks (examples in Figure 7). These include but are not limited to language processing tasks (such as paraphrasing, validating factuality, entailment, sentiments, classification, dialogue quality, and rationale generation) and tasks that involve processing images or videos.

StatisticsThe dataset contain a large sum of input-output instances (36.2K) across 158 tasks. Table 2 shows the overall statistics of TurkishBench. Furthermore, Figure 3 shows the distribution of the input fields represented in our benchmark. The most common field is "checkbox," which is expected as it is commonly used when multiple options could be true. Here, we also distinguish between "text" input (a small, single-line box) and "texture" input (a larger, multi-line box for descriptions and paragraphs) fields.

### Programmatic Interaction with Web

We developed a Python library to streamline the interaction of self-supervised models with our tasks and their evaluations (Figure 4). Our library contains various components. To serve our tasks, we use Turkle3 which is an open-source replication of AMT. The content of these tasks are then accessed through the web-browsers that are loaded by Selenium.4

Footnote 3: [https://github.com/hltcoe/turkle](https://github.com/hltcoe/turkle)

Footnote 4: [https://github.com/SeleniumHQ/selenium](https://github.com/SeleniumHQ/selenium)

Furthermore, we have developed a library of actions for accessing the pages' content and making changes to them. This is discussed in SS3.3.1. The starting point of interaction is our evaluation script that loops over the tasks and their instances. This is fleshed out in SS3.5

\begin{table}
\begin{tabular}{l c} \hline \hline Measure & Value \\ \hline \# of tasks & 158 \\ \# of instances & 36.2K \\ avg. \# of fields per task & 15.6 \\ avg. length (subwords) of the tasks & 16.8K \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of dataset statistics.

Figure 4: The interface linking design segments: our web app serves data, which models access programmatically via our evaluation library.

Figure 3: Distribution of input fields.





#### 3.3.1 A Library of Web Actions

To support model design, we have developed a library of "actions" that can execute various operations on a web page. Our set of actions wrap around the API libraries provided by Selenium (c.f. footnote 4) and PyAutoGUI5 these are sophisticated libraries for web manipulation and the amount of sophistication these libraries provide is significantly more than what is here. Therefore, we build our wrappers around these actions to build an action library with enough complexity to cover the dominant majority of tasks in our benchmark.

Footnote 5: [https://github.com/asweigart/pyautogui](https://github.com/asweigart/pyautogui)

The list of the actions included in our library is shown in Table 3. In terms of the modalities of information, a subset of these actions are either concerned with executing tasks in text (HTML) modality or visual modality. There are fewer actions in the visual modality since visually many actions are combinations of clicking and typing. This is unlike the text modality where various element types require slightly different treatment.

In terms the nature of their roles, one can split these actions into three groups: (i) _Modification_ actions allow the models to modify each page's input elements. (ii) _Navigation_ actions allow the models to explore the page, for example, by scrolling up/down or waiting for all elements to be loaded. (iii) _Sensing_ actions allow models to retrieve the latest information about the web page, for example, by fetching its HTML code, taking screenshots of the active page, or around a given element.

While our actions are designed to cover a broad set of web-based interactions, we acknowledge these are only a subset of what is needed for general-purpose navigation. Actions such as "drag-and-drop" are not included here since we did not have any tasks necessitating such actions.

Web interaction as "tool" resolution.The task here can be framed as iterative _tool resolution_ based on task context. A web agent must execute a sequence of executable actions from our library, guided by task instructions (see Figure 1). This aligns with tool-augmented LMs Schick et al. [2023]; Lu et al. [2024]; Mialon et al. [2023]; Qin et al. [2023]; Gong et al. [2023]; Lu et al. [2024], where tools ground LM generations in an environment. Here, our library's actions function as "tools" for achieving specific outcomes, with an LM aware of the tools, task context, and final goal.

### Evaluation Metrics

We devise an evaluation metric that is sensitive to the output type: (i) _Text fields:_ For a given text response, we compute ROUGE metric Lin [2004] against each label and compute their maximum value. (ii) _Radio/select fields:_ We compute an exact match of the predicted label against the majority vote of gold labels. (iii) _Checkbox fields:_ The results of the checkbox selection is a set. Therefore, we quantify this as intersection over union metric between the two sets (set of predicted labels and the gold labels). (iv) _Range fields:_ The range predictions provide real or integer values. We compute absolute (\(\ell_{1}\)) distance between the prediction and the labels, averaged across the labels. This is normalized by the largest value among the gold labels to normalize it to \([0,1]\) range.

The overall score is an average of all the above responses. Given that the field distribution of each type does not vary from one model to another, averaging these different measures of quality is acceptable when comparing different models.

### Evaluation Protocol

Our evaluation iterates through different choices of evaluation tasks (Pseudocode 1). Specifically, the evaluation script supplies the model with the URL where the task is accessible. Additionally, the model has access to our action library, enabling executable actions.

```
Actionlibrary:act Require: The evaluation tasks:tasks functionEvaluate(tasks, act)for\(t\leftarrow\) tasksdo#Solver receives each task, including its URL (t.URL)#andinputfields.tfields.lthen executes actions#throughtheactionsinact(Table 3).GenericSolver(t, act)#Extracthenewvaluesofeachfieldint.fields#Evaluatethepredictedlabelsvsgoldlabels(SS3.4).
```

**Pseudocode 1** The evaluation protocol

\begin{table}
\begin{tabular}{l c} \hline \hline Actions (Modality) & Description \\ \hline modify\_text (T) & modifies the text of input box \\ modify\_checkbox (T) & modifies the selection of checkbox \\ modify\_radio (T) & modifies a radio button \\ nodify\_select (T) & selects an item in a drop-down menu \\ modify\_range (T) & modifies a range input \\ get\_html (T) & fetches the HTML content of a page \\ capture\_screen (V) & fetches the screenshot of a page \\ click (V) & clicks on a given coordinate \\ type (V) & keyboard type at the selected \\ scroll (W/A) & scrolls up or down \\ maximize (W/A) & maximizes the web page \\ \hline \hline \end{tabular}
\end{table}
Table 3: The supported actions in our library.



Based on this evaluation protocol, it should be evident that our benchmark does not require navigation between web pages, as noted in the limitations section. However, our benchmark does require multiple rounds of interaction to solve a given task. This is because nearly all of our tasks involve more than one step (input field), each needing to be addressed in a different round of interaction.

To reduce the difficulty of the task for our models, we provide the list of field names on which we will evaluate model responses. While this arguably simplifies the task, our benchmark remains open for future analysis with open-ended interaction, without any additional guidance on the names of the target fields.

Task splits for measuring generalization to _unseen_ instructions.To accommodate for scenarios that may involve fine-tuning on supervised data, we create different task splits: (i) _Train_: a set of 125 tasks that can be used for supervised model development. (ii) _Test_: a set of 16 tasks that be used for evaluating model quality (names shown in the caption of Fig. 5). (iii) \({\it Test}_{\text{challenge}}\): a set of 17 tasks that are strictly more difficult than the rest of the tasks, because they require more complex combination of actions or other tasks not covered in Table 3. In this version of the work, we do not provide any experiments on \({\it Test}_{\text{challenge}}\) since they require more sophisticated engineering that is beyond the scope of this work and should be addressed in future work.

Screen Size Considerations.When benchmarking multimodal models, the capture_screen action retrieves visual information from the screen. A potential concern is the impact of screen size on benchmarking consistency. To address this, all evaluations using capture_screen run in headless mode, where the browser operates without a monitor and uses predefined, consistent virtual screen dimensions. This eliminates hardware-specific influences, ensuring robust and reproducible results unaffected by screen size variability.

## 4 Evaluating Models in Solving Web-based Tasks in TurkingBench

We benchmark various models with different architectures on TurkingBench. Our goal is to provide reasonable baselines for our proposed benchmark, so we avoid specialized models that rely on specific assumptions or supervision of our data.

Models.As discussed earlier, our models need to consume information on a mix of information modalities. A model can, therefore, consume the instructions as text, image, or a combination of both. We experiment with state-of-the-art proprietary models like GPT4 and Claude-2.1.6 For GPT4 models, we experiment with both the text models and the vision-language variant that is trained in a joint of visio-textual information. We compare the performance of these models with the latest open-source vision language models like LLaVA-1.6 Liu et al. (2023), InternVL2 Chen et al. (2024) and text-only models like Llama-3.1-Instruct Dubey et al. (2024) and Qwen2 Yang et al. (2024). Our evaluation focused on major model families (GPT4, LLaVA1.6, etc.), as our goal was setting baselines with widely-recognized models.

Footnote 6: GPT4/GPT4-V/Claude accessed via APIs in January 2024; GPT4o accessed in August 2024.

We note that, besides these general-purpose models, there are more specialized text-based models for web exploration, either through pre-training on text data Aghajanyan et al. (2021, 2022); Gur et al. (2022), specialized models for analyzing web-based components Huang et al. (2022); Tao et al. (2022); Ebrahimi et al. (2023); Chen et al. (2022), or visual perception of the web Dosovitskiy et al. (2021); Rust et al. (2023); Lee et al. (2022); Kil et al. (2024). We leave such explorations, which are likely to yield better results, to future work.

Encoding the tasks for evaluation.When processing the HTML content of the tasks, we consider two variants: (1) "full" indicates all the HTML content of the entire page. (2) "relevant" indicates a few neighboring lines of HTML adjacent to (above and under) the input field the model is currently solving (further details in SSC.) To reduce the cost of evaluation, we use 20 instances per each of the 20 evaluation tasks. This adds up to \((20\times 20=)400\) web pages evaluated with a total of roughly \(6k\) input fields evaluated.

An oracle for ceiling performance.We implement an oracle baseline that mimics the gold labels for each input (see the logic in 2). 7 This oracle agent ensures the functional correctness of our evaluation. Our end-to-end evaluation process (model output \(\rightarrow\) execution \(\rightarrow\) lookup answers from web pages \(\rightarrow\) evaluation) is significantly more complex than a typical NLP benchmark, and any of these 

steps can fail. Achieving 100% accuracy with the oracle (ensuring functional correctness) took the lead student several months of effort. The oracle baseline essentially replicates the action sequences of crowdworkers for each HTML input element in the task web pages, executing appropriate actions similar to those shown in Figure 1.

We note that the oracle baseline is limited to tasks that do not require complex annotations (such as drag-and-drop) included in \(Test_{\text{challenge}}\) (discussed in our evaluation split SS3.4). A future use case of this oracle baseline can be to obtain granular action sequences for supervising models.

A "do-nothing" model as a lower-bound.We evaluate a trivial baseline that performs no action (no actions (hence, "do-nothing"). As we see in the results, this baseline scores more than zero because on some tasks doing nothing is the right action (e.g., making grammatical corrections to a given text that happens to be grammatical).

### Empirical Results

We present the results of evaluating our main models in Table 4. We experimented with models of varying parameter sizes. For each row, we indicate whether the model input contains text-only (\(\overline{\mathbf{T}}\)) or text-vision (\(\overline{\mathbf{T}}+\mathbf{V}\)). Additionally, the table shows the size of input prompts measured in

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & Model & \# of Parameters (model size) & Modalities & Text encoding & \# of demos & Input len (subwords) & Maximum Input Sequence Length & Score (\%) \\ \hline  & Do-nothing & – & – & – & – & – & – & 7.8 \\ \hline  & Llama3.1-Instruct & 8B & T & relevant & 3 & 1k & 128k & 23.2 \\  & Llama3.2-Instruct & 3B & T & relevant & 7 & 5k & 128k & 25.0 \\  & Llama3.2-Instruct & 3B & T & relevant & 3 & 1k & 128k & 20.8 \\  & Llama3.2-Vision & 11B & T+ V & relevant & 3 & 1.2k & 128k & 9.28 \\  & Llama3.2-Vision & 11B & T+ V & relevant & 7 & 6k & 128k & 17.8 \\  & LLAVA-VL-Vicuna & 7B & T+ V & relevant & 3 & 1.6k & 4096 & 22.7 \\  & LLaVA-VL-Vicuna & 13B & T+ V & relevant & 3 & 1.6k & 4096 & 19.8 \\  & InternVL2 & 40B & T+ V & relevant & 3 & 1.6k & 8192 & 31.0 \\  & InternVL2 & 40B & T+ V & relevant & 7 & 8k & 8192 & 26.1 \\  & InternVL2 & 76B & T+ V & relevant & 3 & 1.6k & 8192 & 30.4 \\  & InternVL2 & 76B & T+ V & relevant & 7 & 8k & 8192 & 29.6 \\  & Qwen2 & 72B & T+ V & relevant & 7 & 6k & 128k & **34.1** \\ \hline  & Claude2.1 & Unknown & T & full & 7 & 82k & 200k & 22.6 \\  & & T & relevant & 1 & 1.2k & 128k & 19.2 \\  & GPT4 & Unknown & T & relevant & 3 & 4k & 128k & 18.4 \\  & & T & relevant & 7 & 5k & 128k & 21.3 \\  & GPT4 & Unknown & T & full & 1 & 21k & 128k & 18.7 \\  & GPT4 & Unknown & T & full & 3 & 28k & 128k & 35.7 \\  & & T & full & 7 & 82k & 128k & 39.3 \\  & & T + V & relevant & 0 & 900 & 128k & 32.4 \\  & GPT4o & Unknown & T+ V & relevant & 3 & 1.6k & 128k & 30.3 \\  & T + V & relevant & 7 & 8k & 128k & 30.19 \\  & GPT4-V & Unknown & T+ V & full & 1 & 22k & 128k & 19.4 \\  & Unknown & T+ V & full & 3 & 30k & 128k & 41.1 \\  & & T+ V & full & 7 & 86k & 128k & **41.7** \\ \hline  & Oracle & – & – & – & – & – & – & 100.0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of language (\(\overline{\mathbf{T}}\)) and vision-language (\(\bar{\mathbf{V}}\)) model performance on TurkingBench. We evaluate two approaches to encoding HTML documents: “full” includes the entire HTML content, while ”relevant” includes only a few neighboring lines of HTML adjacent to the target input field. Due to context window limits, most open-source models could not accommodate more than 3 demos, all using ”relevant” text encoding. Among the combinations explored, GPT4 outperforms most open-source models but remains far from our ceiling performance.

 

GPT-2 subwords (Radford et al., 2019). Wherever possible, we evaluate the models with 7 in-context demonstrations of tasks and desired actions. The open-source vision-language models (\(\overline{\mathbf{T}}+\nabla\)) had much shorter context windows, so we evaluated them using the "relevant" portion of the HTML code for the task or in-context demonstrations.

Despite the remarkable performance of generalist models, they remain far from our ceiling performance.The best performance \(41.7\%\) is obtained by GPT4 vision-language model (\(\overline{\mathbf{T}}+\nabla\)) with access to full text of each task. This is in-line with other recent observations (Zheng et al., 2024a). This configuration also happens to have a extremely large prompt length (86k subwords) and it shows the remarkable ability of this model to exploit long-range dependencies. We note that the gains of the vision model (\(\overline{\mathbf{T}}+\nabla\)) over text-only model (\(\overline{\mathbf{T}}\)) is minimal (\(41.7\) vs. \(39.5\)).

Open-source models rivaling GPT4.Llama3.1-Instruct (8B params) notably outperforms GPT4 (text-only), achieving a score of 25.0% compared to GPT4's 21.3% with 7 demonstrations when "relevant" bits of the input HTML are supplied. This result is particularly impressive given that Llama3.1-Instruct operates with significantly fewer parameters (8B) than GPT4's rumored size. The comparison highlights Llama3.1-Instruct's ability to efficiently leverage the provided context, potentially making it better suited for certain tasks despite its smaller size. Additionally, Qwen2 (72B) demonstrates strong performance, coming very close to GPT4 when evaluated with the "relevant" HTML. Qwen2's score of 34.1% is not too far off from GPT4-V's 41.7%, which underscores the potential and promise of open-source models over proprietary models.

### Analysis: Test vs Vision Models

For one of our top configurations in Table 4 (GPT-4, 7 demonstrations with "full" HTML encoding), we present a breakdown of model performance across tasks in Figure 5. On 9 out of 16 evaluation tasks, the two models exhibit notable performance differences. While it's intuitive to assume tasks with richer interfaces benefit more from visual input, empirical results do not clearly pinpoint which task design aspects drive these differences. Overall, we conclude that **text-only and vision-language models have complementary capabilities**.

### Analysis: Effect of the Number of Demos

As mentioned earlier, we use few-shot prompting of models in order to steer them their predictions. Here we study the effect of the number of demonstrations in model performance. As the results are shown in Figure 6, **the gains of in-context demonstrations quickly plateaus** when the number of demonstrations just above 3 demonstrations.

### Analysis: Performance Per Field Types

We provide extended details on performances per input field type in Table 5. Note that we distinguish between "text" and "textarea" fields because they are defined with different HTML tags (<input

Figure 5: Performance of GPT4 (7 demonstrations) with two different input modalities across different tasks.

Figure 6: Performance with varying number of demos.

 

type="text"> vs. <texturea>). The former is typically used for short inputs, while the latter is for longer, multi-sentence outputs. There is no consistent patter here. Most models tend to have similar weaknesses and strengths, which is not surprising as they're generally trained on similar-ish [pre-training] data. A notable finding was the surprisingly lower performance of GPT4 compared to other models on "text" input fields, which we could not explain. However, since the data is skewed toward checkboxes (Figure 3), GPT4 performs better in the aggregate evaluations. This suggests the potential for hybrid models by combining those that excel at specific input field types.

### Error Analysis

We conducted human annotations to better understand the results in Table 4. This analysis uses predictions from GPT4o (7 demos and "full" encoding) as it is one of the highest-performing models. One of the authors reviewed one instance from each of the 16 tasks, totaling 144 responses to the input fields.

Evaluating GPT4 responsesOur annotator directly evaluated 134 (out of 144) responses that we successfully parsed by our evaluation metric (i.e., no parsing error). The annotator agreed with GPT4o's responses in 60% (80 out of 134) of the cases, reaffirming that our benchmark remains challenging for the models.

Our analysis revealed several recurring issues. One common problem occurred in binary classification tasks, such as determining whether a word is an adjective with a similar meaning to a given word. For example, GPT4o incorrectly classified "discernible" as describing "modernity," despite the lack of synonymy.

Another significant discrepancy was observed in tasks requiring GPT4o to generate both correct and incorrect answers. Instead of providing actual incorrect answers, GPT4o sometimes returned placeholders like //incorrect options//, failing to meet the task's requirements. In some cases, GPT4o also made syntax errors. However, the evaluation focused on the content of the responses rather than their syntactical correctness, so syntax errors did not affect the assessment of answer accuracy. In Table 6 we highlight examples of model predictions for a given UI.

## 5 Conclusion and Future Work

We introduced TurkingBench to facilitate research on web-based agents. Our benchmark focuses on tasks defined within the context of web pages, such as those commonly found on crowdsourcing platforms. It includes a comprehensive Python-based framework that supports both evaluation and model development. We hope this benchmark will drive further advancements in the development of web-based assistant agents.

Future work should explore modeling improvements for better web agents. For instance, a RAGstyle approach could semantically chunk web pages into meaningful segments, a non-trivial task. Another avenue could involve using these agents as CoPilots for human annotation on crowdsourcing platforms, helping workers identify potential mistakes. We consider these extensions somewhat orthogonal to the primary focus of this work and hope future research will address them.

## Limitations

We discuss several limitations:

Intra page navigationOur benchmark does not require navigation between web pages. While interpage navigation is important, it is not the only challenge. Effective understanding and manipulation of each page, which is our focus, remains a significant challenge for web agents. Our benchmark still requires navigation within each page. This is because almost all of our tasks have more than one step (input field), each needed to be solved in a different round of interaction. As shown in our experiments, the benchmark remains challenging for the models. We accept this trade-off to obtain more natural tasks compared to most existing benchmarks.

Simplified evaluationThis is not an inherent limitation of our benchmark but a simplifying assumption in our evaluation. For simplicity, our

\begin{table}
\begin{tabular}{l l c r|r r r r r} \hline Model & \# of params & \# of dimensions & Modality all & checkbox radio & text & textera \\ \hline Llama-3.2 & 3 B & 3 & T & 20.8 & 58.7 & 25.8 & 19.1 & 0.3 \\ Llama-3.2 & 3 B & 7 & T & 20.3 & 56.8 & 22.9 & 22.0 & 2.9 \\ Genuma-2 & 7 B & 3 & T & 20.4 & 54.1 & 28.7 & 15.1 & 2.0 \\ InternVL & 40 B & 3 & T+ V & 30.7 & 59.0 & 58.0 & 46.0 & 12.0 \\ InterWL & 40 B & 7 & T+ V & 26.1 & 85.0 & 28.2 & 48.1 & 7.5 \\ IntramVL & 76 B & 3 & T+ V & 30.4 & 63.7 & 29.7 & 48.4 & 17.1 \\ IntremVL & 76 B & 7 & T+ V & 29.6 & 56.6 & 30.5 & 46.4 & 12.3 \\ GPT4+VL & - & 3 & T+ V & 30.3 & 73.3 & 34.2 & 16.4 & 15.5 \\ \hline \end{tabular}
\end{table}
Table 5: Performance comparison of models across different modalities and input types.

 

evaluation setup provides some guidance by specifying the names of the fields to be modified. Future work should explore variants of our experiments where such hints are not provided to the model.

ScopeOur benchmark's distribution is biased to crowd-sourcing tasks and hence it is not a holistic measure of web-based agents. That being said, _all_ benchmarks carry their own biases. In the context of web-based tasks, all the existing benchmarks (WebShop, Mind2Web, WebArena, etc.) have their own set of assumptions and restrictions about the scope of web-based agents. We view all these efforts to be complementary, each quantifying a unique aspect of intelligent behavior on the web.

## Ethical Considerations

We recognize concerns that this work could lead to technologies replacing crowd workers, who are vital to AI development. This concern is not unique to our work extends to all AI. Our results show that state-of-the-art models are still far from fully automating crowdsourcing tasks, even with simplified evaluation. Therefore, we hope our work enables benevolent use-cases of AI such as enhancing the quality and productivity of crowd workers rather than replacing them.

## Acknowledgements

This project is partly supported by ONR grant (N00014-24-1-2089) and a generous gift from Amazon. The authors would like to thank Chris Callison-Burch, Mark Dredze, Karthik Narasimhan, Nikhil Sharma, Owen Bianchi, and Elizabeth Salesky for their constructive discussions. GPU machines for conducting experiments are provided by the ARCH (Rockfish) cluster ([https://www.arch.jhu.edu](https://www.arch.jhu.edu)).

## References

* A. Aghajanyan et al. (2021)HTLM: hyper-text Pre-training and prompting of language models. In International Conference on Learning Representations, Cited by: SS3.
* A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, et al. (2022)CMM3: a causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520. Cited by: SS1.
* J. Allen, N. Chambers, G. Ferguson, L. Galescu, H. Jung, M. Swift, and W. Taysom (2007)Plow: a collaborative task learning agent. In AAAI, Vol. 7, pp. 1514-1519. External Links: Document Cited by: SS1.
* R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Socitut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. (2023)Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: SS1.
* C. Bai, X. Zang, Y. Xu, S. Sunkara, A. Rastogi, J. Chen, et al. (2021)UIBert: learning generic multimodal representations for ui understanding. In International Joint Conferences on Artificial Intelligence, IJCAI, Cited by: SS1.
* A. Bordes, X. Glorot, J. Weston, and Y. Bengio (2012)Joint learning of words and meaning representations for open-text semantic parsing. The International Conference on Artificial Intelligence and Statistics, AISTATS. Cited by: SS1.
* A. Burns, D. Arsan, S. Agrawal, R. Kumar, K. Saenko, and B. A. Plummer (2022)A dataset for interactive vision-language navigation with unknown command feasibility. In The European Conference on Computer Vision, pp. 312-328. Cited by: SS1.
* J. Chen, T. Lv, E. Cui, C. Zhang, and F. Wei (2022)XDoc: unified pre-training for cross-Format document understanding. In Conference on Empirical Methods in Natural Language Processing, EMNLP, Cited by: SS1.
* M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, and M. Petrov at al. (2021)Evaluating large language models trained on code. PreprintarXiv:2107.03374. Cited by: SS1.
* Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024)InternVL: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 24185-24198. Cited by: SS1.
* C. Cheng, A. Kolobov, D. Misra, A. Nie, and A. Swaminathan (2023)LLF-bench: benchmark for interactive learning from language feedback. arXiv preprint arXiv:2312.06853. Cited by: SS1.
* J. Clarke, D. Goldwasser, M. Chang, and D. Roth (2010)Driving semantic parsing from the world's response. In SIGINL Conference on Natural Language Learning, CoNLL, Cited by: SS1.
* D. Das, N. Schneider, D. Chen, and N. A. Smith (2010)Probabilistic frame-semantic parsing. In Conference of the North American Chapter of the Association for Computational Linguistics, NAACL, pp. 948-956.

 

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2Web: towards a generalist agent for the web. In _Advances in Neural Information Processing Systems_ (NeurIPS), volume 36.
* Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In _International Conference on Learning Representations_ (ICLR).
* Dubey et al. [2024] Abhimanyu Dubey, Abhinav Jauhiri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Ebrahimi et al. [2023] Sayna Ebrahimi, Sercan O Arik, and Tomas Pfister. 2023. Test-time adaptation for visual document understanding. In _Transactions on Machine Learning Research_ (TMLR).
* Efrat and Levy [2020] Avia Efrat and Omer Levy. 2020. The Turking Test: Can Language Models Understand Instructions? _arXiv preprint arXiv:2010.11982_.
* Furuta et al. [2023] Hiroki Furuta, Yutaka Matsuo, Aleksandra Faust, and Izzeddin Gur. 2023. Language model agents suffer from compositional generalization in web automation. _arXiv preprint arXiv:2311.18751_.
* Gong et al. [2023] Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetti Terzopoulos, Song-Chun Zhu, et al. 2023. Arnold: A benchmark for language-grounded task learning with continuous states in realistic 3d scenes. _arXiv preprint arXiv:2304.04321_.
* Gur et al. [2023] Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafai Safdiari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. _arXiv preprint arXiv:2307.12856_.
* Gur et al. [2022] Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Sadari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. 2022. Understanding html with large language models.
* Gur et al. [2018] Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. 2018. Learning to navigate the web. In _International Conference on Learning Representations_ (ICLR).
* He et al. [2024] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvyager: Building an end-to-end web agent with large multimodal models. _arXiv preprint arXiv:2401.13919_.
* Huang et al. [2022] Yunpan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. _arXiv preprint arXiv:2204.08387_.
* Humphreys et al. [2022] Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachith Chapaparia, Alistait Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. 2022. A data-driven approach for learning to control computers. In _International Conference on Machine Learning_ (ICML), pages 9466-9482.
* Kil et al. [2024] Jihyung Kil, Chan Lee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei-Lun Chao. 2024. Dual-view visual contextualization for web navigation. In _IEEE Conference on Computer Vision and Pattern Recognition_ (CVPR), pages 14445-14454.
* Kim et al. [2023] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. _arXiv preprint arXiv:2303.17491_.
* Koh et al. [2024] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. VisualWeb Arena: Evaluating multimodal agents on realistic visual web tasks. _arXiv preprint arXiv:2401.13649_.
* Ku et al. [2020]Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In _Conference on Empirical Methods in Natural Language Processing_ (EMNLP).
* Lee et al. [2022] Kenton Lee, Mandar Joshi, Julia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandellwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2022. PixExtruct: Screenshot parsing as pretraining for visual language understanding.
* Li and Li [2022] Gang Li and Yang Li. 2022. Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus. _arXiv preprint arXiv:2209.14927_.
* Li et al. [2022] Tao Li, Gang Li, Jingjiie Zheng, Purple Wang, and Yang Li. 2022. MUG: Interactive Multimodal Grounding on User Interfaces. _arXiv preprint arXiv:2209.15099_.
* Li et al. [2020] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Mapping natural language instructions to mobile ui action sequences. In _Annual Meeting of the Association for Computational Linguistics_ (ACL).
* Li et al. [2021] Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani, and Alexey Gritsenko. 2021. VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling. _arXiv preprint arXiv:2112.05692_.
* Lin [2004] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In _ACL Workshop on Text Summarization Branches Out_.
* Liu et al. [2018] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. In _International Conference on Learning Representations (ICLR)_ 

* Liu et al. (2023a) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. In _NeurIPS_.
* Liu et al. (2023b) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xunanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023b. Agentbench: Evaluating LLMs as agents. _arXiv preprint arXiv:2308.03688_.
* Liu et al. (2024) Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long long, Yifan Xu, Xinvan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. 2024. Visualagentbench: Towards large multimodal models as visual foundation agents. _arXiv preprint arXiv:2408.06327_.
* Lu et al. (2024) Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. 2024. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. _arXiv preprint arXiv:2408.04682_.
* Lu et al. (2024) Xing Han Lu, Zdenek Kasner, and Siva Reddy. 2024. Weblinux: Real-world website navigation with multi-turn dialogue. _arXiv preprint arXiv:2402.05930_.
* Lu et al. (2024)Yining Lu, Haoping Yu, and Daniel Khashabi. 2024. GEAR: augmenting language models with generalizable and efficient tool resolution. In _Conference of the European Chapter of the Association for Computational Linguistics_ (EACL).
* Mialon et al. (2023) Gregoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmparantis, Ram Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023. Augmented language models: a survey. _arXiv preprint arXiv:2302.07842_.
* Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. WebGPT: Browser-assisted questionanswering with human feedback. _arXiv preprint arXiv:2112.0932_.
* OpenAI (2023) OpenAI. 2023. GPT-4 Technical Report. _Preprint_, arXiv:2303.08774.
* Pan et al. (2024) Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. 2024. WebCanvas: benchmarking web agents in online environments. _arXiv preprint arXiv:2406.12373_.
* Perrin and Kumar (2019) Andrew Perrin and Madhu Kumar. 2019. About three-in-ten us adults say they are 'almost constantly'online.
* Qin et al. (2023) Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023. Tool learning with foundation models. _arXiv preprint arXiv:2304.08354_.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_.
* Rust et al. (2023) Phillip Rust, Jonas F Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lloneux, and Desmond Elliott. 2023. Language modelling with pixels. In _International Conference on Learning Representations_ (ICLR).
* Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. ToolFormer: language models can teach themselves to use tools. _arXiv preprint arXiv:2302.04761_.
* Sridhar et al. (2023) Abishek Sridhar, Robert Lo, Frank F Xu, Hao Zhu, and Shuyan Zhou. 2023. Hierarchical prompting assists large language model on web navigation. In _Conference on Empirical Methods in Natural Language Processing_ (EMNLP)_Findings_.
* Sun et al. (2022) Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022. META-QUI: Towards Multi-modal Conversational Agents on Mobile GUI. _arXiv preprint arXiv:2205.11029_.
* Tao et al. (2023) Heyi Tao, Sethuraman TV, Michal Shlapentokh-Rothman, Derek Hoiem, and Heng Ji. 2023. Website: Web interface control and sequential exploration with large language models. _arXiv preprint arXiv:2310.16042_.
* Tao et al. (2022) Song Tao, Zijian Wang, Tiantian Fan, Canjie Luo, and Can Huang. 2022. Knowing where and what: Unified word block pretraining for document understanding. _arXiv preprint arXiv:2207.13979_.
* Toyama et al. (2021) Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. 2021. AndroidEnv: A reinforcement learning platform for android. _arXiv preprint arXiv:2105.13231_.
* Trivedi et al. (2024) Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. AppWorld: a controllable world of apps and people for benchmarking interactive coding agents. _arXiv preprint arXiv:2407.18901_.
* Xie et al. (2024) Tianbao Xie, Danyang Zhang, Jiaxuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. 2024. OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments. _arXiv preprint arXiv:2404.07972_.
* Xu et al. (2021) Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. 2021. Grounding open-domain instructions to automate web support tasks. In _Conference of the North American Chapter of the Association for Computational Linguistics_ (NAACL), pages 1022-1032 

An Yang, Baosong Yang, Binyuan Hui, et al. 2024. Qwen2 technical report. _arXiv preprint arXiv:2407.10671_.
* Yao et al. (2022) Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards scalable real-world web interaction with grounded language agents. _Advances in Neural Information Processing Systems_ (NeurIPS), 35:20744-20757.
* Zheng et al. (2024a) Boyuan Zheng, Boyu Gou, Jhyung Kil, Huan Sun, and Yu Su. 2024a. GPT-4V (ision) is a Generalist Web Agent, if Grounded. In _International Conference on Machine Learning_ (ICML).
* Zheng et al. (2024b) Huaixiu Steven Zheng, Swaroo Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. 2024b. Natural plan: Benchmarking llms on natural language planning. _arXiv preprint arXiv:2406.04520_.
* Zhou et al. (2023) Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. WebArena: a realistic web environment for building autonomous agents. _arXiv preprint arXiv:2307.13854_.

 



## Appendix A Additional Related Work

Here we discuss other related work that did not fit in the main text.

**Multi-modal reasoning tasks.**TurkingBench is also related to efforts in multi-modal interactive environments (Gur et al., 2018; Ku et al., 2020; Li et al., 2020, 2022; Li and Li, 2022; Sun et al., 2022; Li et al., 2021; Bai et al., 2021). However, these often feature simple instructions that are only a few sentences long, unlike our more extensive instructions embedded within web pages.

**Web-based agents.** The concept of intelligent automated assistant agents collaborating with humans to complete tasks has been around for some time Allen et al. (2007) and can be seen as an extension of early work on semantic parsing Das et al. (2010); Clarke et al. (2010); Bordes et al. (2012); Gur et al. (2022). Recent literature has explored various forms of supervision, including behavior cloning of actions Gur et al. (2023), reinforcement learning Liu et al. (2018); Nakano et al. (2021); Humphreys et al. (2022); Liu et al. (2023b), and in-context learning (ICL) Kim et al. (2023); Tao et al. (2023); Sridhar et al. (2023). Our work focuses on introducing a new benchmark, providing ICL baselines, and leaving the exploration of more sophisticated models for future research.

## Appendix B Pseudocode for the oracle baseline

```
Requiring: Action library: act Require: The target fields to be modified: fields Require: The gold labels for each field: labels functionOracleSolver(fields, labels) for\(f\leftarrow\)fieldsdo act.Init_llooded(\(f\)) act.scrollo(\(f\)) \(\ell\leftarrow\texttt{labels}(f)\) switch\(f\)>per do case text: Execute act.modify_text(\(f\),\(\ell\)) case radio: Execute act.modify_radio(\(f\),\(\ell\)) case select: Execute act.modify_select(\(f\),\(\ell\)) case range: Execute act.modify_range(\(f\),\(\ell\))
```

**Pseudocode 2** The oracle baseline

We detail the procedure for retrieving "relevant" neighboring HTML, as discussed in SS4 under "Encoding the tasks for evaluation." The "relevant" HTML encoding is implemented by extracting and returning a subset of HTML adjacent to (above and under) the specific input field on a webpage.Specifically: (1) We retrieve the full HTML content of the target elements' grandparents elements. This potentially contains a lot of content. (2.) We split this code into html elements (split based on occurrence of "?") (3) We select 15 elements before and 30 elements below the target element. The original function is accessible in our public code base.



* [14] M. C. Bender, A. Krizhevsky, and D. Vedalov. Learning deep neural networks with convolutional layers: a review of machine learning techniques. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 235-238, 2016.
* [15] J. L. Friedman, R. G. Olshenkov, and I. Goodfellow. Imagenet classification by jointly training data from tiny images. In _Advances in Neural Information Processing Systems (NIPS'97)_, volume 1, pages 1-10, 2006.
* [16] S. Fidler, P. Dollar, and R. Girshick. Generative adversarial nets for large scale image recognition. In _International conference on computer vision and pattern recognition_, pages 135-140, 2016.
* [17] T. H. Lee, Y. Bengio, and Z. Chen. Deep residual learning for large scale image recognition. In _Advances in neural information processing systems_, pages 135-146, 2016.
* [18] W. Li, X. Zhang, and Q. Weinberger. Unsupervised learning for large scale image recognition using deep belief networks. In _IEEE International Conference on Acoustics and Speech (ICASSP)_, pages 1-10, 2016.

[MISSING_PAGE_POST]

 



**Category: Both automatic and human evaluations classified the action as _incorrect_ (true negative).

**Explanation: It is perfectly legal to do carry your pet in public, even as a status symbol.**

**Category: Both automatic and human evaluations classified the action as _incorrect_ (true negative).

How much cultural pressure do you (or those you know) feel about it's okay to carry your pet in public as a status symbol ?**

\begin{tabular}{l



**Category:** Automatic evaluation rated the action as _correct_ even though human evaluations rated it _incorrect_ (false positive).

\begin{table} \end{table}
Table 6: Examples predictions of GPT4-v which is the best performing model in Table 4. For each example, we also highlight the outcome of automatic vs human evaluation.