WebDS: An End-to-End Benchmark for Web-based Data Science

## Abstract

Abstract A large portion of real-world data science tasks are complex and require multi-hop web-based interactions: finding appropriate data available on the internet, synthesizing real-time data of various modalities from different locations, and producing summarized analyses. Existing web benchmarks often focus on simplistic interactions, such as form submissions or e-commerce transactions, and often do not require diverse tool-using capabilities required for web based data science. Conversely, traditional data science benchmarks typically concentrate on static, often textually bound datasets and do not assess end-to-end workflows that encompass data acquisition, cleaning, analysis, and insight generation. In response, we introduce WebDS, the first end-to-end web-based data science benchmark. It comprises 870 web-based data science tasks across 29 diverse websites from structured government data portals to unstructured news media, challenging agents to perform complex, multi-step operations requiring the use of tools and heterogeneous data formats that better reflect the realities of modern data analytics. Evaluations of current SOTA LLM agents indicate significant performance gaps in accomplishing these tasks. For instance, Browser Use, which accomplishes 80 % 80\% 80 % of tasks on Web Voyager, successfully completes only 15 % 15\% 15 % of tasks in WebDS, which our analysis suggests is due to new failure modes like poor information grounding, repetitive behavior and shortcut-taking that agents performing WebDS’ tasks display. By providing a more robust and realistic testing ground, WebDS sets the stage for significant advances in the development of practically useful LLM-based data science. † † footnotetext: * First Author † † footnotetext: † \dagger † Second Author

## 1 Introduction

The web offers a vast repository of knowledge, making automated web traversal a powerful tool for enhancing information retrieval, analytical workflows, and decision-making. Large language models (LLMs) have unlocked new frontiers in knowledge collection, enabling them to interact with websites, synthesize insights, and execute complex tasks with minimal human intervention hong2024data(^1^11For e.g, see recent advances like Deep Research tools by OpenAI (https://openai.com/index/introducing-deep-research/) and Google Deepmind). These agents hold significant promise for data science applications, facilitating autonomous data collection, synthesis, visualization, and predictive analytics peasley2025leveraging. By providing an intuitive natural language interface, LLM-powered agents can democratize access to advanced data analysis, making AI-driven insights more accessible across expertise levels jansen2025leveraging.

Yet, extracting deep insights from websites is not trivial. Modern webpages are complex and dynamic, often containing scattered, unstructured, and multimodal data spread across multiple layers of interactivity. Navigating these environments autonomously presents significant challenges, as web agents must contend with varying page structures, access restrictions, dynamically generated content, and inconsistent data formats. Additionally, effectively synthesizing and analyzing retrieved information requires robust reasoning capabilities and contextual understanding.

While web agents have shown promise in certain settings, particularly in following natural language instructions to accomplish simple tasks such as writing Reddit posts or buying items zheng2024gpt, prior evaluations narrowly targeted web browsing only, without assessing agents on their ability to manipulate and understand data webarena; pan2024webcanvas. Existing web agent benchmarks face trade-offs limiting their long-term usefulness and precision. WebVoyager he2024webvoyagerbuildingendtoendweb, for instance, uses live sites that change over time, hindering reproducibility. Others li2020mapping; deng2024mind2web; webarena rely on coarse metrics that miss finer agent behaviors. Meanwhile, existing data science benchmarks primarily emphasize data manipulation within code-based environments or structured spreadsheets 2502.16395v1, or focus on querying unstructured data from static databases rajpurkar2016squad. Yet in realistic workflows, data scientists and other knowledge workers often begin by browsing the web, navigating across multiple websites, and synthesizing information from heterogeneous sources. Thus, browsing, searching and synthesizing are a fundamental – yet often overlooked –components of real-world web-based data science tasks EnterpriseDataAnalysis; sun2022supporting.

To address the limitations of existing web agent and data science benchmarks, we introduce WebDS, a novel benchmark designed to reflect the full data science workflow—from data acquisition via web browsing to downstream analysis, visualization, reasoning, and reporting. We find that these tasks pose a significantly more challenging testing ground for leading models: for example, while GPT-4o with Browser Use obtains an $81.1\%$ success rate on WebVoyager browser_use2024 it only obtains $13.2\%$ on WebDS. Furthermore, surprisingly, we find that increasing model-capacity does not necessarily lead to increased performance: GPT-4o performs similarly to GPT-4o-mini and Qwen2.5-72b. An error analysis (Section [7](https://arxiv.org/html/2508.01222v1#S7)) shows significant, novel failure modes for all models: poor information grounding (due to the numerical nature of data science tasks), repetitive behavior and shortcut-taking (due to the multi-hop nature of web-based data science). In sum, WebDS introduces a significantly more challenging, real-world benchmark that represents a next frontier for model development. By bridging the gap between web interaction and data science capabilities, we hope that our work grounds the community and pushes us towards developing novel end-to-end AI skills.

Our contributions are threefold:

- 1.
Comprehensive Task Suite We design a benchmark comprising of 870 tasks spanning a wider range of data types, modalities, and domains than any prior benchmark across 29 websites and 10 domains which we discuss more in section [4](https://arxiv.org/html/2508.01222v1#S4) and [3](https://arxiv.org/html/2508.01222v1#S3). These tasks require not only analytical reasoning but also interaction with diverse tools and and interfaces. For example, "Is there an optimal level of healthcare spending to maximize life expectancy, excluding outliers? Write a policy brief to a think tank and a separate document detailing the mathematical analysis.” Solving this task entails identifying relevant data (e.g., from sites like Our World in Data), selecting and applying appropriate analytical techniques (such as nonlinear optimization), and translating technical findings into accessible, domain-specific insights.
- 2.
Realistic End-to-End Evaluation WebDS is the first benchmark to assess the full data science pipeline, simulating end-to-end workflows. Tasks begin with autonomous web browsing for relevant data, followed by analysis and or visualization, and culminate in generating well-reasoned, context-aware outputs.
- 3.
Reproducible and Fine Grained Evaluation Framework. To ensure experimental validity and comparability, we provide a containerized benchmark environment with controlled versions of websites, enabling robust replication. In contrast to prior work that relied on live web content or overly static settings, WebDS strikes a balance between reproducibility and task realism. We also introduce fine-grained metrics that capture subtask completion, tool usage accuracy, data validity, reasoning quality, and report fidelity—allowing for nuanced assessment of agent capabilities. Finally, we also include a diverse mix of end-to-end task and modular skill-specific test, enabling both holistic evaluation and pointed evaluation for targeted diagnostics.

## 2 Related Work

#### Data Analysis Benchmarks

Benchmarks such as SQuAD rajpurkar2016squad and HotpotQA yang2018hotpotqa have played critical roles in evaluating language models’ reasoning over structured and semi-structured data, providing passage-based and multi-hop question-answering tasks, respectively. Further, Wikitable pasupat2015compositional offers a challenge in executing compositional logical queries over semi-structured tables, testing models’ ability to generalize across complex tabular reasoning tasks. Recently, InfiAgent-DABench hu2024infiagent emerged as the first specialized benchmark for data analysis agents, comprising 311 tasks across 55 CSV files with automated evaluation. However, these benchmarks’ focus on structured datasets limits assessment of real-world iterative workflows: data science workflows often heavily involve finding data (e.g. through web navigation), using tools, and analyzing unstructured data 2502.16395v1; rajpurkar2016squad.

#### Web Agent Benchmarks

Evaluating LLM agents on high-level natural language tasks has traditionally relied on surface-form comparisons of action sequences, as seen in VirtualHome puig2018virtualhome and Mind2Web deng2024mind2web, but these fail to capture functional correctness. WebArena improves on this by evaluating functional correctness but has major limitations, including a lack of scoring granularity and rigid state-based evaluation that is not robust to minor variations webarena. Recent approaches use LLMs like GPT-4o to assess task completion pan2024autonomous. However, these works rely only on final screenshots, making the evaluation binary and incomplete. Lastly, methods like WebVoyager consider the last 15 screenshots, greatly strengthening evaluation, but only gives a binary ’SUCCESSFUL / UNSUCCESSFUL’ result, and is unable to account for full trajectory in complex tasks he2024webvoyagerbuildingendtoendweb. Our work avoids this limitation by assessing the full trajectory, and allowing for granular analysis, which is especially important for our tasks which are multihop, multisite and generally involve more steps.

## 3 Benchmark Design

Figure: Figure 1: Example of the task, “I really like the songs ‘Slow Dancing in a Burning Room’ and the Korean song ‘Weather and You’. Which artist has released more songs on average between 2022–2026?” (Subset of full action-sequence shown.) Note: task was selected to save space while still illustrating multi-site attribute
Refer to caption: Task_Demo_Flow.png

In this section, we describe WebDS, the first web-based benchmark to focus on the full data science workflow. Our main objective is to produce a benchmark that is useful, comprehensive, reproducible, and granular. To ground the design of our benchmark and formulate task definitions, we conducted 8 interviews with journalists, data scientists, and other domain experts to identify recurring data-related needs. From these interviews, we derived two broad categories of tasks: those that (1) culminate in a downstream product and (2) those that address a key analytical question. Within these categories, tasks could involve a variety of data modalities and attributes, including: structured data, unstructured textual and non-textual data, multi-hop reasoning, and multi-website data integration, all of which are discussed more in section [4](https://arxiv.org/html/2508.01222v1#S4). In light of these goals we have selected a set of 29 websites that cover 10 high-stakes data-heavy domains, and wrote by hand 870 tasks for LLM agents in these domains.

#### Domains

We chose 29 websites covering 10 high-stake domains, as shown in [7](https://arxiv.org/html/2508.01222v1#Sx1.T7). We selected websites that included various diverse data formats, including table-structured data and downloadable CSVs as well as unstructured data such as text and graphics. Our selection criteria were: (1) websites had to be public, (2) they had to contain data and be used commonly in different kinds of data analysis and (3) they had to have a unique representation of data not found in the rest of our benchmark, or only found in one other website. For these reasons, high scores on our benchmark require an agent to be adept at reasoning and tool usage between different modalities and data representations.

**Table 1: Comparison of dataset features across benchmarks. WebDS is the only benchmark that has tasks spanning Multihop, Structured, Unstructured, Web Nav, QA, Multisite. Actions, and Tool-Use.**
| Dataset | Multihop | Structured | Unstructured | Web Nav | QA | Multisite | Actions | Tool-Use |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SQuAD rajpurkar2016squad | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Wikitable pasupat2015compositional | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| HotpotQA yang2018hotpotqa | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| WebVoyager he2024webvoyagerbuildingendtoendweb | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ |
| WebArena webarena | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| GAIA mialon2023gaia | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| WebWalker wu2025webwalker | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| AssistantBench yoran2024assistantbench | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ |
| WebDS (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

#### Action Space

Unlike prior work webarena, we do not define a fixed action space $\mathcal{A}$, leaving this choice to future developers of web agents. This decision reflects the current landscape, where widely adopted action space representations already exist in frameworks such as BrowserGym workarena2024 and WebArena. Moreover, as a data-science benchmark, imposing strict constraints on the action space would risk limiting future research directions. Researchers may, for example, wish to enable or disable tools such as Python, Bash scripts, MATLAB, and others, depending on their specific needs (as explored in trevino2025benchmarking).

#### Observation Space

Our observation space is defined similarly to existing web benchmarks and consists of HTML, DOM, Screen Shots, and/or AXTrees. A key addition however, is that data can be downloaded and many tasks necessitate the successful downloading and manipulation of data to complete, by default existing agents have loaded into the context of the LLM and the LLM therefore also observes, JSON, CSV, etc.

#### Reproducibility

To ensure consistent and reliable evaluation across runs and models, WebDS provides a fully containerized environment using Docker. Each of the 29 websites in the benchmark is hosted locally within a Docker container, preserving the exact state of the webpages. This helps to eliminate the variability that can arise from dynamic web content, changes to site layout, or server-side updates. We also improved evaluation robustness by addressing language model variability and verified through testing that evaluation scores exhibit minimal to no fluctuation (see Section [5.1](https://arxiv.org/html/2508.01222v1#S5.SS1.SSS0.Px3)).

#### Evaluation Granularity

Although prior benchmarks have provided some granularity in evaluating AI agent performance by producing different scores segmented along difficulties or websites, we are the first to provide evaluation scores along three different dimensions:

- 1.
Domain-wise, along different professional specializations (e.g., performance on demographics-related websites, sports-related websites, etc.)
- 2.
Attribute-wise, along different types of challenges for LLM agents which we call “task attributes” (e.g., multihop, action-based, etc. see more in section [4](https://arxiv.org/html/2508.01222v1#S4))
- 3.
Difficulty-wise, along easy, medium, or hard task complexity

## 4 Benchmark Tasks

### 4.1 Task Generation

We develop a qualitative definition of a web data science task, introduced in Section [4.2](https://arxiv.org/html/2508.01222v1#S4.SS2). We identified 100 candidate websites that met this definition and were freely accessible for use in our benchmark. From this initial pool, we selected a final set of 29 websites to ensure broad coverage of data modalities, enable the construction of multi-site tasks, and represent a diverse range of task attributes. We then grouped these websites into domains with similar content and structure, as shown in Figure [2](https://arxiv.org/html/2508.01222v1#S4.F2).

Using this curated set of websites, we wrote a total of 870 tasks, a simple example of which can be seen in figure [1](https://arxiv.org/html/2508.01222v1#S3.F1). In authoring the tasks, we aimed to achieve a balanced distribution of difficulty levels (easy, medium, and hard), comprehensive coverage of the identified attributes, and strict adherence to our task definition. Task writing emphasized diversity, the inclusion of verifiable ground truths, and the ability to evaluate a wide spectrum of real-world data science capabilities. We find that there is substantial diversity in task difficulty: some tasks can be completed through straightforward web searches, while others require synthesizing information from heterogeneous sources and applying a range of statistical or analytical techniques.

Figure: Figure 2: Distribution of tasks with respect to domain (rounded to nearest decimal), where each domain is a grouping of websites according to table [7](https://arxiv.org/html/2508.01222v1#Sx1.T7). We that Demographics and Policy and Economics and Markets are our most represented domain at 22.4% and 16.3% share respectively. Our least represented domain is sports consisting of 2̃% of tasks, which we note is still 18 tasks.
Refer to caption: treemap_highres.png

### 4.2 Task Formulation

We define a web data science task as: “a query which requires navigation within an environment to learn raw information that is then transformed to a predictive or summarative representation”. Optionally, there can be a downstream action based on the resulting predictive representation. More formally, let:
$\mathcal{W}$ denote the environment (e.g., the space of websites or a dynamic web environment),
$\mathcal{Q}$ denote the space of data science queries,
$\mathcal{D}$ denote the raw data space extracted from $\mathcal{W}$,
$\mathcal{Y}$ denote the space of analytical outcomes (e.g., reports, models, visualizations),
and
$\mathcal{A}$ denote the set of downstream actions, with $\varnothing\in\mathcal{A}$ indicating no action. We define a web data science task as a composite mapping
$f:\mathcal{W}\times\mathcal{Q}\to\mathcal{Y}\times\mathcal{A},$
where an agent, given a browser environment $\mathcal{W}$ and a natural language query $q\in\mathcal{Q}$, returns an analytical outcome $y\in\mathcal{Y}$ and/or an optional downstream action $a\in\mathcal{A}$. This overall process decomposes into three stages—information gathering, data analysis, and optional action—captured by the composition
$f=f_{\alpha}\circ f_{a}\circ f_{d}.$
In the first stage, $f_{d}:\mathcal{W}\times\mathcal{Q}\to\mathcal{D}$, the agent interacts with the environment to extract relevant raw data. In the second stage, $f_{a}:\mathcal{D}\to\mathcal{Y}$, the agent transforms this data into an analytical outcome, potentially through internal steps such as representation learning, tool usage, and prediction. In the final stage, $f_{\alpha}:\mathcal{Y}\to\mathcal{A}$, the agent maps the outcome to a downstream action—or none at all ($a=\varnothing$).

### 4.3 Task Attributes

We manually label each task with one or more of the following 7 labels, or "task attributes":

- •
Question-Answering (QA)  Tasks where the system provides a definitive answer to a user query (e.g. compute the combined total assets for two funds). These tasks have verifiable ground truths.
- •
Action-Based  Tasks that require the system to perform one or more actions after gathering and analyzing information. An example would be analyzing multiple data sources to rank financial indices and then posting an investment recommendation on Reddit.
- •
Single-hop vs. Multi-hop *Single-hop* tasks rely on a single data source to find the answer, while *multi-hop* tasks require combining information from multiple sources in the same website.
- •
Structured vs. Unstructured (textual) vs. Unstructured (contextual) Tasks. Structured tasks utilize data organized in a predefined format (such as databases). *Unstructured (textual)* tasks involve raw text data (e.g. documents or free-form responses), where extracting meaning requires natural language processing. *Unstructured (other)* tasks involve data without a clear textual representation. These tasks require interpretation of external knowledge, images or other multi-modal inputs.
- •
Tool Usage Some tasks necessitate the use of external tools (like Python scripts, Wolfram Alpha, or SQL queries) to perform computations or process information.
- •
Web Navigation (Webnav) Tasks that involve interacting with websites—whether querying a database online or making a post on a forum—relying on navigating through web pages.
- •
Multi-website Tasks Tasks that require gathering information or carrying out actions across multiple websites, where each site contributes separate pieces of the required data or functionality.

In summary, these tasks mirror real-world scenarios where a data analyst or general-purpose assistant must retrieve or compute information (for QA tasks) or perform downstream actions (for action-based tasks). Some tasks remain straightforward, requiring only a single data source or minimal steps, while others demand multi-hop reasoning, multi-website browsing, structured or unstructured data handling, and the integration of external tools. See Table [2](https://arxiv.org/html/2508.01222v1#S4.F2) for a more detailed breakdown of task-categories, but note that because these attributes are multi-label, some attributes (e.g. “Tool Usage”) are in multiple categories. See Appendix [9.1](https://arxiv.org/html/2508.01222v1#Sx1.SS1), Table [4](https://arxiv.org/html/2508.01222v1#Sx1.F4) for more complete counts of tasks.

**Table 2: Illustrative examples of QA vs. Action-Based tasks, showing various properties (single- vs. multi-hop, structured vs. unstructured data, web navigation, and tool usage). Note: Task counts will include any tasks that match the attributes suggested by the table even if other attributes are present.**
|  | Task Category | Description / Example | Properties |
| --- | --- | --- | --- |
| QA | Single-hop QA<br>(344 tasks) | *“What is the lowest AAPL stock price from 2000–2020?”* | - Single data source<br>- Question-Answering<br>- Webnav (likely)<br>- Possibly structured data<br>- No complex tool usage |
| Multi-hop QA<br>(117 tasks) | *“For each person $i$, let $X_{i}$ be the number of times their name appears in New York Times articles between [start date] and [end date]. Find the average and standard deviation of $X_{i}$ for all people mentioned.”* | - Multiple data sources (articles)<br>- Question-Answering<br>- Unstructured data<br>- Tool usage (e.g., Python)<br>- Web navigation |  |
| Actions | Single-hop Action<br>(97 tasks) | *“Delete the review from the user ‘scammer Yoke’ on our product page.”* | - Single data source or site<br>- Requires an action (delete)<br>- Web navigation |
| Multi-hop Action<br>(134 tasks) | *“Buy the $n$th cheapest laptop after comparing prices across Store A, Store B, and Store C.”* | - Multiple data sources (stores)<br>- Requires an action (purchase)<br>- Multi-hop + multi-website<br>- Web navigation |  |
| Action + Tool Usage<br>(139 tasks) | *“Compute the average number of people who got sick per [interval] using CDC data (csv file) and post the result on Reddit.”* | - Structured data (csv)<br>- Tool usage (DB query, analytics)<br>- Action (post on Reddit)<br>- Web navigation |  |

Finally, we categorize tasks into three levels of difficulty – easy, medium hard – based on specific attributes that reflect increasing cognitive and operational complexity, as per table [3](https://arxiv.org/html/2508.01222v1#S4.T3) for details on how we make these determinations.

**Table 3: Task difficulty categorization based on structural and content properties. We find that agents on average perform 2.5x better on easy tasks compared to medium or hard**
| Difficulty | Task Attributes |
| --- | --- |
| Easy (247 tasks) | Does not involve any of the following properties: multihop, nontextual unstructured data, action-based tasks, or tool-use and is single-website. |
| Medium (275 tasks) | Involves exactly one of the above properties and is single-website. |
| Hard (348 tasks) | Involves at least two of the above properties or is multi-website. |

## 5 Experimental Setup

### 5.1 Evaluation Protocol

Our evaluation framework combines automated and subjective assessment to robustly evaluate agents in complex, multi-step tasks. In particular, we provide two evalutions for each task, an automated binary evaluation and a subjective 1–5 score given by an LLM judge.

#### Automated Evalution

For tasks with reference ground truths (e.g. extracting factual statistics) we assign binary labels: SUCCESSFUL / UNSUCCESSFUL. These are determined by a language model comparing the agent’s final output or action to ground truth.

#### Subjective Evaluation via LLM

To evaluate open-ended tasks (e.g. generating a Reddit post with statistics), we build on the LLM-as-a-Judge paradigm introduced by Web Voyager , which classified outcomes based solely on the final 15 screenshots of a trajectory. We extend this method in two key ways (1) we have the LLM provide a more informative 1–5 integer score, along with rationale and failure analysis, enabling more granular agent iteration; and (2) instead of relying on the final states alone, we evaluate the full trajectory by summarizing each observation and analyzing $\left(\text{observation},\;\text{action},\;\text{next observation}\right)$ triplets throughout the task, resulting in more reliable and interpretable scores, which are assigned as follows:

$R_{\text{llm}}(s_{T})=\left\{\!\begin{matrix}5&\text{fully satisfies }G\\
4&\text{mostly satisfies }G\\
3&\text{partially satisfies }G\\
2&\text{major errors, some progress}\\
1&\text{major errors, no progress}\end{matrix}\!\right.$

#### Robustness and Human Study

To validate our subjective scoring method, we conducted a human study and by performing a stability analysis. For the human study we sampled 50 task-trajectory pairs and had them rated independently by three CS undergraduate annotators and found that GPT-4o-based system achieved 88% agreement with the human assessments. For stability analysis, we ran our scoring harness 10 times across 100 tasks for the GPT-4o agents. The standard deviation of scores across runs was $0.027$, confirming the evaluation method’s consistency.

### 5.2 Models

We benchmarked nine state-of-the-art (SOTA) web agents to evaluate performance on our newly proposed benchmark.

To ensure consistency with prior work, we first evaluated the general-purpose GPT-4o and GPT-4o-mini agents using accessibility trees as observations, following the exact implementation described in WebArena webarena. Using the same observation space and implementation pipeline, we additionally benchmarked Claude Sonnet-3.5, Claude Haiku-3.5, and Qwen2.5-72B-Instruct. To explore multimodal capabilities, we modified the observation space to incorporate full-page screenshots and evaluated Qwen2.5-VL-72B-Instruct under this setting.

We also included AgentOccam, the most performant fully open-source agent on WebArena, benchmarking it under the official configuration used for WebArena without altering its implementation. Finally, we evaluated browser_use2024, the current SOTA agent on WebVoyager via both Qwen2.5-VL-72B-Instruct and GPT-4o. In particular, we used the default configuration of Browser Use browser_use2024, the SOTA model on Web Voyager, with minimal adaptations to the observation and action space to better accommodate the structure of our landing pages and to improve handling of CSV files and file downloads (as to not flood the LLM context window).

## 6 Results

**Table 4: Success Rate (SR%) and Score (1–5) of 8 models on WebDS benchmark. Browser Use (Qwen2.5-72b) performs the best with a success rate of 13.2%, most models have below 2% Success Rate. Success rate is measured by the number of tasks evaluated as SUCCESSFUL and score is given by an LLM as a judge, both as defined in section [5.1](https://arxiv.org/html/2508.01222v1#S5.SS1).**
|  | Success Rate (SR%) | Score (1–5) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Agent | Overall | Easy | Medium | Hard | Overall | Easy | Medium | Hard |
| GPT-4o | 1.0 | 1.7 | 0.4 | 0.9 | 1.111 | 1.187 | 1.073 | 1.087 |
| GPT-4o-mini | 0.8 | 1.2 | 1.1 | 0.3 | 1.087 | 1.091 | 1.097 | 1.075 |
| Sonnet-3.5 | 1.3 | 2.9 | 0.7 | 0.6 | 1.087 | 1.168 | 1.056 | 1.054 |
| Haiku-3.5 | 1.6 | 2.4 | 1.5 | 1.2 | 1.135 | 1.200 | 1.119 | 1.102 |
| AgentOccam | 4.8 | 11.3 | 3.0 | 1.8 | 1.658 | 2.121 | 1.417 | 1.524 |
| Qwen2.5-vl-72b | 0.2 | 0.0 | 0.4 | 0.3 | 1.050 | 1.101 | 1.037 | 1.024 |
| Qwen2.5-72b | 1.2 | 2.5 | 0.8 | 0.6 | 1.157 | 1.273 | 1.140 | 1.085 |
| Browser Use (Qwen2.5-72b) | 13.2 | 21.9 | 11.0 | 8.6 | 2.044 | 2.385 | 1.993 | 1.841 |
| Browser Use (GPT-4o) | 12.9 | 21.9 | 7.0 | 11.2 | 2.171 | 2.490 | 1.934 | 2.133 |

Figure: Figure 3: Accuracy of Agents Across Benchmarks. Models perform significantly worse on our benchmark (for e.g. with Browser Use + GPT4o, dropping from 81.9% on WebVoyager to 12.9% on WebDS). Note: WebVoyager Performance is based on GPT-4 not 4o, as there are no verified GPT-4o results he2024webvoyagerbuildingendtoendweb.
Refer to caption: graph_comparison.png

Despite strong performance on current agent benchmarks, all agents perform poorly on WebDS. For instance, AgentOccam —a SOTA open source model that acheives $45.7\%$ on WebArena —achieves only $4.8\%$ on WebDS. Similarly, Browser Use + GPT-4o, the SOTA web agent that surpasses OpenAI’s Operator and Claude Computer Use with an $89\%$ score on the WebVoyager benchmark, achieves just $12.9\%$ on WebDS, even with our modifications [9.2](https://arxiv.org/html/2508.01222v1#Sx1.SS2). The performance drop highlights that data science is a key challenge to current agents. Interestingly, Browser Use paired with Qwen2.5-72b outperforms the same setup with GPT-4o, suggesting that model capability is not the main bottleneck. Rather, the critical limitation lies in the translation layer between reasoning and environment interaction. This is further corroborated by error analysis where there are several UI interaction failures. Future work might focus more on precise control rather than advanced reasoning and planning, we liken the relationship between LLMs and precise control could be analogous to between the motor and prefrontal cortices.

## 7 Analysis

### 7.1 Failure Analysis

We evaluated the reasons behind the failure of these LLM agents, and looked for key reasons behind their failure most failures stemmed from weak grounding, poor feedback handling, and misinterpretation of user intent, more details can be found in table [6](https://arxiv.org/html/2508.01222v1#S7.T6).

**Table 6: Taxonomy of model failure themes with representative examples and underlying causes.**
| Failure Theme | Example | Analysis |
| --- | --- | --- |
| Navigation | The model navigated to the American Physical Therapy Association instead of the American Public Transportation Association.. | The model confuses similarly named entities, interacting with irrelevant domains. It lacks validation steps to check if the information source aligns with the query. |
| UI Feedback | The model repeatedly tried to set a search field to “Abstract,” but the system failed to register this action. | The model cannot confirm whether its UI manipulations are successful. This leads to broken search flows and incomplete query execution. |
| Groundedness | The model failed to mention the 12% bias in generative AI, despite viewing the correct document. | The model fails to extract key details and instead provides vague summaries. This shows a gap between information access and correct synthesis. |
| Failed Repetition | The model failed to mention the 12% bias in generative AI, despite viewing the correct document.. | The model fails to extract key details and instead provides vague summaries. This shows a gap between information access and correct synthesis. |
| Query Interpretation | Asked for a 30% publication growth figure but responded with general research trends. | The model misunderstands that task require certain kinds of reasoning (e.g. a specific numeric trend, not a qualitative description). |
| Effort Allocation | Couldn’t retrieve release count for an artist via MusicBrainz, then gave incorrect Wikipedia total. | The model attempts to use a tool, but after failing, simply googled the task and got it wrong because Wikipedia records unique releases rather than all releases. |

### 7.2 Benchmark Longevity

We design WebDS to stay relevant and avoid saturation by (i) covering diverse domains, data types, and three difficulty tiers, (ii) running its 29 sites in Docker to guarantee exact reproducibility, (iii) offering an extensible task-creation workflow that keeps the benchmark “evergreen,” and (iv) remaining hard enough that humans still outperform the best current models, leaving ample headroom for progress. See Appendix [9.5](https://arxiv.org/html/2508.01222v1#Sx1.SS5) for full details.

## 8 Conclusion

WebDS introduces a novel benchmark that rigorously evaluates AI agents on complex, web-based data science tasks. By encompassing structured and unstructured data analysis, multi-hop reasoning, tool integration, and action-based workflows, it fills a critical gap in current web-based agent evaluation frameworks. Our preliminary results reveal significant performance gaps in SOTA models, highlighting the need for continued innovation in web-based data science automation. We hope that WebDS serves as a foundation for future research in AI-driven web navigation, tool-assisted data analysis, and multi-modal reasoning. Future work in this area would include improving task diversity to encompass more action-based tasks and expansion to OS-level tasks or other enterprise workflows.

## 9 Limitations

While WebDS robust a strong benchmark for evaluating web-based data science agents, it has notable limitations. Despite covering 870 tasks across 29 websites and 10 domains, it lacks full representation of some real-world applications—specialized areas like real-time trading and scientific simulations a remain underrepresented, and focus on QA queries may limit broader relevance. Further, releasing the full evaluation set risks benchmark overfitting, where models are tuned to specific tasks instead of general reasoning. Web agents also pose ethical and technical risks: they may expose PII by linking cross-site data, hallucinate errors, or strain web infrastructure through automated access. As with all agent research, there’s also potential for violating site terms, displacing jobs, or enabling misuse.

## References

- [1]
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su.
Mind2web: Towards a generalist agent for the web.
Advances in Neural Information Processing Systems, 36, 2024.
- [2]
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste.
WorkArena: How capable are web agents at solving common knowledge work tasks?
In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11642–11662. PMLR, 21–27 Jul 2024.
- [3]
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu.
Webvoyager: Building an end-to-end web agent with large multimodal models, 2024.
- [4]
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al.
Data interpreter: An llm agent for data science.
arXiv preprint arXiv:2402.18679, 2024.
- [5]
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al.
Infiagent-dabench: Evaluating agents on data analysis tasks.
arXiv preprint arXiv:2401.05507, 2024.
- [6]
Jacqueline A Jansen, Artür Manukyan, Nour Al Khoury, and Altuna Akalin.
Leveraging large language models for data analysis automation.
PloS one, 20(2):e0317084, 2025.
- [7]
Sean Kandel, Andreas Paepcke, Joseph M. Hellerstein, and Jeffrey Heer.
Enterprise data analysis and visualization: An interview study.
IEEE Transactions on Visualization and Computer Graphics, 18(12):2917–2926, 2012.
- [8]
Qunhua Li.
An analyst-inspector framework for evaluating reproducibility of llms in data science, 2025.
- [9]
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge.
Mapping natural language instructions to mobile ui action sequences.
arXiv preprint arXiv:2005.03776, 2020.
- [10]
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom.
Gaia: a benchmark for general ai assistants.
In The Twelfth International Conference on Learning Representations, 2023.
- [11]
Magnus Müller and Gregor Žunič.
Browser use: Enable ai to control your browser, 2024.
- [12]
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr.
Autonomous evaluation and refinement of digital agents.
arXiv preprint arXiv:2404.06474, 2024.
- [13]
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al.
Webcanvas: Benchmarking web agents in online environments.
arXiv preprint arXiv:2406.12373, 2024.
- [14]
Panupong Pasupat and Percy Liang.
Compositional semantic parsing on semi-structured tables.
arXiv preprint arXiv:1508.00305, 2015.
- [15]
Dale Peasley, Rayus Kuplicki, Sandip Sen, Martin Paulus, et al.
Leveraging large language models and agent-based systems for scientific data analysis: Validation study.
JMIR Mental Health, 12(1):e68135, 2025.
- [16]
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba.
Virtualhome: Simulating household activities via programs.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502, 2018.
- [17]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
Squad: 100,000+ questions for machine comprehension of text.
arXiv preprint arXiv:1606.05250, 2016.
- [18]
Guangyuan Sun, Tanja Friedrich, Kathleen Gregory, and Brigitte Mathiak.
Supporting data discovery: A meta-synthesis comparing perspectives of support specialists and researchers.
arXiv preprint arXiv:2209.14655, 2022.
- [19]
Eduardo Treviño, Hugo Contant, James Ngai, Graham Neubig, and Zora Zhiruo Wang.
Benchmarking failures in tool-augmented language models.
arXiv preprint arXiv:2503.14227, 2025.
- [20]
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Deyu Zhou, Pengjun Xie, and Fei Huang.
Webwalker: Benchmarking llms in web traversal, 2025.
- [21]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
arXiv preprint arXiv:1809.09600, 2018.
- [22]
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant.
Assistantbench: Can web agents solve realistic and time-consuming tasks?
arXiv preprint arXiv:2407.15711, 2024.
- [23]
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su.
Gpt-4v (ision) is a generalist web agent, if grounded.
arXiv preprint arXiv:2401.01614, 2024.
- [24]
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al.
Webarena: A realistic web environment for building autonomous agents.
Proceedings of the 12th International Conference on Learning Representations, 2024.

## Appendix

### 9.1 Dataset Statistics, More Graphs, and Further Details

Figure: Figure 4: Counts of tasks per attribute
Refer to caption: Tasks_per_Attribute.png

Figure: Figure 5: Counts of Task Difficulty
Refer to caption: Task_Difficulty_Distribution.png

Figure: Figure 6: Flow chart for evaluating task difficulty
Refer to caption: task_classification.png

**Table 7: Mapping from Domains to Websites. We note that the Reddit website and Shopping website are the same as in Web Arena, but our tasks are unique**
| Domain | Websites |
| --- | --- |
| Economics/Markets | BEA, FRED, ST LOUIS FED, TRADING ECONOMICS, DATAUSA |
| Demographics | WORLDPOP, WORLDOMETER |
| Music | TUNEBAT, MUSICBRAZ, RIAA, RICHARD POWERS |
| Tourism, Trade, and Airlines | UNWTO, IATA |
| Higher Education | MIT, UCHICAGO |
| Scientific Research | ARXIV |
| Sports | UNDERSTAT |
| Government and Public Policy | CFPB, OUR WORLD IN DATA, REDDIT |
| Energy and Climate | CLIMATE GOV, NOAA |
| Health | CDC MENTAL HEALTH, CDC COVID, CDC OBESITY, NIH |
| E-Commerce | SHOPPING, STOCKNEAR |

### 9.2 Browser Use Implementation

We used the out-of-box version of Browser Use’s agent implementation as of April 22nd 2026 with a few modification to enhance performance on our benchmark. The enhancements were as follows:

- 1.
Changing the observation space to search through the AX Tree of the current state to grab all clickable items and index them $1,...,n$. The original observation space used a get_clickable_elements() function which incorrectly ignored certain links which inhibited the agent from effectively interacting with our landing page and some websites.
- 2.
We improved the agent system prompt to list all actions available to it, as we noticed in initial experimentation that the agent would never take some actions despite it being advantageous.
- 3.
We added a few functions to download and read arbitrary rows of a csv file and write python to analyze analyze said csv.

### 9.3 Compute Used

We used 4x 16-core Xenon CPUs with 258 GB of ram between them to run agents in parallel massively. No GPUs are needed as we make direct calls to provider APIs (no locally hosted models), and running these tasks are CPU and I/O constrained.

### 9.4 Formalization of Agent Approach

#### 1. Navigation and Information Gathering

The agent interacts with the environment to extract raw data relevant to a given query:

$$ $f_{d}:\mathcal{W}\times\mathcal{Q}\to\mathcal{D}.$ (1) $$

Note: In dynamic settings, $\mathcal{W}$ ma y be modeled as an interactive state space or a Markov Decision Process (MDP) to capture sequential interactions.

#### 2. Representation Learning and Outcome Prediction

The agent processes the raw data to obtain an analytical outcome. This stage is further decomposed into:

#### a. Representation Learning

Transform the raw data $d\in\mathcal{D}$ into an intermediate representation $T$ that retains only the information necessary for prediction:

$$ $f_{T}:\mathcal{D}\to\mathcal{T}.$ (2) $$

#### b. Outcome Prediction

Map the learned representation $T$ to an analytical outcome $y\in\mathcal{Y}$:

$$ $f_{y}:\mathcal{T}\to\mathcal{Y}.$ (3) $$

Together, the above steps (a) and (b) define the mapping from raw data to some analytical outcome:

$$ $f_{a}:\mathcal{D}\to\mathcal{Y}.$ (4) $$

#### 3. Optional Downstream Action

Based on the analytical outcome $y$, the agent may execute a downstream action:

$$ $f_{\alpha}:\mathcal{Y}\to\mathcal{A},$ (5) $$

with $a=\varnothing$ indicating that no further action is taken.

- •
HTML - The raw HTML structure of the web pages.
- •
DOM - The Document Object Model representation.
- •
Screenshots - Visual representations of web pages.
- •
AXTrees - Accessibility tree structures for web elements.
- •
Downloaded Data - Unlike prior benchmarks, our setup allows for downloading data, which is then loaded into the LLM’s context. As a result, the LLM can also observe structured data formats such as:
–
JSON
–
CSV
–
Other structured data formats.

### 9.5 More Details on Longevity

- 1.
Fine-Grained Difficulty and Domain Categorization.
WebDS evaluates agents across multiple axes, including task difficulty (easy, medium, hard), data representation (structured, unstructured), and domain (e.g., demography, finance, public health). This multi-dimensional evaluation reduces the risk of models overfitting to a narrow set of tasks and provides continued granular insight even as models improve over time.
- 2.
Containerized and Stable Evaluation.
By deploying all 29 websites in Docker containers, we eliminate the instability associated with live web environments. This ensures that future researchers can reproduce results exactly and fairly compare new methods against older baselines, even years after initial release.
- 3.
Evergreen Task Format and Modifiability
Data science tasks inherently offer broad extensibility: new queries, updated data sources, and evolving analytical goals mean that researchers can continuously generate new tasks using the same framework. WebDS supports community-driven extensions through tools that enable users to create, classify, and evaluate new tasks with minimal overhead. In particular, this can be adapted to tackle many other enterprise workflows, showing huge potential as a benchmark to evaluate general AI agents.
- 4.
High Task Complexity.
Even current frontier models perform poorly, indicating a wide margin for future progress. Additionally, human performance is non-trivial on many tasks, underscoring the benchmark’s difficulty and its grounding in real-world challenges.

### 9.6 Docker Image Safety

We scrape a large portion of our sites from the internet, to avoid unsafe images we picked websites from safe domains such as governmental organizations, nonprofits, and reputable companies with no ads.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The benchmark includes Tool Usage as an attribute. Could the authors clarify which specific tools (e.g., Python code execution, SQL query, spreadsheet manipulation) were enabled for the SOTA agents during the 13.2\% success rate evaluation? Did the 13.2\% success rate fully integrate external code execution or primarily rely on the LLM's internal reasoning over extracted text?

2. The analysis highlights "Poor Information Grounding" as a key failure mode. Can the authors provide a more detailed example of how the "Groundedness" failure was scored? Is the model judged based on whether it extracts the correct data from the HTML (perception error) or whether it uses the correct data in the final report (synthesis/reasoning error)?

3. To maintain relevance and avoid saturation, the paper mentions a plan to "periodically refresh/expand the private test pool." Could the authors provide a brief plan or estimate of the expected frequency or size of this refreshment (e.g., quarterly, or annually adding 100 new tasks)?
soundness: 4
strengths: 1. The benchmark's grounding in practitioner interviews and the inclusion of 29 diverse, data-rich websites ensures the tasks are highly realistic and demand complex generalization, not simple pattern matching.

2. The failure analysis (Table 5) provides excellent, fine-grained insights into unique failure modes not captured by simplistic metrics. Specifically, "Failed Repetition" (due to lacking state-checking heuristics) and "Poor Groundedness" (contradiction between perceived and latent knowledge) are critical problems for future research.

3. The evaluation structure allows for granular analysis across Domain-wise, Attribute-wise (7 labels), and Difficulty tiers (Easy/Medium/Hard), giving researchers precise targets for iterative model improvement.
summary: This paper introduces WebDS, the first end-to-end benchmark for web-based data science workflows. It addresses the gap between existing benchmarks that focus either on simple web navigation or static data analysis. WebDS comprises 870 human-written tasks across 29 dockerized, data-rich websites from 10 domains (e.g., government data, news media). Tasks require complex, multi-step operations (data acquisition, synthesis, analysis, report generation) across heterogeneous sources. Evaluation of current SOTA LLM agents reveals a substantial performance gap: the best agent achieves only $13.2\%$ success (vs. $90\%$ human baseline), exhibiting novel failures like poor information grounding and failed repetition. WebDS is containerized for reproducibility and offers fine-grained evaluation across task attributes and difficulty tiers, positioning it as a new frontier for developing practical, end-to-end data science agents.
weaknesses: 1. The analysis and outputs primarily focus on text (reports, answers). Since the input sites are described as being rich in graphics and non-textual data, the benchmark's current focus may under-represent the full multimodal synthesis challenge (e.g., generating a visual chart or interpreting a complex image-based trend).

2. The action space is not fixed, allowing researchers flexibility, but the implementation relies on existing WebArena/BrowserGym abstractions. A brief discussion on whether these existing abstractions adequately capture the fine-grained data manipulation needed for the "Tool Usage" attribute would be beneficial.

3. The current distribution of tasks may slightly overemphasize QA tasks versus downstream action tasks. While the authors mention future work to include more action-based tasks, the current lack of balance might skew initial model optimization toward data retrieval over action execution.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: Please refer to the weaknesses
soundness: 3
strengths: - The benchmark environment is very comprehensive, covering 29 websites, 10 domains, and 870 tasks. It focuses on the **data science domain**, evaluating the **entire data processing pipeline**.

- The test environment is **dockerized**, offering fixed environments, stable experiments, and reproducible results.

- The authors use **vision-language models** to analyze complete execution trajectories, providing **richer evaluation metrics** beyond success rate.
summary: This paper proposes a new benchmark for the web agent domain: WEBDS, which includes 870 data science tasks across 29 different websites. It also provides a dockerized implementation, ensuring stability and reproducibility of experiments. This work lays a foundation for the development of intelligent agents in the data science domain.
weaknesses: - **Lack of innovation:** The benchmark design does not significantly differ from existing mature web agent benchmarks. The main contributions remain in expanding the testing environment and task set.

- **Insufficient rigor:** Although using vision-language models for trajectory evaluation is common in GUI agent research, the validation experiments for scoring accuracy and stability (on a 1–5 scale) are rather cursory. The authors only mention comparing 50 human-evaluated tasks, yet the claimed 88% agreement seems questionable. Moreover, since most tested agents achieve **less than 10% success rate**, with most scores below 2, the majority of ratings are clustered around 1. Such a narrow score distribution weakens the credibility of the validation.

- **Limited model comparison:** The benchmark does not test more advanced GUI agents such as **UI-TARS** and **GPT-5**. Since most baseline models achieve accuracy below 5%, this raises the question of whether the benchmark is **too difficult**.

- **Lack of analysis on low success rates:** The paper merely lists potential reasons without detailed trajectory-based analysis.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How do you plan to maintain or expand WebDS over time while ensuring reproducibility and preventing overfitting?
2. Could future versions include community-contributed tasks to diversify domains and evaluation scope?
soundness: 3
strengths: This work fills a major gap in current evaluationno existing benchmark assesses full end-to-end data science workflows involving both web interaction and analytical reasoning, captures realistic web-based tasks that better reflect real-world data analysis behavior. The authors evaluates a wide range of SOTA models consistently and highlights key performance bottlenecks.
summary: The paper proposes WebDS, a new benchmark to evaluate the end-to-end web-based data science capabilities of large language model agents. Unlike previous web agent benchmarks that focus on simple browsing, or data science benchmarks that focus on static datasets, WebDS integrates both dimensions — requiring agents to autonomously navigate, collect, analyze, and synthesize data from web-based environments.
weaknesses: 1. The subjective scoring relies on GPT-4o, creating potential evaluation circularity and bias toward similar model families. Including more human evaluations or open-source LLM judges would strengthen reliability.
2. While qualitative error categories are given, there is no quantitative breakdown of which task attributes (multi-hop, tool-use, multi-site) most contribute to failures.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. While WebDS tasks involve data retrieval and basic analysis, many real-world data science workflows include steps such as data cleaning, preprocessing, visualization, and modeling. Do the authors plan to expand WebDS to cover these stages, and how would they envision incorporating them into the benchmark ?
2. Why not adopt **execution-based evaluation**, which would be more realistic than the current rule-based approach?


Miss citations:

I strongly recommend that the authors cite more **data science–related work**, such as:

* *DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?*
* *DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models*
* *Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows*
* *Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?*
* *DABstep: Data Agent Benchmark for Multi-step Reasoning*
* *OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments*
soundness: 3
strengths: The paper is highly original in proposing the first end-to-end benchmark for web-based data science, combining web navigation with real data analysis in a way not covered by prior benchmarks. It demonstrates strong quality and clarity by designing 29 containerized websites and 870 tasks across multiple domains, with clear task categorization that ensures reproducibility. Finally, its significance is clear as results reveal a striking human–agent performance gap, establishing WebDS as a critical benchmark for advancing research on practical, data-science–capable AI agents.
summary: The paper presents WebDS, a benchmark for evaluating LLM agents on realistic web-based data science tasks. It integrates 29 containerized websites and 870 tasks involving data retrieval, multi-hop reasoning, tool use, and report generation. Results show humans succeed ~90%, but state-of-the-art agents achieve <15%, revealing a major performance gap. The key contribution is providing a comprehensive and reproducible benchmark that mirrors end-to-end data science workflows.
weaknesses: 1. Limited Task Scope Beyond Data Science. Although the paper claims to be about Web Agents + Data Science, in reality the data science aspect is minimal, and the amount of required coding is very small.

2. Overly Simple Baseline Design. The paper does not propose its own unique baselines. Considering this is naturally a CLI + GUI hybrid setting, many approaches from WebArena or similar frameworks could have been adapted, but they were not included. Moreover, many closed-source agent frameworks and both open/closed-source products were ignored. Overall, the baseline coverage feels insufficient.

3. Lack of Multiple-Trial Evaluation (pass@n). For each example, it is unclear whether multiple trials were conducted to calculate pass@n. I suspect the low reported performance may be partly due to annotation ambiguity, which makes it hard for agents to succeed on a single attempt.

4. Subjective Task Evaluation Is Too Arbitrary. For open-ended tasks, evaluation is simply based on a 5-level scoring scheme. Ideally, one should measure agreement between humans and LLM-as-a-Judge. If a more reliable scoring method were adopted, I would increase my rating, but the current LLM-based scoring feels too random, and results may vary significantly across repeated runs.


Decision: Accept (Poster)