WebDS: An End-to-End Benchmark for Web-based Data Science
Authors: Ethan Hsu
Hong Meng Yam
Ines Bouissou
Aaron Murali John
Raj Thota
Josh Koe
Vivek Sarath Putta
G K Dharesan
Alexander Spangher
Shikhar Murty
Tenghao Huang
Christopher D Manning
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Benchmark
Data Science
LLM Agents
Primary area: datasets and benchmarks
TL;DR: Data Science with Web Navigation Benchmark for LLM Agents

Abstract: Many real-world data science tasks involve complex web-based interactions: finding appropriate data available on the internet, synthesizing multimodal data from different locations, and producing summarized analyses. Existing web benchmarks often focus on simplistic interactions and often do not require diverse tool-using capabilities. Conversely, traditional data science benchmarks typically concentrate on static, highly structured datasets and do not assess end-to-end workflows that encompass data acquisition, cleaning, analysis, and insight generation. In response, we introduce WebDS, the first end-to-end web-based data science benchmark. It comprises 870 web-based data science tasks across 29 diverse websites from structured government data portals to unstructured news media, challenging agents to perform complex, multi-step, tool-based operations, across heterogeneous data formats, to better reflect the realities of modern data analytics. Evaluations of current SOTA LLM agents indicate significant performance gaps in accomplishing these tasks. For instance, Browser Use, which accomplishes 80% of tasks on Web Voyager, completes only 15% of tasks in WebDS, which our analysis suggests is due to new failure modes like poor information grounding, repetitive behavior and shortcut-taking that agents performing WebDS' tasks display. By contrast, humans achieve around 90% accuracy, highlighting a substantial gap between current agents and human performance. By providing a more robust and realistic testing ground, WebDS sets the stage for significant advances in the development of practically useful LLM-based data science.

Published as a conference paper ICLR 2026
WEBDS: AN END-TO-END BENCHMARK FOR WEB-BASED DATA SCIENCE
Ethan Hsu"r, Hong Meng Yam--2. Ines Bouissou"i, Aaron Murali John",
Raj Thota", Koe*, Vivek Sarath Putta G K Dharesan',
Alexander Spangher'. Shikhar Murty', Tenghao Huang'. Christopher D. Manning'
Sttafford University PPinetree Research UUniverstyy of California, Berkeley "Singapore University of Technology and Design Universityy of Southern California { (ethanhsu, hongmeng, manning @st anord....... edu
"Equal contrihution (first authors). 'Equal contribution (second authors).
ABSTRACT
Many real-world data science tasks involve complex web-based interactions: finding appro- priate data available on the internet, synthesizing multimodal data from different locations, and producing summarized analyses. Existing web benchmarks often focus on simplistic interactions and often do not require diverse tool-using capabilities. Conversely, traditional data science benchmarks typically concentrate on static, highly structured datasets and do not assess end-to-end workflows that encompass data acquisition, cleaning, analysis, and insight generation. In response, we introduce WebDS, the first end-to-end web-based data science benchmark. It comprises 870 web-based data science tasks across 29 diverse websites from structured government data portals to unstructured news media, challenging agents to perform complex, multi-step, tool-based operations, across heterogeneous data formats, to better reflect the realities of modern data analytics. Evaluations of current SOTA LLM agents indicate significant performance gaps in accomplishing these tasks. For instance, Browser Use, which accomplishes 80% of tasks on Web Voyager, completes only 15% of tasks in WebDS, which our analysis suggests is due to new failure modes, such as poor information grounding, repetitive behavior and shortcut-taking that agents performing WebDS's ' tasks display. By contrast, humans achieve around 90% accuracy, highlighting substantial gap between current agents and human performance. By providing more robust and realistic testing ground, WebDS sets the stage for significant advances in the development of practically useful LLLM-base data science.
The WebDS benchmark is available at https: htt:://webbbee /twpww/wwwwhhhark..kkkk. github.hhhuu..... io/
1 INTRODUCTION
Large language models (LLMs) with web-traversal and tool-use abilities, i.e., agents, have shown promise utilizing the web's vast repository of knowledge: navigating websites, synthesizing insights, and executing tasks with minimal human intervention (Hong et al. 2024). These agents hold significant potential for use in data science applications too, with researchers using agents to perform autonomous data querying, visualization, and prediction (Peasley et al. 2025 Jansen et al.) 2025). Yet, combining the two tasks to extract data-driven insights from websites an essential routine for most data science practitioners (Kandel et al. 2012, Sun et al. 2022) is not trivial. Modern webpages are complex and dynamic: unstructured, multimodal data are scattered across multiple layers of interactivity with varying levels of access restrictions.
For examplc, recent advances like Deep Research tools by OpenAl http:://wpea....commmmmm///nnnne research/) and Google DeepMind

---

Published as a conference paper ICLR 2026
Step 3: Extroct MIT lstal eonaiisent aunbers Genographic
Action
CSV
Sourh Conmue
Ted Camu Siarkry
Steg 2: Ratrant 2023
Action
earolinent
CSV
sumnheerlyy
Renographie
Seuch solaage Commessi
1D3 Seaaty
Action
3:
Extaatt nasianal demographic dasa analyne
CSV
Figure I: Example of the task, "Analyze the total enrollment numbers by racial/ethnic category for undergrad- uates (both degree- and seedderre--seeiinn as of October 19, 2022. Cross-reference these numbers with national demographic trends and discuss the potential impact on the university's diversity initiatives. Write a report for the university's 's strategic planning committee on these trends and recommendations. Note: task was selected to save space while still illustrating the multisite attribute.
Autonomously navigating the web to find data, synthesizing and analyzing it repeatedly, requires robust reasoning capabilities and contextual understanding.
Prior benchmarks fail to capture the scope of this challenge. Evaluations are narrowly targeted either towards (a) web browsing only (Zhou et al., 2024; Pan et al. 2024b), or (b) data science tasks only. Existing web agent benchmarks tend to study agents' ability to follow natural language instructions accomplish simple web-based tasks (e.g.. writing Reddit posts or buying items (Zheng et al. 2024)), without assessing agents on their ability to manipulate and understand data. Additionally, these benchmarks often lack long-term robustness: WebVoyager (He et al. (2024). for instance, uses live sites that change over time, hindering reproducibility. Others (Li et al. 2020. Deng et al., 2024, Zhou et al.] (022)) rely on coarse metrics that miss finer agent behaviors. On the other hand, existing data science benchmarks primarily emphasize data manipulation within code-based environments or structured spreadsheets (Li) 2025). or focus on querying unstructured data from static databases (Rajpurkar et al. 20166. However, realistic workflows of data scientists and other knowledge workers often begin by browsing the web, navigating across multiple websites, and synthesizing information from heterogeneous sources. Thus, browsing, searching and synthesizing are fundamental, yet often overlooked, components of real-world web-based data science tasks Kandel et al. 2):12;; (Sun al. (0022).
To address the limitations of existing web agent and data science benchmarks, we introduce novel benchmark, WebDS. WebDS consists of 29 data-rich websites (e.g., CDC, sales) and contains 870 real, human-written tasks that demand complex navigation and analysis. These tasks pose a challenge for leading models: while GPT-4o with BrowserUse obtains an 81.1% 1..% success rate on Web Vovager (üüller & Zunič 2024) it only obtains 13.2% on WebDS; while GPT-4o with AgentOccam (Yang al., 2024 accheves 45.7% 7% success rate on WebArena, it only obtains 4.8% on WebDS. Surprisingly, we find that increasing model capacity does not necessarily lead to increased performance: GPT-4o performs similarly to GPT-4o-mini and Qwen2.5-72b. An error analysis (Section 7 shows significant, novel failure modes for all models: poor information grounding (where grounded knowledge contradicts latent knowledge), repetitive behavior and thortcut-taking (due to the multi-hop nature of web-based data science). Additionally, end-to-end model performance WebDS is markedly below human performance: while the strongest agent reaches 13.2% success, a human baseline
TThis the SOTA open source model WebArena as of 5/15/2025.
2

---

Published as a conference paper ICLR 2026
under identical constraints achieves 90% (#3%). This quantifies large human-agent gap on realistic, long-horizon workflows.
In sum, WebDS introduces a significantly more challenging, real-world benchmark that represents a next frontier for model development. By bridging the gap between web interaction and data science capabilities, we hope that our work guides the community and pushes us towards developing novel end-to-end Al skills. Our contributions are threefold:
1. Comprehensive Task Suite We introduce WebDS, large-scale benchmark consisting of 870 tasks across 29 data-rich websites spanning 10 domains (discussed in section [] [) The benchmark is released in two tracks: WebDS-live, which evaluates agents on live websites, and WebDS-dockerized, which provides a reproducible subset of these environments through containerized deployments and is the most comprehensive dockerized benchmark to date. These websites span more data types, modalities, and domains than any prior benchmark and these tasks require not only analytical reasoning but also interaction with diverse tools and interfaces.
2. Realistic End-to-End Evaluation WebDS is the first benchmark to assess execution of the full data science pipeline, simulating end-to-end workflows. Tasks begin with autonomous web browsing for relevant data, followed by analysis and/or visualization, and culminate in generating well-reasoned, context-aware outputs. A key contribution is that we quantify the large human-agent gap these tasks, highlighting a unique and previously under-measured limitation that today's 's agents cannot yet sustain coherent performance across full data-science workflows.
3. Reproducible, Realistic and Fine-Grained Evaluation. WebDS combines the strengths of both repro- ducible and real- real-world evaluation paradigms. The WebDS-live track captures the evolving nature of real websites and reflects the challenges faced by practical web-based data science agents. In parallel, WebDS- dockerized provides containerized website environments whose state is preserved, enabling fully reproducible experiments and stable longitudinal benchmarking. Beyond prior binary metrics, WebDS introduces fine- grained measures of subtask completion, tool use, reasoning quality, and report fidelity, enabling both holistic evaluation and precise diagnostic analysis of agent performance.
2 RELATED WORK
Data Analysis Benchmarks Benchmarks such as SQuAD RRajpurkar et al. 2016) and HotpotQA (Yang al. 2018) have played critical roles in evaluating language models' reasoning over structured and semi- structured data, providing passage-basee and multi-hop
tasks, respectively. Further, Wikitable (Pasupat & Liang 2015) offers challenge in executing compositional logical queries over semi- structured tables, testing models' ability to generalize across complex tabular reasoning tasks. Recently, InfiAgent-DAABenee (Hu et al. 2024) emerged as the first specialized benchmark for data analysis agents, comprising 311 tasks across 55 CSV files with automated evaluation. Additional data science agent bench- marks study broader workflows, including DSBench (Jing et al. 2024) which evaluates data science tasks with Kaggle and modeloff datasets, DA-Code (Huang et al. (024) which focuses on agentic code generation for data analysis, Spider 2.0 (Lei et al. 2024) which targets real enterprise text to SQL pipelines, Spider2-V (Cao) et al. (2044) which extends these workflows into multimodal settings, and DABStep (Egg et al./ 2025) which benchmarks multi step data reasoning abilities of LLMs. However, these benchmarks' focus on structured datasets limits assessment of real-worke iterative workflows: data science workflows often heavily involve
finding data (e.g., through web navigation), using tools, and analyzing unstructured data et al. 2016).
2025 Rajpurkar
Web Agent Benchmarks Evaluating LLM agents on high-level natural language tasks has traditionally relied on surface-for form comparisons of action sequences, as seen in VirtualHome (Puig et al., 2018) and Mind2Web (Deng al. 2024), but these fail to capture functional correctness. WebArena improves this by evaluating functional correctness but has major limitations, including a lack of scoring granularity and rigid state-based evaluation that is not robust to minor variations (Zhou al. 2024). Recent approaches
3

---

Published as a conference paper ICLR 2026
Dataset
Multhhop
Structured
Castructured
Web Nav
Q4
Multistte
Actions
Tool Use
SQuAD Raiiurrar al.
2016
X
X
v
K
X
X
X
X
Wikitable (Pasupat & lang. Tang.2200 2015
/
X
X
X
X
X
X
HotpotQA Yang el al.
2018
/
X
/
X
X
X
X
X
VebVoyager (He (He et a.
2024
X
X
X
/
/
X
V
WchArena (Zhoü ct al.
20124)
X
X
/
/
/
GAIA (Mialon el al. (0023
X
/
/
/
WebWalker (Wu el 20225
X
V
AssistantBench Yoran
2024)
7
X
WebDS (ours)
/
Table I: Comparison of dataset features across benchmarks. WebDS is the only benchmark that has tasks spanning Multihop, Structured, Unstructured, Web Nav, QA, Multisite. Actions, and Tool-Use.
LLMs like GPT-4o to assess task completion (Pan et al. 2024a). However, these works rely only on final screenshots, making the evaluation binary and incomplete. Lastly, methods like Web Voyager consider the last 15 screenshots, greatly strengthening evaluation, but only gives a binary 'SUCCESSFUL/ WEWSSSRRLLUUSUCCCSSSSSSS UNSUCCESSFUI result, and is unable account for full trajectory in complex tasks (He et al., (024). Our work avoids this limitation by assessing the full trajectory, and allowing for granular analysis, which is especially important for our tasks which are multihop, multisite and generally involve more steps.
3 BENCHMARK DESIGN
this section, we describe WebDS, the first web-based benchmark to focus on the full data science workflow. Our main objective is to produce a benchmark that is useful, comprehensive, reproducible, and granular.
Subject Interviews To ground the design of our benchmark and formulate task definitions, we conducted 8 interviews with journalists, data scientists, and other domain experts to identify recurring data-related needs. From these interviews, we derived two broad categories of tasks: those that (1) culminate in a downstream product and (2) those that address a key analytical question. Within these categories, tasks could involve a variety of data modalities and attributes, including: structured data, unstructured textual and non-textual data, multi-hop reasoning, and weblite data integration, all of which are discussed more in section 4 In light of these goals we have selected set of 29 websites that cover 10 high-stakes data-heavy domains, and wrote by hand 870 tasks for LLM agents these domains.
Domains We chose 29 websites covering 10 high-stakes domains, as shown in In particular, selected websites that included various diverse data formats, including table-structured data and downloadable CSVs well as unstructured data such as text and graphies. Our selection criteria were: (1) websites had be public, (2) they had to contain data and be used commonly in different kinds of data analysis and (3) they had have unique representation of data not found in the rest of our benchmark, or only found in one other website. For these reasons, high scores on our benchmark require an agent to be adept at reasoning and tool usage between different modalities and data representations.
Action Space Gym (Drouin et al. 2024), WebArena). This design allows flexibility: researchers may enable or disable tools (Python, Bash, SQL, etc.) without adapter friction.
We do not fix the action space A, widely adopted abstractions already exist (e.g., Browser-
State and Observation Space The state space consists of website pages and data accessible to the agent, with each state's observation consisting of HTML, DOM, screenshots, and/or AXTrees. Data can be downloaded, and many tasks necessitate the successful downloading and manipulation of data to complete. Each task begins at a fixed entry URL with a natural-language goal, and agents must autonomously discover relevant pages using their own navigation policies. We intentionally do not constrain crawling strategies or actions within each sandboxed-site. enabling users to plug in agents from other frameworks easily and improve compatibility with prior work.
4

---

Published as conference paper ICLR 2026
ARXIV 113 (12.3%)
Thei SOCORRIII
OUR WORLD DATA
ESANI
Gering sudii Favuaatl 1991
Eaergy
Figure 2: Distribution of tasks with respect to domain (rounded to nearest decimal), where each domain is a grouping of websites according to Our domain distribution is chosen based on subject interviews 3
Reproducibility and Realism Our benchmark supports two complementary evaluation regimes. In WebDS- live. agents interact directly with live websites. This track captures the full complexity of real web envi- ronments, including evolving page layouts and continually updated datasets, but may suffer from websites changing. To maintain evaluation stability, we record complete interaction trajectories, including page states, downloaded artifacts, and agent actions, allowing runs to be audited and analyzed post-hoc.
parallel, WebDS-dockerized provides containerized deployments of subset of benchmark websites whose content and structure are preserved at the time of benchmark construction. By freezing these environments within Docker containers, we enable deterministic execution and exact reproducibility of experiments. This dual-track design allows WebDS to balance realism with scientific rigor.
Evaluation Granularity Although prior benchmarks have provided some granularity in evaluating Al agent performance, we are the first provide evaluation scores along three different dimensions: (1) Domain-wise, along different professional specializations (e.g.. performance on demographics-relate websites, sports- related websites, etc.) (2) Attribute-wise, along different types of challenges for LLM agents which we call "task attributes" (e.g., multihop, action-based, etc. see more in section 4 (3) Difficulty, along easy, medium, hard task complexity.
Release Strategy and Benchmark Integrity To mitigate overfitting while preserving reproducibility, we release a 470-task public validation set and hold out 400-task private test set. Leaderboards and official evaluations use rotating subsets the private test set, where we publish prompts and accept predictions (JSON) for scoring Huggingface. We may periodically refreshhexpand the private test pool.
4
BENCHMARK TASKS
4.1 TASK FORMULATION
We define web data science task as: query which requires navigation within EH: browser-like environment to learn raw information that is then transformed to predictive summarized representation" Optionally, there can be a downstream action based on the resulting predictive representation. More formally, let: W denote the environment (e.g., the space of websites the space of data science queries, denote the raw data space extracted from W, y denote the space of analytical outcomes (e.g., reports, models, visualizations). and A denote the set of downstream actions, with
dynamic web environment),
denote
5

---

Published as a conference paper ICLR 2026
E AA indicating no action. We define web data science task as a composite mapping f W Q - O' x A, where an agent, given browser environment W and natural language query g E Q returns an analytical outcome y E y and/or an optional downstream action a E A. This overall process decomposes into three
insess-inoootiioon gathering, data analysis, and optional cctton-c by the composition -= Sa o fa o fa. the first stage, Sa : W 2 + D, the agent interacts with the environment to extract relevant raw data. In the second stage, Ja D - V. the agent transforms this data into an analytical outcome, potentially through internal steps such as representation learning, tool usage, and prediction. In the final stage, fa A. the agent maps the outcome to downstream action- or none all (a Ø).
4.2 TASK GENERATION
To generate data science tasks that meet formal definition, given in Section 4.1 we first identified 100 candidate websites that were freely accessible. Then, we selected final set of 29 websites to ensure broad coverage of data modalities, enable the construction multisite tasks, and represent diverse range of task attributes. Finally, we grouped these websites into domains with similar content and structure, as shown in Figure 2 Using this curated set of websites, we wrote total of 870 tasks, a simple example of which can be seen in Figure
Tasks were authored by primary annotators (paper authors). Every task underwent at least one secondary review for (i) solvability in the live site (at time of publication) and containerized deployment (if site containerized), (ii) clarity of instructions, and (iii) correctness of references/ground truths; lead authors conducted a final pass for consistency and coverage.
In authoring the tasks, we aimed to achieve balanced distribution of difficulty levels, comprehensive coverage of the identified attributes, and strict adherence to our task definition. Task writing emphasized diversity, the inclusion of verifiable ground truths, and the ability to evaluate wide spectrum of real-world data science capabilities. We find that there is substantial diversity in task difficulty: some tasks can be completed through straightforward web searches, while others require synthesizing information from heterogeneous sources and applying a range of statistical or analytical techniques.
4.3
TASK ATTRIBUTES
Our tasks mirror real-world scenarios where a data analyst or general-purpose assistant must retrieve or compute information (which we call Question-Ansse Answering tasks) or perform downstream actions (which we call action- based tasks). Some tasks remain straightforward, requiring only a single data source or minimal steps, while others demand multi-hop reasoning, multi- browsing, structured or unstructured data handling, and the integration of external tools.
We manually label each task with one or more of the following 7 labels, or "task attributes":
Question-Answering (QA) Tasks where the system provides a definitive answer to user query (e.g.. compute the combined total assets for two funds). These tasks have verifiable ground truths.
Action-Based Tasks that require the system to perform one or more actions after gathering and analyzing information. An example would be analyzing multiple data sources to rank financial indices and then posting an investment recommendation on Reddit.
Single-hop vs. Multi-hop Single-hop tasks rely on a single data source find the answer, while multi-hop tasks require combining information from multiple sources in the same website.
Structured vs. Unstructured (textual) vs. Unstructured (contextual) Tasks. Structured tasks utilize data organized in a predefined format (such as databases). Unstructured (textual) tasks involve raw text data (e.g., documents or free-form responses), where extracting meaning requires natural language processing. Un- structured (other) tasks involve data without a clear textual representation. These tasks require interpretation of external knowledge, images or other multi-moda. inputs.
Tool Usage Some tasks necessitate the use of external tools (like Python scripts, Wolfram Alpha, or SQL queries) to perform computations process information.
6

---

Published as conference paper ICLR 2026
Task Category
Description Example
Properties
Single-hop QA (344 tusks)
"Whaz the lowest AAP stock price 2000 26202"
Single Question-Anrwering
Wehnav tiikety)
Pessibly structured data
usage
Multi-hop QA (117 tasks)
"For cach let the uumber of times their appears York urticles between tstaus datel and lend dutel. Fond the average and stundard deviation of X; for all people mentioned"
Multiple data sourves (articles)
OpeetionAAnvveran
Unstrueured data
Toal (e.g.. Python)
Web navigation
Singlo-hop Action luskst
"Delete page."
the
Yoke'
Single
Roguires (dellete)
Web navigation:
Multi-hup Action (134 tasks)
"Buy the with cheqpes: laptng ofter comparing pricez across Store A. Store B, and Store C:"
Multiple data sources (stoees)
Requires action (purchase)
Multi multi website
navigation
Action Tool Usage
"Compute average people who got [imerval)
Seructured (cov)
(1.99 tasks)
using CDC date iere fil) and post the resalt on Reddi."
tool usage (DB query, analytics) Action (post Reddit) Web navigation
Table 2: Illustrative examples of QA vs. Action-Based tasks, showing various properties (single- vs. multi-hop, structured vs. unstructured data, web navigation, and tool usage).
Difficullt
Task Amrilatees
Easy (247 taske) Medium (275 tasks)
Does not involve any of following propertess mullihop. nontettual unciructured data, bloo-- tacks, noll-use and single-wehsite. lnvolves evartly one of the above properties and single website.
Hard (348 tusks)
Involves at least the above pruperties multi website.
Table Task difficulty categorization based on structural and content properties. We find that agents on average perform 2.5x better on easy tasks compared to medium or hard
Web Navigation (Webnav) Tasks that involve interacting with whether querying a database online or making a post on a forum forum-relying on navigating through web pages.
Multi-website Tasks Tasks that require gathering information carrying out actions across multiple websites, where each site contributes separate pieces of the required data or functionality.
We then use these attributes to categorize tasks into three levels of difficulty: easy, medium and hard, based specific attributes that reflect increasing cognitive and operational complexity, per Table See Table 2 for more detailed breakdown of task-categories, but note that because these attributes are multi-label, some attributes (e.g., "Tool Usage") are multiple categories. See Appendix 8.I Table for more complete counts of tasks.
5
EXPERIMENTAL SETUP
5.1 EVALUATION PROTOCOL
Our evaluation framework combines automated and subjective assessment to robustly evaluate agents in complex, multi-step tasks. In particular, we provide two evaluations for each task, an automated binary evaluation and subjective 1-5 score given by LLM judge.
For tasks with reference ground truths (e.g., extracting factual statistics) we assign binary labels: SUCCESSFUL / UNSUCCESSFUL, These are determined by a language model comparing the agent's final output or action to ground truth.
Automated Evaluation
7

---

Published as conference paper ICLR 2026
Metric
5 LLM Judge Runs
Perfect status agreement
96.3%
Perfect score agreement
91.3%
Low variance (0 < stdev S 1) High variance (stdev 1)
6.1%
2.6%
Table 4: Stability results across five runs of the evaluation harness.
Subjective Evaluation via LLM To evaluuat open-ended tasks (e.g., generating Reddit post with statistics),
we build on the LLM-as-a-Judge paradigm introduced by Web Voyager (He et al., 2024), which classified outcomes based solely on the final 15 screenshots of trajectory. We extend this method in two key ways (1) we have the LLM provide a more informative 1-5 integer score, along with rationale and failure analysis, enabling more granular agent iteration; and (2) instead of relying on the final states alone, we evaluate the full trajectory by summarizing each observation and analyzing (observation, action, next observation) triplets throughout the task, resulting in more reliable and interpretable scores, which are assigned as follows:
fully satisfies G mosily satislies G partially majar errurs, some progress majar emors, no progress
Rilm (ST) =
Human Validation of Evaluation Harness To validate subjective scoring method, first conducted human verification study. We had CS undergraduate annotators independently rate a random sample of 400 task-trajectory pairs and found that our evaluation harness achieved 93% status agreement with the human
assessments.
Stability Validation Evaluation Harness Secondly, we ran our scoring harness five times across 800 tasks for the best performing BrowserUse agent. To verify the robustness of the harness, computed both status agreement (SUCCESS vs. NOT SUCCESS) and score agreement across runs. For score agreement, all 5 runs are not in perfect agreement, low variance classified as the scores being generally within +-1 point of each other (standard deviation the 5 scores being less than 1). Results are summarized in Table The standard deviation of per-run average LLM scores was 0.005, indicating very high stability and almost no variability.
look at the standard deviation of the scores across all 5 runs, with
5.2 MODELS
We benchmarked nine state-of-the-art (SOTA) web agents to evaluate performance newly proposed benchmark. To ensure consistency with prior work, we first evaluated the general-purpose GPT-4o and GPT-4o-mini agents using accessibility trees observations, following the exact implementation described in WebArena (Zhou et al.) 2024). Using the same observation space and implementation pipeline, we additionally benchmarked Claude Sonnet-3.5, Claude Haiku-3.5, Dwen2.5--2BBInnttee Claude Sonnet-4.5, GPT-5.1. To explore multimodal capabilities, we modified the observation space to incorporate full-page screenshots and evaluated Qwen2.5 5- 1-.55--LL-7B VL--2L--Innttte We also included AgentOccam, the SOTA SoT.Aoopen-ssurr configuration used for WebArena without altering its implementation. Finally, we evaluated BrowserUse (Müller & Žunië 2024), the SOTA² agent on Web Voyager via both Qwen2.5-VL-720 - and GPT-4o. We made minimal adaptations to the observation and action space to better accommodate the structure of our landing pages and to improve handling CSV files and file downloads.
under this setting.
agent WebArena, benchmarking it under the official
La of 5/15/2025
8

---

Published as conference paper ICLR 2026
Berchmarks
Web Veyager
Mind2Web
Accurscy (N)
Web05 (Ours)
60
40
AgontOccam
GPT T
Browser Use
Figure 3: Agent Performance on Various Benchmarks. Models perform much worse on our benchmark (e.g., BrowserUse + GPT-4o achieves 81.9% on WebVoyager but 12.9% on WebDS), Note: WebVoyager Performance based on GPT-4, as no verified GPT-4o results exist (ee et al. 2024..
5.3 HUMAN PERFORMANCE ON BENCHMARK
We recruited six participants with prior data-science experience. Participants were given the same browsing environment, websites, and tools as the evaluated agents, and were allotted at most 30 minutes per task. Note that the participants were using the live sites, which at the time of experiment were exactly identical to dockerized sites. All participants were familiarized with the interface and permitted one warm-up task for acclimatization before evaluation. Each was then assigned one of two disjoint, randomized subsets of benchmark tasks with reference ground truths. During evaluation, participants followed the same protocol agents: starting from the specified URL,, completing all navigation, data collection, and analysis steps directly in the browser environment.
RESULTS
Success Rate (SR%)
Score (1-5)
Agent
Overall
Easy
Medium
Hard
Overall
Easy
Medium
Hard
GPT-4o
1.0
1.7
0.4
0.9
1.111
1.187
1.073
1.087
GPT-4o-mini
0.8
1.2
1.1
0.3
1.087
1.091
1.097
1.075
Sonnet-3.5
1.3
1.087
1.054
Haiku-3.5
1.6
2.4
1.5
1.2
1.135
1.200
1.119
1.102
Qwen2.5-v1-722
0.2
0.0
0.4
0.3
1.050
1.101
1.037
1.024
Qwen2.5-72b
1.2
2.5
0.8
0.6
1.157
1.273
1.140
1.085
AgentOccam
4.8
11.3
1.658
2.121
1.417
1.524
BrowserUse (Qwen2.5-72b)
13.2
21.9
11.0
8.6
2.044
2.385
1.993
1.841
BrowserUse (GPT-40)
12.9
21.9
7.0
11.2
2.171
2.490
1.934
2.133
BrowserUse (GPT-5.1)
22.2
38.1
15.1
14.3
2.144
2.682
1.916
2.102
Table 5: Success Rate (SR%) and Score (1-5) of 10 models on WebDS-live benchmark. BrowserUse (GPT-5.1) performs the best with success rate of 22.2%. most models have below 2% Success Rate. Success rate is measured by the number of tasks evaluated as SUCCESSFUL and score is given by an LLM as judge, as defined in section 5.T If only the model name is specified, the model run on base framework given in the WebArena Base Agent.
Despite strong performance prior benchmarks, all agents perform poorly WebDS. Agent for example, achieves 45.7% 7% success on WebArena but only 4.8% on WebDS. Likewise, BrowserUse + GPT-4o, which surpasses OpenAl's Operator and Claude Computer Use with 89% drops to just 12.9% on WebDS (even with our adaptations; see Section 8) This sharp decline underscores that web-based data science poses fundamentally different challenges from those captured by existing benchmarks.
Web Voyage,,

---

Published as a conference paper at ICLR 2026
Fallure Theme
Example
Analysis
Navigation
The model novigsted the American Physical Therspy Association instead the American Public Transpotttionn Association.
The model confuses similarly named entniies interneting with irrelevant domains. lacks validation sieps check information sounce aligns with the query.
Feedback
The model repeatedly tried to search fiekd "Abstract" system. failed to registe: this action.
model cannot confirm whether UI manipulatios successful. This leads to broken search flows and inconptete query execution.
Groudeddnsss
The model failed mention 12% in genentive AI, despite viewing the correct mens
model fails to extract key details and indesad provides vague summories This shows gop between infocmation access and correct syathesis
Falled Repetition
The model retries the same search-with- fireer after UI feedback indicates
Lacks loop-brcaking and state checking heursties; does not verify that the previous action changed page state bedore retrying.
the fller did not apply, repeating for dosens of stepc.
Query Interpectation
Asked for 30% publicettio growth figure but responded with general reseurch tremds.
The model miundersttnnds that task require cortain kinds of reasoning (eg, specific mumeric trend, not qualitative description).
Effort Allocation
Couldn't 't retrieve release count for artist via MusicBrainz, then guve incorvect Wikipedia total.
The model anempts use a tool, but after failing, simply guogled the lask und wrung because Wikipedia records snique releases ruther than relenses.
Table 6: Taxonomy of model failure themes with representative examples and underlying causes.
Interestingly, BrowserUse with Qwen2.5-72B outperforms the same setup with GPT-4o, suggesting that raw model capability is not the primary bottleneck. Instead, limitations arise in the translation layer between reasoning and environment interaction, as evidenced by frequent UI handling failures in our error analysis. These findings indicate that progress may depend more on improving control and interaction fidelity than on scaling model capacity alone.
Humans, by contrast, achieve an average success rate of 90%, establishing a human-agent gap of over 75 percentage points. Human participants were able to flexibly adapt strategies, validate intermediate outputs, and recover from interface quirks, which were behaviors current agents lack. This contrast confirms both the solvability and realism of WebDS tasks while exposing critical limitations of today's agents on long-horizon, end-to-end workflows.
7 ANALYSIS
Failure Analysis We evaluated the reasons behind the failure of these LLM agents, and looked for key reasons behind their failure. Most failures stemmed from weak grounding, poor feedback handling, and misinterpretation of user intent, more details can be found in Table 6
Benchmark Longevity We also design WebDS to stay relevant and avoid saturation by (i) covering diverse domains, data types, and three difficulty tiers, (ii) implementing dual evaluation regime, including fixed dockerized subset enabling reproducible comparisons across models and future research even when real websites change over time, (iii) offering an extensible task-creation workflow that keeps the benchmark "evergreen," and (iv) remaining hard enough that humans still outperform the best current models, leaving ample headroom for progress. See Appendix (.ooddx88.6oor for full details.
8 CONCLUSION
WebDS introduces a novel benchmark that rigorously evaluates Al agents on complex, web-based data science tasks. By encompassing structured and unstructured data analysis, multihop reasoning, tool integration, and action-based workflows, it fills a critical gap in current web-based agent evaluation frameworks. Our preliminary results reveal significant performance shortfalls in SOTA agents performance on our benchmark compared to current web agent and data analysis benchmarks. We also show a large gap between current agents (S 13% success) and human baseline (90%). quantifying how far we are from reliable end-to-end autonomy on the web. We hope that WebDS provides a robust target for closing this gap and advancing practically useful web-based data-science agents. Future work in this area would include improving task diversity to encompass more action-based tasks and expansion to OS-level tasks or other enterprise workflows.
10

---

Published as a conference paper ICLR 2026
REFERENCES
Ruisheng Cao. Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yegiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 17:107703-107744, 2024.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How capable are web agents solving common knowledge work tasks? In Proceedings of the 41st International Conference on Machine Learning, pp. 11642-116622 2024.
Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. DABstep: Data agent benchmark for multi-step reasoning. arXiv preprint arXiv:2506.237/1111 2025.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. Proceedings of the
62nd Anmual Meeting of the Association for Computational Linguistics, 2024.
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An Ilm agent for data science. arXiv preprint arXiv:2402. 18679, 2024.
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu et al. InfiAgent-DABench: Evaluating agents on data analysis tasks. arXiv preprint arXiv:2401.0553 2024.
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang. Xiao Liu, Jun Zhao, et al. Da-code: Agent data science code generation benchmark for large language models. Empirical Methods in Natural Language Processing, 2024.
Jacqueline A Jansen, Artür Manukyan, Nour Al Khoury, and Altuna Akalin. Leveraging large language models for data analysis automation. PloS one, 20(2):e0317084, 2025.
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents from becoming data science experts? arXiv preprint arXiv:2409.07703, 2024.
Sean Kandel, Andreas Paepcke, Joseph M. Hellerstein, and Jeffrey Heer. Enterprise data analysis and visualization: An interview study. IEEE Transactions on Visualization and Computer Graphics, 18(12): 2917-2926, 2012. doi: 10.1 1..100922....22..221.
Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, et al. Spider 2.0: Evaluating language models real-world enterprise text-to-SQL workflows. International Conference on Learning Representations, 2024.
Qunhua Li. An analyst-inspector framework for evaluating reproducibility of LLMs in data science, 2025.
arXiv preprint arxiv:2502.163:
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint 2rXiv:2005.03776, 2020.

---

Published as a conference paper ICLR 2026
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia; a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Magnus Müller and Gregor Zunië. Browser use: Enable ai control your browser, 2024. URL https
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.064774 2024a.
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406. 12373, 2024b.
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.000 00305, 2015.
Dale Peasley, Rayus Kuplicki, Sandip Sen, Martin Paulus, et al. Leveraging large language models and agent-based systems for scientific data analysis: Validation study, JMIR Mental Health, 12(():e68135, 2025.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8494-8502, 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint /606.0/000.55220 2016.
Guangyuan Sun, Tanja Friedrich, Kathleen Gregory, and Brigitte Mathiak. Supporting data discovery: A meta- synthesis comparing perspectives of support specialists and researchers. arXiv preprint arXiv:2209.14655, 2022.
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Deyu Zhou, Pengjun Xie, and Fei Huang. Webwalker: Benchmarking Ilms in web traversal, 2025. URL lttpps/ss///xxi abs/2501
Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for LLM-based web agents, 2024. URL | https
//axxiv.org/ab/t 2410 13825
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: /00:: 099.00600, 2018.
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assis- tantbench: Can web agents solve realistic and time-consuming tasks? arXiv preprint arXiv:2407.15711, 2024.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. Proceedings of the 12th International Conference Learning Representations, 2024.
12

---

Published as conference paper ICLR 2026
APPENDIX
8.1 DATASET STATISTICS, MORE GRAPHS, AND FURTHER DETAILS
200
Marber
Hastog
Dructondt
Instrnctinn Rlor)
Coveitary
Figure Counts of tasks per attribute
Difficulty Distribution
350
35
Number Tools
250
30
25
200
20
15
100
Easy
Medium Diffiulity
Hard
Figure 5: Counts of Task Difficulty
13

---

Published as conference paper ICLR 2026
Key Action- Banon-Basdd
Tool Usage
Multihop
Task Olssiifictiion
the task muttisit??
No
Does the Task have >-2 key attributes
No
Does task contain key attributte?
No
Easy Task
Yes
Yes +
Yes Medium
Hard Task
Figure 6: Flow chart for evaluating task difficulty
Domain
Websites
Beunomicc/Markets
BEA, FRED, ST LOUIS FED. TRADING ECONOMICS, DATAUSA
Demagraphics
WORLDPOP, WORLDON DOMMETER
Music
TUNEBAT, MUSICBRAINZ, RIAA, RICHARD POWERS
Tourism, Trade, and Aitlines
UNWTO, IATA
Higher
UCHICAGO
Scientific Research
ARXIV
Sports
UNDERSTAT
Government and Public Policy
CTPB, OUR WORLD DATA, REDDIT
Energy and Climate
CLIMATR GOV, NOAA
Health
CDC MENTAL HEALTH, CDC COVID, CDC OBESITY, NIH
E-Commerce
SHOPPING, STOCKNEAR
Table 7: Mapping from Domains to Websites. We note that the Reddit website and Shopping website are the same in WebArena, but our tasks are unique.
8.2 SCORES FOR SPECIFIC DOMAINS
AgentOccam
Qwen2.5722
Browser Qwen2.5
GPT-6o
Demographics Policy
E-Commence Forams
2.2
0.0
0.0
2,2
2.2
2.2
2.2
13.3
Romomiss/Marrsss
1.0
0.0
3.0
3,0
80
3.0
21.8
28.8
20.7
27.6
34.5
86
18.5
Higher Education
0.0
0.0
0.0
B.0
0.0
800
4.0
Music
1.0
1.0
1.0
8,7
Sclentifc
12.4
Sports
17.6
Tourism, Trade, Airlines
1.2
0.0
2.4
1.2
9.6
1.3
14.5
18.1
Aitribute
GPT-4u
GPT--u-mini
Summet-3.5
Haiku-3.5
AgentOcaam
Qwen2.5-726
Bruwser Qwen2.5
Brewser
Multihop
0.7
1.4
8.7
Unstmucturad (lext)
0.8
044
1.9
1.2
7.0
1.6
15.8
18.1
Unernurrarll (oontext)
1.2
1.2
0.4
2,0
1.2
8.9
10.1
Question
Website
177
0.9
Action-Based
09
B4
D.D
0.9
0.5
7.9
13.2
Tuol
0.3
0.9
0.6
0.3
80
14

---

Published as a conference paper ICLR 2026
Table 8: WebDS benchmark performance: domain-wise and attribute-wise success rates (%).
8.3 BROWSERUSE AGENT IMPLEMENTATION
We build on the open-source BrowserUse framework (snapshot: April 22nd, 2026) and introduce several mod- ifications to increase robustness our benchmark. This appendix summarizes the final agent configuration used in all experiments.
A. Task Initialization and Control Loop Each evaluation episode is defined by a JSON configuration specifying a starting URL and natural-languag intent. This intent is injected directly into the system prompt as the agent's "ultimate task," and the model must terminate using a done action once this task is completed. At every step, the agent receives the browser state (URI., rendered content, DOM/AXTree structure, screenshots) and must respond using a structured Agent object containing:
1. an assessment of whether the previous action achieved its goal,
2. short memory field tracking progress, and
3. the next immediate sub-goal.
This structure enforces continual micro-planniin and self-evaluation, even without an external planning module.
B. Action Space The agent interacts with the environment through high-level semantic actions exposed by BrowserUse, including clicking indexed elements, scrolling, typing, issuing search queries, switching tabs, and extracting text or structured content. We expand the system prompt to list all available actions explicitly, we observed that the default prompting often caused the model to overlook useful but less frequently used actions.
C. Enhanced Observation Space via AXTree Indexing The default BrowserUse implementation relies on a heuristic for identifying clickable elements that frequently misses important links on real webpages. To avoid navigation failures, we replace this mechanism with an AXTree-based traversal that assigns unique index every interactable anchor element. This guarantees that all visible links become addressable by the agent, substantially improving coverage and stability in multi-step navigation tasks.
D. Lightweight Memory BrowserUse includes an optional procedural centuurrllmmmryyy compression module, but we disable it for our experiments. Instead, the model maintains its own high-level task-state ledger in the memory field (e.g.. "3 of 10 pages checked").. For our task lengths, this lightweight in-context memory proved sufficient to maintain coherent long-horizon behavior.
CSV Detection and Data Tools Many benchmark tasks require analysis of structured data presented large CSVs on webpages. To support this efficiently, we extend the agent with:
1. automatic detection of CSV-like page content,
2. conversion of such content into local DataFrames,
3. pickle-based caching for fast re-access, and
4. dedicated actions allowing schema inspection, row extraction, and execution of safe Python analysis code.
These tools allow the agent perform nontrivial data-analysis operations without requiring the model to ingest large tables token-by-token.
G. Parallel Execution Evaluation is performed using an asynchronous worki-worker pipeline. Each worker runs an independent agent and pulls tasks from shared queue with retry logic. When multiple API keys available, they are distributed across workers to avoid rate limits. Full iecittorree-inlluuii browser states, screenshots, and agent outputs-an are logged for post-hoc analysis.
15

---

Published as conference paper ICLR 2026
8.4 COMPUTE USED
We used 4x 16-core Xeon CPUs with 256 GB of RAM between them to agents in parallel massively, No GPUs are needed as we make direct calls to provider APIs (no locally hosted models), and running these tasks are CPU and I/O constrained.
8.5 FORMALIZATION OF AGENT APPROACH
1. Navigation and Information Gathering The agent interacts with the environment to extract raw data relevant given query:
Ja W Q - D.
(1)
Note: In dynamic settings, W may be modeled as an interactive state space or Markov Decision Process (MDP) to capture sequential interactions.
Representation Learning and Outcome Prediction The agent processes the raw data to obtain an analytical outcome. This stage is further decomposed into:
a. Representation Learning Transform the raw data d E D into an intermediate representation that retains only the information necessary for prediction:
Jr T.
b. Outcome Prediction
Map the learned representation T to an analytical outcome y:
fy +
(3)
Together, the above steps (a) and (b) define the mapping from data some analytical outcome:
fa D + V
(4)
3. Optional Downstream Action Based on the analytical outcome y, the agent may execute a downstream
action:
fa : + A. with indicating that no further action taken.
(5)
HTML
The HTML structure of the web pages.
DOM - The Document Object Model representation.
Screenshots Visual representations of web pages.
AXTree Accessibility tree structures for web elements.
Downloaded Data Unlike prior benchmarks, setup allows for downloading data, which is then loaded into the LLM's context. As a result, the LLM can also observe structured data formats such
ISON
CSV
Other structured data formats.
8.6 MORE DETAILS ON LONGEVITY
1. Fine-Grained Difficulty and Domain Categorization. WEBDS evaluates agents across multiple axes, including task difficulty, data representation (structured, unstructured), and domain (e.g., demography, finance, public health). This multi- dintisioian evaluation reduces the risk of models overfitting to a narrow set of tasks and provides continued granular insight even as models improve over time.
2. Containerized and Stable Evaluation. By having dockerized track in WebDS, eliminate the instability associated with live web environments. This ensures that future researchers can reproduce results exactly and fairly compare new methods against older baselines, even years after initial release.
16

---

Published as a conference paper ICLR 2026
Evergreen Task Format and Modifiability Data science tasks inherently offer broad extensibility: new queries, updated data sources, and evolving analytical goals mean that researchers can continuously generate new tasks using the same framework. WEBDS supports community-doe enable users to create, classify, and evaluate new tasks and domains with minimal overhead. In particular, this can be adapted to tackle many other enterprise workflows, showing huge potential as a benchmark to evaluate general AI agents. For example, this can be used to add specialized domains like scientific computing or finance the community finds it useful, with the same Dockerization and rubric-based evaluation.
extensions through tools that
4. High Task Complexity. Even current frontier models perform poorly, indicating a wide margin for future progress. Additionally, human performance is non-trivial on many tasks, underscoring the benchmark s difficulty and its grounding in real-world challenges.
8.7 TASK EXAMPLES
To further illustrate the breadth and depth of the WebDS task suite, we provide representative examples of tasks spanning data gathering, cleaning, transformation, visualization, statistical analysis, and modeling. These examples highlight how WebDS goes beyond retrieval or surface-level question answering to evaluate end-to- end data science workflows. Herein, we provide some examples of tasks. For long tasks, we occasionally provide source context in front of the task which is not shown here for brevity.
Data Gathering, Cleaning, and Preprocessing. Many tasks require the agent to extract heterogeneous data from dynamic web interfaces, resolve formatting inconsistencies, and organize the results into usable structured formats. Examples include:
Extract the most difficult positions to fill in the March 2022 public transit workforce report and organize them into an Excel spreadsheet."
Align unemployment and COVID-19 time series for all countries with GDP per capita above specified threshold and normalize the series for comparison."
"dentiffy all product volumes that appear more than once after applying a multi-stage filtering pipeline."
Rank global GDP levels after cleaning and merging country-level datasets with mismatched units and missing values."
" "Clean long-horizon literacy or poverty datasets spanning multiple decades and resolve missing entries and inconsistent measurement units."
These tasks require structured data extraction, multi-step cleaning, schema reconciliation, and the ability to construct stable tabular outputs.
Visualization and Chart Construction. A large subset of tasks evaluates agent's ability to produce correct and contextually appropriate visualizations. Such tasks include:
Visualize literacy trends the United States and discuss long-run patterns."
Plot multi-city line graph of transit ridership using data from major U.S. metropolitan systems."
"Generate a chart summarizing MIT's undergruduate financial aid statistics and annotate notable trends." "
Solving these tasks requires the agent to retrieve numerical data, structure it uppropriately, select a suitable visualization type, and execute the visualization using an external tool.
Statistical Computation and Analysis. WebDS also contains many tasks assessing competence with descriptive and inferential statistics, requiring nontrivial numerical computation. Examples include:
"Compute the variance of BPM across top-charting music tracks."
"Calculate the standard deviation of average schooling hours from 1900 2005."
" Evaluate efficiency statistics for high-income countries using multi-dimensional indicators."
17

---

Published as a conference paper ICLR 2026
"Compute correlation coefficients and cross-correlations (with optimal lag) between student spending and literacy outcomes."
Such tasks require the agent to derive correct numerical values, interpret temporal structure, and integrate multi-source data.
Predictive Modeling and Regression. number of tasks in WebDS explicitly require model fitting and forecasting using linear, multivariate, or exponential models. Illustrative examples include:
" Model literacy rates using per-capita spending and healthcare expenditure as predictors, and assess model fit."
" a linear regression linking student spending and GDP from 1990 to 2020 and interpres the resulting coefficients."
Perforrm a regression between product weights and prices and evaluate the strength of the relationship." "
Fit an exponential growth model to African tourism arrivals and forecast values for 2030, comparing the result to a linear model."
These tasks evaluate the agent's ability to extract relevant features, apply statistical models, and interpret predictive outputs.
Multisite Data Gathering and Cross-Domain Synthesis. A core feature of WebDS is the requirement to gather data across multiple websites, often with distinet modalities, and to synthesize them to produce a coherent output. Examples include:
"Compare literacy rates between Latin America and the Caribbean and the rest of the world, compute interquartile ranges over the last fifty years for euch region, and visualize the trends."
For countries with GDP per capita above 14,000 USD, compute the average maximum cross-correlation and corresponding lag between unemployment and COVID-I9 cases."
"Extract the most difficult positions to fill in the March 2022 public transit report, compile them in an Excel spreadsheet, and produce relevant chart within the spreadsheet."
" Develop Python script to cluster adults by reported mentally unhealthy days using National Health Statistics Reports data."
" Analyze regional tourism arrivals for 2010, 2015, and 2020; fit an exponential growth model for Africa: predict arrivals for 2030; and compare the forecast to a linear growth model."
These examples highlight scenarios where agents must integrate multi-source information, perform multi-hop reasoning, and execute downstream analytical or programmatic actions.
Interpretation of Technical Content. Finally, WebDS includes tasks assessing the ability to read tables, interpret scientific figures, extract quantitative findings from academic articles, and synthesize domain knowledge, with the ability to do this across a wide variety of figures, papers, time periods etc. Examples include:
Interpret ablation tables from a specific machine learning paper and computing relative performance differences
Extract and compare model performance from figure-based visualizations and describe accuracy trends Identify patterns in ArXiv submission metadata from specific time periods and report anomalous trends These tasks assess high-level reading comprehension, technical grounding, and cross-modal interpretation.
Summary. Together, these tasks demonstrate the breadth of operations WebDS is designed to evaluate, spanning raw data acquisition, multi-website synthesis, structured cleaning, statistical computation, visualiza- tion, and modeling- They reflect real-world analytical workflows that require coordinated reasoning, tool use, and multi-step execution, and therefore serve as a rigorous stress test for modern web-based data science agents.
18

---

Published as a conference paper ICLR 2026
8.8 EXPANDED FAILURE ANALYSIS
Failure Mode
Count
Proportion (%)
Groundedness
309
40.2
Query Interpretation
221
28.8
Effort Allocation
97
12.6
Failed Repetition
49
6.4
Other
42
5.5
Navigation
34
4.4
UI Fecdback
16
2.1
Table Distribution of annotated failure modes across 764 failed trajectories (not all failures are from the same model).
Section we introduced a high-level taxonomy of agent failures on WebDS. Here, we provide more detailed appendix-style analysis of these error modes and their relative prevalence, based on manual annotation of 764 failed trajectories We group errors into seven themes, aligning with the qualitative examples in Table
Groundedness (40.2%) The most common failure mode involves incorrect use of information that is actually available to the agent at some point along its trajectory. In these cases, the agent successfully reaches the relevant page or document but either: (i) mis-reads key numerical or categorical details, (ii) omits crucial values (e.g., reported 12% bias figure), (iii) hallucinates facts not supported by the source (iv) uses information from similar but not correct document or dataset, or (v) does not ground final action in an analysis of the data. Typical instances include reporting incorrect statisties from graphs; summarizing a policy document while attributing it additional constraints not present in the text; conflating similar quantities (c.g., total vs. unique counts); or not correctly using data from a document had visited carlier. These errors indicate that the bottleneck is not necessarily just access to information, but also that agents often fail to grounded answers, especially when dealing with dense tables, multi-column CSVs, or long reports.
Query Interpretation (28.8%) The second- most prevalent category arises when the agent fundamentally misinterprets the user's intent, even if its subsequent reasoning is internally consistent. Common patterns include:
Responding with qualitative trend descriptions when the task requires a specific numerical quantity (e.g.. 30% growth figure).
Treating meta-requests ("extract all links," "compare these two releases") open-ended explanation tasks.
Answering only a subset of multi-part query or focusing on the wrong entity or span of time.
Provides action where analysis is required, provides analysis where action is required.
These cases reveal limitations intent parsing and constraint tracking. The agents often latch onto a salient fragment of the prompt while ignoring explicit requirements about outputs, granularity, or evaluation criteria.
Effort Allocation (12.6%) Effort allocation failures capture mismatches between task complexity and the level of reasoning or exploration the agent performs. At one extreme, agents produce overly terse responses to multi-step analytical tasks (e.g., inspecting only the first table they encounter and drawing conclusions from partial evidence). At the other extreme, they sometimes over-elaborate simple tasks, introducing unnecessary speculation instead of extracting a small set of requested fields. A recurring pattern
*Counts: Groundedness (309, 40.2%), Query Interpretation (221, 28.8%), Effort Allocation (97, 12.6%), Failed Repetition (49, 6.4%), Other (42, 5.5%), Navigation (34, 4.4%), UI Feedback (16, 2.1%).
19

---

Published as a conference paper ICLR 2026
is early abandonment of difficult subproblems: after encountering friction with a primary data source (e.g., a structured API-like site), the agent shorteuts to secondary, less reliable source (e.g.. general search or a more approximate summary page), leading to incorrect results. This suggests that current agents lack robust strategies for budgeting search, analysis, and verification effort over long-horizon tasks.
Failed Repetition (6.4%) Failed repetition errors occur when agents get stuck in behavioral loops or repeatedly apply strategy that has demonstrably failed. Examples include:
Issuing the same search- search-with-file with-filter action multiple times after UI feedback has already indicated that the filter did not apply.
Revisiting the same page and re-reading the same content without updating internal hypotheses or trying alternative navigation paths.
Scrolling without trying another action when page doesn't 't load
These failures highlight weak coop-breaking heuristics and poor use of negative feedback. The agents rarely maintain explicit "this didn't t work" state and therefore lack mechanisms to systematically diversify strategies.
Navigation (4.4%) Navigation failures are defined by "reaching the wrong entity, dataset, or website section, even when relevant resources are present within the environment". For example, agent might navigate to the American Physical Therapy Association instead of the American Public Transportation Association when resolving acronym, or select unrelated subsection of site instead of the specified dataset. This reflects poor ability to disambiguate and over similar datasets and sites. Although less frequent overall, they are particularly problematic in multi-site and multihop settings.
UI Feedback (2.1%) Ul-feedback issues arise when agents fail to correctly infer whether an attempted interaction (e-g., setting a filter to "Abstract"") has succeeded. these cases, the agent continues planning subsequent steps under the assumption that the interface is in desired state, even when the page shows no change. This mismatch leads to broken search flows, empty result sets, or repeated no-op actions. While less common, it still indicates that robust web agents must reason not only over high-level goals, but also over the state transitions induced by actions, including recognizing when an interaction has no effect. Future work may benefit heavily from robust way to predicting the generat effect of an action.
Other (5.5%) The remaining cases fall into a heterogeneous "Other" category. These typically involve minor formatting issues, incomplete outputs that are not clearly attributable to one specific category, or edge cases where multiple error types co-occur without a dominant theme.
Aggregate Takeaways Two conclusions follow from this distribution. First, the dominant failure modes- Groundedness and Query Interpretation-are fundamentally about long-horizon control and alignment; in particular, reasoning with available evidence and user intent, over a long sequence of decisions and history of relevant data. That is, agents often reach the right place but fail to do right thing. Second. the remaining modes (Effort Allocation, Failed Repetition, Navigation, UI Feedback) emphasize limitations in short- control agents perhaps struggle manage search effort, adapt to failed strategies, and maintain consistent internal view of UI state.
8.9
DOCKER IMAGE SAFETY
We scrape a large portion of our sites from the internet, to avoid unsafe images we picked websites from safe domains such as governmental organizations, nonprofits, and reputable companies with no ads.
9 LLM ACKNOWLEDGEMENT
As per ICLR 2026 policy, we used LLMs in this paper to aid or polish writing, including in summarizing long sentences and paragraphs as well proofreading and fixing grammatical errors. Additionally, this is a paper on LLMs, so LLMs were used as part of the experimentation process on the benchmark as required.
20


Reviews:
[{"review_id": "QXkm8Xpjt9", "reviewer": "Reviewer_Jc1x", "summary": "This paper introduces WebDS, the first end-to-end benchmark for web-based data science workflows. It addresses the gap between existing benchmarks that focus either on simple web navigation or static data analysis. WebDS comprises 870 human-written tasks across 29 dockerized, data-rich websites from 10 domains (e.g., government data, news media). Tasks require complex, multi-step operations (data acquisition, synthesis, analysis, report generation) across heterogeneous sources. Evaluation of current SOTA LLM agents reveals a substantial performance gap: the best agent achieves only $13.2\\%$ success (vs. $90\\%$ human baseline), exhibiting novel failures like poor information grounding and failed repetition. WebDS is containerized for reproducibility and offers fine-grained evaluation across task attributes and difficulty tiers, positioning it as a new frontier for developing practical, end-to-end data science agents.", "questions": "1. The benchmark includes Tool Usage as an attribute. Could the authors clarify which specific tools (e.g., Python code execution, SQL query, spreadsheet manipulation) were enabled for the SOTA agents during the 13.2\\% success rate evaluation? Did the 13.2\\% success rate fully integrate external code execution or primarily rely on the LLM's internal reasoning over extracted text?\n\n2. The analysis highlights \"Poor Information Grounding\" as a key failure mode. Can the authors provide a more detailed example of how the \"Groundedness\" failure was scored? Is the model judged based on whether it extracts the correct data from the HTML (perception error) or whether it uses the correct data in the final report (synthesis/reasoning error)?\n\n3. To maintain relevance and avoid saturation, the paper mentions a plan to \"periodically refresh/expand the private test pool.\" Could the authors provide a brief plan or estimate of the expected frequency or size of this refreshment (e.g., quarterly, or annually adding 100 new tasks)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark's grounding in practitioner interviews and the inclusion of 29 diverse, data-rich websites ensures the tasks are highly realistic and demand complex generalization, not simple pattern matching.\n\n2. The failure analysis (Table 5) provides excellent, fine-grained insights into unique failure modes not captured by simplistic metrics. Specifically, \"Failed Repetition\" (due to lacking state-checking heuristics) and \"Poor Groundedness\" (contradiction between perceived and latent knowledge) are critical problems for future research.\n\n3. The evaluation structure allows for granular analysis across Domain-wise, Attribute-wise (7 labels), and Difficulty tiers (Easy/Medium/Hard), giving researchers precise targets for iterative model improvement.", "weaknesses": "1. The analysis and outputs primarily focus on text (reports, answers). Since the input sites are described as being rich in graphics and non-textual data, the benchmark's current focus may under-represent the full multimodal synthesis challenge (e.g., generating a visual chart or interpreting a complex image-based trend).\n\n2. The action space is not fixed, allowing researchers flexibility, but the implementation relies on existing WebArena/BrowserGym abstractions. A brief discussion on whether these existing abstractions adequately capture the fine-grained data manipulation needed for the \"Tool Usage\" attribute would be beneficial.\n\n3. The current distribution of tasks may slightly overemphasize QA tasks versus downstream action tasks. While the authors mention future work to include more action-based tasks, the current lack of balance might skew initial model optimization toward data retrieval over action execution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zyxUmHbDrc", "reviewer": "Reviewer_wYhP", "summary": "This paper proposes a new benchmark for the web agent domain: WEBDS, which includes 870 data science tasks across 29 different websites. It also provides a dockerized implementation, ensuring stability and reproducibility of experiments. This work lays a foundation for the development of intelligent agents in the data science domain.", "questions": "Please refer to the weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The benchmark environment is very comprehensive, covering 29 websites, 10 domains, and 870 tasks. It focuses on the **data science domain**, evaluating the **entire data processing pipeline**.\n\n- The test environment is **dockerized**, offering fixed environments, stable experiments, and reproducible results.\n\n- The authors use **vision-language models** to analyze complete execution trajectories, providing **richer evaluation metrics** beyond success rate.", "weaknesses": "- **Lack of innovation:** The benchmark design does not significantly differ from existing mature web agent benchmarks. The main contributions remain in expanding the testing environment and task set.\n\n- **Insufficient rigor:** Although using vision-language models for trajectory evaluation is common in GUI agent research, the validation experiments for scoring accuracy and stability (on a 1–5 scale) are rather cursory. The authors only mention comparing 50 human-evaluated tasks, yet the claimed 88% agreement seems questionable. Moreover, since most tested agents achieve **less than 10% success rate**, with most scores below 2, the majority of ratings are clustered around 1. Such a narrow score distribution weakens the credibility of the validation.\n\n- **Limited model comparison:** The benchmark does not test more advanced GUI agents such as **UI-TARS** and **GPT-5**. Since most baseline models achieve accuracy below 5%, this raises the question of whether the benchmark is **too difficult**.\n\n- **Lack of analysis on low success rates:** The paper merely lists potential reasons without detailed trajectory-based analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FX7Ak2EbIi", "reviewer": "Reviewer_dTYm", "summary": "The paper proposes WebDS, a new benchmark to evaluate the end-to-end web-based data science capabilities of large language model agents. Unlike previous web agent benchmarks that focus on simple browsing, or data science benchmarks that focus on static datasets, WebDS integrates both dimensions — requiring agents to autonomously navigate, collect, analyze, and synthesize data from web-based environments.", "questions": "1. How do you plan to maintain or expand WebDS over time while ensuring reproducibility and preventing overfitting?\n2. Could future versions include community-contributed tasks to diversify domains and evaluation scope?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work fills a major gap in current evaluationno existing benchmark assesses full end-to-end data science workflows involving both web interaction and analytical reasoning, captures realistic web-based tasks that better reflect real-world data analysis behavior. The authors evaluates a wide range of SOTA models consistently and highlights key performance bottlenecks.", "weaknesses": "1. The subjective scoring relies on GPT-4o, creating potential evaluation circularity and bias toward similar model families. Including more human evaluations or open-source LLM judges would strengthen reliability.\n2. While qualitative error categories are given, there is no quantitative breakdown of which task attributes (multi-hop, tool-use, multi-site) most contribute to failures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6vJZkfztDR", "reviewer": "Reviewer_WpTs", "summary": "The paper presents WebDS, a benchmark for evaluating LLM agents on realistic web-based data science tasks. It integrates 29 containerized websites and 870 tasks involving data retrieval, multi-hop reasoning, tool use, and report generation. Results show humans succeed ~90%, but state-of-the-art agents achieve <15%, revealing a major performance gap. The key contribution is providing a comprehensive and reproducible benchmark that mirrors end-to-end data science workflows.", "questions": "1. While WebDS tasks involve data retrieval and basic analysis, many real-world data science workflows include steps such as data cleaning, preprocessing, visualization, and modeling. Do the authors plan to expand WebDS to cover these stages, and how would they envision incorporating them into the benchmark ?\n2. Why not adopt **execution-based evaluation**, which would be more realistic than the current rule-based approach?\n\n\nMiss citations:\n\nI strongly recommend that the authors cite more **data science–related work**, such as:\n\n* *DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?*\n* *DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models*\n* *Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows*\n* *Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?*\n* *DABstep: Data Agent Benchmark for Multi-step Reasoning*\n* *OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments*", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is highly original in proposing the first end-to-end benchmark for web-based data science, combining web navigation with real data analysis in a way not covered by prior benchmarks. It demonstrates strong quality and clarity by designing 29 containerized websites and 870 tasks across multiple domains, with clear task categorization that ensures reproducibility. Finally, its significance is clear as results reveal a striking human–agent performance gap, establishing WebDS as a critical benchmark for advancing research on practical, data-science–capable AI agents.", "weaknesses": "1. Limited Task Scope Beyond Data Science. Although the paper claims to be about Web Agents + Data Science, in reality the data science aspect is minimal, and the amount of required coding is very small.\n\n2. Overly Simple Baseline Design. The paper does not propose its own unique baselines. Considering this is naturally a CLI + GUI hybrid setting, many approaches from WebArena or similar frameworks could have been adapted, but they were not included. Moreover, many closed-source agent frameworks and both open/closed-source products were ignored. Overall, the baseline coverage feels insufficient.\n\n3. Lack of Multiple-Trial Evaluation (pass@n). For each example, it is unclear whether multiple trials were conducted to calculate pass@n. I suspect the low reported performance may be partly due to annotation ambiguity, which makes it hard for agents to succeed on a single attempt.\n\n4. Subjective Task Evaluation Is Too Arbitrary. For open-ended tasks, evaluation is simply based on a 5-level scoring scheme. Ideally, one should measure agreement between humans and LLM-as-a-Judge. If a more reliable scoring method were adopted, I would increase my rating, but the current LLM-based scoring feels too random, and results may vary significantly across repeated runs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)