KRAMABENCH: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes
Authors: Eugenie Lai
Gerardo Vitagliano
Ziyu Zhang
Om Chabra
SIVAPRASAD SUDHIR
Anna Zeng
Anton A. Zabreyko
Chenning Li
Ferdi Kossmann
Jialin Ding
Jun Chen
Markos Markakis
Matthew Russo
Weiyang Wang
Ziniu Wu
Mike Cafarella
Lei Cao
Samuel Madden
Tim Kraska
Venue: ICLR 2026 Poster (2026)
Keywords: data science agents
data science
data wrangling
data analysis
data management
reasoning
agentic systems
Primary area: datasets and benchmarks
TL;DR: KRAMABENCH is a benchmark of 104 real-world pipelines showing that while LLM systems can generate code and rough plans, they fall short at building data science pipelines that work on real-world data.

Abstract: Discovering insights from a real-world data lake potentially containing unclean, semi-structured, and unstructured data requires a variety of data processing tasks, ranging from
extraction and cleaning to integration, analysis, and modeling. This process often also demands domain knowledge and project-specific insight. While AI models have shown remarkable results in reasoning and code generation, their abilities to design and execute complex pipelines that solve these data-lake-to-insight challenges remain unclear. We introduce KramaBench which consists of 104 manually curated and solved challenges spanning 1700 files, 24 data sources, and 6 domains. KramaBench focuses on testing the end-to-end capabilities of AI systems to solve challenges which require automated orchestration of different data tasks. KramaBench also features a comprehensive evaluation framework assessing the pipeline design and individual data task implementation abilities of AI systems. We evaluate 8 LLMs using our single-agent reference framework DS-Guru, alongside both open- and closed-source single- and multi-agent systems, and find that while current agentic systems may handle isolated data-science tasks and generate plausible draft pipelines, they struggle with producing working end-to-end pipelines. On KramaBench, the best system reaches only 55% end-to-end accuracy in the full data-lake setting. Even with perfect retrieval, the accuracy tops out at 62%. Leading LLMs can identify up to 42% of important data tasks but can only fully implement 20% of individual data tasks.

Published as a conference paper at ICLR 2026
KRAMABENCH: A BENCHMARK FOR AI SYSTEMS ON PIPELINES OVER DATA LAKES
Eugenie Lai'*, Gerardo Vitagliano' *, Ziyu Zhang' Om Chabra', Sivaprasad Sudhir',
Anna Zeng', Anton A. Zabreyko', Chenning Li", Ferdi Kossmann', Jialin Ding", Jun Chen?
Markos Markakis', Matthew Russo', Weiyang Wang', Ziniu Wu!
Michael Cafarella', Lei Cao", Samuel Madden', Tim Kraska!
MITT CSAIL, "Independent, UUniversity of Arizona
denotes equal contribution. Corresponding email: gerarvitémit.edu
ABSTRACT
Discovering insights from a real-world data lake potentially containing unclean, semi-structured, and unstructured data requires variety of data processing tasks, ranging from extraction and cleaning to integration, analysis, and modeling, This process often also demands domain knowledge and project-specific insight. While AI models have shown remarkable results in reasoning and code generation, their abilities to design and execute complex pipelines that solve these data-lake-to- insight challenges remain unclear. We introduce KRAMA KRAMABENCE which consists of 104 manually curated and solved challenges spanning 770 files, 24 data sources, and 6 domains. KRAMA BENCH focuses on testing the end-to-end capabilities of AI systems to solve challenges which require automated orchestration of different data tasks. KRAMABENCE also features a comprehensive evaluation framework assessing the pipeline design and individual data task implementation abilities of AI systems. We evaluate 8 LLMs using our single-agent reference framework DS-Guru, alongside both open- and closed-source single- and multi-agent systems, and find that while current agentic systems may handle isolated data-science tasks and generate plausible draft pipelines, they struggle with producing working end-to- end pipelines. On KRAMA BENCH. the best system reaches only 55% end-to-end accuracy in the full data- lake setting. Even with perfect retrieval, the accuracy tops out at 62%. Leading L.MSs can identify up to 42% of important data tasks but only fully implement 20% of individual data tasks.
INTRODUCTION
The goal of data science is to obtain insights from raw data. A data science workflow typically involves manually selecting data and designing pipelines that perform data wrangling, conduct data analyses, and extract findings, among other data tasks. These workflows (Figure 1) are expected to handle noisy, domain-specific data and scale to data lakes with tens to thousands of files, necessitating
multi-step, data-dependent reasoning and coordination across data tasks (Guo et al. 2024 Shankar al. 2025).
While recent research has advanced individual components of these workflows such as code genera- tion (Nam et al. 2024 Wang & Chen, 2023). tool use Qin al. 2024b a). and natural language question answering (Zhang et al.) 2024b Pu et al.) 2023). the challenge of designing and executing complete end-to-end data science pipelines remains underexplored.
Progress towards practical data-to-insight systems has been hindered by the lack of benchmarks that reflect the real-world complexity of these workflows. Existing benchmarks focus on isolated steps,
Assets available at https:tpp........... /gi thub. cww/miii//cooooo..o. htwpb//..o//iittog///aaaaaaaa

---

Published as a conference paper at ICLR 2026
Data
discovery
Lake
Txn
Payee
Payment Method
Municipalities
Task: According the Consumer Sentinel Network, what is the total amount money defrauded 2024, summing over payment methods? Give an integer in millions of dollars.
Table
001
Alice
Card
Extraction
Albany,
002
Data
Cleaning 90
Theee transactions are From
Consumer Sentinel Network is an anegen that..
2024.
Fraud by Municipalities
Municipality
Fraud Type
5435 million
Albany, NY
Identily Theft
One of the tasks of KRAMA BENCH based on a real data lake of 136 files in the legal discovery domain. Data file sample snippets are simplified.
Figure 1:
Table I: Comparing existing benchmarks. (- indicates partial satisfaction, e.g., not for all tasks)
Sprowee peeeedacatiin ch Chan a (anss)
Lai G012
ARCADE Vin of alaaiin
DA-C Hoang at 20144
Daus Bench Zhnng at MA2225o
DSseenal Bong (222
Go ol 2014)
Ours
Benchmarks
DS Tasks
Data discovery
ARiiNA xx3
Kix3SS
NXXSSS 1SS
A>ssi> ددS ددد
Multi-file insegration
Data cleaning
Data prepartion
Duta amilyss
Modeling
Abilities tested
Dara semantics
XX2
183
Donnin knowledge
Multi Multi-tep reasoning
Evalustion
Implementation
3xx
SIx
Pipeline design
End
such as code generation from fine-grained prompts (Lai al.) 2023 Zhang al. 2025a Huang
et al., 2024]] Yin et al. 2023), text-to-SQL. (Lei et al. 2025| Zhang et al.) 2024a). and modeling
using curated input (Gu et al. 2024 Mitchener et al. [025| Chen et al. 2025). We list these works in Table and discuss more in Section 5 While immensely useful, these benchmarks do not capture
the heterogeneity of data tasks and the accompanying reasoning demands real-world data science involving large, domain-specific, and unclean input datasets.
To bridge this gap, we introduce KRAMABENCE a benchmark designed to evaluate L.M-based systems on complex end-to-end data science pipelines. Bench consists of 104 tasks drawn from 1700 real-world files across 24 sources domains. All tasks are manually curated from fresh, domain-specific sources and paired with expert reference solutions grounded in accessible data. Each task is specified in natural language and requires systems to discover relevant data, perform data wrangling such as cleaning and normalization, and implement statistical computational analyses to produce insights. To study public data's 's leakage into LLM training, we obscured the input of 20% of tasks through replacing real-world identifiers and numeric data with synthetic ones without changing the task structure. We hold them out for evaluation to prevent them from being trained on.
For each task, we provide reference sub-tasks that system capable of solving the end-to-end task should be able to solve. Sub-tasks are also annotated with ground truth results and text descriptors. These assets facilitate our comprehensive evaluation framework with three settings. (1) The most important end-to-end automation setting assesses the ability to solve tasks without human in the loop. (2) The pipeline design setting assesses the ability to reason and identify key components towards successful pipeline design. (3) The individual task implementation setting assesses the ability to act on fine- men-grained descriptions of individual sub-tasks in a correct pipeline.
"The name KramaBench is reference the "Vinyasa Krama" practice of Yoga
2

---

Published as a conference paper at ICLR 2026
We evaluated KRAMABENCH across eight models, along with three different configurations of DS-Guru and three other existing agentic systems (Hugging Face, 2025] OpeenAl 2025] Google
2025). We conducted extensive ablations studies and failure analyses, taking advantage of our comprehensive evaluation framework and obscured inputs.
Through KRAMABENCE
observed multiple insights about where LLM systems successful: (1) Agentic control flow is helpful with KRAMA BENCH'S challenges: a canonical single-agent system (smolagents DR) that iteratively search, plan, and repair achieve 55.83% end-to-end accuracy, outperforming the strongest configurations of DS-Guru (24.98% overall), which uses a structured control flow. (2) canonical multi-agent system (smolagents Reflexion, Siffexioo,,Shinn [Shinn al. (2023) ) using an evaluator agent and reflections achieves 55.37% end-to-end accuracy. The marginal difference (-0.45%) relative the single-agent baseline suggests that simple evaluator-style coordination alone does not substantially improve performance on data-intensive workflows, indicating opportunities for more specialized multi-agent designs. L.MM systems can reason a coarse level about the data operations required by successful workflow and generate plausible pipelines, achieving 41.71% on pipeline design.
Our analyses also reveal some persistent challenges: (1) Retrieval from data lake is problematic, but not the dominating obstacle. Supplying only the gold files improves overall accuracy by only - 7% across systems using different retrieval mechanisms. (2) Weaknesses in fine-grained data-dependent reasoning cause models to fail. Systems even fail most of the time at implementing individual simple sub-tasks, capping at 22.05% when evaluated under the individual task implementation described above. (3) Agents often fail to achieve a holistic understanding of the data lake. We observe that the agents often overly rely on their prior knowledge (at most 43.06% performance fluctuation on obscured inputs), or assume clarifications will be given from a user (24% of failures).
THE DESIGN OF KRAMABENCH
Tasks KRAMABENCH are based on real-world data science challenges from six domains: archeol- ogy, astronomy, biomedical research, environmental science, legal insight discovery, and wildfire prevention. Each domain associated with a data lake containing raw files in structured, semi- structured, or unstructured formats from multiple sources. Each task natural language description of domain-specific data science problem. The goal of a system under test is to design and execute an end-to-end pipeline that takes the entire domain data lake as input and produces the correct output. KRAMABENCH
In addition to the target answer, KRAMA BENCH provides the ground truth solution both in code
and in annotated sub-tasks: natural language descriptions of smaller building-bloct operations that are essential elements within full solution along with prompt and their target answers. These finer- graine references enable the evaluations of pipeline design and individual task implementation.
2.1
TASK DESIGN AND VALIDATION
sul CaNEEa
13
Stant from
iasdy
Create desr language lass
Buid ivita poline
Peproduce aducy resuti
divooeree
incopandixx
Peing er pioekin
Crnoectach
Fieel
umar do guouo
Holine clurly and
A AIR cuncici
trail ppolito to
Dreti tuscforcFiess
of bey
hunc rroil sub
bendimork
bwndhmekk
fanciiona
Figure 2: Workflow for task design and validation in KRAMABENCE detailing the curation, validation, and functional decomposition to ensure quality and consistency across tasks.
3

---

Published as a conference paper at ICLR 2026
Table 2: Detailed breakdown of per-domain tasks in KRAMA MA MABENCH. Hard tasks require multiple files or pipelines with more than three steps.
Table 3: Answer type and example questions.
Type
Metric score
String (exact)
Accuracy (0/1)
Domain
Tasks (sub-)
Hard Tasks
# Files (size)
Archoology
12
50.00%
5 (7.5MBB
String (approximate)
ParaPluie score (0/1)
Astronomy
12 (68)
50.00%
1556 (486MB)
Numeric (exact)
Accuracy (0/1)
Biomedical
66.66%
(175MB)
Environment
20 (148)
70.00%
37 (3IMB)
Numeric (approximate)
1/(1 RAE) (0-1)
Legal
30 (188)
53.33%
133 (L.3MR)
Wildfire
21 (120)
71.42%
23 (IGB)
List (exact)
Fl score (0-1)
Total
(633)
60.58%
1764 (1.7GB)
List (approximate)
Fl score (if match > 0.9)
To curate tasks, started with published studies and reports that (1) contain quantitative graphical findings produced by data analysis. (2) are based on complete and publicly accessible datasets, and (3) require complex multi-step pipelines involving heterogeneous and noisy inputs. Grounding onto these studies and reports ensures that our tasks reflect real-world data science pipelines. We followed 4-step workflow involving tight validations and repeated verifications of reference solutions to ensure the quality of tasks, reference solutions, and fine-grained annotations. We summarize the process in Figure 2
Step 1: Task Curation. For each study or report, we reproduced its important findings using the associated datasets, transforming these findings into problem statements. Within the same domain, more tasks similar to the real-world ones are curated via integrating different data sources. The creator of each task supplies concrete implementation of the pipeline.
Validation. For each task, different second contributor independently attempts to develop a solution. third contributor compares the solution with the one Step 1., resolves ambiguities in the problem statement, and checks in reference pipeline. The execution time of the reference pipeline is also recorded.
Step 2:
Step 3: Key Functionality Identification. A data science problem can have multiple valid solution pipelines. However, certain data processing steps must exist in any correct pipeline. A simple example would be "identifying the column containing the temperature to be Temp". We draft list of these key functionalities for each task using the reference pipeline via instruction-tuning GPT-o3 and manually polish the outputs make sure the description of these sub-tasks do not depend specific implementation choices. The semi- mutomattootionn scripts are available at our repository.
Step 4: Sub-task Curation. We transform each sub-task description into a prompt via instruction tuning local instance of Gemma3-27b and manual inspection. The example in Step would be transformed "which column contains the temperature information"? The target answers each sub-task are manually verified using the reference pipeline.
Table reports the statistics and difficulty distributions of the domains and their tasks. We provide more detailed descriptions and an example of tasks, key functionalities, and sub-tasks in Appendix E
2.2 EVALUATION MECHANISM
As discussed in Section KRAMA BENCH evaluates systems on three capabilities. In Figure 3 we provide an overview of these 3 areas. Our primary focus is (1) end-to-end automation.
(1) End-to-end Automation. For each task, the system output is given score in [o, 1] based on the reference target answer. The scoring schemes for each possible answer type address fuzzy matches and are discussed in Table 3 The string approximation metric uses the method introduced in (Lemesle al.] 22255. Our validation study (details in Subsection G.2p for this LLM-as-a-judee method shows 84% agreement between human annotators and the LLM. Given a domain workload W consisting of numerous tasks T's, the total score of system F for W is ((oce((F((()) The score of F for the entire benchmark suite is analogous.
Results under the following two less- austmateee evaluation settings provide insights into why system may succeed or fail in the end-to-end automation setting and the abilities system to assist with a human-in- inttan----the-loo... Figure & describes the mapping of the following tasks.
4

---

Published as a conference paper at ICLR 2026
1
Proparation
with Goid
Furclionliinn
Bo Lake
teala with Goid
Soaal lack
En Snd
Compus Pinal
Compuie
Syean Under
Produce Oupuut
Piel Code
Design
Check Flar Code Covera
Score Coverage
byckekon
High Boures
Ariwers
Emikat
task Arivver
Figure 3: Overview of KRAMABENCH'S evaluation process.
(2) Pipeline Design. For each task, we assess the system generated pipeline using the key functionali- ties that any correct pipeline needs to contain in some form (from Step ofSSusectioon 2.1p. We score the system with the fraction of key functionalities covered in the pipeline produced by the system. Coverage evaluated via -La-aa---judge following the method in Tong & Zhang (2024) using the description obtained in Step 3.
We provide systems with the problem statements for sub-tasks and compare system outputs to human-curated target answers (as in Step 4 of [Subsectoon 2.1) using the same scoring approach as in end-to-end evaluation. Full technical details of these evaluations are provided in Appendix G
(3) Sub-task Evaluation.
2.3 REFERENCE IMPLEMENTATION
We introduce DS-Guru (Figure 4p, lightweight framework that serves minimal scaffolding enable a single out-of-the-boo LLM to attempt the data science challenges in KRAMABENCH BE DS-Guru has three variants. No-context: The LLM is invoked one-shot with the problem description and the names and paths of the files from the data lake, without any file contents, One-shot: The LLM invoked one-shot with the problem description and sample snippets from each data file. Few-shot: The LLM is first invoked once with the task description and sample snippets, then re-invoked few-shot with execution results and error messages from the pipeline it implemented in the previous shot. With all variants, DS- Guru instructs the LLM to decompose the task into simpler tasks before attempting to implement each task provide the concrete pipeline implementation along with the answer.
Modeetoo robeet
Nocaraeniiuu
evepe
Figure 4: DS-Guru, a lightweight framework that scaffolds out-of-the-box LLMs with multiple variants tackle KRAMA
DS-Guru succinetly addresses where out-of-the-box LLMs struggle with KRAMABENCHE (1) Realistic data lakes exceed LLM context windows. DS-Guru uses budgeted, type-annotated one-pass sampling (OPS) retrieval to make this step tractable. different data operations. DS-Guru uses chain-of-thour
(2) Many data science tasks require many prompting (Wei et al. (022) to encourage decomposition before code synthesis. (3) Code running real-world uncurated data are subject to more sporadic errors compared to code for well-structured tasks. DS-Guru's 's multi-shot approach (Press et al. 2023) can help L.MSs recover from such errors. More details DS-Gure in Appendix
5

---

Published as a conference paper at ICLR 2026
3 EXPERIMENTAL SETUP
We accessed all LLMs in different systems via OpenAl and Together APIs; pipelines generated by systems are executed locally. Whenever possible, we repeated all experiments three times and report average score and standard deviations across the three runs, For some results, marked with an asterisk, could not perform repeated experiments due to high API costs to the backend models being discontinued during the experiments.
DS-Guru: We combine each of the three variants (as in Subsection 2.3) of DS-Guru with six LLMs: GPT-o3,, GPT-4o, Blaude-3.5-Sonnee Llama3.3, 3, Deepseek R1-bek and Qwen2.5 Codeer-
32B [OpenA] 2025 2024: Anthropic 2024 Meta All 2025 DeepSeek-AL et al. 2025 Hui et al.)
2024 totaling to 18 concrete DS-Guru implementations.
smolagents Deep Research (smolagents DR): We use Hugging Face smo 1 agentss (Hugging Face 2025) to evaluate deep research-style agentic systems on our benchmark under both single-agent and multi-agent settings. On the single agent setting (smolagents-singg
DR), smo agent' official single-agent deep research implementation using code actions (git), we report results with two differ- LLMs as the LLM: GPT-o3 and (laude-3.7-Sonnee For the multi-agent setting, we implemented two representative architectures also using smo lagents:: (1) smolagents Reflexion (Shinn et al. 2023) which follows an actor + evaluator + reflection agent loop. (2) smolagents PDT which follows a Planner and Task Decomposer + Tool Executor workflow (PDT), inspired by Fan et al. (2025). We provide more details in Apppendix C We view these systems representative of open-source
"deep research" projects (e.g., Alibaba's (Academy) (2025)).
Closed-Source Deep Research Systems: OpenAl Deep Research (OpenAl DR, OpenAl (0255)
and Gemini Pro-2.5 Agentic Mode (Gemini Agentic) (Google DeepMind 2025 Google 2025)
were evaluated manually through their web interfaces under the end-to-end automation setting. We made best efforts instructing them not search online. However, this restriction not enforceable.
Table 4: Comparison of different mechanisms across systems.
Systems
Retrieval mechanisms
Input modes
Control flow
Internet Access
DS Guru
One Pass Sampling (OPS)
Full, Trimmed, Oracle
Structured loops
OIT OIf
smolagents DR
Agentic retrivaal
Full. Trimmed, Oracle Full, Trimmed, Oracle
Single-agent ReAct loop
smolagents Reflexion
Agentic retrieval
Multi agent
OIT
smolagens PDT
Agentic retrieval
Full. Trimmed, Oracle
Multi-agent
OIf
OpenAl DR
Agentic retrieval
Trimmed, Oracle*
Agentic loops
On
Gemini Agentic
Agentic retrieval
Trimmed, Oracie*
Agentic loops
On
Human Baseline. We conducted small-scale human study in which 9 data-science practitioners solved KRAMA BEnCH under the same conditions and requirements as LLM systems under test in the end-to-end, full input setting. Details can be found in/ppeendxx F
We evaluated six different systems, which differ in four important ways.
Retrieval Mechanisms. DS-Guru employs One-Pass Sampling (OPS) retrieval: a budgeted, type- annotated sample of each file in the data lake (schema summaries + a small row sample) is provided to the LLM once. OPS scales with data lake size but constrains the LLM's direct interaction to sampled views. DR systems employ agentic retrieval: the LLM plans the retrieval and issues file system tool calls iteratively read, filter, and revisit sources, offering richer interaction but a higher cost.
Input Modes. Full: ideally, the entire input lake is available to the system's 's retriever. Oracle: only the gold files are provided (no discovery), isolating non-retrieval failures (planning, reasoning, execution). Trimmed: to respect practical constraints, most notably the UI limit of <10 file uploads imposed by the -lorre-source DR systems, we supply the gold files plus a random subset of distractors up to the limit, testing discovery under budget. Oracle*: #; for tasks where the gold set itself exceeds 10 files, include the task by randomly sampling 10 gold files for upload.
Control flow describes whether system has fully structured loops agentic workflow where agents decide the future courses of actions. Internet access describes whether systems have web search capabilities.
Cost of evaluation. For DS-Guru (few-shot, GPT-o3), evaluating end-to-end answers, pipeline design, and sub-task implementation took 4,501, 116,805, and 10,358 tokens respectively.

---

Published as a conference paper at ICLR 2026
4 RESULTS AND TAKEAWAYS
Jssoos the performance of the systems under the Full, Oracle, and Trimmed input mode. We report only top-perforning configurations here and present full results in Appendix
Table 5: Results by domain for KRA MA BENCH on DS- Guru and smolagents for three input settings. Results marked with an asterisk are not averaged across three executions (mean + standard deviation).
System
Models
Archeology
Astronomy
Biomedical
Domains
Enviroment
Logal
Wildfire
Overall
Full Input Mode
Human baseline
58.33
70.00
100.00
76.90
86.67
66.20
76.75
n0 no-content
DS Gura
GPT-o3
GPT-1o
Claude-3.7 3.7
2.78
0.00
0.00
3.93
3,33 1 4.71
0.00 1. 0.00
4:
0.30 0.14
0.00 0.00
0.00
0.00
5.00
0.00 1 0.00
0.00
1. 0.00
4,08
0.00=
0.00 0.00
1.11 1.57
14.18
0.00
4.38 1 3.92
4.28
0.00
5.87 = 0.71
0.00 2: 0.00
1.88
0.81
DS- Gura
one-shor
Claude-3.7
GPT-4o
GPT-3
19.44
8.33
6.80
3.93
0.00 + 0.00
6.67 + 4.71
3 0.00
0.53 0.75
16.11
0.00
3,33 2.36
0.00
9.26
10.00
4.41= 2.17
0.00
0.00
40.01 + 4.22
13:69
3,06
16.67 = 2.93
6.08
1.31
1.22
1.25
few-shot
DS- Guna
GP
GPT-1o
Claude-3.7 3.7
13,89
3.93 5,56
3.93
0.00 x 0.00
3.33 1. 4.71
0.00 0.00
0.00 a 0.00
0.14
49.56 0.87
15:00 1 4,08
0.00 1 0.00
8.89
9:26= 2.92
4.84 3.16
6.29
25.69 4: 1.15
52,92
10.09 1 9.67
24.95
8.67 L 1.48
8.29
1.12
1.25
smolagents DR
GPT-3
33.33 11.79
23.33 + 9.43
37.04 = 10.48
31,33
8.96
17.78
8.31
35.23 + 11.46
28.07
8.30
44.44
3,93
50,00 + 8.16
38.89 7,86
60.56
7.74
61.23
12.70
60.16
7.43
55.83 = 3.41
smolagents Retlesion
Cla
GPT-o3
4.007
10.39
46.67
45.00
50.00
29
5.24
23.89
56.25 à 8.75
76.67
14.40
65.38
0.86
30.53
55.37 3.336
smolagents PIYT
G3
Claudo-3.7
19:44 + 7,86
8.33 6.80
6.67 + 4.71
0.00 + 0.00
0.00 0.00
5.56
5.56
9.84 + 2,86
1,42 + 1.09
4.51
6.25
0.84
4.95
22,01 #: 1.39
22.58 + 4.05
12.01 + 1.03
7.57 = 1.28
Orucle Input Mode
mu-conlest
DS-Guna
GPT-o3
GPT-4o
Claude-3.7
11.11
0.00+
4.17
0.00
3.93
10.00 +: 8.16
10,00 0.00
0.00 + 0.00
11:11 = 0.00
0.00
0.00= 0.00
0.00
0.00 + 0.00
0.000 + 0.00
1.58
1.67
0.00
0.00
0.00
0.00
1.67
10.17 0.92
0.00+
0.00
0.00 + 0.00
5.36
0.79
1.20
GPT-03
16.67 1
7.51 5.31
23.89 1
15.56 1.57
42,80
21:35
0.81
one
Claude-3.7
GPT-4o
11.31
0.00
7,86
0.00
13.33 a. 4.71
0.00 : 0.00
5.19 7.33
0.00
0.00
12,33 4 3,30
6.67 4 1.67
6.67
3.33
0.00
3.33
19.85 a 3.61
4.76 4 0.00
11.54 8 1.47
3.27 =
1.31
few -hos
DS-Gura
GPT-4o
GPT-o3
Claude-3.7
20,8
16.67
13,89
0.00
3.95
25,00 15.00
20.00 + 0.00
26.67 +: 4.71
7:41 a 5.24
3 5.34
0.00
50.11 + 4.09
18.44 + 6.00
9.17
37.78
61.03
5.78
4.16
52.02 a 3.79
36.67 + 3,53
18.17
43.71 =: 1.94
26.20
3.36
2.34
smolagents
GPT-03
Claude-3.7
27.78
45.83
12.50
65.00 a 5.00
58.33 a 2.78
a: 5.24
46.83 1 1.50
24,00
0.00
17.78
65.00
1.67
32,89
75.08 à 3.78
25.86 4: 1.57
60.67 =: 1.23
smolagents Reflexion
GPT-o3
Claude-3.7 7
37,50
50.56
4.17
3.93
50.00 + 0.00
40.00 + 0.00
83.33 8 5.56
44.44 = 0.00
62.67 a 4.33
31,44 + 2.20
22:22
70.00 8 10.00
8:31
64.45 m
39.78 + 6.14
1.39
32.33
62.81 8 3.10
4.44
smolagents PDT
Claude 3.7
GPT-33
16.67
8.33
6.80
0.00
10.00 : 0.00
0.00 s: 0.00
11.11
0.00
0.00
0.00
11.50
3.26 1. 1.30
3.50
8.62
9.24
3,88
0.35
25,50
15.86 s: 5,53
5.46
14.38
7.69
2.15
2.69
Trimmed Input Mode
DS-GURU few-shot
GPT-3
25.00
6.80
0.00
0.00
3 5.24
48.33
8.50
51.11
3.14
48,87
7.90
37,84
3.90
smolagents- single
Claude-3.7
55.56
40.83
58.12
OpenAl Online Deep Research*
40.00
33.33
44.45
61.67
50.00
67.28
52.18
Google Agentic Gemini-2 P-Pr"
25.00
16.67
33.33
25.00
13.33
24.87
18.48
Agentic control flows drive the largest performance gains on KRAMABENCH. The smolagents DR variant using Claude-3-7 consistently outperforms DS-Guru across all domains (Table reaching an average 55.83% overall score compared to the best DS-Guru variant (few-shot, GPT-o3; 24.98%). The DS-Guru few-shot variant, which enables the LLM catch implementation errors, improves over the one shot variant, e.g., using GPT-o3, with a 8.31% overall improvement. Our detailed studies increased few-shot to 20 iterations yet still showed minor improvements (Appendix Table l3}. This indicates that despite the heterogeneity of data operations, the core challenges are not isolated data operation implementation issues, but instead are to (1) explore and fix the design choices of the end-to-end pipeline; (2) iteratively understand the data and schema in large data lake. Smolagent DR's agentic control flow helps address these challenges. call). OpenAl's own online DeepResearch tool reaches 58.12% overall, partly due to its web search capability. We refer the reader to Appendix for detailed analysis of the human bascline., as well as for detailed analysis of cost and runtimes of each models. In terms of cost, smolagents DR (Claude-3-7) averages 6. 6.10 minutes per faster than OpenAl DR (10.35) but more than 10x slower than DS-Guru few-shot (0.76).
the Trimmed setting (max 10 files per
4.1 ABLATION STUDIES
Retrieval Mechanisms. Overall, using Oracle input for DS-Gum improves the performance across all domains and L.Mss (by 6.38% on average and up to 18. 18%7 except for GPT-o3 for which in the
7

---

Published as a conference paper at ICLR 2026
Table 6: DS-Guru (few-shot, 5 iter- ations, GPT--3):: performance and cost across rows sampled.
Table 7: End-to-end scores of various systems under
obscured vs oracle inputs (mean + standard deviation).
System
Model
Full
Oracle
Obscured
Rows Sampled
Overall Score (%)
Tokens (Mean)
DS-Guru no-context
GPT-o3
5,87+0.71
5.36#1 ,20
3.6921.63
Claude-3.7
1.8840.81
4.11:0.79
0.00=0.00
DS-Guru one-shot
GPT-33
16.6742.93
21 21.3540.84
7.85+1.27
10
22.89
14,077.2
Claude-3.7
1.31+1.22
3.27+1.31
0.50=0.50
50
24.68
37,592.3
DS-Garu
GPT-33
24.98+1.22
43.71
23.02=0).49
100
23.36
64,548.9
few-shot
Claude-3.7
8.29&1.12
19.75s3
3.80+3.80
150
22.58
92,116.1
smolagents
GPT-o3
28.07 80
25.86+4.57
10.07=0.30
DR
Claude-3.7
55.83+3.41
60.67+1.23
12.77=0.00
smolagents
GPT--3
30.53+10.79
32.33+4
9.52+3.95
Reflexion
Claude-3.7
55.37 3:336
62.81
13.9420.00
smolagents
GPT-o3
7.57+1.28
7.6942.69
3.87+3.52
PDT
Claude-3.7
12.01=1.03
14.38+2.15
2.23+0.00
no-context scenario no significant improvement is registered (Table These results under the design of DSD DS-Guru show that supplying samples of the gold files can lead more successful pipelines. We also studied the sensitivity of OPS against the sample size from each file. Table shows that the performance of the system does not meaningfully increase with larger samples.
The benefits of the Oracle in smolagents DR show the same trend, suggesting that agentic retrieval is qualitatively closer perfect retrieval than OPS in terms of file extraction. Even with the Oracle input, the agentic smolagents DR with out-of-the-box LLMs still struggle to solve a large portion of the tasks (62.81 overall with Claude-3.7). These results point weaknesses in data-dependent reasoning ge.g., pipeline design), in addition to extracting the right files.
Data Leakage. To study to what extent different systems are solving tasks via external knowledge present in previous knowledge data instead of producing a reliable data pipeline, we manually curated obscured inputs for all tasks in KRAMA BENCH, where data fields are changed such that a correct pipeline would still produce a correct solution, but system relying on memorization cannot. For example, in a query spanning multiple locations, the real place names may be swapped for fictional ones, i.e., Los Angeles might be changed to "La-La Land."
For both smolagents single DR and DS-Guru few-shot, the performance under the obscured input is 15 - 18% lower compared to the regular and oracle inputs (Table 7). Interestingly, Claude as backend model seems the most sensitive to the obscured workloads, e.g. with a sharp drop from 62.81% to 12.77% in the Reflexion baseline, suggesting that the solution of the benchmark in these variants may heavily rely on parametric knowledge. These observations and the stark difference between the Full and Obscured input performances suggest two distinctive plausible explanations for our observations: (1) Prior knowledge could discourage attempts at data-dependent reasoning. (2) Prior knowledge could be serving as an unintended reward signal in agentic data-dependent reasoning, which possibly can either reduce the performance of the system.
Cross-domain Accuracy Difference The per-domain accuracy of the best performing system (smolagents Reflexion) in KRAMABENCH varies as much as 41.67% on Archeology and 65.38% on Wildfire. Our analysis of system traces show that the primary source of these cross-domain accuracy differences is the differences in the types of data task challenges that each domain emphasize on. We discuss this in more detail in inSubseettion D.I
Diversity of Abilities Required from LLM Agents. Tested independently, pipeline design and subtask implementation outperformed end-to-end automation, e.g, even the no-context scenario, using GPT--3 as a model, the pipeline design score is 39.75% compared to 5.87% end-to-end, In addition, we observe that LLMS also have varying capability profiles: GPT-o3 is strong at high-level pipeline design (39.75% 75%) but weak at implementing those pipelines (10- 22%); interestingly. it scores higher on end-to-end automation (24.98%) than on some implementation tasks. Although with worse absolute results, DeepSeek-RI R1 exhibits the opposite pattern (1% on pipeline design vs 5. 5.79% on implementation). These patterns provide strong evidence that single-mode approaches insufficiently reliable for real-world data science, success depends multiple heterogeneous
8

---

Published as a conference paper at ICLR 2026
Table 8: Lower automation settings results on KRAMABENCH (mean standard deviation).
Models
Variant
Automation setting
GPT-o3
GPT-4o
Llama3 Instruct
DeepSeek-RI
End-to-end automation
5.87 + 0.71
0.00 = 0.00
0.33 + 0,46
0.33 : 0,46
JS-Guru no-context
Pipeline Design
39.75 # 0.73
29.73 1.16
21.80 = 1.30
0.63 # 0,47
Sub-task Implementation
10.69 0.35
3.32 0.54
3.88 0.25
1.48 0.08
End-to-end automation
16.67 2.93
6.08 = 1.25
3,42 # 0.37
0.00 = 0.00
DS-Guru one-shot
Pipeline Design
41.71 1.00
21.26 0.64
15.53 # 0.73
1.59 = 0.77
Sub-task Implementation
17.33 0.48
5.08 0.57
2.80 0.08
3.91 = 0.44
End-to- End-to-end automation
24.98 1.25
8.67 1.48
8,40 # 1.01
0,33 + 0,46
DS-Guru few-shot
Pipeline Design
37.99 2.64
20.31 0.07
12.81 1.04
1.11 = 0.72
Sub-task Implementation
22.05 0.71
6.60 0.29
5.53 0.49
5.79 0.72
skills, such as robust parsing of noisy imputs, query parsing and planning. identifying and performing data-cleaning/ transformation, coding, and iterative debugging.
4.2 DEEPER DIVE: FAILURE ANALYSIS
In this subsection, we closely study two tasks requiring two distinct reasoning capabilities from LLM agents: (1) fine-grained data-dependent reasoning. (2) holistic understanding a potentially domain-specific data lake.
Challenge Fine-grained data-dependent reasoning.
Monthly Precipitation
Water Body Testing
Year
Jan
Feb
Mar
Community
Sample Date
Beach Name
Violation
2015
4.4
3.9
4.7
Chatham
2016-06-13
Bucks Creek
no
2016
5.0
Brewster
2016-08-16
Cliff Pond (DCR) DVS
yes
2017
4.5
M
M
Brewster
2016-05-24
Cliff Pond (DCR) @ Main
no
Figure 5: Data snippets for study cases. Multiple water testing entries for each location may exist.
environment-q17: What is the seasonal bacteria exceedance rate of Chatham's 's Bucks Creek Beach in the June, July, Aug 2016? Impute missing values with median of the month in non-missing years.
To solve this query, a correct pipeline must analyze the data present in both files in Figure 5) DS DS-Guru uses OPS sampling, which may not stands for missing values. Although few-shot prompting enables the agent to see relevant errors, the lack of explicit agentic control flow results in the LLM not connecting the execution errors to these fine-grained data observations. By contrast, on every agentic iteration, smolagents DR conjectures look at next ensure the correctness of the pipeline it has drafted.
realize the "M" buried in the data and deduce that "M"
what the important data This conjecture guides its tool call-enabled retrieval step. It subsequently analyzes the tool call and pipeline execution results before the next iteration. This explicit retrieve-revise revise-repeat pattern tightly couples eтror feedbacks with data retrieval, which helps address the fine grained data- dependent reasoning challenge and leads to working end-to-end pipelines.
fine-grained
Challenge 2: Holistic understanding of the input data and prior knowledge.
environment-q16: How many beuches remained sufe to swimming from 2002 to 2023 inclusive?
emvironment-416-3: How many beuches are there?
enviromment-q16-3 is example sub-task for environment-q16, which also uses files in Figure To solve the full task reliably, a system should be able to identify all beaches to start with. environment- 916-3 prompts the system to carryout this identification and verifies the result.
The challenge with beach identification is that the "Beach Name" column encodes both the beach and sampling location. Cliff Pond (DCR) @ Main refers to the Main (street) sampling location of the Cliff

---

Published as a conference paper at ICLR 2026
Pond beach (Figure 5). Facing many near-duplicate files in the data lake, systems do not have clear global schema or geographical domain knowledge that they could use to understand this encoding scheme. As result, both DS-Guru and smolagents DR failed on this sub-task, despite smolagents DR's agentic control flow. This case highlights the need to incorporate prior knowledge and discover clarifications about under-specified conventions from the data (Mao al., 2019). Towards this end, we analyzed the traces of DS-Guru (few-shot, GPT-o3 & Claude 3.5) with the agentic system diagnosis framework proposed in Cemri al. (2025). Respectively 24% (GPT-o3) and 43% (Claude 3.5) of all 104 tasks suffer from "failure to ask for clarification" and thus were not solved correctly. However, KRAMA BENCH
BRNCABENCH expects that human expert could solve the tasks without additional clarifications by exploring and understanding the data. In this example, human could decipher the beach name conventions with common U.S. geographical knowledge. Reasoning models currently lack similar capabilities to gain holistic understandings of the input data and fail to incorporate prior knowledge.
RELATED WORK
LLM--Powered Agentic Systems. There is a large and fast-growing literature on LLM-powered
Al systems. These systems take on vastly different designs, such as vanilla LLM calls to frontier
pre-trained reasoning models (OpenAl et al. [222]] DeepSeek-AL et al., 2025) Zhong et al. (022)).
-aggeeene generation (Lewis et al. 2020). agentic workflow systems (Zhang et al. 2025b).
chain-of-thoughh and iterative calls (Wei et al.) 2022; Press et al., 2023), reflections (Ji et al 2023)
and task-time verifications (Tang et al. 2024a), structured knowledge representations (Jiang et al. 2024, Su ct al. 2025; Wang et al. 2025). ). and data processing centric systems (Liu et al., 2024 Patel et al. 2025 Shankar et al. (024). Recent work applies these techniques to data science tasks. For example, Doe Wrangler (Shankar et al., 2025) is an integrated development environment that helps the user optimize LLM prompts construct data processing programs. DSAgent (Guo et al., 2024) is a framework that uses LLMs to understand user needs and build data science pipelines. Evaporate (Arora et al. 2023) helps users transform data into queryable tables. AutoPrep (Fan et al.) 2025) constructs a data preparation program over a single table for given question. Despite the progress, evaluating agent performance in real-world end-to-end setting remains a challenge.
Evaluations of LLM--Poerred Agentic Systems. Benchmarks for question answering (QA) have shifted toward evaluating agentic solutions. These benchmarks require iterative retrieval, query parsing, planning, tool use, and temporal awareness. Recent works include FanOutQA (Zhu et al. 2024), MultiHop-RAG (Tang& & Yang 2024), CRAG (Yang et al.] 2024), BrowseComp (Wei et al. 2025), which test end-to-end retrieval systems. MEQA (Li et al. (0224) for multi-hop reasoning with explanation chains, and MINTQA (He et al. 2024) for scaffolding long knowledge. These tasks differ from data science tasks, as they only require information retrieval and joins, but data-intensive processing. Benchmarks such as DS-1000 (Lai et al. (023). DA-Code (Huang et al. 2024), ARCADE (Yin al. 2023). DataSciBench (Zhang al. 2025a). DSEval (Zhang al. (024c) focus instead on implementing detailed instructions in general programming languages, specifically in data science tasks, differentiating themselves from other benchmarks like SWE-Bench (Jimenez
et al. (024)), ML-Bench (Tang el al. 2024b), BigCodeBench (Zhuo et al.] (025). More recently, new benchmarks such as DSBench (Jing et al. 2025) and BLADE (Gu al. 2024 have started to (Chen) al. (025) and BixBench (Mitchener et al.] (2025) evaluate using domain knowledge. Although such benchmarks assess specific capabilities, they fall short of capturing the full complexity of real-world data science pipelines.
evaluate the ability to create an implementation plan. Benchmarks like
CONCLUSION
KRAMABENCH evaluates the capabilities of systems to generate data science pipelines over a data lake consisting of heterogeneous, unclean input. Our comprehensive experiments using 8 LLMs across different agentic systems with KRAMABENCH reveals although current systems are equipped with useful techniques such agentic control flow and generic coding abilities, they are still far from solving wolld data science problems. Our analyses highlight several underexplored challenges such as effective retrieval, data-dependent reasoning, plan revision, and robust prior/domain knowledge integration as meaningful research directions towards practical automated data science systems.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We acknowledge the limitations of KRAMABENCH regarding its scope, language, and cultural biases, and domain coverage, which stem from the human effort required for high-quality curation. All data included is publicly available, anonymized, or pseudonymized, with no personally identifiable information. The biomedical domain contains public data sourced from the cancer data commons (CDC) this data is pseudonymized and does not require confidential access nor specific approvals, with the only sensitive attribute included as part of the workload being the pseudonymized age of patients. We emphasize privacy paramount and warn users of the benchmark against potential identification risks, which we deem unlikely, associated with this data source. In future iterations of benchmark aim broaden domain diversity, include multilingual data, and integrate community contributions to reduce existing biases Furthermore, we comply with licensing practices of data sources. For data sources that are publicly available but have redistribution constraints, do not modify or separately host these datasets. Instead, we point users of our benchmark to the original data sources.
REPRODUCIBILITY STATEMENT
We provide full artifacts artifacss-includin including code, data, workloads, and evaluation scripts- via our public repository at httpt::///////
tuu..hhub.... wwmmmiittt..ccc.c..occomom..m.mo ttmmmmiiii cbbg/ /Kr Kramabench The main paper section and Appendix D describe the process obtained to design and curate the task based on the datasets for each domain. Scripts to reproduce these steps can be found in the main repository. All datasets, benchmark frameworks, benchmark curation semi-automaat accompanying annotations, and reference system DS-Guru are available in our repository. The experimental analysis of different system under test in Section can be reproduced using Python scripts also available in the public repository.
scripts, reference pipelines and other
REFERENCES
smollgeets/cxxmpplss/oopnn deep
letpeddeeeeeeeeeeeeeeeeee research at main huggingfface/smolgeens github.com
https:://github.ccomm/ hww.am/hggggiiig ce smo h/tp:///oaggee/eeeee///aa/a//a/a
gpen_deep_researre [Accessed 01-12-2025).
Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks Microsoft Researcc
microsoft.com.
hrtprt/ccc....caaaa.a...eeanenan [Accessed 01-12-2025).
Alibaba DAMO Academy. Deepresearccher: An openn-sourcce multi-agent system for deep research.
2025. Accessed: YYYY-MM-DD.
Anthropic. Introducing claude 3.5 sonnet. https://ww..athhoopio.cco///
claude 5- 3-5-nneets 2024. Accessed: 2025-09-24.
Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes. PVLDB, 17(2):92-005, October 2023. ISSN 2150-8097. doi: 6292. 362 6294
10. 10.447788/3222292.666224 URL htttts:://oo..... /tt.L//ppp////....o00000011/////
Julia Briden, Peng Mun Siew, Victor Rodriguez-Femandez, and Richard Linares. Transformer-based atmospheric density forecasting. Advanced Maui Optical and Space Surveillance (AMOS) Technologies Conference, 2023. Free preprint available at htttss:/arriv.or//ab//231.1.0 http:/acca...aab/22211011111//// htt.../////laa/////r//////////..
Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt
Keutzer, Aditya G. Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? CoRR, abs/2503. 2025. doi: 40.48500/ARXI.2550...
URL https: setps://doot www.orggl/ooo wwwp::.4.00000...222222
13657
11

---

Published as a conference paper at ICLR 2026
National Interagency Fire Center. Statistiesl National Interagency Fire Center - nifc.gov. https
/ www..nifo.. wwwp:/ireeinooooaiiiooooooooo htw./////stttsiiiiss.. [Accessed 21-05-2025).
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning. Song Gao, Yu Su, and Huan Sun. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery, 2025. URL net forum? 22
battss://openee
F. Clette and I.. Lefèvre. Silso sunspot number v2.0. https://doi.or/// Published by WDC SILSO Royal Observatory of Belgium (ROB).
07 2015.
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang. Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin. Ruyi Chen. Shanghao Lu, Shangyan Zhou Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu. Shunfeng Zhou, Shuting Pan. S. S. Li, Shuang Zhou, Shaoqing Wu. Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning, 2025.
URL http:/ttts/:/xiiiiii org/ abs/ /201. 12948
Yongchao Dou, Emily A Kawaler, Daniel Cui Zhou, Marina A Gritsenko, Chen Huang, Lili Blumen- berg, Alla Karpova, Vladislav A Petyuk. Sara R Savage. Shankha Satpathy, al. Proteogenomic characterization of endometrial carcinoma. Cell, (80(4):729-745) 2020.
European Space Agency. Swarm Satellite Mission Data. htttp:://earth esa. int Cogateway/misssin wttp://wwwssssssmm..m.to....m.tm
swwrw/ data, 2013. Accessed: 2025-05-06.
Meihao Fan, Ju Fan, Nan Tang, Lei Cao, Guoliang Li, and Xiaoyong Du. Autoprep: Natural
language aware data preparation with multi-agent framework, 2025. URL https
//arxiv.orr/abs//211.... 10422
Federal Trade Commission. Age and Fraud Dashboard. https://public.tabeeee..u
hhgp/poofilefedderr: trade htt.aaac..c...sssooooeeee/aaaaaa 2025a. Accessed: 2025-05-06.
Federal Trade Commission. FTC Open Government Data Sets. https://www.ft.. gov/ Seciicy-notices///
government/date sets 2025b. Accessed: 2025-05-06.
Federal Trade Commission. Debt Collection Dashboard. http::/ppb.ic..ab.
com/ hetn//pppppofile//eeee fedeeiial.eeee......... trade. wtt.eecccmmmmiiii/o//oooooooo www.m/ssioon//t///aaa0lllllleeea
Infographic 2025c. Accessed: 2025-055-66
12

---

Published as a conference paper at ICLR 2026
Raphael Fontes. Us election 2020 dataset. https:/ //wwww http//wwwwkagglgee wwwggle.c.m/a...aa...
2020. Accessed: 2025-05-06.
Michael A Gillette, Shankha Satpathy, Song Cao, Saravana M Dhanasekaran, Suhas V Vasaikar, Karsten Krug, Francesca Petralia, Yize Li, Wen-Wei Liang, Boris Reva, al. Proteogenomic characterization reveals therapeutic vulnerabilities in lung adenocarcinoma. Cell, 182(():200-225,
2020.
Google. Agent mode in gemini code assist.
http:://deeeeoorrssscc.....
gemi ni seabbe-assiss//oos//saeeee
2025. Accessed: 2025-09-24.
Google DeepMind. Gemini 2.5 pro. htttps:/deepminue http://ww....oogge/////ee///////
2025. Accessed: 2025-09-24.
Huw Groucutt, Tom S White, Eleanor ML Scerri, Eric Andrieux, Richard Clark- Wilson, Paul Breeze, Simon J Armitage, Mathew Stewart, Nick Drake, Julien Louys, et al. Multiple hominin dispersals into southwest asia over the past 400,000 years. Nature, 597(7876):376-38) 2021.
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang. Tianmai M. Zhang, Lanyi Zhu, Mike A Merrill, Jeffrey Heer, and Tim Althoff. BLADE: Benchmarking language model agents for data-driven science, November 2024. URL
wwtp://oooooo.../22//2///// wtt.:/2......ininnnniiii findings- htt.sseemmlp..................1.
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: automated data science by empowering large language models with case-based reasoning. 2024.
Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, and Jeff Z. Pan. Mintqa: A multi-hop question answering benchmark for evaluating Ilms on new and tail knowledge, 2024. URL https
//arxiv.org//ae hbwgg/bb//4112. 17032
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. Da-code: Agent data science code generation benchmark for large language models, 2024. URL /ttp::///li.crrr/111115555
2024. commp--main... 748
Hugging Face Open deep research smolagents/exxapplee/opnn _eeep_deeee____eeeeeeeee_e _er___ddeeee_eeddl__ee__ research. https:///gtthii com/ huggingface. /smol htgents//ree/mmmm////ee open Bee_n_dee__eeseeeeee
2025. Accessed: 2025-09-24.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jin- gren Zhou, and Junyang Lin. Qwen2.5-Coder Technical Report. arXiv preprint arXiv:2409. 12/86, 2024. URL a arwivv www.:..r/aaa....... 2409.1.1 12186
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigat- ing LLM hallucination via self reflection. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1843, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1208553/11/2223nnnnnnnn--noo findiingseennl..1222.
URL https: ftttss://aclanthology 2orgo//tthooloyyooo/2223323
f Findings emn 1p. 123/
Zhouyu Jiang. Ling Zhong, Mengshu Sun, Jun Xu, Rui Sun, Hui Cai, Shuhan Luo, and Zhiqiang Zhang. Efficient knowledge infusion via KG-LLM alignment. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, pp. 2986 2999, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/ 11/2024.indinns--. 1767]
URL https: ac www.anooooo..oooo/oee..oooo http///224...nn....... findi ngs acl
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bencht Can language models resolve real-world github issues?, 2024. URL
https://cpenreviee net httt:/orummmldeeryyyyyya
13

---

Published as a conference paper at ICLR 2026
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma. Hongming Zhang. Xinya Du, and Dong Yu. Dsbench: How far are data science agents from becoming data science experts?, 2025.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: a natural and reliable benchmark for data science code generation, 2023.
Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models real-world enterprise text- to-SQL workflows. 2025.
httpttppss///peeeeeeeeeee net forum d-dddmmm ro 9cPs
Quentin Lemesle. Jonathan Chevelu, Philippe Martin, Damien Lolive, Arnaud Delhay, and Nelly Barbot. Paraphrase generation evaluation powered by an LLM: A semantic metric,
lexical one. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the International Conference on Computational Linguistics, pp. 8057-8087, Abu Dhabi, UAE. January 2025. Association for Computational Linguistics. httpp://:
www.o/ggyoooooooooooo..o.goeeg 2025 coling 538/
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensee
nlp tasks, 2020.
Ruosen Li, Zimu Wang, Son Quoc Tran, Lei Xia, and Xinya Du Meqa: A benchmark for multi-hop event-centric question answering with explanations. In Advances in Neu- ral Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, newrine www.eeips.....ppeeeeeeeeeeeeeeee wwtpppereppeaeeeeeeeeeee faaeeeerrffeeepppepeeeeeee filles/ppeer/224 _aattasets____________
2024. URL
hash/ Benchmarks Track.htm!
Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, and Gerardo Vitagliano. declarative system for optimiz- ing workloads, 2024. URLhhttss://rxxitt hattt:ss//axxi..orr//////////// 14..00...46966
Yaoli Mao, Dakuo Wang, Michael Muller, Kush R. Varshney, Ioana Baldini, Casey Dugan, and Aleksandra Mojsilović. How data scientistswork together with domain experts in scientific collaborations: To find the right answer to ask the right question? Proc. ACM Hum.-Coppt. Interact,, 3(GRUUP), December 2019. doi: 10.1145/3361118
1145/3361118
Massachusetts
Department of Public Health.
Water
Quality
Mas-
sachusetts Swimming Beaches.
https:/ www. mass. hov/lisss/
aatea-quultey-a--maascccsstttte Doggee 2025-05-06.
2025a.
Accessed:
Massachusetts Department of Public Health. Water Body Testing Report Massachusetts Environmental Public Health Tracking Network (MEPHTN),
hhs
mass gwt
[cl.eeees222eerrnn22eeeeeoooeeee 1B8 1B550008863333887773333333333333
2025b. Accessed: 2025-05-06.
Massachusetts Water Resources Authority.
Beach
Fact Sheets Massachusetts
Water Resources
Authority (MWRA).
https: hampps://wwwt /wwww mwra. com hoomhhabor//
2025a. Accessed: 2025-05-
06.
Massachusetts Water Resources Authority. Download Environmental Data
Mas-
sachusetts Water Resources Authority (MWRA). https: htops://wwy, www.wwwa.....m com/ cww/harbo/
lownload-enviree -ta.eeeaaal a
2025b. Accessed: 2025-05-06.
Meta AI. Llama 3.3: Model cards and prompt formats. ht /tt.pps://ww... www. 11ama. www. com/docs/ docs/
model ca rds- rdd-aandpromppl wwwpppppttoooatto/oaommmmo 1atss//11mmmm3333 2025. Accessed: 2025-019-24
14

---

Published as a conference paper at ICLR 2026
Ludovico Mitchener, Jon M Laurent, Benjamin Tenmann, Siddharth Narayanan. Geemi P Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. Bixbench: a comprehensive benchmark for Ilm-based agents in computational biology, 2025.
00096
Daye Nam, Andrew Macvean, Vincent Hellendoomn, Bogdan Vasilescu, and Brad Myers. Using an Ilm to help with code understanding. In Proceedings of the International Conference on Software Engineering (ICSE), pp. 1-13, 2024.
National Centers for Environmental Information (NCE), Wildfires National Centers for Environ-
mental Information (NCEI). httpps:/ //wp:://wwww..//// www.nneiino... noaa. http://ccees/monttoriiie
Wi wildfirees] 2025. Accessed: 2025-05-06.
National Interagency Fire Center. Fire Information National Interagency Fire Center (NIFC).
2025. Accessed: 2025-05-06.
National Weather Service Climate Data for Boston (BOX) Office - National Weather Service.
https://www.
wwtp://wwww... http://wwwwwtttt...ggggog hww.::thh...o..ooooooooeee http:s//www.vattaee.ggg///////b
wo 2025, Accessed: 2025-05-06.
NCHI.MoniorriggIInfoneee oot.iiirrnnnnnnnooooooooooooooo Monthly Climate Reports I National Centers for Environ-
mental Information (NCEI) ncei.noaa.gov. https:: //www www. ncei noaa. aoeaa.govvccces.
moniitoring/month]] htt.://itnn/nooottt-rrerrrrrr// 2025. [Accessed 21-05-2025).
NOAA Office of Satellite and Product Operations. NOAA Geostationary Operational Envirommental Satellite (GOES) I-M and N-P Series Imager Data. NOAA National Centers for Environmental
Information, 1994. URL ht 2592110259992222992 -
Accessed: 2025-05-
06.
OpenAl. Hello 2025-09-24.
2024. Accessed:
OpenAI. Introducing deep research.
http:::/operai.commmmm
introducing- d 2025. [Accessed 13-05-2025).
OpenAl. Introducing openai 03 and 04-mini.
bttps://openai.com com/inexx index/
Introtueing-o33-and------
2025. Accessed: 2025-09-24.
OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Kl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace. Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattisaa Parascandolo. Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman. Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, 1lya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu. Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko
15

---

Published as a conference paper at ICLR 2026
Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofrr Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders. Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov. Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. URL URL httptttpp:/aaatt
ozg/ abs/ 2412. 16720
Natalia E. Papitashvili and Joseph H. King. Omni daily data. NASA Space Physics Data Facility, 2020a. Ut..htttps://6o..rrr/// 48322/ 00188/2////222222 5fmx- Accessed: 2025-05-06.
Natalia E. Papitashvili and Joseph Joseph H. King. Omni hourly data. NASA Space Physics Data Facility,
2020b. URL
2tt.://///....222.... 48322/ 1shr 18
Accessed: 2025-05-06.
William E. Parker and Richard Linares. Satellite drag analysis during the may 2024 gannon geo- magnetic storm. Journal of Spacecraft and Rockets, 11: September 2024. ISSN 1533-6794. doi: 10.2514/1.a36166
URL http:
doii wrg/o0..o/...o...... wttgg//...///...... 2361441..36111111 2361644
Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. Semantic operators: declarative model for rich, ai-based data processing, 2025. URL
org. abs/://a////// 2407. 1124407.11411 11418
Ofir Press. Muru Zhang, Sewon Min, Ludwig Schmidt. Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3687-5711 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023, findings-emmllp3788. Findings emnlp.
URL
Xiao Pu. Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead. arXiv preprint arXiv:2309.095550 2023.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, al. Tool learning with foundation models. ACM Computing Surveys. 57(4):1-40, 2024a.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong. Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. 2024b. URL https:: /tttp::///penrevie.. nwtp:/oummfffttttt..mee
dling200Jix
Robikscube. Zillow home value index (zhvi). https://www.kaggt httpa/gggllecom///tattttttss robikscube/zillor- value index 2021. Accessed: 2025-05-06.
Fleanor Scerri, James Blinkhorn, Huw S Groucutt, Mathew Stewart, Ian Candy, Ethel Allué, Aitor Burguet-Coca, Andrés Currás, W Christopher Carleton, Susanne Lindauer, et al. Hunter-gatherer sea voyages extended remotest mediterrancan islands. Nature, pp. 1-7, 2025.
Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. Docetl: Agentic query rewriting and evaluation for complex document processing, 2024. URL https 7/a rxivv www.///////aaaeeee /2100 12189
16

---

Published as a conference paper at ICLR 2026
Shreya Shankar, Bhavya Chopra, Mawil Hasan, Stephen Lee, Bjöm Hartmann, Joseph M. Hellerstein, Aditya G. Parameswaran, and Eugene Wu. Steering semantic data processing with doewrangler. 2025. URL/tttss://axxtttttt
org/ htl/aab//200 abs./504.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL
https: htt///axxiiv.. /ttp//xxx.........a./.. http://///////233...333
C. Siemes, de Teixeira da Encarnação, E. N. Doombos,, J. van den IJssel, Kraus, R. Pereštý. L. Grunwaldt, G. Apelbaum, J. Flury, and P. E. Holmdahl Olsen. Swarm accelerometer data processing from raw accelerations to thermospheric neutral densities. Earth, Planets and Space, 68:92, 2016. doi: 10.1186/440633-01-64144 s40623 016-006--100 0474-5
1186/
Hanchen Su, Wei Luo. Yashar Mehdad, Wei Han, Elaine Liu, Wayne Zhang, Mia Zhao, and Joy Zhang. LLM- ffi--riindly knowledge representation for customer support. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Kareem Dar- wish, and Apoorv Agarwal (eds.). Proceedings of the International Conference on Computational Linguistics: Industry Track, pp. 496-504, Abu Dhabi, UAE, January 2025. Association for Com- putational Linguistics. URL https:://///.. //ac:
anthology ortoorgg/22202 http::://acaatthoooogooo co wow.iooonnnnnnnoooo inggstrnduryyy
427
Nan Tang, Chenyu Yang, Ju Fan, Lei Cao, Yuyu Luo, and Alon Y. Halevy. Verifai: Verified generative ai. In Proceedings of the Conference on Innovative Data Systems Research (CIDR), 2024a.
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang. Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang. Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. Ml-bench: Evaluating large language models and agents for machine learning tasks repository-level code, 2024b. URL /ttps::/aaxiv.. org/
2311.09833
Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking retrieval-auggenntee generation for multi- hop queries. In Proceedings of the 1st Conference on Language Modeling (COLM), Philadelphia, PA. USA, 2024. URL https: http::://openeeeee hwwp///wwpee.e.epoo//pppeoe.ee.p net/ www.oooiootteeooo.ooooo Fo COLM 2024.
Weixi Tong and Tianyi Zhang. CodeJudge: Evaluating code generation with large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNIP)), pp. 20032-20051 Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 00111865511/222 emnlp main. mtt.p.aaainn1...............111.
amnlp-main.. 1iaiin.1118. 111. URL attpps://allenthooooyy..oo hwwp:/loooyygg/2222222
U.S. Air Force and NOAA Space Weather Prediction Center. and F10.7cm Flux Forecast. Seaf-45-day-ap-andde fox 2025. Accessed: 2025-05-06.
USAF 45-Day Ap
https://www, /tt::://wwww. wwwpwwwwwwwwwwwwwwww.ppp.www.p.p noaa ttwppooductss/////oooooo
U.S. Census Bureau. National Population Totals: 1atp://00000-aaaaaa-aaaaaaaaaaaa data/tables
gov/
series/ perrleee/eemo/pppee//20000---1 2025. Accessed: 2025-05-06
U.S. Environmental Protection Agency, Air Quality System (AQS) Annual Data Download. https
// //aqs.. epa. gov/ /ooo//aqee agsweb/ att.aaaarraaaaaaaaaaa cessed: 2025-05-06.
files. html 4Annual 2025. Ac-
U.S. Space Command. Two-line element sets (tles) from space-track.org. https: www Sraccettrack.org 2025. Accessed: 2025-05-06.
Jianxun Wang and Yixiang Chen. A review code generation with llms: Application and evaluation. In IEEE International Conference on Medical Artificial Intelligence (MedAl), pp. 284-289. IEEE, 2023.
17

---

Published as a conference paper at ICLR 2026
Xi Wang, Taketomo Isazawa, Liana Mikaelyan, and James Hensman. KBLam: Knowledge base augmented language model. In Proceedings of the Interaational Conference on Learning Repre- sentations (ICLR), 2025. URLhttts://ppeeeeeee httpp//wwwwn.tooooroooaaaaaaaaa
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le. and Denny Zhou, Chain-of-thought prompting elicits reasoning in large language models, 2022. URL http::/ htt.appprrr..atsss.....pppepeeea
/ html
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung. Alex Tachard Passos, William Fedus, and Amelia Glaese Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. 2025.
Wikipedia contributors. Metropolitan statistical area - Wikipedia, The Free Encyclopedia. httpss 7/en. wi wiikipeeen aipedia.ogg//iaieee /Met ropo wttpiiitttttssttttttiaieeee _raa 2025. Accessed: 2025-05-06.
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang. Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang. Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen-tau Yih, and Xin Luna Dong. Crag comprehessive rag benchmark. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024. doi: 104485500amxxx.2444644444
URL
neur www.pppp....pppr.r.e.e. ftap//////1//////////22/44 3ddcb Benchmarks Trasss__rrcrooo html
https: https://proceedingg
Abstract Datasets_aaaaaa and
See arXiv:2406.04744 for the full version.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural language to code generation interactive data science notebooks, July 2023. URL 97
Jesse D. Young, Alexander M. Evans, Jose M. Iniguez, Andrea Thode, Marc D. Meyer, Shaula J. Hedwall, Sarah M. McCaffrey, Patrick Shin, and Ching-Hsun Huang. Large wildfire incident status summary (ics-209) report-generated data for the western united states, 2002-2016. Forest Service Research Data Archive, Fort Collins, CO, 2021. URL
RDS- 20S-202110100)
Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao, Ziyue Li, and Hangyu Mao. Benchmarking the text-to-sql capability of large language models: A comprehensive evaluation. arXiv preprint urXiv:2403.0295. 2024a.
Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang. Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. Datascibench: An Ilm agent benchmark for data science, 2025a.
URL http:httt///aaat /ttp:s////rvvvvaai wwwpxivv.orggaeoo httg/bbs/2502. 13897
Jayy Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng. Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation, 2025b. URL http::tttpp:pprota
Yang Zhang, Hanlei Jin, Dan Meng, Jun Wang, and Jinghua Tan. A comprehensive survey on process-oriented automatic text summarization with exploration of Ilm-based methods. arXiv preprint arXiv:2403.00900 2024b.
Yuge Zhang, Qiyang Jiang, XingyuHan XingyuHan, Nan Chen, Yuging Yang, and Kan Ren. Bench- marking data science agents, August 2024c. URL httpp::/aalaathhoooo...ooooo org.:0/2yoyyyo.2299999
acl acl-long. 3087
Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei
18

---

Published as a conference paper at ICLR 2026
Zhao, Zhenyuan Yang, Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang. Huan Zhao, Jing Zhang, Yiming Ren, Shihuan Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang. Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang, Guoyu Lu, Xianqiao Wang, Lilong Chai, He Li, Jin Lu, Lichao Sun, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang, Ninghao Liu, Bei Jiang, Linglong Kong, Zhen Xiang, Yudan Ren, Jun Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai, Dajiang Zhu, and Tianming Liu. Evaluation of openai ol: Opportunities and challenges of agi, 2024. URL https:// awr.v..rivvv./. org/ abs /ww.////23322 18486
Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch FanOutQA: A multi-hop, multi-document question answering benchmark for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 18-37, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10..18663.11/222:4.ccoshoor
URL
https:/7ee antaanaoooolo wonthoooog...oo .og.lllgg......l..........
wrg. 2024. -cl.-corrll.h22222
Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He. Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li. Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng. Jiawei Liu, Qian Liu, Zijian Wang. David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. 2025. URL /ttep:://penreevew. net www.muummeeeteeeeemmmmm Yrycijlyyyyyi
19

---

Published as a conference paper at ICLR 2026
A EXTENDED EXPERIMENT RESULTS
In this section, we supply the full evaluation results for which we presented a summary of in the main text due to space constraints.
Table 9: Results by domain for KRAMABENCH on DS-Guru and smolagents DR with Full mode.
System
Models
Archeology
Astronomy
Biomedical
Domains
Environment
Legal
Wildfire
Overall
a 00.00
a.30
= 00:00
14.18 +
5.87
GPT-4o
10.00
0:00
+
00:00
0.00
0.00
00000
DS-Giaru
Lima- 3.Issmmttt
0.00
00.00
0.00 a 00.00
0.00 00.00
0.00 1 00.00
0.00 = 00.00
1.59 1: 2:24
0.33 0.46
Do- no-coutest
Deepseok 81
2.78 3.53
0:000 a 00.00
0.00 = 00.00
0.80 2: 00000
0,00 = 00:00
0.00 2 10.000
0.55 0.46
Qwnn? SCalor
0.00
1.37
2.02
1.07
1.44
13.68
3.72
Clande 3.5*
16.67
1.62
2,87
1,17
1.33
13.63
7:45
Clande-3.7
2.78 3,93
3.33 4.71
0.00 00,00
0.00 a 00.00
1.11 1,57
4.38 3,92
1,88 0.81
GPT-63
19 3,93
0,00 00.00
0.53 0.75
16.11 1 9.26
10:00 00,00
60.01 4.22
16.67 2.93
GPT-40
8.33
4.71
3.70
00000
4.41 2.17
13.69
DS-Cum
1ans- I-memtmc
16.67
00
0.10
00.00
sbot
Deepseok
0.00
0:00 00.00
0 00.90
00:00
00.00
00.00
Qwen2-5 Coder?
0.00
1.36
12:59
1.15
16.48
6,45
Chudk-3.5
0.00
4.15
2:15
6.21
6.68
34 99
10.85
Clande 3.7
2.78 4 3.93
0.00 a 00.00
0.00 2: 00.00
3.53 a 2.36
0,00 = 00:00
1.59 4 2.24
1,31
1.22
GP F-
1589 3.33
0,00 a 00.00
0.10 0.14
49,56 2 (187
9.26 = 292
52.92 + 0.72
24.98 1:25
GPT-4o
13.89 + 3.01
3.33 4.71
0.00 + 00.00
0.00 + 00.00
4,84 + 1.16
25.69 + 1.15
8.67
1.48
DS-Gimu
Liama 3.3 Instruct
22.22 3,90
0.00 2 00.00
0.20 0.14
0.00 1 00.00
3.53 a 00:00
23.25 + 3.06
8,40
1.01
lew sbot
Despseck
0.00 80,00
0.00 = 00.00
0.00 00.00
0,00 + 00100
11000 = 00:03
1.59 + 2.34
0.53 0.46
Quen2: Quen2-SCadert* SCader*
8.33
2,40
4.35
12.64
9.06
16.48
9,98
Clack-3.55
16.67
1.56
11:21
14.35
Clande
3,90
00.00
a
6.89 6.22
10.09
1.12
6.30
0:00
0.00 00:30
1.42
22.01 +::
7.57
amollgeets
Chande-3.5
GPT447
33.33
0.00
8:00
22:22
11:11
35.001
60.00
46.67
40:00
$2.38
10
41.35
30.77
Clande-3.7 3.7
44.44 3,93
50,000 a: 8.16
38.89 +: 7,86
00.56 2: 7.34
61.23 = 12.70
50.16 + 7.43
55.83 4 3.41
smolegenss Reflexion
Clanke3.7
GP F-o
30.56 : 10.39
41.67 0.000
46.67 4.71
45.00 - 5.00
50 27.78
23.24
23.89 # 14.55
56.25 8.75
26.67 = 14.40
58.33 1.67
35.04 # 8.36
65. 38 0.86
30.55
55.37
10.79
3.36
smolagents PDT
GPF-63 Chadee3,7
19.44 7,36
$.33 6.30
0:00 a 00.00
6.67 4.71
0.00 00.30
5.56 + 5,56
1.42 1.09
9.84 + 2.86
4,51 s: 0.84
6.25 4.95
22.01 + 1.39
22,58 +
7,57 1.28
12.01 1.03
Table 10: Results by domain for KRAMABENCH on DS-Guru and smolagents DR with Oracle mode.
System
Models
Archeology
Astronomy
Diomedical
Domains
Environment
Legal
Wildiire
Total
GPT-33
11.11 3,93
10:00 8.16
11.11 0,00
0.00 0.00
0:00
0.00
10.17 1
0.92
5.36
1.20
GPT-4o
0.00
0.000
0.30
0.00 0.00
0:00
0.00
0.00
Laamu-333 Inneett
5.56 3.90
0.00
0.00
0.00
0.00
0.00
no-context
Deepseek
0.0 0.00
0.00
0.00
0.30
0.00
0.00
0.00
0.00
Quen2-SCader
6.74
4:53
6.62
Chande 3.5+
16.52
10:63
9.87
12.51
9.80
0.00
11.63
Clande-3.7
4.17 4.17
10:00
0.00)
0.00 0.30
1.5% # 1.58
1,07 = 1.67
8.96 + 0.57
4.11
0.79
GPT-o3
8.33 6.50
16.67 a 4.71
7.51 5: 5.31
23.89 à: 6.71
15.56 à 1.57
42,50 :: 3:48
21.35
0.84
GPT-4e
11.11 7,36
13.33 4.71
5.19 7.33
12.35 à 3.30
6.67 s 0.00
19:85 + 3.61
11.54
1.47
DS-Giui
I Tammo 3-Ieetrue
0.00
0.00
6.67 + 4.71
7.41
5.24
0.00
0.00
0.00
0.00
26.37
+ 5,00
6,74
1.49
une abot
Deepseck
2.78 3.93
6.67 à 4.71
0.00 0,00
0.00
0.00
0.00
0.00 0.00
0,96 0.80
Qwea2 SCnder?
11.57
15.13
8:96
13.22
Clande33.5
22.27
17.93
Claude
0.00
0.00
0.00 0.00
0.00 0.90
6.67 à 1.67
3.33
4.76
3.27
16.67 0.00
26.67
4.71
7 5.24
20.11 +:
01.033
52.02 +: 3.79
43.71
GPT-4o
3.93
20:00
0.00
3,70 + 5.24
6.00
32.78
4.16
36.67
26,20
DS Guru
Lhma 3.3 Instruct
13.89
3,93
10.00
0.00
11.11 =: 0.00
0.00
1 0.00
6.67 =: 0.00
29.72
1.27
11.67 0.65
few-sbot
Deepseek 81
5.56
3.53
6.67 4.71
0.00 0.30
0.00 + 0.000
11.00 = 0.00
0.00 + 0.000
1.51
0.46
SCader
11.83
1491
7.51
18.39
13:70
18:51
15.15
Clande 3.5
16.24
14,02
14.90
13,83
26.36
02
24.22
Clande33.7
20.83 + 4.17
25.00 + 00
0.00 0,00
9.17 9.17
31.67
1.67
18.17
5.20
19.75
3,36
GPT
27.78 7,36
26.67
4.71
37.04
5.24
24.00
0.00
17.78 a 5.67
32.89 1 10.31
25,86
4.57
amolagents
GPI-467
Clande33.5
25.00
16.67
25.00
25,00
22.22
33.33
25.00
20.00
58.10
66
39.00
Clande 3.7
2
65.00
5.00
58.33
2.78
46,83 à:: 1.50
75.08
60.67
smolagenes Retiction
GP Chndd--3.7
30.56 3.55
37.50
4.17
40:00
50:00 + 0.00
0.000
4
83.33
0.30
5.56
31.45
62.67
2:20
4.33
22.22 à
70.00 m 10.00
64.45
39.78 +:
1.30
22.35
62.81
smolagents PDT
GPTe3
Clande 3.7
16.67 4 0.00
* 6.50
10:00
0:00
0.00
0.00
11.11 1 0.00
0.00
0.00
11.50 4
3.26 + 1.30
3.50
9.24 2 0.35
m 3.88
25.50 4: 5.46
15.86
14.38 2:15
7.69
B DS-GURI DETAILS
The baseline system we provide, DS-Guru, follows a simple design. For each task, the system provides the backend LLM with an informative sample of data from each file in the data lake first as well as the task prompt. DS- Gurru leverages instruction tuning to guide the LLM backend to provide Python implementation of the task pipeline well structured explanation of the steps to be
20

---

Published as a conference paper at ICLR 2026
Table 11: Results by domain for KRAMABENCH (Trimmed input lake). marks web-browser on.
System
Metric
Archoology
Astronomy
Biomedical
Enviromment
Legal
Wildfire
Total
DS-Oumm few shot (GFTod)
Score
25 a
0.00 + 0.00
3.70 5.24
48 31
51.11 +
48.87 7.90
37
Arg. tunimelaask (min)
2,8502
1A584
14,4576
2.3543
3.8403
2.3559
3,8076
smolagonts DR
Seore
50,00 0.00
N0.00 11.00
55.56 0.00
40.83 0.00
60,00 & 0J00
67.24 4
58.12 0.00
rumsimettskk (min)
2,3538
5,2044
17,9464
3.6733
3,4209
3.2526
4,8074
smolagents
Score
43.33
52.81
Avg. runtimetuskk (min)
6.5276
1.3909
smolagents PDT
Scure
16.67 a 0.000
0.00 a 0.00
0.00 0.00
14.37 0.00
13.33 0.00
0.00 0.00
8,70 0.00
Avg. tunimeltakk (min)
7,2708
6,304]
15.5661
6.6799
8.031
8.5672
8.2438
OpenAI DR
Scare
40.00
33.33
44.45
61.67
50.00
67.28
52.18
Avg. rumimelask (min)
8.105
20.16
10.67
53
8.68
12.62
10.35
Gemini Prox
Scare
25.00
16.67
33.35
25.00
13.33
24.87
18.48
Arg. (min)
Table 12: Cost-accuracy Tradeoff between different SUTs under Full imput mode.
Accuracy/ Runtime
Accuracy / 1k In Tokens
Accuracy / Ik Out Tokens
SUT
Overall Accuracy
Overall Runtime
GPTo3 Naive
05.87 = 00.71
0lb 01m 41s 06m 26s
00.17 00.02
04.73 = 00.51
02.69 +: 00.31
GPTo3 One Shot
16.67 02.93
08m
13s
00.42 00.12
00.43 = 00.07
09.16 63
GPTb3 Few Shot
24.98 01.25
02h 30m 42s 42m
34s
00.31
00.07
00.29 00.05
08.58 00.61
GPT4o Naive
00.00 = 00.00
38m 42s 02m
23s
00.00 00.00
00.00 00.00
00.00 = 00.00
GPT4a- One Shot
06.08 + 011.25
39m 15x 022m
11s
00.27
00.06
00.45 + 00.09
09.81 # 02.06
GPT4o- Few Shot
08.67 01.48
01h 13m 13s 18m
12s
00.22 00.05
00.26 00.03
07.68 01.33
Llama3 Instruct -Niie
00.33 +: 00:46
35m 09s 03m 53s
00.01
00.02
00.26 00.37
00.37 +: 00.52
Lluma3 Instruct One Shot
03.42 = 00.37
0lb 38m 51s 11m 06s
00.06 00.01
00.25 = 00.03
05. 63 + 00.61
Llama3 Instratt Shot
00.03
00.04
DeepseekR] Naive
00.33 00.46
0lb 36m 07m 12s
00.01 00.01
00.26 37.37
00.16 00.23
DeepseekR1 One Shot
00.00 + 00.00
01h 46m 09% 044 42s
00.00 00.00
00.00 = 00.00
00.00 + 00.00
DeepseekR1 Few Shot
00.33 1: 00.46
0th 56m 12m 06s
00.01 00.01
00.01 1 00.01
00.15 a: 00.22
taken. DS-Guru then executes the implementation and iterate with the LLM pipeline to debug and improve the pipeline by supplying outputs and error messages.
The prompt used to instruct the LLM backend to provide pipeline for the end-to-end task is presented below:
B. SYSTEM PROMPT
You are a helpful assistant that generates plan to solve the given request, and you'll 11 be :iven:Your task is to answer the following question based on the provided data sources. Question: (query)
Data file names:
The following is a snippet of the data files: (data) New think step-by-step carefully. First, provide step-by reasoning of how you would arrive at the correct answer.
Do not assume the data files are clean well-structured (e.g., missing values, inconsistent data type in a column). Do not assume the data type of the columns is what you see in the data snippet (e.g., 2012 in Year could be a string, instead of int). your subsequent code relies on the correct data type (e.g., cast two columns to the same type before joining the two tables).
you need to convert it to the correct type if
You have to consider the possible data issues observed in the data snippet and how to handle them. Output the steps in a JSON format with the following keys:
21

---

Published as a conference paper at ICLR 2026
id: always "main-task" for the main task. For each subtask, use "subtask-1", "subtask-2", etc.
query: the question the step is trying to answer. Copy down the question from above for the main task.
data_sources: the data sources you need check to answer the question. Include all the file names you need for the main task.
subtasks: list of subtasks. Each subtask should have the same structure as the main task. For example, JSON object for the task might look like this: (example_ison)
You can have multiple steps, and each step should be JSON
object. Your output for this task should be a JSON array of JSON objects.
Mark the JSON array with (json_notation) to indicate the start and end of the code block. Then, provide the corresponding Python code to extract the answer from the data sources. The data sources you may need to answer the question are: falle_pahhs..
If possible, print the answer (in a JSON format) to each step you provided the JSON array using the print() function. Use "id" as the key to print the answer.
For example, if you have an answer to subtask-1, subtask-2 2, and main-task (i.e., the final answer), you should print it like this:
print (json.dumps. (("subtask-1": answerl, "subtask-2": answer2,
"main-task"; answer 1,, indent=4))
You can find suitable indentation for the print statement. Always import json at the beginning of your code. Mark the code with (notations) to indicate the start and end of the code block.
B.2 ABLATION STUDIES
We have started conducted ablation studies on key hyper-parameters using the rest-performing configuration of DS-Guru (i.e., self-correcting with GPT-o3), Here our preliminary findings: The quality performance is positively correlated to token usage [1]. When varying the number of sampled per table, our result is consistent success goes up then observed decrease at n=100, which is caused by the limited context window and our naive sampling algorithm. DS-Guru falls back data snippet when the prompt exceeds the context limit. DS-Guru showed consistent success across different numbers of maximum tries, with an initial slight increase. This potentially has two implications: (i) compile/runtime errors are not the major cause of failures; (ii) in a single-agent system, it may be difficult for the agent to get unstuck from loop when fixing the error. We will discuss this in depth in failure analysis. We will update the paper to present these results and discuss them analytically under our 3-level evaluation framework. For reference, the full table of results as follows: Varying the number of rows sampled in the input data snippet.
sample
rows. We
Table 13: Ouru with GPT-o3: performance and cost across different numbers of iterations.
Number Iterations
10
15
20
Overall Performance (%)
23.36
22.83
20.73
21.33
Tokens/Iteration (Mean) 64,548.9
72,926.3
70,845.1
72,301.7
22

---

Published as a conference paper at ICLR 2026
Table 14: Runtime performance by number of sampled rows per file, Runtime in seconds.
SUT
Archeology
Astronomy
Biomedical
Environment
Legal
Wildfire
Overall
Runtime
10 Rows
18.75
12.80
8.63
34.52
13.32
37.42
22.89
732.45
50 Rows
23.48
10.55
7.87
37.60
14.08
40.63
655.61
100 Rows
20.61
11.95
8.53
34.84
12.20
40.60
23.36
1374.82
150 Rows
21.08
10.58
8.64
31.68
13.09
39.22
22.58
802.90
Table 15: Performance by number of tries. Runtime in seconds.
SUT
Archeology
Astronomy
Biomedical
Enviromment
Legal
Wildfire
Overall
Runtime
Tries
20.61
11.95
8.53
34.84
12.20
40.60
23.36
1374.82
10.79
22.83
$75.88
15 Tries
20.47
7.00
8.72
36.84
9.51
31.47
20.73
721.95
C SMOLAGENTS AGENTIC BASELINE DETAILS
In this section, we describe the single-agent and multi-agent baselines systems we evaluated on KRAMABENCH more.
C.I SMOLAGENTS DR
For the single-agent baseline, we use the open source deep research implementation by smolagents (git). This agentic framework follows a canonical think - action - response loop with agentic actions expressed in code. In addition to code, the system is equipped with text inspector capable of processing different common formats originally released with Microsoft Magentic One (mic). While the official implementation also equips the system with web browser by default, we disabled the internet access to allow for direct comparison with DS- DS-Guru.
C.2 SMOLAGENTS REFLEXION
Our first multi-agen baseline is Reflexion (Shinn et al. 2023). In addition an Actor agent, Reflexion (Figure 6) introduces (1) An Evaluator agent which provides internal feedback by evaluating the outcome of each action. (2)A Self-reflection agent which provides external feedback with the outcome and the evaluation of the action. Compared to traditional reinforcement learning techniques, feedback Reflexion are expressed with natural language and stored in agent memory to guide future actions.
C.3 SMOLAGENTS PDT
Our second multi-agent baseline is based on AutoPrep (Fan ct al. 2025), a framework for natural language question answering over tabular data. We augmented AutoPrep with tools for parsing non-tabular data and the hierarchical task decomposition technique similar with DS-Guru to address the complexity of KRAMABENCH tasks. The original AutoPrep pipeline involves three agents: (1) Planner (2) Programmer (3) Executor. With the augmentations we implemented, the system employs two agents respectively playing the roles of (1) Planner and (task) Decomposer (3) Tool Execulor. We implemented this approach also using smolagents and illustrate the architecture in Figure
D DATASET DETAILS
The six input domains with the associated studies that we used to design our benchmark tasks are:
Archeology: the data files consists chronological, archaeological, faunal, and botanical data supporting the presence of Holocene hunter-gatherers on the Maltese Islands in the Mediterranean from roughly 8000 years ago to 7500 years ago. The files were collected from the publicly available
data associated with the papers Groucutt et al. (202 /Sceri et al.) (2025).
Astronomy: the data files consist of the OMNI dataset PPpitashvili & King (2020a b) that contains near-Farth solar wind, plasma, and magnetic field data, the Swarm dataset Siemes et al.) (2016):
23

---

Published as a conference paper at ICLR 2026
Agent
External feedback
Self-reflection (LM)
Internal feedback
Reflective text
Evaluator (LM)
Experience (long-term memory)
Trajectory (short-term memory)
Actor (LM)
Obs / Reward
Environment
Action
Figure 6: Architecture diagram of Reflexion. Reproduced from Figure 2(a) in Shinn et al. 2023
Took
Use of aubtaska
Took Executor
Decomposer
Fencution Engine Environment
Do sub tasks one-by-one
Figure 7: Architecture diagram of smo smolagents-poe
P
Gold Answer
Key Fungtonaiity
Task
Refererce
3
Sub: Task
Intermedidte Arswer 3
Gold Final Artwer
Stn
Key Functionaity
Gold Intermediae
Bub-Taste 2
Figure KRAMABENCH maps system evaluation both pipeline-design and sub-task-level evaluations, enabling analysis of why models succeed or fail beyond end-to-end performance.
24

---

Published as a conference paper at ICLR 2026
European Space Agency (013) that contains the magnetic field and geomagnetic field data, the SILSO Sunspot Number data Clette & Lefevre (2013). Space-Track.org Two-Line Element Sets (TLEs) U.S. Space Command (2025), the National Occanic and Atmospheric Administration (NOAA) Flux Forecast dataset U.S. Air Force & NOAA Space Weather Prediction Center (0025), and NOAA GOES Satellite dataset NOAA Office of Satellite and Product Operations (199aaiions(1994)) The combination of these datasets has been used to analyze how activity from the Sun affects Earth's ss
atmosphere, ocean currents, and weather by the authors of Briden et al. (2023 Parker & Linares (024)..
Biomedical: the data files consist of the Prote-ogenomic characterization of 95 prospectively collected endometrial carcinomas, respectively for 83 endometrioid and 12 serous tumors. Extensive analysis are done on these datasets to understand proteomic markers of tumor subgroups and regulatory mechanisms in the papers Dou et al. (2020); Gillette et al. (2020).
Environment: the data files consist of beach water quality dataset from Massachusetts Environ- ment Public Health Tracking (EPHT) Massachusetts Department of Public Health (2025b). the Massachusetts Bay beach dataset from Massachusetts Water Resources Authority (MWRA) Mas sachusetts Water Resources Authority (20256), and the rainfall dataset from NOAA National Weather Service National Weather Service (2025), from 2002 to 2025. The data has been used in yearly reports Massachusetts Department of Public Health (2025a): Massachusetts Water Resources Authority (2025a) (2055a) to uncover trends in beach water pollution and the correlation between rainfall and water quality.
Legal: the datasets consists of 136 data files, accessible through the Federal Trade Commission (FTC) portal Federal Trade Commission (2025b) and Wikipedia Wikipedia contributors (2025). including information on merger filings, civil penalty actions, etc. The data is used in visual asualizations and dashboards that analyze nation-level debt collection and fraud detection, available at Federal
Trade Commission (202cc a).
Wildfire: the datasets consists of NOAA wildfire dataset National Centers for Environmental
Information (NCEI) (205),, National Interagency Fire Center (NIFC) Fire Information National
Interagency Fire Center (2025), ) US Environmental Protection Agency (EPA) Air Quality Annual
Data U.S. Environmental Protection Agency| (2025), US Election 2020 Dataset Fontes (2020).
Zillow Home Value Index Dataset | [Robikscuhe (2021). J. US Census 2.20 Census Bureau (2025
and the Large wildfire Incident Status Summary Young et al. [02I] to understand wildfire incident
location, cause, and consequences in the US from 2002 to 2016. This data has been used for analysis in the reports published by the NOAA and NIF.//CCE..onntoriingg Info @ noaa.gov (2025)
Center.
D.1 CROSS-DOMAIN ACCURACY DIFFERENCE ANALYSIS
In this subsection, we discuss the findings on likely causes of the differences in accuracy between different domains in KRAMABENCH. We obtained these findings by manually analyzing the traces of smolagennt--reflexionn DR.
1. Archeology (33.33%) In this domain, the system correctly solves questions answerable from single table. However, errors occur for tasks requiring joiniig tables found in different files, because it treats multiple files raw text instead of loading them as tables.
2. Astronomy (16.67%) Astronomy tasks have the lowest average performance. In this domain, a large portion of the required input data is found in proprietary scientifie formats (e.g., FORtRAN-stlle dat files). We observed that the agent struggles whenever it needed to load data from these files, e.g., SP3 orbit files or satellite products.
3. Biomedical (44.44%): When working with biomedical data, the agent is reliable for shallow operations on a single sheet but fails navigate large, multi-sheet workbooks and join data across sheets. Cross- joins, especially between clinical and phosphoproteomics data, are problematic, and errors arise in correlation statisties due to sign miscalculations.
4. Environment (60.00%) In the environmental domain, the system performs well on the relatively tasks involving clean CSV data, such as filtering counting, and averaging. Unlike other tasks which struggle from data retrieval or understanding issues, the main issues arise from implemmenattion/aritmee haritiion//iitmmtiic mistakes, such as incorrect aggregation scopes or rounding errors, which explains the higher score.
25

---

Published as a conference paper at ICLR 2026
Table 16: Detailed breakdown of per-domain tasks in KRAMABENCH. Reproduced from Table
Domain
# tasks
# subtasks
% Hard Tasks
# datasets
# sources
File size
Archeology
12
71
50.00%
5 1556
7.5MB
Astronomy
12
68
50.00%
486MB
Biomedical
9
38
66.66%
7
175MB
Environment
20
148
70.00%
37
31MB
Legal
30
188
53.33%
136
1.3MB
Wildfire
21
120
71.42%
23
1GB
Total
104
633
60.58%
1764
1.7GB
5. Legal (63.33%) In these tasks, the agent handles the straightforward pipelines well but struggles with loading data from messy files. ie.. that contain multi-row headers, partial subtotals, and metadata rows. Amongst the common emrors that stem from these shortcom- ings, one example is that sum, means, and aggregations are only partial due incorrect loading.
6. Wildfire (52.38%) Within wildfire-related tasks, the system faces challenges with geospatial data and temporal/statistica reasoning. It struggles with GeoPackage layers and spatial joins, as well as with colling-window aggregations for weather. However, lookups and simple value comparisons work relatively well.
TASK DETAILS
Across the 6 workloads, supply 104 end-to-end data science pipelines. The table for the overall breakdown of the tasks over the workloads is reproduced at Table 16 for convenience. In this section, we use an example from the archeology workload to explain the organization of tasks.
Each workload is associated with data lake consisting of tabular data and unstructured textual data.
archeology/input//
climateMeasurements xlsx
conflict conflict_brecke.css aaflocrroorddrrlaas_eeggooo xxsxx roman_cities.. cav worldcities.cst
Before tasks in workload are sent to the system under test, the system receives the directory where the data lake resides and may index offline. When tasks are prompted, the system should not receive information which files in the data lake the task pertains to. Each end-to-end task is specified with a high-level natural language prompt. Consider the following example of end-to-end task from the archeology domain:
What is the average Potassium in ppm from the first and last time the study recorded people in the Maltese area? Assume that Potassium is linearly interpolated between samples. Round your answer to 4 decimal places.
For evaluating the performance of our systems, we use three artifacts:
1. The end-to-end ground truth answer used to calculate the overall end-to-end score.
2. A sequence of key functionalities, extracted from a manually verified reference implementa- tion for the solution in Python.
A sequence subtasks, natural language questions whose correct answer depends on correct code implementation of key functionality.
26

---

Published as a conference paper at ICLR 2026
The key functionalities manually refined to correspond to the functionalities that should exist in any pipeline that produces the correct output. The sequence of key functionalities for the example end-to-end task above is the following:
1. Load the halllcaarbbo_dttbbueeeeeeeeeee climateMeasureeeete xlsx and read the first worksheet of each.
xlsx and
2. Remove rows or columns that are entirely NaN or not contain relevant information from both dataframes to ensure clean numeric processing.
3. Convert both chronologies calendar years: for the radio-carbon table get the year as 1950 minus the "date'
4. Convert both chronologies to calendar years: for the climate table get the year as 1950 minus the rounded 'Age_ky.1'' (in thousands of years) multiplied by 1000.
5. Determine the span of human presence in the Maltese area by taking the minimum and maximum 'year' in the radio-carbon dataframe.
6. For every integer year within the human presence span, locate the closest earlier and later rows in the climate dataframe and linearly interpolate (or directly return) the Potassium value 'K' and collect all these values.
7. Compute the mean of the collected Potassium values.
For each key functionality, supply subtask associated with the key functionality. Each subtask annotated with the ground truth subtask answer. These subtasks are used verify the code implementation capabilities of systems under test. Note that among correct pipeline implementations for the end-to-end task, key functionalitics may be ordered or composed differently. The subtasks associated to the end-to-end example task are:
1. Which files contain information about Potassium in ppm and the maltese people?
2. What are the indices (0-indexed) in rows in the climate measurement dataframe that must be cleaned?
3. What the calendar years in the radiocarbon table?
4. What are the calendar years in the climate table?
5. What are the minimum and maximum years of radiocarbon dating for the Malta region?
6. What the Potassium values for each integer year between -7580 and -4050 (included)? ? If the value is not available, use interpolatation between the closest earlier and later values.
7. What is the mean potassium value for the years between -4462 and -4055? Use decimal places.
F HUMAN BASELINE DETAILS
In this section, we summarize how we conducted the human baseline and discuss the results and implications.
27

---

Published as a conference paper at ICLR 2026
To contextualize LLM performance KRAMABENCH, conducted human data science study involving nine participants. Each participant was assigned subset of benchmark tasks and asked to solve them under the same data directory structure, resource constraints, and assumptions provided to our LLM agents. For every assigned task, participants produced:
complete, reproducible end-to-end solution in Jupyter notebook,
a detailed log of their active time, broken down into data exploration, pipeline design, coding, and debugging, and
both draft-stage notes and final clean solution, enabling direct comparison to LLM work- flows and error modes.
Participants followed standardized instructions covering repository setup, task management, time tracking, and reproducibility requirements. Across all human-authore authored solutions, we performed detailed analysis of the underlying failure causes. The distribution of error types is:
Incorrect pipeline design (46%): The largest category. These errors occur when, for example, experts mis-specified join, aggregation rule, grouping key, or filtering logic. This suggests that the most cognitively demanding part of real-world data science is pipeline design, rather than implementation.
Lack of domain knowledge (24%): Many tasks contain implicit domain assumptions (c.g.. definitions of "violation," mapping categorical labels). Experts often produced internally consistent but mismatched interpretations. This shows that even humans struggle with domain-specific task semantics.
Incorrect inputs (12%): Tasks often require gathering information across multiple similarly named or structurally similar files, and even humans sometimes use wrong inputs. These errors reflect the challenge of navigating multi-file filli--flc datasets.
Incorrect answer format (9%): Some errors are due to having the final outputs in the wrong representation (e.g., units, rounding. formatting), which did not match the one requested by the task.
Library/version issues (9%): Minor inconsistencies (e.g., pandas handling) that changed intermediate results enough fail strict correctness checking.
Interpretation and implications. Multi-file, multi-step pipelines are inherently error-prone even for trained experts. Humans have difficulty navigating vast data lake, which we see as an opportunity for L..M-poweeed systems to quickly search through the lake and identify the target files. Having reliable retriever could greatly improve accuracy. Ambiguity and assumed domain knowledge are real factor in real-world data tasks. One possible way for future agentic data-science systems to combat this issue is to ask clarification questions and invite user input. Another approach is to branch on possible solutions by clearly stating the assumptions. Pipeline design is the bottleneck. Nearly half of all errors (45.45%) are due to incorrect pipeline logic, highlighting that the core challenge understanding what transformations perform, not coding them. Overall, most human errors stemmed from misinterpreting ambiguous tasks, selecting the wrong files, or designing incorrect pipelines- challenges that mirror the dominant failure modes of LLM agents. This confirms that KRAMABENCH captures genuinely difficult, real-world data-to data-to-insight tasks where even trained data scientists struggle with pipeline reasoning, multi-fi file navigation, and implicit domain assumptions.
G EVALUATION DETAILS
Considering the broad nature of data science tasks, and the challenges in correctly evaluating their design and implementation, CRAMABENCH evaluates systems on three capabilities. From the most to the least automated: (1) End-to-end automation (2) Pipeline design (3) Sub-task implementation.
We are primarily interested in systems that can solve end-to-end data science tasks fully correctly, which drives main evaluation metric to be the result from the end-to-end automation setting.
28

---

Published as a conference paper at ICLR 2026
Table 17: Answer type and example questions
Type
Example
Metric
Scoring
String (exact)
name a
Accuracy
0/1
String
The month when experiment started.
PuraPluie paraphrase de- tection (emmesle
0/1
(approximate)
ul.
2025
Numeric
Counting the number entries
Accuracy
0/1
(exact)
satisfying predicate.
Numeric
Prediction of future experiment
Relative Absolute Error 1/(1 RAE) (RAF) l sl/brl
(upproximate)
ohservation.
List (exact)
Names of columns to filter data.
(exact match)
score
List
Regression coefficients for dif- ferent variables.
score (approximate score match 0.9)
(approximate)
G.1
MAIN METRIC: END-TO-END AUTOMATION SETTING
Each task in KRAMABENCH has a manually validated target output and is scored from [0.1]. Since pipelines might be composed of steps with varying nature, we identify six possible answer types for the target output. summarized and discussed in Table 3 For each answer type, we choose scoring scheme normalized to the range [0, I,, also shown in Table 3 When tested, the total score of system F for a workload W is defined solely based on the end end-to-end correctness as
Σrew score(F((T) [
Each T' is a task belonging to workload W, and [W] is the number of tasks in workload W. The overall score for the entire benchmark suite is defined analogously.
G.2 LLM-AS-A-JUDGE VALIDATION
To assess the validity of the evaluation for String (approximate) and List (approxi matee with String (approx (nproximmte)) mate) list members conducted via instruction tun- ing an I.M,, we performed a small scale human-L..M evaluator agreement study. We asked three human reviewers manually evaluate the equivalence between the reference solutions and the answers generated by 12 different SUTs (the three variants of DS-Guru across four different LLM backends). We run the LLM-as-a-juke a-jjgge evaluation pipeline three times. We report the Cohen's 's Kappa values for inter-human agreement, human-LLan agreement and inter-LLM calibration (Table 18). The possible values range from -1 (complete misalignment) to 1 (complete alignment). The results show very high inter-human agreement (~~ 95% on average) and moderately high human-LLM agreement (~~ 84% on average). indicating that our usage of L..M--s-a-judge provides meaningful evaluation results.
Table 18: Inter-Human, inter-LLM, and human- LLM agreement on approximate answer evaluation.
Inter-Human Agreement
Inter-LLM Agreement
Human-LLMM Agreement
Rater
Rater 2
K
Rater
Rater
R Rater 1
Rater
Human_ 0
Human_ 1
0.949
LLM judge_0 0
LLM judge_ I
Human 0
LLM judge_0 0
0.870
Human 0
Human_2 2
0.950
LLM judge_0
LLM judge_2
Human I
LLM judge_0
0.867
Human_1
Human_? 2
0.949
LLM judge_1
L.M judge_ 2
Human_2 2
L.M judge_0 0
0.818
G.3 ADDITIONAL EVALUATION SETTINGS
A system that cannot provide fully correct end-to-end results may still be helpful for end-users via assisting them the process of data pipeline design and implementation. Motivated by the goal of assessing this type of helpfulness of systems, conduct evaluations under two less-automated settings. In Section 4 detailing our experiments, we report these results as micro- benchmarks in Table 8
29

---

Published as a conference paper at ICLR 2026
Pipeline Design: This setting evaluates how many essential functions system- system-generate pipeline includes. Here, we ask the system to provide an end-to-end pipeline implemented in Python that solves end-to-end task. For evaluation,
manually curated an explicit list of key functionalities that any correct solution must implement for each task. We evaluate whether the generated pipeline code covers each functionality using the I.M evaluation method proposed in TTng & Zhang (2024).
The score for a single task is computed as
EjEKF(Ty Judge( P) KF(T))
Here, (T) denotes the set of human-annotated key functionalities for task T, [T) is the number of those functionalities, f represents a single functionalityy. P is the pipeline the system generated under test, and Judge a binary decision from LLM-based evaluator indicating wether P contains the key functionality f. The overall score across a workload/the entire benchmark is the average of the individual task scores.
This setting evaluates the system' 's ability to correctly implement simpler, lower-level functionalities and individual data tasks required to solve the entire challenge when explicitly prompted. We provide the system with problem statements of sub-tasks generate in Step 4 of the benchmark curation. Each sub-task corresponds to key functionality and represents an intermediate step within the full end-to-end pipeline, operating over the gold subset of the data lake. assess sub-task performance by comparing the system's 's intermediate outputs to human-annotated references, using evaluation approach similar to the end-to-end automated method described earlier in this section.
Sub-task Implementation:
H SUMMARY OF LLM USAGE
In this section, we summarize our usage of LLMs in compliance with the conference policy. We used L.MMs for the following purposes
1. LLMs were used for the semi-automated generation of fine-grained annotations for the benchmark. However, contributors manually improved and verified all annotations. This described in detail in Subsection 2.1
2. LLMs are an integral part of the systems we evaluated. Their roles in the systems are described in detail in Subsection 2.3 SSction 3
3. LLLM-as-a-jude were used to evaluade string paraphrases and code coverage. This is described in detail inpppendix G
4. LLMs were used to generate better documentations in our repository.
In addition to these rescarch-level involvement of LLMs, we also used LLMs for table formatting and paraphrasing some sentences already written by authors in favor of brevity.
30


Reviews:
[{"review_id": "25oxJRgnyq", "reviewer": "Reviewer_9FHL", "summary": "### Summary\nThis paper proposes KRAMABENCH, a comprehensive benchmark for evaluating AI systems’ end-to-end capabilities on data-intensive tasks. Targeting the gap of existing benchmarks that focus on isolated steps rather than full data science pipelines, KRAMABENCH comprises 104 manually curated tasks across 6 domains (archaeology, astronomy, biomedical research, etc.), covering 1700 files from 24 sources. The benchmark designs three evaluation settings (end-to-end automation, pipeline design, sub-task implementation) and evaluates 8 LLMs alongside multiple agentic systems (e.g., smolagents DR, OpenAI DR). Key findings include: current systems struggle with end-to-end pipeline generation (top accuracy 50%), agentic control flows significantly outperform structured loops, and fine-grained data-dependent reasoning and holistic data lake understanding are major bottlenecks. The paper also provides detailed ablation studies, failure analyses, and full reproducible artifacts, offering valuable insights for future research on automated data science systems.\n\n### Strengths\n1. **Gap-Filling Innovation**: Addresses a critical limitation of existing benchmarks by focusing on end-to-end data science pipelines (from data discovery to insight generation) rather than isolated tasks (e.g., code generation, text-to-SQL), aligning with real-world data science workflows.\n2. **Comprehensive Design**: Covers diverse domains, heterogeneous data types (structured/semi-structured/unstructured), and multi-faceted evaluation dimensions, ensuring the benchmark’s generality and robustness.\n3. **Rigorous Experimentation**: Evaluates a wide range of LLMs and agentic systems, conducts in-depth ablation studies (retrieval mechanisms, data leakage, iteration counts) and failure analyses, providing conclusive evidence for key claims.\n4. **High Reproducibility**: Publicly releases all code, data, workloads, and evaluation scripts, adhering to top conference standards for reproducibility and facilitating follow-up research.\n\n\n\n\n\n### Weaknesses\n1. **Limited Generalizability of Data Leakage Evaluation**: The data leakage assessment only covers 20% of tasks (focused on legal and wildfire domains) with synthetic identifiers/numerics. This narrow scope may not fully reflect how systems rely on memorized data across different domains or task types.\n\n2. **Insufficient Analysis of Domain-Specific Performance Gaps**: The paper reports significant performance variations across domains (e.g., wildfire: ~50% accuracy vs. astronomy: ~16% for smolagents DR), but lacks in-depth analysis of why certain domains are more challenging (e.g., data complexity, domain knowledge requirements).\n\n3. **Lack of Multi-Agent Evaluation**: Current evaluations focus on single-agent systems, but multi-agent collaboration is a promising direction for complex data-intensive tasks—omitting this dimension limits the benchmark’s coverage of state-of-the-art approaches.\n\n\n### Questions\n1. Given that the data leakage evaluation only covers 20% of tasks and is concentrated in the legal and wildfire domains, how do you ensure that the results obtained using synthetic data can generalize to other domains and task types to accurately reflect the characteristics of systems' dependence on memorized data?(Related to Weakness 1)\n\n2. Regarding the significant performance gap of smolagents DR between the wildfire domain (~50% accuracy) and the astronomy domain (~16% accuracy), do you plan to supplement in-depth analysis from dimensions such as data complexity and domain knowledge requirements to explain the root causes of difficulty in different domains?(Related to Weakness 2)\n\n3. Multi-agent collaboration has become a promising cutting-edge direction for solving complex data-intensive tasks. Why did you not incorporate this dimension in the evaluation, and does this benchmark have plans to supplement multi-agent system testing in the future to cover the latest technical approaches?(Related to Weakness 3)", "questions": "See Summary", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "See Summary", "weaknesses": "See Summary", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xe8Xe0lY6d", "reviewer": "Reviewer_fJHC", "summary": "This paper introduce KramaBench, a benchmark to evaluate LLM-based systems on data science tasks. KramaBench includes 104 mannually curated tasks from 1700 real-world files across 24 sources in 6 domains (archaeology, astronomy, biomedical research, environmental science, legal discovery, and wildfire prevention). The authors then provide a comprehensive evaluation framwork with several settings (1) end-to-end automation, (2) pipeline design, and (3) individual task implementation. With experiments on 8 LLMs using DS-Guru reference framework and several agentic systems, the authors find that current systems struggle with end-to-end pipeline generation, even with perfect retrieval.", "questions": "- What is the conceptual or methodological novelty beyond creating a benchmark with more realistic data science tasks?\n- Can you provide validation of the LLM judge against human evaluators, including agreement rates and calibration analysis?\n- Can you provide detailed computational cost analysis (tokens, time, dollars) for running each system on the full benchmark?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- **Comprehensive Evaluation**: The multi-level evaluation framework (end-to-end, pipeline design, sub-task implementation) provides valuable insights into where systems fail.\n- **Rigorous Curation Process**: The 4-step validation process involving multiple contributors, cross-validation, and manual verification of reference solutions demonstrates strong quality control. Grounding tasks in published studies ensures real-world relevance and avoids artificial task design.\n- **Diverse Tasks**: The benchmark spans multiple domains with varying data formats, file counts, and difficulty levels, providing good coverage of real-world scenarios.", "weaknesses": "- **Unclear Motivation and Weak Problem Positioning**: The paper lacks a compelling motivation section explaining why this benchmark is needed now and what specific real-world problems it addresses that existing benchmarks cannot. The introduction jumps directly into the solution without establishing the problem's urgency or providing concrete use cases where current benchmarks fail.\n- **Poor Figure Quality**: Figure 1 is a low-resolution raster image with blurry, barely readable text, which is unacceptable for a top-tier venue. The figure should be vector graphics (SVG/PDF) with crisp, readable text. This significantly impacts the paper's professionalism and readability.\n- **Severely Insufficient Visual Explanations**: The paper has only 2 figures across 9 pages of main content, making it difficult to understand key components. Critical missing visualizations include (1) DS-Guru architecture diagram showing the three variants and data flow, (2) benchmark curation pipeline flowchart for the 4-step process, (3) task structure schema showing relationships between end-to-end tasks, key functionalities, and sub-tasks, (4) evaluation framework diagram.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A5L6lZ8cw5", "reviewer": "Reviewer_WbPU", "summary": "1. This paper introduces KRAMABENCH, a benchmark designed to evaluate AI systems’ ability to solve end-to-end data science pipelines on real-world data lakes (heterogeneous, unclean, multi-source data). It includes 104 manually curated tasks across 6 domains (e.g., archaeology, biomedical research), 1700 files, and 24 data sources, paired with expert reference solutions and a 3-tier evaluation framework (end-to-end automation, pipeline design, sub-task implementation). The authors also propose DS-Guru, a lightweight single-agent framework for baseline testing, and evaluate 8 LLMs + 4 agentic systems (e.g., smolagents DR, OpenAI Deep Research).", "questions": "Beyond Weaknesses. The following is the suggestions for the authors.\n1. Incorporate Human-in-the-Loop Evaluation: Since real-world data science often involves human-AI collaboration, add a fourth evaluation tier (e.g., human oversight of pipeline revisions) to measure how systems assist human experts. This would increase the benchmark’s relevance for industry use cases.\n2. Scalability and Cost Analysis: Extend experiments to larger data lakes (e.g., 10k+ files) to test scalability of retrieval mechanisms (OPS vs. agentic retrieval). Add cost-per-accuracy metrics (e.g., dollars/task, energy consumption) to guide practical adoption.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Fills a critical gap in existing benchmarks (e.g., DS-1000, ARCADE) by focusing on end-to-end data lake processing rather than isolated tasks (code generation, text-to-SQL). Unlike prior work, it emphasizes real-world complexity (noisy data, multi-file integration, domain-specific knowledge) and requires systems to orchestrate all pipeline stages (discovery, cleaning, analysis).\n2. Rigorous task curation via a 4-step validation process (curation → cross-contributor verification → key functionality identification → sub-task curation) ensures high-quality, reproducible reference solutions.\n3. Extensive experiments across 8 LLMs, 4 agentic systems, and 3 input modes (Full, Oracle, Trimmed) provide statistically meaningful results. Ablation studies (e.g., retrieval mechanism, sample size, iterations) further validate conclusions about performance drivers.", "weaknesses": "1. The paper focuses on single-agent systems (DS-Guru, smolagents DR) but barely explores multi-agent architectures, which are increasingly proposed for complex data tasks (e.g., dividing pipeline stages across specialized agents). This is a missed opportunity, as multi-agent systems may mitigate single-agent limitations (e.g., context window constraints, heterogeneous skill requirements).\n2. The results show large performance gaps across domains (e.g., smolagents DR achieves 60% accuracy in Environment but 16.67% in Astronomy), but the paper provides minimal analysis of why (e.g., is Astronomy’s larger file size, more complex data formats, or domain unfamiliarity the cause?). Without this, readers cannot generalize findings to other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KkQoxtmnIY", "reviewer": "Reviewer_4eDt", "summary": "This paper introduces KRAMABENCH, a large-scale benchmark designed to evaluate AI systems on end-to-end data-intensive reasoning—specifically, the ability to design and execute full data-science pipelines across heterogeneous, noisy data lakes.\nThe benchmark comprises 104 curated tasks drawn from 1,700 files and six domains (archaeology, astronomy, biomedical, environmental, legal, and wildfire).\nEach task is derived from real-world studies and paired with reference solutions, sub-task annotations, and evaluation scripts.\n\nThe authors propose three complementary evaluation settings:\n\nEnd-to-end automation – overall task correctness;\n\nPipeline design – coverage of key functional components;\n\nSub-task implementation – correctness on atomic operations.\n\nA minimal baseline framework, DS-Guru, enables single-agent LLMs (e.g., GPT-4o, Claude-3.5, DeepSeek-R1, Llama3.3) to attempt these challenges.\nThe benchmark also evaluates agentic systems such as Hugging Face Smolagents, OpenAI Deep Research, and Gemini Agentic Mode.", "questions": "1-How does KRAMABENCH handle tasks with stochastic or non-deterministic outputs (e.g., probabilistic models)?\n\n2- Could future iterations support incremental learning or multi-agent collaboration settings?\n\n3- Did the authors observe any correlations between model size and success on specific domains (e.g., biomedical vs. legal)?\n\n4- Are the obscured-input experiments sufficient to rule out partial memorization—could LLMs have seen similar structure from public repositories?\n\n5- Would integrating structured metadata (schemas, ontology hints) into DS-Guru improve pipeline design?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality: First benchmark targeting end-to-end data-to-insight reasoning across heterogeneous data lakes.\n\nQuality: Comprehensive curation and evaluation pipeline; transparent methodology with open-source assets.\n\nClarity: Well-written with detailed ablations, case studies, and ethical considerations.\n\nSignificance: Reveals key weaknesses of state-of-the-art LLM agents (e.g., data-dependent reasoning), guiding future research in automated data science.\n\nBreadth: Covers six domains and multiple data modalities (structured, semi-structured, unstructured).\n\nPractical utility: Enables reproducible assessment of both academic and commercial agentic frameworks.", "weaknesses": "Limited evaluation diversity: While the benchmark spans multiple domains, all experiments are English-only and rely on a single execution environment; multilingual or multimodal tasks would strengthen generality.\n\nAbsence of multi-agent baselines: The paper focuses on single-agent and light agentic systems; evaluating collaborative agents or planner–executor architectures would provide additional insight.\n\nComputation cost reporting: Token-level cost is discussed but system runtime and hardware variance could be presented more rigorously.\n\nLimited human comparison: Including a human-in-the-loop baseline would contextualize model performance relative to expert data scientists.\n\nStatistical significance: Some percentage improvements (e.g., +1.3 %) could benefit from confidence intervals or variance analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)