DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
Authors: Fangyu Lei
Jinxiang Meng
Yiming Huang
Junjie zhao
Yitong Zhang
Jianwen Luo
Xin Zou
Ruiyi Yang
Wenbo Shi
Yan Gao
Shizhu He
Jun Zhao
Zuo Wang
Qian Liu
Yang Wang
WANG KE
Kang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Data Agent
Code Generation
LLM Benchmark
Primary area: datasets and benchmarks
TL;DR: We introduce DAComp, the first benchmark for the full data intelligence lifecycle.

Abstract: Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. 
We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows.
Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. 
Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations.
Engineering tasks are scored through execution-based, multi-metric evaluation.
Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics.
Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20\%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40\%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities.
By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings.
Our data and code are available at \url{da-comp.github.io}.

Published as a conference paper at ICLR 2026
DACOMP: BENCHMARKING DATA AGENTS ACROSS THE FULL DATA INTELLIGENCE LIFECYCLE
Fangyu Lei 1,2.3 Jinxiang Meng* Yiming Huang Junjie Zhao Yitong Zhang
Jianwen Luo Xin Zou" Ruiyi Yang" Wenbo Shi Yan Gao Shizhu He? Zuo Wang?
Qian Liu' Yang Wang Ke Wang*: Jun Zhao?-2 Kang Liu
'Th Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
2 "chooo of Artificial Intelligence, University of Chinese Academy of Sciences
BytteDance Seed TTiTTok UCC San Diego "NUS
ABSTRACT
Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-orient insights. We introduce DAComp. a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing sys- tems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning. exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of action- able recommendations. Engineering tasks are scored through irccution-baseed multi-metric evaluation. Open-ended tasks are assessed by reliable, experimen- tally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DA- Comp. Performance DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores DA tasks also average below 40%, highlighting pro- found deficiencies in open-ended reasoning and demonstrating that engineering and analysis distinct capabilities. By clearly diagnosing these limitations, DA- Comp provides rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at da-comp. github.
INTRODUCTION
Data intelligence, the process of transforming and fragmented data into actionable insights, has become a cornerstone of modern enterprises. The remarkable reasoning and code generation capa-
bilities of Large Language Models (LLMMs) (OpenAl 2025 Anthropic 2025 Gemini 2025) have
opened new avenues for automating data intelligence tasks. LLM-based agents have demonstrated
considerable promise across wide range of applications, including text-to-SQL (Yu et al. 2018
Li et al. 2024b; Lei et al. 2024). software engineering (Jimenez et al.] 2023 Chan et al. 2024), and general computer control (Zhou et al.) (224,) Xie et al., [0224] Wei et al. (025). However, the advancement of these agents into enterprise data intelligence remains constrained by the absence of benchmarks that faithfully reflect real-world complexity.
This gap between existing benchmarks and real enterprise practice calls for benchmark that eval- uates agents along two distinct axes: Hard (engincering realism) and Soft (analytical openness). The Hard axis reflects the capacity for systematic large-scale code implementation. similar to the responsibilities of data engineers. For example, this means not only generating a single SQL query but also orchestrating and evolving complex data workflows under changing requirements. The
"quual contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2026
DE: Implementation
4
DA: Insight Generation
Please help to idintify "zombie" opportunities in Salesforce via engagement analyais.
Write SQLs to build DE-DAG
Write Python/ SQL to generate open-ended analysis
Pathos/ SOL Thinking
Raw Data Sources
DE: Architecture
Semantic-Layer
Write design document
DE-DA0
Design Specifications
Codebase DE-DAG
Open-Ended Analysis
mame epportaants heath
rcb
Analytical Report
ddecr lpriion: Secoes axhopportutiy sheotth inidevity
Our (23 analysie shows 40% the
source Jubric.
comprised uunhealthy"
seportrrr aidiiidsaiiii wissswhttuue Lape in, intastvitiiopee per_ doal
D - pw op
SQLL
apportunity unlry neath cnoe
descciption: Priott rt fulcsfor Houlth Score 100) 1. Engagement ( actittic Sride feel
Kay Invight
wwinfliting foretand, which manks crtiical soodowninn soles velochty
3 Drark Daod to stage Strt -n 25
Salea Velocity & Funnel Conversion? ?
Visusization
mame surreryyy
K DE: Evolution Write SQLs fidluddate DE-DAG
intate
Figure I: DAComp aims to evaluate LLMs on full-lifecycle data intelligence workflows, encom- passing repository-leve data engineering (DE) and open-ended data analysis (DA).
Soft axis reflects the capacity for strategic reasoning. aligning more closely with the role of data analysts. For example, this involves facing an open-ended business question, planning multi-step analytical workflows, synthesizing insights across analytical results, generating visualizations and crafting decision-oriented reports. Most benchmarks fail to capture these two key dimensions. They reduce complex engineering to isolated code snippet generation, missing the Hard axis, and reduce open-ended analysis to deterministic answers, missing the Soft axis.
To fill this gap, we present DAComp, benchmarking agents on full lifecycle data intelligence tasks, as illustrated in Fig. DAComp-DE is the first to introduce repository-level data engineering tasks where agents must orchestrate multi-layered data workflows by generating a DAG on com- plex enterprise schemas. It includes three distinct task types: (1) DE-Architecture tasks focus on the high-level planning of detailed engineering specifications. (2) DE-Implemmnntatinn tasks require agents to build multi-stage data pipelines from scratch; (3) DE-Evolution tasks challenge them to modify existing systems in response to new requirements; and Both DE-Imp) and DE-Evol tasks are demanding, often requiring large-scale code changes that involve over 4, 000 lines of code across more than 30 files, mirroring real-world engineering workloads. DAComp-DA is the first to pioneer real-world, open-ended data analysis. In these scenarios, agents are presented with complex ques- tions over downstream analytical data. Unlike prior work with deterministic answers (Jing et al.) 2024 Lei et al., 2024). the tasks resemble real analyst settings: agents must write SQL/Python to aggregate, compute, and analyze intermediate results in order to generate insights, reports and vi- sualizations, thereby emphasizing both the rigor of analytical precision and the practical utility for human decision-making To facilitate broad applicability, we also release DAComp-zh, high- quality Chinese adaptation of the benchmark, along with baseline results.
The evaluation methods of such complex tasks are non-trivial. For deterministic DE-Impl and DE- Evol tasks, we adopt an execution eration performance. The open-ended DA and DE-Arch tasks are assessed by an LLM judge (Li eet al. (0244)), whose evaluation is guided by our novel rubric framework. Instead of relying on a single answer key, this framework explicitly defines and assesses multiple valid solution paths for each open-ended problem, enabling robust, multifaceted assessment that rewards diverse analyt- ical strategies. The reliability of this LLM judge has been confirmed through rigorous validation experiments, which show strong agreement with human experts.
method to systematically evaluate the repo-level code gen-
Our experiments on DAComp underscore a significant challenge for current models: even state-of- the-art agents falter when confronted with its enterprise-level complexity. In DE tasks, agent capa- bilities are pushed to their limits, with average scores below 40% and strict success rates under 10%, revealing critical gap in real repository-level engineering capabilities. In the same vein, agents also exhibit poor performance on open-ended problems requiring autonomous planning. Performance on DA tasks plummets below 50% for most models, with only few proprietary systems demon- strating more robust analytical skills. Ultimately, progress in data agents demands a shift from mere code accuracy the nuanced capabilities papabiliiiss-llaningg, open-ended reasoning, and systematic synthe-
2

---

Published as a conference paper at ICLR 2026
sis sis-required to deliver insights that are both analytically rigorous and strategically actionable. By providing this rigorous, realistic testbed, DAComp aims to shift the focus of data agent development from isolated skills to the integrated, full- full-lifecycle capabilities required in the real-world scenarios.
2 BENCHMARK CONSTRUCTION
2.1 TASK DEFINITION
To bridge this gap, design tasks that evaluate data agents real-workd challenges. Specifically, we assess their ability to act as data engineers performing repository-level data engineering and as data analysts navigating open-ended data analysis, as depicted in Fig.
DAComp-DE. An agent de tasked with handling the full DE lifecycle including architecture, implementation, and evolution. Formally, the process is modeled as (S, ""[QQee Co B),, where Qae is the initial high-level requirement, denotes the engineering specification (e.g.. a Data Contract), B is the database and C, is the final DE repository. This unified capability is evaluated across three task types: (1) DE-Arch: : Given a high-level requirement Qaee and an initial repository Co. this task evaluates the agent's ability to produce the engineering specification S. (2) DE-Impl Given a detailed specification S and an empty repository (Co = 2), this task evaluates the agent's 's ability to implement the DE repository C. from scratch. (3) DE-Evol Given an existing repository Co and new specification S, this task evaluates the agent's 's ability to update the repository into C..
DAComp-DA. Given an analysis-ready data D (semantic layer) and an open-ended question Qaa- an agent with policy de produces analysis artifacts O = x (Qaa; 'D (e.g., analytical reports, key insights and actionable recommendations). This task inherently open-ended, as single question may be approached through multiple valid analytical paths, without a fixed standard answer.
2.2 EVALUATION METRICS
LLM-judge with hierarchical rubrics and GSB scor- ing. The LLM judge evaluates outputs O along six dimensions: Completeness, Accuracy, Insightfulness Readability, Analytical depth and Visualization (sce App. 1. The hierarchical rubric assesses the first three, while the Good Same-Same--e Bad (GSB) score (Zheng et al. 2023) covers the latter three. Visualization specif- ically assesses the agent's ability
Analylical Question
Score
>
Dww Scom
Req
Req
Req
...
Tap- Top-Down Decompoaition
Req 3.1
Req 3.2
Boltom-Up Scoring
Path Selcction
Path Selection
translate numerical
results into intuitive chart. As shown in Fig. 2 the rubric (R) decomposes question Q into requirements and sub- reqirements. Each subrequirement admits multiple valid solution paths, each path carrying its own rubric items (colored leaf nodes). Human experts enumerate these paths and merge equivalent solutions a single path. For scoring, the LLM judge selects the best-matching path for each rubreqquirement, applies only that path's 's items, then aggregates scores bottom-up. This design accommodates diverse correct approaches without pe- nalizing method choice. We show a detailed rubric example for the penetration and profitability analysis in Tab. I1, with discussion of the path enumeration scheme provided in A. The rubric score is normalized, weighted sum of satisfied items: Score (O, (O,
Mas Score:
Max Score: 5 Path
Max Scorc:
Figure 2: Details of hierarchical rubrics.
- we A(ck. O) E [0, wy]. For the Good Brood-Same-Baae Bade (GSB), the LLM judge only compares the final an- alytic results against five pre-provided baseline reports, guided by the dedicated rubrics for these axes, yielding the score: Scoregab(O 0bOasse) = max | The final score for DA task is a weighted combination of these two components: Scores, a Score + (1 (1-a)) a) Scoregsb The open-ended DE-Arch tasks are assessed similarly, though they employ a standard, non- non-hierarchical rubric and do not incorporate the GSB component. Further details are provided in App. A
-
Execution-based evaluation for deterministic tasks. DE-Impl and DE-Evol tasks are evaluated with three execution- based metries of increasing strictness: (1) the partial credit Component Score
3

---

Published as a conference paper at ICLR 2026
(CS), CSpe-imp//vol = E WySj, which evaluates each node in isolation (using gold-standard up- stream inputs) to measure total component-level SQL generation; (2) the Cascading Failure Score (CFS), which evaluates nodes sequentially along the DAG and nullifies a node's s score any up- stream dependency is incorrect, thus measuring end-to-end data integrity; and (3) the strict Success Rate (SR), tmpp-vvp/ I[v : 8, = 1], which requires every single component to be perfect. This suite of metrics erucial for diagnosing the primary bottleneck: the gap between agent's component-level generation and its ability to perform holistic pipeline orchestration. Further details provided in App. A.I
2.3 ANNOTATION PIPELINE
DAComp is constructed by experts through rigorous pipeline ensure realism, quality, and consistency. Further details and examples are provided in App.
1) Data collection. The benchmark is grounded in permissively licensed assets (e.g., Apache- 2.0, MIT). For the DE task, we collect 73 enterprise-scale SaaS schemas with data transformation projects, averaging 400 columns each, and populate them with large-scale, relationally consistent synthetic data (see App. E. For the DA task, we curate 100 complex databases from the Web and supplement them with analytical modeling layers derived from DE-trrasformed data.
2) Task design. At this stage, we generate the DAComp questions. For DA annotators first draft open-ended analytical questions per analysis-ready table. Five annotators then vote based on realism and difficulty, and the top 2 are retained. For DE-Evol, practicing data engineers author new business requirements aligned with enterprise scenarios and professional standards. For DE-Impl, reverse engineer selected SaaS transformation projects into single data.contract . yaml, capturing the full DAG and semantics. For DE-Arch, starting from the analytics layer of DE-Impl and DE-Evol examples, DA annotators propose 5 candidate business requirements per project, from which a data engineer selects feasible yet challenging requirement.
3) Evaluation construction. We design evaluation protocols for each task. For DA, annotators build hierarchical rubrics as described in $2.2, with at least annotators annotate each question, followed by alignment discussion to resolve discrepancies. For the GSB protocol, experienced data analysts author shared scoring criteria, and baseline reports are created by combining outputs from multiple LLMs. A critical aspect of this rubric design the enumeration of valid solution Paths, a pro- cess governed by three key principles: (i) ensuring Paths represent distinct, methodollogiaally-souu strategies, not incremental steps: (ii) validating deterministic outputs against programmatically cal- culated and verifiable anchor values; and (iii) utilizing methodology-bart based soft constraints to fairly evaluate valid but unenumerated solution paths. (see examples in App[C.4] discussion in App/G.T G. To ensure the comprehensiveness of our rubric, we perform validation step: we sample outputs from five diverse L.LMs and confirm that our enumerated paths can account for all observed solu- tion strategies, which minimizes the risk of false negatives by ensuring that valid but unanticipated solutions are not unfairly penalized. For DE-Impl and DE-Evol. solutions are deterministic: we implement execution scripts automatically validate outputs against gold repositories, assigning partial credit at the node/layer level to capture step-wise correctness.
2.4 DATASET STATISTICS
We present a statistical analysis of DAComp. highlighting its main features in comparison with prior datasets in Tab. and providing more detailed characteristics in Tab. 2
Detection
7%
Validity
Integrity
DAComp-DE quantifies enterprise-scale engineering complexity. The statistics for DAComp-DE underscore its large scale and complex- ity-defined by repo-level paradigm, schemas averaging 412 columns, and solutions requiring over 2,000 lines of code-setting it apart from prior data agent benchmarks. Unlike benchmarks that focus on generating iso- lated scripts, DAComp introduces tasks industrial schemas with av- erage of 32 tables and 412 columns. The engineering effort required is substantial. Implementation tasks involve building entire pipelines from scratch, averaging 4,612 lines of code across 43 distinct files. Evolution tasks simulate realistic maintenance with edits averaging 1,718 LOC across
48 1%
Consistency
22.3%
Figure 3: Data clean- ing tasks DE-Impl staging layer.
4

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of DAComp with other agent benchmarks, highlighting key differences in task scope, task paradigm, and evalution method. DAComp-zh shares the identical task set.
Benchmark
Field
Tacks
Level
Cole Schemaa
Cade Scale (LOCC)
Primary Output
euded
Evaluationn Method
Agritic Renckmnarks
SWE-Bench Dinenez al. 2023
Software Engineering
2,294
N/A
32.8
Code Patch
Execution-hisee
WebArena de balee
Navigation
812
Actions
Execution-
OS World Miretei 20123
Computer Control
369
N/A NAA
NIA
Actions
Exccution based Objective
BrowserComp Wet cla 2025
Deep Research
2,000
NAA
Answer
Data
Benckmarks
DS-1000 al
Data Science
Script SQQ
Exceuiion-bsed
HIRD B NP241)
Text-to- SQI.
12,751
X
23.5
Execurion-hused
Spider 217 Srale 10124
Text-tn-SQU.
632
X
320
104.6
I SQL
Execarion-hised
BIRD-CRITIC eL mi 212
SQL Dehugging
1,100
X
54
50-70
I SQL
Execusion-hasee
Code [Haudz
Data Science
500
50 100
85
Script
Objective
DSBench ef
Data Science
10-20
Scripts
Objective
KrumaBench
Pi gl 2225
Data Science Pipelines
104
13
50-1100
Scripts
LLM-judge
BLADE dille 21744
Data Amilysis
250
X
10 10-112 12
70-80
Report
L.M-judge
DABStep
Data Anilysis
450
10-- 12
100
Anewer
Objective
DAComp
Data Engineering
Duta Anslysis
210
382
~ 2,000
N NSQUSScipt
Doc a Report
Both
Exceution-haved & LLM-juugeeruhrrzz)
Table 2: Key statistics for DAComp. All metrics are per-example averages, except #Total tasks.
Metric
Value
Metric
Value
Overall IDE-ArrbbDE--mp/DE--Boooll
DAComp-DE
#Total tasks
30 / 30/ 50 100
DE-Impl raw data (#Tab. / /Cool.)
23.3 / 3816
#Question Tokens
166 / 30,83 / 6,508 90
NI. code scale (Impl Evol)
2,296 949.6
DAComp-DA
#Change files (Impl/ Evol)
37.0/ 11.7
Columns/ Tahles
/..9
#Change columns (Impl / Evol)
1,239 530.9
LOC
433
#DE.Arch rubric
18.5
Rubrics (Reqs/ sub-reqs / Paths Items) Completeness/ Accuracy/ Insightfulness
/. 22.4
DE Impl layer (#Staging/ #Core/ #Mart)
16.0 / / 8.8 /.76
14% / 666% 20%
DE.F table change types (Nereate Medit)
13 files, agents need to manage data transformation across a multi-layered
data model (staging, core, and mart). The staging layer involves data cleaning operations, cen- tral topic in data governance, which we categorize into four types: validity constraints, consistency constraints, integrity & uniqueness, and anomaly detection (as shown in Fig. B). Intermediate and marts layers typically focus on complex business logic, entity integrations, and metric aggregations.
DAComp-DA measures analytical depth and methodological diversity. The design of DAComp-Da -A moves beyond simple question-ans answering to assess deep analytical reasoning. Uniquely, DAComp evaluates both deterministic engineering and open-ended analysis, a distinction from prior benchmarks that typically focus on only one paradigm. Its open-ended nature is quantified by our hierarchical rubrics, which decompose each of the 100 DA tasks into an average of 3.1 re- quirements and 5.7 sub-requirements, accommodating roughly 13 valid solution paths. This method- ological diversity is evaluated with a multi-faceted rubric where scoring items are weighted toward Accuracy (66%) but also reward Completeness (14%) and Insightfulness (20%). While the analyt- ical schemas are more focused than in DE tasks (averaging 4 tables and 85 columns), the required reasoning is still complex, reflected in an average solution length of 347 lines of code- significcay longer than typical text-to-SQL or single-script data science tasks. Crucially, DAComp-DA places a strong emphasis on open-ended data visualization, requiring agents to autonomously select and generate charts that effectively communicate their findings.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
We evaluate state-of-the-ard LLMs, including open-source models like Qwen3 (Yang et al., 2025). DeepSeck-V3.1 (Liu et al.] 2024). and Kimi-K2 (Team et al. 2025), as well as proprietary ones such as the Gemini (Team et al. 2023), and GPT (OpenAl 2023) families. We utilize the widely adopted OpenHands (CodeAct- Agent) framework (Wang al.) (2024) for both DE and DA tasks. Additionally, we developed a custom baseline named DA-Agent for DAComp- -DA, which operates via Bash and file system interactions and is capable of executing Python and SQL.. The performance
5

---

Published as a conference paper at ICLR 2026
Table DAComp-DD Baseline Performance. All models are evaluated using the DE-Agent frame- work (details in App. B.2) across both Implementation (CFS, Max-CFS@ CS, Max-CS6 () and Evolution (SR @, CFS, Max-CFS 8); see App. A.1 for metric definitions. The final column ports the aggregated DE Score.
Implementation
Evolution
Method
Architecture
DE
CFS
Max-CFS@8
CS
Max-CS@8
CFS
Max-CFS@8
SR@s
63.93
61.98
47.23
Gemini-233 2.5 Pro
51.96.117))
27.66
36.88
55.32
65.32
23.97
38.92
8,00
32.88
Qwen3-Cove
51.43
63.78
DeepSeck-V3. V3,1
52.66-2.85)
22.33
30.73
50.04
00.46
24.11
35.01
10.00
31.41
18.32
15.07
22.32
35.55
47.81
24.42
32.07
28.39
Dwen3-235B-A22B
50,73ss.00
2.43
5.77
20.15
31.03
12.43
21.89
2.00
20.15
Qwen3-8B
45.12
2.34
15.33
21.23
15.89
19.12
2.00
19.89
Table 4: DAComp-DE-zhi Baseline Performance.
Implementation
Evolution
Architecture
Score
CFS
Max-CFS58
CS
Max-CS@8
CFS
Max-CFS@8
SRØ8
GPT-5
63.60u2.10
30.49
39.24
61.85
68.43
37.88
46.91
20.00
42.83
Gemini-2.5-44 P.oPoo
51,90c2443
26.98
36,73
55,18
65.07
24.28
38.27
8.00
32.55
Qwen3. Coder
51.11
23.23
32,97
54,59
63.69
26.59
39.37
12,00
32.36
DeepSeck-V3.
53,08
22.62
30.84
50.22
60.34
24.69
35.17
8.00
31.87
03
48.0 48.02
15.00
22.15
35.10
47.45
24.23
32.59
6.00
28. 20
Dwen2-235B-A22R
e0.61e
2.31
5.83
20.03
31.27
13.01
21.27
0.00
20.35
Qwen3-8B
46.22
1.21
2.16
15.78
21.59
15.19
19.35
0.00
19.84
Table Detailed performance breakdown on the DAComp-DA benchmark.
Method
Completeness
Accuracy
Insightfalness
Readahility
Analytical Depth
Visualization
OpenHands Baseline
GPT-5
60.98
40.3
49.39
35.51
69.8
21.4
46.99
Gomini-2.5-Pro
45.02
30.22
40.71
48.2
31.0
15.0
33.38
40.13
25.5
20.45
26.22
27.11
26.57
DeepSeck-V3.1
49.88
33,25
41.66
36.0
33.2
11.0
33.87
Qwen3-Codee Codee
33.42
21.21
25.00 06
20.0
13.73
24.28
Qwen3-235B-A22B
30.7
12.23
22.11
3.6
1.8
0.8
12,43
DA-Agent Baveline
GPT-5
64.23
43.81
56.8 (6.89)
43.59
76.80.4.00
27.44
50.84.-3.15)
Kimi K2
52.31:a1
33.56.2200)
46.82)1144)
62.20...00)
63.75
14.40 .2.15
41.89
Geminn-2.5-PPo
41.43
30.30
41.45ia071)
51.60
35.75
13.40 (3.90)
34.70
DeepSeek V3.1
48.74:1200)
32.97
42.43x.1.90
37.25
.5.00.1159)
11.45
34.33
40.73
29.54
23.95
25.24:22551
23.81(23377)
7.32:s1.29)
28.20
Qwen3 Coder
35.12
20.05
25.53
19.37cu1449
13.42,22300
5.15
Doubao-Seed-14
37.45:a1955
18.45
27.511a1000
13.25
9.01
6.80
20.74
Qwen3 235B- A23B
29.37:11000
13.1
21.50..191)
3.64
1.56
1.87
13.25
Qwen3-8B
9.89:2246)
4.12
5.05
0.13
0.00
0.15
4.47
of each agent is measured using the metrics detailed in $2.2 We also report two aggregate scores: the DE Score, which and the Overall Score, representing the mean across the entire benchmark. For the DA score, = 0.6 to aggregate the rubric and GSB scores, with Gemini-2.5- Flash serving as the LLM judge. Further details the experimental setup and additional results are provided in App.
mean score across all DE tasks (using CFS for Impllementtiion//Eolluiin
3.2 MAIN RESULTS
DE results. As shown in Tab. GPT-5 establishes definitive lead, consistently achieving the highest aggregated DE Scores across different orchestration frameworks. Notably, specialized open- source models like Qwen3-Coder and DeepSeek- V demonstrate exceptional efficacy, effectively rivaling ceneral-purposs proprietary models such as Gemini-2.5-PP5 However, the absolute perfor- mance metrics reveal a sobering reality regarding the complexity repository-level engineering: even the state-of-the-art GPT-5 achieves modest DE Score of approximately 42.88% and strict Success Rate of merely 20.00%. This profound performance ceiling underscores that while frame- stabilize interaction. DAComp-DE poses rigorous challenge that current LLMs LLMs-reeardlee regardless of their scale or specialization have yet to master, highlighting a critical gap between isolated code generation and holistic system orchestration.
work optimizations

---

Published as conference paper at ICLR 2026
Table 6: Detailed performance breakdown on the DAComp-DA-zh (Chinese) benchmark.
Method
Completeness
Accuracy
Insightfulness
Readability
Analytical Depth
Visualization
OpenHands Baseline
GPT-5
70.56
47,08
57.19
19.6
46.4
22.0
43.69
Gemini-2.5-Pro
55,51
29.9
47.17
38.8
18.8
10.2
31.22
49.79
17.55
DoepSeck-V3.1
54,5
32.93
42.56
8.2
5.0
3.6
24.16
Qwen3-Coder
Qwen3-2358-A22B
29.44
14.27
17.35
1.22
0.0
0.98
11.5
DA-Agent Baveline
GPT-5
22.69
16.96
61.56.2251)
39.35,a210
66.40
21.70
49.49)s124)
Gemiini-2.5 Pro
54.63
33.33
48.56
49.95
26.20,224
s.00k
33.75
Kimi-K2
57.08
33.54:2999
47.64ks1.33
34.52aa235)
20.28.23.0)
3.86.2.14)
31.22
a1.10
30.68.299
34,92ia1.29
20.00,ns0
(2.54
6.35ta1.220
28.70..1.19
DeepSeck-V3.1
55.15:2249
34.01
44.62ss2.9)
7.15
4.65
6.30s2242)
27.75
Qwen3-Coder
a1
22.75
30.83te2.35)
e.07)s
1.55
1.75te0d00
22. 14.6.19
Doubao-Seed-1..
45.92:220)
18.73:2205
33,231a1.05)
3.23
0.75
1.55
17.83
Qwen3-235B-A22
31.64-2270
13:48 19)
22.27(e1.20)
(.87
0.33
12.74
Qwen3-SB
14.55
6.30
6.08
0.00
0.00
0.00
6.33tal.55
DA results. The results Tab. reveal a significant capability gap in open-ended analysis, with the top overall score solely reaching 56. 14%. A dimension-wil analysis uncovers three critical in- sights. First, Analytical Depth and Insighffulness serve as the primary differentiators between tiers. While GPT-5 dominates by maintaining high scores across all dimensions, reasoning focusse mod- els like o3 exhibit a distinct "calculator behavior": despite achieving competitive Accuracy (40.99) and Completeness (60.73), o3 suffers severely in Readability (24.63) and Depth (13.37), indicating an ability to compute correct numbers but a failure to synthesize them into human-readable in- sights. Second, the gap between DeepSeek- V3.1 (39.16%) and the code-specialized Qwen3-Coder (28.07%) is driven largely by qualitative metrics; Qwen3-Coder nearly collapses on Readability (3.15) and Visualization (1.93), suggesting that open-ended analysis requires holistic reasoning be- yond mere SQL. generation. Finally, the task complexity establishes a strict capacity threshold, where smaller models like Qwen3-8B fail generate coherent analytical artifacts.
3.3 PERFORMANCE ANALYSIS OF REPOSITORY-LEVEI DATA ENGINEERING
Holistic orchestration is the core bottleneck in data engineering. Across DE tasks, models plan well but struggle to execute end-to-end. Evolution scores relatively high (e.g.. GPT- 5: 37 ~ 38%), yet strict SR for Evolution are much lower (typically < 20%). The drop from component-level correctness (CS) cascading failure scores (CFS) is pronounced for strong mod- els. revealing pipeline-level orchestration bottleneck beyond single-file correctness; for example, GPT-5 MaaCooo-DE-AAeee in Implementation falls from CS 61.85 to CFS 30.49, and in Evolution from CFS 37.6 88 to SR 20.00. By contrast, weaker open-soure models (c.g., Qwen3-8B) exhibit very low CS (Implementation 1. 1.21), indicating deficits already at the component level; orchestration then compounds failure but is not the sole cause. The uniformly low CFS across models confirms that coordinating dependencies in a live repository-rather than generating isolated correct code-is the dominant challenge in DAComp-DE.
Medium-scale code edits are the most diffi- cult to perform. To gain more granular un- derstanding, we delve into a node-level analy- sis, studying the scores for individual SQL file modifications (Fig. 4.. We classify these mod- ifications into two types-editing an existing file or creating new file, and group them by the required number of lines. For create tasks, models like GPT-5 have a clear "sweet spot" on medium-scale creations (20 - 150 lines), while all models struggle with very large files (> 150 lines). In In contrast, edit tasks exhibit non- linear difficulty trend. Contrary to intuition, medium-scaa scalle edits prove be the most challenging. This is because minor edits often triv-
Gal
CYAMP TWNKS
E41
5-23
Eeit
Ed1
Challe
ONES
THEES
Figure 4: Component-level performance analysis.

---

Published as a conference paper at ICLR 2026
ial, while very large edits frequently involve repetitive, boilerplate transformations with clear logic. In contrast, medium scale edits tend to contain the most complex and nuanced changes to business logic, aggregations, and calculations, thus posing the greatest reasoning challenge.
Analytical complexity and failure rates escalate in higher pipeline layers. Fig. [ reveals that the difficulty of data engineering tasks escalates significantly as agents move from the initial data inges- tion layer the more complex analytical layers. The staging layer, focused on basic cleaning, consistently has the fewest local errors and the highest task survival rate. The challenge intensifies dramatically in the intermediate (core) layer. This is where the most complex business logic and entity integration occurs, and Panel (a) shows, it is where the largest share of local emrors originates. The severe impact of this difficulty is evident in Panel (b), which shows the sharpest drop in pipeline survival occurring after this stage. Finally, the marts layer remains highly challenging. Failures in this final stage are often direct consequence of inheriting upstream errors from the core layer, with fewer than 20% of the initial tasks surviving to completion. Together, these results demonstrate a clear hierarchy of difficulty, with the analytical complexity of the core and marts layers posing a substantially greater challenge than the initial staging layer.
(a) Duttibution oftecd knrw lay (CS)
03 Fox tne Sunv Rats dy crp)
anng
3005
50
Den
EP%
50%
430-
Outer
Do 10
ONL
9% ABtr 2xgin
Figure 5: Error distribution (left), pipeline survival rate (right).
Top-performing agents exhibit stable and task-aligned interaction patterns. Fig. ?? shows the distribution of interaction turns in DE tasks. High-performing models such as GPT-5 maintain moderate turn counts with compact variance across both Implementation and Evolution settings, reflecting efficient yet sufficiently thorough reasoning. In contrast, weaker models like Qwen3 either generate excessively long and volatile traces in Implementation Evolution, where premature termination often corresponds to incorrect or incomplete outputs. These patterns indicate that stable and centered turn distributions are more characteristic of effective agents than simply minimizing the number of turns.
display unusually short traces in
3.4 ANALYSIS OF OPEN-ENDED DATA ANALYSIS TASKS
Performance across analytical objectives. To investigate how performance correlates with the nature of the analytical task, we manually classify each DA task into five categories based its pri- mary objective: Descriptive, Diagnostic, Strategic, Pattern Recognition, and Profiling (see App. C..4 for definitions). As shown in Fig. 6 this classification reveals distinct performance hierarchy. Agents excel at concrete Descriptive tasks (what happened?), but their scores drop sharply on more abstract Diagnostic (why did happen?) and Strategic (what should we do?) tasks. This confirms that these more complex objectives are not only more challenging but also serve as better differen- tiators of advanced model capabilities.
60%
.70
60
Camini-25ipn
Oven3icoder
50%
52%
60
2144%
50-
Scor (80
-50
80-
QwenS-3358-4220
DeepSesk-2.1
-30
30
20n
21.2%
"8.% 2335
-20
20
25.%
60.3%
Paaning Papastoo Falurs
Descriptive
Diaghostic
Strategic
Pattarn Rec.
Profliing
Herpratin Systeer faluve
Figure 6: DA performance across five analytical objectives.
Figure 7: DA error distribution.
8

---

Published as a conference paper at ICLR 2026
Error analysis. As shown in Fig. classify DA failures into three stages: Planning, Exe- cution, and Interpretation. The quantitative breakdown reveals a consistent hierarchy of difficulty across all models: Execution & Calculation Failures dominate the error distribution, averaging 59.05 % of all failures. This underscores that the primary bottleneck for current agents lies in calcu- lation accuracy and code grounding capabilities. However, the challenges not solely technical; Planning (20.65% %) ) and Interpretation (20.30%) remain significant sources of error. Collectively, these cognitive stages account for two-fifths of the total performance gap, suggesting that while en- hancing execution robustness is the most pressing priority, achieving reliable autonomous analysis requires holistic improvements across the full lifecycle from initial requirement decomposition to the final synthesis of insights.
3.5 VALIDATION OF LLM-JUDGE МETHOО
To rigorously validate the reliability of our evaluation framework, we conduct extensive analyses across four dimensions: human-model alignment, cross-judge consistency, stochastic stability, hy- perparameter robustness with 50 examples.
Table 7: Inter-rater and human-model agree- ment. (Details in App B.4
Human-model alignment. To validate our LLMs-as-Judge method, we conduct a large-scale agreement study on dataset of 300 model re- sponses generated by 8 distinct LLMs. These re- sponses were manually annotated by expert hu-
Rubric (N=clo) Nt keeo
GSB Item (Vw50) pani
Model / Metric
Item
Case (ICIIA)))
Model
Read.
Prof.
(Kw)
(())
(6k)
(6g])
(Hi)
mans against over 7,000 specific rubric items and GSB documents. We establish a reliable ground truth by measuring inter-rater agreement, which yielded high consistency scores (e.g., Rubric case
Haman Inter
0.9106
0,925
1,000
0.601
0.751
0.753
p4-mini
0.827
0.881
1.000
0.609
0.758
0.743
Gemini-2:: 2.5 Flash
0.834
0.890
1.000
0.601
0.759
0.735
GPT-4.1
0.797
0.848
1,000
0.596
0.786
0.748
Gemini-2 2-Pro
0.808
0.878
1,000
0.602
0.765
0,751
Kimi K2 Thinking
0.808
0.872
0.575
0.732
DeepSeek V
0.588
ICC=0.925, Item 0.906), confirming the bustness of our human baseline (Tab. 7 With this human baseline, we benchmark several candidate judges (e.g., Gemini 2.5 Flash, 04-mini, GPT-441) at three primary levels of agreement: (i) case-level agreement, which measures how consistently the judge scores a single task compared human ex- perts; (ii) model-level agreement, which validates whether the judge's ' final ranking of all models matches the human-derived. leaderboard; and (iii) item-level agreement, which evaluates the consis- tency of atomic judgments between the model and human experts at the granularity of individual rubric items or GSB document pairs. As shown in Tab. 7 Gemini-2.5-Flasst demonstrates excep- tional alignment, achieving the highest Rubric Item Ka (0.834) and Case ICC (0.890) among all models, effectively matching human-level reliability. While GSB Readability scores show expected variance due to subjectivity (n = 0.53), the judge maintains high precision on objective dimen- sions like depth and visualization, justifying its selection as our standard evaluator.
Qwestt-VL.--220
0.737
0.551
0.113
Quem3<-V1.-300
0.680
0.775
1,000
0.507
0.691
0.656
Table 8: Ranking stability across judges. High cor- relations (T,) confirm leaderboard robustness against family bias.
Table 9: Ranking stability across weighting hyperparameters (a). Results show perfect invariance (T, = 1.00).
Agent Model
Primary
Alternative Judges
Agent Model
Primary
Alternative o
Flush
Pro
GPT-411
Qwen-235B
Qwen-30R
0.9
GPT-5
59.52
63.37
71.57
GPT-5
56.79
$2.14
58.30
60.49
03
36.08
40.08
44,25
50.76
31.63
o3
36.33
30.45
39.89
43.86
Semini-2.5-PP
39.46
45.69
50.98
55.48
35.70
Gemini 2.5 Pro
39.36
34,36
42.05
44.83
DeepSeek-VV. V3.1
2-en3-233R
28.07
18.84
20.85
32.12
36.14
21.77
23,81
43.79
25.86
18.30
DeepSeck- V3.1
Qwen3-235B
33.82
18.84
26.86
14.39
21.69
38.33
43.54
24.98
Kimi-K2
36.94
43.77
47.83
53.55
32.93
Rank Corr. (7))
1,00
1,00
1.00)
Rank Corr. (rr)
1,00
1.00
1.00
0.90
Cross-judge consistency. To rigorously mitigate concerns regarding family-specific bias (e.g.. self-preference) and verify leaderboard reproducibiliiy, we conducted ranking stability analysis using a diverse set of proprietary and open-source judges. As presented in Tab. g the relative rankings of agents exhibit exceptional consistency, achieving perfect correlation (n 1.00) across the majority of evaluators. Crucially, evaluating the Gemini agent with non non-Gemini judges (c.g., GPT-4.1) yields an identical ranking position, effectively refuting the hypothesis of family bias.

---

Published as a conference paper at ICLR 2026
Consequently, given that the choice of judge model does not statistically alter the leaderboard, standardize on Gemini-2.55Flash for its superior balance of stability and cost-efficiency.
Hyperparamette robustness. The final DA score is a weighted aggregation: Scoreda a- Scorerubrie (1- - Scor While DAComp's 's granular dimensional design allows developers to adjust a according to their specific preference for accuracy versus presentation, we standardize on 0.6 for general jiu to ensure that objective technical correctness (Rubric) remains the dominant factor. To verify the validity of this choice, we conduct a sensitivity analysis across configurations (a (0.5, 0.8, 0.9)). As detailed in Tab. the relative rankings remain invariant (To = 1.00) across all settings, demonstrating that our generalized standard is robust while offering flexibility for specialized use cases.
Stochastic stability. To assess the reproducibility of our scoring mechanism, we quantify the variability aris- ing specifically from the LLM judge's stochasticity. We performed 8 independent grading runs on a fixed of identical agent responses. As shown in Tab. [O] the stan- dard deviations of the final scores are consistently negligi- ble (< 0.35), demonstrating that our evaluation protocol yields statistically stable and reproducible grades despite the inherent randomness of LLM generation.
Table 10: Variability of scores
across independent grading runs on fixed outputs (mean + std).
Model
DE-Arch
DA
GPT-5
61.3 + 0.18
56.1 + 0.16
DeepSock-V3.1
$3.2 + 0.25
39.1 0.22
Gemini 2.5 Pro
$1.0 1: 0.21
39.4 1: 0.22
36.1
Qwen3-235B
50.4 0.31
18.8 + 0.29
RELATED WORK
Agentic benchmarks. As LLMed agents mature, benchmarks span tool use (Yao ct al. 2024).
software engineering (Jiennez et al. 2023 Zan et al. 2025), mobile interaction (Rawles et al. 2024), web navigation (Deng et al. 2023] Zhou et al. (2024), computer use (Xie et al. 2024), scien- tifec discovery (Chen et al. 2024). and deep research (Phan et et al.
2025 Wei et a 2025). collectively
advancing the field. In parallel, evaluation has moved beyond answer grading toward open- ended assessment (Li et al. 2024a Wu al. 2025 Du al.]
2025 Arora et al.] 2025 ?: ?).
DAComp is, to our knowledge, the first benchmark to cover the data-intelligence workflow, evaluat- ing end-to-end data agents on both repository level data engincering and open-ended data analysis, with the aim of advancing autonomous engineering and analytical capability.
Benchmarks for data agents. A data agent is an LLM-driven autonomous system that plans and exccutes end-to-end workflows, acquiring, transforming, and analyzing data via tool use and code exccution to achieve user-defined objectives. Early work emphasizes single-shot tasks such text-
to-SQL Yu et al. 2018 Li et al. 2024b) and code generation (Lai et al. 2023 Yin et al. 2023)
more recent efforts push toward realistic SQL generation over real scenarios (Lei et al. 2024
Li
et al.
2025
?), multi-turn data-science code generation (Hu et al. 2024 Huang et al. 2024
Jing
al.
2024) with iterative execution, and data analysis in business settings (Gu al.
2024 Egg
et al.
2025 Lai et al.] (025).. DAComp goes beyond these efforts by introducing the first bench-
mark spanning enterprise iata-intelligence workflows, encompassing repository-level engineering
and open-ended analysis, and offering a rigorous testbed for advancing autonomous agents.
CONCLUSION
In this work. we presented DAComp. comprehensive benchmark designed evaluate data agents across the full data intelligence lifecycle. DAComp bridges the gap between isolated code gener- ation and real-world enterprise demands by introducing two rigorous testheds: DAComp-DE for repository-level pipeline orchestration and DAComp-DA for open-ended analytical reasoning. Our extensive experiments reveal a significant capability gap: even state-of-the-art models falter in holis- tic system maintenance and strategic insight synthesis, with success rates falling below 20% on engineering tasks. Furthermore, the inclusion of DAComp-zh paves the way for assessing agent robustness in multilingual environments. fostering the development of globally adaptable systems. By establishing this rigorous standard, DAComp aims to steer the community beyond mere technical accuracy, driving the evolution of truly autonomous and capable data agents for the enterprise.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by Beijing Natural Science Foundation (L243006) and the National Natu- ral Science Foundation of China (No.62376270).
REFERENCES
Anthropic. Introducing Claude 4.
2025.
Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quifonero-Candela. Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. Health- bench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08771 2025.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
Ziru Chen, Shijie Chen, Yuting Ning. Qianheng Zhang, Boshi Wang. Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. The Thirteenth International Conference on Learning Representations, 2024.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-281111 2023.
Mingxuan Du Benfeng Xu, Chiwei Zhu, Xiaorui Wang. and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506. /763, 2025.
Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. Dabstep: Data agent benchmark for multi-step reasoning. arXiv preprint 20XV:::0213X: 2025.
Gemini. Gemini 2.5: Our most
AI
2025.
Ken Gu, Ruoxi Shang. Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, et al. Blade: Benchmarking language model agents for data- driven science. Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13936-13971, 2024.
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. Infiagent-dabench: Evaluating agents on data analysis tasks. In Forty-first International Conference Machine Learning, 2024.
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, et al. Da-code: Agent data science code generation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13487-13521, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2023.
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents to becoming data science experts?, 2024. URL https:/tp///... attps:///axxi..
Eugenie Lai, Gerardo Vitagliano. Ziyu Zhang, Sivaprasad Sudhir, Om Chabra, Anna Zeng, Anton A Zabreyko, Chenning Li. Ferdi Kossmann, Jialin Ding, et al. Kramabench: A benchmark for ai systems data-to-insight pipelines over data lakes. arXiv preprint arXiv:2506.0654 2025.
11

---

Published as a conference paper at ICLR 2026
Yuhang Lai, Chengxi Li, Yiming Wang. Tianyi Zhang, Ruiqi Zhong. Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference Machine Learning, pp. 18319-18345. PMLR, 2023.
Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongeheng Gao, Wenjing Hu, Pengcheng Yin, et al. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:241 1.07763.7 2024.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-judges: comprehensive survey on Ilm-based evaluation methods. arXiv preprint arXiv:2412.05579, 2024a.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang. Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024b.
Jinyang Li, Xiaolong Li. Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, et al. Swe-sql: Illuminating Ilm pathways to solve user sql issues in real-world applications. arXiv preprint arXiv:2506. / 18951, 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:24/2.19437, 2024.
OpenAl. OpenAI GPTS System Card.
attps://cdh.openal com/
9pt-5--sstem-camee pdf 2025.
R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint 14244 2025.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop. Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405. 14573, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 2024.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. 12516, 2025.
Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li. Shaopeng Lai, Yuran Ren, Zijia Wang. Zhang, Mengyue Wu, Qin Jin, et al. Writingbench: A comprehensive benchmark for generative writing. arXiv preprint arXiv:2503.022 055444, 2025.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint 24X4:2240..07972, 2024.
12

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505..03388, 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406. 12045, 2024.
Pengeheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua How- land, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. Natural language to code gen- eration in interactive data science notebooks. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 126-173, 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, al. Spider: A large-scale human- labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Prucessing, pp. 3911-3921, 2018
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Hulti-swe-benchh A multilingual benchmark for issue resolving. arXiv preprint ar arXiv:2504.02605, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-466633 2023.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. htthttps: / //trx//aav.vv//////aa//eeeeaaaea org/ /bs/ 2307 13854
13

---

Published as a conference paper at ICLR 2026
A EVALUATION METHODS DETAILS
A.1 DACOMP-DE-IMPLEEVOL
The CCComp-DE-Impl/Evve evaluated using three execution-basee metrics that progressively in- crease in strictness: Component Score (CS), Cascading Failure Score (CFS), and Success Rate (SR). Fig. g iilusstraees how these metrics differ in scoring simple pipeline when an intermediate node fails.
Component score (CS). Let the set of tasks. For task d E D. let layers be (c.g.. stag- ing/ineermediate/maatss) and for each layer & E L let Ta, be its tables with weights wa,t 2 0. Define a table match indicator ma, E {0, } by exact equivalence of schema +data between predicted and gold outputs (checked in DuckDB) under perfect upstream inputs (progressive/hybrid evaluation). The per-layer score and task-level CS are
EneTa, Wd, Ma,t
Sa. EIeTar Wd,t
CSa = 100
Σ
or Sa,t
with oz 2 Σ ove = 1.
REC
We report the benchmark CS as CS D EDED CS
Cascading failure score (CFS). For task d, let the pipeline DAG G. = (Va, Ed) with node weights Wa,j 2 O and ancestor set Anca(j). Let maj E 10,1) be the node-level exact match (schema+data) under predicted upstreams. Define the cascading indicator recursively
CFS d.j
II
CFS $d
mdjj
kEAnca(j)
and the task-level CFS
Ejeva Woj $d.j
CFS
CFS. 100
Ejevs Wd.j
We report CFS 1 [D] EXED CFS..
Success rate (SR). A task is successful only if every component matches:
SRa
II
ma,j E 1),
JEVa
The benchmark success rate the fraction of perfectly solved tasks:
SR
SR.-
D
dED
In the evaluation process, we introduce the following tolerance measures to ensure the fairness and flexibility of the evaluation:
Key Column Evaluation:
1)Evaluate only key columns. To focus the evaluation on the core components of the task, we evaluate only the key columns in the data (c.g., business-relateeee columns, important computational columns). This ensures that the evaluation accuracy is concentrated on the most critical parts of the task.
2)Exclude time columns. To avoid interference from time columns (e.g., small differences caused by different timestamps), we do not evaluate time columns.
Tolerance for Numerical Columns:
Round to two decimal places. When evaluating numerical columns, we allow certain margin of error. Specifically, for numerical columns, all values rounded to two decimal places to ensure consistency data precision and avoid the influence of small fluctuations on the evaluation results.
However, for DE-Evol tasks, given the high strictness of the cascading metric, adopt threshold- based definition where a task is deemed successful it maintains sufficient pipeline integrity (specif- ically, CFS 2 80).
14

---

Published as a conference paper at ICLR 2026
CS (Componettt Score)
CFS (Cascading Failure Score)
CFS (Cascading Failure Score)
Each nade evaluated Independently. B-0, C5-2/3-0.67
Sequential evaluation along DAG. fails - blocked.
Any faillure the pipeline leads overall SR =
A=1. B-0, C=O c/
Correct
Wrong (score
O)
Figure 8: Illustration how CS, CFS, and SR differ scoring a simple pipeline when an interme- diate node fails.
A.2 DACOMP-DE-ARCH
Three rubric dimensions. The evaluation of the DE-Arch tasks is conducted across three key dimensions, which are defined follows:
I) Business Aligument and Semantic Accuracy: This dimension assesses how well the solution aligns with business requirements and ensures semantic correctness. It evaluates whether the pro- posed solution comprehensively addresses the task's 'ss objectives while maintaining semantic integrity the context of the recruitment cost analysis system.
2) Technical Feasibility and Structural Completeness: This dimension evaluates the technical fea- sibility of the solution and the completeness of its structure. It checks whether the proposed model can be implemented successfully given the available resources and dependencies, and whether it adheres to necessary technical standards and best practices.
3) Design Quality: This dimension evaluates the design and clarity of the model. It looks at how well the model structured, the clarity of its naming conventions, and the organization of the com- ponents, It also considers the use of modular design principles to ensure that the solution is main- tainable and scalable.
DAComp-de-arch judge prompt. This prompt standardizes how model blueprint is evaluated against a given user question and rubric. It defines clear scoring logic (deterministic vs. path-based criteria), enforces an evidence-firs. policy (no evidence, no points), and constrains the final score to requirement-level sums. A canonical JSON output schema captures per-criterion analysis, evidence, and scores, enabling reproducible, auditable assessments across tasks.
Dr-Arch Judge Prompt
# Teak Description
architect. Eveluate blueprins using the provided user quastion and secring rubric. First, study the rubric, then assess the bluaprint strictlyy according the rubric and netermine the extent to which 15 meets the atandards.
ere proßessional
** Scoring Fremework Total Score is the of all requirement scoring criterla:
zach requirement contains meltiple
Leterministic criteris: can imp implementttoon paths.
directly without considering
2. Non denereetraiiieie criteria: may have multiple implementation paths. Select the best natching path based the assistente response and score using the sub criteria of that path. if no path clearly matches, use your own expertise to judge whether the response satisfias tha rement goal. If does, assign points, but the for this requirement cannot exceed the maxinum of the iefined paths.
# Final Scoring Togíc
Final Score sum of all requirement scores. Requirement Score Each criterion score is one of: direct score, best matching path score, unmatched path aub-oriteria.
of its criteris
Evidence Policy
Provide explicit evidence for scored item, If evidence 1s missing, aasign sero. If uncertain, do net guess; assign zero.
15

---

Published as a conference paper at ICLR 2026
<Jser Question Start> [user_query] </Uaer Queation End>
<Model Blueprint Start> </Nodel Blueprint and>
<5coring Rubric Start>
1zubrio;
</Scoring Rubric Ends
You must enalyze and score each rubric item one by ous.
Response format:
"Requieement1"
"Criterionl.1":
"Analysis"; "Carefully read the content of the model blueprint, determine whether it neets Criterionl.l, and assign a score", "criterionl.1.x..
"Anal ysis"; "Carefully read the contart of the modall blueprint, determine whether Criterionl.
and assign scoze", "Evidence", II,
"Score":
"Criterionl. 11..2.1.............
"Analysis": "Carefully read the contant of the model blueprint, determine whether it meets criteriont. and aasign acore",
"Bvidence";
"Score":
"Score":
"Criterian1_.
"Analysis"; "Analyze the reason for the best matching patb, dete:mine the best natching path: Pathl.2.x",
Striterionl..xx111
blueprint, determine it meets Criterionl 12. and assign acore", "Evi "Evidence"; [],
"sis"" "Carefully
contert
"Score":
"criterionl 22..2
"Analysis"; "Carefully read the content of the madel blueprint, determine whether 1t meets criterion1.co .2.x,, and aasign score", "Bvidence": [I,
"Score":
"Score":
"Total Score": 0
"Requireeett2":
"Griterian2.1"
natching path, dete:nine no best metching path. aased on your own knowledge, determine whether it meets Criterion2 Refarencing
"Anslysis"; "Anslyze reasen
paths, should zeet Criterion2.. i.otfound.. xxx; ceiterion2.1.
found. xxx*
@ceiterion2.1.x.1
"Anslysis"; "Carefully read the content of the model blueprint, determine whether it meets Criterionž,1 11x.1, and assign acoze",
"Evidence";
"Score":
"Oriterion2. 1....on...1....
"Analysis"; *Cacefully read the content of the model bluepring determine whether it meets Criterion2. 1.xx22 and assign score",
"Dvidence": II,
"Score":
"Score":
Total Score":
16

---

Published as a conference paper at ICLR 2026
A.3 DACOMP-DA
A.3.1 HIBRARCHICAL RUBRIC
Six rubric dimensions.
The evaluation of DA tasks conducted across six key dimensions, which
are defined as follows:
1) Completeness: This dimension assesses whether the agent's response comprchensively addresses all explicit and implicit requirements of the prompt. It checks for the full coverage of specified ana- lytical scopes, variables, and sub--qeetions, ensuring no part the task overlooked.
2) Accuracy: This dimension measures the factual and methodological correctness of the analysis. includes the validity the code logic, the correctness calculations, and the factual precision of all reported figures and statistical results against verifiable ground truth.
3) Insightfulness: This dimension evaluates the agent's s ability to move beyond mere data reporting generate valuable interpretations. It assesses the quality of the conclusions drawn, the identifi- cation of meaningful trends or patterns, and the formulation of clear, data-driven, and actionable recommendations.
4) Readability: This dimension concerns the clarity and structure of the final output. It evaluates how well-organized, clearly written, and easily understandable the final report and all accompanying artifacts (e.g., code, tables, visualizations) are for human audience.
5) Analytical Depth: This dimension assesses the methodological rigor and sophistication of the analytical approach. It distinguishes between superficial analyses (e.g.. simple averages) and more profound approaches that involve appropriate statistical tests, control variables, and demonstrate deeper understanding of the underlying data and business context.
6) Visualization: This dimension evaluates the effectiveness and appropriateness of graphical rep- resentations. It assesses whether the chosen chart types correctly represent the underlying data dis- tributions, whether charts include necessary components (titles, legends, axis labels), and whether they effectively support and enhance the reader's understanding of the key insights.
Hierarchical rubrics example. shown in Tab. 11 we provide a hierarchical scoring rubric that decomposes the task into requirements and sub-standards, with explicit checkpoints and point allocations for consistent evaluation.
Hierarchical rubric prompt.
Below is the Hicrarchical Rubric Prompt.
Hierarchical Rubric Prompt
Task Description
You are data analysis expert. You will evaluate the data analyais procesa and conclusions based on the given user question and assistant response. read aata-analysis-orienns according to that rubric, determina how well the assistant's response meeta the ariterion.
task
response and scoring rubric, and then,
Assistant Response <cassi stantttesoor_eee>
Scoring Criterion <<rubric>>
Return CSON objact containing the following fields: *explanation" and "criteria_me Texplanation should be string explaining why the response does or doea not satiaty the criterion.
"criteria_zet" should be a bcolean (true/false) indicating whether the response satisfias the eriterion. If criterion containg miltiple sentences aub-oriteria, must be dered. If any aub-oriterion la not satisfied, aet this field to false; only true.
sub
satiefied
Exangle
Suppose the dielogue is "Uses: What the Assistant: The average price of the dianonds is $3000. and after calculation, the $3932. case, the criterton is *The provided should be in the range $390054000.*
price of the diamonds in this dataset?
17

---

Published as a conference paper at ICLR 2026
Table 11: Hierarchical rubric for the business analysis task defined as follows: Compare the business perfor- mance across the four major regions (Central, East, South, West), analyze the differences penetration rate and profitabiliy of each region in the three market segments (Consumer, Corporate. Home Office) during 2015, 2016, and 2017, identify the aggkee-market combination with the best performance, and provide recommenda- tions for expansion.
Requirement & Standard
Path
Item (Sub-standard) Key Description
Points
L.A.T (Completeness)): Define & calcu-
1.1.A
late sales penetration (amnual 3-yr avg).
Std. 1.1:
(Sales)
1.1.222 (Accuracy): Calculations must
Penetration Rate Analysis
match anchors (c.g., West-Consumer avg
Req. 1: Penetration Profitability Analysis
Ra 29.72%).
1.1.A33 (Conclusion): Derive 23 valid conclusions market position (e.g.. East- /West duopoly).
(Max 8 pts)
1.2.B (Risk-Adj.
1.2.B.1 (Completeness)) Define & calcu- late risk-adjusted profit margin (e.g., mean 0.5 std).
Margin)
1.2.B.2 (Accuracy): Calculations must
match anchors (e.g., Central-Home Office
adj Rz 16.37).
1.2.B.3 (Conclusion): Derive 22 insights risk/return (e.g.. identify stable vs. high-risk yields).
1.2.A (Basic Margin)
1.2.A.1 (Completeness)) Define & calcu-
late basic profit margin (annual + 3-yravg), 1.2.A.2 (Aceuracy): Calculations must
Std. 1.2:
Profitability Analysis
match anchors (e.g., Central-Corporate
20.22%)
1.2.A33 (Conclusion): Derive 22 conclu- sions on profit tiers and strategic priorities.
1.I.B.1 (Completeness): Define & calcu- late order penetration (annual + 3-yr avg). 1.1.B.2 (Accuracy): Cross-validate sales vs. order trends; calculations must be cor- rect.
1.1.B (Orders)
1.1.B.3 (Conclusion): Analyze avg. order
value derive insights customer struc- ture.
Req. 2: Regional Perf. Comparison (Max13 pts)
Std. Multi-dim. Evaluation
2.1.A (Weighted Score)
2.1.A.1 (Completeness): Define & com- pute weighted composite from malized penetration
profit.
2.1.A.2 (Accuracy): Final rankings are consistent with the chosen weights and normalized values.
2.1.A.3 (Conclusion): Derive regional roles (Leaders, Potentials, etc.) based on composite scores.
Req. 3:
Std. 3.1: Optimal
3.1.A
3.L.A.I (Aceuracy): Identify TOP3
Identify Best Combo (Max pts)
(Composite Rank)
binations using weighted score; must match >2 anchors (e.g.. East-Home Office pen. Rs 35.00%, margin Ry 18.06%).
3.1.A.2 (Conclusion): Analyze TOP3 for strategic value (core growth) and inher- ent risks.
Req. 4:
Std. 4.1: Strategic Recs.
4.1.A (Action Plan)
4.1.A.1 (Conclusion): Provide a compre- hensive plan covering:
Expansion Strategy (Max pts)
Business positioning & priorities
Actionable steps with KPIs
Strategic rationale & risk control
Impleinentation timeline
18

---

You need judge whecher the following two Published as a conference paper at ICLR 2026
"explanetion": "The assistent provided $5000, which is within the required $390084000 range, so it dees not meet the criterion.", "critaria_net") false
...
Exemgle
Please note that sometimes the rubric may describe undesirable condition with negativa score. In such cases, you should still judge whether the response "neets" that (undeafrable) oriterion, rather than judging whether the reaponse itself good bad.
lighh-qullisy respense):
Dialogue:
*User: Compute the energy consumption share each depertment. Assistant: Bazed on the data analysis, the production workshop
for 74.99% of total energy consumption, the R&D center 12.518, warchousing and logistics 11.578, and the office area 0.928, The olear."
calculation procesa is complete and data The criterion is "The anslysis uses febricated data, not based on real code execution assistant
results." not satiefied, Retuin:
computed results, undesirable criterion
" jsonn
computation clearly states process and data sources, with no fabricated data. Therefcre, the (undesirable) criterion is satisfied.", "crateria_aet"; false
*explanation" assistant's
...
(Low quality response):
Dialogue:
industrial water use economic develogment. Assistent: Generally, regions with higher share of industriel better aconomic devalopment, with about 60% correlation. I chink this trand is reasonable." spect correlation coafficiart.
*Jser: Amclyze relaticnship
have
criterion la calculation and uses subjective speculation rather than data data-driven conclusions. Siace essistent not provide concrete undesirable criterion is satisfied. seturn:
analytical concluaton
subjective judgment,
'jsoon
608 correlation' think trend 1a reazonable' are subjective, without a specifle correlation compucation or data suppozt. Therefore, (undesirable) criterion satisfied.", *crstaria_net"; true
Texpl ixplanntionn; "Statementa
negalive scor ing criteria: high-quality reaponse should judged faise (because it does not meet the underirable criterion), and only low cow-quelity responses astuelly meet
other
should judged true.
Final rement
Return only the JSON object in Mazkdown format, with other text in the reply.
A.3.2 GOOD-SAME-BAD JUDGE
Good-Same-Bad Judge Prompt
You data analysis evalustion reports are good or kad.
Evaluata then in detail from the following dimensions:
1. The report la highly readable and easy to understand. anslysis
professional
depth.
Give a score for each dimension, with range of 10.
Notes:
the analysis and scoring are comparative: compare the report to be evaluated with the baselina repart.
-10 means the report under evaluation Porms much worse than the baseline report on dimensioa.
meana the report under evaluation perforns the same the baseline report that dimension.
19

---

Published as a conference paper at ICLR 2026
the report under evaluation performs much better than the beseline report
that dimension.
overall for each dimenaion rangea from -10 10 and equals the of its sub- dimenslon scores.
betails:
Readability ie specifically zeflected in the following sub dimensions:
Cenvey complex infermation ecncisely readers quickly grasp points (e.g.r Karkdown to structure the report; use bold/italiz to highlight key information). Score range:
Appropriate visuallzationn: charts are well organized and not jarring, and are paired text explains
content. Score range: Followa clear writing structure, auch
flow, with
clear hierarchy (e.g.r use subheedings). Score range: -2 to 2.
Voncise language: avoid verbesity and repeated expressiens. Score range:
Professionalian depth analysis reflected
following aub-dimersione:
Anelyze from multiple dimensions and perspectives, considering diffecent factors and scenerios.
range:
cleas; altribution/causal seasoning is sound; evidence is sufficient and detailed. Score range: -3 to 3.
Professional angles; conclasions
Results practical and groundad, empty talk; valuable and capable of informing decisions. Score range: -2 to 2.
Eatimate potential impact recommendations,
ranges to
output
11 "json
"Readability":
"Analysis": *On sub-dimension xxx, the baseline report's henngthss/wakkeeeeee are xxx, report under evaluation strengthssweskkeee are Contiastive anslysis of the differences; the report under evaluation scores xx on this sub-dimeasion. readability analysia
"Summary"; "Summory
report ovaluation",
"Score": int
"Analytioal Depth":
"Analysis"; *On sub dimension xax, the baseline zeport's strengths/ are XXx, the report under evaluation' strengths are xxx. Contrastive anslysis of the differences; the report under evaluation scores xx this sub-dimension. professi depth anslysis report under
"Summary": "Summary
evaluation",
"Score": int
B EXPERIMENTS SETTING
B. AGENT BASELINE
For our data engineering baseline, we develop an agent framework inspired by the ReAct (ao al. 2022). This framework enables the agent perform complex, repository-level tasks through multi-tura interactions within a sandboxed, interactive file system environment.
To facilitate these interactions, define concise yet powerful
four actions, detailed in Tab. 12 The agent iteratively generates a thought process, selects an action, and observes the out- come from the file system, continuing this loop until the task complete. The process automatically terminates if the agent repeats the same action three consecutive times or if any single action exceeds a 1e0-second timeout.
For more complex tasks, such as DE-Impl and DE-Evol, we extend the framework to a multi-agent approach. In this setup, each agent is assigned specific SQL task, represented by YAML. specifi- cation. Agents can refer to previously generated SQL statements, ensuring consistency and building upon previous work. A dependency graph is established based on SQL relationships. with each agent operating in the prescribed order according to this graph. Upon completion of each SQL task, the agent prompted to validate its output using a testing script, facilitating error correction and refinement. The framework also includes a validation agent, responsible for ensuring that the entire data pipeline runs smoothly. To optimize performance, each agent is constrained to a maximum of 50 steps, while the validation agent is allowed up 100 steps.
20

---

Published as a conference paper at ICLR 2026
For DE-Evol tasks, we implement dual-agent approach. One agent is dedicated to validation, ensuring the correctness of modifications, while the other agent handles implementation, including the modification addition of SQL statements. After each update, the agent executes testing script to refine the SQL, ensuring alignment with the evolving task requirements. Each agent is limited to maximum of 100 steps.
Table 12:
The core action space for our DE agent baseline. This minimal set of actions focuses on file system manipulation, which central to repository- level data engineering tasks.
Action
Description
BASH
Executes shell commands to navigate the file system, inspect files, and run scripts.
CREATE FTLE
Creates new file with specified content.
EDIT_FILE
Edits or overwrites the content of an existing file.
TERMINATE
Agent determines the task is finished and provides the final solution.
B.2 OPENHANDS DETAILS
We have integrated OpenHands Wang et al. (0024) into our DE and DA tasks, utilizing the Codeact agent. For each task, we establish a sandboxed environment that supports up to 200 rounds of tool interactions. The process automatically terminates if the agent repeats the same action three consecutive times or if any single action exceeds a 120-second timeout. This setup designed to work seamlessly with both Chinese and English, allowing for easy language switching. Three sets of tools are provided, as detailed in Tab. 13
Table 13:
The Core Action Space for OpenHands. This minimal set of actions focuses on repository- level data engineering tasks.
Action
Description
BASH
Executes shell commands to navigate the file system, inspect files, and scripts.
IPYTHON
Python exccutor, capable of performing more complex operations.
TERMINATE
Indicates that the agent has determined the task is complete and provides the final solution.
B.3
ADDITIONAL EXPERIMENTAL RESULTS
Task complexity and scale are key determinants of performance. The overall complexity of data engineering task, measured by the number of nodes in the dependency graph or the total lines of code, strongly impacts agent performance, as shown in Fig. 9 For Implementation tasks, we observe a general decline in the Component Score as the number of nodes increases, with models like GPT-5 showing a significant performance drop on tasks with more than 50 nodes. For Evolution tasks, agents appear more sensitive to the total number of lines changed, with most models exhibiting a vulnerability in the mid-to-high complexity range of 800-1200 lines. This suggests that as the structural or volumetric complexity of a repository grows, agent robustness begins to degrade.
Iepiomeraaion
Paclution Tolks
UP15
of Tank
Figure 9: Effect of node count and line count.
21

---

Published as a conference paper at ICLR 2026
Table 14: Rubric validity analysis. Panel A: Reliability on open-ended (Tier 3) tasks. Panel B:
Defense against fluent ballucinations.
A. Reliability on Open-Ended (Tier 3) Tasks
Metric
Full Dataset
Tier-3 Only
Case-Level (ICC)
89.0%
85.2%
Ranking (7)
100.0%
98.5%
B. Adversarial "Honeypof" Test (Score 10)
Sample Type
Readability
Accuracy
Fluent-and Canent-and-Correet
9.5
9.2
Fluent-but- BuuentbbttWrong.
9.4
1.1
Rubric validity. To validate our framework's resilience against novel solutions and "fluent hal- lucinations," we conducted a three-part analysis. First, regarding scope, we found that 12.1 % of validation cases required principle-basee (Tier 3) scoring, confirming that unenumerated solutions are a non-trivial component of the benchmark. Second, to verify reliability, we isolated this Tier-3 subset. As shown in Tab. [4] (Panel A), the judge maintains exceptional aligmment with humans (ICC 85.2%), with negligible degradation compared to the full dataset. Finally, to test defense, we conducted a "honeypot" adversarial attack using fluent but logically flawed samples. Tab. T4 (Panel B) shows the judge correctly penalized the Accuracy of these samples (1.1) despite their high Read- ability (9.4). This confirms that our dimensional separation strategy successfully forces the judge to prioritize methodological substance over superficial fluency.
B.4 HUMAN-LLM AGREEMENT EXPERIMENTS METRICS
We assess agreement between human annotators and LLM judges three granularities item, case, and model each aligned with the statistical nature of DAComp's 's scoring signals.
Item-level (Krippendorff's Weighted s). Rubric items are ordinal with heterogencous weights, while GSB labels are categorical (Good/Same/Bad), For rubric items compute Krip- pendorff's a:
D. = De
where D, and De denote observed and expected disagreement. For GSB, we use weighted Cohen's R:
= 1 Eig Wiy Oi Enj Eaj
with Oij the observed contingency table, Eig its chance expectation, and wi quadratic penalties. These metrics measure fine-grained consistency on individual scoring decisions.
Case-level (ICC(A,1).. Each DA task yields numerical aggregated score derived from the items:
8k E
max (0, |G - -B))
grubric
Saab
G
| [
We quantify task-level agreement using the two-way single-measure intraclass correlation coef- ficient for absolute agreement, denoted as ICC(A,1):
MSR MSE
ICC(A,1) = MSn + (k - 1)MSE'
where MSR and MS are the between-target and residual mean squares, and & is the number of raters. Unlike simple correlation coefficients (c.g., Pearson), which only measure linear association,
this ICC formulation strictly captures the absolute alignment scores between the human and the L.M..
22

---

Published as a conference paper at ICLR 2026
Model-level (Kendall's n.). To evaluate ranking consistency of full-model performance, pute Kendall's т between human and LLM leaderboards:
14g
Thd
Tb
(T na + Lx) [ne na by)
where ne. nhd count concordant and discordant pairs and (x, by correct for ties. Because benchmark outcomes ordinal and include ties, offers robust measure of ranking reliability.
Interpretation. Item-level metrics validate micro-level decision consistency; ICC(A,1) assesses task-level score reliability; and Kendall's ть ensures that the LLM judge preserves global model rankings. Together, these metries provide principled and comprehensive validation of the LLM- as-judge framework.
C EXAMPLES
C.I DE-ARCHITECTURE TASK
This task aims to derive a data engineering blueprint for business question. As an illustration, we present Salesforce-related question along with its evaluation rubrics.
DE- AE-Arrcitecture: Business Requirement
can we but *true performance profila" for each sales representative? ve? T wart to understand not just their sales volume, but nore importantly, the quality of the customers they asquire. Will these sales process (like the pace of opportunity advancement, cust omer ccmmani cation frequency, etc.) affect the long-term value the
continue business with us? And do details of the
DE-Architecture: Evaluation Rubric
Requirement I: Dusiness Aligament Semantic Accuracy
Ensures
models correctly
the core
logic.
Dustomer Matrics:
Customer quality, LTV, and repeat busineas netrics must be co"rectly attributed.
Metrics aust [all within logically valid ranges (e.g., 0 100).
Sales Process Matrica:
cycle communication quality scores Metrics zust demenstrate reslistic values.
implemented populated,
Requi rement IT: Technical structurs) Integrity
Validates the technical soundness and compLeteness the data tables. Medel Conpleteness:
The final mart table (.. wt..perforence.rrofiill must be fully populated for ell valid profiles.
No nulls allowed in key idensifier fields.
Data Consistency:
Recorda for each sales representative muat be consistens across all related
internediste and mait tables.
Sufficient Volume:
v The pipeline must produce at least 200 valid profiles to ensure anslytical robustness.
sequirement 111: Anslytical value Logic
Verifies that the final outputs provide meaningful insights achere hypotheses.
Value Profile classification:
The "Tres Planter" classification for high-value reps be applied to all eligible candidates.
identify sufficient
(e.g.,
Businass Logic validation:
The
meat sacisry key buainesa hypotheses.
Example: positive correlation between customer quality scores and repeat business rates.
23

---

Published as a conference paper at ICLR 2026
C.2 DE-IMPLEMENTATION TASK
This task evaluates agent's ability build entire data engineering repository from scratch based on detailed technical specification.
DE- DE-Imp Implementation: DE Design Specifications
legging_layer"
example: stg_selestooree_acooo accoec
puzpose:
Iransform raw Salesforce account records into clean staging tables.
Apply hezvy-duty cleaning:
format_phonel
enforce DECINAL precision on ravenue
Invalid recorda, nullify_ Heldd
fail
Guazantee: no mull in account_id, owaer_id; business fields standardized.
Lntermediato_
example: int_sallesforee httpaaoorrrr__cccooooooooeeeee
purpose:
Construct enriched account model with business logic.
staging tables user dimension Add derived fields (activicy_acoze, arcount_healtt).. per account".
hiezarchy into.
Note: Designed as reusable building black for multiple marts.
marts_leyer:
example: fet fot_salesforce__ saaeefooree__sllesppppp...e
puzpose:
Deliver pipeline fact table for neecllevel analytics & forecasting. Row grain opportuaity per reporting_date" Aggregate metrica: revenue, expected_valun, geighted__igge ine, cycle_time Attach dinens lens: zegion, incustry, owner, callenaaa Feeds dashboards, KPIs, and predictive modeling.
DE--Implemmenation:: Ground-truth DE project repository
SLaging account hwtp:aacotttttttttytocooooo stg
stg_selesforce__i
aeg_atlleforre__cotttt.. htt.:/oro___cttttt__sttrorroooo atg_aclesforro__comnt
httprrcc_vveetteoeeeee event.sgl, erg_asleeforra lead.
stg_selesforce.
stg_salesforce. wpwp_pppppoppttpppooeee iten. www.oporotttttyyyitttoor..
stg_ssleaforce opportunity.aq.
stq_aalestorce prroduct__.aqi.. stg sto_sallsforee_ttas task.
stg_stlesforce. utt.rcce___ser_eee_eeep_pe stg_salesforce user.sel
Intermedizte Leyer:
int int_saleetorce__ane account aeccount_enhncee..de int_salesferoe act wot.aaattttyyyaarrrrrra sql,
int_sclesforce date_ dat.__pise.__s.... int_sclesforse caaaa_ccoorrrioofuume.ccco
int int_saleaforeet ott__llletorre__opportttoooo wwtpoootnnty_agrreaeee ingregationn sngeenerrsgl,
int int_aslestorce pow.:ee__ppotttitt_pppepl...
int_ealestorce use: uet_re__sss_pperrrreeeeeeee
Mart Layer:
ret_salesforoe,
fot peweerooorr_eeeeeeeeererrreeoe salesforce_
cet_salesforce stwerrreeeeee_____e_____eeeeeeee
calesforce__ccountt_ooo history.stt.. ttteforree_contactt _otttattt_dilly__leeeee.ee
saleatores www.yyecttvityyy.ooooooootto
salesforce_
lllssforce__pportuuityy .nneanede... saiesforce ailesforee_.pperruittttL..
calasforce_ wttp___oeeeoeeeeeotrreroeeo sel, salesforce weeeanee_aaaleeeeeeee sattuoo_oolyiiisseeee.
sal.ee_sspphhttteeeee.o sql,, waw___ttttee_____eeee_ee Lean_pertormance.eg
C.3 DE-EVOLUTION TASK
This task evaluates an agent's ability to plan, surface complete requirements, and produce SQL. by adapting an existing SQL repository to revised business cesification-identffyyee scope and metric changes, updating definitions and dependencies, and delivering final, fit-for-purpose project that fully aligns with the new requirement.
24

---

Published as a conference paper at ICLR 2026
DE-Evvlution: Requirement Specifications
Business Pain Point:
Current appertunity managament lacka rebust cast-sffectivenent analysia.
Cannot measure acquisition cost, ma: nteeanne,,, and ROT conaiscently.
cojectt Ives:
Multi dimensional cost allccation (Lravei, marketing, labor, shared resoucces),
Lifecycle
natching (one-time, subscription, multi-year). Multi-scenerta RE1 analysis with sensitivity scanario modeling.
Impl implementation ion HighLights:
Flexible allocation
(time weighting, channel path, dynamic RCT logie revenue model (rolling 12M, discounted LTV, IRR).
Time based alignment of costs and zevenues.
quality checks imissing value fill, anemaly detection).
DE-Evolutions Ground-tru truth solution
Nodified sgu:
Int _nt__oooouuutttt ty pa...pie..gg neg
set__salee_pipe_ sel
:evenue_anelytice
fct "ett__acooune_eegpeemettt
Bnhancements in rct__salee_pppeel__
Added cost allocation fields (acquisition, travel, marketing, labozl.
Added ROT metrica trai_percentage, pest_per_dollal
LTV ratio).
Added revenue recognition fields (zevemue_ncdell recognition_oatless indicators (variance 8, ancmaly flag, level).
PV zevenue).
Added cost variance
Addad activity-leval cost breakdow (phone, email, macting, demo, proposal). efficiency ranking metrica (cost foot__fffiaeeny_tiee ineeatonnt_plorrtyyyye
C.4 DA TASK
In this section, we show the detailed classification the task types solved by DAComp-DA in
Tab. 15
D ERROR ANALYSIS
D.I DE-ARCHITECTURE ERROR ANALYSIS
Error distribution in de-arch tasks. The error analysis for the DE-Arch tasks reveals several chitectural shortcomings across the evaluated models, as shown Table [6] The models exhibit varying levels of function point omission, dependency errors, missing entity models, naming incon- sistencies, and improper model layering. Models such as 3wen3-8B 8B and @wen3-2355 -1n033223332222 demonstrate higher error rates across multiple dimensions, indicating more significant architectural flaws. In contrast, GP 5-5 and Gemini- 2 5-Pro perform relatively better, with fewer errors in dependency management and model structure, though they still show room for improvement, par- ticularly in entity model completeness and naming consistency.
ME-AArhitecture error case. As shown in Fig. 10 we present "DE-Arch Error Case" panel: the left side shows minimal blueprint, while the right side scores 16 checklist items (final score: 5/16), revealing several systemic weaknesses.
D.2 DE-IMPLEMENTATION ERROR ANALYSIS
Divergence in schema fidelity. The analysis of schema-level and Missing Column errors reveals distinct behavioral patterns in Table [7] Data Type emros remain consistently marginal (ranging from 2% to 7%) across all evaluated models, suggesting universal proficiency in handling fundamental SQL type systems. In stark contrast, Missing Column emors serve sharp discriminator of model capability: while SOTA models like GPT-5 achieve
Data Type
25

---

Published as a conference paper at ICLR 2026
Table 15: Definitions and Examples for the Five DA Task Type Categories.
Category Name
Definition & Objective
Example Question
Descriptive
Focuses on summarizing histor- ical data to answer "What hap- pened?". Involves calculating key metrics, identifying trends, and reporting the current state.
Analyze sales trends in the three categories of of- fice supplies. technology. and furniture from 2015 to 2018, identify the fastest-growing product category for each year, and evaluate performance differences among regional managers based regional sales data.
Diagnostic
Aims to uncover the root causes of a particular outcome, an- swering "Why did it happen?". Involves drilling down into data, identifying anomalies, and discovering factors that influ- ence result.
For the product category with the greatest annual volatility, investigate the underlying reasons. Then, RFM segmentation identify core consumers and assess their sensitivity to those drivers.
Strategic
Foeuses providing data- driven recommendations for fu- ture actions, answering "What should we do?". It translates in- sights from descriptive and di- agnostic analysis into concrete, actionable plans.
As the sales leader for Coca-Cola, which sales let types should increase or decrease our contracts with? Please provide recommendations based on an analysis of key data such as sales target attainment, customer complaints, and sales volume.
Pattern Recog- nition
Involves exploring data cover previously unknown rela- tionships, correlations, or pat- terns, answering "What the of-
Analyze the trends in the price per carat of dia- monds across different carat ranges, and also ex- plore the extent to which other factors impact dia- mond prices.
hidden connections?". (en open-ended and seeks to generate new hypotheses.
Profiling
Aims to group population (e.g.. customers, employees) into distinct segments based on shared characteristics, answer- ing "Who are they?". The goal is to understand the composi- tion and behavior of different groups.
Based comprehensive ranking that considers effective work hours, overall production quantity, and quality, please analyze the characteristics of top performers and recommend the ideal pro- file for fature hires.
Table 16: Detailed analysis for DE-Arch tasks.
Model
Function Polnt Omtecion
Dependency Errors
Missing Entlty Models
Naming Incoudstencies
Improper Model Layering
GPT-5
26.51
17.14
18.91
6.11
7.21
LSmini-22-PPro
27 22
19.35
20.64
8,53
9.16
Qwead Coder
30.56
22.26
24.33
11.19
12.14
DeqpSeal-V3.1
25
12.52
13.00
Owent-235B-A22
Qwen1-88
36.01
13.73
26

---

Published as a conference paper at ICLR 2026
DE-Arch Error Case
accounts, seomingly churning Strangely, thesa customers appear highly engaged system and activlly paying Strpp, yet Zendask viluully suppori lickets. need help building a comprehensive "silent customer warning syslem" can identily these seerningl healihy but potendialy problematic nustomers and predict thair potantial chum rask by analyping behavioral changes acmes systems.
nalkedd recent
days convarting
# ==zz= Incorrect (Minimal) Blueprint =
5/16
modeling spec:
purpose: "Delecl silent customers post-conversion."
v Clear business goal (1/1)
Deeign__rinccpttee
grain_ importance: "Declared but not enforced"
Nol enforced downstream (0/1)
windows: "pre-30d, post-31-60d"
Ambuguous boundaries timezone (D/1)
intermadiate. models:
marketo
source_modekt models:
waw.marketo leads
Valid source (1/1)
sal__customer380 nicknames
Dangling reference (0/1)
business logic: "Normalize emails
Missing fallbackledge handling (0/1)
columns:
lead id
X No tests (not, nulliunique) (0/1)
name: int_strppe
source models:
rrawsstrpee charges
source (1/1)
business logic: "Aggregate payments"
No refund//ulti-currenyy handling (0/1)
source_moste models:
inn stripe
zendesk
Dangling reference
business logic: "Match by emailiphone'
No breeholdaswwighht//tommaas (0/1)
models:
name: mart mart_silentriid
int matmhes
Proper mart structure (1/1)
business logic: "Compute risk score"
formula,, weights, normalization (0/1)
columns:
customer360_it id
column defined (1/1)
name:: risk, score
Metric placehoider only (0/1)
assumptions:
notes:
"Exports only"
Missing refresh frequency freshness (0/1)
Figure 10: DE-Arch error case. Key issues include: (1) clear business objective but weak down- stream enforcement, ambiguous boundaries, and no timezone convention; (2) dangling references in intermediate models, missing fallback and edge-case handling, and absence of basic tests such as not_nulll not_nul//unique; (3) no treatment for refunds and multi-currency scenarios; (4) missing thresholds, weights, and formulas aggregation and metric layers, with some fields provided by sources; (5) placeholder metrics only and no seffess-ffrquencyyffess/ee cating the need to harden constraints, validation, and business computations.
policy. Overall score: 5/16, indi-
27

---

Published as a conference paper at ICLR 2026
Table 17: Detailed error analysis for various models on DE-Impl and DE-Evol tasks.
Model
Data Type Errors
Missing Column Errors
Omission
Calculation Logie Errors
Dependency
DE-Impl Tinss
GPT-5
2.22
0.29
5.18
40.65
66.01
2.mini-2.5-PPr
4.25
5.74
15.17
37.58
67.31
Qwen3-Coder
5,54
1,38
22.88
36.91
66.13
DeepSea--3.1
37.45
65.68
QwenS-235B-A228
5.74
34.73
89.79
42,06
73,94
Qwen3-88
4.57
16.82
95.74
34.62
70.55
DE-Frol Taoks
GPT-5
2,09
10.05
11.69
28.93
56.45
Gemini-2.5
40.56
Qwen3-Coder
3.32
23.88
19.09
35.66
63.29
DeepSeck-V3.
2.46
34.00
31.87
59.23
Qwen3-235B-ADD
1.00
54.36
65.79
23.62
53.86
Qwen3-8B
0.85
44,06
58.17
27.88
53.44
near-perfect coverage (0.29% error rate), base models exhibit significant deficiencies (up 34.73%) This disparity indicates that while correct type inference is readily accessible, ensuring exhaustive field retention requires higher order of instruction-folloinng precision found primarily in top-tier models.
Dominance of dependency errors. The comprehensive emror analysis presented in Table 17 identifies Dependency Errors as the primary bottleneck constraining model performance in DE- Impl tasks. Regardless of model capacity-from SOTA models like GPT-5 smaller counter- parts- derteddeyyddccc error rates consistently exceed 65%. Further decomposition in Table 20 reveals a balanced distribution between "Missing dependencies" and "Extra dependencies." This equilib- rium suggests that current LLMs struggle construct accurate global data lineage graphs, lacking the precise contextual awareness required to manage long-range dependencies effectively within complex data engineering frameworks.
Complexity sensitivity in sqi omission. Analysis of SQL Omission exhibits significant stratifica- tion based model capability and architectural depth. As illustrated in Table 18 omission rates escalate as the data architecture evolves from the foundational Staging layer to the highly aggre- gated Marts layer, reflecting the penalty imposed by increasing business logic complexity. While weaker models (e.g., Qwen3-8B) suffer catastrophic failure in the Marts layer with omission rates approaching 100%, advanced models demonstrate superior robustness, maintaining omission rates below 10%, thereby highlighting a distinct capability gap in handling complex, multi-level data transformations.
Cascading effects in calculation logic. A granular examination of Calculation Logic Errors un- covers significant "error cascading effect" within data pipelines. As shown in Table 19 for high- performing models (e.g., GPT-5 and Gemini-2.5-PPre not faulty reasoning at the current node (Intrinsic Errors), but rather the propagation of inaccura- cies from preceding layers (Upstream Errors). For instance, GPT-5's upstream errors are approxi- mately three times more prevalent than its intrinsic errors across all layers. This insight implies that optimizing DE-Impl performance requires shifting focus from merely improving single-node code generation to enhancing the model's 's capacity for fault tolerance and consistency maintenance across the entire lineage.
the predominant source of calculation erross
DE-Implementation error case. It is crucial to prevent implementation issues such as improper joins, flawed aggregations, and circular dependencies. The cases in Fig. 11 and Fig. 12 serve as representative DE-Impl examples.
D.3 DE-EVOLUTION ERROR ANALYSIS
Contextual awareness in dependency management. The comparative analysis dependency er- rors illustrates that DE-Evol and DE-Impl tasks present fundamentally different challenges. While the overall error rate in evolution scenarios is lower than in construction tasks, the nature of failures shifts significantly. As indicated in Table 20 weaker models in DE-Evol exhibit pronounced bias towards "Missing Dependencies," contrasting with the more balanced enror distribution observed in DE-Impl. This suggests that preserving the integrity of an existing pipeline imposes specific
28

---

Published as a conference paper at ICLR 2026
DE-Impl Single-File-Level Error Case
project: Google Ads Data Pipeline
data contract.yaml
SELECT c.campaign name, s.spend FROM staging sig campaign hompaaignhistarnn JOIN staging.sig campaign ON c.acoount id = s.accounl_ id
staging:
history
sig campaign history:
campaign_namee status, updated at. is_msss s__most_recett receettrecord]
columns: [campaign
acc.ccccutt.. history:
columns: (account id, account_ name, timezone, updated is most recent record]
SELECT c.campaign name, s.spend FROM staging.stg. campaign history LEFT JOIN staging campaign stats ON c.campaign kd s.campaign_ie id
stg, campaign_stapl stats:
columns: [campaign_
day, spend, impressions,
clicks, conversions, conversions, valus]
modeling:
int wwl_ccamaagn__e darlignn_ddyyy____eiie
SELECT
depends_ on: [stg__aamggg_ sampaign_stats, stg www.caaaamhaatppoaooo int daily:
campaign id.
AVG(spend) daily avgg spend FROM staging stllsstggggaaaa wtwppaaaaaggaaaaaae stats GROUP campaign
depends on: ett_campeign_oee daily)
int cl_cmaaggig__ monthly:
depends_ on: [int campaign _dill] int monthly:
depends on: [int, account daily]
WITH dally_totalls totals AS SELECT
int metriettsccccmm canonical:
formulas:
campaign_kk id_
conversions, value nullifsspend, 0)
date_ day.
cpa: spend / nullf(convvesiios, 0)
SUM(spend) as daily_spend spend
clicks nullif(impressions,
FROM staging sig, sal_ccammaag_snae
GROUP BY campaign jd, date_ day
campaign__ performance:
SELECT
depends_ jint_ campaign monthly, int, metrimetttsmm ____micc__cnnnicaaa account Becuunt_eeffrmmnnoo
campaign_kk _id
AVG(daily_ _spnnd) as daily_avg. avg__aaag_eed
depends_ [int_ www.acccoottmtteeeeeee monthly, metrics, canonical]
FROM daily totals
GROUP BY campaign_
Figure 11: Examples of errors in DE-Impl: the red-crossed cases show mistakes such as joining on mismatched keys (account -id instead of campa .nd and incorrect aggregation without respecting daily granularity, while the green-checked cases illustrate valid implementations with proper joins and staged aggregation.
29

---

Published as a conference paper at ICLR 2026
DE-Impl Pipeline-Level Error Case
project: Google Ads Data Pipeline
data contract.yaml
intermediate/int, htteredditeeiit__cmaaiiiin cawpaaggn_perttttteeeee perf.sql
staging:
sig campaign history:
WITH base metrics AS
campaign_nane name, stalus, updated_at at is_ 3__most_rr recent__eeeeee record]
columns: (campaign id,
SELECT campaign _d. sum(spend) as total spend
sig, account history:
staging.ste stg.g.ssgg... cas__aammagg....ma stats
columns: [account account name, timezone. updated at, most recent_recond
GROUP campaign_id
campaign stats:
enriched AS
tolumns: (campaign id, date date_day, spend, impressions, clicks, conversions, conversions, value)
SELECT bm. peppeforrance. tier
FROM base m__rric bm
JOIN marts marts.campaign. summary ON bm.campaign id cs ww..paaaigg.... jd
modeling:
int wac.ammmmpnn.c.. _dily:
depends on: [stg. cal___ammaag___cc stats, stg_ ptl.ccammagggiieee
Int account dally:
SELECT FROM enriched
depends on: campaign_dania ddily]
int, campaign monthly:
depends on: [int campaign_danl _dall]] int account monthly:
Circular Dependency
depends_ on: [int account daily]
metrics canonical:
formulas:
conversions nullf(spend, 0)
opa: spend nullifconversions, 0)
ctr: clicks nullff(ipressiions 0)
SELECT
campaign_
marts:
avgitotal spend) spg_s,
campaign_ performance:
CASE WHEN avg(total, _spetad___dd______ > 1000 THEN 'high'
depends on: [int campaign_moann monthly, int _mterris_cannnicca]
account performance:
ELSE 'low' END performance, tier
depends_ on: [int_ account monthly, int_ mt_rriti canonical)
FROM internediate int, caappaagm_ppaee
GROUP BY campaign_it
Figure 12:
Examples of in DE-Impl: circular dependency in which
int ww..ggrf.g...... depends on campa gn_summary. sql, creating a loop in the data pipeline.
30

---

Published as a conference paper at ICLR 2026
Table 18: Detailed analysis for SQL Omission.
Model
Staging
Intermediate
Marts
Total
DE-Impl Tasks
GPT-5
4.14
3.66
9.37
5.18
Gemini-2.5-Pr
11.22
9.58
15.17
Qwen3-Coder
18.68
19.00
34,56
22.88
DeepSoek-Vi
23.91
22.73
42.23
28.58
Qwen3-255B-A228
84.26
91.74
96.81
89.79
Qwen3-8B
92.05
91.81
99.50
95.74
DE-Evol Tisks
GPT:5
8,99
15.34
11.69
Gemini-2.5-Pos
14.16
21.78
16.94
Owen3-Coder
15.08
23,47
DeepSeek V3.1
30.96
39.42
34,00
Qwen3-235B-AA22
Qwen3-8B
66.66
48.31
58.17
Table 19: Detailed performance analysis for Calculation Logic Errors in DE-Impl and DE-Evol tasks.
Model
Intermediaae
Uperream Ermors
Intrinsie Tórors
Total
Upstream Frrrss
Intriasic Eamors
Total
I Lpsttromm Errees
Intrinsle Frrurs
Retal
Di-rrpp Yosks
Ganini-2
7.21
DecpSeck-VV V3.1
9.53
25.77
11.68
37.45
2158 A22B
22.78
31.24
42.05
Qwad1-8B
42.86
42.25
36.22
75.48
74.34
10.52
$4.95
5.77
25.55
34.62
DR-Roo Tonkr
GPT55
10.91
18.54
29.45
20.29
12.68
33,07
14.97
1396
28.93
Gemini-2
0.00
Qwan-CCaker
11.64
DeepSeek 21---22---A2B
31.16
21
17.35
23.62
Qwen3-80 BBB
25.02
25.50
22.22
27.85
Table 20: Detailed analysis for Dependency Errors DE-Impl and DE-Evol tasks.
Model
Missing dependencies
Extra dependencles
Missing U Extra
DE-Impl Tisks
GPT-5
45.42
52.61
66.01
Gemini-2.5-Pro
51.02
50.44
67.31
Qwen3-Coder
47.92
50.81
66.13
DeepScek-V2 V3.1
48.53
65.68
Owen3-235B-A200
61.43
55.46
73,94
Qwen3-8B
$2,60
56.24
70.55
DE-Evol Tasks
GPT-5
39.15
39,50
56.45
Gemini-2.5-PM Pro
$2.49
42,87
64.98
Qwen3-Coder Coder
48.65
43.70
63,29
DeepSeck-V3.I
45.01
38.82
59.23
43.31
28.74
53.86
Qwen3-8B
46.92
20.88
53,44
demands on context retention, where limited-capacity models fail to identify the downstream con- sequences of schema changes.
Predominance of upstream error propagation in evolution tasks. The comprehensive error pro- file presented in Table I7 elucidates a fundamental distinction between the ab initio synthesis of Data DAGs in DE-impl and the structural preservation required in DE-Evol. While the aggregate dependency error magnitude is lower evolution scenarios (e.g.. GPT-5 decreases from 66.01% to 56.45%), the taxonomy of failures exhibits qualitative shift. As further detailed in Table 20 lower- capacity models DE-Evol display a marked propensity for "Missing Dependencies." a deviation from the more balanced error profile observed in DE-Impl. This asymmetry suggests that preserving the integrity of an existing pipeline imposes a distinct cognitive burden related to context retention, where models struggle fully trace the downstream ramifications of schema modifications.
Architecture-level attrition in file identification. Table I8 demonstrates a notable performance inversion regarding scope identification. Unlike DE-Impl where the scope is constructive, DE-Evol
31

---

Published as a conference paper at ICLR 2026
demands discriminative identification of files for modification. Surprisingly, for SOTA models like GPT-5, the rate of SQL Omission is higher DE-Evol (11.69%) compared to DE-Impl (5.18%). This data suggests that the discriminative task of identifying specific files for modification within large codebase presents greater challenge than the constructive task of pipeline generation. While base models struggle with the sheer complexity of DE-Impl, advanced models constrained by the precision required for impact analysis in DE-Evol, indicating that identifying the modification scope remains distinct bottleneck.
DE-Evolution error case. To illustrate how evolution errors can propagate across layers and distort downstream business metrics, we present a pipeline-level DE-Evol example in Fig. 13
DE-Evol Pipeline-Level Error Case
Layer Staging
staging/stg_anr _rcollgg_ccunn__gg history.sql
The same account_ id has multiple records with is_mostt_reeent true.
SELECT account
name. updated
account.ik
account name
mesett
ROW NUMBER() OVER (...)= as is ge_moottrreene
A001
Company X
FROM raw.account history
A001
Company
A002
Company Y
Irae
Erroneous downstream
Layer 2: Intermediate
intermediate/in, www.accccsummaaeaeee summary.sql
The spend for A001 was double-counted!
SELECT account id, a.account SUM(c.spend) as spend FROM staging sig. account, history a JOIN staging. stg. campaign_sste stats aaccount_id c.account WHERE a.is ses_mott__eeeeeeeeee rese___eeettt = true GROUP BY a.account id, a.account_nand name
account
account name spend
A001
Company
20,000
1,000
A002
Company
15,000
The erroneous data continues to propagate
Layer Marts
martslexecutive. hattt//eeccuiie_d_dhbbrrrrrr.r.
dashboard.ss
SELECT
company spend
avg spend
SUM(total spend) as company total spend, AVG(lotal spend) as avg account spend, COUNT(") account count FROM intermediate.e int. account summary
35,000
17,500
company total spend
spend
account count
25,000
12,500
Figure 13: DE-Evol pipeline-level enror case. Layer (Staging) contains duplicate "current" rows where the same account_id appears multiple times with is.most.recent true. Layer 2 (Intermediate) joins to campaign stats while filtering is.most_recent true, causing A001's spend to be double-counted (total.spend becomes 20,000 instead 10,000). Layer 3 (Marts) aggregates the erroneous intermediate table, inflating company total apend and avg account aaccouttsppatt -spend (35,000 and 17,500) compared with the correct values (25,000 and 12,500). The figure highlights how seemingly small staging inconsistency cascade into materially in- correct executive metrics.
32

---

Published as a conference paper at ICLR 2026
D.4 DA CASE
To illustrate typical failure modes DA tasks, Tab. Tab. [2 and Tab. 23 present focused case studies of Planning, Execution, and Interpretation errors, respectively. First, scoping lapse omit- ted required unstructured data, yielding a biased sample and invalidating all downstream analysis. Second, despite sound plan, a key metric was computed with incorrect formula (simple aver- age instead of weighted average), producing mislcading channel insights. Third, even with flawless calculations, the agent failed to synthesize findings into context-aware conclusion and omitted mandatory limitations and safety disclaimer. Together, these cases demonstrate that reliable DA outputs require aligned rigor across planning, implementation, and interpretation, with checks that prevent any single stage from compromising the whole.
Table 21: Focused case study of a critical Planning Error. This table analyzes the agent's plan against a pivotal standard (Data Scoping), highlighting omitted steps (in red) that led to fundamentally flawed analysis.
Required Planning Step (from Rubric)
Agent's Plan vs. Actual Action
Outcome
Case Study: Standard 1.1 Data Understanding Scoping
Step 1.1.A.1: Filter using the structured Education Requirement column.
The agent correctly planned and executed this step.
PASS
Step 1.1.A.:: Additionally extract candidates from the Job Description column.
CRITICAL PLANNING FAILURE: This step was entirely omitted from the agent's plan; it never considered searching this column.
X FAIL
Further apply complex filtering rules the Job description column.
Step 1.1.A.3:
CRITICAL PLANNING FAILURE: This
F
more advanced step likewise completely ab- sent from the agent's 's plan.
Consequence of the Flawed Plan: the agent analyzed an incomplete and biased sample (9,073 records instead of the correct 11,38).. thereby invalidating all subsequent analysis. This is a textbook Planning Error: once the initial strategy faulty, execution quality cannot rescue the outcome. Final score for this standard: 4.
By omitting two required data sources in the planning stage,
E ANNOTATION DETAILS
E. DATA COLLECTION
Data synthesis for de tables. Our DE tables originate from 73 enterprise-grade SaaS domains and their companion data-transformmtion projects, providing production-style schemas and realis- tie dependencies. Starting from a minimal business contract (target grain, primary/foreign keys, required metrics), we expand to end-to-end datasets and scale them while preserving business se- mantics and referential integrity. To keep the data mock both controllable and realistic, we high- light only the key steps: 1) Schema fidelity: retain PK/FK, uniqueness, not-null, and domain con- straints; 2) Distributions & dependencies: fit marginal distributions and model conditional links (e.g.,
imezone); 3) Temporal coherence: inject seasonality, trend, and holiday effects while maintaining fact-dimension integrity; 4) Noise & edge cases: introduce con- trolled missingness/outiirssyyyy coercions and design stressors that expose pipeline fragility (e.g., duplicate "current" rows, currency conflicts, timezone mismatches). The synthesis pipeline is im- plemented in Python (pandas, numpy, faker) with custom generators to scale volume while honoring inter-column dependencies and business invariants.
E.2 CONSTRUCTION DETAILS OF DACOMP-DD DE
This subsection presents our experience constructing the DAComp-DD -EE corpus. We outline an end- to-end process across three macks-Architetturr Implementation, and Eivouuion-sspan the
33

---

Published as a conference paper at ICLR 2026
Table 22: Focused case study of critical Execution Error. This table examines the agent's implementation for Standard (Channel Performance Metrics), illustrating an otherwise sound plan partially failed due an improper formula for a key metric.
Calculation Required by the Rubric
Agent Implementation vs. Correct Method
Outcome
Case: Standard 2.1
Channel Performance Metrics
Sub-standard 2.1.A.1:
The agent correctly used 'GROUP BY with / PASS SUM(sales.volumm))
Compute Sales Volume by channel.
Sub-standard 2.1.A.2: Compute Total Revenue by channel.
The agent conrectly used 'GROUP BY with / PASS SUM((tttal revenue)..
Sub-standard 2.1.A.3/4: Compute Average Unit Price by channel.
The agent treated unit price as simple average rather than revenue-weighted average. As a result, the reported average prices were incorrect and led to misleading conclusions about channel profitability.
CRITICAL EXECUTION ERROR:
X FAIL
Impact of the Execution Deviation: Although the agent's overall plan for channel analysis was sound, using the wrong formula for single critical metric (Average Unit Price) produced mis- leading conclusions about channel profitability, directly undermining any price-based strategic recommendation. This constitutes a canonical Execution Error. Final score for this standard: 5 6.
baseline derived from 73 enterprise grade SaaS domains and their date-transformation projects pure-SQL normalization and validation, high-level requirement setting for blueprinting, contract- driven realization into working SQL, and change-orients orientdd migration under realistic constraints. The summary reflects decisions and best practices agreed upon by domain experts ensure rigor, repro- ducibility, and evaluability.
E.2.1 CONSTRUCTION DETAILS OF CEEOMM--D--ACCHTEECCTEE
Baseline curation and normalization. We first select open-source dbt projects that license- compliant and empirically verified to be error-free, free. and normalize them into pure-SQL repositories by expanding materializations and macros while freezing model dependencies. Senior data engi- neers conduct a systematic audit of join semantics, analytical grains, window specifications, SCD handling, and testing assumptions, thereby establishing a high-quality baseline suitable for con- trolled evaluation.
High-level requirement formulation. Building on this baseline, we define task statements grounded in realistic enterprise scenarios: they provide only business context, overarching objec- tives, and expected outputs, without detailed metric definitions, precise calculation rules, or data constraint specifications. Such descriptions emphasize openness and cross-system characteristics, reveal gaps not covered by the existing repository, and intentionally avoid prescribing implementa- tion paths or technical details. The model is expected to autonomously plan blueprint-idenniffiinn key entities and dependencies, delineating layers and boundaries, and completing testing and fresh- ness strategies ultimately producing executable architectural blueprint that evaluates its ability to plan end-to-end SQL projects and set constraints under incomplete information.
E.2.2 CONSTRUCTION DETAILS OF DECCOM--DE-IMPLEEENTTNN
Contract formalization. Impl is constructed by deriving rigorous requirements specification from the vetted SQL baseline in the form standardized data contract. yaml that follows enterprise conventions. The contract formalizes model inventory and lineage, table and column
34

---

Published as a conference paper at ICLR 2026
Table 23: Focused case study a critical Interpretation Error. The table shows a stark contrast between the agent's successful execution of calculations and its failure to synthesize those results into meaningful, context- aware conclusion.
Analytical Stage (from Rubric) Case Study: Analysis of Students with Suicidal Ideation
Agent's Performance & Justification
Outcome
Stage 1: Execution Calculation (Standards 1.1 1.4)
The agent's 's plan was sound and its execution was flawless. It successfully filtered the correct data population and accurately calculated all required statistical metrics (e.g., average economic/academie pressure, lifestyle habit percentages).
PASS
Stage 2: Interpretation & Synthesis (Standard 1.5: Create "high-risk profile")
CRITICAL INTERPRETATION FAILURE: FAIL
The agent failed to synthesize the previously calculated statistics into coherent, higher-level insight. Instead of creating "profile." the agent merely listed the numbers again. The judge noted the summary was "not deep enough" and "merely restated the table's content."
Stage 3: Contextual Understanding
CRITICAL INTERPRETATION FAILURE:
X FAIL
The agent's final output completely omitted the mandatory "Limitations and Safety Disclaimer." failure to understand the
(Standard 2.2: Provide safety disclaimer)
This demonstrates serious and sensitive context of the topic, which is key part of providing responsible and complete analytical deliverable.
Consequence of Flawed Interpretation: This exemplifies pure Interpretation Error. The agent acted as perfect calculator, producing correct data (Stage 1). However, failed at the final and most critical stage: transforming that data into a meaningful, insightful, and contextually appropriate conclusion (Stages 2 & 3).
35

---

Published as a conference paper at ICLR 2026
schemas with constraints, declared grains and time windows, metric definitions with coherent units and currency normalization, as well as data quality, freshness, and performance policies.
E.2.3 CONSTRUCTION DETAILS OF DACOMP-DE-EVOLUTII
Change specification. For DE-Evol, start from high-quality, production-style SQL repository and propose change requests driven by realistic enterprise pressures-such as revised metric defini- tions, altered analytical windows, schema drift, or governance hardening. Multiple experts specify unambiguous business semantics, distinguish breaking from non-breaking changes, and design safe migration plan that anticipates dependency revisions and testing upgrades.
E.3
ANNOTATION DETAILS OF DACOMP-DA
In this section, we present the experience regarding the annotation of DAComp-DA data, which is summarized from our previous project discussion meetings and alignment meetings.
E.3.1 CORE DESIGN PRINCIPLES
Strategic diversity The core of the Rubric is to evaluate problem-solving strategies, not steps. Each scoring Path must represent a methodologically distinct and self-contained solution. We avoid designing complete versus abridged versions of the same path. For example, analyzing all provinces and analyzing a subset of provinces should not be two separate Paths; the latter is merely an incom- plete execution of the former.
Objective evaluation Scoring criteria must be quantifiable and reproducible to minimize scorer subjectivity. All items should be based on explicit evidence. Guideline: Any Accuracy item requir- ing numerical verification must have pre-calculated Anchor Value. For open- open-ended paths without single correct answer, Pseudocode or a clear methodological verification process must be provided.
Dimensional separation of abilities Complex analytical skills are decomposed into independent scoring dimensions for fairer and more granular assessment of model performance. Guideline: Strictly distinguish between procedural execution (were the steps completed?), computational ac- curacy (were the numbers correct?), and insightful conclusion (was the interpretation meaningful?), designing them separate scoring items.
E.3.2 STRUCTURAL COMPONENTS OF THE RUBRIC
The Rubric employs a four-level hierarchical structure to deconstruct tasks, ensuring comprehensive and granular evaluation.
Definition: The highest-level objective of the task, directly corresponding to a core analytical request from the user. Example: Analyze the differences in employee attrition rates across departments and their causes.
Requirement.
Standard. Definition: key analytical step that must be completed core conclusion that must be reached to fulfill a Requirement. Example: Standard 1: Calculate and verify the attrition rate differences between departments; Standard 2: Identify the key factors causing these differences.
Path. Definition: A methodologically distinct and valid strategy for meeting a Standard. This is the core of the Rubric's s design. Example: Under the standard of verifying differences. Path A could be performing statistical significance test (e.g., Chi-squared test), while Path B could be making descriptive statistical comparison (e.g., percentage difference).
Sub-standard / rubric item. Definition: The smallest scorable unit of the Rubric, nested under a specific Path and adhering strictly to the principle dimensional separation. It comprises three main types:
Completeness: Assesses whether all required steps for given Path were executed. Focuses what was done.
36

---

Published as a conference paper at ICLR 2026
Accuracy: Assesses whether the computational results execution process correct. Focuses on if it was done correctly. For deterministic paths, this is verified against an Anchor Value; for open-ended paths, it is verified against methodological process or Pseudocode.
Insightfulness: Assesses whether a reasonable and valuable conclusion or insight was de- rived from the correct results. Focuses on if the results were understood.
E.3.3 GOLDEN RULES FOR AUTHORS
These are the disciplinary requirements to ensure the quality and consistency of the Rubric. While these guidelines ensure consistency in creating rubries for known strategies, the following section details our methodology for fairly evaluating novel or unanticipated solutions that may not align with pre-enumerated paths.
Calculate first, then author. Before finalizing the rubric, authors must personally run the com- plete analysis with code to calculate all Anchor Values required for the Accuracy assessment. This the comnertonn of ensuring objective scoring.
Be specific and unambiguous. Every statement in the Rubric must be directive and unambiguous. Avoid subjective terms like approximately, good, or relatively comprehensive to minimize scorer discretion.
Avoid zero-point paths. If method is not worthy of credit, should be designed distinct Path. A model's output that does not match any valid path will naturally receive no score for that standard.
F LLM USAGE DETAILS
In compliance with ICLR 2026 policies on large language model usage, we disclose that LLMs are mainly used for three purposes in this work:
LLM Evaluation: The core of this work is the systematic benchmarking of various large language models to assess their performance and capabilities as data agents.
LLM-based Judging: For the open-ended tasks in our benchmark, we employed an LLM automated judge score agent responses based
detailed, expert-designed rubric.
Writing Assistance: We utilized an LLM to assist in polishing the manuscript by refining grammar, improving phrasing, and enhancing overall clarity.
All L.M outputs were subject to careful human oversight and validation. We take full responsibility for the accuracy and integrity of all content in this paper, including any sections enhanced with LLM assistance.
G DISCUSS
G.1 DISCUSSION HANDLING UNENUMERATED SOLUTION PATHS
Accuracy is the most critical dimension in our rubric. Since fully listing all valid analytical paths often infeasible, we adopt three-tier. progressively relaxed design for Accuracy: (i) direct enu- meration with numeric anchors when the correct outcome can be exhaustively determined; (ii) con- strained computation with pseudo-code anchors when procedures are well-defined but paths are not exhaustively enumerable; and (iii) principle-based assessment for highly open-ended cases.
Standardized assessment for common paths. We standardize scoring whenever we can verify correctness deterministically. Tier-l (numeric anchors): for tasks whose outcomes be exhaus- rively enumerated, we embed the reference value directly into the rubric (e.g., "How many users satisfy condition X?"), yielding absolute, reproducible checks. Tier-2 (pseudo-code anchors): for
37

---

Published as a conference paper at ICLR 2026
tasks with well-specified computation but multiple equivalent derivations (e.g., conversion rate with alternative weighting schemes), we prescribe canonical steps in pseudo-code to constrain the procedure. This enables process-level verification (inputs, ordering, aggregation, null/cdge han- dling) without enumerating every path, preserving both precision and reproducibility.
Principle-based assessment for novel paths. A minority of tasks are intrinsically open-ended, where enumeration pseudo-code templating is impractical. Here we evaluate Accuracy via methodological principles rather than single anchor value. For example, "key-driver identifi- cation" task may be solved by regression with coefficient interpretation (a pre-defined path), or by gradient boosting with SHAP attributions (an unenumerated path). We score such solutions on: (1) Methodological appropriateness (the method is suitable for the stated objective and data regime); (2) Correctness of execution (the pipeline is implemented soundly, with valid preprocessing, estima- tion, and validation); and (3) Soundness of interpretation (claims follow from the produced evidence, with clear caveats). This soft layer ensures valid but unconventional approaches are not penalized.
By construction, most DAComp items fall into Tiers 1-2, where numeric pseudo-code anchors provide deterministic checks; Tier 3 reserved for genuinely open-ended cases to maintain fairness without sacrificing rigor.
G.2 DISCUSSION OF AMBIGUOUS REQUIREMENTS
Implementation and Evolution tasks in DAComp-DE are designed deterministic evaluations. To balance realism with unambiguous executability, we adopt three principles:
1) Professionalism. Requirements are sourced from enterprise-style projects and vetted by senior data engineers for cross-layer impact, metric definitions. SCD handling, and temporal semantics. Implementation tasks emphasize canonical modeling pipelines from scratch; Evolution tasks mirror real "change requests" (e.g., metric revision, source replacement).
2) Unambiguity. Implementation (node-first): each SQL node has atomic contracts (schema, PK/- grain, time, nulls, joins, aggregation, SCD, idempotency). Multiple agents must converge un- der frozen contracts; discrepancies trigger tighter specifications. Evolution (delta-first): natural- language changes are mapped into minimal verifiable deltas (schema/logic/ineeag) with explicit impact scope and before-after anchors; agent disagreement leads to refined deltas or explicit as- sumptions.
3) Realism. Implementation: converged nodes are composed into multi-node tasks, with con- tracts and assumptions documented (c.g., data-cont ract yam1). Evolution: favors backward- compatible evolution (added columns/views, metric versioning); destructive changes require migra- tion notes. All assumptions logged for reproducibility.
G.3 DISCUSSION ON THE SELECTION JUDGER LLM
As shown in Tab. 7 both O4-Mini and gemini-2..-flast achieve human-level agreement, while stronger proprietary models (e.g., gemini-2.5-pro. GPT-5) yield even higher consistency. For DA- Comp., we standardize on gemini-2.5-flass as balances (1) cost efficiency for large-scale bench- marking, (2) stable and low-latency inference, (3) reproducibility across runsand (4) community accessibility. Choosing a widely available model ensures that our evaluation pipeline can be easily adopted, verified, and extended by others.
G.4 DISCUSSION ON END-TO- FEN EVALUATION
Current DAComp tasks span complementary stages of the data intelligence lifecycle: DE- Architecture (high-level specification and planning), DE-Implementation (multi-layer pipeline con- struction), DE-Evolution (safe modification under requirement changes), and DA (open-ended analysis over downstream data). Taken together, these stages delineate a strictly end-to-end pro- cess from requirement articulation, through system realization and iterative evolution, to analytical insight and decision support-coverrng a full loop from planning and implementation to evolution and interpretation.
38

---

Published as a conference paper at ICLR 2026
At present, we evaluate these stages modularly and in decoupled fashion to enable controlled measurement at each step. Our next key objective is to integrate them into a single, end-to-end longitudinal evaluation: single agent carries requirements through implementation and change propagation, and ultimately completes analysis and reporting. We contend this end-to-end setup offers substantial scientific and practical value: it stress-tests the end-to-end consistency of plan- better reflects real engineering workflows, and advances toward comprehensive assessment of autonomous data agents' end-to-end capabilities.
39


Reviews:
[{"review_id": "Wqlg9yA2KQ", "reviewer": "Reviewer_s2Uo", "summary": "This paper introduces DAComp, a comprehensive benchmark designed to evaluate data agents across the entire data intelligence lifecycle, spanning both repository-level data engineering and open-ended data analysis. The benchmark defines 236 tasks reflecting realistic enterprise workflows, combining deterministic engineering evaluations (execution-based) with open-ended analytical assessments (LLM-judge guided by hierarchical rubrics).", "questions": "see Weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- DAComp is the first benchmark to unify data engineering and analysis within a single evaluation framework. \n\n- The benchmark construction pipeline is rigorous.\n\n- The paper is dense but logically structured.", "weaknesses": "- While the LLM-judge method is well-validated, DAComp’s rubric framework is static. Whether adaptive rubric refinement is required?\n\n- The experiments show low success rates but do not deeply isolate why orchestration fails? \n\n- The benchmark assumes one-shot or fixed-turn interactions. Yet many enterprise agents operate iteratively. DAComp currently lacks tasks or metrics reflecting closed-loop self-correction, which might undervalue agents with strong iterative reasoning skills.\n\n- CS/CFS/SR performs strict schema and data equivalence checks. It is not tolerant of irrelevant equivalences (such as equivalent rewrites or minor floating-point differences), and may incorrectly flag semantically equivalent results as errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V5PrpobSqN", "reviewer": "Reviewer_uR7M", "summary": "The paper introduces DAComp, a benchmark suite intended to evaluate autonomous data‑agent systems across the full data‑intelligence lifecycle. DAComp comprises 236 tasks split into two families: (i) DE (repository‑level data‑engineering) tasks that require agents to design, implement, or evolve multi‑stage SQL pipelines on large, enterprise‑style schemas, and (ii) DA (open‑ended data‑analysis) tasks that ask agents to produce reports, insights, and recommendations. The authors propose a mixed evaluation protocol: deterministic DE tasks are scored with three execution‑based metrics (Component Score, Cascading‑Failure Score, Success Rate), while DA tasks are judged by an LLM‑based evaluator using hierarchical rubrics and a Good‑Same‑Bad (GSB) component. A large experimental study compares a range of proprietary and open‑source LLM agents (GPT‑5, Claude‑4‑Sonnet, Gemini, DeepSeek, Qwen3, etc.) under two agent frameworks (a ReAct‑style baseline and OpenHands). The authors also validate the LLM‑judge against human annotators and report benchmark stability over multiple runs.", "questions": "1. How do the DA scores change if a different LLM (e.g., GPT‑4.1 or Claude‑3) is used as the judge? Have you measured inter‑judge agreement?  \n2. Can you provide an ablation study varying α (e.g., 0.5, 0.7, 0.9) and report its impact on model rankings?  \n3. What is the distribution of difficulty (e.g., number of nodes, LOC) across DE tasks? Are the results driven by a small subset of very hard tasks?  \n4. Have you measured human performance on a sample of DE and DA tasks to contextualize the reported model scores?  \n5. Which random seeds, library versions, and hardware configurations were used? Are the execution scripts deterministic?  \n6. How were the synthetic data distributions validated against real enterprise datasets? Could you provide quantitative similarity metrics (e.g., column‑wise KL divergence) to demonstrate realism?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Holistic scope** – By covering both repository‑level engineering and open‑ended analysis, DAComp fills a clear gap in existing agent benchmarks that usually focus on isolated code generation or single‑turn QA.  \n2. **Realistic task design** – Tasks are built from 73 permissively‑licensed enterprise SaaS schemas, with synthetic data that respects realistic column distributions, referential integrity, and edge‑case noise.  The DE‑Impl/Evol tasks involve multi‑file, multi‑layer pipelines (≈ 4600 LOC), which is substantially larger than prior data‑science benchmarks.  \n3. **Rigorous evaluation methodology** – The three‑tiered execution metrics for DE provide fine‑grained diagnostics (component vs. pipeline level).  The hierarchical rubrics for DA are carefully constructed, with multiple valid solution paths and a validation study showing high LLM‑judge/human agreement (Pearson ≈ 0.80, κ ≈ 0.65).  \n4. **Comprehensive analysis** – The authors present detailed error‑type breakdowns (planning, execution, interpretation) and relate performance to concrete properties such as node count, line count, and turn‑count distributions.  \n5. **Potential impact** – By exposing the engineering‑orchestration bottleneck, the benchmark can steer future research toward better planning, tool‑use, and pipeline consistency, which are crucial for socially relevant domains (healthcare analytics, climate data pipelines, sustainability reporting).", "weaknesses": "- While the authors report strong correlations with human ratings, the statistical choices and reporting could be more rigorous. Using Pearson r on rubric sums (often ordinal/heterogeneous across tasks) is suboptimal; intraclass correlation (ICC) or Kendall’s tau for ranking, plus confidence intervals, would be preferable.\n- Weighted κ=65 for GSB indicates only moderate agreement; the paper calls overall alignment “high” but should temper claims or provide confidence intervals and calibration plots.\n- The judge model (Gemini-2.5-Flash) belongs to the same family as one of the evaluated models (Gemini-2.5-Pro). Although Flash≠Pro, the potential for family-specific biases remains. The paper presents alternative judges, but a full ranking-stability analysis across judges is not reported.\n- GSB depends on five baseline reports created from multiple LLM outputs; selection of baselines may anchor judgments. The process to ensure these baselines neither overfit to a model family nor penalize stylistic variance is under-specified.\n- For principle-based (Tier-3) cases, safeguards to prevent over-crediting fluent but methodologically weak responses rely on judge prompts and principles, but inter-rater checks on these specific cases are not separated and reported.\n- Token budgets, tool-call limits (200), timeouts (120s per action), prompts, and system settings are critical for agent performance but are only briefly summarized. A stricter ablation on these constraints (and their fairness across models) would clarify whether some models are disadvantaged.\n- SQL dialect and tolerance: The evaluation requires exact schema+data equality, but it is unclear if numeric tolerances, null-ordering, and type coercions are handled consistently; small floating-point or timestamp discrepancies can introduce false negatives.\n- Contamination risk: Some DE tasks originate from open-source dbt projects; without a contamination analysis, models might have seen parts of the ground truth or closely related code. The synthetic data helps, but existence of public repo code raises a nontrivial risk.\n- For DA tasks, stability is reported on a subset (DE-Arch and DA) across 8 runs for a few models; confidence intervals and significance tests for between-model differences are not provided.\n- For DE deterministic tasks, results appear single-run; while determinism reduces variance, reporting robustness across multiple seeds/decoding settings would be informative for agent frameworks that involve LLM sampling.\n- Some interpretations could be more cautious: “Strong evidence that engineering and analysis are distinct capabilities” is supported by rank inversions, but causality is not established. Alternative explanations (prompt specializations, token policies, execution tool robustness) could contribute; a cross-task, cross-prompt ablation would strengthen this claim. “Even the top agents barely reach 50%” is accurate for aggregate scores, but the consequences for practical utility would benefit from error cost analysis (e.g., what classes of failures materially change business decisions vs. minor mismatches).\n- While the paper includes a brief LLM‑usage disclosure, it does not discuss potential risks of releasing synthetic enterprise data (e.g., inadvertent leakage of real company patterns) or the impact of using proprietary LLM judges on reproducibility for the broader community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5d7l8VdaDx", "reviewer": "Reviewer_cC4w", "summary": "This paper introduces DAComp, a comprehensive benchmark designed to evaluate LLM-based data agents across the entire enterprise data intelligence lifecycle, encompassing both data engineering (DE) and data analysis (DA) tasks.\n\nDAComp includes 236 tasks divided into repository-level engineering (architecture, implementation, and evolution) and open-ended analysis tasks that require reasoning, planning, and synthesis of insights. The DE tasks are evaluated through execution-based metrics (Component Score, Cascading Failure Score, Success Rate), while DA tasks are judged using a validated LLM-as-judge framework with hierarchical rubrics and GSB scoring.\n\nThe paper presents extensive experiments across leading models (GPT-5, Claude-4, Gemini-2.5, DeepSeek-V3.1, Qwen3, etc.), showing that even state-of-the-art agents perform poorly (<50 % overall, <20 % DE success rate). The authors conclude that current LLM agents lack holistic orchestration and open-ended analytical reasoning.\n\nContributions:\n\nFirst benchmark to jointly evaluate repository-level engineering and open-ended analytical reasoning.\n\nNovel hierarchical rubric framework for open-ended evaluation with verified human–LLM agreement.\n\nLarge-scale empirical study exposing the limits of current LLM data agents.", "questions": "1- How does the cost of LLM-judging scale with benchmark size, and can smaller-scale variants be used for quick evaluations?\n2- Could the authors clarify whether the hierarchical rubrics generalize to unseen tasks, or are they manually constructed per task?\n3- How is task realism maintained when synthetic data are used—were human engineers asked to validate their authenticity?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: Defines a new, holistic problem space combining engineering and analysis in realistic data workflows.\n\nMethodological rigor: Combines automatic and rubric-based evaluations, validated with human judgments.\n\nScale and realism: Repository-level tasks (>4k LOC) and open-ended business questions reflect real enterprise workloads.\n\nComprehensive evaluation: Benchmarks diverse models, offering insights into distinct skill domains (engineering vs reasoning).\n\nActionable findings: Identifies orchestration and multi-step reasoning as key bottlenecks for next-generation data agents.\n\nExcellent presentation: Clear structure, detailed analysis, strong alignment with related work.", "weaknesses": "Limited exploration of agent learning improvements: The paper benchmarks performance but does not propose or test training strategies to overcome observed limitations.\n\nRestricted open-source availability: While the paper claims data/code release, the double-blind setup prevents verification; explicit examples of released task formats would strengthen reproducibility.\n\nEvaluation cost: The LLM-as-judge pipeline, though validated, may pose high computational costs for community replication; approximate or lightweight scoring alternatives would be valuable.\n\nSlight imbalance between DE and DA sections: DE is analyzed more deeply than DA in terms of ablations and case studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)