Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agents
Authors: Irene Testini
Lorenzo Pacchiardi
Jose Hernandez-Orallo
Venue: Accepted by TMLR ()

Abstract: Data science aims to extract insights from data to support decision-making processes. Recently, Large Language Models (LLMs) have been increasingly used as assistants for data science, by suggesting ideas, techniques and small code snippets, or for the interpretation of results and reporting. Proper automation of some data-science activities is now promised by the rise of LLM agents, i.e., AI systems powered by an LLM equipped with additional affordances--such as code execution and knowledge bases--that can perform self-directed actions and interact with digital environments. In this paper, we survey the evaluation of LLM assistants and agents for data science. We find (1) a dominant focus on a small subset of goal-oriented activities, largely ignoring data management and exploratory activities; (2) a concentration on pure assistance or fully autonomous agents, without considering intermediate levels of human-AI collaboration; and (3) an emphasis on human substitution, therefore neglecting the possibility of higher levels of automation thanks to task transformation.

Published in Transactions on Machine Learning Research (10/2025)
Measuring Data Science Automation:
A Survey of Evaluation Tools for AI Assistants and Agents
Irene Testini* Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK
st370@com.ace
José Heraández-Orallo
jorallo Cuppe.es
Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de Valéncia, Spain
Lorenzo Pacchiardi*
1p666@com.ac..u uk
Leverbulme Centre for the Future of Intelligence, University of Cambridge, UK
Reviewed OpenReview: ht tp en net /f 3 d= fL
Abstract
Data science aims to extract insights from data to support decision-making processes. Re- cently, Large Language Models (T.LMs) have been increasingly used as assislants for data science, by suggesting ideas, techniques and small code snippets, or for the interpretation of results and reporting. Proper automation of some data-science activities is now promised by the rise of LLM agents, i.e., AT systems powered by an LLM equipped with additional affordances- as code exccution and knowledge bases that can perform self-directed actions and interact with digital environments. In this paper, we survey the evaluation of LLM assistants and agents for data science. We find (1) a dominant focus on a small subset of goal-oriented activities, largely ignoring data management and exploratory activities; (2) a concentration on pure assistance or fully autonomous agents, without considering intermedi- levels of human-AI collaboration; and (3) an emphasis on human substitution, therefore neglecting the possibility of higher levels of automation thanks to task transformation.
1 Introduction
Large Language Models (LLMs) (Brown et al. 2020) and their multimodal extensions first caught public prominence by powering capable chatbots that are now widely used as assistants to humans in several tasks, such as summarising documents (Liu et al. 2023ch, performing translations (Zhu et al. 2023), and creating code snippets (Guo al., (023)). These LLM assistants take instructions from a human the form of prompt and return answer, with the human retaining control over planning and decision-making by determining the sequence of actions to follow and deciding how much to rely the assistant's output; while they can use tools, their usage is generally prescribed at certain steps. Now, attention is increasingly dedicated to "LLM agents" (Wang et al. 2024) that can autonomously and iteratively decide sequence of actions to take and repeatedly interact with an external (digital) environment, being equipped with affordances and tools such as code execution (Huang et al. 2024b), internet access (Zhou et al., (2044), knowledge bases (Chen al. 2024), and operating system control (Liu al. 2023a),, which the agents can decide to use of autonomously.
In this paper focus on how assistants and agents for data science applications are being evaluated. Our aim is to provide overview of current evaluation set ups, and highlight the gaps in scopes of evaluation, rather than reviewing the performance of the models themselves. Data science is the process of handling and analysing data to extract insights that support decision-making science, business, or other contexts.
"Equal contribution.

---

Published in Transactions on Machine Learning Research (10/2025)
Data science may deal with different modalities of data (tabular, images, audio, etc.) but it always involves writing and interpreting information represented as text, whether in the form of code or natural language, and processing images, such as to present information or models. This dominance of textual modalities, combined with the vast amount of relevant online material that LLMs loverage during both training and inference, makes data science well-suited for automation by LLMs. Indeed, since the early days of LLMs (Chandel ct al. 2022), LLM assistants and agents have been used in data science applications. In this work, we survey evaluation tools measuring performance on tasks across the data science pipeline, offering, to our knowledge, the first comprehensive review of LLM evaluation for core data science taske]]
Data science is highly multidisciplinary and involves a breadth of activities, combining fields such as statistios, machine learning, and data engineering with tasks such as understanding business needs, writing reports, and preliminary research. Therefore, to compare the data science automation evaluation tools in our survey, we adopt the data science task taxonomy of Startnnez-Plumee et al. (2019), which expands the traditional CRISP-DM framework to include activitics related to data management and exploratory analysis (see Fig. [ reproducing Fig- 3 from Martinez-Plumed et al. 2019). and classify each evaluation lool by the activities it requires subjects to perform and explicitly evaluates. Moreover, in our survey we specifically consider the level of autonomy cach evaluation targets-whether the subjects are agents, assistants, or intermediate forms, such as LLM agents operating under close human supervision that correct their actions as needed. Further, we also analyse the way in which the tasks are framed and evaluated, to understand if they measure the ability of Al systems to simply substitute humans or if instead they consider that the Al system can transform the task in deeper ways, such as by bringing functional improvements (for instance, LLMs may not need to produce visualisations to perform data exploration; see Sec. 2.1
It is worth stressing how, in our work, we do not overview developments in state-of-the-art LLM assistants or agents for data science (referring interested readers to other works, Sun et al.) 2024) nor analyse the current performance of such systems across the various data science activities and autonomy levels. While such efforts would be valuable and may reveal strong corrolations between LLM systems' performance on specific lasks, our work addresses a more foundational need: to establish laxonomies across which data science evaluations can be structured and to identify gaps in the evaluation landscape that must be filled before comprehensive evaluation of LLM abilities can be carried out. Thercfore, by focusing on activities and autonomy, make the following findings:
Most evaluation works individually target (small) subset of data science activities (Tables 2 and 3); a few works cover multiple activities (Secs. 4.2 and 4.3 but none cover all of them. Taken to- gether, the surveyed works cover the landscape of data-science activities in biased fashion, chiefly over-representing "goal-oriented" activities, such as data preprocessing, producing plots in specified formats, or building predictive models for predetermined targets. This prevents the identification of correlations between AT systems' performance on different activities. Only a few studies (Cheng let al.) 2023 Sahu et al., 2025 Majumder al., 2024) give prominence to open-ended, exploratory as- pects of data science (such as interpreting client needs within business context, creatively exploring datasets and proposing potential uses) or data management Yu et al. 2018 2019b
Lei et al. 2024
Most studies focus either on assistants following human- defined actions or fully autonomous agents, overlooking more realistic scenarios of intermediate L.MM-human collaboration (also referred to
as "centaur evaluations", Haupt & Brynjolfsson 2025 Exceptions include Li et al. (025cc
(agents with simulated human users) and Yu et al. (2019a) (assistants aiding users in clarilying data tasks). Due to this near-binary focus, we organise the surveyed works according to the focus on assistants (Sec. g) and agents (Sec. 4h, highlighting when they assess intermediate autonomy.
More than half of the evaluations we survey implicitly assume that AT will substitute humans without functionally changing the tasks, either in assuming the steps by which a task is solved
are the same a human would follow (Yu el al. 2019a Zhang el al. 2024b: Chen el al. by scoring the task referring to human- human-producea produced output, despite there not being a single ground
2024) or
While many studies assess LLMs on general coding tasks (Jimenez al. 2024 and planning (Valmockam al.
2023 our
survey focuses specifically on evaluations within the data science domain. This includes works that target data science-specific roding as well as other data science activities that are often overlooked in existing evaluations.
2

---

Published in Transactions on Machine Learning Research (10/2025)
truth (Song al. 2025 Huang et al. 2024b Hu et al. 2024 Jing et al. 2024 Chen et al. 2024
Examples of works also rewards agents that functionally transform the task to solve it by scoring the final output are Pietruszka et al. 20024);
Cheng et al. (2023) Li et al. (022bb); Chan et al.
(024] ; Lei ct al. (024)) Majumder et al. [0024]]] Sahu et al. (025
The paper is structured as follows: Sec. discusses some fundamental concepts on technological transforma- tion, assistance and autonomy, LLM evaluation, and data science automation and its activities. Secs. 3 and
dive deep into works evaluating LLM assistants and agents, respectively. Sec. Dec.Ee summarises the challenges in data science evaluation and suggests future directions for more effective and comprehensive evaluation.
2 Background and related work
2.1 Levels of technological transformation
A naive perception of how tochnology transforms processes and activitics is automation by substitution: a human performing lask is replaced by machine without functionally transforming the task. To evaluate this, sample of tasks representing how humans tackle an activity is collected and the machine is tested on them (Eriksson et al. 2025). The wwtpttutoooonggmmmtttttoooooiiio inlffcattionReedeinittton (SAMR) model (Puentedura] 2006; Hamilton et al. 2016) identifies substitution as the lowest level of transformation and outlines su subsequent levels with progressively greater degrees of transformation: augmentation, where the ma- chine substitutes the human with some functional improvement; modification, where the task is significantly redesigned to allow automation; redefinition, in which the whole activity is redesigned, even creating new tasks. Much of the debate around A AT-powered automation focuses on the two bottom levels (augmentation and substitution), but the real penetration of AI technology is happening at the top levels of modification and redefinition (Brynjolíssoni 2022 Brynjolfsson et al. 2025), which hold the potential to achieve higher levels of automation. Indeed, we did not "automate away the jobs of lamplighters by building robots capa- ble of carrying ladders and climbing lampposts" " (Frey & Osborne 2023). Importantly, activities that have already been substituted can be iteratively transformed further as technology improves. Evaluating progress is therefore much more complex than if substitution was the only force at. play: a robot substituting human lamplighters in carrying and climbing ladders would have scored highly in turning on gas lamps, but the rodefinition afforded by electricity led to automating street lights, rendering robotic lamplighter needless. Similar considerations apply when evaluating AI progress in aulomating complex processes composed of many activities, such as data science: for instance, LLMs may not need to produce high quality visualisa- tions to perform successful data exploration, as they may be able to directly interpret large tables of data, To effectively evaluate modification and redefinition, AI evaluation should allow AI systems to perform activities differently from humans by rewarding the achievement of broad objectives.
2.2 Assistance and autonomy
Related yet orthogonal to the SAMR model is the distinction between assistance and autonomy Shneider- man 2020): in an assistive situation, a human uses the technology while retaining control of the process and only having some well-defined parts automated or improved. Assisted driving or writing are good examples: the process becomes more efficient and safe because of the use technology. On the contrary, in an au- tonomous situation, the technology performs the task independently and has more freedom choose how. Of course, autonomy exists on spectrum: intermediate levels include, for example, technology operating inde- pendently while human oversees the sequence of steps and retains the ability to halt operations. In relation to this, Cihon et al. 2024 defined levels of agent features relevant to autonomy. Their classification assigns high autonomy to agents acting fully autonomously, whereas the intermediate and lower levels correspond to agents consulting humans either at termination or at each step. This aligns with our understanding of autonomy levels, which additionally includes an even lower level where a human assigns a specific task to an assistant. Hence, holistic AI evaluation should take into account quality of the result and the level of human labour, which limits the impact of the technology in the long term. Importantly, for all levels of autonomy, the technology can perform the task in way that places the automation at any level of the SAMR hierarchy.
3

---

Published in Transactions on Machine Learning Research (10/2025)
2.3 LLM evaluation
The area of AI evaluation (Burden al. 2025 mostly relies tasks encapsulated in input-output bench- marks with a reference output for each example. For LLMs, these input-output pairs are most often Q&A examples used to evaluate assistants (Chang et al.) 2023 Guo et al.)
2023 or autonomously acting agents
(Wang et al. 2024 Yehudai et al.) 2025). While the use of natural language gives a perspective of breadth, recent works highlighted how current evaluation practice fails to measure realistic human-LLan interaction
(Haupt & Brynjolfsson/ 2025) and real-world impact (Burden et al., 2025 Reiter 2025). This agrees with our findings that evaluations for data science mostly fail to capture intermediale levels of LLM-human col- laboration and concentrate on evaluating substitution rather than higher levels of transformation (Sec. 2. Indeed, Chang et al. (2023) and Haupt & Brynjolfsson (2025) highlighted human-in-the-loop testing and evaluations in an open environment as important directions, and [Wang et al. (2024) identified shift to- wards end-to-end tasks requiring human evaluators and versatile metrics, yet most evaluations today only consider a subset of tasks in the data science pipeline, with a few exceptions (Sec. 4.3).
A few studies evaluate truly long-horizon scenarios or quantify the human effort they still require. Wang et al. (2023) and Park et al. 2023) showed that agents can sustain hours-to-days of open-ended play or social simulation, but both exposed failure modes that need periodic human mudges. Quantitatively, Liu et al. (2023a) found that commercial models needed median of 2.4 human corrections per lask a general agent benchmark, whereas open-source models needed 5-8. Recently, [wa et al. (2025) showed that autonomous agents are progressively conquering tasks that take humans longer to complete when considering a fixed success rate (e.g,, 50%), but performance still progressively degrades on tasks requiring more than 10 seconds. Recently, Kwa et al.] (2025) stratified tasks according to the mean execution time humans require to complete them, then measured Al model performance on each subset of tasks within specific time ranges (c.g., tasks taking around 10 minutes for humans). Plotting human-estimated task complction time against AT success rate, they demonstrated that AT success rates decrease as human completion time increases. However, when examining the human time range at which different Al models achieve 50% success rate, more recent models consistently reach this threshold on tasks requiring longer human completion times, indicating progressive improvement in tackling more complex, time-intensive tasks.
2.4 Data science automation
Automating data science was a topic of research even before LLMs became commonplace. Bie et al. (2021) argued that the technical and domain knowlodge required to solve data science tasks motivated efforts to- ward automation. The authors calegorised data science lasks into four main quadrants, defined by two axes degree of open-endedness and dependence on domain context-highhiggttee highlighting that model-buildnng ac- tivitics are more casily automated (c.g., through AutoML approaches, Hutter et al. 2019 Gijsbers et al. 2024) due to their lower open-endedness and context dependence. They also identilied three forms of au- tomation: mechanisation, composition, and assistance. Assistance corresponds to our interpretation of the term while mechanisation and composition can be grouped under our umbrella of automation (Sec. 2.2), but differ in focusing respectively small parts of the process or the overall pipeline; in our work, do not make this distinction and instead rely on the activities of Martinez--Puumed et al. (2099) to identify how many elements of the pipeline each evaluation work covers. After Bie et al. (2021) published their survey, numerous works built LLM agents to automate data science. Their evolution, capabilities, and applications across the data science pipeline are reviewed by Sun et al. (2024); the authors, however, did not address LLM evaluation for data science, which is the focus of our work. More recently, (Hu al.] 2025) proposed a taxonomy for the data ecosystem: Data Management, which includes data collection, data storage, and data preprocessing; Data Analysis, which includes model evaluation, data interpretation, and decision making; and Data Visualisation. This taxonomy partly overlaps with that in Slartnnez-Plumeee et al. (2019) which we use in our work (Sec. 2.5), but misses some of the most exploratory aspects. Finally, Chintakunta et al. (2025) conduct a systematic mapping study examining the application of LLMs in data science. Considering a 5-stage decomposition of data science, they find that most papers apply LLMs in Data Exploration and Analysis, followed by Model Building and Evaluation, and Data Collection and Preparation, with Deploy- ment and Problem Definition unexplored which mostly matches our findings, obtained with a finer task decomposition. Moreover, from the surveyed papers, consensus on research gaps emerges, including the
4

---

Published in Transactions on Machine Learning Research (10/2025)
need to expand to complex, real-world evaluation tasks and to enhance human-AI interaction by creating more user-friendly interfaces; these echo our findings on the lack of human-AT interaction evaluations and the artificial nature of many evaluation tools, for instance by the adoption of a "substitution" perspective. In addition, our survey complements Chintakunta et al. 2025 by providing a deeper analysis evaluation.
2.5
The activities of data science
Table 1: Dala-science activities and brief definition (complete definitions in Appendix A
Activity (abbr.)
Brief definition
Goal-oriented (CRISP-DM)
Business Understanding (BU)
Define the problem and draft plan that meets business requirements Collect and explore data spot useful subsets, insights, issues
Data Understanding (DU)
Data Preparation (DP)
Build the final analysis dataset via selection, cleaning, and transformation Apply modelling techniques, tune their parameters and evaluate models Check that the business objectives are met, with no overlooked issues Deliver the model's outputs in a usable form (report, integration, etc.)
Modelling (M)
Evaluation (E)
Deployment (Dep)
Explovatory
Goal Exploration (GE)
Identify business goals that could be addressed with cata Data Source Exploration (DSE) Discover new, valuable data sources
Data Value Exploration (DVE)
Judge the potential value that can be extracted from the data
Result Exploration (RE)
Connect data-science results back to business goals
Narrative Exploration (NE)
Craft meaningful (visual textual) stories from the data
Product Exploration (PE)
Devisc services or applications that turn extracted value into products
mata-menngtmnnt
Data Acquisition (Acq)
Obtain or generate relevant data (e.g., via sensors or apps)
Data Simulation (Sim)
Simulate systems to generate data and explore causal "what-if" sccnarios
Data Architecting (Arch)
Design the logical/ logical/physical /physical layout and inlegration of data sources Make data accessible through databases, APIs, or visualisations
Data Release (Rel)
Many taxonomies of data science activities exist (see Martínez-Pluuse popular CRISP-DM (Cross Industry Standard Process for Data Mining, Chapman 2000), which considers projects goal-oriented, with pre-defined objective that can be approached by "mining" data through an approximately sequential process, from problem framing to solution delivery. However, Martinez-Plumeel ct al. (2019) argues that this goal-oriented,, pre-collected-data perspective ignores many tasks of modern data science, where exploration is essential and the data takes centre stage rather than serving as a fixed backdrop. Consequently, Saartinez-Plumeee et al. (0199) expands this taxonomy by proposing a sequence of exploratory activities that underscore the less prescriptive nature of data science, re-framing as an investigative endeavour; and data management activities that do not assume data is already given and require to fetch more data from different sources. We provide a list. of the activities introduced in Mlarinez-Plumee) et a.l. (2019) and concise definition in Table [ Note that not all modern data-science projects include every activity, nor is the order of activitics fixed as in the CRISP-DM framework. Instead, cach project follows ils own "trajectory" in the space of data-science tasks MMrrlínez-lunnnn el. al. 2019). Moreover, the distinction between activities may be blurred.
al. 2019 See 2). One of the most
3 Evaluating LLM assistants in data science
In this section, we focus on evaluations of LLMs as assistants, namely prompting them in a fixed, pre- determined manner without letting them independently determine the sequence of steps. Table showss the
5

---

Published in Transactions on Machine Learning Research (10/2025)
surveyed papers and the activities (Sec. 2.5) they cover; double tick marks an activity that is explicitly assessed, whereas single tick marks an activity that is required for completing the tasks but not directly assessed.
Data science activitics covered by the surveyed LLM assistants evaluation works. See Sec. 2.5 for definition of the acronyms. A double tick refers to an activily explicitly evaluated, while a single lick refers to an activity necessary for succeeding in the tasks but not explicitly evaluated.
Table 2:
Goal-oriented
Exploratory
Data Management
Papers
BU DU DP M E Dep GE DSE DVE RE NE PE Acq Sim Arch Rel
ARCADE Yin et al. 2022
AssistedDS Luo et a.. 2025
V
d
v
CERT [Zan et al. 2022
CoSQL Yuu al.
201 9a
DS-1000 1000 Lai et a..
20033
-SSBhncch (Oyanng et al.
2025
DSP CCannel
2022
FeatEng
Pietruskka et al.
2024
GPT4-DA [Cheng et al.
2023
HardML Pricope 2025
LIDA Dibia 2023
SParC Yu et al.
2019b
Spider Yu et al.
2018
Spider 2.0-Lite (Lei et al.
2024
Spider 2.0-Snow (Lei et al.
2024
StatLLM (Song et al. 2025
First, many works evaluate LLMs used to generate code for specific steps data science, such as prepro- cessing data given a lemplate, fixing bugs, or producing visualisations given instructions or prerequisites. In particular, ARCADE (Yin et al. 2022) and CERT (Zan et al. 2022) focus on Data Preparation and related activities with specific Python libraries. ARCADE lems involving data wrangling and Exploratory Data Analysis (EDA), defined as Jupyter notebooks, that require Python's is Pandas library; for example, a problem could involve extracting min and max values from a dataframe and answer questions such as "In which year was the most played game added?". CERT instead introduces two benchmarks (PandasEval and NumpyEval), each consisting of 101 tasks manually reworked for coherence and consistency from StackOverflow problems tagged as relevant to Pandas and NumPy re- spectively; for problems whose solution a function, test cases are included, while the correctness of the predicted variable is checked for the other problems. Relatedly, DSP (Chandel et al.) 2022) contains problems instantiated in 306 pedagogical Jupyter notebooks with 92 associated datasets, covering data ma- nipulation, cleaning, and wrangling (parts of Data Preparation). Similarly to CERT, the correctness of the task is automatically graded with test cases. An example problem would be "Show the correlation between population density in 2023 and 2050, rounded to 2 decimals". Similarly, DS-1000 (Lai et al., 2023 consists of 1,000 coding problems extracted from StackOverflow, spanning Python libraries such as NumPy, Pandas, SciKit-Learn, TensorFlow, matplotlib, SciPy, and PyTorch. The problems are manually perturbed circumvent the issue of memorisation in LLMs and cover Data Preparation and Modelling. The problems are scored through multi-criteria execution-based evaluation metrics that rely on test cases and constraints to check whether the output relies specific packages and functions. See Fig. 2 for an example problem and evaluation set-up. Ouyang et al. (2025) extend DS-1000 to create DS-Bench by adding Seaborn, Keras and LightGBM. To build the benchmark, they first define a broad task scope and collect Python seed code from GitHub using DS-1000's reference code and corresponding StackOverflow answers. An automated LLM pipeline then transforms each snippet to avoid memorisation. Candidates are filtered by properties (compilability, stars, API calls) and functionality (must pass least one LLM-generated test). For each
benchmark consisting of 1,082 coding prob-
httpe://stacckorrffoo..ooo// com.uusstoossess

---

Published in Transactions on Machine Learning Research (10/2025)
surviving candidate, an LLM generates 200 test cases and problem cenription-compller with an intro- duction, function signature, input//ouiput formats and examples. After manual review, 1,000 problems are selected. Performance is measured with pass@k: a problem is solved if any k generated samples pass the unit tests. Instead, FeatEng (Pictruszka et al., 2024), evaluates LLMs' ability to produce a Python function for engineering data features (thus addressing Data Understanding and Preparation) suitable for downstrean modelling tasks. The authors select datasets based on their popularity on Kaggle and ensuring broad domain coverage, reaching a total of 101 tasks. Notably, and in contrast to the older works described above, where questions admitted fixed ground truths, performance is measured in terms of the reduction in error of a model trained on the extracted features compared to a baseline model trained on the original, untransformed data. Therefore, this allows Al models to go beyond simply substituting humans and reach higher levels of task transformation (Sec. 2.1). StatLLM [Song et al., 2025) instead focuses on statistical Modelling, assessing LLMs' ability to generate code to solve a dataset of 207 statistical analysis tasks assembled from various public online resources, including descriptive statistics, hypothesis testing, regression and ANOVA, gener- alised linear models, survival analysis, model selection, and non-parametric statistics; tasks might require the LLM to run specific model on variable in a given dataset, or to plot a variable. Uniquely, the LLM has to generate SAS code; evaluation is carried out using Natural Language Processing (NLP) metrics to compare the generated code against a human gold standard, thus being grounded in substitution (Sec. 2.1
Considering Data Management activities, Spider (Yu et al. 2018 SParC (Yu et al., 2019b) and CoSQL (Yu et al.] 2019a) (all from the same research group) evaluate conversational database querying systems translating natural language into SQL queries (part of Data Architecting but also requiring Data Under- standing). These works build the same 200 databases from 138 domains: Spider consists of 10,181 manually crafted questions and 5,693 unique SQL reference queries and evaluates the generated queries with matching of SQL components or the overall query to the reference one, or with the accuracy of the execution. SParC expands Spider, which contains only single-turn questions, by simulating multi-turn interactions and therefore introducing context dependence: annotators chained Spider tasks together in a conversational flow resulting in 4,298 question sequences with 12,726 questions. Performance is evaluated in terms of exact set match (per turn), and interaction match (full sequence accuracy); however, this does nol evaluate the ability of the Al system to interact with a user successfully. This is done in CoSQL, which also includes task where the system must identify ambiguous questions needing clarifications and unanswerable queries (accuracy is evaluated using dialogue act labels). This makes CoSQL unique in addressing intermediate levels of automation for assistants (Sec. 2.2). The clarifications are then included in the context the system uses to determine the correct SQL query, scored using exact match or component match. However, despite being inserted in the context of a conversation, only one system answer at a time is evaluated, therefore still considering the paradigm of substitution (Sec. 2. Natural language summaries of the query output produced by the system are also evaluated (with the BLEU score). Overall, CoSQL comprises over 30,000 dialogue turns and 10,000 annotated SQL queries, derived from approximately 3,000 dialogues collected by having users interact with mock interface controlled by an expert and simulating real-world database query sconarios. Finally, the same authors recently introduced (Lei et al. 2024). Spider 2.0-Lite, consisting of 547 test instructions mapped to 158 real databases hosted on BigQuery Snowflake and SQLite and solely scored based on execution accuracy, and Spider 2.0-Snow, re- re-hosting the same 547 questions on Snowflake to spotlight one dialect while keeping identical self-contained evaluation.
Moving to the exploratory aspects of data science, LIDA (Dibia 2023) introduces a system generating data visualisation and infographics by prompting L.MSs in structured manner to provide a summary of the dataset (Data Understanding), formulate data exploration goals (Data Value Exploration), generate code specifications for the visualisations (Goal Exploration), and generate stylised graphics based on the previous output (Narrative Exploration). This also covers aspects of Data Release as it involves making data accessible through visualisations. The system is accompanied by an evaluation tool, based on 57 datasets sourced from the Vega d repository; two metrics are used: visualisation error rate, computed as the percentage of generated visualisations that result in code compilation errors; and visualisation quality, in which GPT-4 (Achiam et al.) 2023) is tasked with assessing the quality of the generated visualisations across 6 dimensions: code accuracy, data transformation, goal compliance, visualisation type, data encoding, and

---

Published in Transactions on Machine Learning Research (10/2025)
aesthetics. Instead, AssistedDS (Luo et al. (2025) is benchmark designed to evaluate how well LLMs leverage domain knowledge to improve predictive performance on tabular datasets. Tt tests the model's ability to critically assess, filter, and apply both helpful and harmful external information, using synthetic datasots with controlled feature-label relationships and Kaggle datasets augmented with high- and low-rated notebook insights. Evaluation focuses on code quality and predictive accuracy across, and the same lask is evaluated by providing different configurations of domain knowledge. By measuring how effectively models identify and apply valuable domain insights, AssistedDS directly targets the core of Data Value Exploration.
While the above works focus on single steps of the data science pipeline, Cheng et al. 2023) evaluates GPT-4 as a data analyst on end-to-end data mining problems (excluding several exploratory steps and the entirety of data management). In particular, they provide GPT-4 with a database schemn (Data Understanding) and a real-world business question (Business Understanding) and tasks it with extracting the relevant data (Data Preparation, Data Value Exploration), conducting Modelling, generating visualisations and producing an analysis (Deployment, Narrative Exploration, Result Exploration). GPT-4 embedded within framework (referred to as GPT-4DA); in which it is first. prompted to generate code that is executed to produce graphs and text file containing the generated data, and then prompted again generate an analysis comprising five insights derived from the textual data (excluding the figures). They devise three evaluation metrics for the generated figures (correctness of data and information, chart type, and aesthetic), and four evaluation metrics for the generated insights (correctness of data and information, alignment with question, complexity, and flu- ency). By using these broad metrics, GPT-4 is free to solve the task in ways different from what humans would do, thus reaching higher levels of transformation (Sec. 2.1p. They test this pipeline on the NvBench dataset (Tuo et. al. 2021 and employ six human professionals evaluate GPT-4 (using rubric detailing the above metrics) and a professional (human) data analyst as bascline. While involving humans leads to more compre- hensive understanding of performance, it also makes rumning the evaluation more costly and less reproducible.
Data science involves additional skills other than coding, For example, domain knowledge in data science is essential. To evaluate this, Pricope (2025) introduce HardML, a benchmark of 100 multiplo-choice questions, designed to challenge experienced data science professionals, assessing advanced reasoning skills and domain knowledge. The questions are original, handcrafted, and may include multiple correct answers; they span various topics such as natural language processing, computer vision, statistics and statistical modelling, classical machine learning, and cover activities such as Data Understanding and Preparation, Modelling, Data Value Exploration and Narrative Exploration. An example question is: "An AT company just shipped a new foundational language model. They claim they have trained it for 2.79M H800 hours on 14.8T tokens. Upon further rescarch, looking at Nvidia card specs, you find 3,026 TFLOPs/s of FP8 performance with sparsity, or typically half this (1.513e15 FLOPs/s) without sparsity. Moreover, you find out that they used FP8 FLOPs without structured sparsity. Given that the model has 37B activated parameters, roughly what hardware utilization did they achieve? Select the closest." " Importantly, whilst several activities are evaluated by the benchmark, each question only targets a single activity; moreover, the majority of the questions focus on reasoning capabilities and coding for machine learning and various aspects of deep learning engineering.
From this overview, it is evident how nearly all LLM assistant evaluation works focus on code generation, and how there is a concentration on the goal-oriented activities of Data Understanding, Data Preparation and Modelling (Table 2). with a paucity of evaluation works for exploratory and, even more, data management activitics, except for Data Value Understanding, and few works touching on Narrative Exploration, Goal Exploration, Data Source Exploration, Result Exploration, Data Architecting, and Data Release.
4 Evaluating LLM agents in data science
In this section, we consider evaluations of LLM agents, which augment LLMs with a set of affordances and allow them to determine the sequence of steps they go through by iterative prompting. We also consider works evaluating agents with (simulated) user interaction. Table 3 shows the papers we overview and the activities they cover (Sec. 2.5 a double tick marks an activity that explicitly assessod, whereas a single tick marks one that is vital for completing the tasks but not directly assessed.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Data science activities covered by the surveyed LLM agent evaluation works. See Sec. 2.5 for definition of the acronyms. A double tick refers to an activity explicitly evaluated, while single tick refers an activity necessary for succeeding in the tasks but not explicitly evaluated.
Table 3:
Data Management BU DU DP M E Dep GE DSE DVE RE NE PE Aeq Sim Arch Rel
Goal-oriented
Exploratory
Papers
BLADE Cuu et al.
2024
BissBenchmark el al.
2025b
CoTa (Li et al.
202bc
V
V
BSR-Bench (Niao et al.
2025
Data- Cola-Copilot [Zhang el al. 2024a
DA-Code (Huang et 20246
DiscoveryBencl (Ma uunder ler
et al.
2024
DSBench (Jing et 2024
DS-Eval Zhang et al.
2024b
IDA IDA-Bench (Li al.
2025a
InfiAgent- DABnchh Hu et al.
2024
Bench (Sahu al. 2025
BenghntBBnnch (Huang et al.
20244
MLE-Bench (Chan
2024
MLGym (Nalhami et al.
2025
RE-Bench Wiik el
2024
SgienceAgenlBeeng Bench (C Chen el al.
2024
Spider 2.0 (Lei et al.
2024
SUPER Bogin et al. 2024
WebDS Hsu et al.
2025
4.1 Works targeting specific goal-oriented activities
Many works aim to evaluate LLM agents on individual goal-oriented activities of the data science pipeline. Starting from Data Understanding, [i et al. (2025b) introduce BiasBenchmark, which evaluates the ability to detect biases in datasets. To build the benchmark, they select 5 datasets from prior bias mitigation rescarch and 100 demographie--elated features their combinations and craft (using an LIM playing the role of a user) possible bias detection queries, including intentionally ambiguous questions. Then, during evaluation, a bias detection query is passed to the evaluated LLM agent to test whether it is able to effectively detect if the bias exists by analysing the provided data; clarification questions posed by the agent are automatically answered by the LLM-based user simulator according to the original task specifications, ensuring consistency and reprodueibility. The agent must. quantify the bias level according to a 5-level scale, which is then compared the ground truth obtained by measuring five widely-used bias detection metrics. However, scoring the final bias level allows to agent to determine it in potentially novel ways, therefore going above human substitution (Sec. 2.1 They also evaluate the agent's intermediate process, by developing agent-based automated evaluation system that looks at the evaluated agent's 's logs and produces performance rating levels for five aspects: user communication, task planning, tool invocation, dynamic plan adjustment and result analysis. Interestingly, the first one (user communication) scores the ability of the agent to ask clarification questions to the (simulated) user who set the task.
Instead, Data-Copilot (Zhang al. 2024ah introduces an LLM agent for data wrangling that, given a dataset schema, independently explores potential user requests and generates modular code to address them, which is then leveraged in the deployment stage. To benchmark it, the authors release 547 test requests drawn from 173 human seeds plus larger 3547-request self-exploration pool. The tasks rely on financial data and touch upon Data Value Exploration, Data Understanding and Preparation, and Modelling. Each test case is accompanied by a human-curated answer table and they are jointly (manually) annotated with four labels for dataset analysis: task difficulty, request rationality, expression ambiguity, answer accuracy. System performance measured with GPT-4-based @ scoring against the gold tables (plus an image check) and with the number of tokens used.

---

Published in Transactions on Machine Learning Research (10/2025)
Moving towards Modelling, MLE-Bench (Chan et al. 2024) and RE-Bench (Wijk et al. 2024 are both learning engineering benchmarks, but differ in the complexity of the tasks and scenarios: passes 75 tasks sourced from Kaggle 4 whose deterministic scoring functions are taken from the corresponding Kaggle competitions- as they score the final result, this allows agents to solve the task in ways potentially dif- ferent from humans, going beyond substitution (Sec. 2.1;; ; however, because these functions vary across lasks, each score is compared against a snapshot of the (human) leaderboard. Instead, RE-BBench includes envi- ronments each presenting unique Machine Learning (ML) task focused on optimising either the loss function or the run-time; the value of these scoring functions is manually inspected for evaluation and evaluators need to have access to a reference solution. Relatedly, ALAgent Bench (Huang et al. 2024a) comprises 13 tasks specified by a goal, occasionally constraints or specific instructions, starter files, and an evaluator; the tasks are collected and adapted from recent Kaggle challenges, CLRS (Veličković et al. 2022 BabyLM the starter files consist of data, description of data and metric, and initial code; each task has its own goal metric to im- prove on, whose mcasure is used for automated evaluation. For an overview of MI.Agent workflow and evalu- ation, see Fig. 7 Instead, Nathani et al. (2025) introduce MLGym, an environment to train LLM agents on ML tasks using reinforcement learning. Given task description, initial codebase, and actions and observa- tions history, the agent generates action (shell commands executed by the environment) to accomplish re- search objectives iteratively; the execution feedback can then be used to refine the agent. MLGym is equipped with a benchmark consisting of 13 tasks spanning data science, game theory, computer vision, NLP,, and reinforcement learning, and selected from sources such as Kaggle's House Price Prediction [ 1971), CIFAR-10's image classificationf] and more; they require the agent to perform Data Understanding, Modelling and Evaluation. As the various tasks have different performance metric, they score each agent by a quantity that reflects how closely, on average across a range of tolerance levels, it matches the best performer every individual task. MLGym differs from MLAgent Bench for the larger complexity of its tasks. Finally, [L et al. 2025a introduce IDA- IDA-Bench, which attempts to evaluate LLMs on their ability to perform guided predictive Modelling tasks; the benchmark includes an LLM-simulated user with domain knowledge and sub- jeetive insights who interacts with an agent to provide instructions throughout a multi-turn itcrative process; the agent is then lested on adapting its goal and following instructions. The lasks are obtained from Kaggle; an LLM distils reference insights in natural language format from an optimal solution. These, together with information such as hyperparameters, serve task-specific template for the simulated user, which requests the agent to perform certain steps, without offering all insights up-front, and offers clarifications when asked. Results of the trained model are evaluated against ground truth using task-specific evaluation functions; and compared with a human baseline, obtained by running the notebook the simulated user has access to. They also dotermine the ability to interact by considoring how the prodiction accuracy changes by increasing the number of interactions. Fig. shows an example "trajectory" of the guided data analysis process.
encom-
3-SAT (Cook,
Next, CSR-Bench (Xiao et al., 2025) and SUPER (Bogin et al., 2024) test whether agents can correctly deploy code from a project repository when given instructions an important, though not exclusive, part of the data-science Deployment stage (Sec. 2.5) CSR-Bench and SUPER both transform GitHub repositories into end-to-end "run-the-code" challenges in which an autonomous LLM agent must parse documentation, install dependencies, debug failures, and produce a outcomes assessed by an automatic completion metric. CSR-Bench supplies 100 diverse repositories, each constituting one comprehensive task that typically involves environment setup, data and model acquisition, model training, inference, and evaluation. In contrast, SUPER targets reproducibility in machino-lcarning and NLP research across 801 repositorics, organised into three nested nubsets-erppe (45 manually authored full-pipeline problems with human gold standards), masked (152 focused subtasks derived from the expert set), and auto (604 GPP----o-generated tasks created from repository READMEs)-cach accompanied by task-specific metrics or expected outputs for evaluation.
Finally, considering agents performing Data Management tasks, Spider 2.0 Lei et al. 2024 the most recent iteration of Spider (Yu et al. 2018), is a benchmark of 632 real-world text-to-SQL. workflow problems derived from enterprise-level database use cases; the agent's 's answers are evaluated using completion rate, accuracy, and coherence, therefore allowing agents to solve the task in ways different from pure human
htttps//www.kagr www.s//wwwwgglc..oooooo com/ httpp://bbbyll baabbyyll...ghuub github. 10/
https://wwx. htwp://wwww.......oootoeeeeeeeee cang--cnnnBB-
10

---

Published in Transactions on Machine Learning Research (10/2025)
substitution (Sec. 2.1 Spider 2.0 differs from previous benchmarks by the same authors YYu et al. 2018
2019b al discussed in Sec. 3), in its more complex set-up: the tasks do not consist of pre-prepared inputs (question and database schema) or expected outputs (predicted SQL), but a real project codebase and a database interface; the agent interacts with the codebase through command scripts, as well as SQL queries.
4.2 Evaluating multiple activities explicitly
Some works target a broader spectrum of data science activities and evaluate each explicitly. To start with, DA-Code (Huang et al.) 2024b),
(Hu et al. 2024) and DSBench (Jing et al, 2024) all predominantly consider Data Preparation and Modelling, and mostly score the agent-producod solution by closely comparing it with reference ones, thus being anchored in the "substitution" paradigm (Sec. 2.1]. In particular, DA-Code consists of 500 tasks sourced from Kaggle, GitHub, and other sources, each primarily covering exploratory data analysis (which roughly includes Data Understanding and Data Modelling-thie
Value Exploration), Data Preparation, multiple activities, each task is more narrow. DA-Code includes a variety of data structures and requires the use of SQL, Python, and Bash. Each task is accompanied by a single canonical artefact (table, chart, text file or hidden test-sct labels) created by experienced annotators except for predictive modelling tasks. For grading, a solution is stripped down to the elements explicitly constrained by the instructions (such as required columns, the numeric data underlying plot, or specified visual metadata) before applying a strict equality check against the reference artefact. For machine-learning tasks, the grader instead computes the task-specific metric (e.g. F1, MAE, Silhouette) the hidden labels and awards partial credit in proportion to performance above baseline. Relatedly, InfiAgenttDABeneee generated closed-form questions, such as "Is there a lincar relationship between the GDP per capita and the life expectancy score in Happiness rank.csv? Conduct linear regression and use the resulting coefficient of determination (R-squared) to evaluate the model's goodness of fit ... [omitted for brevity]", derived from csv files sourced from GitHub repositorics, with respective gold-standard answers generated by OpenAT's Advanced Data A The benchmark covers a broad range of tasks, such as feature engineering, correlation analysis, data preprocessing, distribution analysis, summary statistios (all representing Data Preparation and Understanding), and machine learning (Modelling). The evaluation relies on caleulating the portion of questions for which all subquestions exactly match the reference solution. Finally, DSBench obtains tasks from ModelOrf and Kaggle and split them into two categories: data analysis, 466 tasks characterised by long text context, various modalities, and a wide scope for solutions, and evaluated in terms of accuracy by an LLM which compares the responses to a human solution; and data modelling, 74 tasks requiring the LLM to build a predictive model with performance scored by the ability of the agent to generate and submit a bug-free model. Beyond Data Understanding, Data Preparation and Modelling, some tasks also cover Evalustion, Deployment, and exploratory activities.
even though the overall benchmark consider
introduces DAFval, dataset of 257 GPT-4
Moving to broader range of activities, DSEval Zhang et al. 2024bb contains chains of inter- dependet problems (based on data from StackOverflow, Pandas-cxerccisey 10 CoetCode and Kaggle) where each highlights a different stage of the data-science lifecycle-Data Understanding and Preparation, Modelling, or interpretation (belonging to Deployment and Narrative Exploration) while re-using the runtime context left by the previous problems. By doing so, agents must solve the overall task by following the same steps that humans would follow; thus, DSEval only evaluates agents' substitution ability rather than their potential to transform lasks (Sec. 2.Ip. For each problem, they employ custom validator modules to check correctness against the solution or run unit. tests. Relatedly, CoTa (Li et al. 2025c) obtains a set of tasks by simulating (with LLMs) a company setup composed of an administrator and data scientist solving a client's 's problem making use of an AI Chatbot Agent; they then manually filter those interactions where the Chatbot Agent produced correct code and obtain 1024 interactions where agents are asked to write code to solve a problem or answer a multiple-choice question. Overall, these tasks cover Data Understanding and Preparation, Data Value Exploration, Modelling, Deployment, Results and Narrative Exploration (by converting plots into answers or summarising findings in prose). They test agents both in "normal" mode,
$ http:///eeenn...ooooccccccccoeee
https://github. com/ htwp:l///ooonnuggoronaaoao guwpsamora/pandes. exercises
https://leetcode.com
11

---

Published in Transactions on Machine Learning Research (10/2025)
where all requirements and details are specified by the user, and in an "action" mode, where the agent has to perform an action such as asking for clarification, updating code based on user-reported error, and others. For code updating and normal turns, the agent's interactivity is disabled, effectively falling back to an assistant setup, while in the other cases the agent may iteratively call a sandboxed Python exccutor. Further, for the "clarification" " lasks, the agent has to pose follow-up questions that are answered by a LLM-simulated user, making this one of few benchmarks (Li et al. 2025b that evaluate interactivity. An example of the different interaction modes can be found in Fig.
Overall, however, scoring is purely based on the outcome (thus not judging the quality of the interaction): ): the agent's artefact is compared to a gold reference with task-specific comparators. Finally, Science Agent (Chen et al. 2024) builds on 102 tasks from scientific peer-reviewed publications, validated by subject experts; each task includes a data-driven discovery goal, information on the data, expert-provided knowledge, and a reference Python program. The questions are challenging, such as "Develop a darug-target interaction model with the DAVIS dataset to repurpose the antiviral drugs for COVID", or " "Analyze Toronto fire stations and their service coverage to identify coverage gaps". The performance of the LLM agent on each task is scored against 3 metries: Program Evaluation (itself consisting of: Valid Execution Rate, Success Rate, API Cost and embedding similarity computed by CodeBERT Zhou et al. 2023); Figure Evaluation (using GPT--4);; and Stubrie-Based Evaluation based on 5 fundamental steps (Data Loading, Data Processing, Modelling or Visualisation, Output Formatting, and Output Saving)- Therefore, this mostly evaluates substitution (Sec. [.1] both by closely referring to human- pronan-rovvied solutions and by splitting the task in the same sequence of steps humans would follow.
4.3 End-to-ent tod tasks scored by their final result
A few works instcad evaluate agents on end-to-end questions--nvolyee formulating plans, generating code and plots, and producing coherent results and insights-and score the final output of the task (in contrast to individual steps as in Sec. 4.2h- This naturally allows to reward higher levels of task transformation beyond mere human substitution (Sec. 2.1p. First, Insight Bench (Sahu et al. 2025) includes 100 tabular datasets of 500 wathettccaallggnertteeee entries cach, organised in structures obtained from a real-world enterprise data management plat.orm. When generating the synthetic data, a set of insights is manually "planted" in them. The insights (a total of 475) are divided into four families: descriptive, consisting of plots that describe the data; diagnostic, analysing the cause behind trends; predictive, consisting of visualisations that summarise model predictions: prescriptive, that explain actionable insights. The LLM agents are evalu- ated based on how many insights they recover, when provided with the dataset and an open-ended goal formulated by non-expert users, such as "Analyse incident trends in the data.csv file". In particular, Llama- 3-Eval, lechnique inspired by G-Eval (Liu et al. (023b) which uses Llama-3 (Dubey et al., 2024), is used to compare the agent-produced insights with the reference ones, both at a summary level and at a deeper description level. The tasks require touch upon multiple data seience activitics from Goal Exploration to Data Understanding, Value Exploration and Preparation, and to Modelling and Narrative Exploration but only the final insights are directly evaluated (Deployment, Narrative Exploration, and Result Exploration). Similarly, BLADE (Gu et al.) (024)) and Discovery Bench (Majumder et al.) (224)) both challenge LLM agents to explore complex dataset with a vaguely defined goal, such as "Are soccer players with dark skin tone more likely than those with a light skin tone to receive red cards from referees?" (Gu et al. 2024). However, differently from Bench,, they consider scientific datasets and focus on agents' ability to integrate statistical knowledge with understanding of data from a broad range of scientific domains. Both BLADE and DiscoveryBench include end-to-end scientific anaa-analysis tasks that begin with genuine re- search questions and necessitate multistep solution workflows. BLADE includes 12 carefully curated datasets collected from statistical textbooks, research papers, and crowd-sourced studies, while Discovery Bench offers 264 real-world tasks drawn from published studies plus 903 synthetic tasks spanning 48 domains. Tasks cover Data Understanding, Preparation, statistical and Machine Learning Modelling, Narrative Exploration, as well as Deployment, Data Value Exploration and various degrees of domain understanding (Business Understanding and Data Understanding). Both BLADE and BeecoveryBenchh grade solutions aulomatically with LLMs so that multiple defensible workflows can receive credit. Their emphases, however, diverge: BLADE chocks if cach analytical step- concepaull variable selection, admissible transformations, model family, byper-parameters corresponds to one of multiple expert- sroouc-produced solutions (to account for alter- natives), which however still limits the amount by which the agent can transform the task; BeecoveryBench
12

---

Published in Transactions on Machine Learning Research (10/2025)
instead scores at the level of the final context, variables and relationship identified, using GPT-4 to judge the semantic match ween the agent's 8 claim and the human-produced reference solutions, without considering how the former was obtained. An overview of BLADE and DiscoveryBench can be found in Figs. 3 and G A recent paper (Hsu et al. 2025) has developed a benchmark, WebDS, to tackle the limitations of current benchmarks in evaluating LLM agents on their ability to browse and search the internet over multiple, multi- modal, unstructured websites to identify and collect datasets dynamically and perform the full trajectory of web data science activities, where they define a web data science task as one that requires navigation within environment to acquire new raw information, that is then manipulated. They manually write 870 tasks based on 29 websites covering 10 data-heavy domains; they evaluate both the end results (using automated evaluation metrics where ground-truths are available), as well as the intermediate steps (such as subtask completion, tool usage, data validity, reasoning quality) using an LLM-as-ajjuge which gives scores on a scale from 1 to 5, and then validating the protocol with human studies and stability analysis.
From Table 3 we can see that agents have been evaluated on more data science activities than assistants (Table 2. particularly considering goal-oriented activities (with the exception of Business Understanding):
however, there is still lack of evaluations for data management and, to a lesser extent, exploratory activities. Additionally, of all the surveyed works, only BiasBenchmark (Li et al.) 2025b), Co (Li ct al., 2025c). and IDA-Bench [Li et al.) 2025a) evaluate agents in a collaborative framework with (simulated) users; whilst simulated users may be an oversimplification of real users interactions, and ignores the employment of domain experts in real-world scenarios, it is a step in the direction of evaluating interactions and their impact on performance.
5 Challenges and future directions
Our analysis shows that most evaluation works focus either on assistance, looking at isolated tasks that require LLMs to provide an answer on single-turn basis (without access to tools and under human super- vision) or on full automation, where LLMs are wrapped in agents that act autonomously. A few notable exceptions exist YYu et al.] 2019a Li et. al. 2025c bla,, which primarily rely on other LLMs to simulate hu- man users. While this approach ensures cost-effective evaluation and reproducibility, exploratory tasks may lead the agent to attempt novel solutions, such that the simulated user may be unable to assist it as the most suitable answer may not be within its knowledge base. This can be partly addressed by employing humans to answer such unprecedented queries and progressively enriching the knowledge base. Overall, however, the evaluation ecosystem should be enriched by more fully-flegged "centaur evaluations" (Haupt & Brynjolfisson, 2025) that directly evaluate human-AT collaborations, reorienting AI development lowards augmentation rather than substitution and allowing researchers to directly measure human-centred desiderata, such as perceived helpfulncss; among the surveyed works, only Li et al. (2025a) goes towards these more complex metrics, by attempting to quantify trade-off between autonomy and Sitiability/perorrint
We found data management and exploratory activities remain mostly uncovered. This is due to 1) the inher- ent difficulty of scoring exploratory activities, which lack fixed ground truth, and 2) the complex real-world interactions that certain exploratory activities (such as Business Understanding and Goal Exploration) and data management activities (Acquisitinn and Simulation) demand. To address this issue, simulated environ- ments where data management and client interaction can occur should be developed, analogous to related developments in scientific research evaluation (Jansen et al. 2024 Cerrato et al.] 2025). Such environments would enable the evaluation of agents or assistants that function holistically as data scientists by under- standing business requirements, facilitating data collection, and adapting customer requests through data exploration. Simultaneously, realistic evaluation should progress toward end-to-end tasks that do not de- pend on strict ground truth comparisons or simple sppivity metrics. Instead, these evaluations should reward insight generation (such as the works in Sec. 4.3) in potentially original ways, thereby properly in- centivising systems that fundamentally redefine activities rather than focusing solely on human substitution.
Overall, these gaps in the evaluation landscape make it impossible to obtain a comprehensive characterisation of the performance of L.LM-basod systems across data science tasks and activitics. This obscures potentially strong correlations between specific activities, which may better illuminate the underlying capability space and provide insights on how to best improve these systems going forward, following the traditional path in
13

---

Published in Transactions on Machine Learning Research (10/2025)
machine learning where better evaluations in particular domains precede better systems. Thus, propose the following actions to improve data science evaluation of AT systems:
More comprehensive benchmarks covering most activities in Table I considering intermediate steps and preparatory activities, for evaluating substituution-occused approaches.
Greater emphasis on incorporating human assistance (either real or simulated) in the evaluation, and developing methods to quantify the trade-off between autonomy and reliability.
Development of comprehensive simulated environments that enable testing Al systems as holistic data scientists performing data collection and client interaction activitics.
Evaluations incorporating end-to-end tasks and broad objectives that allow and reward systems that redefinc activitics and propose original solutions differing from the reference oncs.
Field studies to validate the measurements obtained through these evaluation tools by comparing them to the real-world impact of human-Al collaborations.
Once gaps in the evaluation landscape are filled, conduct correlation studies performance across the different data science activities.
There has been enormous progress in data science automation, compared to the state of the art just a few years ago (De Bie et al. 2022). It is in open-ended tasks, the use of domain context and human-AI collaboration where data science automation is lagging behind, but upcoming tools may be able to conquer these domains: we must make sure our evaluations allow us to properly track progress.
References
OpenAI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shappiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Made laine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dun- ning, Adrien Ecoffet, Atty Elcti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim 'o Posada Fishman, Juston Forte, abella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Toopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He Mike Heaton, Jo hannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jomn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Lo- gan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Li, Rachel Lim, Molly Lin, Stephanie Lin, Ma teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katic Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christinc McLcavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M' Ashvin Nair, Reiichro Nakano, Rajeev Nayak, Arvind Neclakantan, Richard Ngo, Hyconwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavloy, Andrew Peng,
14

---

Published in Transactions on Machine Learning Research (10/2025)
Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rim- bach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Shep- pard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer'on on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll L. Wainwright, J Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welin- der, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tian- hao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report. 2023. URL attps://arxis org/pdf/ httppdfff//3..3
Tijl De Bie, Luc de Raedt, José Heraández-Orallo, Holger H. Hoos, Padhraic Smyth, and Christopher K. I. Williams. Automating data science. Communications of the ACM, 65:76-87, 2021. URL https:
7/al. acm. wwt...org//o/i...o. 1145/349526/
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabhar- wal, and Tushar Khot. SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories. ArXiu, abs/2409.0745 044404 2024. URL https:///rxii: org/arg/// aa////bs///99/ 000p///4400044000000
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo pher Berner, Sam McCandlish, Alec Radford, Πyy Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (,ds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877 1901. Curran Associates, Inc., 2020. URL co470 (tt.06666d0cb44444444444600 http:///peeeeieeenoooo....aoaaaa httpa///////////////////////////
attps://procedings..u neuripp 142f64a-Paper pdf
Erik Brynjolfsson. The turing trap: The promise & peril of human-like artificial intelligence. Daedalus, 151 (2):272-288) 2022.
Erik Brynjolfsson, Danielle Li, and Lindsey Raymond. Generative AT at work. The Quarterly Journal of Economics, pp. qjae044, 2025.
John Burden, Marko Tesić, Lorenzo Pacchiardi, and José Hernández-Orallo. Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture. arXiv preprint arXiv:2502.15690 2025. URL httph:ttt///dtt org/10. (tgg/0..85550/// arXiv 2502. 15620
Mattia Cerrato, Nicholas Schmitt, Lennart Baur, Edward Finkelstein, Sclina Jukic, Lars Münzel, Felix Peter Paul, Pascal Pfannes, Benedikt Rohr, Julius Schellenberg, Philipp Wolf, and Stefan Kramer. Science- gym: A simple testbed fornbsp:ai-drrven scientific discovery. In Discovery Science: 27th International Conference, DS 2024, Pisa, Italy, October 14-16, 2024, Proceedings, Part I, pp. 229-243, Berlin, Hei- delberg, 2025. Springer-Verlag. ISBN 978-3-031-78976-2. doi: 00.1007/998--30311-770 https://dor /ttp://ww00000000000/003-----110
15. URL
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal A. Patwardhan, Lilian Weng, and Alcksander Mkadry. ML.E-bench: Eval- uating Machine Learning Agents on Machine Learning Engineering. ArXiv, abs/2410.07095,, 2024. URL https:/ //tpsii////xv.../// www//bs//////000 07095
15

---

Published in Transactions on Machine Learning Research (10/2025)
Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. Training and Evaluating a Jupyter Notebook Data Science Assistant. In Proceedings of the 36th AAAT Conference on Artificial Intelligence (AAAI), 2022. URL https://github. wwom///ccrooott/aattciiieee
Yu-Chu Chang, Xu Wang, Jindong Wang, Yuanyi Wu, Kaijie Zhu, Hao Chen, Linyi Yang. Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Weirong Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qian Yang, and Xingxu Xie. A Survey on Evaluation of Large Langnage Models. ACM Transactions on Intelligent Systems and Technology, 15:1-45, 2023. CRL https://dd. acm.
/tt.:////...cc..orr/////////// /ttp///o1100.11//34412220 1145/3641289
Peter Chapman. CRISP-DM 1.0: Step-by-step data mining guide. 2000. URL https:///ww: kde. CS. uni assel. http://aawaaapalaa//////////////
pdf
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienoeAgentBench: Toward Rigorous Assossment of Language Agents for Data-Driven Scientific Discovery. ArXiv, abs/2440.050000, 2024. URL https:
7/arxiv. /aaaxii..g//bba........ 2410. 05080
Liying Cheng, Xingxuan Li, and Lidong Bing. Is GPT-4 a Good Data Analyst? In Findings of the Associalion for Computational Linguistics: EMNLP 2023, pp. 9496 9514, 2023. URL
488500//
TXiv. 2305.15038
Sai Sanjua Chintakunta, Nathalia Nascimento, and Everton Guimaraes. Large language models in the data science lifecycle: A systematic mapping study. arXiv preprint arXiv:2508. 1 11698, 2025.
Peter Cihon, Merlin Stein, Gagan Bansal, Sam Manning, and Kevin Xu. Mcasuring AI Agent Autonomy: Towards Scalable Approach With Code Inspection. In Workshop on Socially Responsible Language Modelling Research, 2024. URL htttps://operrevien http:://pppeeeeeeeeeee///eoeeees htt..//oorrmmm/dd///upopcoooo.
Stephen A. Cook. The complexity of theoren--roving procedures. In Proceedings of the Third Annuual ACM Symposium on Theory of Computing, STOC '71, pp. 151-158, New York, NY, USA, 1971. Association for Computing Machinery. ISBN 9781450374644. doi: 00555/00155..00055 URL https://doi... /rtp://////o...0/11.111
1178000177.800444
Tijl De Bic, Luc De Raedt, José Hernández-Orallo Holger H. Hoos, Padhraic Smyth, and Christopher K. I. Willians. Aulomating data science. Commun. ACM, 65(3):76 87, February 2022. ISSN 0001-0782. doi:
10.1145/3495256. URL htttps://doi.org/11 /t/p://60..0g///00111///49/999 1145/3495256
Victor Dibia. LIDA: A Tool for Automatic Generation of Crammar-Agnostic Visualizations and Info- graphics using Large Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), Volume 3: System Demonstrations, pp. 113-126, 2023. URL
https::/accanthholgge wwwp:/hhoooogyo.o/222222.2 acl 1emo..mm... 11/
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur' elien Rodriguez, Austen Gregerson, Ava Spataru, Bap tiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Chris- tian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Cantón Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovie, Frank Zhang, Gabriele Synnaeve, Gabrielle Lec, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang, Guillem Cucurell, Hai- ley Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Joet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Leo, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden Alwala, K. Upasani,
16

---

Published in Transactions on Machine Learning Research (10/2025)
Kate Plawiak, Kegian Li, Ken-591 neth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Niko lay Bashlykov, Nikolay Bogoy- chev, Niladri S. Chatterji, Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasié, Peter Weng, Prajiwal Bhargava, Pratik Dubal, Praveen Krishman, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Sil- veira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Ro main Sauvestre, Ron- nie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Van- denhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujiwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ra- manathan, Viktor Kerkez, Vincent Gonguet, Virginic Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whit ney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aaditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yumus, Andrei Lupu, Andres Al- varado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisen- man, Azadch Yazdan, Beau James, Ben Maurer, Ben Leonhardi, Po-Yao (Bernie) Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tin- dal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang- Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ongenel, Francesco Caggioni, Francisco Guzmn'a 'an, Frank J. Kanayet, Frank Seide, Gabricla Medina Florez, Gabriclla Schwarz, Gada Badcer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayanan, Hamid Shojanaz- eri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Trina-Flena Veliche, Itai Gat, Jake Weissman, James Ge- boski, James Kohli, Japhet Asher, Jcan- -aattttt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cumminns, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kaixing (ai)) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Voer- araghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Takshya Garg, A Lavender, Leandro Silva, Tee Bell, Tei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Mar- tynas Mankus, Matan Hasson, Matthew Lennic, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Tathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Nor- man Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pe dro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollár, Polina Zvyag- ina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky
17

---

Published in Transactions on Machine Learning Research (10/2025)
Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sncha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanic Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupla, Sung-Bac Bang-Bae Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Tonescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wen- wen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosubrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Llama 3 Herd of Models. ArXiv, abs/2407.21788. 2024. URL https://arxix: wwwpiis/.r//b//////e/e/oo/oo aalaaba/400777 24077 21783
Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation. arXiv preprint Sft::5022005559 2025. URL http::///ii...//....00..
iv. 06559]
Carl Benedikt Frey and Michael Osborne. Generative Al and the future of work: a reappraisal. Brown J. World Aft., 30:161, 2023.
Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. Amlb: an automl benchmark. Journal of Machine Learning Research, 25(101):1 65, 2024. URL https://www.jait
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Tong Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Mcrrill, Jeffrey Hoer, and Tim Althoff. BLADE: Benchmarking Language Model Agents for Data-Driven Science. arXiv, 2024. URL
https://arxixs orgg hrrg/bbs//0n 2408. 09667v2
Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, and Deyi Xiong. Evaluating Large Language Models: A Comprehensive Survey. ArXin, abs/23 abs/2310.19936, 2023. URL https:///arti::
/33100.199/3
Erica R Hamilton, Joshua M Rosenberg, and Mete Akcaoglu. The substitution augmentation modification redefinition (SAMR) model: A critical review and suggestions for its use. Tech Trends, 60:433-4h4 441,, 2016.
Andreas Haupt and Erik Brynjolfsson. Position: AI should not be an imitation game: Centaur evaluations. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. URL https:
Ethan Hsu, Hong Meng Yam, Ines Bouissou, Aaron Murali John, Raj Thota, Josh Koe, Vivek Sarath Putta, G K Dharesan, Alexander Spangher, Shikhar Murty, Tenghao Huang, and Christopher D. Manning, Webds: An end-to-end benchmark for web-based data science. 2025. URL httpp:://app sew../iicc..smmmticcccocoooo
lar.org/ owwoorg//ooo Trng/CorpusID. : 280420999
Chuxuan Hu, Dwip Dalal, and Xiaona Zhou. A Dataset-Centric Survey of LLM-Agents for Data Science. OpenReview, 2025. URL httpp::://ppaeeeeeee.eeeeeeeeeee het:/ppdPPid=iWAheemmgglll
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. InfiAgent-DA Bench: Evaluating Agents on Data Analysis Tasks. ArXiv, abs/2401.05507, 2024. URL https://aaxxv..orrgg 8/2401.05507
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MeAggetBench: Evaluating Language Agents on Machine Learning Experimentation. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024a. URL http::ttt///et ott/ arrvv 2310. 03302
18

---

Published in Transactions on Machine Learning Research (10/2025)
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Läu, Jun Zhao, and Kang Liu. DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 13487-13521, 2024b. URL https://arxiv http//rxxi..orga/baa/sa /ttps//11..0100 07331
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
Peter Jansen, Marc-Alexandie Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bod- hisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. Advances in Neural Information Processing Systems, 37:10088-10116, 2024.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. SWF-Bench: Can Language Models Resolve Real-World GitHub Issues? In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview mett/oorunii
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How Far Are Data Science Agents to Becoming Data Science Experts? ArXie, abs/2409.00 000907009,7773, 2024. URL
atg/abbs/409.077/0/
Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Scraphina Nix, Tao Lin, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, and Lawrence Chan. Measuring AI Ability to Complete Long Tasks, Xix preprint Xiv::22003 14499, 2025.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. URL https://do1. http://doo..r///////
org/10 10. 48550/ 2211. 11501
Fangyu Lei, Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Hongcheng Gao, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQT.
Workflows. ArXiv, abs/2411.. 0as/2411.077633 2024. URL https://arxiv. http::///aaa....oggga/////1.1.
Hanyu Li, Haoyu Liu, Tingyu Zhu, Tianyu Guo, Zeyu Zheng, Xiaolie Deng, and Michael I. Jordan. IDA- Bench: Evaluating LLMs Interactive Guided Data Analysis. arXiv preprint arXic:2505. 18223, 2025a.
URL htttp:///arxa http:/wwwworr/bb//555..... 18223 Submitted on 23 May 2025.
Haoxuan Li, Mingyu Derek Ma, Jen tse Huang, Zhaotian Weng, Wei Wang, and Jicyu Zhao. BI- ASINSPECTOR: Detecting Bias in Structured Data through LLM Agents. 2025b. URL https:
04855
Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Bowen Qin, Yurong Wu, Xiaodong Li, Chenhao Ma, Jian-Guang Lou, and Reynold Cheng. Are large language models ready for multi-turn tabular data analysis? In Forty-second International Conference on Machine Learning, 2025c. URL https://openreviee rett/forrumm/d=111xxxx7711
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Hangliang Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Shengqi Shen, Tianjun Zhang, Shong Shen, Yu Su, Huan Sun, Minlic Huang, Yuxiao Dong, and Jie Tang. AeentBench: Evaluating LLMs Agents. ArXio, abs/2308 03668, 2023a. URL htttps://artt www.ii.rg//bbbbbb..o.a.o
3/2308.03688 03688.
19

---

Published in Transactions on Machine Learning Research (10/2025)
Yang Liu, Dan Iter, Yichong Xu, Shuo Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In Conference Empirical Methods in Natural Language Processing, 2023b. URL https://doi. 0eeg/00.85500 htte//axivvvvv.2 16634
Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Dragomir R. Radev, and Arman Cohan. On Learning to Summarize with Large Language Models as References. In North American Chapter of the Association for Computational Linguistics, 2023c. URL https://doi.ors: 10. 200..8850/aaa /a1500/rxxi..v 2305 14239
An Luo, Xun Xian, Jin Du, Fangqiao Tian, Ganghua Wang, Ming Zhong, Shengchun Zhao, Xuan Bi, Zirui Liu, Jiawei Zhou, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, and Jie Ding. Assistedds: Benchmarking how external domain knowledge assists llms in automated data science. ArXin, abs/2506.13992. 2025. URL https://api ser.riiemaaniiiii cacioaaasccholar..ommmmm http:/aaawrrreeoppopre//////0/00 2794144444
Yuyu Luo, Jiawei Tang, and Guoliang Li. nyBench: A Laage-Scale Synthesized Dataset for Doos-- Natural Language to Visualization Task. ArXiv, abs/22112.12920 2021. URL https://arxiv.orgg
2112. 12926
Bodhisaliwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBench: Towards Data-Driven Discovery with Large Language Models. ArXiu, abs/2407.011225, 2024. URL https://arxi
w.org/abs/2407.00722 017225
Fernando Martfnez-Plumed
Plumed, Lidia Cohtroras-Ochando, Cesar Ferri, José Hernández-Oralo,, Mcelis Kull, Nicolas Lachiche, María José
and Peter Flach. CRISP-DM twenty years later: From data mining processes to data science trajectories. IEEE transactions on knowledge and data engineering, 33(8))3048 3061, 2019. URL
ieee. www..ee..ee..aiaeeeee.e http//aaattractttcccmttt////////
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dicuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileamu MT.Gym: A New Framework and Benchmark for Advancing AT Research Agents. 2025. URL https: orgoi.org/00 48550/ 2502..5522200 2502.14499
Shuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun, Qihao Zhu, and Jie M. Zhang. DS- DS-Bench: A Realistic Benchmark for Data Science Code Generation. 2025. URL https://doii org/ 48555//rr455///xxxxx 2505. 15
621
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST 223, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701320. doi: 10. /3////568883366666333333 URL https://doi.oor orgp:io.rg/1// www.1111//35118..100 3606763
Michal Pietruszka, Lukasz Borchmann, Aleksander Jędrosz, and Pawel Morawiecki. Can models help us create better models? Evaluating LLMs as data scientists. arXée, 2410.233311 23331v3,,v 2024. URL https:
7/axxiv.org abs///bs/2244/ 2410. 22331v1
Tidor- Vlad Pricope. HardML: A Benchmark for Evaluating Data Science and Machine Learning Knowledge and Reasoning in AI. Xix 2025. URL https://arxiv. org/ aass//55..11550
Ruben R Puentedura.
Transformation, technology, and education. /tte/, 2006.
[Retrieved May],
Ehud Reiter. We should evaluate real-world impact. Computational Linguistics, 0891-2017. doi: 10.1162/coli.a.1.. URL http:://ooo..oogg/0.11/////.....
Lingsstttiioniiiniiiinniniiti, pp. 13, 07 2025. ISSN
Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, Nicolas Chapados, Christo- pher Pal, Sai Rajeswar Mudumba, and Issam Hadj Laradji. InsightBennh: Evaluating Business Analytics
20

---

Published in Transactions on Machine Learning Research (10/2025)
Agents Through Multi-Step Insight Generation. arXiv, 2025. URL http:::://arrrk.oo/////////0.0 23v3
Ben Shneiderman. Human-centered artificial intelligence: Reliable, safe & trustworthy. Internationalll Journal of Human- Computer Interaction, 36(6):495 -04,,-55,,,, 2020. URL ottpe:///ool.ogg///...000/111122
0.1741118
Xinyi Song, Lina Lee, Kexin Xie, Xueying Liu, Xinwei Deng, and Yili Hong. StatLLM: A Dataset for
Evaluating the Performance of Large Language Models in Statistical Analysis. 2025. URL https:
//doi.org//1 48510./////a.ii......... 2502. 17657
Maojum Sun, Ruijian Han, Binyan Jiang, Houduo Qi, Defeng Sun, Yancheng Yuan, and Jian Huang. A Survey on Large Language Model-basod Agents for Statistics and Data Science. ArXiv, abs/2412.14222,, 2024. URL https: http:///xxvvvv.g///aa/ /402. 14222
Karthik Valmockam, Matthew Marquez, Alberto Olmo, Sarath Sreodharan, and Subbarao Kambhampati.
PlanBench: An extensible benchmark for evaluating large language models planning and reasoning about change. Advances in Neural Information Processing Systems,
2023. URL https:
7/doi. org/10 48550/arXiv 2206 10498
Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Mikhail Dashevskiy, Raia Hadsell, and Charles Blundell. The CLRS Algorithmic Reasoning Benchmark. In International Conference Machine Learning, 2022. URL https:///ot:::: org. 200.0.8550//r/// http/axxvvvvv.0.
15659
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhn, Linxi (Jim) Fan, and Anima Anandkunar. Voyager: An Open-Ended Embodied Agent with Large Language Models. Trans. Mach. Learn. Res., 2024, 2023. URL https:://ooo http:::///oooo//1///// 48550/arXi1 2305. 16291
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6))186345, 2024. URL https: "link.springer com/ http///rrieeee1.11/0111.. 001.000000001100000444001------1
Hjalmar Wijk, Tao R. Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Jun Koba Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. RE-Bench: Evaluating fronticr AI R&D capabil- ilies of language model agents against. human experts. ArXiv, abs/2411.15114, 2024. URL https:
//axxiv. peexxiv.or//ae www.oorg//b/bba/eeeeeeeee abs//ab1/////../////////
Yijia Xiao, Runhui Wang, Luyang Kong, Davor Golac, and Wei Wang. CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories. 2025. URL https://doi.org//
3550/arXivy D.cXii.2502.0111 2502.06111
Asaf Ychudai, Lilach Eden, Alan Li, Guy Uziel, Yilum Zhao, Roy Shmueli-Scheuer. Survey on Evaluation of LLM-based Agents, 2025. URL https://arxiv> wwwzsivv.rrr//.oo 2
Arman Cohan, and Michal
03. 16416.
Pengcheng Yin, Wen-Ding Lú, Kefan Xiao, A. Eashaan Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural Language to Code Generation in Interactive Data Science Notebooks. ArXiv, abs/2212.09248, 2022. URL actps://arxivv
Tao Yu, Rui Zhang, Kai-Chou Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Z Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A Largo-Scale Human- Human-Labeled Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. ArXiv, httr/1009.0888 2018. URL https://aaaat wttr:/..orr/aba/1800.000
21

---

Published in Transactions on Machine Learning Research (10/2025)
Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang. Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Ti, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander R. Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter S. Lasecki, and Dragomir R. Radev. CoSQL: A Conversational Text-to-SOL Challenge Towards Cross- Domain Natural Language Interfaces to Databases. Arxiv, abs/ 0bs/1199.05377 2019a. URL https:
7arxiv .rgg htte//bs///e 1909. 053788
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, He Yang Er, Trene Z Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir R. Radev. SParC: Cross- Domain Semantic Parsing in Context. ArXin, abs/1906. 1906.02285, 2019b. URL.
www. org.oogg////o.oooo.ooo...o.. /a//abb////// /900. 02288
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. CERT: Continual Pre- Pre-Training on Sketches for Library-Oriented Code Generation. In Proceedings of the Thirly-Firy First International Joint Conference on Artificial Intelligence (IJCAI), pp. 2022. URL https:///gitaei httpa//......cccoooooot///oooae.
Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. Copiloopilot: Bridging Billions of Data and Humans with Autonomous Workflow. arXiv, 2024a. URL httppt://axxxi..orrr ab3// 07209v7
Yuge Zhang, Qiyang Jiang, Xingyu Han, Nan Chen, Yuqing Yang, and Kan Ren. Benchmarking Data Science Agents. In Annual Meeting of the Association for Computational Linguistics, 2024b. URL https:
//doi. 485504455////// arxiv.. 17168
Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig, CodeBERTscore: Evaluating code generation with pretrained models of code. arXiv preprint or orXiv:2302.055 05527, 2023.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents. In The Twelfth International Conference on Learning Representations, 2024. URL https:://penrreview wwwp:ppwwwweeeeeneeeeeeeeeeeeeee httt://rrrumm..ookssss......
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis. Arxie, abs/ abs/2304.04665, 2023. URL htttttt////xxxi orgg httpa/ab//233004555
A Appendix: definition of data-science activities from Martinez-Plumed et al. (2019)
We reproduce here for convenience the definition of the data-science activities as used in Martinez- P) Plumed et al. (2019).
The original stages of the CRISP-DM (Cross Industry Standard Process for Data Mining, Chapman 2000 framework are follows:
Understanding the project objectives and requirements from a business perspective, then converting this knowledge into a data mining problem definition and a preliminary plan designed to achieve the objectives.
Business Understanding:
Data Understanding: Beginning with an initial data collection and proceeding with activities to familiarize oneself with the data, identify data quality problems, discover initial insights, and detect interesting subsets for hypothesis formation.
Data Preparation: Encompassing all activitics required to construct the final datasct from the initial raw data. This includes selecting tables, records, and attributes, as well as transforming and cleaning data for modelling.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Modelling:
Selecting and applying various modelling techniques while calibrating their parameters to optimal values. Some techniques have specific data requirements, often necessitating return to the data preparation phase.
Evaluation: Evaluating the constructed model(s) to ensure they properly achieve business objec- tives. The steps taken in the modelling process are reviewed to confirm that important business issues have been overlooked.
Deployment:
Applying the model in a way that useful for the customer, such as generating report, implementing a repeatable data mining process, or integrating it into peeisinn-making systems. While the customer typically executes deployment, the analyst ensures that all necessary steps are understood.
As argued in Martínez-Plumed et al. (201 this framework assumes a well-defined business goal and pre- collected data. Additionally, it follows a fairly linear process, similar to mining metal in known location. Thus, it is goal-oriented and process-centric, with data serving as an essential ingredient rather than the focal point. However, in exploratory data science, data takes centre stage, akin to prospecting rather than direct mining. Martinez-Plumed el al. (2019) introduces the following additional exploratory activities:
Goal Exploration: Identifying business goals that can be achieved through lata-driven approaches.
Data Source Exploration: Discovering new and valuable data sources.
Data Value Exploration: Assessing the potential value that can be extracted from the data.
Result Exploration: Relating data science results to business goals.
Narrative Exploration: Extracting meaningful stories, whether visual or textual, from data.
Product Exploration: Identifying ways to transform extracted data value into services or appli- cations that provide new and valuable benefits to users and customers.
Furthermore, Maartnez-Plumeee et al. 20019 critiques the CRISP-DM model for representing data static entity within the process, assuming that data has already been collected and merely needs understanding and preparation for modelling. However, modern data science projects often involve dynamic data management activities, including:
Data Acquisition: Obtaining or generating relevant data, such through the installation of sensors applications.
Data Simulation: Simulating complex systems to produce useful data and explore causal relation- ships (c.g., "what-if" sccnarios).
Data Architecting:
Designing the logical and physical layout of data and integrating different data sources.
Data Release: Making data accessible through databases, interfaces, and visualisations.
23

---

Published in Transactions on Machine Learning Research (10/2025)
Business Understanding
Data Understanding
Data Source Exploration
Data Value Exploration
Acquisition
Goal Exploration
Deployment
Data Simulation
Data Preparation
Result Exploration
Data Architecting
Release
Product Exploration
Evaluation
Modelling
Narrative Exploration
Figure 1: Maartnez-Plumece et al. (201)): "The DST map, containing the outer circle of exploratory activities, inner circle of CRISP-DM (or goal-directed) activities, and the core the data management activities."
B Appendix: examples of tasks
df pd. pd..aataFram (("A": [1, 3], "s": [4, 5, 6]))
Here is a sample dataframe:
Problem
Prompt
Language Models (GPT-3 Codex)
to inverses each existing column to the dataframe name them based on existing column names with prefix, e.g. inv_A an inverse of column and so an.
Predict
The resulting dataframe should look like so: result pd. DataFrane [1, 3], [4, 6], "inv_A": [1/1,
Replace [insert] code following predicted code snippets
1/2, 1/3], "inv_r = [1/4, 1/5, 1/6]})
Obviously there are redundant methods doing this in a loop, there should exist much more pythonic ways of doing it ... [omitted for brevity]
result df df.div(1). add, prefix(" ("inv.
Execute to evaluate
A:
<code>
Code Context
Multi-criteria Execution-based Evaluation Test case
Inport pandas as pd
pd.DataFrame(("A" [1, 2, 3], "B": [4, 5, 613)
</code>
df pd. Dataaaanne(""""" [1, 2, 3), 18": [4, 5, 6}}) pd.DataFrave((MM) [1, 2, 3],
SOLUTION
[4, 6),
<code>
"inv_A": [1/1, 1/2, 1/3], "inv.B": [1/4, 1/5, 1/6)))
[insert]
</cade>
Test case
END
df,ans ... /onit for brevity)
<code>
print(result)
pdi tolktsstinggco assert equal(result, ans)
</code>
Surface-form constraints
and should appear Syntax
Reference Solution
result df.jjiin(d-aappyy(ammaa x: add)).add_prffix(111
Correct/wrong?
" "An example problem in DS-1000. The model needs fill in the code into linsert]in the prompt on the left; the code will then be executed to pass the multi-criteria automatic evalu- ation, which includes the test cases and the surface-form constraints; reference solution is provided at the bottom left."
Figure Lai et al. (2023):
24

---

Published in Transactions on Machine Learning Research (10/2025)
Reseerch Questiorutypotheek
Data Teble
Crowd-Sourced Annolations
Ground Truth Decision Space
Are soccar players with dark akin some more lkaly then those with fight skin tone to receive red carts from referees?
Tiarsfoms
Conceptual Variatley
DV Received
Trater
Totaro moodd))
Generated Analysis
Automatic Evaluation
IN Plagrr dke Tare
Vos
Language Model Agent
Conasb Prosic
Depertel
Det Sad Carde sdig riSCand cakanne
Cunwert gandsined analyses
Contrd Paction
proeee "riee on weeehhie
Frowe adge
Araanooaiii you Sverty dapaas tava)
matea kg
Vets cooply Ingit coll 2015
Zata Inderstnding
extrical noces concepiu
Speerras Commitiion
remalo recars pistiics zael aeltiteewiaa (P1/1303
Vilue and graph-baned
Lugalio
depeebecy
HLar gieee" (waw?)
Feraons
Vetupar
Inputcis "puno]
Gu et al. 2024):: "Overview of BLADE. Given research question and dataset, LM agents generate full analysis containing the relevant conceptual variables, data transform function, and statistical modeling function (boxes 1-4-5). BLADE automatically evaluates this against the ground truth (box 6)."
Figure 3:
I1
Show me the mean age of customers.
Okay, will computer the mean age of customers using Python code.
mean_age df ['age" 1 Ieeant))
print age custorers: (mean_age)"
Outputs:
Mean age of customers: 53.3
Answer: 53.3
I1 V
I2
Okay, they's re middle-aged. Then, let's 's see what people aged 40-60 bought and how often.
Figure 4: Li 2025ap:: "Example task trajectory for Walmart sale prediction, showcasing the iterative interaction between the simulated user providing instructions and the agent executing code within the sandbox to achieve the analysis goal."
25

---

Published in Transactions on Machine Learning Research (10/2025)
Are there palierns trends in the 10 player's performncc gress courts, opponents they struztle against? Opoonents who have wan at least 3 matches are notable opponents. Return me a list of their names.
there any patterns or trends top player's 's performance on grass courts, such as: notable opponents they struggle against? Return me list of Uheir names
Under: Specific
opponents players
won apainst
opporents, won-grass, matcheslgaass, metches["in
To anower these questions, we'll analyre the dataeet to find notable apponents the top 10 players struggle against, identify factors discuss
isinttpp player__aaaaaaaa names) "=Fanse]lWinnerr volue, counts)
each their per formance on grass courts. More Importantly, can you make sure what kind of opponents notable?
Filter the opponents have
matches against the top 10 players
notable opporinnts
Clarification
opporents, won[spponentt, won >-
Action (Clarification)
any patterns or performance on grass courts, such as: notable ogponents they struggle against? Opporintts who have won at least matches are notable opponents. Return list their
player's
Are there any patterns trends the top 10 player's performance on grass courts, such as: notable opponents they strupple against? Return of
Under Speci File
More Importantly, make what kind opponents
from decision, company import read cav. file,
count, count_unioue, values, check, eelements, In, lit.. reset, Index, convert,to ttt.tooooooooooooo pororry, convert, Lnnoto, Jist
scatirscation
Find the opoonents who have against
Opponents
3 matches notable
valusslgrass matchesif hack, clements, irlliiiggsss matchhs((Waorrl player Fabel[Winner"
opponents,
Find opponents who won against the top 30 players epponents won-osunt unique welussgrees matehennchee ck elementss in, Jisttgress, matchess WWnnerr] top. playyay names) -- Falso) WWhnnr Il...
Private
Private Action (Clarification)
Figure Li et al. (2025c): The figure shows the "Normal" mode, with the agent being provided the relevant information and tasked with writing code address the task, and "Action" mode, where the agent has to take specific action (in this case, asking for clarification). "Private" refers to tasks requiring the use of bespoke software libraries to which the agent has access to.
Dataset:
Workflow:
Has
1. the data (missing
etc.)
Gold
Predicted
Score
Height
Weight
Age
Savings
2. Derive BMI, time pref. from existing columns
for
valid
Joe
178cm
160lb
30
yes
3. Compute correlation of column(s) indicating
Context:
dalaset
dataset
1.0
Sue
168cm
130lb
45
no
time preference to BMI
HasSavings,
HasSavings,
4. Sclect report those strong correlations
Variables:
BMI
BMI
1.0
person's time preference (bias for immediate gratification delayed rewards) related to BMI?
Goal:
Final Answer:
pos.
pos.
Time preference indicators ("HasSavings') is positively correiated with BMI with coeff. 0.3199
Relation- ship:
correlated with 0.32
correlated with 0.3199
1.0
DiscoveryBench Task
Discovery Agent Workflow
Evaluation
Figure 6: Majumder et al. (024): "Each DISCOVERYBENCH task consists of goal and dataset(s) (left). Solving the task requires both statistical analysis and scientific semantic reasoning, e.g., deciding which analysis appropriate for the domain, and mapping goal terms to column names (center). A faceted evaluation allows open-ended final answers to be rigorously evaluated (right)."
26

---

Published in Transactions on Machine Learning Research (10/2025)
Starter Files
Evaluator
Task Deseription
workspace/
images/
Task:
data description.ss
Improve the performance of the current model in train.py... ...
Test Accuracy
evaluation, tx
train.py
for Let's see how doas!
Increasee learning rate!
Edit train. pV
python train. py
Edited! Here the diff
Training Log
workspace'
workspacel
images!
images!
description
data description txt
evaltaaton details.tst
evalusation details IXt
train.py
sabmission.cm
9--
train.py
Figure 7: Huang et al. (2024a): "Overview of Bench. Each environment in MLAgent includes a task description, a set starter files, and an evaluator. An agent can read/write /write files and execute Python code repeatedly, eventually producing final file (e.g., test predictions in submission.csv) The agent is evaluated based on the quality of this file.""
27


Reviews:
[{"review_id": "Vkoqw0nuQ0", "reviewer": "Reviewer_GfA1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- At this stage the paper needs clarifications (see below, “” quoted text are citations from the paper):", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- As a survey paper, the work of the authors necessitated a good deal of finding and analyzing relevant papers to the subject of the survey.\nAlthough the domain is quite recent (data science automation), many works have been undertaken and are reported in the paper.\n- The paper focuses on surveying evaluation of large language Models’ (LLMs) use by such automation approaches. The following assertion summarizes the difficulty of the task: “Evaluations often implicitly assume that AI will substitute humans without functionally changing  the tasks, either in assuming the steps by which a task is solved are the same a human would follow … or by scoring the task referring to human-produced output, despite there not being a single ground truth”.\n- An interesting finding is that data management activities automation are seldom assessed although they are the basis of data science activities and have a tremendous impact on the subsequent ones.\n- The paper has also good readable user (real or simulated) interaction illustrations at the end.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- 1.\tSection 1. Introduction: \n    - You mention “In this paper we focus on how to evaluate assistants and agents”,\n        - As a survey paper, wouldn’t that be “how they are being evaluated”?\n    - “and processing image”, any modality for that matter (not only images).\n    - “1 While many studies assess LLMs on tasks related to data science, such as coding (Jimenez et al., 2024) and planning (Valmeekam et al., 2023), our focus is on those that explicitly target data science.”\n        - Might lead to think that you are not considering coding as part of Data Science (DS), although the rest of the paper states otherwise “Data science involves additional skills other than coding”. So are you focusing on works that go beyond just coding automation evaluation?\n    - “we adopt the widely used data science task taxonomy of Martínez-Plumed et al. (2019)”:\n         - The referenced paper does not state its activity pipelines or maps as a taxonomy nor does it use the terminology. Can you reference the paragraph or most likely figure you are referring to?\n- 2.\tSection 2.3 LLM evaluation: “progressively conquering tasks that take humans longer to complete when considering a fixed success rate (e.g., 50%), but performance still progressively degrades on tasks requiring more than 10 seconds.”\t:\n    - Where two different metrics are used within a comparison (Success rate & execution time) is somewhat confusing. Care to clarify?\n    - These references seem to be missing from the references section: [Hu et al., 2024], [Hu et al., 2025]  \n- 3.\tSection 2.5: “data management activities that treat data as dynamic rather than static.” \n    - Dynamic and Static need to be explicitly defined in this context.\n- 4.\tSection 3 Evaluating LLM assistants in data science:\n    - “a double tick marks an activity that is explicitly assessed, whereas a single tick marks one that is vital for completing the tasks but not directly assessed”: \n         - Double tick activities do not seem vital! May be: “Task vital activities are either represented with double tick marks for explicitly assessed or single tick marks for not directly assessed”. Just a suggestion.\n    - “… constraints to check for the presence of specific APIs.” Unclear without context.\n- 5.\tSection 4.1 Works targeting specific goal-oriented activities:\n    - Paragraph 1: Is creating bias detection queries considered a task?\n    - Paragraph 2: “task difficulty, request rationality, expression ambiguity, answer accuracy.” \n         - How do these labels compare to other ones/metrics in other discussed papers in this section?\n    - Paragraph 4: “test whether agents can correctly deploy code from a project repository when given instructions”: \n         - Does “deploying code here” correspond to the definition of ”Deployment” of goal-oriented (CRISP-DM) activity (Table 1)? \n- 6.\tSection 4.3 End-to-end tasks scored by their final result:\n    - “GPT-4 to judge the semantic match between the agent’s claim and without considering how the former was obtained.” \n         - \"agent’s claim and the ‘defined goal’?\n\n\nImprovement suggestions:\n- Appendices:\n    - Please keep Appendix naming coherent (Appendix A etc.)\n- The paper is well written overall, except notably for cross-referencing and formatting of the references section which is not well structured but needs to:\n    - Better streamline author list. Any reason for the use of the lengthy ‘all authors’ form and not  “Et al.” form.\n    - Cross-referencing is not clearly used: \n         - [Authors, Date, optional index] cross-reference should be tied to same form indexing in the references section:\n              -\tExample [Martínez-Plumed et al. (2019)] to [Martínez-Plumed et al. (2019)]: Fernando Martínez-Plumed, Lidia Contreras-Ochando, Cesar Ferri, José Hernández-Orallo, Meelis Kull, etc.”  or \n         - Use a standard numbering scheme or what the journal recommends.\n    - Ordering references’ list helps a lot: Alphabetically for example.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gSFeTvPxmA", "reviewer": "Reviewer_UPiS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Comprehensive survey. The paper comprehensively surveys the existing LLM benchmarks in data science automation, using different tasks and aspects to categorize different benchmarks.\n\n## Weakness\n- Lack of exploration on the correlations between different data science activities. The paper first introduces the definition of the data science activities from Martínez-Plumed et al. (2019). Then, the paper categorizes existing LLM benchmarks based on the taxonomy. However, why such categorization is important to benchmark the different capabilities of LLMs is unknown. For example, if LLMs show strong correlations between different tasks (e.g., DU and DP), such a detailed categorization becomes unnecessary.\n\n- Lack of in-depth discussion. The paper straightforwardly introduces existing benchmarks without showing an in-depth discussion. Most of the discussion still focuses on the taxonomy for either assistance or agent. The paper is more about summarizing existing knowledge than discovering new knowledge.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper conducts a comprehensive survey on existing benchmarks in data science automation, focusing on LLM as an assistant and LLM as agents. By utilizing the taxonomy in data science activities, the paper summarizes how existing benchmarks fall into different categories, given how they are designed to evaluate LLMs' capability. Finally, the authors summarize current benchmark limitations and propose potential future directions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "As mentioned in the weakness. The authors are suggested to conduct more in-depth quantitative analysis based on existing benchmarks.\n\n- First, the authors are suggested to justify why such a taxonomy is needed for evaluating LLM in data science in the first place. For example, do benchmarks on different taxonomies show strong relevance? If LLMs' capability in the two categories is strongly correlated, do we necessarily need to cover both?\n\n- Next, the authors are suggested to give a more in-depth discussion on these benchmark results, instead of solely focusing on how the benchmark itself is designed. This analysis can give more insights into understanding how well current LLMs can handle different tasks. For example, in which domain are the LLMs still performing unsatisfied. These insights can further inspire the community instead of simply introducing benchmarks themselves.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "psfFNZWPRJ", "reviewer": "Reviewer_V9Xp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n- Comprehensive Taxonomy and Analysis​​: The survey establishes a rigorous framework for evaluating AI assistants/agents by integrating Martinez-Plumed et al.'s data science activities (Table 1) with the SAMR model (Sec. 2.1) and autonomy spectrum (Sec. 2.2). This dual-axis approach effectively categorizes 50+ evaluation tools, revealing critical gaps in coverage (Tables 2, 3). For instance, Table 2 demonstrates that 80% of assistant evaluations focus solely on goal-oriented activities (DU/DP/M), neglecting exploratory and data management tasks.\n- Novel Critique of Evaluation Paradigms: The paper compellingly argues that prevailing benchmarks overemphasize human substitution (e.g., exact code/output matching in DS-1000, StatLLM, and DA-Code) while ignoring higher-order task redefinition (Sec. 2.1). This is exemplified by benchmarks like FeatEng and InsightBench (Sec. 4.3), which reward functional improvements (e.g., error reduction) rather than replicating human steps. The distinction between assistance and autonomy (Sec. 2.2) further clarifies why hybrid human-AI collaboration remains underexplored.\n\n# Weaknesses:\n\n- Methodological Gaps:  While the survey critiques the lack of exploratory/data management evaluations (Sec. 3, 4), it does not deeply analyze why these gaps persist. For example, challenges in ground-truth definition for tasks like Business Understanding (BU) or Data Simulation (Sim) are noted but not quantified. A meta-analysis of evaluation feasibility (e.g., cost, reproducibility) would strengthen the argument for simulated environments (Sec. 5).\n\n- ​​Limited Discussion of Real-World Validity: The reliance on simulated users (e.g., Li et al., 2024; 2025a,b) is presented as a strength but risks oversimplifying human-AI dynamics. Real-world factors—such as domain expertise asymmetry or ambiguous client requirements—are underexplored. Incorporating case studies of field deployments (e.g., CoSQL’s expert-annotated dialogues) could contextualize scalability concerns.\n\n- ​​Inconsistent Treatment of \"Autonomy\":  Section 2.2 conflates technical autonomy (tools/affordances) with process autonomy (goal-setting flexibility). For instance, Spider 2.0 (Table 3) tests SQL generation autonomy but not goal refinement, while DiscoveryBench (Fig. 5) evaluates exploratory autonomy. A unified autonomy metric (e.g., Cihon et al.’s levels) would clarify this spectrum.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey makes the first comprehensive assessment of evaluation tools for LLM-based assistants and agents in data science, introducing a novel analytical framework that critically examines existing literature through the dual lenses of Martinez-Plumed's activity taxonomy and the SAMR transformation hierarchy. The analysis reveals three critical gaps: a pronounced bias toward narrow goal-oriented activities like Data Preparation and Modeling while neglecting exploratory and data management tasks; a polarized focus on either pure assistants or fully autonomous agents that overlooks intermediate human-AI collaboration; and a pervasive \"substitution paradigm\" where benchmarks prioritize replicating human workflows rather than rewarding functional innovation. To address these limitations, the work advocates for developing holistic benchmarks covering underrepresented activities, creating simulated environments for client interaction testing, designing end-to-end evaluations that reward insight generation beyond reference solutions, and conducting field studies to validate real-world impact—ultimately proposing a roadmap to track progress in open-ended tasks where data science automation currently lags. This structured assessment of 30+ evaluation tools establishes foundational criteria for measuring how LLM agents transform rather than merely replicate data science workflows.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- ​​Expand coverage of exploratory and data-management activities: \n  - Add experiments assessing agents on ​​Data Source Exploration (DSE)​​ and ​​Data Value Exploration (DVE)​​ using datasets requiring discovery of novel data sources (e.g., simulate \"what-if\" scenarios via Data Simulation).\n  - Incorporate benchmarks like DiscoveryBench (Sec. 4.3) to evaluate end-to-end exploratory workflows.\n- ​​Address autonomy spectrum gap\n  - Introduce tasks with ​​simulated human interactions​​ (e.g., BiasBenchmark's clarification protocol) to measure adaptability to real-time feedback.\n  - Quantify trade-offs between autonomy and reliability using metrics like human correction frequency (Sec. 2.3).", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)