DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?

## Abstract

Abstract Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have demonstrated impressive language/vision reasoning abilities, igniting the recent trend of building agents for targeted applications such as shopping assistants or AI software engineers. Recently, many data science benchmarks have been proposed to investigate their performance in the data science domain.
However, existing data science benchmarks still fall short when compared to real-world data science applications due to their simplified settings. To bridge this gap, we introduce \ours , a comprehensive benchmark designed to evaluate data science agents with realistic tasks. This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions. \ours offers a realistic setting by encompassing long contexts, multimodal task backgrounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling tasks. Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG). These findings underscore the need for further advancements in developing more practical, intelligent, and autonomous data science agents. 1 1 1 Our code and data are available at Github https://github.com/LiqiangJing/DSBench . .

## 1 Introduction

Figure: Figure 1: The overall workflow of our proposed \ours benchmark. The figure illustrates the process from task description and data file processing through model/agent execution to the final evaluation, demonstrating how a data science agent approaches a given problem and compares its solution to the ground-truth answer.
Refer to caption: x1.png

Large Language Models (LLMs) (OpenAI, ; Touvron et al., ) and Large Vision-Language Models (LVLMs) (OpenAI, ; Liu et al., ) have achieved compelling success on various vision and language tasks, such as natural language understanding (Wang et al., ), visual question answering (Antol et al., ), and image captioning(Lin et al., ), demonstrating their adaptability and effectiveness.
However, despite their achievements, LLMs and LVLMs face limitations when applied to certain real-world tasks due to the lack of integration with practical applications, such as computer manipulation. To address this, advanced LLMs and LVLMs are increasingly being incorporated into interactive intelligent systems, enabling them to tackle complex tasks with additional tools and interfaces. A prominent example of this is the data science agent, an emerging research area that assists individuals and organizations in making informed decisions, predicting trends, and improving processes by analyzing large volumes of data (Hong et al., ; Guo et al., ; chapyer team, ; Zhang et al., ).

The data science agent aims to address data-centric scientific problems, including machine learning, data analysis, and mathematical problem-solving, which present unique challenges, such as complex and lengthy task-handling steps. For example, Jupyter AI
(jupyter-ai team, ) connects generative language models with Jupyter notebooks(^2^22[https://jupyter.org/](https://jupyter.org/).) and provides a user-friendly and powerful way to improve developer productivity in the Jupyter Notebook. MLCopilot (Zhang et al., ) leverages LLMs to make solutions for novel real-world machine learning tasks, based on the existing experiences from historical tasks.
To evaluate the performance of the data science agent, the existing work focuses on developing either code generation benchmarks (Zhang et al., ; Zan et al., ; Chandel et al., ) or math problem benchmarks (Lu et al., ; Cobbe et al., ). For example, DS-1000 (Lai et al., ) introduce one thousand code completion problems covering seven widely-used Python data science libraries: NumPy, Pandas, TensorFlow, PyTorch, SciPy, Scikit-learn, and Matplotlib. Hendrycks et al. () introduce the MATH benchmark, which enables the community to measure the mathematical problem-solving ability of models.
Although these benchmarks can be applied to investigate the performance of data science models, they still do not closely reflect real-world data science tasks.

Building an effective benchmark for data science agents/models still presents a significant challenge. The tasks included must be sufficiently complex to simulate real scenarios, yet the predictions made by these models must remain straightforward to verify. Although existing benchmarks achieved compelling success, they are still under a simplified setting compared to real-world data science tasks.
Firstly, the task instructions in existing benchmarks are often brief and limited to single modalities. In contrast, real-world tasks typically involve lengthy instructions and multiple modalities.
Furthermore, some existing benchmarks only provide incomplete evaluations, focusing primarily on the simple code completion or in-filling capabilities of LLMs/LVLMs, which can be resolved by a few-step reasoning or a few lines of code, overlooking the end-to-end evaluation of the whole system’s performance.
Additionally, the evaluation of existing benchmarks may be biased because it is limited to certain environments, such as specific Python packages. However, real-life data science tasks are tool-irrelevant and data-centric.

To tackle these limitations, we introduce a comprehensive data science benchmark, \ours, and the workflow of our benchmark is shown in Figure 1. Our benchmark contains two categories of tasks: *data analysis* and *data modeling*.
The former focuses on answering a data analysis question that needs the agent to fully understand the data and the question’s intent, and the latter requires the agent to build predictive models to learn from the data and generalize to the test set.
Specifically, our dataset is built on data competitions ModelOff(^3^33[https://corporatefinanceinstitute.com/resources/financial-modeling/modeloff-guide/](https://corporatefinanceinstitute.com/resources/financial-modeling/modeloff-guide/).) and Kaggle(^4^44[https://www.kaggle.com/](https://www.kaggle.com/).). In total, we collected 466 data analysis tasks and 74 data modeling tasks.
\ours offers several advantages over existing data science benchmarks.
The rationale for using these two platforms is that ModelOff and Kaggle are among the most popular data science competitions, offering tasks that closely resemble real-world scenarios, where each task requires extensive data manipulation.
These include a more realistic data science benchmark setting that encompasses understanding long contexts and multimodal task backgrounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling, as shown in Table 1.

**Table 1: Comparison with existing agent benchmarks. Columns include the research field (Field), whether the task instruction includes the data file (Data File?), table (Table?), and image (Image?), the word length of the task description (#Len), whether an executable evaluation function is provided (Exec. Eval.?), whether the benchmark asked the agent to finish the task in a fixed/constrained environment (Env. Fix.?), whether the benchmark only asked the agent to generate code (Code-only?) and the number of tasks (Tasks).**
| Benchmark | Field | Data File? | Table? | Image? | # Len. | Exec.<br>Eval.? | Code<br>only? | Env.<br>Fix.? | Tasks |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Spider (Yu et al., ) | Text-to-SQL | ✗ | ✗ | ✗ | - | ✗ | ✗ | ✓ | 1034 |
| MLAgentBench (Huang et al., ) | Machine Learning | ✓ | ✗ | ✗ | - | ✓ | ✗ | ✓ | 13 |
| SWE-Bench (Jimenez et al., ) | Software | ✗ | ✗ | ✗ | 195.1 | ✓ | ✗ | ✗ | 2,294 |
| Mind2Web (Deng et al., ) | Web | ✗ | ✗ | ✗ | - | ✗ | ✗ | ✗ | 2,000 |
| WEBLINX (Lu et al., ) | Web | ✗ | ✗ | ✗ | - | ✗ | ✗ | ✗ | 2337 |
| WorkArena (Drouin et al., ) | Web | ✓ | ✓ | ✓ | - | ✓ | ✗ | ✓ | 29 |
| AndroidWorld (Rawles et al., ) | Android | ✓ | ✗ | ✓ | - | ✓ | ✗ | ✓ | 116 |
| WebArena (Zhou et al., ) | Web | ✗ | ✗ | ✗ | - | ✓ | ✗ | ✓ | 812 |
| OSWorld (Xie et al., ) | Computer Control | ✗ | ✗ | ✗ | - | ✓ | ✗ | ✓ | 369 |
| DS-1000 (Lai et al., ) | Data Science | ✗ | ✗ | ✗ | 140.0 | ✗ | ✓ | ✗ | 1,000 |
| Arcade (Yin et al., ) | Data Science | ✗ | ✗ | ✗ | 18.4 | ✗ | ✓ | ✗ | 10,082 |
| Spider2-V (Cao et al., ) | Data Science | ✗ | ✗ | ✗ | - | ✓ | ✗ | ✓ | 494 |
| DSEval (Zhang et al., ) | Data Science | ✓ | ✗ | ✗ | - | ✓ | ✓ | ✗ | 825 |
| \ours (Ours) | Data Science | ✓ | ✓ | ✓ | 797.9 | ✓ | ✗ | ✗ | 540 |

For data analysis tasks, we mainly utilize the accuracy rate as the metric. On the other hand, for the data modeling tasks, it is non-trivial to investigate their overall performance because of inconsistency in numerical ranges and evaluation dimensions for metrics of different data modeling tasks. Therefore, we further propose the Relative Performance Gap (RPG) to normalize the different metrics in our data modeling tasks.
We evaluate multiple state-of-the-art LLMs, LVLMs, and agents, discovering that they fail to solve most of the tasks. The best-performing agent in our experiments only achieves 34.12% accuracy for data analysis and achieves a 34.74% RPG.

Our contribution can be summarized as follows,
(1) We construct a data science benchmark, \ours, which consists of 466 data analysis tasks and 74 data modeling tasks.
(2) To comprehensively evaluate different existing approaches for the data modeling tasks, we propose the Relative Performance Gap metric that can normalize various evaluation metrics for data modeling.
(3) We evaluate representative state-of-the-art LLMs, LVLMs, and agents including the most recent GPT-4o, Claude, and Gemini models, and find that our benchmark is challenging for most of the existing approaches.

## 2 Data Science Agent Benchmark

Our proposed data science benchmark \ours mainly consists of two parts: the data analysis task and the data modeling task.
These tasks were chosen because they mirror the real-world challenges that data science experts face in their professional work. Data science often requires handling complex data, extracting insights, and building models to solve problems. To ensure \ours reflects these practical demands, we focus on these two task types, making the benchmark more realistic and aligned with the actual needs of data scientists. In our search for appropriate datasets and challenges, we found that both ModelOff and Kaggle provide excellent examples of tasks that fit our requirements, as they cover diverse and realistic problems, making them well-suited for our benchmark.

### 2.1 Data Analysis

Data analysis tasks focus on answering a data analysis question that needs the agent to fully understand the data and the question’s intent. The answer is unique and certain.

#### 2.1.1 Data Collection

Modeloff is a global data analysis competition that challenges contestants to use Excel to solve data-centric questions and case studies.
It mostly focuses on independent questions that involve mini exercises in Excel.
We found that the questions in Modeloff challenges consist of data analysis tasks that can be solved by different tools, such as Python, Excel, and Matlab. Therefore, we resort to the Modeloff challenge for the evaluation of data analysis ability in data science agents. We collected all Modeloff challenges and then filtered all the challenges that didn’t contain any questions. Finally, the original 43 challenges are filtered down to 38 challenges with 466 questions. The question types can be categorized into multi-choice questions and fill-in-the-blank questions. All the statics information of our data analysis tasks is detailed in Table 2.

**Table 2: Summary of dataset characteristics for data analysis tasks.**
|  | Mean | Max | Min | Total |
| --- | --- | --- | --- | --- |
| #Challenges | - | - | - | 38 |
| #Questions | 12.26 | 50 | 3 | 466 |
| Length (Introduction) | 749.58 | 28,487 | 0 | 28,484 |
| Length (Question) | 65.86 | 406 | 6 | 30,691 |
| #Excel | 0.82 | 2 | 0 | 31 |
| File Size of Excel (KB) | 236.56 | 2,755.92 | 0.16 | 7,333.44 |
| #Image | 0.13 | 1 | 0 | 5 |
| #Sheets | 2.3 | 4 | 1 | 69 |
| #Table | 1.29 | 12 | 0 | 49 |

#### 2.1.2 Task Formulation

##### Input and output.

Suppose we have the task introduction $I$, the data files $D=\{d_{1},\cdots,d_{N_{d}}\}$, and the question $Q$, we then feed them into a data science agent $\mathcal{G}$ to answer the question $Q$, *i.e., *$\hat{A}=\mathcal{G}(I,D,Q)$. $N_{d}$ is the total number of data files and it can be $1$. $\hat{A}$ is the generated answer by the $\mathcal{G}$ with the tool set $\mathcal{T}$. The toolset $\mathcal{T}$ can contain any tools that can operate the data and be utilized by LLMs/LVLMs.

##### Evaluation metrics.

To evaluate the performance of the whole agent system, we compare the semantics of ground-truth answer $A$ and the predicted answer $\hat{A}$ by $S(A,\hat{A})$. If the semantics of ground-truth answer $A$ and the predicted answer $\hat{A}$ are the same, we consider the generated answer to have successfully answered the question. $S(\cdot)$ is the semantics comparison function that is implemented by a LLM and prompt in Appendix A.
The metric for our benchmark is the percentage of data science questions that are answered correctly, *i.e., *task-level accuracy. In addition, we also introduce competition-level accuracy for comprehensive evaluation. Competition-level accuracy is calculated by averaging the accuracy scores obtained from each competition.

#### 2.1.3 Features

Different from the previous data analysis benchmark, our data analysis tasks, have several distinct properties, which are discussed as follows:

##### Varies Modalities.

Different from the previous works which mainly focus on textual modality (*e.g., *(Lai et al., ; Cobbe et al., )), our task consists of various modalities, such as images, Excel files, and tables. To show the distribution of the different modalities in different challenges, we visualized the number of challenges in different modalities, as shown in Figure LABEL:fig:modality.

##### Complex Table.

Various tables are contained in this dataset. There may be several tables for one question. Therefore, the data science agent should identify which table is important for the current question. Furthermore, some questions are solved depending on multiple tables.
Different from the previous benchmark, the length of the table is longer. It is unlikely to solve this kind of question without additional tools, even for humans. In addition, the formats of different tables in one challenge are changeable. For example, some tables have properties in their first row, and some tables have properties in their first column.

##### Diverse Long Context.

The context in our dataset consists of long textual descriptions (815.44 words on average) as well as multiple modality content. Resolving this kind of data analysis question requires a full understanding of the long description text and the corresponding tables, data files, and images, and identifying the important context semantic content with the current question that needs to be answered.

##### Wide Scope for Possible Solutions.

Our evaluation task provides a critical platform for assessing the capabilities of data science agents and the corresponding systems.
Our data analysis tasks can be designed to compare a variety of approaches, from pure LLMs/LVLMs to cutting-edge data science agents.
The task setting greatly expands the freedom of tool use and encourages developers to employ creative strategies that may diverge from established norms (such as using Excel). For example, we can use either Excel or Python to calculate the amount of tax a company pays.

### 2.2 Data Modeling

Data modeling tasks focus on using machine learning technology to learn from data and generalize to unseen data, making predictions with the task instruction.
Different from data analysis tasks that aim to make a single and verifiable result, data modeling results in predictions that are evaluated based on metrics for a testing set, such as accuracy.

#### 2.2.1 Data Collection

To evaluate the performance of data science agents on data modeling tasks, we resort to machine learning competitions. Kaggle is a data science competition platform and online community for data scientists and machine learning practitioners. From the platform, we found there are total 648 competitions. Since the testing set in the Kaggle competition is inaccessible, we split the original training set into the training set and testing set as an 8:2 ratio for evaluation.
In this way, we could directly get the performance of the solution devised by a data science agent, avoiding submitting the solution to the Kaggle website.
To split the dataset easily, we only retain the competitions with a training file, a testing file, and a sample of submission file. Then we can use an automatic code to split the original data files in the Kaggle competition.
Finally, we got 74 data modeling competitions in our data science benchmark. All the statics information of our data modeling tasks is detailed in Table 3.

**Table 3: Summary of dataset characteristics on data modeling tasks.**
|  | Mean | Max | Min | Total |
| --- | --- | --- | --- | --- |
| #Competitions | - | - | - | 74 |
| #Metrics | - | - | - | 18 |
| Context Length | 687.5 | 2,505 | 216 | 50,875 |
| #Training samples | 287k | 4,828k | 200 | 21,270k |
| File Size of Training set | 61.07 GB | 486.66 GB | 11.17 KB | 4519.31 GB |

#### 2.2.2 Task Formulation

##### Input and output.

Suppose we have the competition description $E$, the training set $A$, the testing set $S$, and the sample of the submission file $M$, we then feed them into a data science agent $\mathcal{G}$, which could devise an algorithm and implement the corresponding code to generate the submission file $\hat{F}$ which is the predicted result for the input testing set, *i.e., *$\hat{F}=\mathcal{G}(E,N,S,M)$. $\hat{F}$ is the generated submission file by the $\mathcal{G}$ with the toolset $\mathcal{T}$ and it has a similar format with the sample of submission file $M$. The toolset $\mathcal{T}$ can contain any tools that can operate the data and be utilized by LLMs/LVLMs.

##### Evaluation metrics.

To evaluate the overall performance in all competitions, we first adopted Task Sucess Rate as the evaluation metric, *i.e., *whether the agent can build an ML model and generate the submission file in a bug-free manner within a fixed number of steps.
To further learn the performance of the model devised by the agent, we also
evaluate the predicted submission file $\hat{F}$ with the original metric corresponding to the competition as $p=f(F,\hat{F})$. $F$ is the ground-truth file for the testing set, $f(\cdot)$ is the evaluation function of the specific metric (such as F1 and accuracy), and $p$ is the performance.
However, the metrics are different across different competitions, so we can not directly take the average across tasks. Thus we calculate the gap between the performance of the file submitted by the agent and the performance of the human expert as an evaluation indicator of the data modeling task.
Specifically, we devise the Relative Performance Gap (RPG) metric to show the performance of the data science model, formulated as

$$ $\frac{1}{N}\sum_{i=1}^{N}\max((p_{i}-b_{i})/(g_{i}-b_{i}),0).$ (1) $$

$N$ is the total number of competitions. $p_{i}$ is the performance of the predicted submission file for the $i$-th competition and $g_{i}$ is the highest performance value = for the $i$-th competition.
$b_{i}$ is the performance of a baseline. More details about how to get the performance of baseline can be found in Appendix B.

#### 2.2.3 Features

##### Long context.

For each competition in Kaggle, we crawled the corresponding task, evaluation, and dataset description, as shown in Figure 3(^5^55The sample in the figure is from the URL [https://www.kaggle.com/competitions/bike-sharing-demand](https://www.kaggle.com/competitions/bike-sharing-demand).). The description depicts the task background and the aim of the competition. The evaluation introduces what metric is used to evaluate the performance of the competition and how the metric is computed. The data description contains the overall description of all data sets and the explanation of each attribution in the data file. The average length of context text and data of our data modeling tasks is 79187.

Figure: Figure 3: The content of a Kaggle competition which contains task description, evaluation, and dataset description.
Refer to caption: x2.png

##### End-to-end setting.

Different from the existing work that only asks the model to generate the code, our task setting is more difficult and requires a wider range of capabilities of agents, *e.g., *model design, code implementation, self-debugging, *etc. *In other words, our data modeling task is end-to-end and every step in task resolving focus on the different ability of agents. The end-to-end setting allows the model to complete the task with minimal constraints, which is similar to the real-life task the data science experts face. Hence, our data modeling task is not only to test the capability of the underlying LLMs/LVLMs but also to test whether the interactions and environment in the system are properly designed.

##### Execution-based Evaluation.

We use execution Python script to verify the usability of the submission file and evaluate the performance of the submission file from data science agents. Hence, the generated submission file from data science agents should strictly comply with the file format requirements in the input. For different competitions, we may use different metrics. In total, the number of metrics in our dataset is 18 and the distribution of the number of competitions in each metric is shown in Figure 4.

Figure: Figure 4: The chart displays the count of Kaggle competitions (vertical axis) categorized by the evaluation metrics used (horizontal axis). Each bar represents the number of competitions that employed a specific metric, highlighting the diversity of evaluation criteria in \ours. “m0-m17” denote metrics Accuracy (m0), ROC (m1), Normalized Gini Coefficient (m2), Root Mean Squared Logarithmic Error (m3), Root Mean Squared Error (m4), R2 Score (m5), Mean Columnwise Root Mean Squared Error (m6), Macro F1 (m7), Micro-averaged F1 (m8), Mean Absolute Error (m9), Word-level Jaccard Score (m10), Quadratic Weighted Kappa (m11), Pearson Correlation Coefficient (m12), Median Absolute Error (m13), Symmetric Mean Absolute Percentage Error (m14), Mean Column-wise Spearman’s Correlation Coefficient (m15), MPA@3 (m16), and Logarithmic Loss (m17).
Refer to caption: x3.png

## 3 Experiment

### 3.1 Experimental Setups

As we mentioned before, we mainly use the accuracy for the data analysis task, and RPG score for the data modeling task. In addition, we also report costs (if the agent utilizes a charging API) and the average time of resolving a task. We select two kinds of models for evaluation: (1) vanilla language model and (2) agent (*i.e., *LLMs/LVLMs+interaction environment). The vanilla language model contains open-source LLMs (including Llama3-8b, Llama3-70b Touvron et al. () and LLaVA Liu et al. () ) and closed-source LLMs (including GPT-3.5, GPT-4, GPT-4o, GPT-4o mini), Gemini and Claude. The agent system includes closed-source system Code Interpreter(^6^66[https://platform.openai.com/docs/assistants/tools/code-interpreter](https://platform.openai.com/docs/assistants/tools/code-interpreter).)) and open-source
agent systems (including Autogen (Wu et al., )).
For the Code Interpreter, we selected GPT-3.5, GPT-4, GGT-4o, and GPT-4o mini as base models, respectively.
For the Autogen, we use Llama3-8b, Llama3-70b, GPT-3.5, GPT-4, GPT-4o, and GPT-4o mini as agents, respectively.
All the open-source models are run on a 4 $\times$ NVIDIA A100 GPU server.

### 3.2 Quantative Analysis

#### 3.2.1 Data Analysis

We show the performance comparison among different baselines on data analysis tasks in terms of accuracy rate, cost, inference time, and challenge accuracy in Table 4.
From the Table, we observed that: (1) Models that perform better on basic language tasks are more capable of performing data analysis tasks. For example, GPT-4o achieves the best performance among all vanilla model-only baselines and it also shows advanced performance on several general language tasks(^7^77[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/).), such as MMLU (Hendrycks et al., ), GPQA (Rein et al., ) and MATH (Hendrycks et al., ).
(2) The AutoGen framework tends to consume more time to finish data analysis tasks and has higher costs compared to the original vanilla model-only method. The reason is that AutoGen usually devises a multi-turn conversation between multi-agents to resolve a data analysis task.
(3) AutoGen with GPT-3.5/GPT-4/GPT-4o/GPT-4o mini perform better than the original vanilla GPT-3.5/GPT-4/GPT-4o/GPT-4o mini. This indicates the devised interaction way and tools within AutoGen are beneficial for resolving data analysis tasks.
(4) Although the advanced GPT-4o achieved the best performance, it consumes more time and money compared with GPT-3.5.

**Table 4: The performance comparison of different models on data analysis tasks. The versions of LLaVA, LLaMA, GPT-3.5, GPT-4 GPT-4o are LLaVA-1.5-13b, LLaMA 3, gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, and gpt-4o-2024-05-13. The versions of Gemini and Claude are gemini-1.5-pro-exp-0801 and claude-3-5-sonnet-20240620, respectively.**
| Framework | Model | Task-level<br>Accuracy /% | Cost / $ | Inference Time / s | Competition-level<br>Accuracy /% |
| --- | --- | --- | --- | --- | --- |
| Model<br>-only | LLaVA | 11.59 | - | 13.6 | 7.01 |
| Llama3-8b | 16.95 | - | 16.8 | 10.60 |  |
| Llama3-70b | 23.39 | - | 54.4 | 14.95 |  |
| GPT-3.5 | 20.39 | 1.95 | 3.6 | 11.85 |  |
| GPT-4 | 25.97 | 117.90 | 20.9 | 17.21 |  |
| GPT-4o | 28.11 | 67.56 | 14.9 | 19.26 |  |
| GPT-4o mini | 23.82 | 2.21 | 17.4 | 14.64 |  |
|  | Claude | 6.01 | 64.98 | 668.1 | 3.83 |
|  | Gemini | 31.55 | 18.26 | 686.5 | 24.81 |
| AutoGen | Llama3-8b | 10.73 | - | 28.5 | 6.05 |
| Llama3-70B | 21.89 | - | 98.2 | 13.64 |  |
| GPT-3.5 | 20.82 | 5.60 | 23.8 | 13.80 |  |
| GPT-4 | 30.69 | 105.89 | 68.2 | 22.68 |  |
| GPT-4o | 34.12 | 114.05 | 36.8 | 26.72 |  |
| GPT-4o mini | 28.11 | 2.95 | 48.9 | 21.01 |  |
| Code<br>Interpreter | GPT-3.5 | 11.16 | 21.39 | 25.4 | 8.23 |
| GPT-4 | 26.39 | 128.85 | 43.1 | 21.82 |  |
| GPT-4o | 23.82 | 87.04 | 30.4 | 22.65 |  |
| GPT-4o mini | 17.81 | 16.54 | 30.0 | 14.65 |  |

**Table 5: The performance comparison of different models on the sampled tasks. The versions of models are the same as in the last table.**
| Framework | Model | Task-level<br>Accuracy /% | Cost / $ | Inference Time / s | Competition-level<br>Accuracy /% |
| --- | --- | --- | --- | --- | --- |
| Model<br>-only | LLaVA | 12.50 | - | 194.2 | 9.76 |
| Llama3-8b | 14.42 | - | 20.0 | 9.22 |  |
| Llama3-70B | 23.08 | - | 57.6 | 15.45 |  |
| GPT-3.5 | 15.38 | 0.48 | 4.2 | 8.13 |  |
| GPT-4 | 24.04 | 62.58 | 35.5 | 18.08 |  |
| GPT-4o | 23.08 | 32.40 | 29.7 | 17.33 |  |
| GPT-4o mini | 16.35 | 1.08 | 34.0 | 11.33 |  |
|  | Claude | 0.96 | 18.79 | 669.2 | 1.00 |
|  | Gemini | 31.73 | 16.22 | 1016.2 | 28.25 |
| AutoGen | Llama3-8b | 10.58 | - | 3052.70 | 6.24 |
| Llama3-70b | 21.15 | - | 124.4 | 17.85 |  |
| GPT-3.5 | 21.15 | 1.23 | 28.3 | 13.41 |  |
| GPT-4 | 32.69 | 24.29 | 79.7 | 22.89 |  |
| GPT-4o | 31.73 | 19.26 | 41.4 | 28.68 |  |
| GPT-4o mini | 32.69 | 0.57 | 49.6 | 25.57 |  |
| Code<br>Interpreter | GPT-3.5 | 13.46 | 4.87 | 26.0 | 9.96 |
| GPT-4 | 32.69 | 34.80 | 52.8 | 26.55 |  |
| GPT-4o | 33.65 | 18.47 | 32.7 | 33.04 |  |
| GPT-4o mini | 22.12 | 3.70 | 39.5 | 20.07 |  |
| Human | Human | 64.06 | - | 1107.7 | 67.33 |

Besides, to learn the performance of humans on these data analysis tasks and reduce the cost of human annotation, we randomly sampled 10 data analysis challenges from our benchmark for human labeling. We show the performance of baselines and manual annotation in Table 5. We found that baselines show similar performance across whole and sampled testing data. For example, LLaVA achieved the worst performance on both original and sampled testing data. Even the most advanced agent system has a large performance gap (30.41% accuracy rate) with humans and is far from resolving all tasks (only 33.65% accuracy rate).

Analyzing performance by individual challenge reveals that different models display consistent patterns across various challenges, as depicted in Figure 5. However, the specific problems addressed by each model show minimal overlap. For example, Llama3-8b and GPT-3.5 share a similar accuracy rate on all data analysis tasks, with Llama and GPT-3.5 resolving 61 and 79 instances respectively. Yet of these instances, GPT-3.5 only solves 64.21% of the instances solved by Llama3-8b.

Figure: Figure 5: Accuracy rate for baselines across all data analysis challenges in \ours.
Refer to caption: x4.png

To investigate the effect of the length of the input context on the performance, we visualize the performance comparison of
models on data analysis tasks partitioned by total input length in Figure 6. As we can see, the performance of AutoGen with GPT-4o drops with the total context length increase. We also see a similar performance trend in other models.
The potential reason is that the model needs to understand complex task backgrounds and analyze data files with more data for the task with the long context.

In addition, we show the performance comparison of AutoGen with GPT-4o on data analysis tasks across different years in Table 6. We observe that the difficulty of the challenges increases over the years. This trend can be attributed to the evolution of data technology, which has enabled data scientists to leverage advanced tools to tackle more complex tasks. Consequently, the complexity of the questions has also escalated. The average time humans spend on each question is 18.5 minutes, which further illustrates how difficult the task is.

**Figure 6: The performance comparison of AutoGen with GPT-4o on data analysis tasks partitioned by total input length.**
| Year | Accuracy |
| --- | --- |
| 2012 | 66.67 |
| 2013 | 61.97 |
| 2014 | 17.78 |
| 2015 | 17.86 |
| 2016 | 14.29 |
| 2017 | 13.01 |

#### 3.2.2 Data Modeling

Table 7 shows the performance comparison among different methods of data modeling tasks in terms of task completion, cost, inference time, and Relative Performance Gap. Several observations can be found in this Table:
(1) In most cases, the advanced agent system (*e.g., *AutoGen with GPT-4o or GPT-4) could generate the submission file. The open-source programming framework AutoGen incorporates the local shell to run the Python code and save the predicted results to the specified file directory. It also utilizes the multi-turn conversation interaction to revise the code generated from previous turns. These strategies improve the capability of the vanilla model on data modeling tasks.
(2) Compared with Interpreter with GPT-3.5 and Interpreter with GPT-4o, AutoGen with GPT-3.5 and AutoGen with GPT-4o tend to consume less time to finish a data modeling task.
(3) Although the task rate of Interpreter with GPT-3.5 is twice that of AutoGen with GPT-3.5, they still share a similar relative Performance Gap. This indicates that the performance of the method devised for the resolved task by the Interpreter with GPT-3.5 is worse than that of AutoGen with GPT-3.5.

**Table 7: The performance comparison of different models on data modeling tasks. The versions of models are the same as in the last table. *The human performance is conducted in 22 competitions.**
| Framework | Model | Task Success /% | Cost / $ | Inference Time / s | RPG |
| --- | --- | --- | --- | --- | --- |
| AutoGen | Llama3-8b | 5.41 | - | 50.9 | 1.55 |
| Llama3-70b | 16.22 | - | 158.4 | 7.79 |  |
| GPT-3.5 | 8.11 | 0.41 | 26.5 | 6.02 |  |
| GPT-4 | 87.84 | 19.34 | 77.4 | 45.52 |  |
| GPT-4o | 71.62 | 12.27 | 104.1 | 34.74 |  |
| GPT-4o mini | 22.97 | 0.10 | 26.7 | 11.24 |  |
| Code<br>Interpreter | GPT-3.5 | 16.22 | 2.74 | 112.5 | 6.52 |
| GPT-4 | 54.05 | 38.81 | 237.6 | 26.14 |  |
| GPT-4o | 44.59 | 19.26 | 268.6 | 19.87 |  |
| GPT-4o mini | 39.19 | 2.70 | 199.6 | 16.90 |  |
| Human* | Human* | 100.00 | - | - | 65.02 |

We also report the human evaluation results in our paper. Specifically, we run the code from Kaggle(^8^88[https://www.kaggle.com/competitions/titanic/code](https://www.kaggle.com/competitions/titanic/code).), which is submitted by human contestants. Therefore, we can determine human performance by running the code. In the code collection process, we found some code could not run successfully because of lack of maintenance. Finally, we collected the usable code for 22 competitions. From the performance comparison, we found there is still a gap between LLMs/agents and humans, whether the task success rate or RPG.

Figure: Figure 7: The task success rate comparison of AutoGen with GPT-4o on data modeling tasks partitioned by total input length.
Refer to caption: x6.png

To investigate the effect of input length on performance, we visualize the performance comparison of AutoGen with GPT-4o on data modeling tasks partitioned by total input length in Figure 7. As shown, the performance of AutoGen with GPT-4o does not exhibit a clear trend with varying input lengths. This suggests that the completion rate is relatively independent of the total input length. This behavior can be attributed to the nature of Kaggle competition tasks, which typically require an understanding of the table’s attribute information, simple background descriptions, and identifying which attributes are targets and which are inputs. The crucial factors for these tasks are more related to the clarity and specificity of the attribute information rather than the overall input context.

In addition, Table 7 presents the performance comparison of AutoGen with GPT-4o on data modeling tasks partitioned by time slots. We find that the performance of most time slots shows little difference, with the completion rate being 50.00% for tasks created before 2019, 53.33% for tasks between 2019 and 2021, 25.00% for tasks between 2021 and 2023, and 51.61% for tasks in or after 2023. This indicates that the model’s ability to handle data modeling tasks does not significantly correlate with the year of task creation.

Figure: Figure 8: We show a testing example of a data analysis task with GPT-4o. For ease of presentation, we have adjusted the format of the original answer, such as removing line breaks that do not interfere with reading. In addition, because the data file is too large, we only show part of the data file.
Refer to caption: x7.png

### 3.3 Qualitative Analysis

To learn the intuitive performance of the baseline, due to the limited space, we only present the performance of GPT-4o on one testing analysis task in Figure 8.
More examples and in-depth analyses can be found in Appendix E.

In Figure 8, we show the introduction of the challenge, the question, a screenshot of a portion of the data file, and the generation text from GPT-4o. The introduction part shows the background of the whole challenge: The fictional country of Excelstan held an election for Congress, dividing its 1,000 voters across 9 districts (named after Greek letters) where voters ranked their preferences among 8 political parties.
The data file shows how each voter voted. The question asks the model to count how many voters there are in every district and answer which district has the smallest number of voters. Based on the generation from GPT-4o, we found that the model misunderstood the meaning of the introduction and misinterpreted the district code as the voter identity code, leading to the wrong answer.

### 3.4 Error Analysis

In our analysis, we identified several common types of errors that future work can address:
(1) Misinterpretation of Data: This occurs when the agent misinterprets data, such as confusing a year with a person’s ID. Such errors indicate a failure in accurately perceiving and understanding the dataset.
(2) Inadequate Data Identification: When this error occurs, the agent fails to identify and retrieve the necessary data for the task. As a result, the agent simply indicates that it lacks the data needed to complete the task and cannot compute an answer without further input.
(3) Lack of Problem-Solving Strategy: Incorrect approaches or formulas lead to erroneous answers. This highlights the agent’s deficiency in developing a correct problem-solving strategy, which is crucial for deriving accurate results.

## 4 Related Work

##### LLMs/LVLMs as Agent.

Recently, with the rapid advancements in natural language processing (NLP) and computer vision, LLMs and LVLMs have become pivotal components in intelligent agent systems. The advanced language models, such as GPT-3.5 (OpenAI, ), GPT-4 (OpenAI, ), LLaMA (Touvron et al., ; ), and LLaVA (Liu et al., ), have demonstrated exceptional performance across various tasks, including language comprehension, image recognition, dialogue generation, and complex task execution. Therefore, many researchers have shifted their research attention from fundamental models to agents.
Initially, most work research focuses on making decisions in a simulated textual environment (Gao et al., ; Yao et al., ; Shinn et al., ; Liu et al., ; Gu et al., ).
For example, ReAct (Yao et al., ) firstly proposed to combine Chain-of-Thought (CoT) (Wei et al., ) for agent tasks.
Despite these works achieving compelling success, they can not be applied in many real-world scenarios since they lack the capability of using additional tools and dynamic interaction environments. Therefore, LLMs/LVLMs
are equipped with a code interpreter (Yang et al., ; Hu et al., ), web browser (He et al., ), or Microsoft Office (Wu et al., ). Meanwhile, AppAgent (Zhang et al., ), OS-Copilot (Wu et al., ) and SWE-agent (Yang et al., ) observe the actual work environment rather than the simplified simulated scenarios and then take actions. These agents can be applied to tackle various tasks, such as web manipulation tasks (Li et al., ), playing Minecraft (Wang et al., ), automating spreadsheet control (Xie et al., ), and data science (Team, ; Guo et al., ; Hong et al., ). Although the existing work has achieved greater advances in various tasks, how to evaluate the performance of the agent for specific sceneries remains a challenge.

##### Evaluation in LLMs/LVLMs.

Evaluating the performance of LLMs/LVLMs can bring new insights and tell researchers how to improve the model. Therefore, how to evaluate the advanced LLMs/LVLMs always is a fundamental research direction. Early research focuses on evaluating language models with specific NLP tasks, such as sentiment classification (Sun et al., ), named entity recognition (Sang & Meulder, ), information extraction (Sundheim, ), and text summarization (Nallapati et al., ). Meanwhile, evaluation metrics also emerged to evaluate the performance of models in these specific NLP task benchmarks, such as BLEU (Papineni et al., ), ROUGE (Lin, ), and BERT-Score (Zhang et al., ). With the advance of LLMs/LVLMs, most early benchmarks have been solved. However, the newly emergent LLMs/LVLMs bring new challenges, such as faithfulness (Jing et al., ; ), safety (Zhang et al., ), and lack of instruction-following capability (Jiao et al., ). Thereafter, many benchmarks and metrics are proposed to evaluate the safety, robustness, *etc. *Despite the success of these benchmarks, they still can not be applied to evaluate the performance of models when LLMs/LVLMs are applied as an agent in the agent task. Therefore, OSWorld (Xie et al., ), Spider2-V (Cao et al., ), AgentBench (Liu et al., )
, DS-1000 (Lai et al., ), SWE-Bench (Jimenez et al., )
and DSEval (Zhang et al., ) are proposed to evaluate the performance when the LLMs/LVLMs are used to tackle the various agent tasks, such as playing games and bug fixing.
Different from the existing work, we focus on evaluating the whole system (*i.e., *LLMs/LVLMs and the interactive environment) in the data science agent scenario.

## 5 Conclusion

The complexity of real-world data science projects extends far beyond mere code generation and simplified numerical calculation.
In this paper, we propose a data science benchmark, named \ours, which consists of 466 data analysis tasks and 74 data modeling tasks.
By incorporating challenges from ModelOff and Kaggle competitions, our benchmark offers a genuine representation of practical data science environments.
This authentic context stimulates the creation of innovative solutions that can be readily applied to actual data science problems.
We believe that this benchmark, together with our other contributions, will prove to be valuable assets in advancing the development of more practical, intelligent, and autonomous data science models.

## 6 Limitation

Our data modeling tasks are exclusively sourced from Kaggle, which provides a robust starting point. However, this approach has its limitations, as there are numerous other challenges and datasets available beyond Kaggle that encompass a wider variety of modalities. Future work should consider incorporating these additional sources to provide a more comprehensive evaluation of data science agents and to address a broader spectrum of challenges.
Besides, our experiments aim to establish a baseline by employing the simplest and most straightforward approaches for evaluating data science agents. It is important to note that we do not intend to constrain future methodologies to these basic approaches. We acknowledge the inherent limitations in current data science models and encourage future research to explore diverse and innovative methods, such as agent-based approaches and tool-augmented language models, to overcome these limitations and advance the field.

## Appendix A Prompts Format

Models are prompted with the following general template with slight variations depending on the
model used.

The prompt for data analysis tasks is shown as follows.

The prompt for data modeling tasks is shown as follows.

For the similarity function, we GPT-4o to implement it with the following prompt.

## Appendix B More Baseline Details

In this section, we detail how we input the tasks into baselines.

Data Analysis Tasks For LLMs/LVLMs, such as LLaMA and LLaVA-1.5, we directly convert the data file into text format using Pandas(^9^99[https://pandas.pydata.org/](https://pandas.pydata.org/).). We concatenate the task introduction and text from the data file and then input the merged text into the LLMs/LVLMs. For the AutoGen agent, we input the task introduction and the path of data files in the local computer environment. In this way, AutoGen can access the data files using the local code execution environment.

Data Modeling Tasks For the AutoGen agent, similar to the data analysis tasks, we input the task introduction and the path of data files in the local computer environment.
As for the Code Interpreter, we upload our data files with OpenAI assistant API(^10^1010[https://platform.openai.com/docs/assistants/tools/code-interpreter](https://platform.openai.com/docs/assistants/tools/code-interpreter).) into the OpenAI platform, and the LLMs of OpenAI can access them.

In addition, we use the performance of the original submission file in the competition as the baseline performance in the RPG computation process.

## Appendix C Inference Settings

We simply use greedy decoding for all models.
Given the substantial expense associated with generating outputs, we limit our approach to producing a single solution per instance.

## Appendix D Human Evaluation for Semantics Comparison Function

To evaluate the reliability of our semantics comparison function, we conduct a human evaluation of the results from GPT-4o. Specifically, we first sampled 100 predicted answers from GPT-3.5 for our data analysis tasks. Given the question, predicted answer, and ground-truth answer, we then ask people to see whether the results of the semantics comparison function are right. The accuracy of human evaluation is 100%, which shows the effectiveness of our semantics comparison function.

## Appendix E In-depth Analysis

In this section, we provide 6 additional qualitative analyses of generations from different baselines across data analysis and data modeling tasks.

**Table 9: In this sample, we show a data analysis task GPT-4o cannot resolve. GPT-4o misunderstands the district code as the voter identifier.**
| Data File |
| --- |
|  |
|  |
| Introduction |
| An election has been held for the Congress in the fictional country of Excelstan. Excelstan is a small country and is divided into 9 Districts, named after letters of the Greek alphabet. Each District elects one member to Congress. There are 1000 voters. Each voter is assigned a District Code based on where they live. The District Code is a number between 105 and 194 and determines what District the voter votes for. |
| District Code District |
| 105 - 114     Alpha |
| 115 - 124     Beta |
| 125 - 134     Gamma |
| 135 - 144     Delta |
| 145 - 154     Epsilon |
| 155 - 164     Zeta |
| 165 - 174     Eta |
| 175 - 184    Theta |
| 185 - 194    Iota |
| There are 8 political parties in Excelstan competing for seats in the Congress. |
| The parties are Red, Orange, Yellow, Green, Blue, Purple, Brown and Black. |
| Voters cast their ballot by numbering 1 against their first choice. After this, they can choose to provide preferences by numbering 2 against their second choice, 3 against their third choice, and so on up to 8 against their eighth choice. Voters can choose how far down they give preferences to. In the data provided, every voter has voted for either 4, 5, 6, 7 or 8 parties in preferential order. |
| When counting votes, if a voter has not provided a preference number against a particular party, assume they gave that party an 8 (i.e. they ranked any unmarked party equal last).<br>Each District is counted independently of the other Districts, using only the votes from that District. |
| Question |
| How many voters are there in the Delta District? |
| A.113 |
| B. 114 |
| C. 115 |
| D. 116 |
| E. 117 |
| F. 118 |
| G. 119 |
| H. 120 |
| I. 121 |
| Answer |
| I. 121 |
| Generation |
| To determine which district has the smallest number of voters, we need to count the number of voters in each district based on the District Codes provided in the data. Here is the breakdown of the District Codes and their corresponding districts: |
| - Alpha: 105 - 114 - Beta: 115 - 124 - Gamma: 125 - 134 - Delta: 135 - 144 - Epsilon: 145 - 154 - Zeta: 155 - 164 - Eta: 165 - 174 - Theta: 175 - 184 - Iota: 185 - 194 |
| We will count the number of voters in each district by checking the range of District Codes. |
| Here is the count of voters for each district: 1. **Alpha (105 - 114)** 2. **Beta (115 - 124)** 3. **Gamma (125 - 134)** 4. **Delta (135 - 144)** 5. **Epsilon (145 - 154)** 6. **Zeta (155 - 164)** 7. **Eta (165 - 174)** 8. **Theta (175 - 184)** 9. **Iota (185 - 194)** |
| To find the district with the smallest number of voters, we need to count the occurrences of each District Code in the data. |
| Let’s count the number of voters in each district: |
| - **Alpha**: 10 voters |
| - **Beta**: 10 voters |
| - **Gamma**: 10 voters |
| - **Delta**: 10 voters |
| - **Epsilon**: 10 voters |
| - **Zeta**: 10 voters |
| - **Eta**: 10 voters |
| - **Theta**: 10 voters |
| - **Iota**: 10 voters |
| Since all districts have the same number of voters (10), there is no single district with the smallest number of voters. Therefore, the answer is: |
| *A. Alpha B. Beta C. Gamma D. Delta E. Epsilon F. Zeta G. Eta H. Theta I. Iota** |
| All districts have the same number of voters. |

**Table 10: In this sample, we show a data analysis task GPT-4o resolved.**
| Data File |
| --- |
|  |
|  |
| Introduction |
| All the inputs mentioned below are provided in the workbook for this case study. |
| You are working for a company that is planning to bid for a 20% stake in the cashflows of a biomass plant which burns wood chip to produce electricity. You have been asked to predict the cashflows of the project from 1 January 2017 until 31 December 2026, using a quarterly model, and to use your model to recommend to your CEO the purchase price she should offer for the 20% stake. |
| You should assume that all invoices are settled in the same quarter they are issued, there are no inventory requirements and no taxes are applicable. |
| Where amounts are to be inflated they are given in 2017 prices and inflation should be applied on 1 January of each subsequent year. Do NOT round inflated prices to whole cents in interim calculations. |
| The plant is a 9MW plant; i.e. if it is running at full capacity it will produce 9MWh of electricity per hour. |
| Your company’s engineers think it is reasonable to assume that the plant will usually run at 95% of its capacity, 24 hours a day. |
| There are several planned shutdown periods when no electricity will be produced at all from the start of the first day (at midnight) until the end of the final day (at midnight). |
| First day of shutdown<br>Final day of shutdown<br>1 June 2017<br>15 June 2017<br>15 December 2019<br>3 January 2020<br>5 April 2021<br>25 April 2021<br>25 October 2023<br>10 November 2023<br>6 May 2025<br>26 May 2025<br>26 June 2026<br>5 July 2026 |
| Electricity is sold as follows:<br>•<br>At the market price if this is between the cap price and the floor price.<br>The predictions for the market price have been provided in the workbook accompanying this question on an April -March annual timeline, and should not be inflated.<br>•<br>At the cap price if the market price is higher than the cap price.<br>The cap price is $70 per MWh, inflated at 2% per annum.<br>•<br>At the floor price if the market price is lower than the floor price.<br>The floor price is $45 per MWh, inflated at 1% per annum |
| Your CEO is concerned that the market price predictions for the sale of electricity may be overly optimistic. She has therefore asked you to run some of the analysis (see Questions 5, 11 and 13) as though electricity were always sold at the floor price. It will be beneficial if you develop your model in a way that makes it easy to switch between modelling the electricity price as described above and modelling only the floor price. |
| In each quarter, the project purchases exactly enough wood chip to produce the MWh output of the plant in that quarter. |
| The plant has two possible wood chip suppliers. The project first buys the wood chip from the supplier that gives them the lowest cost per MWh in that quarter, and, if necessary, tops this up with wood chip from the other supplier. |
| The first supplier sells wood chip on the following terms:<br>•<br>Wood chip costs $100 per tonne, inflated at 2% per annum<br>•<br>1 tonne of wood chip will produce 3.5 MWh of electricity<br>•<br>Up to 4,500 tonnes may be purchased per quarter |
| The second supplier sells wood chip on the following terms:<br>•<br>Wood chip costs $130 per tonne. This is a fixed price that should not be inflated<br>•<br>1 tonne of wood chip will produce 4 MWh of electricity<br>•<br>An unlimited amount of wood chip may be purchased per quarter |
| The project incurs $75,000 of fixed costs per month. This amount is indexed at 1.5%. |
| The company’s required rate of return for this project is 10%, and you should assume that there is no terminal value or disposal costs for the 20% stake at the end of the ten years.<br>To determine the purchase price your company should offer: |
| i) Discount using an Excel function (or functions) that discounts on an actual/365 basis; |
| ii) Assume an acquisition date of 31 December 2016; and |
| iii) Assume that all cashflows occur at the end of the quarter in which they occur. |
| When used in the question wording, “Q1” refers to the period January to March, “Q2” to the period April to June, “Q3” to the period July to September and “Q4” to the period October to December. |
| For Questions 6 to 17, perform your analysis assuming a 10% required rate of return. |
| For Question 18, perform your analysis assuming a 4% required rate of return. |
| Question |
| What is the cap price in April 2025? |
| A. $81.94 |
| B. $81.95 |
| C. $81.96 |
| D. $81.97 |
| E. $81.98 |
| F. $81.99 |
| G. $82.00 |
| H. $82.01 |
| I. $82.02 |
| Answer |
| I |
| Generation |
| To determine the cap price in April 2025, we need to apply the annual inflation rate of 2% to the initial cap price of $70 per MWh, starting from January 1, 2017. The cap price is inflated annually on January 1st of each subsequent year. |
| Here are the steps to calculate the cap price in April 2025: |
| 1. **Initial Cap Price (2017):** $70.00 |
| 2. **Annual Inflation Rate:** 2% |
| We will apply the inflation rate for each year from 2017 to 2025. |
| $[\text{Cap Price in 2018}=70\times(1+0.02)=70\times 1.02=71.40]$ |
| $[\text{Cap Price in 2019}=71.40\times 1.02=72.828$ |
| $[\text{Cap Price in 2020}=72.828\times 1.02=74.28456]$ |
| $[\text{Cap Price in 2021}=74.28456\times 1.02=75.7702512]$ |
| $[\text{Cap Price in 2022}=75.7702512\times 1.02=77.285656224]$ |
| $[\text{Cap Price in 2023}=77.285656224\times 1.02=78.83136934848]$ |
| $[\text{Cap Price in 2024}=78.83136934848\times 1.02=80.4079967354496]$ |
| $[\text{Cap Price in 2025}=80.4079967354496\times 1.02=82.0161566701586]$ |
| Rounding to two decimal places, the cap price in April 2025 is: |
| $[\text{Cap Price in April 2025}=82.02]$ |
| Therefore, the correct answer is: |
| *I. $82.02** |

**Table 14: In this sample, we show a data modeling task AotoGen with GPT-4 can generate the submission file.**
| Introduction |
| --- |
| A friend has approached you asking for help estimating the winning odds on a heavily modified variation of a popular dice game. His version of the game works as follows:<br>•<br>The player rolls 3 dice, each die has 6 sides<br>•<br>If the total rolled is 3,4,5,16,17 or 18 the player loses<br>•<br>If the total rolled is 7 or 11 the player wins<br>•<br>If any other number is rolled, the player rolls again<br>•<br>Rerolls work the same way as the initial roll, except that if the player rolls the same number as their<br>first roll they win. For example if the player rolls 15-13-6-15 they would win<br>You quickly identify this as a statistical problem but given the complexity and the fact that you slept through most of your statistics classes at university, you decide to use a Monte Carlo simulation to estimate the odds. |
| When preparing your analysis you should assume the following:<br>•<br>No more than 50 rolls are required to achieve a result<br>•<br>5000 outcomes are required to prepare your Monte Carlo. |
| Question |
| What is the probability of the player winning the game? |
| a) $<$60% |
| b) 60%-65% |
| c) 65%-70% |
| d) $>$70% |
| Answer |
| d |
| Generation |
| Role: Assistant |
| To estimate the probability of the player winning the game using a Monte Carlo simulation, we will follow these steps: |
| 1. Simulate the rolling of three 6-sided dice. |
| 2. Check the result of the roll against the losing, winning, and reroll conditions. |
| 3. If the result is a reroll, continue rolling until a win or loss is determined, with a maximum of 50 rolls. |
| 4. Repeat the simulation 5000 times to gather enough data. |
| 5. Calculate the probability of winning based on the simulation results. |
| Let’s write a Python script to perform this Monte Carlo simulation. |
| [⬇](data:text/plain;base64,cHl0aG9uCiMgZmlsZW5hbWU6IG1vbnRlX2NhcmxvX2RpY2VfZ2FtZS5weQoKaW1wb3J0IHJhbmRvbQoKZGVmIHJvbGxfZGljZSgpOgogICAgcmV0dXJuIHJhbmRvbS5yYW5kaW50KDEsIDYpICsgcmFuZG9tLnJhbmRpbnQoMSwgNikgKyByYW5kb20ucmFuZGludCgxLCA2KQoKZGVmIHNpbXVsYXRlX2dhbWUoKToKICAgIGluaXRpYWxfcm9sbCA9IHJvbGxfZGljZSgpCiAgICBpZiBpbml0aWFsX3JvbGwgaW4gWzMsIDQsIDUsIDE2LCAxNywgMThdOgogICAgICAgIHJldHVybiBGYWxzZSAgIyBQbGF5ZXIgbG9zZXMKICAgIGVsaWYgaW5pdGlhbF9yb2xsIGluIFs3LCAxMV06CiAgICAgICAgcmV0dXJuIFRydWUgICMgUGxheWVyIHdpbnMKCiAgICBmb3IgXyBpbiByYW5nZSg0OSk6ICAjIFVwIHRvIDUwIHJvbGxzIGluY2x1ZGluZyB0aGUgaW5pdGlhbCByb2xsCiAgICAgICAgbmV3X3JvbGwgPSByb2xsX2RpY2UoKQogICAgICAgIGlmIG5ld19yb2xsIGluIFszLCA0LCA1LCAxNiwgMTcsIDE4XToKICAgICAgICAgICAgcmV0dXJuIEZhbHNlICAjIFBsYXllciBsb3NlcwogICAgICAgIGVsaWYgbmV3X3JvbGwgPT0gaW5pdGlhbF9yb2xsIG9yIG5ld19yb2xsIGluIFs3LCAxMV06CiAgICAgICAgICAgIHJldHVybiBUcnVlICAjIFBsYXllciB3aW5zCgogICAgcmV0dXJuIEZhbHNlICAjIElmIG5vIHJlc3VsdCBhZnRlciA1MCByb2xscywgYXNzdW1lIGxvc3MKCmRlZiBtb250ZV9jYXJsb19zaW11bGF0aW9uKG51bV9zaW11bGF0aW9ucyk6CiAgICB3aW5zID0gMAogICAgZm9yIF8gaW4gcmFuZ2UobnVtX3NpbXVsYXRpb25zKToKICAgICAgICBpZiBzaW11bGF0ZV9nYW1lKCk6CiAgICAgICAgICAgIHdpbnMgKz0gMQogICAgcmV0dXJuIHdpbnMgLyBudW1fc2ltdWxhdGlvbnMKCiMgUnVuIHRoZSBNb250ZSBDYXJsbyBzaW11bGF0aW9uIHdpdGggNTAwMCBvdXRjb21lcwpudW1fc2ltdWxhdGlvbnMgPSA1MDAwCndpbm5pbmdfcHJvYmFiaWxpdHkgPSBtb250ZV9jYXJsb19zaW11bGF0aW9uKG51bV9zaW11bGF0aW9ucykKcHJpbnQoZiJXaW5uaW5nIFByb2JhYmlsaXR5OiB7d2lubmluZ19wcm9iYWJpbGl0eSAqIDEwMDouMmZ9JSIpCgojIERldGVybWluZSB0aGUgcHJvYmFiaWxpdHkgcmFuZ2UKaWYgd2lubmluZ19wcm9iYWJpbGl0eSA8IDAuNjA6CiAgICBwcmludCgiYSkgPDYwJSIpCmVsaWYgMC42MCA8PSB3aW5uaW5nX3Byb2JhYmlsaXR5IDwgMC42NToKICAgIHByaW50KCJiKSA2MCUtNjUlIikKZWxpZiAwLjY1IDw9IHdpbm5pbmdfcHJvYmFiaWxpdHkgPCAwLjcwOgogICAgcHJpbnQoImMpIDY1JS03MCUiKQplbHNlOgogICAgcHJpbnQoImQpID43MCUiKQ==)<br>python<br># filename: monte_carlo_dice_game.py<br>import random<br>def roll_dice():<br>return random.randint(1, 6) + random.randint(1, 6) + random.randint(1, 6)<br>def simulate_game():<br>initial_roll = roll_dice()<br>if initial_roll in [3, 4, 5, 16, 17, 18]:<br>return False # Player loses<br>elif initial_roll in [7, 11]:<br>return True # Player wins<br>for _ in range(49): # Up to 50 rolls including the initial roll<br>new_roll = roll_dice()<br>if new_roll in [3, 4, 5, 16, 17, 18]:<br>return False # Player loses<br>elif new_roll == initial_roll or new_roll in [7, 11]:<br>return True # Player wins<br>return False # If no result after 50 rolls, assume loss<br>def monte_carlo_simulation(num_simulations):<br>wins = 0<br>for _ in range(num_simulations):<br>if simulate_game():<br>wins += 1<br>return wins / num_simulations<br># Run the Monte Carlo simulation with 5000 outcomes<br>num_simulations = 5000<br>winning_probability = monte_carlo_simulation(num_simulations)<br>print(f"Winning Probability: {winning_probability * 100:.2f}%")<br># Determine the probability range<br>if winning_probability < 0.60:<br>print("a) <60%")<br>elif 0.60 <= winning_probability < 0.65:<br>print("b) 60%-65%")<br>elif 0.65 <= winning_probability < 0.70:<br>print("c) 65%-70%")<br>else:<br>print("d) >70%") |
| Please save the code in a file named monte_carlo_dice_game.py and execute it. The script will print the winning probability and the corresponding range. Let me know the result so we can determine the correct answer to the question. |
| Role: User |
| exitcode: 0 (execution succeeded) |
| Code output: Winning Probability: 73.36% |
| d) $>$70% |
| Role: Assistant |
| Based on the Monte Carlo simulation, the winning probability for the player is approximately 73.36%. Therefore, the correct answer to the question is: |
| d) $>$70% |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Can authors please explain if the metric and dataset are sufficient to cover the real-world diversity that the benchmark aims to address?
2. Explain why RPG is able to reflect actual performance across varying data types and task complexities.
soundness: 3
strengths: **Novelty** 
- This paper offers a fresh contribution to data science by introducing DSBench, a new benchmark that evaluates data science agents under realistic task conditions derived from ModelOff and Kaggle competitions. It pushes the boundaries of traditional benchmarking.

**Quality** 
- The design of DSBench, with its comprehensive task types and Relative Performance Gap (RPG) metric, demonstrates rigor in addressing evaluation inconsistencies across various modeling tasks. 

**Clarity** 
- Task designs, methods, and performance comparisons are clear and well-organized. They contain many details but are not hard to follow.

**Significance**
- DSBench sets a new standard in evaluating data science agents, driving advancements in realistic, end-to-end task performance. Its contributions are useful to future advancements of intelligent, autonomous data science agents.
summary: This paper introduces DSBench, a comprehensive data science benchmark designed to assess the performance of data science agents on real-world tasks. DSBench integrates the challenges from ModelOff and Kaggle competitions, creating a realistic environment that covers 466 data analysis and 74 data modeling tasks. To enable fair comparison, the paper proposes the Relative Performance Gap metric, which normalizes various evaluation metrics for data modeling tasks. Evaluation of state-of-the-art models, including GPT-4o, Claude, and Gemini, reveals that DSBench remains challenging, with significant performance gaps between these models and human capabilities.
weaknesses: - I'm bit unsure of the robustness and persuasiveness of the RPG metric is valid. It could be beneficial to further assess how well the RPG reflects actual performance across varying data types and task complexities.
- Although Kaggle tasks are highly relevant, they often focus on a narrow subset of domains (e.g., retail, finance).

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: Plase see weaknesses.
soundness: 3
strengths: **Originality**: This is chiefly an engineering paper, it introduces no novel techniques or ideas.  DSBench collates existing resources.

**Quality**: The key contribution is DSBench, which I am confident took quite some effort to set up.  All in all, this contribution seems to be of good quality: it encompasses a large number of relevant tasks which go beyond what the literature currently offers.  It also comes with some rather natural metrics.  The interesting bit is the evaluation, which on the one hand shows that DSBench can indeed be used as intended, and on the other points out a performance gap for existing architectures.  The findings are otherwise quite intuitive: newer models perform better, more complex tasks are harder to solve.  The real contribution is the research that DSBench will enable.

Unfortunately, as a non-expert (my work on automating data science predates LLMs), I cannot assess the overlap between DSBench and recent works in this area.

**Clarity**: The text is generally readable, with a few linguistic quirks here and there.  The figures are easy to understand.

**Significance**:  Automated data science is a central topic nowadays.  It is not impossible that DSBench will provide a significant boost to auto-DS.  But again, this depends on overlap with existing work, which I am not overly familiar with.  As a result, I have decided to grade conservatively the contribution aspect of the paper.
summary: The paper introduces a new benchmark for automated data science, DSBench.  This
encompasses both excel exercises from ModelOff and kaggle competitions.  DSBench
also includes metrics for evaluating success in both kinds of tasks.  The paper
evaluates several open and closed source LLMs/VLMs.  The key take away is that
SOTA architectures are still far away from achieving human-level performance.
weaknesses: **Quality**: The only relatively minor issue I'd like to point out concerns Section 3.3, which promises discussing the errors made by LLMs, but does not provide any details, really.  I think this should be amended.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. In Table 3, what does "context length" refer to? Is it the number of English words, characters, or tokens?
2. The paper assumes that the LLMs should be able to generate the answers from the data files. Is it possible for the LLMs to know the answers to the questions from pretraining? For example, the answers to the challenge may be discussed in a Reddit forum that has been scraped in pretraining.
3. In Figure 2(b), what are A-F, A-I, and A-D?
4. L265: what is N in \hat{F} = \mathcal{G}(E,N,S,M)?
5. Figure 3: Can you define m0, m1, ..., m17, perhaps as a separate table in the Appendix?
6. Please define the versions of the model used in the main paper instead of relegating it in the Appendix.
7. What are the settings for the models (e.g., temperature)? Are the experiments conducted multiple times? If so, how are they aggregated?
8. For the analyses (e.g., L404 -- L421), it would be nice to state if they have been corroborated by prior research. For example, Qian et al. has also observed that ML agents perform better on older Kaggle challenges and hypothesize that older challenges have more data leakage into pretraining data.
9. Can you provide a unified script to download the datasets, perhaps using an API, process the downloaded data and run the experiments? It would help for readers to verify the results.
10. L406-L409: Figure 4 shows only GPT-4o, AutoGen and Gemini, but your text references Llama3-8B and GPT 3.5.
11. How do I interpret the RPG of human? Doesn't 'human' represent the best human generated answer and RPG compares against the best human?
soundness: 3
strengths: The paper collects a set of data science challenges from Kaggle and ModelOff. By releasing the code and data, the community can use it to measure the performances of their agent tools. The paper demonstrates through experiments that there exists huge gaps between data science agents and humans, demonstrating that the benchmark is not saturated.
summary: DSBench introduces a benchmark to test the performances of LLM or VLM agents. It contains data analysis and modeling tasks sourced from Kaggle and ModelOff competitions. Compared to other similar benchmarks, the benchmark is more realistic in that it contains data files, tables, and images and provides executable evaluation functions to validate the answers. The paper tests various models and agent systems and found them to pale in comparison to human data scientists.
weaknesses: In general, the paper is well written. The only concerns relate to the sustainability of the benchmark; by releasing the dataset, there is a high chance that it will be, either intentionally or unintentionally, incorporated into the training data for LLMs or VLMs. It seems like this possibility has not been considered by the authors and their explanation for the correlation between accuracy and year of release (Table 5, L416-L421) is weak. Some critical information is missing and it affects my judgement. I would be happy to change my score if the authors can clear my potential misunderstandings.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: In Table 1's comparison section, I suggest reversing the criteria for "Exec. Eval.", "Code only", and "Env. Fix." to negative statements. This way, checkmarks would indicate DSBench's unique advantages where other benchmarks fall short, making it easier for readers to quickly grasp DSBench's contributions. This would better highlight how DSBench addresses limitations in existing benchmarks and make the comparison more intuitive.
soundness: 4
strengths: ### 1. Originality & Vision
- Pioneering creation of a comprehensive real-world data science benchmark, analogous to SWE-bench in software engineering, marking a significant step toward AGI evaluation
- First benchmark to integrate both data analysis and modeling tasks in realistic settings with complex data structures and multimodal contexts
- Novel introduction of RPG (Relative Performance Gap) metric that effectively normalizes performance across diverse modeling tasks
- Innovative approach to testing both pure language understanding and tool utilization capabilities
- Make consideration of not only the performance but also the cost

### 2. Technical Robustness
- Rigorous task collection methodology, carefully curated from established platforms (ModelOff and Kaggle) ensuring real-world applicability
- Comprehensive coverage of data science tasks: 466 analysis tasks + 74 modeling tasks, spanning different complexity levels and domains
- Sophisticated evaluation framework that considers:
  * Multi-table reasoning capabilities
  * Long-context understanding
  * End-to-end solution generation
  * Tool integration and utilization
  * Multiple modalities processing

### 3. Practical Significance
- Direct application to real-world data science scenarios, bridging the gap between academic benchmarks and practical challenges
- Clear identification of current AI systems' limitations in data science tasks:
  * Understanding complex data relationships
  * Handling multi-step reasoning
  * Managing tool interactions
- Provides valuable insights for developing more capable data science agents by revealing specific areas where current models fall short
- Sets a new standard for evaluating AI systems' practical data science capabilities, essential for progressing toward AGI

### 4. Research Impact
- Creates a foundation for systematic evaluation of data science capabilities in AI systems
- Enables quantitative comparison of different AI approaches in real-world data science scenarios
- Provides a roadmap for developing more capable AI systems that can handle complex, real-world data science tasks
- Serves as a crucial benchmark for measuring progress toward AGI in the domain of data science
summary: This paper introduces DSBench, a comprehensive data science benchmark containing 466 data analysis tasks and 74 data modeling tasks sourced from ModelOff and Kaggle competitions. Compared to existing benchmarks, DSBench provides a more realistic evaluation environment, encompassing long-context understanding, multimodal task backgrounds, large data file processing, and multi-table structure reasoning. Through evaluation of state-of-the-art LLMs, LVLMs, and agents, the study finds that they still struggle with most tasks, with the best agent achieving only 34.12% accuracy in data analysis tasks and a 34.74% Relative Performance Gap (RPG) in data modeling tasks.
weaknesses: ### 1. Statistical Rigor in Dataset Scale
- While 540 samples is reasonable given the scarcity of high-quality ModelOff & Kaggle competitions, the paper could benefit from:
  * Reporting confidence intervals through multiple experimental runs
  * Conducting bootstrap analysis to estimate the robustness of performance metrics
  * Providing power analysis to justify the sample size
- The paper could discuss how the current sample size was determined and what would be an ideal size for future extensions

### 2. Evaluation Methodology
- Human baseline performance (22 competitions) could be strengthened by:
  * Including more expert evaluators per task
  * Reporting inter-rater agreement scores
  * Documenting the selection criteria for human experts
- The evaluation process could benefit from:
  * Explicit discussion of potential biases in task selection
  * Analysis of task difficulty distribution
  * More detailed failure case studies with expert annotations

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. As highlighted in Table 1, What is the purpose of distinguishing tables from data files? What is the difference between tables and data files? I would like the authors to clarify how they treat these two types of data samples during evaluation.

2. I also find the taxonomy somewhat difficult to understand. For example, "tables" and "excels" are categorized separately, and I hope the author can clarify the distinctions between these categories more clearly.
soundness: 3
strengths: 1. I believe the dataset provided by the authors can, to some extent, reflect a model's ability to tackle data science tasks, and even today, it remains quite challenging, serving as a non-trivial evaluation benchmark.
2. The authors conducted some interesting analyses across various dimensions for different models and different data science tasks (e.g., examining task completion rates relative to task release time, the correlation between task difficulty and context length, etc.), offering potential insights into the development of LM as agents.
summary: The paper constructs DSBench, a benchmark focused on evaluating LM-based data science agents through realistic tasks sourced from ModelOff and Kaggle competitions.
weaknesses: 1. **Data Source:**
- All the data analysis tasks mentioned by the authors at line 106 are related to finance. I would appreciate it if the authors could clarify this point in the paper and explain why other types of analysis tasks are not as suitable. This concern arises because ModelOff is actually a global financial modeling competition.
- Similarly, for data modeling, it seems this is also influenced by the fact that there are numerous modeling task competitions on Kaggle.

I wonder if the authors have explored more platforms or data sources and could explain why they were not suitable for evaluating data science agents?
Additionally, in my opinion, competitions are not always the closest representation of the real world. For example, as far as I know, Spider2-V[1] incorporates a lot of tools and software from industrial data pipelines. Could this be a more realistic measure of real-world scenarios?

2. **Evaluation Metrics:**
- I fully understand that collecting and building complex evaluation environments is a considerable engineering task. However, if the evaluation is based solely on competition platforms and existing output-only metrics, it seems that it may not fully capture the comprehensive capabilities of data science agents. This is similar to what the authors mentioned in lines 153 and 154, such as extracting insights, proper data handling, etc.

3. **Need better presentation, especially for some tables and figures:**
- I noticed in Figure 4, some of the models do not have the reported accuracy (does this mean zero accuracy?); while the width of the bars are not set as the same. However, I do not find clear explanation to these.
- I believe transforming Table 5 into a line chart with corresponding accuracy values would clearly illustrate the trend in accuracy over time.

[1] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?, NeurIPS 2024.

Meta-review: The paper introduces DSBench, a comprehensive benchmark for evaluating data science agents through 540 tasks sourced from ModelOff and Kaggle competitions, offering a significant contribution to the field of AI-driven data science evaluation (Reviewers TVnT and EZxL). The benchmark's key strengths include its realistic task environment, covering 466 data analysis and 74 data modeling tasks, and the novel Relative Performance Gap (RPG) metric (Reviewer cMgg). Reviewers consistently highlighted the benchmark's importance in revealing substantial performance gaps between current AI models and human data scientists, with the best agent achieving only 34.12% accuracy in data analysis tasks (Reviewer TVnT). The benchmark's originality lies in its comprehensive approach, integrating multi-table reasoning, long-context understanding, and multimodal task backgrounds, making it a pioneering effort in systematically evaluating AI's data science capabilities (Reviewer TVnT).

Despite the overall positive reception, several weaknesses were identified. Reviewers raised concerns about the benchmark's limited data diversity, noting that the tasks are primarily finance-focused (Reviewer XL8J), and questioned the robustness of the evaluation methodology (Reviewer EZxL). The potential for data leakage through model pre-training was a significant concern, with one reviewer highlighting the risk of benchmark tasks being inadvertently incorporated into future model training data (Reviewer EZxL). Additionally, reviewers suggested improvements in statistical rigor, such as reporting confidence intervals, conducting bootstrap analysis, and providing more detailed explanation of the evaluation process (Reviewer TVnT). The paper also received critiques about its presentation, with some reviewers noting difficulties in understanding the taxonomy and requesting clearer explanations of data categorization and evaluation metrics (Reviewer XL8J). Despite these limitations, the majority of reviewers rated the paper as marginally above or at the acceptance threshold, recognizing its potential to drive advancements in automated data science research.

Decision: Accept (Poster)