DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
Authors: Liqiang Jing
Zhehui Huang
Xiaoyang Wang
Wenlin Yao
Wenhao Yu
Kaixin Ma
Hongming Zhang
Xinya Du
Dong Yu
Venue: ICLR 2025 Poster (2025)
Keywords: data science
agent
benchmark
llm
Primary area: datasets and benchmarks

Abstract: Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have demonstrated impressive language/vision reasoning abilities, igniting the recent trend of building agents for targeted applications such as shopping assistants or AI software engineers. Recently, many data science benchmarks have been proposed to investigate their performance in the data science domain. However, existing data science benchmarks still fall short when compared to real-world data science applications due to their simplified settings. To bridge this gap, we introduce DSBench, a comprehensive benchmark designed to evaluate data science agents with realistic tasks. This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions. DSBench offers a realistic setting by encompassing long contexts, multimodal task backgrounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling tasks. Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG). These findings underscore the need for further advancements in developing more practical, intelligent, and autonomous data science agents.

Published as a conference paper at ICLR 2025
DSBENCH: How FAR ARE DATA SCIENCE AGENTS FROM BECOMING DATA SCIENCE EXPERTS?
Liqiang Jing!
Zhehui Huang*
Xiaoyang Wang²
Wenlin Yao
Wenhao Yu?
Kaixin Ma²
Hongming Zhang²
Xinya Du
Dong Yu²
UUniversity of Texas at Dallas
Teneent Al Lab, Seattle UUniverstty of Southern California
ABSTRACT
Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have demonstrated impressive language/vision reasoning abilities, igniting the re- cent trend of building agents for targeted applications such as shopping assistants or AI software engineers. Recently, many data science benchmarks have been proposed to investigate their performance the data science domain. However, existing data science benchmarks still fall short when compared to real-world data science applications due to their simplified settings. To bridge this gap, we in- troduce DSBench, comprehensive benchmark designed to evaluate data science agents with realistic tasks. This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from ModelOff and Kaggle competitions. DSBench offers a realistic setting by encompassing long contexts, multimodal task back- grounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling tasks. Our evaluation of state-of-the-an LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34. 34%12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG). These findings underscore the need for further advancements de- veloping more practical, intelligent, and autonomous data science agents.
INTRODUCTION
Large Language Models (LLMs) OpenAl 2023a Touvron al. 2023b) and Large Vision- Language Models (LVLMs) OOpenAl 2023b Liu et al. 2023b) have achieved compelling success on various vision and language tasks, such as natural language understanding (Wang et al., 2019). visual question answering (Antol et al., 2015). and image captioning(Lin et al., 2014), demonstrat- ing their adaptability and effectiveness. However, despite their achievements, LLMs and LVLMs face limitations when applied to certain real-world tasks due to the lack of integration with practical applications, such as computer manipulation. To address this, advanced LLMs and LVLMs are in- creasingly being incorporated into interactive intelligent systems, enabling them tackle complex tasks with additional tools and interfaces. A prominent example this is the data science agent, an emerging research that assists individuals and organizations in making informed decisions, predicting trends, and improving processes by analyzing large volumes of data (Hong et al., 2024 Guo et al. 2024 chapyer team 2023 Zhang et 2024c).
The data science agent aims to address data-centric scientific problems, including machine learning, data analysis. and mathematical problem-solving, which present unique challenges, such complex _uuapee__ppteerrr and lengthy task-handn hass-hhndling steps. For example, Jupyter AI (jupyter-ai team 2023) connects gener- ative language models with Jupyter notebooks and provides a user-friendly and powerful way to improve developer productivity in the Jupyter Notebook. MICCopilot (Zhang et al. 2024a) lever- ages L.LMs to generate solutions for novel real-world machine learning tasks, based on the existing experiences from historical tasks. To evaluate the performance of the data science agent, the ex- isting work focuses on developing either code generation benchmarks (Zhang et al.] 2024c Zan al. 2022 Chandel et al. 2022) or math problem benchmarks (Lu et al. 2023 Cobbe al. 2021).
"This work is done during Liqiang Jing and Zhehui Huang's 's internship at Tencent AI Lab, Bellevue, USA. https://jupyterr.orgg

---

Published as a conference paper at ICLR 2025
An election has been held in Excelstan. Excelstan small country including Districts. There are 1000 voters. Each voter is assigned District Code based on where they live. The Dist- rict Code is number between 105-194 and determines what District the voter votes for.
Text
Image
No.
District Code
Distrit
ID
Age
Age
Code
105 114
Alpha
T00/35
200842
40
154
T02052
15
147
212251
122
Table
File
Gamma
11N072
102.28
335
160
2226822
237420
192
161
135 144
Delta
113,005
142
251,005
151
5
145-151
Epailon
T22875
256,30
140
6
155 155-166 164
Zeta
128527
239
139
Question Inputs
146,040
252
279.014
142
T4K,02
337
$19,000
135
8
181
9
185 194
Tota
182,55
122
457785
138
Table district code in Excelstan.
Excel file: Voter information and their voted district (not show).
How many voters are there in the Delta District?
A. 113
B. 114
115
D. 116
E. 117
Given the problem description and data files, DS agent generates executable codes to solve the problem.
DS Agent
jupyter
Solutio__fx Ellcctintnnoveee Voting uur Checapoire hous ago
Age
District
Edtt
View
Run
Kamel
Settings
Traiie
100050
Code
Pythan3 3 fostenet
O
[1]: Import pandas as pd import
[2]: date_encel peedees_excel /Clectionneeetttteeeei vettoooootttnngixss" "imet_r
Solution
dsta_excel
605
964309
[3] delta_vsting_.comt
996
919459
for dis, code in date, l Code"):
957
291887
128
if sutidis_ccdee n- 135 and intidis_code) xw 144:
delta, delta_voting yotinn__count count +x Nuster
898
092377
44
144
print( 'Nusber of vaters in Delsa district:', aatta_votiingcoumm count)
590
887305
84
547
Delta district:
1000 rows x.3 cllcumm
After executing the code, the DS agent finalize the answer based on C
Evaluation
execution results: There are 115 voters in Delta district, the answer is C.
Figure 1: The illustration of the complete workflow of our proposed DSBench benchmark, from task description and data file processing to model or agent execution, followed by the final evaluation. demonstrates how a data science agent approaches problem and processes the data, while the evaluation process compares its solution with the ground-truth answer.
For example, DS00 (Lai et al., 2023) introduce one thousand code completion problems cover- ing seven widely-used Python data science libraries: NumPy, Pandas, TensorFlow, PyTorch, SciPy, Seikit-learn, and Matplotlib. Hendrycks et al. (2021b) introduce the MATH benchmark, which en- ables the community measure the mathematical problem-solving ability of models. Although these benchmarks can be applied to investigate the performance of data science models, they still do not closely reflect real-world data science tasks.
Building an effective benchmark for data science agents/models still presents a significant chal- lenge. The tasks included must be sufficiently complex simulate real scenarios, yet the predic- tions made by these models must remain straightforward to verify. Although existing benchmarks achieved compelling success, they are still under a simplified setting compared real-world data science tasks. Firstly, the task instructions in existing benchmarks are often brief and limited to single modalities. In contrast, real-world tasks typically involve lengthy instructions and multiple modalities. Furthermore, some existing benchmarks only provide incomplete evaluations, focusing primarily on the simple code completion or in-filling capabilities of L.L.s/LVV.MMs, which can be resolved by few-step reasoning or a few lines of code, overlooking the end-to-end evaluation of the whole system's 's performance. Additionally, the evaluation of existing benchmarks may be biased because it is limited to certain environments, such as specific Python packages. However, real-life data science tasks are tool-irrelevant and data-centric.
To tackle these limitations, introduce a comprehensive data science benchmark, DSBench, and the workflow of our benchmark is shown in Figure [ Our benchmark contains two categories of tasks: data analysis and data modeling. The former focuses on answering financial data analysis
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with existing agent benchmarks. Columns include the research field (Field), whether the task instruction includes the data file (Data File?), table (Table?), and image (Image?), the word length of the task description (#Len), whether an executable evaluation function provided (Exec. Eval.?), whether the benchmark asked the agent finish the task in fixed/constrained environment (Env. Fix.?), whether the benchmark only asked the agent to generate code (Code-
only?) and the number of tasks (Tasks).
Exce. Eval.?
Code only?
Eav. Fix.?
Benchmark
Field
Data File? Table? Image? Len.
Spider et al. 2018
Text-to-SQL
X
1,034
MLAgentBench (Huang et
2023
Machine Leariing
X
V
13 2,294
SWE-Bench Jimenez
2023
Software
X
195.1
X
Mind2Web
Dang al.
Web
X X V
X 2,000
WEBLINX
Ju al. 2024
Weh
2337
WorkArena Drouin
2024
Web
AndroidWorld
Rawlles
2024
Android
116
WebArena
Zhou
2023
Web
X
812
OSWorld Xic
Computer Control
DS-0000 Lai et al.
202.3
Data Science
X
140.0
X
1,000
Arcade im al. 2023
Data Science
X
18.4
X
10,082
Spidcr2-V (Cao et al. 2024
(Cao
Data Science
X
X
X
V
494
DSEval Zhang ct al.
2024c
Data Science
X
X
825
DSBench (Ours)
Data Science
/
797.9
X 540
question that needs the agent to fully understand the data and the question's 's intent, and questions either multiple-choice or fill-intthe-blank The latter requires the agent to build predictive models learn from the training data and make predictions for the testing data. Specifically, our dataset is built on data competitions ModelOff and Kaggle" In total, we collected 466 data analysis tasks from Modeloff and 74 data modeling tasks from Kaggle. DSBench offers several advantages over existing data science benchmarks. The rationale for using these two platforms is that ModelOff and Kaggle are among the most popular data science competitions, offering tasks that closely resemble real-world scenarios, where each task requires extensive data manipulation. These include more realistic data science benchmark setting that encompasses understanding long contexts and multi- modal task backgrounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling, as shown in Table
For data analysis tasks, we mainly utilize the accuracy rate the metric. On the other hand, for the data modeling tasks, it is non-trivial to investigate their overall performance because of incon- sistency in numerical ranges and evaluation dimensions for metrics of different data modeling tasks. Therefore, we further propose the Relative Performance Gap (RPG) to normalize the different met- rics in our data modeling tasks. We evaluate multiple state-of-the-art LLMs, LVLMs, and agents, discovering that they fail to solve most of the tasks. The best-performin agent in our experiments achieves only 34. 12% accuracy for data analysis tasks and 34. 34.74% RPG for data modeling tasks.
Our contribution can be summarized as follows: (1) We construct data science benchmark, DS- Bench, which consists of 466 data analysis tasks and 74 data modeling tasks; (2) To comprehensively evaluate existing approaches for the data modeling tasks, we propose the Relative Performance Gap metric that can normalize various evaluation metrics for data modeling; (3) We evaluate represen- tative state state-ofthee th-the-art LLMs, LVLMs, and agents including the most recent GPT-4o, Claude, and Gemini models, and find that our benchmark is challenging for most of the existing approaches.
DATA SCIENCE AGENT BENCHMARK
Data science often requires handling complex data, extracting insights, and building models to solve problems. To ensure DSBench reflects these practical demands, focus these two task types: data analysis, and data modeling. In our search for appropriate datasets and challenges, we identified
modeloff-quide//
https://wwe kaggle... com/
We released all our data and code Githuh
3

---

Published as a conference paper at ICLR 2025
Summary of dataset characteris- ties for data analysis tasks. Intro and Len Lue are the length of the task introduc- tion and questions
Table 2:
Table 3: Summary of dataset characteristics on data modeling tasks. File size is the size of Train- ing set.
Mean
Max
Min
Total
#Challenges
38
Mean
Max
Min
Total
eQuestions
12.3
466
#Competitions #Metrics
74
Ien_I
749.58
28,487
28.484
18
Len.Que
65.9
406
30,691
Context Length
688
2,505
216
50,875
#Excel
0.8
31
#Training samples
287k
4,828k
200
21,270k
Excel Size (KB)
236.6
2,755,9
0.2
7,333,4
File Size
61 GB
487 GB
11 KB
4,519 GB
#lmage
0.1
5
#Sheets
2.3
69
#Table
1.3
12
49
that ModelOff and Kaggle provide diverse and realistic tasks that align well with requirements for data science and data modeling tasks, respectively.
2.1 DATA ANALYSIS TASKS
2.1.1 DATA COLLECTION
Modeloff is global financial data analysis competition that challenges contestants use Excel to solve data-centric questions and case studies. It mostly focuses on independent questions that involve mini exercises Excel. The questions Modeloff challenges consist of data analysis tasks that different tools, such Python, Excel, and Matlab can solve. Therefore, we resort to the Modeloff challenge for the evaluation data analysis ability data science agents. We collect all Modeloff challenges and then filter all the challenges that do not contain any questions. Finally, the original 43 challenges are filtered down to challenges with 466 questions. The question types can be categorized into multi-choice questions and fill-in-the- tantthh-bblnkk questions. The data statics of our data analysis tasks are detailed in Table
2.1.2 TASK FORMULATION
Input and output. Suppose we have the task introduction I, the data files D = {di,... .dN.k. and the question Q, we then feed them into a data science agent g to answer the question Q, i.e, À = G[, D, Q). N& is the total number of data files and it can be 1. Å is the generated answer by the G.
Evaluation metrics. To evaluate the performance of the whole agent system, we compare the semantics of ground-truth answer and the predicted answer À by S(A, Å). If the semantics of ground-truth answer A and the predicted answer A are the same, we consider the generated answer have successfully answered the question. 5(-) is the semanties comparison function that implemented by I.LM and prompt in Appendix The metric for our benchmark is the percentage of data science questions that are answered correctly, i.e., task-level accuracy. In addition, we also introduce competition-level accuracy for comprehensive evaluation. Competition-lewe accuracy is calculated by averaging the accuracy scores obtained from each competition.
2.1.3 FEATURES
Various Modalities. Different from the previous works which mainly focus textual modality (e.g., (Lai et al. 2023 Cobbe al. (02T)), our task consists of various modalities, such as images, Excel files, and tables. To show the distribution of the different modalities in different competitions,
we visualized the number of competitions in different modalities, as shown in Figure 2(a)
Complex Table. Various tables are contained in this dataset. There may be several tables for one question. Therefore, the data science agent must identify which table is important for the current question. Furthermore, some questions require analyzing data across several tables. Unlike previous benchmarks, the tables in this dataset are longer, making it difficult to solve these questions without additional tools, such as Python or Excel, even for humans. In addition, the formats of different
4

---

Published as a conference paper at ICLR 2025
Text+Excel
Texle-ableEExee
Gap Filling
28.9%
A-D
34.2%
30.3%
35.2%
5.3%
18.4%
2.6%
12.2%
22.3%
10.5%
Text+Table
Text-only
Text+Image
Text+Table-Eaee Exbl--Exoo-
A-I
A-F
(a)
(b)
Figure 2:
Visualization of the distribution of the datasets. (a) Distribution of data analysis challenges in terms of modalities. Every challenge may contain several questions. (b) Distribution data analysis challenges in terms of question types.
tables in one challenge are changeable, For example, some tables have properties in their first row, and some tables have properties in their first column.
Diverse Long Context. The context in our dataset consists of long textual descriptions (815.44 words average) well as multiple modality content. Resolving this kind of data analysis question requires I) a full understanding of the long description text and the corresponding tables, data files, and images, and 2) the ability to identify the important context semantic content relevant to the current question.
Wide Scope for Possible Solutions. Our evaluation task provides critical platform for assessing the capabilities of data science agents and the corresponding interactive environment. Our data anal- ysis tasks can be utilized to compare a variety of approaches, from pure LLMs/LVLMMs to cutting- edge data science agents. The task setting greatly expands the freedom of tool use and encourages developers to employ creative strategies that may diverge from established norms (such as using Excel). For example, we can use either Excel or Python to calculate the amount of tax a company pays.
2.2
DATA MODELING TASKS
2.2. DATA COLLECTION
To evaluate the performance of data science agents on data modeling tasks, we resort to machine learning competitions. Kaggle is a data science competition platform and online community for data scientists and machine learning practitioners. From the platform, we find there are total 648 competitions. Since the testing set in the Kaggle competition is inaccessible, we split the original training set into the training set and testing set as an 8:2 ratio for evaluation. In this way, we could directly get the performance of the solution devised by data science agent, avoiding submitting the solution to the Kaggle website. To split the dataset easily, we only retain the competitions with a training file, a testing file, and sample of submission file. Then split the original data files the Kaggle competition. Finally, we get 74 data modeling competitions in our data science benchmark. All the statics information of our data modeling tasks is detailed in Table 3
can
automatic code to
2.2.2 TASK FORMULATION
Input and output. Suppose we have the competition description E, the training set A, the testing set S, and the sample the submission file M, we then feed them into a data science agent G. which could devise an algorithm and implement the corresponding code to generate the submission file F which is the predicted result for the input testing set, ie.. F = G(E, A, M.). f is the generated submission file by the and it has similar format to the sample of the submission file M.

---

Published as conference paper at ICLR 2025
25
Number of Challenges
20
15
5
m0
m1
m2
m13
m4
m1
m6
m7
m8 m9 m10 mll ml2 ml3 ml4
m15
m16
ml7
Evaluation Metrics
Figure 3: The chart displays the count of Kaggle competitions (vertical axis) categorized by the evaluation metrics used (horizontal axis). Each bar represents the number of competitions that em- ploy specific metric, highlighting the diversity of evaluation criteria in DSBench. The denotation of "m0-m17" can be found in Appendix D The top-3 used metrics are ROC ("m1") Root Mean Squared Logarithmic Error ("m3") and Root Mean Squared Error ("m4").
Evaluation metrics. Our evaluation focuses on two key aspects of data science agents: whether they can generate a submission file and the performance of the models they develop. Specifically, we first adopt Task Success Rate as the evaluation metric, i.e., whether the agent can build an MI. model and generate the submission file in bug-free manner within fixed number of steps. To further learn the performance of the model devised by the agent, we also evaluate the predicted submission file f with the original metric corresponding to the competition as p f(F, F). F is the ground truth file for the testing set, f()) is the evaluation function the specific metric (such as Fl and accuracy), and p is the performance. However, the metrics different across different competitions, can not directly take the average across tasks. Thus we calculate the gap between the performance of the file submitted by the agent and the performance of the human expert as an evaluation indicator of the data modeling task. Specifically, we devise the Relative Performance Gap (RPG) metric to show the performance of the data science model, formulated as N Σ max((p;- -bb///// b.), 0)- N the total number competitions. p; is the performance the predicted submission file the i-th competition and is the highest performance value = for the i-th competition. 6; is the performance of a baseline. More details can found in Appendix
2.2.3 FEATURES
Long context. For each competition in Kaggle, we crawl the corresponding task, evaluation, and dataset description, shown Appendix G The deseription depicts the task background and the aim of the competition. The evaluation introduces what metric is used to evaluate the performance of the competition and how the metric is computed. The data description contains the overall de- scription of all data sets and the explanation of each attribution in the data file. The average length of context text and data of our data modeling tasks 79.187.
End-to-end setting. Unlike existing work that focuses solely on code completion, our task is more challenging and demands a broader range of agent capabilities, such model design, code im- plementation, and self-debugging. Our data modeling task is end-to-end and every step in task resolving focuses on the different abilities of agents. The end-to-end setting allows the model to complete the task with minimal constraints, which is similar to the real-life task the data science ex- perts face. Hence, our data modeling task evaluates the ability of the whole agent systems, including LLMs/LVLMss tools, and agent interactive environment design.
Execution-basse Evaluation. We use execution Python script to verify the usability the sub- mission file and evaluate the performance of the submission file from data science agents. Hence, the generated submission file from data science agents should strictly comply with the file format requirements in the input. For different competitions, may different metrics. In total, the number of metrics in our dataset is 18 and the distribution the number of competitions in each metric is shown in Figure 3

---

Published as a conference paper at ICLR 2025
Table 4: The performance comparison of different models on data analysis tasks. mance is based on results from 10 sampled competitions.
*Human perfor-
Task-level Accuracy %%
Competition-leve Accuracy 7%
Framework
Model
Cost / $
Inference Time /
LLaVA
11.59
13.6
7.01
Llama3-8b
16,95
16.8
10.60
Llama3-70b
23.39
54.4
14.95
Model
GPT-3.5
20.39
1.95
3.6
11.85
-only
GPT-4
25.97
117.90
20.9
17.21
GPT-4o
28.11
67.56
14.9
19.26
GPT-4o mini
23.82
2.21
17.4
14.64
Claude
6.01
64.98
668.1
3.83
Gemini
31.55
18.26
686.5
24.81
Llama3-Sb
10.73
28.5
6.05
Llama3-70B
21.89
98.2
13.64
GPT-3.5
20.82
5.60
23.8
13.80
AutoGen
GPT-4
30.69
105.89
68.2
22.68
GPT-4o
34.12
114.05
36.8
26.72
GPT-4o mini
28.11
2.95
48.9
21.01
GPT-3.5
11.16
21.39
25.4
8.23
Code
GPT-4
26.39
128.85
43.1
21.82
Interpreter
GPT-4o
23.82
87.04
30.4
22.65
GPTT-4o mini
17.81
16.54
30.0
14.65
Human
64.06
1107.7
67.33
3 EXPERIMENT
3.1 EXPERIMENTAL SETUPS
We select two kinds of models for evaluation: (1) vanilla language model and (2) agent (i.e. LLM- s/LVtsMsiittrraationn environment). The vanilla language model includes open-source LLMs (in- cluding Llama3-8b, Llama3-70b (Touvron al. 2023a) and LLaVA (Liu et al. 2023b) and closed- source LLMs (including GPT-3.5, GPT-4 GPT-4o, GPT-40 mini). Gemini and Claude. The agent system includes closed-source system Code Interpreter and open-source agent systems (includ- ing Autogen (Wu al.] 2023)). For the Code Interpreter, we selected GPT-3.5, GPT-4, GGT-4o, Llama3-8b, Llama3-70b,
and GPT-4o mini base models, respectively. For the Autogen, GPT-3.5, GPT-4, GPT-4o, and GPT-4o mini as agents, respectively. More details can be found in Appendix A
3.2 QUANTITATIVE ANALYSIS
In this section, we conduct a quantitative analysis for baselines, Due to the limited space, we further conduct a qualitative analysis Appendix
3.2. DATA ANALYSIS TASKS
We show the performance comparison among different baselines data analysis tasks in terms of accuracy rate, cost, inference time, and challenge accuracy in Table 4 From the Table, we observe that: (1) Models that perform better basic language tasks tend also excel in data analysis tasks. For example, GPT-4o achieves the best performance among all vanilla model-only baselines and it
also shows advanced performance on several general language task such as MMLU (Hendrycks)
et al. 2021a), GPQA (Rein et al. 2023) and MATH (Hendrycks et al.) 2021c). (2) The AutoGin framework tends to consume more time to finish data analysis tasks and higher costs compared

---

Published as a conference paper at ICLR 2025
GPT-40
AutoGen with GPT-40
Gemini
0.8
Acquracy
0.4
8
9
10
11
12
15
16
17
18
20 Challenges
19
21
22
23
24
25
26
27
28
29
30
31
32
33
34
36
37
38
Figure 4: Accuracy for baselines across all data analysis challenges in DSBench.
1.0
Table 5: The performance comparison of Au- toGen with GPT-4o on data analysis tasks across different years. The performance of most time slots shows little difference.
0.8
0.74
Accuracy
0.6
0.4
0.30
Year
Accuracy
0.2
0.14
0.13
2012
66.67
0.0
2013
61.97
<500
500-5k
5k-20k
>02k
2014
17.78
# Input Tokens
2015
17.86
Figure 5: The performance comparison of Au- toGen with GPT-4o on data analysis tasks parti- tioned by total input length.
2016
14.29
2017
13.01
the original vanilla model-only method. The reason is that AutoGen usually devises multi- turn conversation between multi-agents to resolve data analysis task. (3) AutoGen with GPT- B.5GPTT-4GPPT44//GTT-4 mini outperform the original vanilla PPT335/GFT-IIGPT-O//GP-.. mini This indicates the interaction mechanism and tools within AutoGen beneficial for data analysis tasks. (4) Although the advanced GPT-4o achieved the best performance, it consumes more time and money compared with GPT-3.5. (5) Even the most advanced agent system has large performance gap with humans. More analysis of human results shown in Appendix B
Difficulty across different competitions. Analyzing performance by individual challenge reveals that different models display consistent patterns across various challenges, as depicted in Figure 4 However, the specific problems addressed by each model show minimal overlap. For example, Llama3-8b and GPT-3.5 share a similar accuracy rate on all data analysis tasks, with Llama and GPT-3.5 resolving 61 and 79 instances respectively. Yet of these instances, GPT-3.5 only solves 64.21% of the instances solved by Llama3-8b.
Difficulty correlates with context length. To investigate the effect of the length of the input context on the performance, we visualize the performance comparison of models on data analysis tasks partitioned by total input length in 33 As we can see, the performance AutoGen with GPT- 4o drops with the total context length increase. We also see similar performance trend in other models. The potential reason is that the model needs to understand complex task backgrounds and analyze data files with more data for the task with the long context.
Difficulty correlates with release time. In addition, we show the performance comparison of Au- toGen with GPT-4o on data analysis tasks across different years in Table 5 We observe that the difficulty of the challenges increases over the years. This trend can be attributed to the evolution of data technology, which has enabled data scientists to leverage advanced tools to tackle more com- plex tasks. Consequently, the complexity of the questions has also escalated. The average time humans spend on each question is 18.5 minutes, which further illustrates how difficult the task is.
8

---

Published as a conference paper at ICLR 2025
Table The performance comparison of different models on data modeling tasks. The versions of models are the same as in Table "Human performance is based on results from 22 competitions.
Framework
Model
Task Success 1%
Cost
Inference Time /s
RPG
Llama3-8b
5.41
50.9
1.55
Llama3-70b
16.22
158.4
7.79
GPT-3.5
8.11
0,41
26.5
6.02
AutoGen
GPT-4
87.84
19.34
77.4
45.52
GPT-4o
71.62
12.27
104,1
34.74
GPT-4o mini
22.97
0.10
26.7
11.24
GPT-3.5
16.22
2.74
112.5
6.52
Codc Interpreter
GPT-4
54.05
38.81
237.6
26.14
GPT-4o
44.59
19.26
268.6
19.87
GPT-4o mini
39.19
2.70
199.6
16,90
Human *
100.00
65.02
3.2.2 DATA MODELING TASKS
Table the performance comparison among different methods of data modeling tasks in terms of task completion, cost, inference time, and Relative Performance Gap (RPG). Several observations can be found in this Table: (1) In most cases, the advanced agent system (e.g., AutoGen with GPT-4o or GPT-4) could generate the submission file. The open-source programming framework AutoGen incorporates the local shell to run the Python code and save the predicted results the specified file directory. It also utilizes the multi-turn conversation interaction to revise the code generated from previous turns. These strategies improve the capability of the vanilla model on data modeling tasks. (2) Compared with Interpreter with GPT-3.5 and Interpreter with GPT-4o, AutoGen with GPT-3.5 and AutoGen with GPT-4o tend to consume less time finish data modeling task. (3) Although the task success rate of Interpreter with GPT-3.5 is twice that of AutoGen with GPT-3.5, they still share a similar RPG. This indicates that the performance of the method devised for the resolved task by the Interpreter with GPT-3.5 3.5.5 is worse than that of AutoGen with GPT-3.5.
Large gap between models and human performance. We also report the human evaluation re- sults in our paper. Specifically, we run code from Kaggle which is submitted by human contestants. Therefore, we can determine human performance by running the code. In the code collection pro- cess, find that collect the usable code for 22 competitions. From the performance comparison, we find persistent gap between L.Ms/agents and humans in both task success rate and RPG.
code could not successfully due to lack of maintenance. Finally,
0.69
Table 7: The task success rate comparison of AutoGen with GPT-4o on data modeling tasks partitioned by time slots. The performance of most time slots shows minimal difference.
0.54
0.48
Task Sucvess
0.26
Year Range
Task Success (%)
0.0
<011
50.00
<500
500-750
750-1k
>1k
2019 & <2021
53.33
# of Input Tokens
2001 & <2023
25.00
Figure 6: The task success rate comparison of Au- toGen with GPT-4o on data modeling tasks parti- tioned by total input length.
20033
51.61
Difficulty independent length. To investigate the effect input length on performance, we vi- sualize the performance comparison of AutoGen with GPT-4o data modeling tasks partitioned by total input length in Figure 6 As shown, the performance of AutoGen with GPT-4o does not exhibit clear trend with varying input lengths. This suggests that the completion rate is relatively independent the total input length. This behavior can be attributed to the nature of Kaggle com- petition tasks, which typically require an understanding of the table's attribute information, basic
https: wwwp://wwwwwwwooooc.ooo/..w... kaggle wtt.:om//cc///// wwt.///mmetiiiin////oooooooeoeee

---

Published as a conference paper at ICLR 2025
background descriptions, and identifying which attributes are targets and which are inputs. The cru- cial factors for these tasks are more related to the clarity and specificity of the attribute information rather than the overall input context.
Difficulty independent of release time. In addition, Table 7 presents the performance compari- son of AutoGen with GPT-4o data modeling tasks partitioned by time slots. We find that the performance across most time slots shows little difference, with the completion rate being 50.00% for tasks created before 2019, 53.33% for tasks between 2019 and 2021, 25.00% for tasks between 2021 I and 2023, and 51.61% % for tasks from 2023 onwards. This indicates that the model's ability to handle data modeling tasks does not significantly correlate with the year of task creation.
4 RELATED WORK
LLMs/LVLMs as Agents. Advancements in NLP and computer vision have positioned LLMs and LVLMs as pivotal components in intelligent agent systems. Models like GPT-3.5 (OpenAl 2022). GPT-4 (OpenAl 2023a), LLaMA (Touvron et al. 2023a b), and LLaVA (Liu et al. 2023b)
have excelled in tasks such as language comprehension. image recognition, dialogue generation, and complex task execution, prompting research shift towards agent applications. Initially,
search focused on decision-making in simulated textual environments (Gao et al.] [0033] Yao et al.]
2023a Shhinn et al.] 2023: Liu al. 2223a Gu et al. 2023), with ReAct (Yao et al.
2023b) pi-
oneering the integration of Chain-of- Thought (CoT) (Wei et al. 2022) for agent tasks. However, these approaches lack real-world applicability due to limitations in tool usage and dynamic interac- tions. Consequently. LLMs/LVLMs have been equipped with functionalities like code interpreters
Yang et al. 2024 Hu et al. 2024), web browsers (He et al. 2024), and Microsoft Office integration (Wu et al.) 2024). Agents like AppAgent (Zhang et al.) 2023a), OS-Copilot (Wu et al. 2024). and
SWE--ggent (Yang et al. 2024) operate in actual work environments, enabling applications in web manipulation (Li ct al.) (023). playing Minecraft (Wang et al. (023). spreadsheet automation (Xic)
ct al. 2024). and data science tasks (Team] 2023 Guo et al. 2024 Hong et al. 2024). Despite these advances, evaluating agent performance in real-world scenarios remains challenge.
Evaluations of LLMs/LVLMs. Evaluating LLMs/LVL.M. is essential for gaining insights and guid- ing model improvements. Early evaluations focused specific NLP tasks like sentiment classifi- cation (Sun et al. 2023), named entity recognition (Sang & Meulder 2003), information extraction (Sundheim 1992). and text summarization (Nallapati 20066) using metrics such as BLEU (Pa- pineni et al. 2002), ROUGE (Lin 2004). and BERT-Score (Zhang et al. 2020). As LLMs/LVLMM advanced, they surpassed these benchmarks but introduced new challenges like faithfulness (Jing ct al, [ Chang et al., 2024). safety (Zhang et al. (023b). visual reasoning (Zhang et al. 2024b), and istructiio-followinn capabilities (Jiao et al. 2023). Furthermore, to evaluate the per- formance of agent with LLMs on real-world scenarios, new benchmarks like OSWorld (Xie et al.) 2024]. Spider2-V (Cao et al. 2024). Spider 2.0 (Lei et al.) 2024). DA-Code (Huang et al. 2024). AgentBench (Liu et al., (023cc) DS-1000 (Lai et al. 2023), SWE-Bench (Jimenez et al. 2023 BigCodeBench (Zhuo et al. 2024), CodeRAGBench Waag et al. 2024), RepoBench (Liu et al. 2024), ML-Bench (Liu et al. 2023d) and DSEval (Zhang et al.) 2024c) have been proposed to eval- uate performance in agent tasks like gaming and bug fixing. In contrast to these works, we focus on evaluating the entire system- that is, both LLMs//VLLMs and the agent interactive environment-
real-world data science scenarios.
CONCLUSION
The complexity of real-world data science projects extends far beyond mere code generation and basic numerical calculation. In this paper, we propose a data science benchmark, named DSBench, which consists of 466 data analysis tasks and 74 data modeling tasks. By incorporating challenges from ModelOff and Kaggle competitions, our benchmark provides a genuine representation of prac- tical data science environments. This authentic context stimulates the creation of innovative solu- tions that can be readily applied to actual data science problems. We believe that this benchmark, together with our other contributions, will prove to be valuable assets in advancing the development of more practical, intelligent, and autonomous data science models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In IEEE International Conference on Computer Vision, pp. 2425-2433. IEEE Computer Society, 2015.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu. Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, and Tao Yu. Spider2-v: How far multimodal agents from automat- ing data science and engineering workflows?, 2024.
Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. Training and eval- uating jupyter notebook data science assistant. CoRR, (bss/201.11900) 2022.
Yue Chang, Liqiang Jing. Xiaopeng Zhang, and Yue Zhang. A unified hallucination mitigation framework for large vision-language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL http::::////ppeeeeeeeeee net, www.mronoommmmiiiii
chapyer team. chapyer. GitHub, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.141 2021.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vázquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks? CoRR, abs/2403.07718, 2024.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamic Callan, and Graham Neubig. PAL: program- aided language models. In International Conference on Machine Learming, volume 202 of Proceedings of Machine Learning Research, pp. 100644-0799 PMLR, 2023.
Yu Gu, Xiang Deng, and Yu Su. Don't generate, discriminate: A proposal for grounding language models to real-world environments. In Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 4928-4949. Association for Computational Linguistics, 2023.
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Au- tomated data science by empowering large language models with case-based reasoning. CoRR, abs/2402. 17453, 2024.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. CoRR, abs/2401.139199 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations. OpenReview.net, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021c.
11

---

Published as a conference paper at ICLR 2025
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Danyang Li. Jiaqi Chen, Jiayi Zhang, Jinlin Wang. Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Wenyi Wang, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zongze Xu, and Chenglin Wu. Data interpreter: An LLM agent for data science. CoRR, abs/2402. 18679, 2024.
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Guoyin Wang. Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. Infiagent-dabench: Evaluating agents on data analysis tasks. CoRR, abs/2401 fbs/2401.05507, 2024.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Benchmarking large language models as Al research agents. CoRR, abs/2310.03323 2023.
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. Da-code: Agent data science code generation bench- mark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 13487-13521. Association for Com- putational Linguistics, 2024. URL /atps::/aclantholooyoooooo owg. 22044 emn mamnpppmmii.....
748
Fangkai Jiao, Bosheng Ding, Tianze Luo, and Zhanfeng Mo. Panda L.M:: training data and evaluation for open-sourced chinese instruciionfollooinng large language models. CoRR, abs/2305.03025, 2023.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? CoRR, abs/2310.06770, 2023.
Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du Faithscore: Fine-grained evaluations of hallucinations in large vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 5042-5063. Association for Computational Linguistics, 2024a.
Liqiang Jing, Jingxuan Zuo, and Yue Zhang. Fine- graeegraiedd and explainable factuality evaluation for multimodal summarization. CoRR, Abs/2402.11411 2024b.
jupyter-ai team. jupyter-ai. GitHub, 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 18319-18345. PMLR, 2023.
Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida I. Wang, and Tao Yu. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. CoRR, abs/2411.07766 2024. doi: 104855ARRX122410077666 URL 07763
https: htttps://do..org// .ww.oooorggooo htt.:////......//////
Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. Sheetcopilot: Bringing software productivity to the next level through large language models. In Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Processing Systems, 2023.
Chin-Yer Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81. Association for Computational Linguistics, July 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014 - 13th European Conference, volume 8693 Lecture Notes in Computer Science, pp. 740-755. Springer, 2014.
12

---

Published as a conference paper at ICLR 2025
Bo Liu, Yugian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+P: empowering large language models with optimal planning proficiency. CoRR, abs/2304. 111777 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, 2023b.
Tianyang Liu, Canwen Xu, and Julian J. McAuley, Repobench: Benchmarking repository-Ievel code auto-completion systems, In The Twelfth International Conference on Learning Representations, ICLR, 2024.
Xiao Liu, Hao Yu, Hanchen Zhang. Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang. Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang- Agentbench: Evaluating ilms agents. CoRR, abs/2308. 0066888 2023c.
Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, et al. MI-bench: Large language models leverage open-source libraries for machine learning tasks. arXiv preprint arXiv:2311. 1.0985,, 2023d.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chus Zhu. Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Eleventh International Conference on Learning Representations. OpenReview.net, 2023.
Xing Han Lu, 7denek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi- turn dialogue. CoRR, abs/2402.05930, 2024.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. Abstractive text summarization using the 20th SIGNLJ. Conference on Computational Natural Language Learning, pp. 280-290. ACI., 2016.
mns and beyond. In Proceedings of
OpenAl. Chatgpt blog post. httpps://pennaa..c/////ll//////h
2022.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023a.
OpenAl. Gpt-4v(ision) system card. OpenAl Blog Post, 2023b. URL vittps://opent com/
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318. ACL, 2002.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E. Bishop, Wei Li. Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy P. Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. CoRR, abs/2405. 145733 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. CoRR, abs/231 1.122 2023.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction the conll-2003 shared task: Language- independent named entity recognition. In Proceedings of the Seventh Conference Natural Language Learning, pp. 142-147, ACL, 2003.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reftection. CoRR, abx/2303. 1366. 2023.
Teng Sun, Liqiang Jing, Yinwei Wei, Xuemeng Song. Zhiyong Cheng, and Liqiang Nie. Dual consistency-enhanne semi-supervised sentiment analysis towards COVID-19 tweets. JEEE Trans. Knowl. Data Eng. 35052): 2023.
13

---

Published as a conference paper at ICLR 2025
Beth Sundheim. Overview of the fourth message understanding evaluation and conference. In Proceedings of the 4th Conference on Message Understanding, pp. 3-21. ACL, 1992.
XAgent Team. Xagent: An autonomous agent for complex task solving, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.0922 2023b.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations. OpenReview.net, 2019.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. CoRR, abs/2305. 16291, 2023.
Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. Coderag-bench: Can retrieval augment code generation? CoRR, abs/2406.14499 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, abs/2308.08 155.088155,10 2023.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. CoRR, abs/2402.07456. 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. CoRR, abs/2404.07972, 2024.
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. CoRR, abs/2405.15793. 2024.
Shunyu Yao. Dian Yu. Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, 2023a.
14

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations. OpenReview.net, 2023b.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua How- land, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sut- ton. Natural language code generation in interactive data science notebooks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 126-173. Association for Computational Linguistics, 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 3911-3921. Association for Computational Linguistics, 2018
Daoguang Zan, Bei Chen, Dejian Yang. Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian- Jian-Guang Lou. CERT: continual pre-training on sketches for library-oriented code generation. In Proceedings of the ThirttyFFrst International Joint Conference on Artificial Intelli- gence, pp. 2369-236210 2375. ijcai.org, 2022.
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang. Bin Fu, and Gang Yu. Appagent: Multimodal agents smartphone users. CoRR, abb/2312.11777 2023a.
Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. Micopilot: Unleashing the power of large language models in solving machine learning tasks. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2931- 2959. Association for Computational Linguistics, 2024a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evalu- ating text generation with BERT. 8th International Conference on Learning Representations. OpenReview.net, 2020.
Yue Zhang, Liqiang Jing, and Vibhav Gogate. Defeasible visual entailment: Benchmark, evaluator, and reward-driven optimization. CoRR, abs/2412.16232, 2024b. doi: 10.48550/ARXIV24412. 16232. URL URL httpl:ttt:///ettm
org. 4084085555511i www.a.iwwi................ 16232
Yuge Zhang, Qiyang Jiang, Xingyu Han, Nan Chen, Yuqing Yang, and Kan Ren. Benchmarking data science agents. CoRR, abs/2402.111168, 2024c.
Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang. Chong Long, Xiao Liu, Xuanyu Lei. Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions. CoRR, abs/2309.0704. 2023b.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environ- ment for building autonomous agents. CoRR, abs/2307.13854, 2023.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo. Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. CoRR, abs/2406.15877, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXPERIMENTAL SETUPS
In all settings, the versions of LLavA, LLaMA, GPT-3.5, GPT-4, and GPT-4o --. 13b, LLaMA 3, spt-3.5-turbo-0125, gp--4-turbo-2022400-00 and gpt-4o-2024-05-13 The sizes of LLaMA and LLaVA are 8B and 13B, respectively. The versions of Gemini and Claude are gemini- 1.5-pro-exp-000000 and --sile-----sonnt-2200
respectively. As we mentioned before, we mainly use the accuracy for the data analysis task, and RPG score for the data modeling task. In addition, we also report costs (if the agent utilizes charging API) and the average time of resolving a task. We simply use greedy decoding for models. Given the substantial expense associated with generating outputs, limit approach producing single solution per instance. All the open-source models are run on 4 X NVIDIA A100 GPU server.
B HUMAN PERFORMANCE FOR DATA ANALYSIS TASKS
Besides, to learn the performance humans on these data analysis tasks and reduce the cost of human annotation, we randomly sampled 10 data analysis challenges from our benchmark for human labeling. We show the performance of baselines and manual annotation in Table the 10 sampled data analysis tasks. We found that baselines show similar performance across whole and sampled testing data. For example, LLaVA achieved the worst performance on both original and sampled testing data. Even the most advanced agent system has large performance gap (30.41 % accuracy rate) with humans and is far from resolving all tasks (only 33.65% accuracy rate).
Table 8: The performance comparison of different models on the sampled tasks.
Task-level Accuracy /%
Competition-level Accuracy /%
Framework
Model
Cost / $
Inference Time /s
LLaVA
12.50
194.2
9,76
Llama3-8b
14.42
20.0
9.22
Llama3-70B
23.08
57.6
15.45
Model -only
GPT-3.5
15.38
0.48
4.2
8.13
GPT-4
24.04
62.58
35.5
18.08
GPT-4o
23.08
32.40
29.7
17.33
GPT-4o mini
16.35
1.08
34.0
11.33
Claude
0.96
18.79
669.2
1.00
Gemini
31.73
16.22
1016.2
28.25
Llama3-8b
10.58
3052.70
6.24
Llama3-70b
21.15
124.4
17.85
GPT-3.5
21.15
1.23
28.3
13.41
AutoGen
GPT-4
32.69
24.29
79.7
22.89
GPT-4o
31.73
19.26
41.4
28.68
GPT-4o mini
32.69
0.57
49.6
25.57
GPT-3.5
13.46
4.87
26.0
9.96
Code
GPT-4
32.69
34.80
52.8
26.55
Interpreter
GPT-4o
33.65
18.47
32.7
33.04
GPT-4o mini
22.12
3.70
39.5
20.07
Human
Human
64.06
1107.7
67.33
C PROMPTS FORMAT
Models are prompted with the following general template with slight variations depending on the model used.
The prompt for data analysis tasks shown as follows.
'//
16

---

Published as a conference paper at ICLR 2025
You are data analyst. will give you background introduction and data analysis question. You must answer the question.
The introduction is detailed as follows.
cintroductions
(introduction text, table, and image)
</introductions>
The workbook is detailed follows.
<excel>
(excel content)
</excei>
The questions are detailed as follows.
<question>
(question content)
</question>
Please anawer the above question. '..
The prompt for data modeling tasks is shown follows.
'..
You are a data scientist. I have a data modeling task. You must give mo
+ the predicted results as a CSV file as detailed in the following content. Please don't ask any questions. I provide you with
three files. One is training data, one is test data. There is also
sample file for submission
The task roducttio is detailed follows.
<introduction>
(introduction text
</introduction>
The training data, resting data, and sample of submission files are path /xxx/xx/xx.
For the similarity function, we GPT-4o to implement it with the following prompt.
'''
Please judge whether the generated
is right wrong. We require that the correct answer to the prediction gives a clear answer, S not just a calculation process or disassembly of ideas. The + question is (question).
The true answer is (answer).
The predicted answer is (prediction).
If the predicted answer is right, please output True. Otherwise output Flase. Don't output any other text content. You only output True or False.
17

---

Published as a conference paper at ICLR 2025
D METRICS OF DATA MODELING TASKS
In total, the number of metrics of data modeling tasks in our dataset is 18 and the distribution of the number of competitions in each metric is shown in Figure
Number of Challenges
23
20
11
10
m0
m1
m2
m3
m4
m5
m6
m?
m8
m9
m10
ml1
m12
m13
m14 m15
m16
m17
Evaluation Metrics
Figure 7: The chart displays the count of Kaggle competitions (vertical axis) categorized by the evaluation metries used (horizontal axis). Each bar represents the number of competitions that employed specific metric, highlighting the diversity of evaluation criteria in DSBench. "m0-m17" denote metrics Accuracy (m0), ROC (ml), Normalized Gini Coefficient (m2), Root Mean Squared Logarithmic Error (m3), Root Mean Squared Error (m4), R2 Score (m5), Mean Columnwise Root Mean Squared Error (m6), Macro F1 (m7), Micro-averaged
(m8), Mean Absolute Error (m9), Word-level Jaccard Score (m (0),), Quadratic Weighted Kappa (ml1), Pearson Correlation Coefficient (m12), Median Absolute Error (m13), Symmetric Mean Absolute Percentage Error (m14), Mean Column-wise Spearman's Correlation Coefficient (m15), MPA @3 (ml6), and Logarithmic Loss
(m17).
E MORE BASELINE DETAILS
In this section, we detail how we input the tasks into baselines.
Data Analysis Tasks For LLMs/LVLMMs such LLaMA and LLaVA-1.5, directly convert the data file into text format using Panda We concatenate the task introduction and text from the data file and then input the merged text into the L.LMs/LVLMS. For the AutoGen agent, input the task introduction and the path of data files in the local computer environment. In this way, AutoGen can access the data files using the local code execution environment.
Data Modeling Tasks For the AutoGen agent, similar to the data analysis tasks, we input the task introduction and the path of data files in the local computer environment. As for the Code Interpreter, we upload our data files with OpenAl assistant API into the OpenAl platform, and the LLMs of OpenAl can access them.
In addition, we use the performance of the original submission file in the competition as the baseline performance in the RPG computation process.
F HUMAN EVALUATION FOR SEMANTICS COMPARISON FUNCTION
To evaluate the reliability of our semantics comparison function, conduct human evaluation of the results from GPT-4o. Specifically, we first sampled 100 predicted answers from GPT-3.5 for our data analysis tasks. Given the question, predicted answer, and ground- answer, we then ask people to see whether the results of the semantics comparison function are right. The accuracy of human evaluation is 100%, which shows the effectiveness of our semanties comparison function.
https://platfors
18

---

Published as a conference paper at ICLR 2025
Description
Dataset Description
renting bicycles where the process obtaining membership, rental, bike automated via network kiosk locations throughout Using systems, people are
Bike sharing systems
You are provided hourly rental data spanning two years. For this competition, the training set is comprised of the first 19 days of each month, while the test set the 20th to the end of the month. You must predict the total count of bikes rented during each hour covered by the test set, using only information available prior to the rental period.
location and return to different place on an as-needed basis. Currently, there world.
500 bike-sharing programs around the
The data generated by these systems makes them attractive for researchers because the duration travel, departure location, arrival location, and time elapsed is explicitl recorded. Bike sharing systems therefore function as a sensor network, which can be used for studying mobility in a city. In this competition, participants are asked combine historical usage patterns with weather data in order to forecast bike rental demand in the Capital Bikeshare program in Washington, D.C.
Data Fields
datetime hourly date timestamp
season = spring, summeer 3 fall, 4 winter holiday whether day considered holiday
workingday whether the day neither weekend nor holiday
weather 1: Clear, Few clouds, Partly doudy, Partly doudy
2: Mist + Cloudy, Miat + Broken clouds, Mist Few clouds,
Evaluation
3: Light Snow. Light Rain Thunderstorm Scattered clouds, Light Rain Scattered clouds
Submissions are evaluated one the Root Mean Squared Logarithmic Error (RMSLE). The RMSLE is calculated
4: Rain lce Pallets Thunderstorn Mist, Snow
Fog
temp temperature atemp "feels like" temperature in Celslus humidity - relative humidity
Submission File
Your submission file must have header and should be structured in the following format:
windspeed wind speed
casual- number of non-registered user rentals initiated registered number registered user rentals initiated count number of total rentals
datetime, count
2011-01-28 00:00:00,0 2011-01-20 01:00:00,0
Figure 8: The content of Kaggle competition which contains task description, evaluation, and dataset description.
G KAGGLE EXAMPLE
We show kaggle competition example in Figure
H ERROR ANALYSIS
In our analysis, we identified several common types of errors that future work can address: (1) Misinterpretation of Data: This occurs when the agent misinterprets data, such as confusing year with a person's ID. Such errors indicate failure in accurately perceiving and understanding the dataset. (2) Inadequate Data Identification: When this error occurs, the agent fails to identify and retrieve the necessary data for the task. As result, the agent simply indicates that it lacks the data needed to complete the task and cannot compute an answer without further input. (3) Lack of Problem-Solving Strategy: Incorrect approaches formulas lead to erroneous answers. This highlights the agent's s deficiency in developing correct problem-solving strategy, which is crucial for deriving accurate results.
I IN-DEPTH ANALYSIS
I.1 QUALITATIVE ANALYSIS
To learn the intuitive performance the baseline, we first present the performance of GPT-4o on one testing analysis task in Figure 9
In Figure 9
we show the introduction of the challenge, the question, screenshot of portion of the data file, and the generation text from GPT-4o. The introduction part shows the background of the
10 sample in the figure is from the URI. https: /wwww kagg" e com cot.mm/mmmttttiioon......
inke-shhaiin--demmnn
19

---

Published as a conference paper at ICLR 2025
Data File
Jone
2011
1091
173
Darg
Desa
199
195.00
189
198
11110
175
116.0
133
183
195
Introduction
An election has been held for the Congress in the fictional country of Excelstan, Excelstan is small country and is divided into 9 Districts, named after letters of the Greek alphabet. Each District elects one member to Congress. assigned District Code based where they live. The District Code number between 105 and 194 and determines what District the voter votes for. District Alpha
There 1000 voters. Each
105 114
115 124
Bela
134
Gamma
135 144
Delta
145 154
Epsilon
155 164
Zeta
165 174
Ela
175 184
Theta
194
There are political partles in Excelstan competing for seats in the Congress.
parties are Red, Orange, Yellow, Green, Blus, Purple, Brown and Black.
Voters cast their ballot by numbering against their first choice. After this, they can choose provide preferences by numbering 2 against their second choice, 3 against their third choice, and so on up to against their eighth choice. Voters can choose how down they give preferences either 5, 7 or 8 parties preferential order.
provided, every
voted
When counting votes, if voter has not provided a preference number against particular party, assume they gave that party (i.e. they ranked any unmarked party equal last).
Each District counted independently of the other Districts, using only the votes from that District.
Question
How many
Delta District?
A.113 B.
114 C. 115 D. 116 E. 117 F. 118 G. 119 H. 120 121
Generation
To determine which district has the smallest number of voters, we need to count the number voters in each district the Codes provided the Here corresponding districts:
District Codes their
Alpha: 105 114 Beta: 115 124 Gamma: 125 - 134 Della: 135 144 Epsilon: 145 154 Zeta: 155 164
Eta: 165 174 Theta: 175 184 lota: 185 194
We will count the number of voters in each district by checking the range of Districl Codes.
Here is count of voters for district: 1, "Alpha (105 114) "Beta (115 124 *'aamma (125- 134)*
"Della (135 144)" * "Epsilon (145 154y" ZZea (155 164)" "Ela (165 174)" B. "Theta (175
184)" 9. **lota (185 - 194)*
number voters,
count occurrences District
data.
Let's count the number voters each district:
"'Alpha"*:
""Btt""
"*amma**: 10 voters ""Delta**: 10 "Epailon"*
"Zeta"": 10 voters "Eta"": 10 voters "Theta"*: 10 voters - "*lota": 10 voters
Since all districts have the same number voters (10), there na single district with the smallest number of voters.
Therefore, the
"A. Alpha B. Beta C. Gamma D. Delta E. Epsiton F. Zeta G. Eta H. Theta I. lota"
districts have same number voters.
Figure 9: We show testing example of data analysis task with GPT-4o. For of presentation, we have adjusted the format of the original answer, such as removing line breaks that do not interfere with reading. In addition, because the data file is too large, only show part of the data file.
whole challenge: The fictional country of Excelstan held an election for Congress, dividing its 1 1,000
20

---

Published as a conference paper at ICLR 2025
voters across 9 districts (named after Greek letters) where voters ranked their preferences among political parties. The data file shows how each voter voted. The question asks the model to count how many voters there in every district and answer which district has the smallest number of voters. Based on the generation from GPT-4o, we found that the model misunderstood the meaning of the introduction and misinterpreted the district code the voter identity code, leading to the wrong answer.
We further provide 6 additional qualitative analyses of generations from different baselines across data analysis and data modeling tasks. Table 9 shows a data analysis task GPT-4o cannot resolve. Table shows data analysis task GPT-4o resolved. Table [1] data analysis task AutoGen with GPT-4o resolved. Table [2] data analysis task AutoGen with GPT-4o cannot resolve. Table 13 shows a data modeling task AutoGen with GPT-4o cannot generate the submission file. Table 14 shows a data modeling task AutoGen with GPT-4 can generate the submission file.
21

---

Published as a conference paper at ICLR 2025
Table 9: In this sample, show data analysis task GPT-4o cannot resolve. GPT-4o misunder- stands the district code as the voter identifier.
Data File
Voloe
Dianket
837
890
165 355
138 730
834
848
193 898
138.71
181.400
899
189.41
191.35
112
251
185
287
22

---

Published as a conference paper at ICLR 2025
Formating Legend
User Varaale Assumption (0 decimal places)
1,00
User Varablle Assumption (2 decimal piacos)
1 1:0.00
User Varabbl Assumption (percentage)
50.00%
User Varabbl Assumption (sate)
11-06-155
Calculation decimal places)
1,009
Calculation iregative, descmal pleces)
(1,000)
Calculatior decimal places)
1,00000
Calculatier (percantage)
50
Call to populsied by model author
Calculatior (date)
31-Dec-15
formula
fon:
Uiits label
Variable Assumptton places)
1,309
Label
Label
Workinga
1,000
Top Sheet header
Label
Sedion
Labal
Introduction
An election has been held for the Congress the fictional country of Excelstan. Excelstan a small country and is divided into 9 Districts, named after letters of the Greek alphabet. Each District elects member to Congress. There are 1000 voters. Each voter is assigned a District Code based on where they live. The District Code is number between 105 and 194 and determines what District the voter votes for.
District Code District
105 114 Alpha
115 124
Beta
125 134
Gamma
135 144
Delta
145 154
Epsilon
155 164
Zeta
165 174
Eta
175 184
Theta
185 194
Iota
There are 8 political parties in Excelstan competing for seats in the Congress. The parties are Red, Orange, Yellow, Green, Blue, Purple, Brown and Black.
Voters cast their ballot by numbering against their first choice. After this, they choose provide preferences by numbering 2 against their second choice, 3 against their third choice, and so on up to 8 against their eighth choice. Voters can choose how far down they give preferences to. the data provided, every voter has voted for either 4, 5, 6, 7 or parties in preferential order.
When counting votes, if voter has not provided preference number against a particular party, assume they gave that party an 8 (i.e. they ranked any unmarked party equal last). Each District is counted independently of the other Districts, using only the votes from that District.
Question
How many voters there in the Delta District?
A.113
B. 114
C. 115
D. 116
E. 117
118
G. 119
H. 120
I. 121
23

---

Published as a conference paper at ICLR 2025
Answer I. 121
Generation
To determine which district has the smallest number voters, we need count the number of voters in each district based on the District Codes provided in the data. Here is the breakdown of the District Codes and their corresponding districts:
Alpha: 105 114 Beta: 115 124 Gamma: 125 134 Delta: 135 144 Epsilon: 145 154 Zeta:
155 164 Eta: 165 174 Theta: 175 184 Iota: 185 194
We will count the number of voters in each district by checking the range of District Codes.
Here the count of voters for each district: ** Alpha (105 14))** 2. **ta (115 124)** 3. **Gamma
(125 134)** 4. **Delta (135 144)** 5. **pssilon (145 154)** 6. *ete (155 164)*# 7. **Eta (165
1744** 8. **Theta (175 184)** **Iota (185 - 194)**
To find the district with the smallest number of voters, need to count the occurrences of each District Code in the data.
Let's count the number of voters in each district:
** Alpha** 10 voters
* *Beta** 10 voters
*Gamma*** **; 10 voters
**eelta**; 10 voters
**Epsilon**; 10 voters
** 10 voters
**Eta*# 10 voters
**Theta*#; voters
**Iota**: 10 voters
Since all districts have the same number of voters (10), there is no single district with the smallest number of voters. Therefore, the answer is: ** A. Alpha B. Beta C. Gamma D. Delta E. Epsilon F. Zeta G. Eta H. Theta I. Iota** All districts have the same number of voters.
24

---

Published as a conference paper at ICLR 2025
Table 10: In this sample, show data analysis task GPT-4o resolved.
Data File
Period Start Peried End
Uiibs
Sum
1.01
Production Information
Plant capacily
[MW]]
Availabiiity
95 00'
Hours in day
[P]
Shudown
Shutdown perio
T-Jun-17
T5-Ju-3 37
Shuldown per
15-0
Shutdown per
StAp-211
Shutdown per
6-
Shutdown perood
Jun 20
5-30-22
[date]
Feed costs
1:05
Revenues
Market price
Date Start
Date End
Price [$ Mahj
[dane]
[date]
Year
16
51 M7
78.00
Year
Tegg--7 7
31 S1-Mar-18
65.00
Year
1-Agr:15
S1 Mar-T 19
21.00 0.00
Year
1-Agr:19 19
31 21-Mar-00 20
20, 0.00
Yoar
1-Agr-30 20
31 Ma-Mor
0.00
Year
1:Agr21
31-Mer-22
00
Year
1=Agr-22
31-Mer-21
Year
1=Agr233
31-M8m-34
Year
1:Agr-24
31-Mer-25
Year
1:Aor:25
Year
1:Apr-25
31-Mar-27
Cap
por MWh]
price
1.04
Costs
Supplier
Price
15 per tonne)
100.00 .0
MWh produced per tonno
(MWh por tonne)
8 5.00
Maximum amount available quarter
[tonnes]
4.500.00 00
Supplier
Price
15 per tonne)
130.00
MWh produced per tonne
[MWh per tonne]
4.00
Fixed costa
per month)
75,000.00
25

---

Published as a conference paper at ICLR 2025
Formatting Legend
User Assumption decimal places)
Assumption places)
User Variable Assumption (percentage)
50.00%
User Variable Assumption (date)
ST-Dec-15
Calculation decimal places)
1,000
Calculation (negative, decimal places)
(1,000)
Calculation (2 decimal places)
1,000.00 00
Calculation (percentage)
50.00%
Cell to be populated by model author
Calculation (date)
31-Dec-15
Unique formula row
red font
Units label
[9]
Variable Assumption decimal places)
Workings
Sheet header
Label
Section header
End Sheet
Introduction
All the inputs mentioned below are provided in the workbook for this case study.
You working for company that is planning bid for 20% stake in the cashflows of biomass plant which burns wood chip to produce electricity. You have been asked to predict the cashflows the project from January 2017 until 31 December 2026, using quarterly model, and to your model recommend to your CEO the purchase price she should offer for the 20% stake.
You should assume that all invoices are settled in the same quarter they issued, there no inventory requirements and no taxes are applicable.
Where amounts are be inflated they are given in 2017 prices and inflation should be applied on January of each subsequent year. Do NOT round inflated prices to whole cents in interim calculations.
The plant is a 9MW plant; i.e. if it is running at full capacity it will produce 9MWh of electricity per hour. Your company's s engineers think it is reasonable to assume that the plant will usually run 95% of its capacity, 24 hours day.
There are several planned shutdown periods when no electricity will be produced at all from the start of the first day (at midnight) until the end of the final day (at midnight).
First day of shutdown
Final day shutdown
June 2017
15 June 2017
15 December 2019
January 2020
5 April 2021
25 April 2021
25 October 2023
10 November 2023
May 2025
26 May 2025
26 June 2026
5 July 2026
26

---

Published as a conference paper at ICLR 2025
Electricity sold as follows:
At the market price if this is between the cap price and the floor price.
The predictions for the market price have been provided the workbook accompanying this ques- tion on an April -March annual timeline, and should not be inflated.
At the cap price if the market price is higher than the cap price.
The cap price is $70 per MWh, inflated at 2% per annum.
At the floor price if the market price lower than the floor price.
The floor price is $45 per MWh, inflated at I % per annum
Your CEO is concerned that the market price predictions for the sale of electricity may be overly optimistic. She has therefore asked you to run some the analysis (see Questions 11 and 13) as though electricity were always sold at the floor price. It will be beneficial if you develop your model in way that makes it easy to switch between modelling the electricity price as described above and modelling only the floor price.
In each quarter, the project purchases exactly enough wood chip produce the MWh output of the plant in that quarter.
The plant has two possible wood chip suppliers. The project first buys the wood chip from the supplier that gives them the lowest cost per MWh in that quarter, and, if necessary, tops this up with wood chip from the other supplier.
The first supplier sells wood chip the following terms:
Wood chip costs $100 per tonne, inflated at 2% per annum
tonne of wood chip will produce 3.5 MWh electricity
Up to 4,500 tonnes may be purchased per quarter
The second supplier sells wood chip on the following terms:
Wood chip costs $130 per tonne. This fixed price that should be inflated
tonne of wood chip will produce 4 MWh of electricity
An unlimited amount of wood chip may be purchased per quarter
The project incurs $75,000 of fixed costs per month. This amount is indexed at 1.5%.
The company's s required rate of return for this project is 10%, and you should assume that there is terminal value or disposal costs for the 20% stake at the end of the ten years. To determine the purchase price your company should offer:
i) Discount using an Excel function (or functions) that discounts on an actual/365 basis;
fi) Assume acquisition date of 31 December 2016; and
iii) Assume that all cashflows occur at the end of the quarter in which they occur.
When used in the question wording, "Q1" refers to the period January to March, "Q2" to the period April June, "Q3" to the period July to September and "Q4" to the period October December. For Questions 6 to 17, perform your analysis assuming 10% required rate of return. For Question 18, perform your analysis assuming 4% required rate of return.
Question
What is the cap price in April 2025?
A. $81.94
B. $81.95
C. $81.96
D. $81.97
$81.98
F. $81.99
G. $82.00
H. $82.01
27

---

Published as a conference paper at ICLR 2025
I. $82.02
Answer
Generation
To determine the cap price April 2025, we need to apply the annual inflation rate of 2% to the initial cap price of $70 per MWh, starting from January 1, 2017. The cap price is inflated annually on January 1st of each subsequent year.
Here are the steps to calculate the cap price in April 2025:
Initial Cap Price (2017);** $70.00 ** Annual Inflation Rate:** 2%
We will apply the inflation rate for each year from 2017 2025.
[Cap Price in 2018 70 x + 0.02)
1.02 71.40)
[Cap Price in 2019 71.40 x 1.02 = 72.828
[Cap Price in 2020 = 72.828 x 1.02 = 74.28456]
[Cap Price in 2021 74.28456 1.02 75. 75.7702512
[Cap Price in 2022 75.7702512 1.02 77. 77.285656224]
[Cap Price in 2023
77. 17.285656224 1.02 78.83136934848
[Cap Price in 2024 78.831 88.83336934848 1.02 40.4079967354496
[Cap Price in 2025 30.40799673544466 x 1.02 82.01615667000
Rounding to two decimal places, the cap price in April 2025 is:
[Cap Price in April 2025 82.02]
Therefore, the correct answer is:
I. $82.02
28

---

Published as a conference paper at ICLR 2025
Table 11: In this sample, show data analysis task Aotogen with GPT-4o resolved.
Introduction
friend has approached you asking for help estimating the winning odds on a heavily modified variation of popular dice game. His version of the game works as follows:
The player rolls 3 dice, each die has 6 sides
If the total rolled is 3,4,5,16, 16,17 77 or 18 the player loses
If the total rolled 7 or 11 the player wins
If any other number rolled, the player rolls again
Rerolls work the same way the initial roll, except that if the player rolls the same number their first roll they win. For example if the player rolls 15-13-6-15 they would win
You quickly identify this as a statistical problem but given the complexity and the fact that you slept through most of your statistics classes university, you decide
a Monte Carlo simulation to estimate the odds.
When preparing your analysis you should assume the following:
No more than 50 rolls are required achieve result
5000 outcomes are required to prepare your Monte Carlo.
Question
What is the probability of the player winning the game?
a) <60%
b) 60%-65%
c) 65%-70%%
700%
Answer
d
Generation
Role: Assistant
To estimate the probability of the player winning the game using Monte Carlo simulation, will follow
these steps:
1. Simulate the rolling of three 6-sided dice.
Check the result of the roll against the losing, winning, and reroll conditions.
3. If the result is reroll, continue rolling until a win loss is determined, with maximum of 50 rolls. Repeat the simulation 5000 times to gather enough data.
5. Calculate the probability of winning based on the simulation results.
Let's write Python script to perform this Monte Carlo simulation.
29

---

Published as a conference paper at ICLR 2025
pythen
filename: monte_ caare__crll__oe die___ice________e______ game.
import random
def roll_dice
return random. randint (1, + random. randint (1, + random, randint (1,
def simulate_game ():
initial_roll - _el1_dice[ () if initial_roo roll in 13, 4, 5, 16, 17, 18) return False Player loses elif initial roll 17, 111: return True / Player wins
for in range (49) Up to 50 rolls including the initial roll new_roll coll_dice() new_roll [3, 4, 5, 16, 17, 18]:
return False # Player loses
elif new_roll == initial_roll or new_roll in [7, 11]: return True $ Player wins
return False # If no result after 50 rolls, assume loss
def monte_carlo_simulat in (nur_simulations)
wins for range (num simulatiens):
if simulate_game gameatteeeam@(a_ (): wins += 1
return wins num_simulations
Run the Monte Carlo simulation with 5000 comes num sum_simulations = 5000
winning_probabinn = (winning_probability 100: 2f)**)
siuu_simllation)
print (f"Rinning Probability:
if Determine the probability range
0.60;
print("a <60*")
elif 0.60 wimming_probabllity 0.65:
print ("rt(((b) ("b) 60$-654*)
elif 0.65 <= sinning_probabiii < 70:
print ("c) 658-708")
else:
print ("d) >708")
Please save the code in a file named monte connteccrlloddee dice www.dddiceeeeeeee ggame.py and execute it. The script will print the winning probability and the corresponding range. Let me know the result so we can determine the correct answer to the question.
Role: User
exitcode: 0 (execution succeeded) Code output: Winning Probability: 73.36%
d) 70%
Role: Assistant
Based on the Monte Carlo simulation, the winning probability for the player is approximately 73.36%. Therefore, the correct answer to the question is:
d) 70%
30

---

Published as a conference paper at ICLR 2025
Table 12: In this sample, we show a data analysis task AutoGen with GPT-40 cannot resolve.
Data File
PK Kos 24th Mar- 2014 0.364 Ivb
16
fo- 2014
1.
1.30
koh
8PM
Thu
20-
1.933 keb
6PM
2br6-tor-9914
0.421
keh
14M
Priday
0,209 kanh
S4M
Tue
19th-kug-2014
1.224
127M
Mon
2th-
201-14 1,216 bub
AM
Tue
25-Nox-2014
0.437 Anh
BAM
14-Aux-2014
0. 523 keh
10
4PM
25ch-Jas
[Bth-Jas=2014
652besh
11
4PM
11th-Feb-2916.00
O. 509 keh
12
1M
Friday 11-Jul-2014
9.547 bsb
15
121M
Sue 26-9ee-2014.0 D. 885kes
14
PM
Tue 01-1pr-2014 9.557 ket
16
3MM
Tue 9412-Feb-2014 0.293
16
11
Sunday M211- 0.314 keb
17
PM
24:2an-2014
2,94 keh
19th:_un-2014.00 378 knb
33re-Sep-2014.00
963
12PM
26-Jul-2014
leth
11AM
Mendar 23rd-Jen-20044 010
Teendar 01-1u1-80144 0.626
Tursday 11it-Feb-2814
0.547
11
13-34e-8014. 578kkb
Sut. 36-3ee- 2014. 0.392 koh
10AM Sat. 2211-War-2011 0,511 keh
Mon 2814-A0--3901.000
Rth-Mar-2014
B1-Jas-3014
17:1
Ist-Jan-2014
24:1-0e1-2014
03-0e1-2014 625kkh
10FM
15-Nor-2014
kes
08- 05-Mey-0014
0.88kkhh
10PM
25-1
2054
Sop--5pp-2011
24-1ee-1014 385
San 23-Mar-2014
1.67 keh
12AM 9-1an-9014 0.54 keb
AM Thu 15th-Mer-2014
6.
keb
44
11PM Pri 31-0et-20014
588 keb
12PM Frider 18sb-hog-2014
6.852 Jnb
8AM 15- Mar 2014 6. 959 keh
47 PM 21st- (ass-lla-22210 2014 2,92 bob
48
10PM 2eii--0e-2011,..30
kuh
49
A 02nd Mayd-May--000 2014
0.272khh
50
PM 30-8ov-2014
0.403 keh
51
6AM Sun 13s
0,345 bab
52 1PM 3 2-Jui-2814.00
0,4 bub
55 PM Priday
17-140-2014 0. 6.23hbh
54 PM 7-Jan=2014_ 6. 84keh
55 8PM 31- 31-May-20144 0. 326 kah
56 PM Thu 18th- 2etb-9ee-2014
1. 1.50keh
57
1PM The F5t1-Febb2214__
0. 6.2 keh
58
1PM Tue 17- J01--00-
514 Jeh
59
11 Mon 11-1ug-2014 0.618
60
13PM 1D-Feb-2914 0.591 keh
61
1IPM San. 511-901-2014__ 254hhh
62
9PM Fri 14:1 0.314
63
3AM Sth:
185
64
6PM The 2-14 6, 731 kmh
2AM Thu 0b-1an-2014__ 5.kkb
18th-Bec-2014
11
knb
9-Jas-2014_ 8.8 kos
5-Feb-2014
228ksh
1611-)a1-2014_ 5.
12AM 20-4
0,345
12
2014
2914-Jes:2914.00
3-Sep-1014 c.0kksh
0.624
7-Ju1-2014
15th-Sor-3314
05 05-Mar- 2014
0.20khh
18th-Aug 2014 611teh
20-8 151
Monday 17th-
keh
Introduction
Just over a year ago you had a smart electricity meter installed at your home. This partieular model of smart meter reports your usage to your electricity provider every hour. Your electricity contract is due for renewal and being a top financial analyst you decide to identify which contract will minimise your annual electricity cost.
There are 3 different types of electricity contracts available to you. Each are described in the table below:
31

---

Published as a conference paper at ICLR 2025
Plan
Explanations
No Flex
The cost per kWh of electricity constant for the entire year The cost per kWh of electricity fluctuates depending on the month
Monthly Flex
Hourly Flex
The cost per kWh of electricity fluctuates based on the time of day
To decide which contract optimal, you decide to compare the costs under each of the contracts assuming your usage remains exactly the same as last year. You've contacted your current electricity supplier to obtain your electricity usage history for the last year. Unfortunately they provided to you poorly structured and unsorted format, so you will need to clean it up prior to doing your analysis. In the data the hour field identifies when the hour starts, so 8AM would from 8:00AM until 8:59AM.
You'l ve placed your usage history and the specifics of each of the electricity contracts into the supplied work- book. Your usage history located the "Usage" sheet while the specifics about each of the electricity contracts is located on the "Contracts" sheet.
Question
What is your average electricity usage per hour in February?
0.760kWh
O. Wh
0.808kWh
0.833kWh
Answer
Generation
Role: User
To answer the question, we need to follow these steps: Load the Excel workbook and read the "Usage" sheet. Filter the data to include only the usage for February. Calculate the average electricity usage per hour for February.
Let's start by writing Python script perform these steps.
filename: calculate_februuryyuue
import pandas pd
load the excel workbook file_ = usage_ df pd. read excel (file pathh sheet name=' Usage')
xisx'
Convert the 'Date' column to datetime format ( = pd.to_ to__attttiime (usage_df ['Dae''))
Filter the data include only the usage for Februazy february_usage usage of [usage_de df D' Date' jtd........ month 2]
Calculate the average electricity usage per hour for February vverage_usage_februar tebruary_usage Usage (kWh) mean
Print the result
print (f"Average electricity usage per hour in rebruary: kwh")
Please save the code in file named calculate february usage.py and execute it. The script will print the average electricity usage per hour in February.
Role: Assistant
Exit code: (execution failed)
Code output:
32

---

Published as a conference paper at ICLR 2025
Traceback (most recent call last): File
indexes/base.py" line 3805, in get_loc
return self._engine.go. get lcc (casted key) catiineeee___ooccaeeeeeeeeeee File "index.pyx"" line 167, in paddes_ File "index.pyx", line 196, in aa File oan.aeeeiLlllaaaaaaaaaaaaaaaaaaa 1ine
line 7081, in
4 [aa0ss__hisshaaaaaaaeeee.p.ppaa
File henaaaaeiiiiaaaaaaaaaaaaaaaaaaaa
7089, in
4 pandas._ htw.aa_aaeibshhhaaaeeeeeeeeeee
KeyError: 'Date'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "//sers/ htt.ee/nwnncne/ecchaaeooo/e/ee/e Secculaat__februrr__ussss
line 10, in <module>
usage_df["
pd.to_datetinne
File
frame.py", 1ine 4102, in getitem___
indexer self. columes get__oo (key) File attaacicc./assatttto///////ttooo
indexes/base.pe py", line 3812, in get_loc raise KeyError (key) from err keyError: 'Date'
Role: User
The error indicates that the 'Date' column does not exist in the "Usage" sheet. Let's inspect the columns of the "Usage" sheet understand the structure of the data.
We will modify the script to print the column names of the "Usage" sheet.
filename: inspect_ peapectt_uaae___ee Seragg_shhee.pppe
import pandas as pd
Load the Excel workbook file_path
-att xlax'
usage_df pd. read_ (file_path, sheet_name- Usage')
Print the uumn names to inspect the structure print ("Columns in 'Jsage' sheet:", usage wow.um...c.llmmmooooo tolist))
Please save the code file named inspect pheett and execute The script will print the column names of the "Usage" sheet. This will help us identify the correct column names to use in calculations.
Role: Assistant
Exit code: 0 (execution succeeded)
Code output:
Columns in 'Usage' sheet: :
3 PM Mon 2014
0.384 kwh
Role: User
appears that the "Usage" sheet contains a single column with poorly structured and unsorted format. We need to clean and parse this data to extract the date, time, and usage information.
33

---

Published as a conference paper at ICLR 2025
Let's write a script to clean the data and extract the necessary information. We will then filter the data for February and calculate the average electricity usage per hour.
filename: clee__enkccclllllacccoorrrrrrrr
import pandas pd import
Load the Excel workbook filc_path =
xlsx'
usage_df pd. readd_eeceee (file_path) "aaaee"
Axtract the single column raw_data Saage_df.1lcch 0]
Define function parse
data
def parse_usage garre__ssge__entry(eee
match - re. re. (r' \s* (\d+ [APM] +) 1s+ (ww+
+ [1]112)(aa2)) (2) (2)-W+11::4) (1ah--d:4)_((dddd_____ ([Vd.1+)
kwhls*', entry)
if match:
time, date, usage match. groups)
return pd. SerSeries [time, date, float (usage)
else:
return pd. Series ( (None, Nene, Nonel)
Apply the parsing function to the raw data parsed raw_data aapllyppppyyl... (parse_usage_enta
['mime', "Date', 'Usage (kwh) []
Drop rows with missing values parsed data.dropna (inplace=True)
Convert the "Date' column to datetime format datac Datee' pd.te_datetine (parsed_data Date'', format=/ id-bb-8Y")
Pilter the data to include only the usage for February february_usage parsed_ [pared_d [parsed_data Date' l.tee]]ddd.mttttt == 2]
Caicuiate the average electricity usage per hour for February sebruary_usage Usage (kWh) meea
Print the result
print (f"Average electricity usage per hour in February: kWh")
(averaag_uagee_eerrraae
Please save the code in a file named clean_and htt.aandddccllaa calculate februuay .usage.py and execute it. The script will clean the data, filter for February, and print the average electricity usage per hour.
Role: Assistant
Exit code: (execution failed)
34

---

Published as a conference paper at ICLR 2025
Code output:
Traceback (most recent call last):
File http:////cooooooeeooooooocaaaaaa
line 30, in
+ <rodule>
parsed_datal =
format=' Sd-bb--Y
File
tools/datetiims.ppy, 1ine 1067, in to_datetime
values convert_listtlkkeerrg_....o
format)
Pile
tools/datet mools//atetimes.pp" 1ine 433, in _tt.ooeerr_liaalkke_daeeeeeeee ims return array_ strptimessetmmeeeeeeee setttime_with__ee with aath_fallback (arg, name, utc, format, exact, errors)
File "strptime.pyx", line 501, in tiii wtwry_strrttiiinneeeee File "strotime.pyx", line 451, pandas. sttp::......s.....tt.pp.pp..p..e stapy File "strptime.pyx", line 583, in htt.:stssllsssstttee.epppe..e
+
ValueErrer: time data "Wed 16th-Jul-2014 doesn't match format "id-ab-8Y", position 0. You might want try: passing 'format
your strings have a consistent format;
passing 1a0ma0==I0086011 if your strings are all IS0860: but not + necessarily in exactly the same format; passing 'format='mixdd"> ', and the format will be inferred for each element individually. You might want to use 'dayirrst" alongside this.
Role: User
The error indicates that the date format in the data does not match the format specified in the pd.to_datetime function. The date format in the data includes the day of the week and ordinal suffixes (c.g.. "16th-Jul- 2014").
We need to adjust the date parsing to handle this format. Let's update the script correctly parse the date format.
35

---

Published as a conference paper at ICLR 2025
filename:
import pandas pd import re
Load the Excel workbook file_path =/ usage_df pd. read_exe (file_path, sheet_name- Usage')
xlax'
extract the single column raw data = usage df. dfocll 0]
Define a function to parse the raw data def parse_usage_entry" (entry):
match = re. re.match (r' \8* (Nd+ [APM] +) (s+
\d(11, [a-z] (2) -w ([ kwhle*', entry)
if match:
time, date, usage match. matchh groups
# Remove the day of the week and ordinal suffix from the date date seb ()d(1,21) Ind/ 11' date.split('' ' 11 (1))
return pd.Series ([time, date, float (usage)
else:
return pd. Series ( ([None, None, Nonel)
Apply the parsing function to the raw data parsed_data raw_data ( (parse_usage_ene parsed data. columns ('Time', "Date', "Usage (kWh)
Drop rows with missing values dropa (inplace-True)
Convert the "Date' column to datetime format parsed_data Datee' = pd.to_datetime (parsed_datal Date'), ), format-/ id-
Pilter the data include only the usage for February tebruary_usage dateeddaatll
Date' ] l.dt .onthh 2
Calculate the average electricity usage hour for February ferrageyusag__eebrrr february_usage Usage (kWh) ( m.meammenn.
Print the result
print (E"Average electricity usage per hour in February: Laverage_usage feaggurrebuuryy 3f; kWh")
Please save the code in a file named clean and calculate february eebruary-usage.pp .usage.py and execute it. The script will clean the data, filter for February, and print the average electricity usage per hour.
Role: Assistant
Exit code: 0 (execution succeeded)
Code output:
Average electricity usage per hour in February: 7.0 kwh
Role: User
The calculated average electricity usage per hour in February is 0.720 kWh. However, this value does not match any of the provided options (0.760 kWh, 0.784 kWh, 0.808 kWh, 0.833 kWh).
36

---

Published as a conference paper at ICLR 2025
Let's re-examine the data and ensure that all entries for February are correctly included and that the calcu- lations are accurate. We will also print sample of the filtered data for February to verify its correctness.
filename: verify_ fer.uyy_ebbrrrry_uaag.....
import pandas pd import
Load the Excel workbook filc_path =
xlsx'
usage_df pd. read_exceel (file_path, theet_name== "aaee"
Axtract the single column raw_data
0]
Define function parse
data
def harse_usage wwrre__ssge_enntrylee ent
match - re. re.match (r' \s* (\d+ [APM] 1s+ (1w+
+ [a-1)))[[a2] (2)-W=+-\d:4)) ( ([Vd.1+) kwhls*', entry)
if match:
time, date, usage match. groups) # Remove the day of the week and ordinal suffix from the date date re. sub ()d(1,2)) rdd x''1', date.split ' 1) [1))
return pd. Series ([time, date, float (usage)
else:
return pd. Series ( ([Nene, None, None])
Apply the parsing function to the raw data parsed_data raw_data apply pparse_usage_entee parsed dola.columns ['Time', 'Date', 'Usage (kKh)
Drop rows with missing values basseddddta.drrppp.. (inplaco-True)
Convert the "Date' column to datetime format peased_datae ('ate') pd.to_datetime (parsed_ data Date"", format-' 3d-
Pilter the data include only the usage for February february_usage parsed datalparsed data 'Date' l..d..mmott. == 2.
Print a sample of the filtered data for February print ("Sample of February usage data:") print (february_usage.. h))
Calculate the average electricity usage per hour for February = february_usage usage I' Usage (kWh) mean ()
Print the result
print (f"Average electricity usage per hour in February: kWh")
laveragg_usgee_ferruuyy
Please save the code in file named fenffaffeburryuuuse jusage.py and execute it. The script will print sample of the filtered data for February and the average electricity usage per hour. This will help us verify the correctness of the data and calculations.
Role: Assistant
Exit code: 0 (execution succeeded)
37

---

Published as a conference paper at ICLR 2025
Code output:
Sample of February usage data:
Time Date Usage (kWh)
571 11 PM 2014-02-20 0.72
Average electricity usage per hour in February: 0 720 kwb
Role: User
The output indicates that the filtered data for February contains only entry, which is likely why the average usage is 0.720 kWh. This suggests that the data might be incomplete or not properly parsed. Let's re-examine the raw data and ensure that all entries are correctly parsed. We will print sample of the raw data to understand its structure better.
filename: inspect_ saapect_raw_coo raw_data.ppappp
import pandas as pd
Load the Excel workbook
file path '..000000000 usage_df = pd. read excel (file (file_path, sheet sheet_name= name=' Usage')
xlsx'
Print sample of the raw data inspect its structure print ("Sample raw usage data:")
print (usage_df. head (10))
Please save the code in file named inspect_raw_datte and execute it. The script will print sample of the raw data to help us understand its structure and identify any issues with parsing.
Role: Assistant
Exit code: 0 (execution succeeded)
Code output:
Sample of raw usage data:
PM Mon 24th-Mar-2014
0.384 kwh
5AM 15-Aug-2014, 1.201 kwh
8PM Thu 20-Mar-2014
1.523 kwh
2
6PM 23rd-Agr-2014
0.424 kwh
3 1AM Friday 19th-Dec-2014
0.209
kwn
5AM Tue 19th-Aug-2014
Role: Assistant
The raw data contains various inconsistencies, such as extra spaces, underscores, and different formats for the date and time. We need refine our parsing function to handle these inconsistencies more robustly. Let's ss update the script to clean and parse the data more effectively
38

---

Published as a conference paper at ICLR 2025
filename:
import pandas pd import re
Load the Excel workbook file_path 110000---00a---aaaaa111111110000 usage_df read_exe (file_path, sheet_name= Usage')
xlax'
extract the single column raw data = usage df. dfolll 0]
Define a function to parse the raw data def
(entry):
$ Remove extra spaces and underscores entry re. sub (r' [_Ns]',,
entry) strrpp
match re.match (r' ()d[1,2) [APM]+] Ast (Ar
4 (d(1,2) [a-z] (2) (w)=)))-dd44))) Ast ([)d.]+) kwh', entry)
if match:
time, date, usage = match. match.groups. groups
# Remove the day of the week and ordinal suffix from the date date re. (r' ()d(1,21) (nd s'\1', date.split (" ' (11)
return pd. ([time, date, float (usage) ))
else:
return pd. Series ( ([None, None, None])
Apply the parsing function to the raw data parsed data = raw raw__a__ ley_dataaapp appla (parse_usae___enne wuwguuusaaeeeeee entree coaseedddata.collll
= ['Time', "Date', 'Usage (kKh)
Drop rows with missing values datse (inplace=Irue)
Convert the 'Date' column to datetime format parsed_data Date'] pd.to_datet (parsed_datal Date'), format-' 8d-
Filter the data to include only the usage for February eebruary_usage parsed dataaaaaaaaeeeeeeeee data Dattt J month == 2
Print a sample of the filtered data for Pebruary print (oatt""aaal ("Sample of February usage data:") print Stebruary_usage.heed
Calculate the average electricity usage per hour for February averagc__sage__ecnuu
february_usage I' Usage (kWh) mman ()
Print the result
print (f"Average electricity usage per hour in February:
[average www.saageeeruurrr.33333 kWh")
Please save the code file named clean and calculate february usage v2.py and execute it. The script will clean the data more effectively, filter for February, and print the average electricity usage per hour.
Role: Assistant
Exit code: 0 (execution succeeded)
39

---

Published as a conference paper at ICLR 2025
Code output:
Sample of February usage data:
Time
Date Usage (kWh)
254
AM 2014-02-04
0.264
571
11
FM 2014-02-20
0.720
960
11
PM 2014-02-08
1.116
1644
AM 2014-02-03
0.230
1767
12 PM 2014-02-15
0.576
Average electricity usage per hour in rebruary: 0. 855 kwh
Role: User
The calculated average electricity usage per hour February now 0.855 kWh. This value still not matching any of the provided options (0.760 kWh, 0.784 kWh, 0.808 kWh, 0.833 kWh).
Let's verify the correctness the data and calculations by printing the total number of entries for February and the sum of the usage values. This will help ensure that all entries are correctly included and the calculations are accurate.
40

---

Published as a conference paper at ICLR 2025
# filename:
ve::fyy_rrrrary.waaa_....
import pandas pd import re
Load the Excel workbook file_path 11AA011-oouc-1rea1a1111:10000000 usage_df pd. read_exe (file_path, sheet_name- Usage')
xlax'
extract the single column raw data = usage df. dfolll 0]
Define a function to parse the raw data def
(entry):
$ Remove extra spaces and underscores entry re. sub (r' [_Ns]',,
entry) strrpp
match re.match (r' ()d[1,2) [APM]+] As+ (Ar
4 (d(1,2) [a-z] (2) -w)=))--d:44))) Ast ([)d.]+) kwh', entry)
if match:
time, date, usage = match. groups) # Remove the day of the week and ordinal suffix from the date date re. (r' ()d(1,21) (ndi s'\1', date.split ' (11)
return pd. ([time, date, float (usage) ))
else:
return pd. Series ( ([None, None, None])
Apply the parsing function to the raw data parsed data = raw raw__a__ ley_dataaapp apply (parse_usage_eenee entree coassedddata.collllc
= ['Time', "Date', 'Usage (kKh)
Drop rows with missing values daraa (inplace=Irue)
Convert the 'Date' column to datetime format parsed_data Date' pd.to_datet (parsed_datal Date'), format-' (d-
Filter the data to include only the usage for February gebruary_usage parsed data parsed data DDatt" J.dt.month == 2
Print the total number entries for February and the sum of the usage 4 values
print(""Tota (E"Total number of entries for February: (fe (")
print (f"Sum of usage values for February: (february_usage U'age +kWh) (].uum())"
Calculate the average electricity usage per hour for February average_ = february_usage Usage (kWh) m.man
Print the result
print (f"Average electricity usage per hour in February:
[average_ usage Sebruary:.33. kWa")
Please the code in file named verify cecbruary-uaaee... total number of entries for February, the sum of the usage values, and the average electricity usage per hour. This will help verify the correctness of the data and caleulations.
and execute it. The script will print the
Role: Assistant
Exit code: 0 (execution succeeded)
41

---

Published as a conference paper at ICLR 2025
Code output:
Total number of entries for February: 34 Sum of usage values for February: 29.08 29.0800000000006
Average electricity usage per hour in February: 0. kwh
Role: User
The output indicates that there are 34 entries for February with total usage of 29.083 kWh. The average electricity usage per hour is calculated as 0.855 kWh, which is consistent with the previous result. Given that the calculated average does not match any of the provided options, let's re-evaluate the possible reasons:
42

---

Published as a conference paper at ICLR 2025
Data Completeness: Ensure that all February data entries are included. Data Accuracy: Verify that the parsing function correctly extracts all relevant data. Let's print all the entries for February to manually verify the data.
filename: prinn__february_enrriee.
import pandas as pd import re
Load the Excel workbook filc_path
xlsx' usage_df = pd. excel (file_path, sheet_name=' Usage')
extract the single column rat_data usage_df. ilo
Define a function to parse the raw data def parse sarse_usage wst.u_saatyyyeeeeeeeeee (entry)
entry re. sub (r' [_ts]+', underscores
Remove
spaces
entry) (strip()
match = re. ma.hatch (r' ()d(1,2) [APM] 1s+ (vet
4 \d(1, [a-z] (2)- \st ([1d.1+) kwh', entry)
if match:
time, date, usage match. ()oups))
# Remove the day of the week and ordinal suffix from the date date re.sub ()d(1,2)) (st Indirdith))) V''1' date.split ' 1) (1))
return Serier ([time, date, float (usage)
else:
return pd. Series [[one,, None, None]]
Apply the parsing function to the raw data parsed data = data.aaaaaa..o a (parse usage enttry) parsed columcolumns ['Time', 'Date', 'Usage (kWh) ']
Drop rows with missing values dabbeddddtaaddrppp... (inplaco-True)
Convert "Date' column datetime format
persed_datall ('ate'' pd.to_datetimg (parsed_ data [" Date'), format-/ 3d-8b-8Y
Pilter the data include only the usage for February february_usage parsed parsee data| DDte'' l....t.ooo month == 2.
Print all entries for February print ("All ("All entries for February:") print (february_usage)
Calculate the average electricity usage per hour for February = february_useee usage [' Usage (kWh) '] mean ()
Print the result
print (f"Average electricity usage per hour in February: kWh")
faverage_usge_feeuuzyy....
Please save the code in file named print february entries.py and execute it. The script will print all entries for February and the average electricity usage per hour. This will help us manually verify the data and identify any discrepancies.
Role: Assistant
Exit code: 0 (execution succeeded)
43

---

Published as a conference paper at ICLR 2025
Code output:
Al1 entries for February:
Time
Date Usage (kWh)
254
AM 2014-02-04
0.264
571
11
FM 2014-02-20
0.720
960
11
PM 2014-02-08
1.116
1644
AM 2014-02-03
0.230
1767
12
PM 2014-02-15
0.576
1869
AM 2014-02-16
1.685
2250
AM 2014-02-16
0.414
2261 AM 2014-02-17
0.262
2454 AM 2014-02-18
0.264
2611 AM 2014-02-09
0.421
2718 PM 2014-02-22
0.896
2813
12 AM 2014-02-23
0.922
2851 PM 2014-02-27
0.785
3118 11 AM 2014-02-11
0.570
3141 PM 2014-02-07
1.630
3329 11 PM 2014-02-15
1.008
3530 7 PM 2014-02-12
1.793
3623 11 AM 2014-02-05
0.492
3976 PM 2014-02-11 2.333
4191 2 AM 2014-02-08
0.371
4376 PM 2014-02-19
0.408
4739 10 AM 2014-02-17
0.475
5058
2014-02-08
1.145
5838 12 AM 2014-02-06
0.335
5879 7 PM 2014-02-22
1.793
6033 PM 2014-02-17
0.814
6155 AM 2014-02-26 0.250
6318 PM 2014-02-21 2.203
7214
PM 2014-02-16 1.145
7523 7 PM 2014-02-20
1.901
7570 11 AM 2014-02-22
0.698
8047
2014-02-09
0.356
9274 10 PM 2014-02-02
0.606
8371 AM 2014-02-10
0.202
Average electricity usage per hour in February: 0.855 kWh
Role: User
The data for February appears to be correctly parsed and includes 34 entries. The average electricity usage per hour is calculated as 0.855 kWh, which still does not match any of the provided options. Given the discrepancy, let's consider the possibility that the provided options might be based on a different interpretation of the data. For example, the options might be based on a different subset of the data or a different calculation method.
To ensure thoroughness, let's calculate average usage per day in February and see matches any the provided options when divided by 24 (the number of hours in day).
44

---

Published as a conference paper at ICLR 2025
filename; oalouleee_gellll...eesse.e/oooee
import pandas pd import re
Load the Excel workbook
file_path ./
MO1 4014--ound-1-Dea ing th-Data waraattawrrkoook... x11xx
usage_df pd. rxce_ (file (file_path, sheet_name- Usage')
Extract the single column raw data usage df. ifliloci;
Define a function to parse the zaw data def parse_usage_entry (entry):
Remove extra spaces and underscores
entry re. (r' [[s]]',
entry) (srrpp()
match re. ra.catch (r' (1)(1,2) [APM] +) \s+ (1ot
4 (d(1,, (2)- \st ([1d.]+) kwh', entry)
if match:
time, date, usage match. groups
& Remove the day of the week and ordinal suffix from the date date = re.sub ()d(1,21) (st Ind]rd]th) 1'\1', date.split((' ' 1) 111)
return pd.Series ([time, date, float (usage)
else:
return pd. Seriess ([Nne,, None, Nonel]
Apply the parsing function to the raw data parsed data = raw htw_ddta..ae aata..pply (parse usage (eage_entry) collu.cmmss = ['Time', 'Date', 'Usage (kWh)
Drop rows with missing values parsed_data. droppa (implace-True)
Convert "Date' column to datetime format carsed_datal Date' pd.to_dateting (parsed_datal datal'Datat' DDte"" I, format=' 8d-8b-81')
Filter the data to include only the usage for February february_usage parsed_dat data (parsed_data parsed_dattpaarss____ta D'tat month -- 21
Calculate the total usage for February Lataal_saaee_eebie = february_usagel Usage (kih) '] ].sum
Calculate the number of days in February
days_in_february fcbruary_usage ('Date' Date' ] .t wap.ddy.nnnnnn.. nunique
Calculate the average usage per day in February average_usagg_per___n
februuurr dayb_in_
Calculate the average usage per hour based on the daily average
average usage per hour average Saarage_usaae__eeee per day / 24
Print the results print (f"Total usage for February: ttotal_.sag_feerrryy.. kwh")
print (f"Number of days in February: (daya_in
print (f"Average usage per day in February: (average_usage_pr__cdl ..3.: kWh") print (f"Average usage per heur in February (based on daily average) .3f; kWh")
Laverage_usage_per__ree
Please save the code in a file named calculate daily average february.py and execute it. The script will calculate the total usage for February, the number of days in February, the average usage per day, and the average usage per hour based on the daily average. This will help us verify if the provided options are based a different calculation method.
45

---

Published as a conference paper at ICLR 2025
Role: Assistant
Exit code: 0 (execution succecded)
Code output:
Tetal usage for February: 29.083
Number of days in February: 22 Average usage per day in February: 1 1.322 kwh
Average usage per hour in February (based on daily average) 0. 0.055 kwh
Role: User
The calculated average usage hour based on the daily average is 0.055 kWh, which is significantly lower than the provided options. This suggests that the provided options are not based on daily average calculation.
Given that the initial calculation of 0.855 kWh is consistent and thorough, it seems that the provided options might be incorrect or based on different dataset interpretation.
To conclude, the average electricity usage per hour in February, based the provided data and thorough analysis, is 0.855 kWh. This value does not match any of the provided options (0.760 kWh, 0.784 kWh, 0.808 kWh, 0.833 kWh), indicating potential issue with the provided options.
you have any further instructions need additional analysis, please let me know. Otherwise, can conclude that the correct average usage per hour in February is 0.855 kWh based on the given data.
46

---

Published as a conference paper at ICLR 2025
Table 13: In this sample, show data modeling task AutoGen with GPT-4o cannot generate the submission file. The AutoGen system generates the code required for our data modeling task, but it does not execute the code to produce the submission file. This limitation highlights challenge with current models in terms of instruction-following particularly in tasks that require both code generation and execution to complete the data modeling process.
Introduction
This is one of the two complementary forecasting tasks to predict COVID-19 spread. This task based on various regions across the world. To start single state-level subcomponent, please the companion forecasting task for California, USA.
The White House Office of Science and Technology Policy (OSTP) pulled together a coalition research groups and companies (including Kaggle) to prepare the COVID-19 Open Research Dataset (CORD-19) to attempt to address key open scientific questions on COVID-19. Those questions are drawn from Na- tional Academies of Sciences, Engineering, and Medicine's (NASEM) and the World Health Organization (WHO).
Kaggle is launching two companion COVID-19 forecasting challenges to help answer subset of the NASEM/WHO questions. While the challenge involves forecasting confirmed cases and fatalities between March 25 and April 22 by region, the primary goal isn't 't'to produce accurate forecasts. It's identify factors that appear to impact the transmission rate of COVID-19.
You are encouraged to pull in, curate and share data sources that might be helpful. If you find variables that look like they impact the transmission rate, please share your finding in notebook.
As the data becomes available, we will update the leaderboard with live results based on data made available from the Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE).
We have received support and guidance from health and policy organizations in launching these challenges. We're re hopeful the Kaggle community can make valuable contributions to developing a better understanding of factors that impact the transmission of COVID-19.
Evaluation
Submissions are evaluated using the column-wise root mean squared logarithmic error.
The RMSLE for a single column calculated
(log(p; 1) log(a; + 1))²
(I)
where:
n is the total number of observations
-P; is your prediction
is the actual value
log(z) the natural logarithm of
The final score is the mean of the RMSLE over all columns (in this case, 2).
Submission File
For each ForecastId in the test set, you'l Il predict the cumulative COVID-19 cases and fatalities to date. The file should contain header and have the following format:
Forecast Id, Confi rmedCases, Fatalities
1,10,0 0
2,10, 0
3,10, 0 etc.
47

---

Published as a conference paper at ICLR 2025
Data
In this challenge, you will be predicting the cumulative number confirmed COVID19 cases in various locations across the world, well the number of resulting fatalities, for future dates. We understand this a serious situation, and in no way want to trivialize the human impact this crisis causing by predicting fatalities. Our goal is provide better methods for estimates that assist medical and governmental institutions to prepare and adjust as pandemics unfold. File descriptions
train.csv the training data up Mar 18, 2020.
test.csv the dates to predict; there is a week of overlap with the training data for the initial Public leaderboard. Once submissions are paused, the Public leaderboard will update based on last 28 days of predicted data.
submission.csv a sample submission in the correct format; again, predictions should be cumula-
tive
Generation
Role: Assistant
Assistant: To tackle this task, we will follow these steps: Load the Data: Read the training and test datasets. Preprocess the Data: Handle missing values, convert date columns to datetime objects, and perform any necessary feature engineering.
Model Training: Train model to predict the cumulative number of confirmed COVID-19 cases and fatali- ties.
Make Predictions: Use the trained model to make predictions on the test dataset.
Save Predictions: Save the predictions the required format CSV file.
Let's start by implementing the solution in Python:
48

---

Published as a conference paper at ICLR 2025
filename; covid19_forecasttie
import pandas pd
from sklearn. wwwellinnneoiieee model import LinearRegression
from sklearn. preprccsspiii import LabelEncoder import numpy as np Load the data
test_pathh
output_path o train = h test htw.read_csvtestt.eeeesc submissicn csv
s on.cav
path)
Preprocess the data def (c]
df['Date'] tme (df[' Ddt(')ate
dfl'Year'] df df['Date'] yda.
df['Month'] df['Date'] mothh
df('Day'] = [ellDDaee].dttdde
df = df drop (columns=('
return df
train preprocess_coe (ta (train)
test = aaaaprocessddatateeees
# kncode categorical features
le LabelEncoder
trainn"Province__tee
fe.ffittrranffrm (train provinentrrovinc__sstteeee aastye (str))
train|' Craint'Country Country_Regionn Region' ]
le. tr.fit_transfommtttee Country Country_Region! asttype (str)
test ('Province_Statee
Le.transform(tess ('areesssttroooooe ('Povvince State' assype (str))
test /Country_Region'
(rrasfformttest Regiiny astype (str))
Prepare data for modeling
X_train = train.drow (columns-( i "Fatalitios"] [)
y_train_cases train (oedd
y_train_fatalities train(" X_test Lest.drop (columns= ( ('Forccastdd'))
Train the model for Confirmed Cases model_cases = LinearRegressiont)) model_cases. fit (K_train,, y_train_cases)
Train the model for Patalities model_ fatalitiess = LimearRegression// model fadel__faallties.fiis fit (X train,
Make predictions
predictions_caset modelccaoes.predict (X_test) fattaliiitsssttliiiis = model_fattllttooopprrddcc (X_test)
Prepare the submission file submi ssi
predictions_case
submi ssi on [ ['ataliiies'] predictions_fatte
Ensure no negative predictions
submissien| ConfirmedCases:
submission I' 'Confirmedaasss apply (1ambda
max (0, x))
ssionssionn Fatalities'] submission Fatalities apply (lambda x: max (0, x)
Save the submission file submissicn.to. (output_ap coutput_path, index=False) print (E"Predictions saved (output_pathh")
Please run the above script to generate the predictions and save them to the specified CSV file.
Once the script is executed, the predictions will be saved in the path Jcovid19 aooall--ggoobll-rccettiingggg week-
1.csv.
49

---

Published as a conference paper at ICLR 2025
Table 14: In this sample, show data modeling task AotoGen with GPT-4 can generate the submission file.
Introduction
Essay writing is an important method to evaluate student learning and performance. It is also time- consuming for educators to grade by hand. Automated Writing Evaluation (AWE) systems can score essays to supplement educator's s other efforts. AWEs also allow students to receive regular and timely feedback on their writing. However, due to their costs, many advancements in the field are not widely available to stu- dents and educators. Open-source solutions assess student writing needed reach every community with these important educational tools.
Previous efforts to develop open-source AWEs have been limited by small datasets that were not nationally diverse or focused on common essay formats. The first Automated Essay Scoring competition scored student-wring short-answer responses, however, this is a writing task not often used in the classroom. To improve upon earlier efforts, a more expansive dataset that includes high-quality, realistic classroom writing samples was required. Further, to broaden the impact, the dataset should include samples across economic and location populations to mitigate the potential of algorithmic bias.
In this competition, you will work with the largest open-access writing dataset aligned to current standards for student-appropriate assessments. Can you help produce open-source essay scoring algorithm that improves upon the original Automated Student Assessment Prize (ASAP) competition hosted in 2012?
Competition host Vanderbilt University is a private research university in Nashville, Tennessee. For this competition, Vanderbilt has partnered with The Learning Agency Lab, an Arizona-based independent non- profit focused on developing the science of learning-basel based tools and programs for the social good.
To ensure the results of this competition are widely available, winning solutions will be released as open source. More robust and accessible AWE options will help more students get the frequent feedback they need and provide educators with additional support, especially in underserved districts.
Evaluation
Submissions are scored based on the quadratic weighted kappa, which measures the agreement between outcomes. This metric typically varies from 0 (random agreement) to 1 (complete agreement). In the event that there is less agreement than expected by chance, the metric may below 0.
The quadratic weighted kappa is calculated as follows. First, N x N histogram matrix O is constructed, such that Oi, corresponds to the number of essay iids (actual) that received a predicted value An N-by-N matrix of weights, w, is calculated based on the difference between actual and predicted values:
j)²
Wi.j (N - 1)
(2)
An N-by-N histogram matrix of expected outcomes, E, calculated assuming that there no correlation between values.
This is caleulated the outer product between the actual histogram vector of outcomes and the predicted histogram vector, normalized such that E and have the same sum
50

---

Published as a conference paper at ICLR 2025
From these three matrices, the quadratic weighted kappa is calculated
Eis Wh.5 A = Eis Ws,3 Ei,
Submission File For each essay_i _id in the test set, you must predict the corresponding score (described on the Data page). The file should contain a header and have the following format:
essay_id, score
000d118, 3
000fe60, 3
001ab80, 4
Data
The competition dataset comprises about 24000 mudentt-ritted argumentative essays. Each essay scored on scale of to (Link to the Holistic Scoring Rubric). Your goal is to predict the score an essay received from its text.
File and Field Information
train.csv Essays and scores be used training data.
essay_id -say_id- The unique ID of the essay
full_text The full essay response
score Holistic score of the essay on 1-6 scale
data. Contains the same fields train.csv, aside from exclusion of score. (Note: The rerun test set has approximately 8k observations.)
test.csv The essays be used
sample_submission.sss A submission file in the correct format.
essay id The unique ID of the essay score The predicted holistic score of the essay on a 1-6 scale
Please note that this is Code Competition.
Generation
Role: User
To tackle the task of predicting essay scores using the provided dataset, will outline plan and then provide you with the Python code to execute. The plan involves the following steps:
Data Loading: Load the training and test datasets. 2. Data Preprocessing: Prepare the data for modeling. which might include cleaning the text data.
Feature Engineering: Convert the essay texts into numerical format that can used by machine learning models, using techniques like TF-ID.
Model Training: Train machine learning model using the training data. Given the nature the task, a regression model specialized text model like BERT could be appropriate. Prediction: Use the trained model to predict scores for the essays the test dataset. Output: Save the predictions in the specified CSV format and location.
51

---

Published as a conference paper at ICLR 2025
Here is the Python code that encapsulates these steps. You should run this code in your Python environment where the data files are located.
filename: esay__cooring_prrdccccooo
import pandas pd
import ffidfVectorizer from calcarn.ensemil import RandomporestRggresso::
from sklearn. wew.treeeeteeeeee deraatureeettactioncto
from Allcaarn.odel_solee...
import train_test_eplii
from sklearn. metrics import sean_squurred_rror import
l.ad data train_path http///mrnnnggggrraar//tmaeeerrr cav! test_path
train data pd. rea.readdee (train path) test data rearead Csv (test_ path)
Preprocess data tfidf Ifidfvectorizer (ttprrrrrssstgooood//o//ooosaaa train featuree tfidf. tfidf. wtw..rtaaaoooommraainnnnncn d f'ull"tttltettt
foatures===000)
test_ features tfidf. tww..ttrsssfcmmttt_deeeeee ffll fall_text
Model training model RandomF PandomforettRegesssttt (osttmttss00 model. fit (train train data score' ))
Prediction
predictions model. feeelpreeict (test_features)
predictions round(score) for score in predictions] Round to the nearest integer
Prepare submission file
submissicn Dataarmma ((
'sssay_id'; test_datal" test_data[cessay_dde
}) 'score': predictions
Save submission file
output. (output (aut submissicn.to..
path, index=False)
print (f"Submission file saved to (output_path)")
This script uses a RandomfForestRegresoor for prediction, which is simple yet effective model for regres- sion tasks. The TF-IDF vectorizer is used to convert text data into a format suitable for machine learning. After running this script, the predictions will be saved in the specified output directory. Make sure to check the output directory for the results.
Role: Assistant
Exit code: 124 (execution failed)
Code output:
Timecut
User: Assistant
The timeout error suggests that the process took too long execute, likely due the size of the data and the complexity of the operations (such as TF-IDF transformation and RandomFores training). To address this, we can simplify the model and reduce the computational load. Here's an updated approach: 1. Reduce Feature Space: Decrease the number of features in the TF-IDF vectorizer.
52

---

Published as a conference paper at ICLR 2025
Simplify Model: Use a simpler model that requires less computation time, such linear regression model.
Here's 's the revised Python code with these adjustments:
from sklearn. feature_extract text import uffifvecctorieer
from sklearn. linear_model import Linearkegression import cs
Load data train_path
test_path h/tp////rrngggg//////.oaaaaa////
train_data pd. ottpa//ccccaaa......oo.oaoooaaaa
test_data = pd. ctst___
Preprocess data tfidf 1fidfvectorizer (stg max nax_features-100 1000)
Reduced
+ number of features
ffidf.fit tfiid.iitttaasffomm transform(train datal' full
full _ull_textt ]]
test features tfidf. transformmttetttttt.t.tttotttttt data full text'
Model training model = LinearRegression ()
model. fit (train_featureed train_das data score'
Prediction
predictions = model.predie (test_ features)
predictiens = [rund for score in predictions] $ Round scores to the
nearest integer
Prepare submission file submi ssien pd. DataFrame ((
esssy id' test datal' lasallessay..o id' ), 'score predictions ))
Save submission file output_path = w:loaiiggggggggbbbauuttteeerarar submission tsv (output_path, index-False)
print (f"Submission file saved to (output_path/")
This version of the script uses a Linear Regression model, which is computationally less intensive than RandomForest, and reduces the number of features in the TF-IDF vectorizer to speed up the text processing. Please run this updated script and check if it completes within the allowed time.
Role: Assistant
Exit code: 0 (execution succeeded) Code output:
Submission file saved ///p////nrrggggggl/looooooeeeeer
Role: User
Great! The script executed successfully and the submission file has been saved to the specified location. You can now check the output file to ensure that it contains the predicted scores formatted correctly according to the competition's requirements.
everything looks good, you can proceed with any further steps required by the competition, such as submitting the results performing additional analysis. you need further assistance or have more tasks, feel free to ask. Otherwise, good luck with your compe- tition entry!
53


Reviews:
[{"review_id": "ffJLgUw67W", "reviewer": "Reviewer_cMgg", "summary": "This paper introduces DSBench, a comprehensive data science benchmark designed to assess the performance of data science agents on real-world tasks. DSBench integrates the challenges from ModelOff and Kaggle competitions, creating a realistic environment that covers 466 data analysis and 74 data modeling tasks. To enable fair comparison, the paper proposes the Relative Performance Gap metric, which normalizes various evaluation metrics for data modeling tasks. Evaluation of state-of-the-art models, including GPT-4o, Claude, and Gemini, reveals that DSBench remains challenging, with significant performance gaps between these models and human capabilities.", "questions": "1. Can authors please explain if the metric and dataset are sufficient to cover the real-world diversity that the benchmark aims to address?\n2. Explain why RPG is able to reflect actual performance across varying data types and task complexities.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Novelty** \n- This paper offers a fresh contribution to data science by introducing DSBench, a new benchmark that evaluates data science agents under realistic task conditions derived from ModelOff and Kaggle competitions. It pushes the boundaries of traditional benchmarking.\n\n**Quality** \n- The design of DSBench, with its comprehensive task types and Relative Performance Gap (RPG) metric, demonstrates rigor in addressing evaluation inconsistencies across various modeling tasks. \n\n**Clarity** \n- Task designs, methods, and performance comparisons are clear and well-organized. They contain many details but are not hard to follow.\n\n**Significance**\n- DSBench sets a new standard in evaluating data science agents, driving advancements in realistic, end-to-end task performance. Its contributions are useful to future advancements of intelligent, autonomous data science agents.", "weaknesses": "- I'm bit unsure of the robustness and persuasiveness of the RPG metric is valid. It could be beneficial to further assess how well the RPG reflects actual performance across varying data types and task complexities.\n- Although Kaggle tasks are highly relevant, they often focus on a narrow subset of domains (e.g., retail, finance).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R8gHoWEq7v", "reviewer": "Reviewer_2rKh", "summary": "The paper introduces a new benchmark for automated data science, DSBench.  This\nencompasses both excel exercises from ModelOff and kaggle competitions.  DSBench\nalso includes metrics for evaluating success in both kinds of tasks.  The paper\nevaluates several open and closed source LLMs/VLMs.  The key take away is that\nSOTA architectures are still far away from achieving human-level performance.", "questions": "Plase see weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality**: This is chiefly an engineering paper, it introduces no novel techniques or ideas.  DSBench collates existing resources.\n\n**Quality**: The key contribution is DSBench, which I am confident took quite some effort to set up.  All in all, this contribution seems to be of good quality: it encompasses a large number of relevant tasks which go beyond what the literature currently offers.  It also comes with some rather natural metrics.  The interesting bit is the evaluation, which on the one hand shows that DSBench can indeed be used as intended, and on the other points out a performance gap for existing architectures.  The findings are otherwise quite intuitive: newer models perform better, more complex tasks are harder to solve.  The real contribution is the research that DSBench will enable.\n\nUnfortunately, as a non-expert (my work on automating data science predates LLMs), I cannot assess the overlap between DSBench and recent works in this area.\n\n**Clarity**: The text is generally readable, with a few linguistic quirks here and there.  The figures are easy to understand.\n\n**Significance**:  Automated data science is a central topic nowadays.  It is not impossible that DSBench will provide a significant boost to auto-DS.  But again, this depends on overlap with existing work, which I am not overly familiar with.  As a result, I have decided to grade conservatively the contribution aspect of the paper.", "weaknesses": "**Quality**: The only relatively minor issue I'd like to point out concerns Section 3.3, which promises discussing the errors made by LLMs, but does not provide any details, really.  I think this should be amended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kcEzkrJcWH", "reviewer": "Reviewer_EZxL", "summary": "DSBench introduces a benchmark to test the performances of LLM or VLM agents. It contains data analysis and modeling tasks sourced from Kaggle and ModelOff competitions. Compared to other similar benchmarks, the benchmark is more realistic in that it contains data files, tables, and images and provides executable evaluation functions to validate the answers. The paper tests various models and agent systems and found them to pale in comparison to human data scientists.", "questions": "1. In Table 3, what does \"context length\" refer to? Is it the number of English words, characters, or tokens?\n2. The paper assumes that the LLMs should be able to generate the answers from the data files. Is it possible for the LLMs to know the answers to the questions from pretraining? For example, the answers to the challenge may be discussed in a Reddit forum that has been scraped in pretraining.\n3. In Figure 2(b), what are A-F, A-I, and A-D?\n4. L265: what is N in \\hat{F} = \\mathcal{G}(E,N,S,M)?\n5. Figure 3: Can you define m0, m1, ..., m17, perhaps as a separate table in the Appendix?\n6. Please define the versions of the model used in the main paper instead of relegating it in the Appendix.\n7. What are the settings for the models (e.g., temperature)? Are the experiments conducted multiple times? If so, how are they aggregated?\n8. For the analyses (e.g., L404 -- L421), it would be nice to state if they have been corroborated by prior research. For example, Qian et al. has also observed that ML agents perform better on older Kaggle challenges and hypothesize that older challenges have more data leakage into pretraining data.\n9. Can you provide a unified script to download the datasets, perhaps using an API, process the downloaded data and run the experiments? It would help for readers to verify the results.\n10. L406-L409: Figure 4 shows only GPT-4o, AutoGen and Gemini, but your text references Llama3-8B and GPT 3.5.\n11. How do I interpret the RPG of human? Doesn't 'human' represent the best human generated answer and RPG compares against the best human?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper collects a set of data science challenges from Kaggle and ModelOff. By releasing the code and data, the community can use it to measure the performances of their agent tools. The paper demonstrates through experiments that there exists huge gaps between data science agents and humans, demonstrating that the benchmark is not saturated.", "weaknesses": "In general, the paper is well written. The only concerns relate to the sustainability of the benchmark; by releasing the dataset, there is a high chance that it will be, either intentionally or unintentionally, incorporated into the training data for LLMs or VLMs. It seems like this possibility has not been considered by the authors and their explanation for the correlation between accuracy and year of release (Table 5, L416-L421) is weak. Some critical information is missing and it affects my judgement. I would be happy to change my score if the authors can clear my potential misunderstandings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WjABvATHuU", "reviewer": "Reviewer_TVnT", "summary": "This paper introduces DSBench, a comprehensive data science benchmark containing 466 data analysis tasks and 74 data modeling tasks sourced from ModelOff and Kaggle competitions. Compared to existing benchmarks, DSBench provides a more realistic evaluation environment, encompassing long-context understanding, multimodal task backgrounds, large data file processing, and multi-table structure reasoning. Through evaluation of state-of-the-art LLMs, LVLMs, and agents, the study finds that they still struggle with most tasks, with the best agent achieving only 34.12% accuracy in data analysis tasks and a 34.74% Relative Performance Gap (RPG) in data modeling tasks.", "questions": "In Table 1's comparison section, I suggest reversing the criteria for \"Exec. Eval.\", \"Code only\", and \"Env. Fix.\" to negative statements. This way, checkmarks would indicate DSBench's unique advantages where other benchmarks fall short, making it easier for readers to quickly grasp DSBench's contributions. This would better highlight how DSBench addresses limitations in existing benchmarks and make the comparison more intuitive.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "### 1. Originality & Vision\n- Pioneering creation of a comprehensive real-world data science benchmark, analogous to SWE-bench in software engineering, marking a significant step toward AGI evaluation\n- First benchmark to integrate both data analysis and modeling tasks in realistic settings with complex data structures and multimodal contexts\n- Novel introduction of RPG (Relative Performance Gap) metric that effectively normalizes performance across diverse modeling tasks\n- Innovative approach to testing both pure language understanding and tool utilization capabilities\n- Make consideration of not only the performance but also the cost\n\n### 2. Technical Robustness\n- Rigorous task collection methodology, carefully curated from established platforms (ModelOff and Kaggle) ensuring real-world applicability\n- Comprehensive coverage of data science tasks: 466 analysis tasks + 74 modeling tasks, spanning different complexity levels and domains\n- Sophisticated evaluation framework that considers:\n  * Multi-table reasoning capabilities\n  * Long-context understanding\n  * End-to-end solution generation\n  * Tool integration and utilization\n  * Multiple modalities processing\n\n### 3. Practical Significance\n- Direct application to real-world data science scenarios, bridging the gap between academic benchmarks and practical challenges\n- Clear identification of current AI systems' limitations in data science tasks:\n  * Understanding complex data relationships\n  * Handling multi-step reasoning\n  * Managing tool interactions\n- Provides valuable insights for developing more capable data science agents by revealing specific areas where current models fall short\n- Sets a new standard for evaluating AI systems' practical data science capabilities, essential for progressing toward AGI\n\n### 4. Research Impact\n- Creates a foundation for systematic evaluation of data science capabilities in AI systems\n- Enables quantitative comparison of different AI approaches in real-world data science scenarios\n- Provides a roadmap for developing more capable AI systems that can handle complex, real-world data science tasks\n- Serves as a crucial benchmark for measuring progress toward AGI in the domain of data science", "weaknesses": "### 1. Statistical Rigor in Dataset Scale\n- While 540 samples is reasonable given the scarcity of high-quality ModelOff & Kaggle competitions, the paper could benefit from:\n  * Reporting confidence intervals through multiple experimental runs\n  * Conducting bootstrap analysis to estimate the robustness of performance metrics\n  * Providing power analysis to justify the sample size\n- The paper could discuss how the current sample size was determined and what would be an ideal size for future extensions\n\n### 2. Evaluation Methodology\n- Human baseline performance (22 competitions) could be strengthened by:\n  * Including more expert evaluators per task\n  * Reporting inter-rater agreement scores\n  * Documenting the selection criteria for human experts\n- The evaluation process could benefit from:\n  * Explicit discussion of potential biases in task selection\n  * Analysis of task difficulty distribution\n  * More detailed failure case studies with expert annotations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pZ0254Odx7", "reviewer": "Reviewer_XL8J", "summary": "The paper constructs DSBench, a benchmark focused on evaluating LM-based data science agents through realistic tasks sourced from ModelOff and Kaggle competitions.", "questions": "1. As highlighted in Table 1, What is the purpose of distinguishing tables from data files? What is the difference between tables and data files? I would like the authors to clarify how they treat these two types of data samples during evaluation.\n\n2. I also find the taxonomy somewhat difficult to understand. For example, \"tables\" and \"excels\" are categorized separately, and I hope the author can clarify the distinctions between these categories more clearly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I believe the dataset provided by the authors can, to some extent, reflect a model's ability to tackle data science tasks, and even today, it remains quite challenging, serving as a non-trivial evaluation benchmark.\n2. The authors conducted some interesting analyses across various dimensions for different models and different data science tasks (e.g., examining task completion rates relative to task release time, the correlation between task difficulty and context length, etc.), offering potential insights into the development of LM as agents.", "weaknesses": "1. **Data Source:**\n- All the data analysis tasks mentioned by the authors at line 106 are related to finance. I would appreciate it if the authors could clarify this point in the paper and explain why other types of analysis tasks are not as suitable. This concern arises because ModelOff is actually a global financial modeling competition.\n- Similarly, for data modeling, it seems this is also influenced by the fact that there are numerous modeling task competitions on Kaggle.\n\nI wonder if the authors have explored more platforms or data sources and could explain why they were not suitable for evaluating data science agents?\nAdditionally, in my opinion, competitions are not always the closest representation of the real world. For example, as far as I know, Spider2-V[1] incorporates a lot of tools and software from industrial data pipelines. Could this be a more realistic measure of real-world scenarios?\n\n2. **Evaluation Metrics:**\n- I fully understand that collecting and building complex evaluation environments is a considerable engineering task. However, if the evaluation is based solely on competition platforms and existing output-only metrics, it seems that it may not fully capture the comprehensive capabilities of data science agents. This is similar to what the authors mentioned in lines 153 and 154, such as extracting insights, proper data handling, etc.\n\n3. **Need better presentation, especially for some tables and figures:**\n- I noticed in Figure 4, some of the models do not have the reported accuracy (does this mean zero accuracy?); while the width of the bars are not set as the same. However, I do not find clear explanation to these.\n- I believe transforming Table 5 into a line chart with corresponding accuracy values would clearly illustrate the trend in accuracy over time.\n\n[1] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?, NeurIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)