DA-Code: Agent Data Science Code Generation Benchmark for Large Language
  Models
Authors: name: Yiming Huang; name: Jianwen Luo; name: Yan Yu; name: Yitong Zhang; name: Fangyu Lei; name: Yifan Wei; name: Shizhu He; name: Lifu Huang; name: Xiao Liu; name: Jun Zhao; name: Kang Liu
Published: 2024-10-09 18:00:05

Abstract: We introduce DA-Code, a code generation benchmark specifically designed to
assess LLMs on agent-based data science tasks. This benchmark features three
core elements: First, the tasks within DA-Code are inherently challenging,
setting them apart from traditional code generation tasks and demanding
advanced coding skills in grounding and planning. Second, examples in DA-Code
are all based on real and diverse data, covering a wide range of complex data
wrangling and analytics tasks. Third, to solve the tasks, the models must
utilize complex data science programming languages, to perform intricate data
processing and derive the answers. We set up the benchmark in a controllable
and executable environment that aligns with real-world data analysis scenarios
and is scalable. The annotators meticulously design the evaluation suite to
ensure the accuracy and robustness of the evaluation. We develop the DA-Agent
baseline. Experiments show that although the baseline performs better than
other existing frameworks, using the current best LLMs achieves only 30.5%
accuracy, leaving ample room for improvement. We release our benchmark at
https://da-code-bench.github.io.