ML-Bench: Large Language Models Leverage Open-source Libraries for
  Machine Learning Tasks
Authors: name: Yuliang Liu; name: Xiangru Tang; name: Zefan Cai; name: Junjie Lu; name: Yichi Zhang; name: Yanjun Shao; name: Zexuan Deng; name: Helan Hu; name: Zengxian Yang; name: Kaikai An; name: Ruijun Huang; name: Shuzheng Si; name: Sheng Chen; name: Haozhe Zhao; name: Zhengliang Li; name: Liang Chen; name: Yiming Zong; name: Yan Wang; name: Tianyu Liu; name: Zhiwei Jiang; name: Baobao Chang; name: Yujia Qin; name: Wangchunshu Zhou; name: Yilun Zhao; name: Arman Cohan; name: Mark Gerstein
Published: 2023-11-16 12:03:21

Abstract: Large language models have shown promising performance in code generation
benchmarks. However, a considerable divide exists between these benchmark
achievements and their practical applicability, primarily attributed to
real-world programming's reliance on pre-existing libraries. Instead of
evaluating LLMs to code from scratch, this work aims to propose a new
evaluation setup where LLMs use open-source libraries to finish machine
learning tasks. Therefore, we propose ML-Bench, an expansive benchmark
developed to assess the effectiveness of LLMs in leveraging existing functions
in open-source libraries. Consisting of 10044 samples spanning 130 tasks over
14 notable machine learning GitHub repositories. In this setting, given a
specific machine learning task instruction and the accompanying README in a
codebase, an LLM is tasked to generate code to accomplish the task. This
necessitates the comprehension of long and language-code interleaved documents,
as well as the understanding of complex cross-file code structures, introducing
new challenges. Notably, while GPT-4 exhibits remarkable improvement over other
LLMs, it manages to accomplish only 39.73\% of the tasks, leaving a huge space
for improvement. We address these challenges by proposing ML-Agent, designed to
effectively navigate the codebase, locate documentation, retrieve code, and
generate executable code. Empirical results demonstrate that ML-Agent, built
upon GPT-4, results in further improvements. Code, data, and models are
available at https://ml-bench.github.io/.