ML-Bench: Large Language Models Leverage Open-source Libraries for
  Machine Learning Tasks
Authors: name: Yuliang Liu, name: Xiangru Tang, name: Zefan Cai, name: Junjie Lu, name: Yichi Zhang, name: Yanjun Shao, name: Zexuan Deng, name: Helan Hu, name: Zengxian Yang, name: Kaikai An, name: Ruijun Huang, name: Shuzheng Si, name: Sheng Chen, name: Haozhe Zhao, name: Zhengliang Li, name: Liang Chen, name: Yiming Zong, name: Yan Wang, name: Tianyu Liu, name: Zhiwei Jiang, name: Baobao Chang, name: Yujia Qin, name: Wangchunshu Zhou, name: Yilun Zhao, name: Arman Cohan, name: Mark Gerstein

Large language models have shown promising performance in code generation
benchmarks. However, a considerable divide exists between these benchmark
achievements and their practical applicability, primarily attributed to
real-world programming's reliance on pre-existing libraries. Instead of
evaluating LLMs to code from scratch, this work aims to propose a new
evaluation setup where LLMs use open-source libraries to finish machine
learning tasks. Therefore, we propose ML-Bench, an expansive benchmark
developed to assess the effectiveness of LLMs in leveraging existing functions
in open-source libraries. Consisting of 10044 samples spanning 130 tasks over
14 notable machine learning GitHub repositories. In this setting, given a
specific machine learning task instruction and the accompanying README in a
codebase, an LLM is tasked to generate code to accomplish the task. This
necessitates the comprehension of long and language-code interleaved documents,
as well as the understanding of complex cross-file code structures, introducing
new challenges. Notably, while GPT-4 exhibits remarkable improvement over other
LLMs, it manages to accomplish only 39.73\% of the tasks, leaving a huge space
for improvement. We address these challenges by proposing ML-Agent, designed to
effectively navigate the codebase, locate documentation, retrieve code, and
generate executable code. Empirical results demonstrate that ML-Agent, built
upon GPT-4, results in further improvements. Code, data, and models are
available at https://ml-bench.github.io/.