BigCodeBench: Benchmarking Code Generation with Diverse Function Calls
  and Complex Instructions
Authors: name: Terry Yue Zhuo, name: Minh Chien Vu, name: Jenny Chim, name: Han Hu, name: Wenhao Yu, name: Ratnadira Widyasari, name: Imam Nur Bani Yusuf, name: Haolan Zhan, name: Junda He, name: Indraneil Paul, name: Simon Brunner, name: Chen Gong, name: Thong Hoang, name: Armel Randy Zebaze, name: Xiaoheng Hong, name: Wen-Ding Li, name: Jean Kaddour, name: Ming Xu, name: Zhihan Zhang, name: Prateek Yadav, name: Naman Jain, name: Alex Gu, name: Zhoujun Cheng, name: Jiawei Liu, name: Qian Liu, name: Zijian Wang, name: David Lo, name: Binyuan Hui, name: Niklas Muennighoff, name: Daniel Fried, name: Xiaoning Du, name: Harm de Vries, name: Leandro Von Werra
Organization: BigCode

Automated software engineering has been greatly empowered by the recent
advances in Large Language Models (LLMs) for programming. While current
benchmarks have shown that LLMs can perform various software engineering tasks
like human developers, the majority of their evaluations are limited to short
and self-contained algorithmic tasks. Solving challenging and practical
programming tasks requires the capability of utilizing diverse function calls
as tools to efficiently implement functionalities like data analysis and web
development. In addition, using multiple tools to solve a task needs
compositional reasoning by accurately understanding complex instructions.
Fulfilling both of these characteristics can pose a great challenge for LLMs.
To assess how well LLMs can solve challenging and practical programming tasks,
we introduce Bench, a benchmark that challenges LLMs to invoke multiple
function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained
programming tasks. To evaluate LLMs rigorously, each programming task
encompasses 5.6 test cases with an average branch coverage of 99%. In addition,
we propose a natural-language-oriented variant of Bench, Benchi, that
automatically transforms the original docstrings into short instructions only
with essential information. Our extensive evaluation of 60 LLMs shows that LLMs
are not yet capable of following complex instructions to use function calls
precisely, with scores up to 60%, significantly lower than the human
performance of 97%. The results underscore the need for further advancements in
this area.