BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

## Abstract

Abstract Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls.
Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development.
In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions . Fulfilling both of these characteristics can pose a great challenge for LLMs.
To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench , a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks.
To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%.
In addition, we propose a natural-language-oriented variant of BigCodeBench , BigCodeBench - Instruct , that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97% . The results underscore the need for further advancements in this area.

### 1 Introduction

Task automation, including competitive programming (Li et al., ; Hendrycks et al., ; Jain et al., ), GitHub issue resolution (), and question answering (Gao et al., ; ), has attracted significant interest from academia and industry to facilitate the development of advanced models for code, especially in Python (). With recent advances in data-centric deep learning techniques, large language models (LLMs) trained on large-scale corpora have shown superior capabilities of translating textual inputs to syntactically correct and functional programs.
However, widely-used benchmarks like HumanEval (Chen et al., ) and MBPP (Austin et al., ) only contain short, self-contained, and algorithm-focused programming tasks and have been saturated by recent model releases.
In this work, we aim to close the evaluation gap between these isolated coding exercises and real-world programming, asking *Can LLMs solve more challenging and more practical tasks via programs?*

Solving programming tasks in real-world scenarios typically has two main characteristics: (1) *Diverse Function Calls as Tools*.(^1^11In this work, we refer to “tools” as library-oriented but non-object-oriented Code Function Call APIs, which are discussed in [Gorilla OpenFunctions](https://gorilla.cs.berkeley.edu/blogs/4_open_functions.html) The Code Function Calling APIs are typically seen in common external Python packages like Numpy, Sklearn, and Pandas.)
A complex programming task often requires the invocation of diverse function call sequences as tools (Robillard & DeLine, ; Hu et al., ; Qin et al., ).
To avoid reinventing the wheel, domain-specific libraries are designed to cover function calls (or APIs) with comprehensive functionalities; and
(2) *Complex Instructions*. Due to the complexity to perform various programming tasks, instructions may require compositional reasoning ability to perform a sequence of functionalities in the correct order (e.g., input data manipulation, error message handling, and specific output formatting) (Wiegers & Beatty, ; Paetsch et al., ; Partsch, ). For instance, creating a network application that includes functionality for retrieving responses from an HTTPS server (Figure 1) requires integrating several components with multiple function calls, such as managing SSL contexts, handling socket connections, and ensuring the response is returned in the specific format.

Building a high-quality execution-based benchmark and the environment that simulates aforementioned tasks with practical and challenging constraints is non-trivial.
First, it is hard to naturally source self-contained programming tasks with complex instructions, unlike the short code exercises in HumanEval.
Although most GitHub repositories contain realistic source code, the functions inside these repositories often require cross-file information (Ding et al., ).
Second, real-world programming scenarios are extremely diverse (Zhout et al., ). Existing benchmarks (Zan et al., ; Lai et al., ) only focus on popular scenarios like data science.
Third, mainstream programming benchmarks have a significant number of ambiguous or under-specified specifications, resulting in inaccurate evaluation results (Siddiq et al., ; ). While there have been attempts to improve the data quality with LLMs (Jain et al., ), LLMs have their own bias and cannot reliably perform refinement (Zheng et al., ).

To construct massive high-quality programming tasks, we propose a novel framework (Figure 2) that uses collaboration between LLMs and human experts to build a rigorous execution-based benchmark, BigCodeBench. Particularly, we utilize LLMs to source programming tasks, refactor programs, and add test cases, under constant human supervision.
Our benchmark contains 1,140 rich-context and multi-tool-use programming tasks in Python, covering 723 function calls from 139 popular libraries across 7 domains.
As we aim to let models reason any suitable function calls to complete the tasks via code, we design the unit test cases in an open-ended manner and examine certain behaviors based on different inputs (). We assess two common programming scenarios: (1) Code Completion (BigCodeBench-Complete), which evaluates the capability of code generation based on the structured docstrings; and (2) Instruction to Code (BigCodeBench-Instruct), which evaluates the ability to complete programming tasks based on natural-language-oriented (NL-oriented) instructions. While BigCodeBench-Complete emphasizes structured docstring prompts, BigCodeBench-Instruct challenges LLMs to generate precise code without relying on non-essential details like [interactive Python examples](https://docs.python.org/3/library/doctest.html).

Through extensive studies on 60 models, we find that LLMs still struggle to invoke multiple function calls from cross-domain libraries and follow complex instructions to solve programming tasks using Python. Specifically, the best performing LLM, GPT-4o, solves merely 60% of tasks on BigCodeBench-Complete and less than 50% on BigCodeBench-Instruct, indicating that LLMs themselves lack the ability to align with human expectations when instructions are more natural. Interestingly, we find that some instruction-tuned LLMs like GPT-4 constantly refuse to follow long instructions to repeat essential context and thus fail the test cases.
Furthermore, LLMs perform differently when using domain-specific function calls as tools.
We also demonstrate the strongly positive correlations between mainstream benchmarks and BigCodeBench, validating our evaluation results.

### 2 Benchmark Construction

In this section, we introduce our human-LLM-collaboration framework for the BigCodeBench construction. As shown in Figure 2, the process of BigCodeBench-Complete encompasses three stages: Data Synthesis, Semi-automatic Program Refactoring and Testing Case Generation, and Human Curation. In addition, we construct BigCodeBench-Instruct in Section 2.4, an NL-oriented benchmark for programming-task solving. Further information is available in Appendix B and Appendix J. The construction is progressively contributed by 20 authors as annotators for one year in total, with 75% of them having more than 5 years of experience in Python programming. The progress is continually managed by the lead annotator to control the data quality. Each constructed programming task is expected to: (1) have clear instructions inside [PEP-257](https://peps.python.org/pep-0257/)-structured docstrings; (2) use multiple libraries as tools to solve the problem; and (3) be paired with at least five test cases encapsulated in the [unittest](https://docs.python.org/3/library/unittest.html) framework, with a complex test setup (e.g., database connection, and directory creation), to verify program behaviors instead of relying on simple input-output assertions.

Figure: Figure 2: Each programming task in BigCodeBench is created through a three-stage construction process. The task quality is controlled by the human-LLM collaboration.
Refer to caption: x2.png

#### 2.1 Data Synthesis

One intuitive method would be to rely on source code repositories to construct the function-level programming tasks. However, most repository-level functions require cross-file information, such as customized modules (Ding et al., ), which are hard to self-contain and document. We argue that leveraging LLMs to synthesize customized programming tasks can be more viable (Wei et al., ; Luo et al., ), especially when overseen by humans.
Given a code snippet of API usage with a brief human instruction as the seed example (Figure 2), an LLM is instructed to enrich the programming intent and refine the corresponding implementation by using diverse libraries. Specifically, we use seed examples from ODEX (Wang et al., ), a benchmark containing intent-paired code skeletons from Stack Overflow. We use the GPT-4 API(^2^22We use the model version gpt-4-0613.), the strongest LLM at the time of data sourcing, to synthesize the programming tasks. To help the LLM synthesize self-contained and relevant programming tasks based on the seed example, we instruct the model with a 2-shot in-context demonstration (Appendix J.1) crafted by the lead annotator.

As previous studies have shown that LLMs favor their own generations (Zheng et al., ; Panickssery et al., ), such phenomena can make the model evaluation unfair.
We mitigate the model biases with an obfuscation and perturbation process. We first replace the semantic-rich program entry points with dummy function names. In addition, we perturb the natural language descriptions in docstrings with the back-translation method of NL-Augmenter (Dhole et al., ), inspired by ReCode (Wang et al., ). After validating the post-processed programs with an abstract syntax tree parser, we collected 4,718 function-level programming samples.

#### 2.2 Semi-automatic Program Refactoring and Testing Case Generation

Programs synthesized by LLMs may contain various issues, such as undeclared variables and runtime bugs. Without proper verification, the implementation cannot directly serve as a ground-truth solution. To construct a high-quality execution-based benchmark, we need to add test cases that can rigorously verify the correctness of programs and identify any bugs.
However, it takes non-trivial effort for human developers to understand synthesized programs and properly refactor them with thorough testing. To improve the program quality and ease human annotation, we propose a conversion-driven framework for program refactoring and test case generations inspired by (Xia & Zhang, ). Specifically, we utilize the [Code Interpreter](https://openai.com/index/chatgpt-plugins/) session in the web-based GPT-4, which is back-ended by a Linux-based virtual environment with pre-installed Python packages. We engage 13 authors as human annotators (including the lead annotator) for this step, assigning each annotator 100 randomly sampled programming tasks based on their preferences of code topics (Appendix J.2.1). We detail the design of this human-in-the-loop framework from human and LLM aspects as follows:

###### Human Aspect

Human developers possess varying preferences and levels of familiarity with specific data types and programming scenarios. To aid human annotators in providing more precise feedback for refining programs with GPT-4, we have defined 10 data types (e.g., SQL, CSV, and Python built-in types) and task scenarios (e.g., data analysis, networking, and visualization). GPT-4 API is utilized to automatically classify each program according to these categories, with detailed descriptions available in Appendix J.2.1. The annotators’ role is to continually instruct GPT-4 to refractor the programs and to provide continuous feedback to guide the model whenever it fails to self-debug or incorrectly refactor the program.

###### LLM Aspect

To effectively guide GPT-4 in the iterative refinement of programs and test cases, we provide detailed annotation guidelines in Appendix J.2.2 as an initial prompt. These guidelines encompass two high-level instructions: (1) Refine the function, including its docstrings, to enhance realism and reduce ambiguity; and (2) Write unit tests to ensure the functional correctness of the given program description. Specifically, the model is taught to follow a step-by-step refinement process: (1) Remove unused libraries and add necessary libraries if they are missing in the code snippet; (2) Reformat docstrings to adhere to [PEP-257](https://peps.python.org/pep-0257/) conventions; (3) Align program implementations with the instructions inside docstrings; (4) Write and execute unit tests to ensure they pass; and (5) Add refined programs and test cases to files for downloading.

During interactions with GPT-4, we identify two main drawbacks in the Code Interpreter session. First, GPT-4 struggles to write proper test cases when mocking tests are employed. While the model can generate high-level designs of mocking tests, it often fails to understand how test cases should be constructed based on execution feedback. Second, GPT-4 can become stuck while resolving runtime bugs, leading to iterative refinement until the session times out. Continuous human feedback on viable solutions is essential to address these issues and ensure the model stays on track.

During the post-processing of the annotated data, we observe that a significant number of test cases were incomplete. This issue arises because GPT-4 often omits partial content when writing long contexts into files. After removing all invalid programs via program analysis, we end up with 1,223 refactored programming tasks with paired test cases.

#### 2.3 Human Curation

To enhance the benchmark quality, we implement a three-fold human curation process:

###### Examination

We first perform a rigorous manual examination of the benchmarks, guided by a set of detailed guidelines (Appendix J.3) to add more test cases and resolve a list of runtime issues due to the LLM-generated flaky tests (Luo et al., ). In addition, we aim to formalize further the programming tasks based on the following criteria: (1) The task should utilize at least two libraries to enrich the task scopes; (2) Import statements should be limited to those used in the task implementations and be natural based on human understanding; (3) The docstring should follow PEP-257 to ensure consistent styles; (4) Both the task implementation and test cases should be clearly aligned with the task docstrings, based on human understanding; and (5) Required library modules in the docstrings should be aligned with the import statements.
For test case construction, the criteria are as follows: (1) Test cases should be encapsulated by the [unittest](https://docs.python.org/3/library/unittest.html) framework, making them [pytest](https://docs.pytest.org/en/stable/)-compatible; and (2) Test cases need to be deterministic, asserting expected program behaviors based on the inputs ().

###### Pre-Evaluation

To enhance the benchmark quality, we perform a dry run to pre-evaluate an LLM other than GPT-4 used in previous steps. We choose GPT-3.5-Turbo API(^3^33We use model version gpt-3.5-turbo-1106.) to generate solutions based on the examined task prompts. By understanding how the model fails on the tasks, annotators may add essential details to the docstrings to clarify the task instructions but avoid describing step-by-step implementations.

###### Cross-Checking

To further validate data quality and ensure consistency across all programming tasks in the testing environment, 7 additional annotators refine and finalize the data annotated by others. This round focuses on the utility of docstrings and test cases. We automatically parse the docstrings and ask annotators to manually correct the docstring structures, specifically addressing task descriptions, function parameters, expected returns, exception handling, required modules, and interactive examples. Additionally, we remove unused imported modules via program analysis. For the interactive examples, we ensure their correctness via [doctest](https://docs.python.org/3/library/doctest.html), except for those requiring system and network access.
The confirmed programming tasks are finally validated by the automated test workflows in [GitHub Container Registry](https://github.com/features/packages), where the test cases are automatically run against the task implementations in a configured sandbox environment. To ensure the finalized data quality, we randomly assign 33 finalized task prompts to the 11 annotators (the lead annotator was excluded; one annotator was unavailable at the time), to write the solutions. The lead annotator conducts the evaluation of the solutions, finding that 97% (32 out of 33) of sampled tasks can pass all test cases.

Figure: Figure 3: Each task prompt in BigCodeBench-Instruct are transformed from BigCodeBench-Complete via the pre-defined rules. We omit non-essential details during the transformation.
Refer to caption: x3.png

#### 2.4 Benchmarking NL-oriented Instructions to Code Generation

When instruction tuning LLMs using NL data, the input is mainly an NL instruction, and the target output is the NL or source code for completion (Muennighoff et al., ). This training objective aligns with the downstream applications such as multi-turn dialogue, where users ask questions or provide task-specific instructions to the models. While existing programming benchmarks commonly format the verbose instructions in docstrings (Chen et al., ), users may instruct the models to generate code samples with the less verbose NL context. Despite that there have been similar attempts like HumanEvalPack (Muennighoff et al., ) addressing this limitation, their inputs still lack some naturalness. Generally, users tend to describe the high-level idea of functionalities and avoid redundant information (e.g., parameters) or too-specific details (e.g., interactive examples). Thus, we create BigCodeBench-Instruct, a benchmark variant that prompts LLMs to solve programming tasks with more NL-oriented instructions, assessing the model’s ability to understand human requirements correctly.
Based on the task prompts created in BigCodeBench-Complete, we design a set of parsing rules and transform them into more natural instructions (Figure 3). For quality control, 5 authors who do not participate in previous stages inspect the randomly sampled prompts and their corresponding ground-truth solutions and reach an agreement on the alignment between the instruction prompts and task implementations.

### 3 Benchmark Statistics

Figure: Figure 4: Examples of tools in BigCodeBench are illustrated. Each function call belongs to a domain-specific library. The distribution of each domain is computed based on the frequency of domain-specific libraries appearing per task. For example, “63%” in “Computation” means that there are 63% tasks in BigCodeBench using at least one computation library.
Refer to caption: x4.png

**Table 1: Summarized statistics of representative function-level Python programming benchmarks , partially inspired by (Zan et al., ). BigCodeBench are more complex regarding the depth of task complexity and breadth of tool-use knowledge. PL.: Programming Language. Cov.: Branch Coverage. Char.: Characters. C.C.: Cyclomatic Complexity (starting from 0, higher values indicate more complex code). Lib.: Library. Dom.: Domain. Std.: [Python Standard Library](https://docs.python.org/3/library/index.html). Ext.: [Python External Library](https://pypi.org/).**
| Overall Statistics |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Benchmark | # Task | Test (Avg.) | Prompt (Avg.) | Solution (Avg.) |  |  |  |  |
| # | Cov. | Char. (Code) | Line | Char. | Line | C.C. |  |  |
| HumanEval | 164 | 7.8 | 98% | 450.6 (450.6) | 13.7 | 180.9 | 6.8 | 3.6 |
| DS-1000 | 1,000 | 1.6 | 98% | 871.8 (193.9) | 29.1 | 138.1 | 5.1 | 1.6 |
| DS-1000 (Orig.) | 452 | 1.5 | 98% | 831.4 (201.2) | 26.2 | 115.5 | 4.2 | 1.4 |
| ODEX | 945 | 1.8 | 96% | 87.5 (87.5) | 1.0 | 50.4 | 1.9 | 1.4 |
| BigCodeBench-Complete | 1,140 | 5.6 | 99% | 1112.5 (1112.5) | 33.5 | 426.0 | 10.0 | 3.1 |
| BigCodeBench-Instruct | 663.2 (124.0) | 11.7 |  |  |  |  |  |  |
| Tool Statistics |  |  |  |  |  |  |  |  |
| Benchmark | # Dom. | # Lib. | # Call | Tasks (Avg.) | Combination |  |  |  |
| Std. / Ext. | Std. / Ext. | # Lib. | # Call | # Lib. | # Calls | # Dom. |  |  |
| HumanEval (Chen et al., ) | 3 | 4 / 0 | 7 / 0 | 0.1 | 0.1 | 6 | 8 | 5 |
| DS-1000 (Lai et al., ) | 4 | 5 / 9 | 7 / 321 | 0.8 | 1.1 | 66 | 331 | 24 |
| DS-1000 (Orig.) (Lai et al., ) | 4 | 4 / 9 | 5 / 289 | 0.9 | 1.3 | 59 | 260 | 23 |
| ODEX (Wang et al., ) | 7 | 40 / 26 | 128 / 102 | 0.6 | 0.5 | 105 | 202 | 20 |
| BigCodeBench | 7 | 77 / 62 | 281 / 442 | 2.8 | 4.7 | 577 | 1045 | 56 |

###### Overall Statistics

The first part of Table 1 presents an overview comparison between BigCodeBench and other mainstream function-level Python programming benchmarks. We note that the full DS-1000 dataset emphasizes the perturbed tasks to avoid model memorization and thus includes an original subset for reference.
Therefore, we also include statistics for non-perturbed problems (DS-1000 Orig.). As the provided statistics suggest, BigCodeBench contains much more rigorous execution-based evaluation and has longer task prompts that contain complex instructions. The ground-truth solutions are also longer than in prior benchmarks, indicating that the tasks inside BigCodeBench require more complex implementation logic. To illustrate the programming complexity, we measure cyclomatic complexity, which is the number of independent paths through the task solution. We notice that BigCodeBench has a similar complexity to HumanEval, much higher than DS-1000 and ODEX. The high cyclomatic complexity indicates that solving the tasks in BigCodeBench requires non-trivial reasoning ability from the programming perspective.

Figure: Figure 5: Complexity - tool comparisons with various benchmarks.
Refer to caption: x5.png

###### Tool Statistics

In this work, we highlight any libraries and function calls from these libraries as program-based tools for LLMs (Qin et al., ). Therefore, we compare the tools used among the mainstream benchmarks in the second part of Table 1. First, BigCodeBench covers 281 function calls from 77 standard libraries and 442 from 62 external libraries, which is far more diverse than the other benchmarks. The examples of these tools are shown in Figure 4.
Based on the [PyPI](https://pypi.org/) download statistics, the mean download counts of the recent 30 days are 56.9M+ for the 62 external libraries, suggesting their high popularity in real-world software development.
Second, BigCodeBench frequently invokes a sequence of function calls from multiple libraries to solve a single task, requiring significant compositional reasoning ability for task-solving. Third, BigCodeBench has more diverse combinations among libraries, function calls, and domains in the ground-truth solutions.
We visualize the library and function call density used by different benchmarks in Appendix I, showing the even broader tool-use diversity in BigCodeBench.
To better understand solution complexity (measured by characters) and tool-use diversity (measured by distinct function calls), we compare the tasks in BigCodeBench with those in representative benchmarks in Figure 5 (details are provided in Appendix I.2). We find that BigCodeBench requires more complex reasoning and problem-solving skills to implement comprehensive functionalities.

### 4 Evaluation

Our evaluation uses the unbiased version of Pass@$K$ (Chen et al., ) to accurately assess the functional correctness of generated code snippets by LLMs. To make general observations, we extensively evaluate 60 state-of-the-art LLMs on BigCodeBench-Complete and 35 instruction-tuned LLMs on BigCodeBench-Instruct. Specifically, following prior works (Roziere et al., ; Liu et al., ; Lai et al., ), we report Pass@1 with greedy decoding for the main experiments in the zero-shot setting. To investigate more thoroughly, we compute Pass@1 and Pass@5 results with random sampling to generate $N$ ($N$=5) samples with a temperature of 0.8 and top-p of 0.95 in Appendix L. While it is encouraged to generate much more ($N\geq K$) samples to avoid bias, we take the lower bound due to limited computational resources. We use the same prompts for code generation from (Liu et al., ), given in Appendix K.

#### 4.1 Measuring Task-Level Performance

We first evaluate the task-solving performance of each LLM and summarize the findings as follows. We show the main results in Figure 6. As models constantly omit the essential code in the generation and hence fail the tests, we calibrate the generation quality by adding the missing setup. The Pearson’s $r$ correlation between the model ranks on BigCodeBench-Complete and BigCodeBench-Instruct is 0.982, indicating a strong alignment. In addition, model rankings show the signs of scaling laws (Kaplan et al., ), where bigger models can solve more tasks. We also observe that there is still some performance gap between the top closed models and open ones. Detailed studies can be found in Appendix M. We highlight a few findings as follows:

Figure: Figure 6: Pass@1 results of instruction-tuned LLMs on BigCodeBench-Complete (Top) and BigCodeBench-Instruct (Bottom). We only highlight the calibrated results having at least a difference of 1% from the original Pass@1.
Refer to caption: x6.png

###### Instruction-tuned LLMs omit essential details of long code prompts

Interestingly, we observe that instruction-tuned LLMs can omit the essential import statements of the given prompts in BigCodeBench-Complete, which can lead to task failure due to the lack of proper module and constant definitions. The omission is likely to happen when models need to repeat the long context in the response. Such behaviors are denoted as “model laziness” in long-context interactions, similar to the observations in Section 2.2. Due to the limited prompt length of existing programming benchmarks (Table 1), there is no quantitative evidence of the laziness phenomenon in prior code generation benchmarks. To understand how laziness can affect the model performance, we calibrate the generation quality by adding the missing setup (e.g., import statements and global constants). When comparing the calibrated Pass@1 and the original ones in the top figure of Figure 6 and Appendix L, we find that GPT-4 tends to omit much more context and perform poorly on BigCodeBench-Complete, consistent with the previous community discussion(^4^44[https://community.openai.com/t/why-i-think-gpt-is-now-lazy](https://community.openai.com/t/why-i-think-gpt-is-now-lazy)) and confirmed by OpenAI (OpenAI, ). While instruction-tuned LLMs have an average performance degradation of 0.8% on BigCodeBench-Complete, there is a less than 0.3% difference on BigCodeBench-Instruct, which validates the hypothesis that models omit more information of longer inputs.

###### Instruction tuning helps to follow programming constraints

When comparing instruction-tuned LLMs with their base ones (Appendix L), we observe that instruction tuning can improve the capability of following complex constraints of the prompts. The mean calibrated Pass@1 on BigCodeBench-Complete for instruction-tuned LLMs and paired base LLMs are 40.7% and 35.7%, respectively, indicating the great performance gap. The performance disparity is further magnified when the instruction-tuned LLMs are calibrated. We also note that the task performance of CodeQwen1.5-7B is not enhanced by instruction tuning, possibly due to the lack of fine-grained data during training.

###### LLMs are sensitive to the verbosity of programming instructions

From the bottom figure of Figure 6, we notice that LLMs perform much worse on BigCodeBench-Instruct than BigCodeBench-Complete with an average decrease of 8.5% on Pass@1, while maintaining similar rankings. This observation indicates that LLMs still lack the proper understanding of condensed human requirements since the task instructions of BigCodeBench-Instruct are less verbose. While it is possible that the lower verbosity may introduce more ambiguity, we make sure that the instructions transformed from BigCodeBench-Complete do not lose the key information from the human perspective. In addition, we find that models with lower scores on BigCodeBench-Complete degrade less on BigCodeBench-Instruct, limited by their programming capability.

Figure: Figure 7: Top 5 instruction-tuned LLMs on BigCodeBench-Complete.
Refer to caption: x8.png

#### 4.2 Measuring Tool-Level Performance

Besides evaluating task completion performance, we try to understand how well LLMs can use tools for task-solving. As it is hard to evaluate the accuracy of each independent function call regardless of the correctness of the model solutions, we rely on the calibrated greedy decoding results on BigCodeBench-Complete and deem the use of the associated library correct if the task is passed by all test cases.

###### Jack of all trades, master of most.

Figure 7 shows the top 5 calibrated instruction-tuned LLMs ranked on BigCodeBench-Complete. The best overall performing LLMs, like GPT-4o, excel in most domains but still fall short in certain ones. We suggest that domain specialization can result from the training data. Similar findings can be found in the base models, which are visualized in Appendix L.2.

**Table 2: Tool-use comparisons between all generated solutions (Sol.) and ground truths (GT.) on BigCodeBench-Complete.**
|  | Library (%) | Function Call (%) |  |  |
| --- | --- | --- | --- | --- |
| $Sol.\subseteq GT.$ | $Sol.\nsubseteq GT.$ | $Sol.\subseteq GT.$ | $Sol.\nsubseteq GT.$ |  |
| Pass | 79.84 | 20.16 | 40.46 | 59.54 |
| Fail | 71.78 | 28.22 | 22.47 | 77.53 |

###### LLMs use different function calls to solve tasks.

We assess how all 60 calibrated models use tools on BigCodeBench-Complete and report the mean statistics in Table 2. By inspecting the library usage, we find that models use imported libraries in more than 70% of tasks. For the remaining 20%, we notice that models tend to import additional libraries to solve the tasks. After some inspection, we notice that most of these added libraries are standard, explaining why they can still pass the tests. When analyzing the function calls, we find that models tend to use different function calls from the ones in ground truths.
Due to the open-endedness of the programming tasks, the invocation of function calls is expected to be flexible. However, using more diverse function calls is more likely to introduce task failures. We provide several examples where the model-generated solution contains function calls different from the ground truth in Appendix M, Examples 9 and 10.

### 5 Comparison to Existing Programming Benchmarks

**Table 3: Correlation coefficients measured by Pearson’s $r$ and Spearman’s $p$. against HumanEval+ and LiveCodeBench.**
|  | BigCodeBench-Complete | BigCodeBench-Instruct |  |  |
| --- | --- | --- | --- | --- |
|  | $r$ | $p$ | $r$ | $p$ |
| HumanEval+ | 0.849 | 0.861 | 0.864 | 0.894 |
| LiveCodeBench | 0.853 | 0.898 | 0.778 | 0.815 |

We compare model performances on BigCodeBench-Complete against existing benchmarks like HumanEval+ (Liu et al., ) and LiveCodeBench (Jain et al., ) used for evaluating the coding capabilities of LLMs. We compute the Pearson and Spearman correlation coefficients for the calibrated model ranks on BigCodeBench-Complete and BigCodeBench-Instruct against HumanEval+ and LiveCodeBench. From Table 3, we observe the strong correlations for both coefficients, suggesting that BigCodeBench is well aligned with the mainstream evaluation trend. However, as BigCodeBench measures the different aspects from HumanEval+ and LiveCodeBench, some models are expected to have various ranks among these benchmarks. In addition, we note that models which have saturated HumanEval like GPT-4o, still have room for improvement on BigCodeBench in comparison to human performance.

### 6 Related Work

###### Large Language Models for Code

With the rise of LLMs, there have been various models trained on code. Codex (Chen et al., ) marks the first base LLM pre-trained on code, which was used as the backbone model for GitHub Copilot. More recently, pre-trained base code models (Nijkamp et al., ; Li et al., ; Lozhkov et al., ; Roziere et al., )
have been built to perform accurate code completion.
Later, with the advance in instruction tuning (Ouyang et al., ), LLMs can generate the code snippet that aligns with the given NL instruction.
Instruction-tuned code models (Muennighoff et al., ; Luo et al., ; Wei et al., ), are generally better at programming than their base ones.

###### Programming Benchmarks

To assess the programming capability of LLMs, researchers have proposed many programming benchmarks. Most existing benchmarks focus on short, self-contained, and algorithm-specific programming tasks, such as HumanEval (Chen et al., ) and MBPP (Austin et al., ).
Recently, open-domain coding benchmarks (Wang et al., ; Lai et al., ) have been built to challenge code LLMs on application-specific scenarios by using specific sets of tools and libraries.
However, they focus on simple intents and use limited specific function calls per programming task, making their evaluation less challenging and realistic.
Benchmarks like SWE-bench (Jimenez et al., ) are built to evaluate the performance of a code agent framework (e.g., iterated prompting, real-time environment interaction, and long-context exploration).
Our BigCodeBench focuses on evaluating the fundamental code generation capability of LLMs, which is also an essential path toward strong code agents.
In addition, SWE-Bench is constructed from GitHub repositories with existing test cases — which limits the diversity of the tasks considered. In contrast, our collaborative LLM-human annotator procedure allows generating tasks from seed queries collected from (Wang et al., ), tackling a broader range of software tasks.

### 7 Conclusion

We introduce BigCodeBench, a new high-quality programming benchmark constructed via the collaboration between human experts and LLMs, assessing the capability of tool use and complex instruction following. Through the extensive evaluation of 60 LLMs, we find that there is a long way for models to achieve perfection on this benchmark and share a few findings that can potentially improve the performance. We urge the community to work on more advanced LLMs for code and continually build upon our benchmark and extended BigCodeBench-Hard (Appendix F),as discussed in our long-term roadmap (Appendix G).

### Acknowledgements

We truly appreciate BigCode making many great works happen, including The Stack (Lozhkov et al., ), SantaCoder (Allal et al., ), StarCoder (Li et al., ), OctoPack (Muennighoff et al., ), Astraios (Zhuo et al., ), StarCoder2 (Lozhkov et al., ), and StarCoder2-Instruct (BigCode, ). BigCodeBench cannot be built without the support of the BigCode community. We thank Sea AI Lab and MASSIVE for providing the computational resources. The project is also partially supported by Terry Yue Zhuo’s CSIRO’s Data61 PhD Scholarships and Xiaoning Du’s Google Research Scholar Program Award. For the benchmark, we particularly thank Xiangru Tang, Dmitry Abulkhanov, Noah Ziems, Chengran Yang, Jiamou Sun, Nickil Maveli, and Lili Bo for their eagerness to participate. We are extremely grateful to the EvalPlus team for their open-source evaluation framework. We also thank Loubna Ben Allal, Zhiruo Wang, Zhensu Sun, Sean Hughes, and Zhou Yang for their valuable comments.

## 

### Appendix

parttocContents
\parttoc

### Appendix A Contributions

#### Project Leadership

Terry Yue Zhuo

#### Benchmark Construction

Chien Vu,
Jenny Chim,
Han Hu,
Haolan Zhan,
Xiaoheng Hong,
Wenhao Yu,
Niklas Muennighoff,
Jean Kaddor,
Wen-Ding Li,
Junda He,
Ming Xu,
Zhihan Zhang,
Ratnadira Widyasari,
Indraneil Paul,
Simon Brunner,
Imam Nur Bani Yusuf,
Thong Hoang,
Chen Gong,
Armel Zebaze,
Prateek Yadav,
Terry Yue Zhuo

#### Experiment

Terry Yue Zhuo,
Binyuan Hui

#### Evaluation Framework

Terry Yue Zhuo,
Indraneil Paul,
Jiawei Liu,
Zhoujun Cheng

#### Analysis

Terry Yue Zhuo,
Binyuan Hui,
Zhoujun Cheng,
Alex Gu,
Naman Jain

#### Paper Writing

Terry Yue Zhuo

#### Presentation Editing

Niklas Muennighoff,
Indraneil Paul,
David Lo,
Zijian Wang,
Daniel Fried,
Binyuan Hui,
Qian Liu,
Jean Kaddor,
Jiawei Liu,
Imam Nur Bani Yusuf,
Chen Gong

#### Long-term Advice & Support

Daniel Fried,
Niklas Muennighoff,
Qian Liu,
Zijian Wang,
Binyuan Hui,
Xiaoning Du,
David Lo,
Jiawei Liu,
Harm de Vries,
Leandro von Werra

### Appendix B Datacard

We follow (Bender & Friedman, ) to create the datacard for BigCodeBench, where we tend to summarize and centralize all information that might be relevant for the benchmark analysis.

###### Curation Rationale

This is detailed in Section 2 and Appendix J.

###### Language Variety

Information about our annotators’ nationality will not be provided, as the constructed benchmark is hardly related to regional or social dialects. However, we confirm that all communications during the annotation process are in mainstream English (en-US). We note that the first language of some annotators is not English, which can introduce some inaccurate expressions to the task prompts in BigCodeBench.

###### Curators Demographic

The benchmark construction requires the great annotation effort of Curators, who are involved in the process detailed in Section 2. They come from the following population:

[itemize]
@ Age:
@@ 18-25: 25% (5/20)
@@ 26-35: 70% (14/20)
@@ 36-45: 5% (1/20)
@ Experience in Python Programming (Years):
@@ 1-3: 5% (1/20)
@@ 3-5: 20% (4/20)
@@ 5+: 75% (15/20)
@ Academic Background:
@@ Bachelor: 5% (1/20)
@@ Master: 20% (4/20)
@@ PhD: 75% (15/20)

###### Text Characteristics

This is detailed in Section 3.

### Appendix C Data Sheet

Besides the provided Datacard, we follow the documentation frameworks provided by (Gebru et al., ).

#### C.1 Motivation

##### C.1.1 For what purpose was the dataset created?

Our dataset aims at providing a thorough assessment of the capability of solving programming tasks. Particularly, we focus on the challenges and practicability of the tasks, and pinpoint two main characteristics that few benchmarks highlight: (1) Diverse Function Calling; and (2) Complex Instruction Following. This dataset will help stakeholders better understand the fundamental abilities and limitations associated with deploying LLMs.

We believe that there are three main expectations of a good execution-based programming benchmark:

- •
The benchmark should be easy to use and efficient in evaluating the fundamental capabilities of LLMs. Repository-level benchmarks (e.g., SWE-bench ()) are not suitable for this purpose.
- •
The benchmark should be practical, covering various programming scenarios. Algorithm-specific benchmarks (e.g., HumanEval (Chen et al., ) and MBPP (Austin et al., )) are unsuitable. Domain-specific benchmarks (e.g., DS-1000 (Lai et al., )) are also unsuitable for this purpose.
- •
The benchmark should be challenging, where the tasks require LLMs’ strong compositional reasoning capabilities and instruction-following capabilities. The benchmarks with simple tasks (e.g., ODEX (Wang et al., )) are unsuitable.

BigCodeBench is the first benchmark that meets all three expectations. It is an easy-to-use benchmark that evaluates LLMs with practical and challenging programming tasks, accompanied by an end-to-end evaluation framework BigCodeBench. We aim to assess how well LLMs can solve programming tasks in an open-ended setting.

#### C.2 Composition/Collection Process/Preprocessing/Cleaning/Labeling and Use

The answers are described in our paper as well as the GitHub repository: [https://github.com/bigcode-project/bigcodebench-annotation](https://github.com/bigcode-project/bigcodebench-annotation).

#### C.3 Distribution

##### C.3.1 Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?

No. Our dataset will be managed and maintained by the BigCode community ([https://www.bigcode-project.org/](https://www.bigcode-project.org/)).

##### C.3.2 How will the dataset be distributed (e.g., tarball on website, API, GitHub)?

The evaluation dataset will be released to the public, and hosted on GitHub and Hugging Face.

##### C.3.3 When will the dataset be distributed?

It has been released now.

##### C.3.4 Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?

Our dataset is distributed under the Apache-2.0 license.

#### C.4 Maintenance

##### C.4.1 How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

Please contact Terry Yue Zhuo ([terry.zhuo@monash.edu](mailto:terry.zhuo@monash.edu)) and the BigCode Project ([contact@bigcode-project.org](mailto:contact@bigcode-project.org)), who are responsible for maintenance.

##### C.4.2 Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?

Yes. If we include more tasks or find any errors, we will correct the dataset hosted on Hugging Face and GitHub and update the results in the leaderboard accordingly. It will be updated on our website.

##### C.4.3 If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?

For dataset contributions and evaluation modifications, the most efficient way to reach us is via GitHub pull requests. For more questions, contact Terry Yue Zhuo ([terry.zhuo@monash.edu](mailto:terry.zhuo@monash.edu)) and the BigCode Project ([contact@bigcode-project.org](mailto:contact@bigcode-project.org)), who are responsible for maintenance.

### Appendix D Data Contamination

While BigCodeBench tasks are constructed from scratch, there are still some concerns regarding the potential data contamination. Therefore, we conduct N-gram contamination experiments on ODEX intents (English) and an anonymized archive of Stack Overflow used by StarCoder2 Lozhkov et al. () which may be correlated to BigCodeBench instructions. We also evaluated StarCoderData (Python) (Li et al., ), which has been widely used as the code training data for various LLMs. We focus on the overlaps between the BigCodeBench instructions and the queries contained by these data sources, using 10-gram and 13-gram setups (Brown, ; Elazar et al., ; Shao et al., ; Guo et al., ; Bai et al., ) to indicate potential data contamination. Due to the significant computational resources required, the 10-gram overlap on StarCoderData is timeout and thus omitted.

**Table 4: Comparison of N-Gram data across different datasets.**
| N-Gram | Overlap Percentage (%) |  |  |
| --- | --- | --- | --- |
| ODEX | Stack Overflow | StarCoderData |  |
| 13 | 0.00 | 0.18 | — |
| 10 | 0.09 | 1.49 | 2.49 |

As shown in Table 4, the likelihood of our task descriptions being contaminated by existing data is extremely low. With a stricter 10-gram configuration, no more than 2.5% of BigCodeBench tasks overlapped with the tested data sources.

### Appendix E Extended Related Work

###### Large Language Models for Code

With the rise of LLMs, various models have been trained on code. The first series of models, like CodeBERT (Feng et al., ) and GraphCodeBERT (), are built based on the BERT (Kenton & Toutanova, ) encode-only architecture, which specializes in code-understanding tasks such as defect detection and clone detection (). As there is an increasing need for automated code completion, the T5-based encoder-decoder architecture has been applied to train code-specific models like PyMT5 (Clement et al., ), PLBART (Ahmad et al., ), CodeT5 (Wang et al., ), CodeT5+ (Wang et al., ), and GrammarT5 (Zhu et al., ). While these models are better at generating code and comments, they are not fully optimized. More LLMs for code have become decoder-only, following the design of GPT (Radford, ). The early decoder-only LLMs for code, trained with left-to-right token prediction, cannot assist developers in inserting code between programs. To fulfill this need, InCoder (Radford, ) has been designed to allow code infilling. Subsequently, the combination of left-to-right and fill-in-the-middle code generation has been widely adopted in later model development, such as StarCoder (Li et al., ), CodeLlama (Roziere et al., ), and DeepSeek-Coder (Guo et al., ). However, these models are pre-trained only and are denoted as base models. With advances in instruction tuning (Ouyang et al., ), more LLMs for code are further fine-tuned on natural language instructions to better understand user queries. For example, WizardCoder () and MagiCoder (Wei et al., ) are instruction-tuned with synthetic data containing diverse instruction-code pairs. Later, OpenCodeInterpreter (Zheng et al., ) developed a multi-turn instruction dataset and achieved better coding performance. More recently, there has been a growing interest in agentic programming  (), developing prompting systems to enhance the capabilities of performing software engineering tasks, like GitHub issue resolution and code search. However, they are agnostic regarding the backbone models. In this work, we aim to assess the fundamental capabilities of LLMs in practical code generation without any customized prompting systems.

###### Programming Benchmarks

To assess the programming capability of LLMs, researchers have proposed various programming benchmarks. Most existing benchmarks focus on short, self-contained, algorithm-specific programming tasks, such as HumanEval (Chen et al., ) and MBPP (Hendrycks et al., ). While there are other programming benchmarks like APPS (Hendrycks et al., ) and CodeContests (Li et al., ), they are not frequently used due to the lack of an easy-to-use evaluation infrastructure. Additionally, some benchmarks focus on real-world program-based tool use, such as DS-1000 (Lai et al., ), ODEX (Wang et al., ), NumpyEval (), and TorchDataEval (Zan et al., ). These benchmarks are designed to evaluate how well LLMs can handle specific libraries and programming scenarios, simulating more realistic and complex coding tasks. However, these benchmarks often focus on simple intents and use a limited number of specific function calls per programming task, making their evaluations less challenging and realistic. Meanwhile, there has been another line of work on multi-programming-language code generation. For example, HumanEval-X (Zheng et al., ), MultiPL-E (Cassano et al., ), and CodeScope (Yan et al., ) extend Python-only programming benchmarks via code translation, without considering programming-language-centric downstream scenarios. While there are some benchmarks addressing language-specific limitations like ClassEval (Du et al., ) and OOP (Wang et al., ), their tasks are limited in terms of quantity, complexity, and diversity. Later, benchmarks like RepoBench (), CrossCodeEval (Ding et al., ), CoderEval (Yu et al., ), and SWE-bench () are designed to evaluate the performance of a code agent framework, which includes iterated prompting, real-time environment interaction, and long-context exploration. Our benchmark focuses on evaluating the fundamental code generation capability of LLMs, which is also an essential path toward strong code agents. Additionally, SWE-bench is constructed from GitHub repositories with existing test cases, which limits the diversity of the tasks considered. In contrast, our collaborative LLM-human annotator procedure allows for generating tasks driven by real-world software engineering requirements, similar to the queries from StackOverflow, tackling a broader range of software tasks. Furthermore, our benchmark emphasizes some important aspects that have not been well discussed in the programming domain, like open-endedness (Hughes et al., ), multi-tool use (Qin et al., ), and instruction-following (Lou et al., ).

### Appendix F BigCodeBench - Hard

Running the full set of BigCodeBench will be burdensome for common users, especially when evaluating a large model on both BigCodeBench-Complete and BigCodeBench-Instruct. In order to save budgets, we release a minimal high-quality subset of BigCodeBench-Hard, serving as a proxy for the full set.

As illustrated in Figure 8, the workflow to construct BigCodeBench-Hard is mainly inspired by MixEval (Ni et al., ), which utilizes a small number of benchmark samples to align user-facing evaluation. While MixEval focuses on general-domain evaluation and considers only code generation tasks with minimal samples from MBPP and HumanEval, we extend the idea to make code generation evaluation more user-centric. Specifically, we follow these steps to create BigCodeBench-Hard:

Figure: Figure 8: BigCodeBench-Hard construct workflow.
Refer to caption: x9.png

First, we choose [an anonymized archive of Stack Overflow](https://huggingface.co/datasets/bigcode/stack-exchange-preferences-20230914-clean-anonymization) that has been preprocessed by the BigCode community. Details of the preprocessing can be found in the StarCoder2 (Lozhkov et al., ). The archive contains 10.4 million questions and answers, covering diverse programming languages and topics, making it a good source of user queries.

To bridge the query source and BigCodeBench, we leverage all-mpnet-base-v2, a pre-trained sentence embedding model recommended by the Sentence Transformers documentation (Reimers, ). This model, trained on a mixture of text and code data, is suitable for identifying similarities between programming queries and BigCodeBench tasks.

We use the model to retrieve the most similar tasks for each query in the Stack Overflow archive, ranking them by the dot product between normalized embeddings. Based on manual inspection of the retrieved tasks, we conclude that a similarity score above 0.7 is a good threshold for task selection. By applying this threshold, we obtain 6,895 queries and 626 BigCodeBench tasks after deduplication.

We illustrate the alignment between the Stack Overflow queries (Figure 9) and the BigCodeBench tasks (Figure 10). As shown in the figures, both the query and the task prompt revolve around web scraping to extract hyperlinks from web pages, using Python libraries to handle HTTP requests and parse HTML. Both involve interaction with CSV files to either read input URLs or write output data. While the specific implementation details differ, the core objective of extracting and handling hyperlink data from web pages is a shared aspect, aligning their overall scope closely.

Figure: Figure 9: StackOverflow user query.
Refer to caption: extracted/5908016/query.png

Figure: Figure 10: BigCodeBench task prompt.
Refer to caption: extracted/5908016/task.png

However, the retrieved 626 tasks are still infeasible for evaluation. To improve evaluation efficiency, we further filter the tasks by difficulty. Unlike the construction of MixEval-Hard, we define the following more explainable criteria: (1) Library Usage: Each task in BigCodeBench emphasizes the compositional reasoning ability for coding and requires the use of at least two libraries. For BigCodeBench-Hard, we keep only the tasks that require more than two libraries, challenging the models to choose more diverse function calls as tools to solve the tasks; (2) Solution Length: We set the threshold at 426 tokens, which is the average solution length of the tasks in BigCodeBench. The ground-truth solution provides a reference for the task complexity, and tasks with longer solutions are more challenging to solve; and (3) Solve Rate: We compute the solve rate per task based on all the evaluated models on the leaderboard. The solve rate is defined as the number of models that can solve the task divided by the total number of models. Specifically, we deem tasks with a solve rate below 50% as hard tasks.

Figure: Figure 11: BigCodeBench rankings v.s. BigCodeBench-Hard rankings
Refer to caption: x10.png

Through comparison, we notice that the model performance on BigCodeBench-Hard differs significantly from the one on the full set of BigCodeBench. We suggest that these differences arise from the imbalanced distribution of target domains and a large number of easy tasks in BigCodeBench, resulting in a slight misalignment between the evaluation and user-facing tasks. For example, GPT-4o-2024-05-13 and GPT-4-0613 may be overfitting to the easy tasks in BigCodeBench, leading to low performance on BigCodeBench-Hard.

To validate the effectiveness of BigCodeBench-Hard, we use a private leaderboard, [SEAL-Coding](https://scale.com/leaderboard/coding) curated by Scale AI, as a reference. The SEAL-Coding leaderboard is designed to evaluate models on a set of user-facing tasks across various application domains and programming languages. Specifically, SEAL-Coding compares [four of the best closed LLMs on Python](https://x.com/summeryue0/status/1807804955136598061), with the following rankings: (1) GPT-4-Turbo Preview, (2) Claude 3.5 Sonnet, (3) GPT-4o, and (4) Gemini 1.5 Pro (May 2024). These rankings align with our results based on the average score of the Complete and Instruct splits of BigCodeBench-Hard, indicating that BigCodeBench-Hard is more user-centric and challenging for model evaluation.

We encourage the community to use BigCodeBench-Hard when the budget is limited, and the evaluation needs to be more user-centric. Additionally, we note that BigCodeBench-Hard can be dynamic by design, depending on user queries and the evaluated models. We can periodically update BigCodeBench-Hard to keep the evaluation challenging and user-centric.

For dataset contributions and evaluation modifications, the most efficient way to reach us is via GitHub pull requests. For more questions, please contact Terry Yue Zhuo ([terry.zhuo@monash.edu](mailto:terry.zhuo@monash.edu)) and BigCode Project ([contact@bigcode-project.org](mailto:contact@bigcode-project.org)), who are responsible for maintenance.

### Appendix G Long-Term Roadmap and Call for Contributions

In this section, we share a long-term roadmap to address the limitations of BigCodeBench, and sustainably build with the community. We believe that program-aided language models (Gao et al., ) for task completion and reasoning provide the possibility towards artificial general intelligence. Our goal is to provide the community with the most open, reliable, and scalable evaluations to truly understand the fundamental capabilities of LLMs for programming, pinpointing the ways to unleash their power.

#### G.1 Limitations

Given the limited time and budget we have to develop the initial benchmark, we have foreseen several limitations and aim to address them step-by-step.

###### Multilingualism

One of the main limitations is that BigCodeBench is Python-only and cannot be easily extended to other programming languages. As the function calls are mostly language-specific, it is hard to find a package or library with exactly the same functionalities other than Python. However, given the fact that Python is the most flexible and popular programming language to automate various tasks, BigCodeBench may fulfill most of the community needs. In the meantime, we still seek efficient approaches to construct BigCodeBench-like programming tasks with tools in other languages, without much human effort.

###### Saturation

Another potential criticism is that some LLMs can still perform reasonably well BigCodeBench, considering that the best models can only resolve no more than 30% of real-world GitHub issues on SWE-bench. One might indicate that our benchmark is not challenging enough. However, we note that the low performance on SWE-bench is likely due to the under-specified instructions and misaligned test cases(^5^55[https://github.com/princeton-nlp/SWE-bench/issues/72](https://github.com/princeton-nlp/SWE-bench/issues/72)). Compared to SWE-bench, we tend to make the programming tasks much less ambiguous and ensure that the authors can pass their own solutions during annotation.

###### Reliability

During the execution-based evaluation, we notice that some test cases are flaky (Luo et al., ), which results in uncertainty across multiple test runs without any code changes. We have progressively resolved some identified issues, such as missing setup of random states and improper removal of non-existent files. However, the remaining cases are trickier. For example, the socket query can be timed out or refused due to the unstable connection. With that being said, we try our best to make the uncontrollable changes of Pass@1 under 0.6%. We plan to continually enhance the reliability of all test cases, with the help of the community. To maintain the high reproducibility, we host a real-time code execution sandbox in the Hugging Face space.

###### Rigorousness

While we achieve high test coverage for the ground-truth solutions in BigCodeBench, it does not guarantee that any code generated by LLMs will be correctly assessed against existing test cases. Previous works like EvalPlus (Liu et al., ) have attempted to extend the limited test cases by augmenting the input-output pairs via LLM- and mutation-based strategies. However, it is challenging to adapt EvaPlus to the test harness in BigCodeBench, as the harness only examines the expected program behaviors during the runtime (e.g., mocking tests). Furthermore, the function calls used to pass test cases by LLMs are more nondeterministic, making traditional test generation (Anand et al., ) methods cover all possible scenarios. Therefore, we still consider LLM-based test generation (Wang et al., ) promising, but with proper designs. Specifically, a possible approach is to collect all the generated solutions that pass the current test cases and make a capable LLM (e.g., GPT-4o) harness with self-refinement in a sandbox environment.

###### Generalization

One intuitive question is “How well do the models generalize to the unseen tools and tasks?” Current BigCodeBench only covers the common libraries and daily programming tasks. It will be more interesting to benchmark models on the programming tasks that use emerging libraries like [transformers](https://github.com/huggingface/transformers) and [langchain](https://github.com/langchain-ai/langchain). Crafting new high-quality programming tasks requires huge effort, as demonstrated in this paper. There are two efficient approaches for building a programming benchmark for model generalization. First, we can instruction tune an LLM on BigCodeBench data with additional information of libraries and function calls. The trained model is expected to generate programming tasks with proper test cases based on the given library or function call details.
However, the quality of such data synthesis is unknown, making the practicability questionable.
Another way is to replace the function calls and libraries in BigCodeBench with the synthetic names, simulating the unseen ones. A similar approach (Zan et al., ) has been used for code generation on unknown libraries. Although this method may lose the naturalness of software development, the construction process is more controllable and practical.

###### Evolution

Naturally, the libraries can be obsolete or updated (Lamothe et al., ), which means that the source code data for model training will constantly evolve. Thus, the models may not memorize function calls from a deprecated library version. This poses a challenge for any tool-dependent programming benchmarks to correctly examine the model capability without periodic updates. Another related concern is the test set contamination issue due to the evolving training data. It is suggested to have both a public set and a private set for better evaluation in a recent blog(^6^66[https://www.jasonwei.net/blog/evals](https://www.jasonwei.net/blog/evals)). For future releases, we aim to perform the benchmark evolution both publicly and privately, and host the private test set internally.

###### Interaction

Recent interests are around the concept of LLMs as Agents (Xi et al., ), which is deemed as a way towards artificial general intelligence. Specifically, LLMs will be grounded in a less constrained sandbox environment, where they can interact with any given applications, such as the web browser and terminal. The environment can help unlock the capabilities like self-debugging (Chen et al., ) and self-reflection (Shinn et al., ). We tend to work in this direction and see how well LLMs as Agents can perform on BigCodeBench.

#### G.2 BigCodeBench -OOD

We will construct BigCodeBench-OOD, a variant of BigCodeBench that assesses the generalization ability on out-of-distribution (OOD) libraries and programming tasks (Zhout et al., ). We note that any features that do not frequently appear can be considered OOD. We tend to build BigCodeBench-OOD with two different types of OOD: (1) Completely synthetic features; and (2) Real but long-tail features. The first type can represent the private libraries that are unknown to the models, and the latter one may represent the ones trending in the recent programming practice but have not yet played major roles in the training data.

#### G.3 BigCodeBench -Interact

To examine the agent-centric programming ability, we will adapt BigCodeBench to an interactive sandbox environment, with a list of applications that can help models program and repair iteratively as humans. Unlike existing programming benchmarks such as HumanEval and MBPP, BigCodeBench covers a wide range of tasks that require profound programming knowledge. Without the proper understanding, models are likely to fail in debugging and repair. There are potentially two options for the evaluation. One naive option is to allow models to see the backtrace of how they fail in the test cases, which can guide them to repair the bugs with a clear goal. A more challenging setup is to make models fix bugs independently without seeing any backtrace. Capable models are expected to resolve issues by generating their own test cases and inspecting the execution outcomes.

### Appendix H Artifacts

**Table 5: Artifacts for reproducibility.**
| Name | Public Link or Endpoint |
| --- | --- |
| BigCodeBench (v0.2.0) |  |
| GitHub | [https://github.com/bigcode-project/bigcodebench/releases/tag/v0.2.0.post3](https://github.com/bigcode-project/bigcodebench/releases/tag/v0.2.0.post3) |
| Hugging Face | [https://huggingface.co/datasets/bigcode/bigcodebench](https://huggingface.co/datasets/bigcode/bigcodebench) |
| Croissant | [https://huggingface.co/api/datasets/bigcode/bigcodebench/croissant](https://huggingface.co/api/datasets/bigcode/bigcodebench/croissant) |
| Annotation Framework |  |
| GitHub | [https://github.com/bigcode-project/bigcodebench-annotation](https://github.com/bigcode-project/bigcodebench-annotation) |
| Evaluation Framework |  |
| GitHub | [https://github.com/bigcode-project/bigcodebench](https://github.com/bigcode-project/bigcodebench) |
| PyPI | [https://pypi.org/project/bigcodebench/](https://pypi.org/project/bigcodebench/) |
| Datasets for Comparisons |  |
| HumanEval | [https://github.com/openai/human-eval/blob/master/data/HumanEval.jsonl.gz](https://github.com/openai/human-eval/blob/master/data/HumanEval.jsonl.gz) |
| ODEX | [https://github.com/zorazrw/odex/tree/master/data](https://github.com/zorazrw/odex/tree/master/data) |
| DS-1000 | [https://github.com/xlang-ai/DS-1000/blob/main/data/ds1000.jsonl.gz](https://github.com/xlang-ai/DS-1000/blob/main/data/ds1000.jsonl.gz) |
| APPS | [https://github.com/hendrycks/apps](https://github.com/hendrycks/apps) |
| APIBench | [https://github.com/ShishirPatil/gorilla/tree/main/data/apibench](https://github.com/ShishirPatil/gorilla/tree/main/data/apibench) |
| MBPP | [https://github.com/google-research/google-research/tree/master/mbpp](https://github.com/google-research/google-research/tree/master/mbpp) |
| NumpyEval | [https://github.com/microsoft/PyCodeGPT/tree/main/cert/pandas-numpy-eval](https://github.com/microsoft/PyCodeGPT/tree/main/cert/pandas-numpy-eval) |
| PandasEval | [https://github.com/microsoft/PyCodeGPT/tree/main/cert/pandas-numpy-eval](https://github.com/microsoft/PyCodeGPT/tree/main/cert/pandas-numpy-eval) |
| TorchDataEval | [https://github.com/microsoft/PyCodeGPT/blob/main/apicoder/private-eval/data/real_torchdata_eval_v3.jsonl.gz](https://github.com/microsoft/PyCodeGPT/blob/main/apicoder/private-eval/data/real_torchdata_eval_v3.jsonl.gz) |
| Models for Evaluations |  |
| GPT-4o | gpt-4-turbo-2024-04-09 |
| GPT-4-Turbo | gpt-4-0125-preview |
| GPT-4 | gpt-4-0613 |
| GPT-3.5-Turbo | gpt-3.5-turbo-0125 |
| Claude-3-Opus | claude-3-opus-20240229 |
| Claude-3-Sonnet | claude-3-sonnet-20240229 |
| Claude-3-Haiku | claude-3-haiku-20240307 |
| DeepSeek-V2-Chat | deepseek-chat (API, 32K) |
| Mistral-Large | mistral-large-2402 |
| Mistral-Small | mistral-small-2402 |
| DeepSeek-Coder-33b-Instruct | [https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct) |
| DeepSeek-Coder-6.7b-Instruct | [https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct) |
| DeepSeek-Coder-1.3b-Instruct | [https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-instruct) |
| CodeLlama-70B-Instruct | [https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf) |
| CodeLlama–34B-Instruct | [https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf) |
| CodeLlama-13B-Instruct | [https://huggingface.co/codellama/CodeLlama-13b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-13b-Instruct-hf) |
| CodeLlama-7B-Instruct | [https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf) |
| Llama-3-70B-Instruct | [https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) |
| Llama-3-8B-Instruct | [https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) |
| Qwen2-72B-Instruct | [https://huggingface.co/Qwen/Qwen2-72B-Instruct](https://huggingface.co/Qwen/Qwen2-72B-Instruct) |
| Qwen2-7B-Instruct | [https://huggingface.co/Qwen/Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct) |
| Qwen2-57B-A14B | [https://huggingface.co/Qwen/Qwen2-57B-A14B](https://huggingface.co/Qwen/Qwen2-57B-A14B) |
| CodeQwen1.5-7B-Chat | [https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat](https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat) |
| Qwen1.5-110B-Chat | [https://huggingface.co/Qwen/Qwen1.5-110B-Chat](https://huggingface.co/Qwen/Qwen1.5-110B-Chat) |
| Qwen1.5-72B-Chat | [https://huggingface.co/Qwen/Qwen1.5-72B-Chat](https://huggingface.co/Qwen/Qwen1.5-72B-Chat) |
| Qwen1.5-32B-Chat | [https://huggingface.co/Qwen/Qwen1.5-32B-Chat](https://huggingface.co/Qwen/Qwen1.5-32B-Chat) |
| Yi-1.5-34B-Chat | [https://huggingface.co/01-ai/Yi-1.5-34B-Chat](https://huggingface.co/01-ai/Yi-1.5-34B-Chat) |
| Yi-1.5-9B-Chat | [https://huggingface.co/01-ai/Yi-1.5-9B-Chat](https://huggingface.co/01-ai/Yi-1.5-9B-Chat) |
| Yi-1.5-6B-Chat | [https://huggingface.co/01-ai/Yi-1.5-6B-Chat](https://huggingface.co/01-ai/Yi-1.5-6B-Chat) |
| Mixtral-8x22B-Instruct | [https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1) |
| Magicoder-S-DS | [https://huggingface.co/ise-uiuc/Magicoder-S-DS-6.7B](https://huggingface.co/ise-uiuc/Magicoder-S-DS-6.7B) |
| CodeGemma-7B-Instruct | [https://huggingface.co/google/codegemma-7b-it](https://huggingface.co/google/codegemma-7b-it) |
| StarCoder2-Instruct | [https://huggingface.co/bigcode/starcoder2-15b-instruct-v0.1](https://huggingface.co/bigcode/starcoder2-15b-instruct-v0.1) |
| Granite-34B-Code-Instruct | [https://huggingface.co/ibm-granite/granite-34b-code-instruct](https://huggingface.co/ibm-granite/granite-34b-code-instruct) |
| Granite-20B-Code-Instruct | [https://huggingface.co/ibm-granite/granite-20b-code-instruct](https://huggingface.co/ibm-granite/granite-20b-code-instruct) |
| Granite-8B-Code-Instruct | [https://huggingface.co/ibm-granite/granite-8b-code-instruct](https://huggingface.co/ibm-granite/granite-8b-code-instruct) |
| Granite-3B-Code-Instruct | [https://huggingface.co/ibm-granite/granite-3b-code-instruct](https://huggingface.co/ibm-granite/granite-3b-code-instruct) |
| CodeLlama-70B-Base | [https://huggingface.co/codellama/CodeLlama-70b-hf](https://huggingface.co/codellama/CodeLlama-70b-hf) |
| CodeLlama-34B-Base | [https://huggingface.co/codellama/CodeLlama-34b-hf](https://huggingface.co/codellama/CodeLlama-34b-hf) |
| CodeLlama-13B-Base | [https://huggingface.co/codellama/CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf) |
| CodeLlama-7B-Base | [https://huggingface.co/codellama/CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf) |
| Llama-3-70B-Base | [https://huggingface.co/meta-llama/Meta-Llama-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B) |
| Llama-3-8B-Base | [https://huggingface.co/meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) |
| DeepSeek-Coder-base-33B | [https://huggingface.co/deepseek-ai/deepseek-coder-33b-base](https://huggingface.co/deepseek-ai/deepseek-coder-33b-base) |
| DeepSeek-Coder-base-6.7B | [https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base) |
| DeepSeek-Coder-base-1.3B | [https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base) |
| CodeQwen1.5-7b | [https://huggingface.co/Qwen/CodeQwen1.5-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B) |
| Yi-1.5-34B | [https://huggingface.co/01-ai/Yi-1.5-34B](https://huggingface.co/01-ai/Yi-1.5-34B) |
| Yi-1.5-9B | [https://huggingface.co/01-ai/Yi-1.5-9B](https://huggingface.co/01-ai/Yi-1.5-9B) |
| Yi-1.5-6B | [https://huggingface.co/01-ai/Yi-1.5-6B](https://huggingface.co/01-ai/Yi-1.5-6B) |
| Mixtral-8x22B-Base | [https://huggingface.co/mistralai/Mixtral-8x22B-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1) |
| CodeGemma-7B | [https://huggingface.co/google/codegemma-7b](https://huggingface.co/google/codegemma-7b) |
| CodeGemma-2B | [https://huggingface.co/google/codegemma-2b](https://huggingface.co/google/codegemma-2b) |
| StarCoder2-15B | [https://huggingface.co/bigcode/starcoder2-15b](https://huggingface.co/bigcode/starcoder2-15b) |
| StarCoder2-7B | [https://huggingface.co/bigcode/starcoder2-7b](https://huggingface.co/bigcode/starcoder2-7b) |
| StarCoder2-3B | [https://huggingface.co/bigcode/starcoder2-3b](https://huggingface.co/bigcode/starcoder2-3b) |
| Granite-34B-Code-Base | [https://huggingface.co/ibm-granite/granite-34b-code-base](https://huggingface.co/ibm-granite/granite-34b-code-base) |
| Granite-20B-Code-Base | [https://huggingface.co/ibm-granite/granite-20b-code-base](https://huggingface.co/ibm-granite/granite-20b-code-base) |
| Granite-8B-Code-Base | [https://huggingface.co/ibm-granite/granite-8b-code-base](https://huggingface.co/ibm-granite/granite-8b-code-base) |
| Granite-3B-Code-Base | [https://huggingface.co/ibm-granite/granite-3b-code-base](https://huggingface.co/ibm-granite/granite-3b-code-base) |

### Appendix I Tool Statistics

#### I.1 Analysis

Figure: (a) BigCodeBench
Refer to caption: x11.png

Figure: (a) BigCodeBench
Refer to caption: x16.png

#### I.2 Comparison to Existing Programming Benchmarks

**Table 6: Depth (Complexity — Solution Characters) and breath (Diversity – Function Calls) comparisons to existing programming benchmarks in Python.**
| Benchmark | Complexity | Diversity |
| --- | --- | --- |
| APPS (Hendrycks et al., ) | 352.9 | 137 |
| DS-1000 (Lai et al., ) | 138.1 | 328 |
| ODEX (Wang et al., ) | 50.4 | 230 |
| APIBench (Patil et al., ) | 77.5 | 171 |
| MBPP (Austin et al., ) | 181.1 | 51 |
| NumpyEval (Zan et al., ) | 30.1 | 52 |
| PandasEval (Zan et al., ) | 45.8 | 12 |
| HumanEval (Chen et al., ) | 180.9 | 7 |
| TorchDataEval (Zan et al., ) | 52.8 | 6 |
| BigCodeBench | 426.0 | 723 |

#### I.3 Version Control

#### I.4 Domain Classification

### Appendix J Detailed Benchmark Construction

#### J.1 Data Synthesis Prompt

#### J.2 Semi-automatic Program Refactoring and Testing Case Generation

##### J.2.1 Programming Task Classification Prompt

##### J.2.2 Guidelines

#### J.3 Human Curation Guidelines

### Appendix K Evaluation Setup

#### K.1 Inference

We perform all the model inference on A100 GPUs, except for the closed ones. For the closed models, we rely on their official APIs provided in the documents.

#### K.2 Execution

We conduct the execution mainly on the Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz, composed of 2 sockets, with 18 cores per socket.

#### K.3 Prompt Template

Figure: Figure 14: Prompt template for models supported by vLLM (Kwon et al., ).

Figure: Figure 15: Prompt template for OpenAI/DeepSeek APIs.

Figure: Figure 16: Prompt template for Mistral model APIs.

Figure: Figure 17: Prompt template for Anthropic model APIs.

### Appendix L Detailed Benchmarking Results and Analysis

#### L.1 Detailed Results

**Table 7: Evaluating LLMs on BigCodeBench-Complete. Due to the training flaws in StarCoder2 and Granite-Code, we additionally strip the trailing newlines for model inference.**
| Model | Size / Checkpoint | Greedy Decoding | Random Sampling |  |  |
| --- | --- | --- | --- | --- | --- |
| Original | Calibrated | $Pass@1$ | $Pass@5$ |  |  |
| Instruction-tuned LLMs |  |  |  |  |  |
| GPT-4o (OpenAI, ) | 2024-05-13 | 0.602 | 0.611 | 0.557 | 0.711 |
| GPT-4-Turbo (OpenAI, ) | 2024-04-09 | 0.582 | 0.582 | 0.563 | 0.699 |
| GPT-4 (Achiam et al., ) | 0613 | 0.484 | 0.572 | 0.407 | 0.682 |
| GPT-3.5-Turbo (OpenAI, ) | 0125 | 0.506 | 0.506 | 0.504 | 0.657 |
| Claude-3 (Anthropic, ) | Opus | 0.568 | 0.574 | — | — |
|  | Sonnet | 0.534 | 0.538 | — | — |
|  | Haiku | 0.486 | 0.501 | — | — |
| Mistral (AI, ) | Large | 0.379 | 0.383 | 0.365 | 0.539 |
|  | Small | 0.390 | 0.413 | 0.398 | 0.601 |
| DeepSeek-Chat (DeepSeek-AI, ) | V2 | 0.494 | 0.494 | 0.482 | 0.596 |
| DeepSeek-Coder-instruct (Guo et al., ) | 33B | 0.511 | 0.511 | 0.491 | 0.687 |
|  | 6.7B | 0.432 | 0.438 | 0.426 | 0.624 |
|  | 1.3B | 0.293 | 0.296 | 0.270 | 0.468 |
| CodeLlama-instruct (Roziere et al., ) | 70B | 0.489 | 0.496 | 0.429 | 0.681 |
|  | 34B | 0.351 | 0.356 | 0.266 | 0.518 |
|  | 13B | 0.297 | 0.317 | 0.247 | 0.470 |
|  | 7B | 0.254 | 0.257 | 0.193 | 0.403 |
| Llama3-instruct (AI@Meta, ) | 70B | 0.539 | 0.545 | 0.522 | 0.650 |
|  | 8B | 0.368 | 0.369 | 0.338 | 0.562 |
| Qwen2-Instruct (Bai et al., ) | 72B | 0.539 | 0.540 | 0.489 | 0.682 |
|  | 57B-A14B | 0.465 | 0.468 | 0.399 | 0.648 |
|  | 7B | 0.407 | 0.421 | 0.329 | 0.577 |
| CodeQwen1.5-Chat (Bai et al., ) | 7B | 0.437 | 0.443 | 0.408 | 0.632 |
| Qwen1.5-Chat (Bai et al., ) | 110B | 0.432 | 0.444 | 0.425 | 0.590 |
|  | 72B | 0.398 | 0.403 | 0.373 | 0.569 |
|  | 32B | 0.410 | 0.420 | 0.375 | 0.557 |
| Yi-1.5-Chat (Young et al., ) | 34B | 0.433 | 0.428 | 0.411 | 0.622 |
|  | 9B | 0.422 | 0.424 | 0.379 | 0.601 |
|  | 6B | 0.338 | 0.339 | 0.290 | 0.496 |
| Mixtral-instruct (AI, ) | 8x22B | 0.496 | 0.502 | 0.458 | 0.677 |
| Magicoder-S-DS (Wei et al., ) | 6.7B | 0.475 | 0.476 | 0.424 | 0.643 |
| CodeGemma-instruct (Team et al., ) | 7B | 0.383 | 0.393 | 0.334 | 0.568 |
| StarCoder2-Instruct (BigCode, ) | 15B | 0.437 | 0.451 | 0.404 | 0.610 |
| Granite-Code-Instruct (Mishra et al., ) | 34B | 0.438 | 0.444 | 0.399 | 0.613 |
|  | 20B | 0.421 | 0.421 | 0.375 | 0.600 |
|  | 8B | 0.392 | 0.397 | 0.367 | 0.581 |
|  | 3B | 0.315 | 0.315 | 0.250 | 0.466 |
| Base LLMs |  |  |  |  |  |
| CodeLlama (Roziere et al., ) | 70B | 0.443 | — | 0.364 | 0.639 |
|  | 34B | 0.371 | — | 0.297 | 0.570 |
|  | 13B | 0.320 | — | 0.243 | 0.527 |
|  | 7B | 0.287 | — | 0.207 | 0.457 |
| Llama3-base (AI@Meta, ) | 70B | 0.433 | — | 0.375 | 0.625 |
|  | 8B | 0.288 | — | 0.223 | 0.466 |
| DeepSeek-Coder-base (Guo et al., ) | 33B | 0.466 | — | 0.401 | 0.661 |
|  | 6.7B | 0.418 | — | 0.343 | 0.599 |
|  | 1.3B | 0.223 | — | 0.174 | 0.412 |
| CodeQwen1.5 (Bai et al., ) | 7B | 0.456 | — | 0.411 | 0.650 |
| Yi-1.5 (Young et al., ) | 34B | 0.400 | — | 0.302 | 0.575 |
|  | 9B | 0.355 | — | 0.295 | 0.563 |
|  | 6B | 0.274 | — | 0.201 | 0.435 |
| Mixtral-base (AI, ) | 8x22B | 0.455 | — | 0.355 | 0.633 |
| CodeGemma (Team et al., ) | 7B | 0.373 | — | 0.295 | 0.557 |
|  | 2B | 0.239 | — | 0.153 | 0.375 |
| StarCoder2 (Lozhkov et al., ) | 15B | 0.383 | — | 0.332 | 0.609 |
|  | 7B | 0.285 | — | 0.232 | 0.514 |
|  | 3B | 0.214 | — | 0.178 | 0.416 |
| Granite-Code (Mishra et al., ) | 34B | 0.385 | — | 0.333 | 0.582 |
|  | 20B | 0.258 | — | 0.286 | 0.552 |
|  | 8B | 0.356 | — | 0.287 | 0.536 |
|  | 3B | 0.202 | — | 0.177 | 0.406 |

**Table 8: Evaluating instruction-tuned LLMs on BigCodeBench-Instruct. We report the results of greedy decoding, paired with calibrated results. To understand the performance difference between BigCodeBench-Complete and BigCodeBench-Instruct, we compute $\Delta(NL2C-C2C)$ for each model. We do not include the results of Granite-Code-Instruct 8B & 3B as they constantly have empty outputs.**
| Model | Size / Checkpoint | Greedy Decoding | $\Delta(NL2C-C2C)$ |  |  |
| --- | --- | --- | --- | --- | --- |
| Original | Calibrated | Original | Calibrated |  |  |
| GPT-4o | 2024-05-13 | 0.499 | 0.511 | -0.103 | -0.100 |
| GPT-4-Turbo | 2024-04-09 | 0.480 | 0.482 | -0.102 | -0.100 |
| GPT-4 | 0613 | 0.459 | 0.460 | -0.025 | -0.112 |
| GPT-3.5-Turbo | 0125 | 0.382 | 0.391 | -0.124 | -0.115 |
| Claude-3 | Opus | 0.452 | 0.455 | -0.116 | -0.119 |
|  | Sonnet | 0.425 | 0.427 | -0.109 | -0.111 |
|  | Haiku | 0.392 | 0.394 | -0.094 | -0.107 |
| DeepSeek-Chat | V2 | 0.404 | 0.404 | -0.090 | -0.090 |
| DeepSeek-Coder-instruct | 33B | 0.418 | 0.420 | -0.093 | -0.091 |
|  | 6.7B | 0.353 | 0.355 | -0.079 | -0.083 |
|  | 1.3B | 0.227 | 0.228 | -0.066 | -0.068 |
| CodeLlama-instruct | 70B | 0.405 | 0.407 | -0.084 | -0.089 |
|  | 34B | 0.289 | 0.290 | -0.062 | -0.066 |
|  | 13B | 0.282 | 0.285 | -0.015 | -0.032 |
|  | 7B | 0.218 | 0.219 | -0.036 | -0.038 |
| Llama3-instruct | 70B | 0.432 | 0.436 | 0.107 | -0.109 |
|  | 8B | 0.317 | 0.319 | -0.051 | -0.050 |
| Qwen2-Instruct | 72B | 0.382 | 0.385 | -0.157 | -0.155 |
|  | 57B-A14B | 0.357 | 0.361 | -0.108 | -0.060 |
|  | 7B | 0.280 | 0.291 | -0.127 | -0.130 |
| CodeQwen1.5-Chat | 7B | 0.393 | 0.396 | -0.044 | -0.047 |
| Qwen1.5-Chat | 110B | 0.336 | 0.350 | -0.096 | -0.094 |
|  | 72B | 0.326 | 0.332 | -0.072 | -0.071 |
|  | 32B | 0.318 | 0.323 | -0.092 | -0.097 |
| Yi-1.5-Chat | 34B | 0.335 | 0.339 | -0.098 | -0.089 |
|  | 9B | 0.343 | 0.345 | -0.079 | -0.079 |
|  | 6B | 0.255 | 0.256 | -0.083 | -0.083 |
| Mistral | Large | 0.296 | 0.300 | -0.083 | -0.083 |
|  | Small | 0.318 | 0.321 | -0.072 | -0.092 |
| Mixtral | 8x22B | 0.404 | 0.406 | -0.092 | -0.096 |
| Magicoder-S-DS | 6.7B | 0.360 | 0.362 | -0.115 | -0.114 |
| CodeGemma-instruct | 7B | 0.321 | 0.323 | -0.062 | -0.070 |
| StarCoder2-Instruct | 15B | 0.367 | 0.376 | -0.070 | -0.075 |
| Granite-Code-Instruct | 34B | 0.358 | 0.361 | -0.080 | -0.083 |
|  | 20B | 0.337 | 0.340 | -0.084 | -0.081 |

#### L.2 Further Analysis

###### Bigger models yield better programming performance

The performance on BigCodeBench displays signs of scaling laws, where the models with more parameters generally solve more programming tasks. While the scaling laws hold for most instruction-tuned and base LLMs, Mistral-Large appears less capable than Mistral-Small on both BigCodeBench-Complete and BigCodeBench-Instruct, implying that Mistral-Large may be under-fitting. Our findings are consistent with (Yan et al., ).

###### Closed LLMs perform better than open LLMs

We notice that most strong LLMs on BigCodeBench are non-permissive, led by the models from OpenAI and Anthropic. Among the open LLMs, the best model, Llama3-instruct-70B, slightly outperforms Claude-3-Sonnet and ranks 5th on both BigCodeBench-Complete and BigCodeBench-Instruct. The most permissive LLMs are relatively small and, hence, remain in a gap from the non-permissive ones.

###### Jack of all trades, master of most

Figure 7 shows the top 5 for instruction-tuned ranked on BigCodeBench-Complete. Best overall base LLMs like Dpsk-Coder-Base-6.7B excel in most domains but still fall short in certain ones. We suggest that the domain-specific specialty will likely result from the training data.

Figure: Figure 18: Top 5 base LLMs
Refer to caption: x21.png

###### LLMs are better at computation, cryptography, and general domains

We further investigate the performance among different domains and find that models tend to be more capable of the tools in computation, cryptography, and general domains. On the other hand, models frequently fail when the tasks involve network tools. Therefore, we encourage the development of models specializing in such low-performing domains. A few examples of failures in the network domain can be found in Appendix M Examples 6, 7, and 8.

#### L.3 Prompting Techniques

**Table 9: Calibrated Pass@1 comparison of plain prompting and zero-shot CoT prompting on both BigCodeBench and BigCodeBench-Hard.**
| Split-Subset | GPT-4o-2024-05-13 (%) | Gemini-1.5-Pro-API-0514 (%) |
| --- | --- | --- |
| Full-Complete | 61.1 $\rightarrow$ 59.4 | 57.5 $\rightarrow$ 55.9 |
| Full-Instruct | 51.1 $\rightarrow$ 49.5 | 43.8 $\rightarrow$ 44.2 ($\uparrow$) |
| Hard-Complete | 29.1 $\rightarrow$ 34.5 ($\uparrow$) | 31.1 $\rightarrow$ 27.7 |
| Hard-Instruct | 25.0 $\rightarrow$ 23.0 | 19.6 $\rightarrow$ 18.2 |

We provide some preliminary studies on the effectiveness of the zero-shot chain-of-thought (CoT) (Kojima et al., ) on GPT-4o and Gemini-1.5-Pro by appending "$\backslash$nLet’s think step by step." to the end of the original prompt. From Table 9, we observe that there is no significant advantage to using zero-shot CoT for performing BigCodeBench tasks, indicating that the prompting technique is not sufficiently reliable for complex compositional reasoning. Therefore, we conclude that prompt design is very tricky and should not be recommended for the official evaluation setup. For our prompt design, we followed the same approach used in EvalPlus for consistency.

### Appendix M Qualitative Studies

Our benchmark exposes instances where even the strongest models struggle to use APIs, which may not be exposed by simple benchmarks like HumanEval. Here, we show some examples with GPT-4o.

Example 1: In the following example, the model fails to use itertools.cycle correctly, producing an infinite loop.

Example 2: In this example, the model uses the semantics of instanceof incorrectly. Here, isinstance(j, (int, float)) does not actually check if $j$ is an int or float, but rather the type (int, float).

Example 3: Here, the model mishandles the case where the directory does not exist with incorrect use of os.path.dirname.

Example 4: In this example, the model misunderstands how to use the itertools.zip_longest method. The arguments must be iterable, but the written function feeds single values.

Example 5: In the following example, the model outputs code that has a different encoding for password_hash and expected_password_hash so that even if the passwords are the same, the response will indicate a failed one.

Example 6: This example showcases a failure to understand the nuances of the wikipedia library. When page_title is not found in Wikipedia, verbatim, the wikipedia.page method returns a wikipedia.exceptions.DisambiguationError rather than a wikipedia.exceptions.PageError.

Example 7: In this Network example, the error actually originates from the regex. For the first example, instead of www.google.com, the regex actually extracts www.google.com,, which leads to errors down the line.

Example 8: In this example, the model misunderstands the host naming structures. The issue is in the line response = requests.get(f"https://ipinfo.io/ip_address/json"), where the /json should not be present. This leads to a hostname not found error.

Example 9: This example shows an example where there is a simple solution using flatten, but the generated solution does not use this function and therefore gets the function incorrect.

Example 10: This is another example where the model chooses to use the wrong function. Here, it uses np.mean instead of np.nanmean (which the ground truth solution uses). This leads to the failure of a test with incomplete data.

### Appendix N Unit Test Design

HumanEval Test: The HumanEval tests only consider input-output assertions, which only work for simple programs without configuration and environment setup. We use several tests below as an example.

BigCodeBench Unit Test: We demonstrate the design of BigCodeBench unit tests as follows, where we mock various scenarios for the network connection and validate the program behaviours. Compared to the ones in HumanEval and other benchmarks like APPS using input-output assertions, our unit tests require great human effort to design and cover various settings.

### Appendix O bigcodebench : Evaluation Infrastructure

In this section, we document the usage of bigcodebench, the evaluation infrastructure for BigCodeBench. We note that the prototype of bigcodebench is based on EvalPlus (Liu et al., ).

### Appendix P Development Timeline

###### 04/2023 - 05/2023

Project Initiation.

###### 06/2023 - 07/2023

Benchmark Construction — Data Synthesis.

###### 07/2023 - 11/2023

Benchmark Construction — Semi-automatic Program Refactoring and Testing Case Generation.

###### 12/2023 - 04/2024

Benchmark Construction — Human Curation; BigCodeBench Evaluation Tool Development.

###### 04/2024 - 05/2024

Benchmark Finalization; Experiment; Analysis; Drafting; Code-Eval Development.

###### 06/2024

Initial BigCodeBench Release.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - The Python libraries used in this study are mostly popular ones, likely present in the pretraining data of most LLMs. Future work could explore how LLMs adapt to newer, domain-specific libraries that may not have been included in their training data.
- Based on the benchmark results, what potential approaches could improve the performance of existing LLMs in handling complex instructions and function calls? Insights on targeted training techniques or model adjustments would be valuable.
soundness: 3
strengths: - The paper makes a valuable contribution to code generation benchmarks by incorporating diverse function calls and complex instructions, offering a novel perspective on evaluating LLM capabilities in library and tool use. Results indicate that even top-performing LLMs face challenges with these tasks, highlighting opportunities for further advancements.
- The paper is well-structured and easy to follow. The benchmark curation framework is technically sound, back with a systematic design and thorough quality checks. The extensive evaluation provides some insights into how current LLMs handle library usage and follow complex instructions.
summary: This paper introduces BigCodeBench, a novel code generation benchmark that improves existing ones like HumanEval by incorporating more diverse function calls and complex instructions. BigCodeBench comprises 1,140 Python code generation tasks, necessitating the use of 139 libraries across 7 domains. The dataset was constructed through a semi-automated framework leveraging collaboration between LLMs and human experts. In this process, LLMs are prompted to generate coding tasks based on sampled code snippets, performing both initial generation and refinement of programs and test cases, while human annotators continuously provide feedback to enhance quality. Extensive experimental evaluations reveal that current LLMs struggle with accurately executing complex instructions and function calls, achieving scores of up to 60%, significantly below the human performance of 97%. These results highlight the ongoing challenges and the need for further advancements in the field of code generation.
weaknesses: - The technical contribution of the benchmark curation framework appears incremental, as the collaborative approach between LLMs and human is already widely adopted in LLM research.
- While Python libraries are essential for task solutions in this benchmark, the rapid evolution and versioning of these libraries can introduce compatibility issues. The authors should consider methods to mitigate the effects of version incompatibility in the evaluation process.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: - How to prevent future data contamination? 
- How are test cases constructed to evaluate visualization tasks requiring graph outputs?
soundness: 3
strengths: - The proposed benchmark introduces a challenging task that involves diverse function calls as tools and complex instruction-following for code generation. Current state-of-the-art LLMs achieve up to 60% Pass@1, while human performance reaches 97%, providing a clear benchmark for future research to improve on.
- By evaluating a wide range of both open-source and closed-source LLMs on this benchmark, the study establishes a strong foundation and adds credibility to the proposed framework.
summary: This paper introduces a novel benchmark to evaluate the ability of cutting-edge LLMs in function calling and complex instruction following for code generation.
weaknesses: - There is a risk of future data contamination [1, 2]. To mitigate this, it may be beneficial to release only the validation set while keeping the test set hidden, similar to practices used in NL2SQL benchmarks [3,4].

- Some domains, such as visualization tasks returning graph outputs, would benefit from examples illustrating how the test cases are constructed for evaluation.

[1] Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039–12050, Bangkok, Thailand. Association for Computational Linguistics.

[2] Jain, Naman, et al. "Livecodebench: Holistic and contamination free evaluation of large language models for code." arXiv preprint arXiv:2403.07974 (2024).

[3] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.

[4] Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, & Yongbin Li (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

---

confidence: 3
contribution: 4
initial_rating: 10
presentation: 4
questions: - How do you distill observations such as "Interestingly, we observe that instruction-tuned LLMs can omit the essential import statements of the given prompts", do you rely on static analysis of automatic error logs? are humans also involved in the process?
soundness: 4
strengths: ### Originality
The benchmark proves to be original for the following reasons:
- It contains diverse code, sourced from widely adopted Python libraries from seven domains. 
- It contains more complex programming tasks than other benchmarks. Specifically, the authors report on task cyclomatic complexity compared to other function-level Python programming benchmarks, and a larger coverage of function calls compared to state-of-the-art function calling benchmarks. 

### Quality

#### Benchmark Construction
The authors followed best practices to select, clean, and filter the seed data. They also integrate the use of LLMs into the process by keeping humans in the loop when needed. Moreover, code execution in a controlled environment is performed to ensure code quality.

#### Validation
- The authors extensively evaluate 60 state-of-the-art LLMs on BigCodeBench-Complete (code-completion based on docstrings) and 35 instruction-tuned LLMs on BigCodeBench-Instruct (programming tasks based on natural language instructions). 

### Clarity
- The benchmark construction and evaluation processes are remarkably explained.
- Limitations of the dataset as well as future work are addressed in detail as part of the appendix.

### Significance
As BigCodeBench will be open-sourced it represents a significant contribution to the research community. Moreover, the study reveals interesting observations about the ability of LLMs in complex code-completion tasks, which sets a roadmap for the development of new training datasets to improve these capabilities.
summary: The authors present BigCodeBench, a massive high-quality benchmark to test Large Language Models (LLMs) ability to perform diverse function calls and solve complex code instructions. BigCodeBench comprises 1,140 fine-grained Python programming tasks after proper cleaning and filtering and covers functionality from 139 widely adopted programming libraries. Specifically, the benchmark evaluates two common programming scenarios (1) Code completion based on structured docstrings, and (2) the completion of programming tasks based on natural language instructions. The authors used powerful LLMs to source programming tasks, refactor code, and generate test cases, under constant human supervision. This construction process is a remarkable example of a perfect balance between LLM automation and human curation to ensure data quality and reliability. The authors conduct an extensive evaluation of state-of-the-art LLMs on BigCodeBench. Among others, the results show that LLMs are not yet capable of following complex instructions to use function calls precisely.
weaknesses: Please fix the typo in line 182 (refractor -> refactor) to make even better this amazing work.

---

confidence: 4
contribution: 4
initial_rating: 10
presentation: 4
questions: - With respect to line 237, what were the reasons or examples of model failure in the dataset that was improved and captured by the annotators? Subsequently, what steps were taken to overcome it?
- In case of BigCodeBench-Instruct benchmark, each user query is structured well in terms of sequence and format of instruction for code generation. This might not be the case when it comes to user-like query. Analysis on part of robustness to ambiguity in NL, variation in structured format needs to be further analyzed and presented within the paper.
- Line 300, 'PL: Programming Language', is not used within context of the table and is unnecessary.
- Table 1: Tool Statistics, # Call column, format the column to have all '/' inline, like the adjacent columns.
- Perhaps, Figure 6 can be more visually summarized using scatter plot with grid colored based on size of parameter of LLM. Can be updated, if it helps provide better display of story.
- A definition of what calibrated scores should be added in description.
- Table 3 comparison can also be done against other datasets like MBPP Plus dataset, NaturalCodeBench and EvalPlus.
- Add on analysis of how new benchmark overcomes the issue of data contamination due to the benchmarks being used as training data in several LLM. What rephrasing or variation in query structuring is ensured to tackle this?
- Add in example of failure cases and categorize to why LLM still fails on new benchmark. 
- How well does an expert perform comparatively?
soundness: 4
strengths: Quite a thorough analysis and structured approach of generation of benchmark is taken into account. With detailed literature review of past works in the field, the paper also demonstrates its effectiveness as an improved code generation benchmark and evaluation.
summary: This benchmark paper proposes improved execution-based evaluation for solving complex real-time application like code generation problems. It focuses on both the effective instruction query and the detailed test case setup for unit test sets, curated using human-LLM collaboration.
weaknesses: Apart from tiny bits of improvements needed, the paper is quite detailed in its analysis. However, it lacks the essence of what are the areas where these LLM are failing on the new benchmark, and a compared analysis of how an expert human would perform in the same. So as to note the scope of baseline improvement.

Meta-review: The paper presents a new code generation benchmark for evaluating large language models. Overall, the benchmark is comprehensive and more challenging with diverse function calls and complex instructions as opposed to previous generation of benchmarks like HumanEval. I believe this will be a useful contribution to the community. I will request the authors to consider making it as easy possible though. A point about "compatibility issues with future python versions" was mentioned in the review which I feel is critical for widespread use of the benchmark.

Decision: Accept (Oral)