BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Authors: Terry Yue Zhuo
Vu Minh Chien
Jenny Chim
Han Hu
Wenhao Yu
Ratnadira Widyasari
Imam Nur Bani Yusuf
Haolan Zhan
Junda He
Indraneil Paul
Simon Brunner
Chen GONG
James Hoang
Armel Randy Zebaze
Xiaoheng Hong
Wen-Ding Li
Jean Kaddour
Ming Xu
Zhihan Zhang
Prateek Yadav
Naman Jain
Alex Gu
Zhoujun Cheng
Jiawei Liu
Qian Liu
Zijian Wang
Binyuan Hui
Niklas Muennighoff
David Lo
Daniel Fried
Xiaoning Du
Harm de Vries
Leandro Von Werra
Venue: ICLR 2025 Oral (2025)
Keywords: Code Generation
Tool Use
Instruction Following
Benchmark
Primary area: datasets and benchmarks

Abstract: Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks range from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing **diverse function calls as tools** to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding **complex instructions**. Fulfilling both of these characteristics can pose a great challenge for LLMs. To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions containing only essential information. Our extensive evaluation of 60 LLMs shows that **LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%**. The results underscore the need for further advancements in this area.

Published as a conference paper at ICLR 2025
BIGCODEBENCH: BENCHMARKING CODE GENERA- TION WITH DIVERSE FUNCTION CALLS AND COMPLEX INSTRUCTIONS
Terry Yue Zhuo Minh Chien Vur
Jenny Chim
Han Hu1.3
Wenhao Yu
Ratnadira Widyasari
Imam Nur Bani Yusuf Haolan Zhan!
Junda He3
Indraneil Paul Simon Brunner
Chen Gong"
Thong Hoang? Armel Zebaze
Xiaoheng Hongl1 Wen-Ding Lja3 Jean Kaddour Ming Xu Zhihan Zhang
Prateek Yaday15 Naman Jain
Alex Gu Zhoujun Cheng
Jiawei Liu
Qian 2iu Zijian Wang" Binyuan Hui Niklas Muennighoff David Lo
Daniel Fried 24 Xiaoning Du Harm de Vriest25
Leandro von Werra
Core contributors, random ordering Additional contributors, random ordering
Senior contributors, random ordering
Monash University CCIRR's Data61
"Singapore Management University
*Detomo Inc., Japan $Queen Mary University of London University of Notre Dame
'TU Darmstadt 8 IIndependent "University of Virginia Inria 11 Intel Teneeent Al Lab
18 Cornell University 14Universtyy College London USC Chapel Hill
UC Berkeley
17 MIT 18 Shanghai Jiaotong University Uher UIUC 21 Sea AI Lab
22 AWS AI Labs"
23 Contextual AI Carneeie Mellon University 25 ServiceNow Research
26 Hugging Face
io/
(terry zhuo, xiaoning @monash. edu contactteblgoode-rrjjeee.
ABSTRACT
Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks range from software engineering development to
reasoning. While current benchmarks have shown that L.MSs can solve tasks using programs like human developers, the majority their cvaluations are limited to short and self-contained algorithmic tasks standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristies can pose great chal- lenge for LLMs. To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCode ch, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and domains for 1,140 fine- mne-grained tasks. To evaluate L.MSs rigorously. each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose natural- language-oriented variant of B BigCodeBench, BigCodeBench-Inn that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to function calls pre- cisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.
'The work does relate the author's position Amazon. Corresponding authors.

---

Published as a conference paper at ICLR 2025
Suport http.eliont
Parameters
Requirements
inport socket
server_ name [ste]: Hews of the server to which the request is made
http.etient
Export
kocket
SAL
der task, wwneerurvereeeeeee server_port, path):
server-.port (int]: Port number of the server to which the recuest is made
Examplles
Makes
request speciried
HTTP request
Task, Funce (w, can' ponnn,e 6 ' Isiinsaancennss, str)
server and path, and retrieves the response. Porooeeters:
True
Returnes
Returns
Test Case Class
Moisest...
stn: Response body from the serven
Setup
Teardoan()
Requinesants:...
test_re rest_retumn_ return_yyooe....
Exceples: ...
Raises
hing (ig(...)
***
sEL SSLarro:: on SSL handshake error
test, stt..,sss...adds handshakee hangt
test. ponteent
Figure 1: Programming tasks in BigCodeBench are structured with complex instructions in the docstrings, annotated by experts. The behavior of the solution is evaluated against a class of rigorous test cases with the proper environment setup.
INTRODUCTION
Task automation, including competitive programming (Li et al., 2022; Hendrycks et al., 2021; Jain al., 2024). GitHub issue resolution (Yang et al.), and question answering (Gao al., 2023; Chen et al.), has attracted significant interest from academia and industry to facilitate the development of advanced models for code, especially in Python (Wang al.). With recent advances in data-centric deep learning techniques, large language models (LLMs) trained on large-scale corpora have shown superior capabilities of translating textual inputs to syntactically correct and functional programs. However, widely-used benchmarks like HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) only contain short, self-contained, and algorithm-focue focused programming tasks and have been saturated by recent model releases. In this work, aim to close the evaluation gap between these isolated coding exercises and real-world programming. asking Can LLMs solve more challenging and more practical tasks via programs?
Solving programming tasks in real-world scenarios typically has two main characteristics: (1) Diverse Function Calls as Tools' A complex programming task often requires the invocation of diverse function call sequences as tools (Robillard & DeLine, 2011; Hu et al., 2018; Qin et al., 2023). To avoid reinventing the wheel, domain-specific libraries are designed cover function calls (or APIs) with comprehensive functionalities; and (2) Complex Instructions. Due to the complexity to perform various programming tasks, instructions may require compositional reasoning ability perform sequence of functionalities in the correct order (e.g.. input data manipulation, eтror message handling, and specific output formatting) (Wiegers Beatty, 2013; Paetsch et al., 2003; Partsch, 2012). For instance, creating network application that includes functionality for retrieving responses from an HTTPS server (Figure 1) requires integrating several components with multiple function calls, such managing SSL contexts, handling socket connections, and ensuring the response is returned in the specific format.
Building high-quality execution- based benchmark and the environment that simulates aforemen- tioned tasks with practical and challenging constraints non-trivial First, is hard to naturally source self-contained programming tasks with complex instructions. unlike the short code exercises in HumanEval, Although most GitHub repositories contain realistic source code, the functions inside these repositories often require cross-file information (Ding et al., 2024). Second, real-world program- ming scenarios are extremely diverse (Zhout et al., 2023). Existing benchmarks (Zan et al., 2022b; Lai al., 2023) only focus on popular scenarios like data science. Third, mainstream programming benchmarks have a significant number of ambiguous or underspecified specifications, resulting in inaccurate evaluation results (Siddiq et al., 2024; Jain et al.). While there have been attempts to improve the data quality with LLMs (Jain et al., 2023), LLMs have their own bias and cannot reliably perform refinement (Zheng et al., 2024a).
In this work. we refer to "tools" as oriented but non orjecct-rriieteed Code Function Call APIs, which are discussed in Gori i la OpenFunct onns The Code Function Calling APIs are typically seen in common external Python packages like Numpy, Skleam, and Pandas.
2

---

Published as a conference paper at ICLR 2025
Data Synthesis
H Program Refactoring
Human Curation
value of datetine. today()
& Test Case Generation
in tiee
Instruct
SPT-3.3 Terno
SPT-4
def, gencrate, acathor raporttuto, timc):
Cross-Check
Ro Refine
roport fon of cities across vardous time zones
Fendback
Add Test
st a given time (UTD).
Satation
Figure 2: Each programming task in BigCoeBBench is created through a three-stage construction process. The task quality controlled by the human-LLAn -LLM collaboration.
To construct massive, high-quality programming tasks, propose novel framework (Figure 2) that uses collaboration between LLMs and human experts to build a rigorous execution-basel benchmark, BigCodeBench. Particularly, we utilize LLMs to source programming tasks, refactor programs, and add test cases, under constant human supervision. Our benchmark contains 1, 1,140 rich-context and multi-tool-use programming tasks in Python, covering 723 function calls from 139 popular libraries across 7 domains. As we aim to let models reason any suitable function calls to complete the tasks via code, we design the unit test cases in an open-ended manner and examine certain behaviors based on different inputs (Jain et al.). We assess two common programming scenarios: (1) Code Completion [BigCodeBench-Congg lete), which evaluates the capability of code generation based on the structured docstrings; and (2) Instruction to Code (BigCodeBen.cooo ch- which evaluates the ability to complete programming tasks based on natural- laarrrallnnuugg-orree oriented (NL-oriented) instructions. While Bi SigCodeBench-Comg lete emphasizes structured docstring prompts, BigCCoeBench--Tnne Sich-Instruue ruct challenges LLMs to generate precise code without relying non-essential details like interactive Python examples.
Inst ruct),
Through extensive studies on 60 models, we find that LLMs still struggle to invoke multiple function calls from cross-domain libraries and follow complex instructions to solve programming tasks using Python. Specifically, the best performing LLM, GPT-44,, solves merely 60% of tasks on BigCodeBench-Conm Cwwpnhhh--olleeeeeeeee and less than 50% BigCodeBench-Inste LLMs themselves lack the ability to align with human expectations when instructions are more natural. Interestingly, we find that some instruction-tuned LLMs like GPT-4 constantly refuse to follow long instructions to repeat essential context and thus fail the test cases. Furthermore, LLMs perform differently when using domain-speciin function calls as tools. We also demonstrate the strongly positive correlations between mainstream benchmarks and BigCodeBench, validating our evaluation results.
indicating that
2 BENCHMARK CONSTRUCTION
In this section, we introduce our human-LLMM-oollborrtion framework for the BigCodeBench construction. As shown in Figure 2, the process of SiigCodeBenchh-Cmpllee. ett encompasses three stages: Data Synthesis, Semi-automatic Program Refactoring and Testing Case Generation, and Human Curation. In addition, we construct wil.CodeBer.c--Inss...... oriented benchmark for programming- and Appendix The construction is progressively contributed by 20 authors as annotators for one year in total, with 75% of them having more than 5 years of experience in Python programming. The progress is continually managed by the lead annotator to control the data quality. Each constructed programming task is expected to: (1) have clear instructions inside PEP-257-structures docstrings; (2) use multiple libraries as tools to solve the problem; and (3) be paired with at least five test cases encapsulated in the unittest framework, with a complex test setup (c.g., database connection, and directory creation), to verify program behaviors instead of relying on simple input-output assertions.
Section 2.4, an NL- solving. Further information is available in Appendix B
2.1 DATA SYNTHESIS
One intuitive method would be to rely on source code repositories to construct the function-level programming tasks. However, most repository-level functions require cross-file information, such as
3

---

Published as a conference paper at ICLR 2025
customized modules (Ding et al., 2024), which are hard to self-contain and document. We argue that leveraging LLMs to synthesize customized programming tasks can be more viable (Wei et al., 2023; Luo et al., 2023), especially when overseen by humans. Given code snippet of API usage with a brief human instruction as the seed example (Figure 2), an L.MM is instructed to enrich the programming intent and refine the corresponding implementation by using diverse libraries. Specifically, use seed examples from ODEX (Wang et al., 2023c), a benchmark containing intent-paired code skeletons from Stack Overflow. We use the GPT-4 APP, the strongest LLM at the time of data sourcing, to synthesize the programming tasks. To help the LLM synthesize self-contained and relevant programming tasks based on the seed example, we instruct the model with 2-shot in-context demonstration (Appendix J.1) crafted by the lead annotator.
As previous studies have shown that L.L.Ms favor their own generations (Zheng et al., 2024a; Pan- iekssery et al., 2024), such phenomena can make the model evaluation unfair. We mitigate the model biases with an obfuscation and perturbation process. We first replace the semantic-rich program entry points with dummy function names. In addition, we perturb the natural language descriptions in docstrings with the back-translation method of NL-Augmenter (Dhole et al., 2023), inspired by ReCode (Wang et al., 2023a). After validating the post-processee programs with an abstract syntax tree parser, we collected 4,718 function-level programming samples.
2.2 SEMI-AUTOMATIC PROGRAM REFACTORING AND TESTING CASE GENERATION
Programs synthesized by LLMs may contain various issues, such undeclared variables and runtime bugs. Without proper verification, the implementation cannot directly serve as a ground-truth solution. To construct high-quality exccution-base benchmark, we need to add test cases that can rigorously verify the correctness of programs and identify any bugs. However, it takes non-trivial effort for human developers understand synthesized programs and properly refactor them with thorough testing. To improve the program quality and ease human annotation, we propose a conversion-driven framework for program refactoring and test case generations inspired by (Xia & Zhang, 2023). Specifically, we utilize the Code Interpreter session in the web-based GPT-4, which is back-ended by a Linux-based virtual environment with pre-installed Python packages. We engage 13 authors as human annotators (including the lead annotator) for this step, assigning each annotator 100 randomly sampled programming tasks based on their preferences code topics (Appendix J.2.1). We detail the design of this -oop--o- framework from human and LLM aspects as follows:
Human Aspect Human developers possess varying preferences and levels of familiarity with specific data types and programming scenarios. To aid human annotators in providing more precise feedback for refining programs with GPT-4, we have defined 10 data types (e.g.. SQL, CSV, and Python built-in types) and task scenarios (e.g., data analysis, networking, and visualization). GPT-4 API is utilized to automatically classify each program according to these categories, with detailed descriptions available in Appendix 1.2.1. The annotators' role is continually instruct GPT-4 to refactor the programs and to provide continuous feedback to guide the model whenever it fails to self-debug or incorrectly refactor the program.
LLM Aspect To effectively guide GPT-4 in the iterative refinement of programs and test cases, provide detailed annotation guidelines in Appendix J.2.2 as initial prompt. These guidelines encompass two high-level instructions: (1) Refine the function, including its docstrings, to enhance realism and reduce ambiguity; and (2) Write unit tests to ensure the functional correctness of the given program deseription. Specifically, the model is taught to follow a step-by-step refinement process: (I) Remove unused libraries and add necessary libraries if they are missing in the code snippet; (2) Reformat docstrings to adhere to PEP-257 conventions; (3) Align program implementations with the instructions inside docstrings; (4) Write and execute unit tests to ensure they pass; and (5) Add refined programs and test cases to files for downloading.
During interactions with GPT-4 we identify two main drawbacks in the Code Interpreter session. First, GPT-4 struggles to write proper test cases when mocking tests are employed. While the model can generate high-level designs of mocking tests, often fails to understand how test cases should be constructed based on execution feedback. Second, GPT-4 can become stuck while resolving runtime
We use the model version gpt-4-0613,
4

---

Published as a conference paper at ICLR 2025
bugs, leading to iterative refinement until the session times out. Continuous human feedback on viable solutions is essential to address these issues and ensure the model stays on track.
During the post-processing of the annotated data, observe that significant number of test cases were incomplcte. This issue arises because GPT-4 often omits partial content when writing long contexts into files. After removing all invalid programs via program analysis, end up with 1 1,223 refactored programming tasks with paired test cases.
2.3
HUMAN CURATION
To enhance the benchmark quality, we implement a three-fold human curation process:
Examination We first perform rigorous manual examination of the benchmarks, guided by set of detailed guidelines (Appendix J.3) to add more test cases and resolve a list of runtime issues due the LLM-generated flaky tests (Luo et al., 2014). In addition, we aim to formalize further the programming tasks based on the following criteria: (I) The task should utilize at least two libraries to enrich the task scopes; (2) Import statements should be limited to those used in the task implementations and be natural based on human understanding: (3) The docstring should follow PEP-257 to ensure consistent styles; (4) Both the task implementation and test cases should be clearly aligned with the task docstrings, based on human understanding: and (5) Required library modules in the docstrings should be aligned with the import statements. For test case construction, the criteria are as follows: (1) Test cases should be encapsulated by the unittest framework, making them pyptest-compaibble and (2) Test cases need to be deterministic, asserting expected program behaviors based on the inputs (Jain et al.).
To enhance the benchmark quality, we perform dry run to pre-evaluate an LLM other than GPT-4 used in previous steps. We choose GPT-3.5-Turie API to generate solutions based on the examined task prompts. By understanding how the model fails on the tasks, annotators may add essential details to the docstrings to clarify the task instructions but avoid describing step-by-step implementations.
Pre-Evaluation
Cross-Checde Cross-Checking To further validate data quality and ensure consistency across all programming tasks in the testing environment, 7 additional annotators refine and finalize the data annotated by others. This round focuses on the utility of docstrings and test cases. We automatically parse the docstrings and ask annotators to manually correct the docstring structures, specifically addressing task descriptions, function parameters, expected returns, exception handling, required modules, and interactive examples. Additionally, we remove unused imported modules via program analysis. For the interactive examples, we ensure their correctness via doctest, except for those requiring system and network access. The confirmed programming tasks are finally validated by the automated test workflows in GitHub Container Registry, where the test cases automatically run against the task implementations in a configured sandbox environment. To ensure the finalized data quality, we randomly assign 33 finalized task prompts the 11 annotators (the lead annotator one annotator was unavailable at the time), to write the solutions. The lead annotator conducts the evaluation of the solutions, finding that 97% (32 out of 33) of sampled tasks can pass all test cases.
excluded;
2.4 BENCHMARKING NL-ORIENTED INSTRUCTIONS TO CODE GENERATION
When instruction tuning LLMs using NL data, the input is mainly an NL instruction, and the target output is the NL or source code for completion (Muennighoff et al., 2023). This training objective aligns with the downstream applications such as multi-turn dialogue, where users ask questions or provide task-specific instructions to the models. While existing programming benchmarks commonly format the verbose instructions in docstrings (Chen al., 2021), users may instruct the models to generate code samples with the less verbose NL context. Despite that there have been similar attempts like HumanEVallPack (Muennighoff al., 2023) addressing this limitation, their inputs still lack some naturalness. Generally, users tend to describe the high-level idea of functionalities and avoid redundant information (e.g. parameters) or too-specific details (e.g., interactive examples). Thus, we create B
benchmark variant that prompts L.LMs to solve
'ee use model version -pt-3.5 -ttubbo--0
5

---

Published as a conference paper at ICLR 2025
Complete Prompt
Instruct Promрt
Inport
Write a function cet tock, func(sontpts
anport Joon
backupp.ll log htop/log.j/ee Jaon
def task_func(
should exception
sorist= waaktttbacppppeee. sh"
Log hog=//tmp/// Jogo ):
The function shoutd output with:
Ratarna
Description
You should start with:
Parerwtene:
saport
Returnss
Raisess
Requiineeents
oef fesk_fune(
scripti script:"bcckop.et backup.sh", tog /tt////////gg/smm
Laamplex
):
Figure 3: Each task prompt in BigCodeeench--Inttrrus are transformed from BigCodeeercchhCooppeeee via the pre-defined rules. We omit non-essential details during the transformation.
programming tasks with more NL-oriented instructions, assessing the model's ability to understand human requirements correctly. Based on the task prompts created in BigCoceBenchh-omp ette, design set of parsing rules and transform them into more natural instructions (Figure 3). For quality control, 5 authors who do not participate in previous stages inspect the randomly sampled prompts and their corresponding ground-truth solutions and reach agreement the alignment between the instruction prompts and task implementations.
3 BENCHMARK STATISTICS
Domain
Library
Function Call
Computation (63%)
pandas, sktearn, rath, statistics, ov2, statemodels, tensorfion, textbleb, skisage.
pandas. DataFrase, randon, randon.eddddd seed,
pandas. mendas.reed.coom read_csv, randon.,antiin,, pandas. Serins.
General (44%)
randes, re, collactions, itertools, string, operator, heape, functools, regex, inspect,
collactions. Counter, randos, seed, choice, se.sub, re.iindal,, itertools.chaiie
randin,
Visualization (31%)
latplot seaborn, PIL, folius, sordclous, turtla, epl_toolests
natplotlib, siplottib.pyplot, natplotlib. www.ott...pplo..eee.. pyplottsuuupllass
satplortib.pyplon pyplot..
System (30%)
Jeon, subprocoss, pathlib, sqlites, io, zistile, logging, pickle, struct, puttil.
os. ps.path.join,
os.rakedirs, glob. glob. os:tistdir, loon.load,
shutit.nove.
Time (10%)
datetine, time, pytz, dateutil, holidays,
srrtiii.datatton datetiso. deriiiinddaatiinn... lise, time.alesp, batatiae.datetiee strrtii..stpptie..
Network (8%)
requests, wrllib, b54, socket, django, flask, ipaddress, setplib, http, flask, rail, egi, sal, masil, eecheniza.
arse urlparse, django.http. HttpResponse, ipaddress, IPvilletion, setplib. SHTP, requests. post, socket. gaierron.
Cryptography (5%)
binsacii, codaca, cryptagraphy, Crypto
cryptography weeeeet.tt.et.... prwettiies.. padding. padding.........p... cryptographyy wwwpsaawt.piiitissspppeeoos wtw.daat.iiss...dddd....... C57_
generate, key,
Figure 4: Examples of tools in BigCo are illustrated. Each function call belongs to domain-speciffe library. The distribution of each domain is computed based on the frequency of domain-specific libraries appearing per task. For example, "63%" in "Computation" means that there are 63% tasks Bi BigCodeBench using at least one computation library.
Overall Statistics The first part of Table presents an overview comparison between BigCodeBench and other mainstream function-level Python programming benchmarks. We note that the full DS-1000 dataset emphasizes the perturbed tasks to avoid model memorization and thus includes an original subset for reference. Therefore, we also include statistics for non-perturbed problems IDS-1000 Orig.). As the provided statistics suggest, BigCodeBench contains much more rigorous execution -aasut evaluation and has longer task prompts that contain complex instructions. The ground-truth solutions also longer than in prior benchmarks, indicating that the tasks in- side BigCodeBench require more complex implementation logic. To illustrate the programming complexity, measure eyclomatic complexity, which is the number of independent paths through

---

Published as a conference paper at ICLR 2025
Table 1: Summarized statistics of representative function-level Python programming benchmarks partially inspired by (Zan et al., 2023). B BigCoceBench are more complex regarding the depth of task complexity and breadth of tool-use knowledge. Cov. Branch Coverage. Char: Characters. C.C.: Cyclomatic Complexity (starting from 0, higher values indicate more complex code). Lib.; Library. Dom.: Domain. Std.: Python Standard Library. Ext.: Python External Library.
Overall Statistics
Test (Avg.)
Prompt (Avg.)
Solution (Avg.)
Benchmark
Task
Cov.
Char: (Code)
Line
Char.
Line
C.C.
HumanEval
164
98%
450.6 (450.6)
13.7
180.9
6.8
3.6
DS-1000
1,000
1,6
98%
871.8 (193,9)
29.1
138.1
5.1
1.6
DS- 1000 (Orig.)
452
1.5
98%
831.4 (201.2)
26.2
115.5
4.2
1.4
ODEX
945
96%
87.5 (87.5)
1.0
50.4
gcodeRench--Complees
1112.5 (1112.5)
33.5
1,140
5.6
99%
426.0
10.0
3.1
Big LigCodBBencc-Insseeett
663.2 (124.0)
11.7
Tool Statisties
Lib.
# Call
Tasks (Avg.)
Combination
Benchmark
Dom.
Std. Ext.
Std. Ext.
#ib
# Call
Lib.
Calls
Dom.
HumanEival (Chen al., 2021)
3
4/0
7/0
0.1
0.1
6
8
DS- DS-1000 (L.a et al., 2023)
4
5/9
7/321
0.8
1.1
66
331
24
DS-1000 (Orig.) (Lai al., 2023)
4
4/9
5/289
0.9
1.3
59
260
23
ODEX (Wang 2023c)
7
40/26
128
0.6
0.5
105
202
20
BigCodeBench
7
72
281 /442
2.8
4.7
577
1045
56
the task solution. We notice that BigCodeBench has similar complexity to HumanEval, much higher than DS-1000 and ODEX. The high cyclomatic complexity indicates that solving the tasks in B @CCodBeenchh requires non-trivial reasoning ability from the programming perspective.
Tool Statistics In this work, we highlight any libraries and function calls from these libraries as program- tools for LLMs (Qin et al., 2023). Therefore, compare the tools used among the mainstream benchmarks the second part of Table First, BigCodeBench covers 281 function calls from 77 standard libraries and 442 from 62 external libraries, which is far more diverse than the other benchmarks. The ex- amples of these tools are shown in Figure 4. Based on the PyP[ download statistics, the mean download counts of the recent 30 days are 56.9M+ for the 62 external libraries, suggesting their high popularity in real-world software development. Second, BigCodeBench frequently invokes sequence of function calls from multiple libraries to solve single task, requiring sig- nificant compositional reasoning ability for took-nolving. Third, Bi BigCodeBench has more diverse combinations among li- braries, function calls, and domains in the ground-truth solu- tions. We visualize the library and function call density used by different benchmarks in Appendix I, showing the even broader tool-use diversity in Bi Big SiggCodeBench, To better understand solution complexity (measured by characters) and tool-use diversity (measured by distinct function calls), we compare the tasks DigCCddeBencr with those in representative benchmarks in Figure (details are provided in Appendix 1.2). We find that Bi gCodeBench requires more complex reasoning and problem-solving skills implement comprehensive functionalities.
Depth (Comptexity)
BigCodesench
Breadth (Tool)
0
APPS
APIBench
PandasEvol
HBPP
HunanEval
MumpyEvot
TorchlataEvat
Figure Complexity tool com- parisons with various benchmarks.
EVALUATION
Our evaluation uses the unbiased version of Pass (Chen et al., 2021) to accurately assess the functional correctness of generated code snippets by LLMs. To make general observations, we extensively evaluate 60 state-of-the LLMs on BigCodieBench-Comol Specifically, following prior works (Roziere et al.,
and 35 instruction-
tuned LLMs on eiigoodeeench-Inssrrae 2023; Liu et al., 2024; Lai et al., 2023), we report Pass@1 with greedy decoding for the main experiments in the zero-shot setting. To investigate more thoroughly, compute Pass @ and
7

---

Published as a conference paper at ICLR 2025
@ results with random sampling to generate N (N=5) samples with temperature of 0.8 and top-p of 0.95 in Appendix L. While it is encouraged to generate much more (N 2 K) samples to avoid bias, we take the lower bound due to limited computational resources. We use the same prompts for code generation from (Liu et al., 2024), given in Appendix K.
4.1 MEASURING T'SK-LEVER PERFORMANCE
We first evaluate the task-solving performance of each LLM and summarize the findings as follows. We show the main results in Figure 6. As models constantly omit the essential code in the generation and hence fail the tests, we calibrate the generation quality by adding the missing setup and calculate Pass @ 1, which is denoted as calibrated Pass @1 1. The Pearson's i r correlation between the model ranks on BigCodeBench-CompL
is 0.982, indicating a strong alignment. In addition, model rankings show the signs of scaling laws (Kaplan et al., 2020), where bigger models can solve more tasks. We also observe that there is still some performance gap between the top closed models and open ones. Detailed studies can be found in Appendix M. We highlight few findings as follows:
and BigCodeBench-Inste
Passart
Big Biggloleeeench-Comlllte
B
-S4n
-14B
-0B
JB
60
CP T0n.--tube Tar
Cloude 20
Clodlo Coode-attha bateea
Coal--t
Cold 2oe
Code Cole--one-eeee lacck
Combie Code Code-
Code ha-
Megicaee 5.70 w7
Up Tpli-Code--innd 2n
0ot tns
Comslee Comilte-Coonnnn Code tooe
Codet tourp
Comble Cole---ee--i
Dps Upt.coderies
Models
Passol1
BigC Innthheenn-nsttruuct
70B
-34B
-14B
-7B
3:30
Clos Middde
Open Modala
Callbrased Scoee
Gyrann
cFt4
Closke Sppnn
Chaclo Manode-blueeeet Momttet
Chocdl Mttie
Llanaa Foa.aaeeeee
Macoot 200-
Dpple Chat-1
Oweekt 20
Upyle OppleCader CydeCCotrehnnaat he
Cranit Caw.eaCllleeeeeee 3e0
111 11t.Chha-30
Ste Wwf.oorr---e---p
Cronte Colee Codee
Magionder S. 78 Cu Chatee
Dryle Caw.---ae---eaeee 1277
111
VILS-
Crdell Codellaanabbeht
Dysle DydleCaateeteee
Models
Figure 6: @ results of instruction-tuned LLMS on BigCodeBench-Compleer (Top) and SesCodtBench--Inttree (Bottom). We only highlight the calibrated results having at least difference of 1% from the original Pass 1.
8

---

Published as a conference paper at ICLR 2025
Instruction-tuned LLMs omit essential details of long code prompts Interestingly, ob- serve that instruction-tuned LLMs can omit the essential import statements of the given prompts in BesCoddBenchhComp...
which can lead to task failure due to the lack of proper module and constant definitions. The omission is likely to happen when models need to repeat the long context in the response. Such behaviors are denoted as "model laziness" in long-context inter- actions, similar to the observations in Section 2.2. Due to the limited prompt length of existing programming benchmarks (Table 1), there no quantitative evidence of the laziness phenomenon in prior code generation benchmarks. To understand how laziness can affect the model perfor- mance, we calibrate the generation quality by adding the missing setup (e.g., import statements and global constants). When comparing the calibrated Pass@) I and the original ones in the top figure of Figure and Appendix L., we find that GPT-4 tends to omit much more context and perform poorly on B gCode ete, consistent with the previous community discussion' and confirmed by OpenAI (OpenAI, 2024b). While instruction-tunet LLMs have an average perfor- mance degradation of 0.8% on CenCooeeeeccc-oom.... Big BigCodeBench- Inse ruct, which validates the hypothesis that models omit more information of longer inputs.
there is less than 0.3% difference on
Instruction tuning helps to follow programming constraints When comparing instruction- tuned LLMs with their base ones (Appendix L), we observe that instruction tuning can improve the capability of following complex constraints of the prompts. The mean calibrated Pass @ on biigoodeBench-Comllee
for iustruction-tundd tuned LLMs and paired base LLMs are 40.7% and 35.7%, respectively, indicating a great performance gap. The performance disparity is further magnified when the instruction-tuned LLMs are calibrated. We also note that the task performance of CodeQwenl.5- is not enhanced by instruction tuning, possibly due to the lack of fine-grained data during training.
LLMs sensitive to the verbosity of programming instructions From the bottom figure of Figure 6, we notice that LLMs perform much worse on BigCodeeench-Instzee Biggoddeeench-Coplleet with an average decrease of 8.5% on Pass@1 1, while maintaining similar rankings. This observation indicates that LLMs still lack the proper understanding of condensed human requirements since the task instructions of BigCodeBenchh-Inse less verbose. While it is possible that the lower verbosity may introduce more ambiguity, we make sure that the instructions transformed from BiggCodeBench-Come Compeeeepeeeeeee lete do not lose the key information from the human perspective. In addition, we find that models with lower scores on BeigCodeeench-Compl.ee programming capability.
than
are
degrade less on BigCodeBench- Instnchh-Instuut.... limited by their
4.2 MEASURING TOOL-LEVEL PERFORMANCE
Besides evaluating task completion performance, we try to understand how well LLMs can tools for task-solving. As it is hard to evaluate the accuracy of each independent function call regardless of the correctness the model solutions, we rely on the cali- brated greedy decoding results Bi chhh Comples lete and deem the use of the associated li- brary correct if the task is passed by all test cases.
System
Computation
Visualization
64
40
20
oneral
GFT44
Jack of all trades, master of most. Figure shows the top 5 calibrated instruction-tunet LLMs ranked on Big.CoeeBechh-Cooo Cete. lete. The best overall- performing LLMs, like GPT-4o, excel in most do- mains but still fall short in certain ones. We suggest that domain specialization can result from the train- ing data. Similar findings can be found in the base models, which are visualized in Appendix L.2.
Time
Guto
Gaude 3-0ps
0114
Loma) 70o
Cryptography
Network
Figure Top 5 instruction-tume LLMs on BenchcoeBnnch--mmpeette....
"httpss//ccmmet openai wwt.t//t//t/cc// hann/t/why-itte wwwnn-hgtn----- -at--ppt-s----- 1azy

---

Published as a conference paper at ICLR 2025
Table 2: Tool-use comparisons between all generated solutions (Sol.) and ground truths (GT.. on BigCoddeBench-Complee
Library (%)
Function Call
Sol. C. GT.
Sol. ş GT.
Sol. GT.
Sol. & GT.
Pass
79.84
20.16
40.46 22.47
59.54
Fail
71.78
28.22
77.53
LLMs use different function calls to solve tasks. We assess how all 60 calibrated models use tools on BigCoodeBnchh-ommpleeee mean statistics in Table 2. By inspecting the library usage, we find that models use imported libraries in more than 70% of tasks. For the remaining 20%, we notice that models tend to import additional libraries solve the tasks. After some inspection, are standard, explaining why they can still pass the tests. When analyzing the function calls, we find that models tend to use different function calls from the ones ground truths. Due to the open-endedness of the programming tasks, the invocation of function calls is expected to be flexible. However, using more diverse function calls is more likely introduce task failures. We provide several examples where the ground truth Appendix M, Examples 9 and 10.
and report the
notice that most of these added libraries
solution contains function calls different from the
5 RELATED WORK
Large Language Models for Code With the rise of LLMs, there have been various models trained on code. Codex (Chen et al., 2021) marks the first base LLM pre-trained on code, which was used as the backbone model for GitHub Copilot. More recently, pre-trained base code models (Nijkamp al., 2022: Li et al., 2023; Lozhkov et al., 2024; Roziere et al., 2023) have been built to perform accurate code completion. Later, with the advance in instruction tuning (Ouyang et al., 2022), LLMs can generate the code snippet that aligns with the given NL instruction. Instruction-tus tuned code models (Muennighoff et al., 2023; Luo et al., 2023; Wei et al., 2023), are generally better at programming than their base ones.
Programming Benchmarks To assess the programming capability of LLMs, researchers have proposed many programming benchmarks. Most existing benchmarks focus short, self-contained, and algorithm-specifie programming tasks, such as HumanEval (Chen et al., 2021) and MBPP (Austin al., 2021). Recently, open-domain coding benchmarks (Wang et al., 2023c; Lai al., 2023) have been built to challenge code LLMs on application-specific scenarios by using specific sets of tools and libraries. However, they focus on simple intents and use limited specific function calls per programming task, making their evaluation less challenging and realistic. Benchmarks like SWE- bench (Jimenez et al., 2023) are built to evaluate the performance of code agent framework (e.g., iterated prompting, real-time environment interaction, and long-context exploration). Our BigCodeBench focuses on evaluating the fundamental code generation capability of LLMs, which is also an essential path toward strong code agents. In addition. SWE-Bench is constructed from GitHub repositories with existing test cases - which limits the diversity of the tasks considered. In contrast, collaborative LLM-human annotator procedure allows generating tasks from seed queries collected from (Wang et al., 2023c), tackling a broader range of software tasks.
6 CONCLUSION
We introduce 3i gCodeBeenh,, new high-quality programming benchmark constructed via the collaboration between human experts and LLMs, assessing the capability tool use and complex instruction following. Through the extensive evaluation of 60 LLMs, we find that there is a long way for models to achieve perfection on this benchmark and share a few findings that can potentially improve the performance. We urge the community to work more advanced LLMs for code and continually build upon our benchmark and extended BigCodeBench- (aaarrdApppppaa (Appendix F),a discussed in our long-term roadmap (Appendix G).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We truly appreciate BigCode making many great works happen, including The Stack (Lozhkov al., 2024), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023), OctoPack (Muennighoff et al., 2023), Astraios (Zhuo et al., 2024), StarCoder2 (Lozhkov et al., 2024), and StarCoder2- Instruct (BigCode, 2024). BigCodeBench cannot be built without the support of the BigCode community. We thank Sea AI Lab and MASSIVE for providing the computational resources. This research project is supported by Terry Yue Zhuo's CSIRO's Data61 PhD Scholarships, the National Research Foundation, under its Investigatorship Grant SRAF-NRFIO8-2022UU0 Xiaoning Du's Google Research Scholar Program Award. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. For the benchmark, we particularly thank Xiangru Tang, Dmitry Abulkhanov, Noah Ziems, Chengran Yang, Jiamou Sun, Nickil Maveli, and Lili Bo for their eagerness to participate. We are extremely grateful to the EvalPlus team for their open-source evaluation framework. We also thank Loubna Ben Allal, Zhiruo Wang, Zhensu Sun, Sean Hughes, and Zhou Yang for their valuable comments.
and
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877. 2023.
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Kai-Wei Chang. Unified pre-training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655-2668, 2021.
Mistral AI. Mistral. http:://ttssraaaaa///////saaaaas 2024a.
Mistral AL Mixtral-&x22b-v..1.
hetps://huggingfant http://gggggg.......oommmsrtrrla
Mixt ra 1- 2x2a1-002222
Mistral AI. Mixtral-k222b-inssmuxx--01.
http::///uggiingaaee..////ssttt Mixtra - 2111-=22222--sstuttt-00 2024c.
AltMeta. Llama model card. 2024. URL WuRLhttp:///tthttt blob/mainn htt.mmain//// M mdd
llama/aaa/////// /1mmm/3/
Loubna Ben Allal, Raymond Li, Denis Kacetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't reach for the stars! In Deep Learning for Code (DL4C) Workshop, 2023.
Saswat Anand, Edmund K Burke, Tsong Yueh Chen, John Clark, Myra B Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil McMinn, Antonia Bertolino, al. An orchestrated survey of methodologies for automaatdd software test case generation. Journal of systems and software, 86(8):1978 1901-2001, 2013.
AI Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Hemryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint wwX.::22008.073...2. 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
11

---

Published as a conference paper at ICLR 2025
Emily M Bender and Batya Friedman. Data statements for natural language processing: Toward miti- gating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604, 2018.
BigCode. starooder2-15b-iisttucttyy starcoder2-15b- www.iiinsttttti 2024.
httpss//haggiigaccc.co/ooooooo
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 2020.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49(7):3675-361,, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint XIXx:2107700374,, 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang. and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research.
Xinyun Chen, Maxwell Lin, Nathanael Schaerli, and Denny Zhou. Teaching large language models to self-debug. In The 61st Annual Meeting Of The Association For Computational Linguisties, 2023.
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. Pymt5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9052-9065, 2020.
DeepSeek-AL. Deepseck-v2. v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
Kaustubh Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahadiran, Simon Mille, Ashish Shrivastava, Samson Tan, et al. Nl-augmenter: A frame- work for tusk-sensitive natural language augmentation. Northern European Journal of Language Technology, 9(1,, 2023.
Yangruibo Ding. Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36, 2024.
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng. Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for evaluating Ilms on llaes code generation. arXiv preprint arXiv:2308.01 01611, 2023.
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, al. What's in my big data? arXiv preprint Xrx: 10.20077, 2023.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020,
pp. 1536- 2020.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program--tte -ided language models. In International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jemnifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12): 86-92, 2021.
12

---

Published as a conference paper at ICLR 2025
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. In International Conference on Learning Representations.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Wu, YK Li, et al. Deepseek-coded When the large language model meets progranming-the the rise of code intelligence. arXiv preprint arXiv:2401. 14196, 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. Summarizing source code with transferred api knowledge. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 2269-2275, 2018.
Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktaschel. Open-endedness essential for artificial superhuman intelligence. arXiv preprint arXiv:2406.0422. 044668 2024.
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and lon Stoica. R2e: Turning any github repository into programming agent environment. In ICML 2024.
Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E Gonzalez, Koushik Sen, and Ion Stoica. Llm-assisted code cleaning for training accurate code generators. arXiv preprint arXiv:2311.14900 2023.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig. Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001 008361, 2020.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLL,, volume 1, pp. Minneapolis, Minnesota, 2019.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626,50 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 1001:00: natural and reliable benchmark for data science code generation. In International Conference Machine Learning, pp. 2023.
PMLR,
Maxime Lamothe, Yann-Gaël Guéhéneuc, and Weiyi Shang. A systematic review of api evolution literature. ACM Computing Surveys (CSUR), 54(8):1-36, 2021.
Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, LI Jia, Jenny Chim, Qian Liu, et al. Starcoder: may the source be with you! Transactions Machine Learning Research, 2023.
13

---

Published as a conference paper at ICLR 2025
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition--eeee code generation 2022.
with alphacode. Science, 378(6624)
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code
auto-completion systems. In The Twelfth International Conference on Learning Representations.
Renze Lou, Kai Zhang, and Wenpeng Yin. A comprehensive survey on instruction following. arXiv preprint arXiv:2303 10475, 2023.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poiries Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402. 19173, 2024.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round I).
Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp. 643-653, 2014.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2023.
Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza So- ria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al. Gran- itc code models: A family of open foundation models for code intelligence. arXiv preprint M.iiv:2405.0432,, 2024.
Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng. Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, 2023.
Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures. arXiv preprint arXiv:2406.06566 2024.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-tum program synthesis. In The Eleventh International Conference on Learning Representations, 2022.
OpenAl.
Gpt-3.5-turbo.
https: /ettss://ppeai..com comdi//ee// index/
OpenAI. Gpt-4o. http:///ggaaaaaccco/o//////////0
OpenAI.
htttps://ppete new-embedding-mode www.elnn--oooes---eeee balss-and-as and-aai----a-ea apd.a-aaaee-pp---dateeeeeeee/ 2024b.
Gpt-4-turbo.
com/ com/index/ iddx/
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
14

---

Published as a conference paper at ICLR 2025
Frauke Pactsch, Armin Eberlein, and Frank Maurer. Requirements engineering and agile software de- velopment. WET ICE 2003. Proceedings. Twelfth IEFE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 2003., pp. 308-313. IEEE, 2003.
Arjun Panickssery, Samuel R Bowman, and Shi Feng. Lim evaluators recognize and favor their own generations. arXiv preprint 1101::2004.137766 2024.
Helmut Partsch. Specification and transformation of programs: a formal approach to software development, Springer Science & Business Media, 2012.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15322 15334, 2023.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang. Chaojun Xiao, Chi Han, al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023.
Alec Radford. Improving language understanding by generative pre-training, 2018.
N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1988.10084 2019.
Martin P Robillard and Robert DeLine. field study of api learning obstacles. Empirical Software Engineering, 16:703-732, 2011.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12500, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingehuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXivv2402.03300, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mohammed I Latif Siddiq. Simantika Dristi, Joy Saha, and Joanna Santos. Quality assessment of prompts used in code generation. arXiv preprint arXiv:2404.100 10155, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08295 2024.
Minyang Tian, Luyu Gao, Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, HAO TONG, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu A Huerta, and Hao Peng. Scicode: research coding benchmark curated by scientists. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: //openreview net forum? ADLaALt
Jiawei Wang. Li Li, Kui Liu, and Haipeng Cai. Exploring how deprecated python library apis are (not) handled. In Proceedings of the 28th acm joint meeting european software engineering conference and symposium on the foundations of software engineering, pp. 233 233-244, 2020.
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engincering, 2024a.
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang. Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. Recode: Robustness evaluation of code gen- eration models. The 61st Annual Meeting Of The Association For Computational Linguistics. 2023a.
15

---

Published as a conference paper at ICLR 2025
Shuai Wang, Liang Ding, Li Shen, Yong Luo, Bo Du, and Dacheng Tao. Oop: Object-oriented programming evaluation benchmark for large language models. arXiv preprint arXiv:2401.0663 2024b.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better I1m agents. In Forty-first International Conference on Machine Learning.
Yue Wang, Weishi Wang, Shafiq Jory, and Steven CH Hoi. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696 8708, 2021.
Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. Codet5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1669-1088, 2023b.
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-basee evaluation for open- domain code generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1271- 2023c.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint arXiv:23 12.122202222 2023.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, 2024.
Karl E Wiegers and Joy Beatty. Software requirements. Pearson Education, 2013.
Zhiheng Xi. Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: survey. arXiv preprint arXiv:2309.0744 0.844, 2023.
Chunqiu Steven Xia and Lingming Zhang. Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. arXiv preprint arXiv:2304.00385 2023.
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, lon Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorille: .s.. berkeley ey. edu/bbuu haatt//los///// 2024.
html,
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang. Tingyu Lin, Weishan Zhao, Li Zhu, Shuiguang Deng, al. Codescope: An execution- based multilingual multitask multidimensional benchmark for evaluating Ilms on code understanding and generation. arXiv preprint X.X.x23311.088588 2023.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024.
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. Codereval: benchmark of pragmatic code generation with generative pre- trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-12, 2024.
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. Cert: Continual pre- pre-training sketches for library-oriented code generation.
Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Wang Yongji, and Jian- fian-Guang Lou. When lan- guage model meets private library. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 277-288, 2022a.
16

---

Published as a conference paper at ICLR 2025
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. Cert: continual pre-training on sketches for library-oriented code generation. arXiv preprint arXiv:2206.0663. 06888, 2022b.
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. Large language models meet n12code: A survey. In Proceedings of the st Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp. 7443-7464, 2023.
Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts. arXiv preprint arXiv:2405.044. .45520. 2024.
Zhaoxu Zhang, Hengcheng Zhu, Ming Wen, Yida Tao, Yepang Liu, and Yingfei Xiong. How do python framework apis evolve? an exploratory study. In 2020 ieee 27th international conference on software analysis, evolution and reengineering (saner), pp. 81-92. IEEE, 2020.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a- aa--de--- with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36, 2024a.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufci Xue, Lei Shen, Zihan Wang. Andi Wang, Yang Li, al. Codegeex: A pre-trained model for code generation with multilingual bench- Proceedings of the 29th ACM SIGKDD Conference on Knowledge
marking on humaneval-x. Discovery and Data Mining, pp. 5673-5684, 2023.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreeerr: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14655. 2024b.
Xin Zhout, Kisub Kim, Bowen Xu, Jiakun Liu, DongGyun Han, and David Lo. The devil is in the tails: How long-tailed code distributions impact large language models. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 40-52. IEEE, 2023.
Qihao Zhu, Qingyuan Liang, Zeyu Sun, Yingfei Xiong, Lu Zhang, and Shengyu Cheng. Gramman55: Grammar-integrated pretrained encoder-decoder neural model for code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1-13, 2024.
Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameter-efffcient instruction tuning code large language models. arXiv preprint 2400:2200.000888. 2024.
17

---

Published as a conference paper at ICLR 2025
APPENDIX
Contents
A Contributions
20
B Datacard
20
C Data Sheet
21
C.I Motivation
21
C.2 Composition/Collectioon
Process/Preprocessingcco Clcaaing/Laaaeeenee and Use
21
C33 Distribution
21
Maintenance
22
D Data Contamination
22
E Extended Related Work
23
F BigCodeBench-Hard
24
G Long-Term Roadmap and Call for Contributions
27
G.1 Limitations
27
G.2 BigCodeBench-OOD
28
G.3 BigCodeBench-Imencc
28
G.4 BigCooeBBech--Bvlvee
29
H Artifacts
29
I Tool Statisties
31
L.1 Analysis
31
1.2 Comparison to Existing Programming Benchmarks
....ea..........................
31
1.3 Version Control
32
1.4 Domain Classification
32
J Detailed Benchmark Construction
34
J.1 Data Synthesis Prompt...
34
J.2 Semi-automatic Program Refactoring and Testing Case Generation
36
J.3 Human Curation Guidelines
38
K Evaluation Setup
41
K.1 Inference
41
K.2 Execution
41
K.3 Prompt Template
41
L Detailed Benchmarking Results and Analysis
43
L.1 Detailed Results
43
1.2 Further Analysis
45
18

---

Published as a conference paper at ICLR 2025
L.3 Prompting Techniques
45
M Qualitative Studies
46
N Unit Test Design
52
Comparison Existing Programming Benchmarks
54
P Evaluation Less-Structured Instructions
54
Q bigcodebench: Evaluation Infrastructure
55
R Development Timeline
55
19

---

Published as a conference paper at ICLR 2025
A CONTRIBUTIONS
Project Leadership
Terry Yue Zhuo
Benchmark Construction Chien Vu, Jenny Chim, Han Hu, Haolan Zhan, Xiaoheng Hong, Wenhao Yu, Niklas Muennighoff, Jean Kaddor, Wen-Ding Li, Junda He, Ming Xu, Zhihan Zhang, Ratnadira Widyasari, Indraneil Paul, Simon Brunner, Imam Nur Bani Yusuf, Thong Hoang, Chen Gong, Armel Zebaze, Prateek Yadav, Terry Yue Zhuo
Experiment
Terry Yue Zhuo, Binyuan Hui
Evaluation Framework
Terry Yue Zhuo, Indraneil Paul, Jiawei Liu, Zhoujun Cheng
Analysis Terry Yue Zhuo, Binyuan Hui, Zhoujun Cheng, Alex Gu, Naman Jain
Paper Writing Terry Yue Zhuo
Presentation Editing Niklas Muennighoff, Indraneil Paul, David Lo, Zijian Wang, Daniel Fried, Binyuan Hui, Qian Liu, Jean Kaddor, Jiawei Liu, Imam Nur Bani Yusuf, Chen Gong
Long-term Advice & Support Daniel Fried, Niklas Muennighoff, Qian Liu, Zijian Wang, Binyuan Hui, Xiaoning Du, David Lo, Jiawei Liu, Harm de Vries, Leandro von Werra
B DATACARD
We follow (Bender & Friedman, 2018) to create the datacard for Bi BigCodeBnnh,, where we tend summarize and centralize all information that might be relevant for the benchmark analysis.
Curation Rationale
This is detailed in Section 2 and Appendix J.
Language Variety Information about our annotators' nationality will not be provided, as the constructed benchmark is hardly related to regional or social dialects. However, we confirm that all communications during the annotation process are in mainstream English (en-US). We note that the first language of annotators is not English, which can introduce some inaccurate expressions to the task prompts in BigCodeBench.
Curators Demographie
The benchmark construction requires the great annotation effort of Cura- tors, who are involved in the process detailed in Section 2. They come from the following population:
Age:
18-25: 25% (5/20)
26-35: 70% (14/20)
36-45: 5% (1/20)
Experience Python Programming (Years):
1-3: 5% (1/20)
3-5: 20% (4/20)
5+: 75% (15/20)
Academic Background:
Bachelor: 5% (1/20)
20

---

Published as a conference paper at ICLR 2025
Master: 20% (4/20)
PhD: 75% (15/20)
Text Characteristics
This is detailed in Section 3.
C DATA SHEET
Besides the provided Datacard, we follow the documentation frameworks provided by (Gebru et al., 2021).
C.I MOTIVATION
C.I.1 FOR WHAT PURPOSE WAS THE DATASET CREATED?
Our dataset aims to provide a thorough assessment of the capability solving programming tasks. Particularly, we focus the challenges and practicability of the tasks, and pinpoint two main characteristics that few benchmarks highlight: (1) Diverse Function Calling: and (2) Complex Instruction Following. This dataset will help stakeholders better understand the fundamental abilities and limitations associated with deploying LLMs.
We believe that there three main expectations of good execution-based programming benchmark:
The benchmark should be easy to use and efficient in evaluating the fundamental capabilities of LLMss. Repository-Ievel benchmarks (e.g.. SWE-bench (Yang et al.)) are not suitable for this purpose.
The benchmark should be practical, covering various programming scenarios. Algorithm- specific benchmarks (e.g.. HumanEval (Chen et al., 2021) and MBPP (Austin al., 2021)) are unsuitable. Domain-specific benchmarks (e.g., DS-1000 (Lai et al., 2023)) are also unsuitable for this purpose.
The benchmark should be challenging, where the tasks require LLMs' strong compositional reasoning capabilities and instruction-followe capabilities. The benchmarks with simple tasks (e.g., ODEX (Wang et al., 2023c)) are unsuitable.
BigCodeBench is the first benchmark that meets all three expectations. is an easy-to-use benchmark that evaluates LLMs with practical and challenging programming tasks, accompanied by an end-to-end evaluation framework Bi gCodeBench. We aim to assess how well LLMs can solve programming tasks in an open-ended setting.
C.2 COMPOSITION/COLLECTOO PRCEESSPEEROCESS//CLAAAAAAAALAEE AND USE
The answers are described our paper as well as the GitHub repository: https: /ttps://github..r b ggodee gww.eeenchheeeeeecccccccccccccce annotation
C.3 DISTRIBUTION
C.3.1 WILL THE DATASET BE DISTRIBUTED TO THIRD PARTIES OUTSIDE OF THE ENTITY (H.G., COMPANY, INSTITUTION, ORGANIZATION) ON BEHALF OF WHICH THE DATASET WAS CREATED?
No. Our dataset will be managed and maintained by the BigCode community (https:/ senttps://ww..r. projjcttorg//)
C.3.2 How WILL THE DATASET BE DISTRIBUTED (E.G., TARBALL ON WEBSITE, API, GITHUB)?
The evaluation dataset will be released to the public, and hosted on Hugging Face.
21

---

Published as a conference paper at ICLR 2025
C.3.3
WHEN WILL THE DATASET BE DISTRIBUTED?
It has been released now.
C.3.4 WILL THE DATASET BE DISTRIBUTED UNDER COPYRIGHT OR OTHER INTELLECTUAL PROPERTY (IP) LICENSE, AND/OR UNDER APPLICABLE TERMS OF USE (ToU)?
Our dataset is distributed under the Apache-2.0 license.
C.4 MAINTENANCE
C.4.1 HOW CAN THE OWWER/CURAAOOR/MAAAAAO OF THE DATASET BE CONTACTED (E.G., EMAIL ADDRESS)?
Please contact Terry Yue Zhuo (terry. zhuo@monash. edu) and the BigCode Project (contactt@iigoodeporoo org), who responsible for maintenance.
C.4.2 WILL THE DATASET BE UPDATED (E.G., TO CORRECT LABELING ERRORS, ADD NEW INSTANCES, DELETE INSTANCES)?
Yes. If we include more tasks or find any errors, we will correct the dataset hosted on Hugging Face and update the results in the leaderboard accordingly. It will be updated on website.
C.4.3 IF OTHERS WANT TO EXTEND/AUGMENT//RIII ON/CONTRIBUTE TO THE DATASET, IS THERE A MECHANISM FOR THEM TO DO so?
For dataset contributions and evaluation modifications, the most efficient way to reach us is via GitHub pull requests. For more questions, contact Terry Yue Zhuo (terry zhuoêmona sh edu) and the BigCode Project (contact wwwpiiioooeoooeeeeeeeeeeee org), who are responsible for maintenance.
D DATA CONTAMINATION
While Bi CCodeBench tasks are constructed from scratch, there are still some concerns regarding the potential data contamination. Therefore, we conduct N-gram contamination experiments on ODEX intents (English) and anonymized archive of Stack Overflow used by StarCoder2 (Lozhkov et al., 2024), which may be correlated to BigCodeBench instructions. We also evaluated StarCoderData (Python) (Li et al., 2023), which has been widely used the code training data for various LLMs. We focus on the overlaps between the Bi Bi gcodeBencch instructions and the queries contained by these data sources, using 10-gram and 13-gram setups (Brown, 2020; Elazar et al., 2023; Shao et al., 2024; Guo et al., 2024; Bai et al., 2023) to indicate potential data contamination. Due to the significant computational resources required, the 10-gram overlap on StarCoderData is timeout and thus omitted.
Table 3: Comparison of N-Gram data across different datasets.
Overlap Percentage (%)
N-Gram
ODEX
Stack Overflow
StarCoderData
13
0.00
0.18
10
0.09
1.49
2.49
As shown in Table 3, the likelihood of our task deseriptions being contaminated by existing data is extremely low. With a stricter 10-gram configuration, no more than 2.5% of BigCodeBench tasks overlapped with the tested data sources.
In addition, we have carefully considered potential future data contamination issues before releasing BigCodeBench. To mitigate this, we have decided to release the data on Hugging Face rather than directly on GitHub, Based past experiences, most contamination stems from the unintentional inclusion GitHub source code, as seen with datasets like HumanEval and MBPP Unlike GitHub, Hugging Face does not support the kind of automated scraping that typically leads contamination,
22

---

Published as a conference paper at ICLR 2025
making BigCodeBench relatively safer from this issue. That said, acknowledge that is impossible to ensure complete privacy of benchmark data. For instance, when closed-souree model APIs are used for inference, companies may collect and use the data for training if it is deemed high-quality. Preventing this would require access to their model weights and the ability to run them locally, which is not feasible in most cases.
E EXTENDED RELATED WORK
Large Language Models for Code With the rise of LLMs, various models have been trained on code. The first series of models, like CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo et al.), are built based on the BERT (Kenton & Toutanova, 2019) encode-only architecture, which specializes code-understanding tasks such as defect detection and clone detection (Iu et al.). As there is an increasing need for automated code completion, the T5-based encoder-decoder architecture has been applied to train code-specific models like PyMT5 (Clement et al., 2020), PLBART (Ahmad et al., 2021), CodeT5 (Wang et al., 2021), CodeT5+ (Wang al., 2023b), and GrammarTS TS (Zhu et al., 2024). While these models are better at generating code and comments, they are not fully optimized. More LLMs for code have become decoder-oni only, following the design of GPT (Radford, 2018). The early decoder-only LLMs for code, trained with left-to-right token prediction, cannot assist developers in inserting code between programs. To fulfill this need, InCoder (Radford, 2018) has been designed to allow code infilling. Subsequently, the combination of left-to-right and fill-n-tthe-middle code generation has been widely adopted in later model development, such as StarCoder (Li et al., 2023), CodeLlama (Roziere et al., 2023), and DeepSeek- -Coder (Guo et al., 2024). However, these models are pre-trained only and are denoted as base models. With advances in instruction tuning (Ouyang et al., 2022), more LLMs for code are further fine-tuned on natural language instructions to better understand user queries. For example, WizardCoder (Luo et al.) and MagiCoder (Wei et al., 2024) are instruction-tung with synthetic data containing diverse instruction-code pairs. Later, OpenCodelnterpreter (Zheng et al., 2024b) developed a multi-turn instruction dataset and achieved better coding performance. More recently, there has been growing interest in agentic programming (Yang et al.). developing prompting systems to enhance the capabilities of performing software engineering tasks, like GitHub issue resolution and code search. However, they agnostic regarding the backbone models. In this work, we aim to assess the fundamental capabilities of LLMs in practical code generation without any customized prompting systems.
Programming Benchmarks To assess the programming capability of L.Mss, researchers have proposed various programming benchmarks. Most existing benchmarks focus on short, self-contained, algorithm-speciff programming tasks, such HvaanEval (Chen et al., 2021) and MBPP (Hendrycks et al., 2021). While there are other programming benchmarks like APPS (Hendrycks et al., 2021) and CodeContests (Li et al., 2022), they are not frequently used due to the lack of an easy-to-ure evaluation infrastructure. Additionally, some benchmarks focus on real-world program-based tool use, such as DS-1000 (Lai et al., 2023), ODEX (Wang et al., 2023c), NumpyEval (Zan et al.), and Dataaal (Zan et al., 2022a). These benchmarks are designed to evaluate how well L.LMs can handle specific libraries and programming scenarios, simulating more realistic and complex coding tasks. However, these benchmarks often focus simple intents and use limited number of specific function calls per programming task, making their evaluations less challenging and realistic. Meanwhile, there has been another line of work on mulliiproorammmigg language code generation. For example, HumanEval-X (Zheng et al., 2023), MultiPL-E (Cassano et al., 2023), and CodeScope (Yan al., 2023) extend Python-only programming benchmarks via code translation, without considering benchmarks addressing language-specife limitations like ClassEval (Du et al., 2023) and OOP (Wang et al., 2024b), their tasks are limited in terms of quantity, complexity, and diversity. Later, benchmarks like RepoBench (Liu et al.), Fvoss (Ding et al., 2024), CoderEval (Yu et al., 2024), and SWE-bench (Yang et al.) are designed to evaluate the performance of code agent framework, which includes iterated prompting, real-time environment interaction, and long-context exploration. Our benchmark focuses on evaluating the fundamental code generation capability of LLMs, which is also an essential path toward strong code agents. Additionally, SWE-bench is constructed from GitHub repositories with existing test cases, which limits the diversity of the tasks considered. In contrast, our collaborative LLM-human annotator procedure allows for generating tasks driven by real-world software engineering requirements, similar to the queries from StackOverflow, tackling broader
downstream scenarios. While there are some
23

---

Published as a conference paper at ICLR 2025
range of software tasks. Furthermore, our benchmark emphasizes some important aspects that have not been well discussed in the programming domain, like open-endedness (Hughes et al., 2024), multi-tool use (Qin et al., 2023), and instruction-following (Lou et al., 2023).
F BIGCoDEBENCH-HARD
Running the full set of BigCoeeBench will be burdensome for common users, especially when evaluating a large model on both Bi gCCodeRench-Comgg eete ete and Bi BigCodeBench- Instteernnc----tutt...... In order to save budgets, serving as a proxy for the full set.
release minimal high-quality subset of BigCodeBench-ffard.
As illustrated in Figure 8, the workflow construct Bi SiigCodeBencc--Haree is mainly inspired by MixEval (Ni et al.. 2024), which utilizes a small number of benchmark samples to align user user-facing evaluation. While MixEval focuses -eneral-domain evaluation and considers only code generation tasks with minimal samples from MBPP and HumanEval, we extend the idea to make code generation evaluation more user-centric. Specifically, follow these steps to create Ri Rendeeennh-hHeeee rd:
stackoverflow Embeddings
slackoverflow
Dot Product
Retrieved Tasks
Sentence Transformers
Library >
Solution Beeriagg
Astrage 6.%
Flltering
# BigCodeBench Embeddings
BigCodeBench
$ BigCodeBench-HHrr
Figure 8: BigCodeBench-Haard construct workflow.
First, we choose an anonymized archive of Stack Overflow that has been preprocessed by the BigCode community. Details of the preprocessing can be found in the StarCoder2 (Lozhkov et al., 2024). The archive contains 10.4 million questions and answers, covering diverse programming languages and topics, making it good source of user queries.
To bridge query source and Bi BigCh we leverage all trained sentence embedding model recommended by the Sentence Transformers documenta- tion (Reimers, 2019). This model, trained on a mixture of text and code data, is suitable for identifying similarities between programming queries and BigCodeBench tasks.
base-v2, a pre-
We use the model to retrieve the most similar tasks for each query in the Stack Overflow archive, ranking them by the dot product between normalized embeddings. Based on manual inspection of the retrieved tasks, we conclude that a similarity score above 0.7 is a good threshold for task selection. By applying this threshold, we obtain 6,895 queries and 626 Bi CCodeBench tasks after deduplication.
We illustrate the alignment between the Stack Overflow queries (Figure 9) and the BigCodeBench tasks (Figure 10). As shown in the figures, both the query and the task prompt revolve around web scraping to extract hyperlinks from web pages, using Python libraries to handle HTTP requests and parse HTML. Both involve interaction with CSV files either read input URLs or write output data. While the specific implementation details differ, the core objective of extracting and handling hyperlink data from web pages is shared aspect, aligning their overall scope closely.
However, the retrieved 626 tasks are still infeasible for evaluation. To improve evaluation efficiency, we further filter the tasks by difficulty. Unlike the construction of MixEval-Hard, we define the following more explainable criteria: (1) Library Usage: Each task in B BigCoeBBench emphasizes the compositional reasoning ability for coding and requires the use of at least two libraries. For BigCodeBen ch-reeeenchaarddd we keep only the tasks that require more than two libraries, challenging the models to choose more diverse function calls as tools to solve the tasks; (2) Solution Length: We set the threshold at 426 tokens, which is the average solution length of the tasks in BigCodeBench. The ground- truth solution provides a reference for the task complexity, and tasks with longer solutions challenging solve; and (3) Solve Rate: We compute the solve rate per task based all
24

---

Published as a conference paper at ICLR 2025
Pythes script inporta srt's from Usst.cw, and aty oncher text und lire Links Feard on coch ar from the coV
reference the erts the cov are BUU coluen
Tree EeGuReT Lhtall Isper:: 33t Secolon
wourt ffiliess
veport requests
scent Listudoer, (1T isefi tuen file real mole
witss
Lostents apprater AM exnh on be tot contents
for: 1rt. contenter
Faze artionnjur" [e]l rest))
papt.
for Lisk uog.
prentcuri, Link Link-toet, Lest, 1.,, tok, aref")
dempezi//uae soople mew.rry-att wrr----r----uu -enus are the art ralume is the rwc
Hee pu www.ww....o. comple www.w//////ee.gr....... par--aatr-rrr/1 Creat tn shww remmpla ceal wny comple ai
Jezign sithwe cuappe cocc Basggh ppc1
beoy1]
hosse wur www. coopple consrior arl beoy S85 INEtI unowei chansla.cmm/soo cam/soo
seeue
Hile
before euch recll /4 bave arede after nach tine fram the cse:
Hoopp 1/w//0 coumpi cont/cca--ae -amour11
Hel rttax counee exueg ti cole-caaaoo caruuaccy ritps Unuhee exoelee c392//00
possiate?
Figure 9: StackOverflow query.
requnste
srtlis parae lngert krtjpon
def tack feect
sitv
"tttpss////ww eocclo..ces
Us Fite air "screps, Jete..eetteoooo Jete.ow's
faartion terapec vnlsage hyperitnic oore woollte 19x file.
Paranatert:
retaa
vedpage scrape.
Pooppa/////oww com.ccccceec..m con file 1otr, optiorati
Unis
let:
mave of urique woolitee Ttrke crrspet fime the wetpage
recueats
srtlih so.ue
Exceples:
Faxi "wwweeeeee
task, Faxc "eareceeragge 1o Mttacc///ewwc wiffereet.ccmm cca"", owr www.rreeerrrre..... www.rrrter..........o.oo Lewsoow
Figure 10: BinCh task prompt.
25

---

Published as a conference paper at ICLR 2025
BigCodeBench (Complete + Instruct)
GPT40:2028-5513
hts.Feeeettaaaee Ina (20ze
OPF-4 Turbe 2028
Cloude-1 2tt.--2----nmmet22000000
12
Hernee 2-thets Llama d-208i Pro- d514
20
Gemini- 1.5 A-flash-AP--052 0514
Claude 2-d-mmee-2222202222
Reffectidennnee 2:278
Autolddip Seab Coder 338 Iestruct
Qwen2-71B-Cher
Hermas2--rr-Laammah 3:200
46
Gamelama-77lkttttrrut
Mastral-
Clasding Har 1o
Model Type
Open 158)
Open (> 358)
125
15.
175
20.0
20.5
25.2
33.5
33.0
32.5
Hard Set
Figure 11: BigCodeBench rankings v.s. BigCodeBench-Harze rankings
the evaluated models on the leaderboard. The solve rate is defined as the number of models that can solve the task divided by the total number of models. Specifically, we deem tasks with a solve rate below 50% as hard tasks.
Through comparison, we notice that the model performance on BigCodeBencha Hard differs signif- icantly from the the imbalanced distribution target domains and a large number of easy tasks in Bi BigCode resulting in slight misalignment between the evaluation and user-facing tasks. For example, GPT- 40-2024-05-13 13 and GPT-4-0613 may be overfitting to the easy tasks in BigCodeBench, leading to low performance Bi BCodeBenchh-Harrd
the full set of B Bech We suggest that these differences arise from
To validate the effectiveness of BigCodeBenchh-iarr.. we use a private Icaderboard, SEAL-Coding curated by Scale AI, as a reference. The SEAL-Coding leaderboard is designed evaluate models on a set of user-facing tasks across various application domains and programming languages. Specifically, SEAL-Coding compares four of the best closed LLMs on Python, with the following rankings: (1) GPT-4-Turbo Preview, (2) Claude 3.5 Sonnet, (3) GPT-4o, and (4) Gemini 1.5 Pro (May 2024). These rankings align with our results based on the average score of the Complete and Instruct splits of wee.CoeeBenchhaareee indicating that B BigCodeBBnch--irrd is more user-centric and challenging for model evaluation.
We encourage the community to BigCodeBench--fard when the budget is limited, and the evaluation needs to be more user-centric, Additionally, we note that BigCodeBench-HBe dynamic by design, depending BigCodeBench-Haaze to keep the evaluation challenging and user-centric.
can be queries and the evaluated models. We can periodically update
For dataset contributions and evaluation modifications, the most efficient way to reach is via GitHub pull requests. For more questions, please contact terry zhuo@monash. edu and contact @
Project, who are responsible for maintenance.
26

---

Published as a conference paper at ICLR 2025
G LONG-TERM ROADMAP AND CALL FOR CONTRIBUTIONS
In this section, we share a long-term roadmap to address the limitations of Bi BigCodeBench, and sustainably build with the community, We believe that program- aided language models (Gao et al., 2023) for task completion and reasoning provide the possibility towards artificial general intelligence. Our goal is to provide the community with the most open, reliable, and scalable evaluations truly understand the fundamental capabilities of LLMs for programming, pinpointing the ways to unleash their power.
G.1 LIMITATIONS
Given the limited time and budget we have to develop the initial benchmark, we have foreseen several limitations and aim to address them step-by-step.
Multilingualism One of the main limitations is that BigCcheBerch is Python-only and cannot be easily extended other programming languages. As the function calls are mostly language- specific, it is hard to find package or library with exactly the same functionalities other than Python. However, given the fact that Python is the most flexible and popular programming language to automate various tasks, Ri cench may fulfill most of the community needs. In the meantime, we still seek efficient approaches to construct Bi geoogeeeccc-hiieee programming tasks with tools in other languages, without much human effort.
Saturation Another potential criticism is that some LLMs can still perform reasonably well BigCodeBench, considering that the best models can only resolve no more than 30% of real-world GitHub issues on SWE-bench. One might indicate that our benchmark is not challenging enough, However, we note that the low performance on SWE-bench is likely due to the under-specified instructions and misaligned test cases". Compared to SWE-bench, we tend to make the programming tasks much less ambiguous and ensure that the authors can pass their own solutions during annotation.
Reliability During the execution- evaluation, we notice that some test cases are flaky (Luo et al., 2014), which results in uncertainty across multiple test runs without any code changes, We have progressively resolved some identified issues, such as missing setup of random states and improper removal of non-existen files. However, the remaining cases are trickier. For example, the socket query can be timed out or refused due to the unstable connection. With that being said, we try our best to make the uncontrollable changes of Pass@1 under 0.6%. We plan to continually enhance the reliability of all test cases, with the help of the community. To maintain the high reproducibility, we host real-time code execution sandbox in the Hugging Face space.
Rigorousness While we achieve high test coverage for the ground-truth solutions in BigCodeBench, does not guarantee that any code generated by LLMs will correctly as- sessed against existing test cases. Previous works like EvalPlus (Liu et al., 2024) have attempted to extend the limited test cases by augmenting the input-output pairs via LLM- and mutation-baset strategies. However, it is challenging adapt EvaPlus to the test harness in BigCodeBench. as the harness only examines the expected program behaviors during the runtime (c.g., mocking tests). Furthermore, the function calls used to pass test cases by LLMs traditional test generation (Anand et al., 2013) methods cover all possible scenarios. Therefore, we still consider LLM--bsed test generation (Wang al., 2024a) promising, but with proper designs. Specifically, a possible approach is to collect all the generated solutions that pass the current test cases and make capable LLM (c.g., GPT-4o) harness with self-refinement in a sandbox environment.
more nondeterministic, making
Generalization One intuitive question is "How well do the models generalize the unseen tools and tasks?" Current Bi BigCodeBench only covers the common libraries and daily programming tasks. It will be more interesting to benchmark models on the programming tasks that use emerging libraries like transforme r and 1 langchain in. Crafting new high-quality programming tasks requires huge effort, as demonstrated in this paper. There two efficient approaches for building a programming benchmark for model generalization. First, we can instruction tune an LLM on
'https:// http://gtttttttt./ http:g/th......oooe.tttoooommmma p/ww:: issues/7
27

---

Published as a conference paper at ICLR 2025
BigCodeBench data with additional information of libraries and function calls. The trained model is expected to generate programming tasks with proper test cases based on the given library or function call details. However, the quality of such data synthesis is unknown, making the practicability questionable. Another way is to replace the function calls and libraries in Bi gCodeRench with the synthetic names, simulating the unseen ones. A similar approach (Zan al., 2022b) has been used for code generation on unknown libraries. Although this method may lose the naturalness of software development, the construction process more controllable and practical.
Evolution Naturally, the libraries can be obsolete or updated (Lamothe et al., 2021), which means that the source code data for model training will constantly evolve. Thus, the models may not memorize function calls from a deprecated library version. This poses a challenge for any tool- dependent programming benchmarks to correctly examine the model capability without periodic updates. Another related concern is the test set contamination issue due to the evolving training data. suggested to have both a public set and private set for better evaluation in a recent blog". For future releases, we aim to perform the benchmark evolution both publicly and privately, and host the private test set internally.
Interaction Recent interests are around the concept of LLMs as Agents (Xi et al., 2023), which deemed as a way towards artificial general intelligence. Specifically, LLMs will be grounded in a less constrained sandbox environment, where they can interact with any given applications, such as the web browser and terminal. The environment can help unlock the capabilities like self- debugging (Chen et al., 2023) and self-reflection (Shinn et al., 2024). We tend to work in this direction and see how well L.MS as Agents can perform on Bi @CodeBennhh.
Diversity There might be some concerns regarding the diversity of Python libraries covered by BigCodeBench. While we agree that Bi BigCcheBench can be extended with more domain- specific libraries like the ones in SciCode (Tian et al., 2024), we believe that the usage of popular libraries can be more aligned with real-world software development and maximally benefit the community. In addition, due the limited number of domain experts participating in our data annotation, they may not have enough knowledge of the latest or domain-specific libraries. Annotating the programming tasks involved with such libraries may result in errors and ambiguities. Furthermore, the emerging libraries are immature and usually propose breaking changes, posing challenges to constructing programming tasks with stable APIs.
G.2 BIGCODEBENCH-OOD
We will construct Bi BigCodeBench-OOD. variant of B BigCodeBench that assesses the general- ization ability on out-of-distribttion (OOD) libraries and programming tasks (Zhout et al., 2023). We note that any features that do not frequently appear can be considered OOD. We tend build BigCodeBench-OOO with two different types of OOD: (1) Completely synthetic features; and (2) Real but long-tail features. The first type can represent the private libraries that are unknown to the models, and the latter one may represent the ones trending in the recent programming practice but have not yet played major roles in the training data.
G.3 CECCODEBENCC-INNEE
To examine the agent-centric programming ability, we will adapt BigCodeBench to an interactive sandbox environment, with a list of applications that can help models program and repair iteratively as humans. Unlike existing programming benchmarks such as HumanEval and MBPP, BigCode covers wide range of tasks that require profound programming knowledge. Without the proper understanding, models are likely to fail in debugging and repair. There are potentially two options for the evaluation. One naive option is to allow models to the backtrace of how they fail in the test cases, which can guide them to repair the bugs with a clear goal. A more challenging setup is to make models fix bugs independently without seeing any backtrace. Capable models are expected to resolve issues by generating their own test cases and inspecting the execution outcomes.
'https:/ ://w://///ww//////////// jasoowwi........................ net www.tettllllloe hbtc///l///eeee eevellll
28

---

Published as a conference paper at ICLR 2025
G.4 BreCosBenci-EvoLvar
To mitigate the effects of API evolution, we consider LLM-based API updates as a promising approach to explore. However, several challenges need be addressed from the software engineering aspect. First, there are many types of API evolution in Python. According to the study on 288 releases of six popular Python libraries (Zhang al., 2020), there are different types of API evolution (e.g.. class removal, method removal, parameter reordering, and field addition). Among them, 11 types are breaking changes, meaning they will directly affect the program's 's behavior. Resolving the API removal requires non-trivial effort, as the evolved APIs may not have any replacement in the future library version. Second, Wang et al. (2020) mention that a significant number deprecated APIs not documented properly, making it hard for library users to mitigate the usage of deprecated APIs. To potentially mitigate all these issues, we suggest combining static program analysis for library versions and multi-tury LLM interaction. Specifically, we utilize static program analysis to compare APIs across versions and make LLMs to identify the potential changes, even they are not officially documented. Then, will apply rule-based methods analyze whether the target programming tasks inside BigCodeBench have deprecated APIs. For the ones to be updated, we will further provide LLMs with the new library documentation and identified API changes, let them propose new implementations for the ground-truth solutions, and revise the deprecated APIs inside test cases. Regarding the cases of API removal, LLMs may need more turns to generate the valid implementation, there is no reference for the replacement. The revision To automatically validate the updated solutions and test cases, we will ground LLMs inside the code sandbox for multi-turn execution.
H ARTIFACTS
29

---

Published as a conference paper at ICLR 2025
Table 4: Artifacts for reproducibility.
Endpsint
Gilbib
hutga hiitti con//bllge jecprojject dedoeeceeeeenee releesees lases 2ag/ hsspa:/ / h/thhhigggaas-..... co/ dat saeta// bodeodd bigeodabanch
Buzzing
Areevation Pratesae
hiSpa / /tt.bbb..bb...ooo com bulquede peojjctt/ bagoodaaanndde cnnotatiion
Bullstio Praseasol
Cldbbbb
hiA /oitbbb com luieedde peojuct./ haggodabandd. haspa ptppp usg you juctt Bugecdabanndd
Dowsees for Comparions
/tw.a///aaaatee..... Juunk -i4
OREEX
httpa:/ / com/ zoranmu /dex/tree weattae data
geond.gg
APPS
hetps:/ //gtt/ob. /henddycka
AFfBeexh
hettpa /.gttbbb con cows/shilll ohi eha 11 ggo 1. La wwwee//eeeaaaaee dataa/ dataaap lbottoo
MBFP
sttpp///ggttt con fgoogl ses--essacch ch/ googlee researduute seurcrreed wewee///aaaaeeee. en
NEspa:// (.ggt//ghttcbb www.eeeoottttt reyto
Ln pappaaaaaae mane
Pardueeal
In/eert// hESpa: //gittub...... www.oo /ByCoded /ooo /ea codaz: pwiiaataii /ataa//aaa////aaa/a// http///esaaaceeeeooeooooo www.s_oeeeccaaeeeeeo.o...o gaonk.ga
hetpar/2g11
nunpy-eeaa
TrechittBBval
Modeis for Atollattion
GPT-4-
2124-04-0
GPT-4-Tub
jJK-4-0125ggree greciin
GPT44
apt-4-0613
2p--3 54
Claste- S-teeet
chaste-1-eetet-- 3134422
DecySeck V2 Chot
dos (AP). 32K)
lage
Misial Small
nital 2402
DespSadk Cades Iaelnaaa
h/har//agtasse wwt.c///eddde-eeeeeee dewpseek-ccneee 520 inatoruct
DespSeck Caico67b-lnime
httpa: /tw.g//ggaacs....oo. co de///deee-eeaee a deessesk-ccod codec----- 6. Th- inttract
DespBock Cales Ldb ab-imintt
httpa: //w.ceingssss.....
dassseek--at al/ desosenk- cod-aee---cd----ee 1. inatraet
Coddlana-- -ath Instrct
/chogg ngfaos co/codel
rust Laae- 340-
Coodd Jana-am-1i 130 Insrect
hetpe perpe//2unt /2hggg! ng ho coleodel and/ 1 wa The Then ront-th
Cook lama to-eem
httpes/ /Zhagg ng toe coleadel bod// aoee- TheT ruct -hi
Limm
hiipas /hwchinggaaeeeeooooo
/oaba blana 102 Inakeuec
Ques2 22B lntroct
hi Aattar//hheinng lase Quen/ 725 Instsuct.
Quee2 fnetret
hikppa Bttpee//haaaanggg. ansee geand Oven2 78 instauck
Qwen2-570-A14
httpe://hage niggtng.ts...oo Owwn//en22-77000 55882-578-83365
Q- 1-105--hhi
https:/hhugginge whg.gingtees..... Deen/ Quenn/ 2- Chat
Oweel 5-220---00 Casi
Oweel 1-5228 Clat
Sttpa//haggl.o pyg co/pen//eennn 32a-diat
hetpas/ (app://hgggaae www.aao.coooooooo.oo..ooooo
Yi-
Y-- ni Chut
httpa: happa///hreiggteee
12- Chat
Misnnd 8x238-Iasnes
httpa:/ /tw.hhiggaaas....... /istttcaaa (Xaat -e223
Migicoles
hsspar/ htt.hhhhinnnssee.eo
Surtwder--lnan
Comtte-itt Sill-Date Iratrcc
hetper/ /hhpp! ni hos cortanggaan wew.eeeeeeeeeeeeeee 10p-od team ruct
Comme-2ll Cate-trarnct
herper/ /hhugg nig teg grannggantte www.teeeggeeeeeeeeeeeee 2tw--te--t--co----------
hiipas /happinggane
grank co-eode instruct
CodeLlana TUB Base
hilpur/ /uu slan
Codellama 145 Buse
clane 340-ic
Codkilama-
httpa:/ /ttgt///hhgaaat
Librm- 3010--------- Hise
hetpa /ttet///uuaaaae
blara------------- 1-10
Llone 3 Buse
httpas/ /togg///hgggnnm aao
Decplees Ooden 1300-331 338
nttpas/ /hoggl ng faoe co/deegeed--al //degpeek--ooo cooopeee--oo------
3tb-base
httpas/ /choggl ng co/deepeeee--aa
DeppSerk Cidenbee-1 38
3t-bade
CookOwenl.S .b-7b
httpa: //w.reingfass
Cwen//odaaaeee 2-78
15-15 34B
haspa:/ httpa://hurs /hw.c//agaaaaa......o co/01-1000000000 001-1//22
hsspa:/ htwpa///whhiiinaiseccoom
Miboral--c221--n Bloo
httpe:/
attpe://hagy nigh htt.:////oogeee/ceeeoeeeo 2ooogg///coohoeeaa0000
CockGGemma-11
httpe://hugg. www.gningnne.....oo.... co//googee/ccooeeeeee
158
www.ilooo////aoooooooooooo wtw.od//aaaaaaooooooooo 2odu2-11
StrCwdar2
StaCoda22 3B
Hilpuu// Thapppinng
/a
2odur
Graite S4B Code Bese
Hi HLAga:///uu hagp:://ugggingg aoo
grand bib-eode-bba
Graste-2th Cade- Buse
httpa:/ httpa:/hhagginee netteee
c0b-code-bbee
Grante di Codo Haas
hetpa: /ttts/////eieae
30

---

Published as a conference paper at ICLR 2025
1 TOOL STATISTICS
1.1 ANALYSIS
Frequency
426
Short Head
80% + + 20%
Proquency
80% - 20% Short
Long Tail
Frquenzyy
64
Short Head
Long
Long Tail
18
66
19
139
(b) HumanEval
(c) ODEX
(a) BigCodeBench
Frequency
80% 20%
Frequancy
80% 20%
Long liil
Long Tail
14
(d) 1000 (Orig.)
(e) DS-1000
Figure 12: Library density comparisons. We sort the libraries by frequency count, showcasing the long-tail distribution to highlight the broad diversity within BigCodeBench.
Frequency
350
Short Head
80% 20%
Frequency
Short Head
Long Tail
-
Fequency
22
Short Head
|
Long Tail
154
723
(b) HvaanEval
ODEX
(a) BigCodeBench
Frequency
80% - 20%
Short Head
Frequency
Short Head
80% 1 20%
Long
Long
294
156
328
(d) DS-1000 (Orig.)
(e)
DS-1000
Figure 13: Function call density comparisons. We sort function calls by frequency count, showcasing the long-tail distribution highlight the broad diversity within BigCodeBench.
1.2 COMPARISON TO EXISTING PROGRAMMING BENCHMARKS
Table 5: Depth (Complexity Solution Characters) and breath (Diversity Function Calls) compar- isons to existing programming benchmarks in Python.
Benchmark
Complexity
Diversity
APPS (Hendrycks et al., 2021)
352.9
137
DS- DS-1000 (Lai et al., 2023)
138.1
328
ODEX (Wang el al., 2023c)
50.4
230
APIBench (Patil et al., 2023)
77.5
171
MBPP (Austin al., 2021)
181.1
51
NumpyEval (Zan et al., 2022b)
30.1
52
PandasEval (Zan et al., 2022b)
45.8
12
HumanEval (Chen et al., 2021)
180.9
7
TorchDataEval (Zan et al., 2022a)
52.8
6
BigCodeBench
426.0
723
31

---

Published as a conference paper at ICLR 2025
1.3
VERSION CONTROL
pandlasee2...3
acikdt-leazees.33
nelpoo-1ibe3.7.3
scabornes0,23.2
numpy==1.21.2
aurba=@0.55.0
cxypteesstyy-s88.coo
scipy==1_7,22
altke=3,8
Fyts=2023...20mm
nelworizee2.6.3
ata:amodeig==0.14.
1xmi w...3
1lu.13=65.9.5
0jangown4.2.7
selert selleriumee4.15.
2illowee10.3.0
detet:ineeef:
Pytlion ducx==1_1..0
openpyx1==3.1.2
1evershtt/nne002555
3ordringe==@2,.0
1ake:=e20.110
Lemeor/uuve2,2111
9..
-ytesseract==0...11
etarset==3.2.3
Python-dateut-
blacc33=004411
onspythan==2.6.1
Fla.k==3.0.3
2lack_Logi.==+0..3
Flask se0
2lask_wtt-22...1
tol:um==0.16.0
geopy==2.4,1
scras-22.11.0
itbrosa=m0,20.11
p:0ttycable=3.13.00
pysryp:oddmm=-5.11.11
python_ntts_cllcatt"
Rcquests==2.3..0
requests_
rsam=0_$
sendgrid==6.1..0
=oxctable=1.7.0
Merszeeg==3.0.22
WIFo:maen3.2.2
x1.d-22.0.1
x1w:==1,3.0
am1::dd:ttim013.3
gensin=m4.3.2
sympy=#1.12
9ytakots=$.4.1
=ex:blop==0.10.0
docxtp1=m0.11.5
sto::modelo-99.14.0
9yquszye=1.4.3
3olidays=o0.25
scistt-tmage=0.18..
nataorgew7,2.11
gcopandas=0.15.2
x1.d-2.0.11
py1msL=e8.2.0
six:podia==2.4.0
1.4 DOMAIN CLASSIFICATION
"Cryptats "Cyypoogragly', 5 "pritt rVianal cant "artay"; *Gemerals "haswSd"; "cyploggragiy" "gineael the ogryptogrephy"" "bisect tomneralf,
"blakej*; "Crystography", "eetwork",
32

---

Published as a conference paper at ICLR 2025
"ealondaz"; "rine", "caifs "Katwosk", "chardet"; "Metwork", "onath": "Computatioon,
"codoca"; "ceystograshy", "collcstians"; "Goncral", "ccyptooraphy*s "Cryptooraphy", "cay": "Systen", MoLypea*; "Byalen", "datetine"; rcine", "datoutii"; "Time", "diiflib"; nGeneral", "djanea"; "Kecwork", "dosx"; "Systen", "enall"; "Heiwork", "fakez"; "Ganoral", "Elask"; "Hetwork", "Elask_ioqin": "Hacwosk", "Flask_maii"; "Notvork", "Elask_ "Heework" "fanatch": "deneral",
"Coliun": "Visualization',
"Functoulls" " "geapy": "s "Heiweck", "actpass"; "3yaton"" "ylob": "Syales*,
"azip": "Syaten*,
"hashl ils"; "Gryptooraphy"" "Iwapng": "Genesal", "hnac*: "Csystoaraphyy',
"hemi*: "Network"
"hitp"; "Nelwuck", "inportlls"; "Gonara"",
Minspect :; "General", "lulf "Systen",
"ipaddreas"; "Kttorrk "icertonls's "Gneerll", "jonn": "Syalwa",
"beraa":: "Computat
"libross"; "Corgutat "logging": "ayalen*, "1 "; "Network",
"math": "Comutattion
"ratplatlib"s iiiguniizattion, "mechani ze*; "Network", teinmi-ypes' "Nebwork", "multiprrceessin"" "Syeten", "alti*; "Corgutation"" "mungyy" "Comuuaaiion" "openpyxi" "Systen", "operstor"; "General", "aotts "Systen", "pandas"; "Computation", "pathl iis; "Syster", "pickie": "Systen", "pegutil"; "General", fprattfora": *dysten", "pretcytaale" "Generai", "psutil"; "Systen", "sytosseract": "Computation", "pytz": "Time", "queue"; "General", "randon": "Concral", "re"t: "Genersi", "reqiesta"; "Hetwork", "rsat: "Coyttograpy", "scipy": "Computation",
"searcts"; "Cryptography", "select"; "Syeten", "scaderid": "Hozwork", "shutil*: "Systen", "scles:n": "Corputation", "sntplib": "Network", "socket": "Hetwork*, "sound:lle"; "Computation", "selica3"; "Syston*"
"sclf: "Ketsork", "statiatias"; "Computation", "scatemodala": *computation", "sczing": "Cenerai*, "s:zuct"; *Systen", "Systom", "sys*: "Eysten",
"tazfile"; "Systen*, "tonsocflav"; "Corpetation", "Lestcablat: "oaneral", "Lostwsap": *Ganeral", "throading"; "Systen", "Lime": "Tine",
"turtia": w/iusiltatton", "typos": "dcacrai",
33

---

"Network", Published as a conference paper at ICLR 2025
"untoodedata": "duneral", "urllis": "Network*, "ucid"; "General", "waznlags": "Ganeral", "weckzoug"; "Hatwork", "wozdninja"; "Computation", "acCorna"; "Network", "xixt"; "Systen", "Betausk",
"Anltodict"; "Netwosk", "yemi": "Systen", "sipfilu"; "Syslen*, "Levenshtein": rcomputation", "ast": "Gonerel", "confiqpacsee" 18ystent, "av2": "Canputatioa", "decinal"; "General", "enun"; "deneral", "Grzno": "Systen" "Elask_
"Iuplib": "Ketwosk",
"gensim": *Computation",
"geopandas": *Congutation iaim" "holldays": "Tine",
"api_caolkita". "Viualliation", "nataori"; "General",
"pyquezy*: "Network", hytthon.htpp.nlai "regez": "Geieeal", "shapely"; "Corputation",
"shlex": "Systenn",
"signal*; "Systenn", "skinagu"; "Conoutation",
"sympy": "Comput ion",
"Lexiblub"; "Compulalion
"eyping*s "Cenersl*,
"ulkipedia"; "Network",
"wordclonä"; ication,, "glib": "Systen", "axpoze*s "8ysten", "buillina iill::n" "General",
"lncelet lef; "Systen", "inp"s *Systen", "docxtpi"; "Systen", "selaniun": "Hetwork", "IPython"; "Cosguial fon', "eilecmp"; "Systen", "ruleid "Ceners)*, "h "Syaten" "obspy": "Computation", "sprint tceneral*, "Miod": "Systen", "argpanse" regpaese"; "General", "torchit: "Computat ton*, "copy": "Generai"
J DETAILED BENCHMARK CONSTRUCTION
J.1 DATA SYNTHESIS PROMPT
Ful simple example,
complex sommarios and Invoke nulciple Python libraties
to solve each problem.
The writien inteat should allgn with a nere specidic and practical scenario, but should still be easy to de Fonelionnll nees asn.
function wish the cessitton intent. terminal-base input-ourpuc exanples The Function sloald costain complex logic like if wlaw atasenenta and loupe. You have to use noze than three Python libraries for acanario. Brite inports and variable definitions furesian.
For acenario,
single
Please includa sequirencats
function docstring.
Tay La avoid using web
if possible.
If chere are any ceastants (e.g. atrings and namaric values) used in the functions, you need to dectare
then beforw the funct ion.
provide sanple Tzy securn valuss corractness assertion.
data
you
comment.
Tach programning scenaric and intent should be segarated by the agecial boken "GPT_OORX__NAE RBRAK
Generaae exaeples
("Lask_id" 4550369, "prompi"; *def (_650009 ", "suffixts "osmonsee's solut
fatalinetion nos (pyberace) "Last "Inisport pylzlaiaport Limelafron datst ine ingues datatine, cinezore in check leandidste) "test"; (*)n assert (omeeistte)) datet 11970, 1. 11. ( stosel swconds tlne.tine 1'n"), "entry-point 1E_45300505,
replace fexinfost
Inteat"; "gel, value dateLine, Loday () in tine"]]
zonaf, "libbary": Platetiims, "pyle", 1
Scenario 1: pandes, syts, ins, randon, netpletiin
34

---

Published as a conference paper at ICLR 2025
""python
impozt pendas z4 pd
smport pytz
fzon datecine Laport datatine
from random impeet
impozt metplot.ia.pyppi. as pi:
Constants
('Nea York', "aaddan (Bwi Hing", "Takyo", "Bydnwy"]
CEADHER_CONDDTICNS [ ('Sunny", roloudy", "Bainyfy "Snowy", (Bcorny']
aities
TINEZCNES
"Nox Yorkf; "Mnrricz/Hew /ookk//
"London's Sarepe/Lomdon""
Ling "Asia/Bhanchal
"Tokyofs (Aaia/Tokys"
"Bydney's : Mustzalla/Sydine
Generate seport of weather conditionn for 1iat of cillex acrosa various tina (U2C).
Paramelwes:
itc_datetine idatetine): The datetine in UTC.
beluenes
DacaPrane: pandas DataPrana with veather conditions for the cities.
Iequiremente
pytz
dateLine
nstplot l.b.otlbbbppp....... piblpy
Exangles
intettine 15, txinfompytx.0ne)
repozt generalu,wwatheessppee.. (uteetttitttttimm
330 print (regort)
393 report (f ('eatber Condi Lion' wwaloe_pounta C1 plot bind""
report date []]
city CITIES:
city pytx. sitts-iineoon (MIM0O leityll
city_tlne dat www..ettiiasseeeo Llawsone (cily_ix)
weather KEATHER CONDITIONS sancint 10, ler (MEATHER_CONDDTTTEEE CONDIT:ONS)-11
regort_data sarrtt_dtttagaaee (leity, city_tine, weather))
zeport dt Dd. Ireport data, columre-("Cityy,, "Locel "ime", "Meather Condition'
1GP__DDEX_
Scenario 2:
pytz, datetime, aumpy, ">python
sngort pytr
fzon datetine import datetimo
sngost nanpy a
From dateutil 1pareer inport paree
Constants
wp..rray (1992, 1974, 1976,
1981, 1982, 1983, 1985, 1988, 1990, 1993, 1990, 1997,
1999, 2006, 2009, 2012, 2015, 2016, 202011
#son_tz, to_t21:
Calculate the total seconds that have passed since given datoring from the currect tine timosones considering
Farancters:
sate_str (stcr: The date string in "yyy--m-dd haimm:so" fornat. date string. tine ocaverted.
fronce2 (stz): The tinazone
to_tz (stel: timezone
Recurns:
seconds.
Requirenents:
datetimo
pumpy
catcurtl.parser
Example:
wtw___ccond___aac___oo (11900001011 00:03:03', (ogk")
35

---

Published as a conference paper at ICLR 2025
fronLtz tyta.tinegonssttonn. to_tz pytz.timezona (to_tz)
giver_data
cursont_cate datetine.naw .ascimezone (to_tz)
tocal_seconda jeusroni__ate given_date). tocaa_
leag_years SECONDE leg.degicei
given_datt..jess
[Vaar]]
1cap_Focenda len (1cap_y3ors)
tocal_soconda seap_seconds
swcurn int [ooa [Lotal_seeee secenda))
Above the iilastration.
Generabe complex
Following simple exampee:
J.2
SEMI-AUTOMATIC PROGRAM REFACTORING AND TESTING CASE GENERATION
J.2.1
PROGRAMMING TASK CLASSIFICATION PROMPT
given progrem:
sot
CSV
DataFrames
JSCN
XME
1TML
Image
Tex:
Built-in Data St recture
Analysis
Networking
Processing
visusalization
File Storage
Eccrypcion
You should output the suitable Labels in list tornst, auch "CSV", "DataFranee" I.
J.2.2 GUIDELINES
Annotation Guidsline:
You are aiven function inside "function.pp", The qoal to: refine the ruaction iaclucing its cocstrings in order to make the runction more realistic and less + arbiguous. This
you ile funstion exactly sana functicaslity with given function bodys 2) vrite blackbex unit tosto to casurs the functioral cosrestnese of the given functior. You should also +- nake the funasion
docstring, you should able
teat.
StepliChezk Library Imports
#!#8 Import Statonent
Renove library importa not Inport libraries before the function declaration.
code.
Cleckk if the usage of these Libracies is cessonable. Por esample, if the desuription aska to cumplete (-) functionalaty that can be implemerted without any of these libraries, then the usace of these (-) You
Libraries
Step for that san nake use of the inposted librariex.
ipt on
+4# Stepés Check ringtring
Description
Check if the expresston of the descciplion is clear and concise, IC not, aeed Lo modify Uhe (
desoription to descript ittrripplle must sention the loving five things:
-Funstionallly
Input
Ourput returned
haquirementa
imported Sbbraries/nudater o- examples of the ieput and euspot of the farasion
used
lontion the necessery dat strudiure if the faresion requicea dats menipulation ion..
renove any Libraries/nooules are imported in the funct fon to accomodace the ackbox testing.
caenge
tonctionality
36

---

Published as a conference paper at ICLR 2025
1181 Inpuc Parematars
Check if the fanction takes
you nodify the description and the f
Function make input paranotezs.
#:81 Example
Provide exanpiea the ingut outgut
"bash
5>> [_D1""ol1"")
"hello wosld"
(_D(-1*)
"love you"
bbash
250 in..gi. jpg"""
snodule "matplettile tib.lettbb..yyoots
>>> F_Di 192*)
Check Function Implenentation
Check if the FaneLian implemental iun is worsect. If nut, you need Lo repaie Ube funct fun Inglemettation to nake it correct.
Fanction
any
deseription has nut nent loned any of then, you need to wither leave off arganeat it takes the cafault value, or modify the deseription ion to mection then.
yea and
For example, for the 'ple.hist Ieounts, birserp. prrarrp (len toounta) 11)-0.5, nelethe0.a)" in 10.0%, thee deseription sbould mention the apecific way curgube with "len fcounial) 0.5" and 'awidih =0.8".
Check if the Fanctian has the return values. If rot, you need to modify the funosion fon to make it return to check.
the valuwa for the Leat
data, you storage or variable for data visualisation Por exapple, if the Funstion segultes to ahov plot, you mua relurn the glot (via Axest.
requires
either recurn
+
Funct take path for
If the Functioon requires checking
properties and 'print to show the results, you need propert outputa
nodify fy this furosion
results. The revised facetion should analgenate presxiating return calasa, thereby facilitating sore versa.lle utiliaation in the reat of your code.
Consider this original funct ions
check propert les leriiginn lis))
is_eoply = not bool berigine_lise) print i=*Is Hist enpty? (is_enpty!")
length (origini
peintit" of the list is Llnnghhi")
(rokk_ppopetties ((1, (1)
This funat ion checks two propert les of Hist: whet it's enpty and itx length, Tt chen prints theee results. Howsves, these results be elsemhere in
New, let'x modi fy the functioon to recurn these resules:
def check ee parriiiinll_leee (1ac)
is_empty (orggooiiol)l__11e))) length ientorigtnsi
is_onpyy, longth
enpty, Hist length check properties((), 2, 4)1
print (e"e the List empty? clist_enptyy engthh"]
print (f"llengh
In this moditied version, the tunction returne the tvo propertion instead ot printing then. This allove list_lenysh variabies
the your progzon.
clszwbere
If you retuzn any fornats of values "string"), make sure that you mentica the fornat in the + description.
correctness the function implementation.
** Stopf: Run The Function and Write Blackbox Tast cases
"run_tests!)" that conteing assertion-based blackbox test blackbox (- "test.py" It any of the tollowing sata types are used for maipolaction, you need to manuelly design the datz or utilize 3rd party libraries and vebsites fa.g. 'Faker' and "unittest.mock to genorate or nock the "filos/"* protecol should work corsectly
function is contained in file
'faresien.py'
requized
testing function
EINL protozol. (See https://chat. openai. htt.aa..oo/aarre//sieeeeeeeeeeee
=id4-d8244772c13)
hinl (xuboage, page 1ink)
CBV
15on
wnl
inage
37

---

modity the Published as a conference paper at ICLR 2025
You should test the poszibla at:ributes of that weitten data, For example, if you return plet, yea f
need to tost it the plot contains the correct titie, x-axis, y-axis and data pointe.
fosmalizo test writing, you to the collowing function:
"python
suile tunitcest wtwiittta..naaaaaaa.ee makesuutt (TestCases))
sulto.adcTon
runner unittest. weitteet...x..ettBBieee TestRunnee)))
claps TestCasen (untteast.osttiass.
del Lest (self):
Input 1:
provide series of unis testa to test atcributes the returned pass
del
(swll))
Input 2: waite your Leaz case
Provide a saries of uni: testa to test all stcributes of the returned valcos
test_ (solf)
Toput: 3: wiile your Les- case here.
Provide series of uni: teata to test all atcxibutes of the returned valcea pass
veice moze tasta here
Each blackbox teat nothod be tested with anique Input value assertad seturn valces.
#16 Warking
After reading guidel ine, refine "function.gs and write singgee test progran nared "run_testx()" Lest i which should contain as leass ifiver diflerent inpul, cases.
besting
3rd party libraries and websites genezate nock Lest data. exanple, you ahould design the henl webpage to neet the fonctions foral ity and test scenarios, For any numerie values, you need to regresent then the data.
you le.g. "aker and "unilieat
generate
by yourself
Make the test data as corplex es possible to minia the real-world date, For example, if the funct reguires smading usv file, you reed to design the uav file to neet the funct lunality and Lext scenarios.
You can not renove any libraries or nodulea used original Eunction Hovever, you can add new Libraries modules to the turction.
suze you have tested return values the possible attributes the writton data.
Yeep testing ng the funct for until you are set istisfied wish che function implerentat ion and the test ice casea.
any tested properties
nencioned desoription, you to modi cescription
ment then.
will provide the funet scup for programers to implenent by their shore is any anbiquity, you need
is
ne ambiquizy in the deseription. te modity the cescription and the function rrg-ementation to make clear and ccncise, Think sbout the possible quest ions that programers may asx and try to ansver then in the descrigtion.
function impiementation.
paes blackbox test cases. Otherwise, you function implementation to make it pass all blackbox test cases.
'ruaction.py"
++ Retine "tunction.py" and xrite bLackbox
'test.py'
Note that "akerr" library has already botn instelled in the anvironment and you can use it rreely.
Download rarined 'tunction.py', written "cest.py" and created "test_data.zig" i2 there exists.
J.3 HUMAN CURATION GUIDELINES
Annoaationn Guidel inse
+4 Rrvi conment Setuug
Preguirements.coo 1 please up Pythen can use Anwourde or Python Virtuel Environment for this purpose.
given
Use the follewt lewiing cormand to install all. requiied libraries: trghh
sip install zuguissments..kx
note envizonment will seme fer Code Interpreter). Although it expected that nost APTs are stable, it is safer to use
by OpenAT Advanced Anelyeis
38

---

Published as a conference paper at ICLR 2025
consistent library versions. You are encouzaged to use nore librariea covered the requiraments. Ext to anrich cach: sample.
Annotation Goal
The goal is to:
Jafine the function, including ita docscrings it order to nake the fanction noze realistic and leas ( and doestring, should
you the function exaeily the same functionaliyy with the yiven function body. Add noce librasy APIs meceasasys Weite additional black hox unst zeata to ensuze the functional correctness of the given function. Yea (
enbiquous. This
to implament
should consider nany corner cases as possiale.
Exposted Annctation
You are given Pythun script execute and amnoiate.
define the follewing tesra:
"Progranning Problen' centaina three pazts; [Stardalona] "Import Statement"
The imported librarins, auteodales or funat should be used in the following Function implemental ion. add nissing Libzaries, submodules or functiona if one used but not inparted. "Erobiom Functicn''
Lon Signature" and its curresgunding 'Doostring'
The 'Funetion ion Hanu" should be obfuseated in the fomat of (SUMMERR") to anonynicy. [NUKBER]
should be the one insida the tile nane.
The docstrings (example can be found at Guogle Fyshin BLyle Guide) should comtain: A "Pnnctionality beseription', "
"Funct Peraneters'
*Punet Paraneters Descriptions
"Bunning Examlees' in Python Intmspreter and theis expecled oulputa. "saiction"
The fanction implesentat ion to fulfit the Functionns dexoribed In the "Doostringg, Buile' cuntains theee paris: [Standalona] "Import Bratement" The inported Librarie, suberddules or funct ions should be used in the fol lowing tests. [Standalum]] "DeaLCaswa" class
class should contain least five distinct
cases.
The text cases should be aligned with the docstring descript ion. Text cases should nos exsert any (-) altiibutes whith ase not. apucifically neat, iuned.
The test cases should cover complete poverage, you should
branches in the "Problen Funct on as possible. In order (_0Nee) [NAKR) py
the
the commnn 'coverage coverage zeport "mt. Ruplace the "[_[NDEER] (WANE) .py" with the file name you ase lesting with. Tgnore the nissing iine in the 'Test Suite',
' roggrnning Problen en should be able
all these cest casea. This means the acrapts should run (- succesetully withous any tailed test cases when you run "python xxx.py" the terninal.
[Stardelane] 'run teats' funct
The function should only cont ain the code helping to execace the Lest cases.
Adcressed
You may notice the existence at the tellowing issues in the given Python script: 'Punct ion Yane" hax not been obfuscated. Replace nane
't', 'Docetring" is unciest, anbiguous, impractioal or not well aligned with 'Solution', 'Funct ion "excrigt fon should describe a pract inell fares ionaliiiitty. You should either retine the 'Fenction Descriptionl or "Solution", cheose the more teasible to + done.
Make sure at Teast correct Yonning Reamples' are ircluded. "Solution" does not use all imporsed libreries or APIe.
Try refine the 'Programing Froblen' so that the 'Pinction Sessrigtion' implies the corresponding APT
usage and such It (s) 15 difcicuit to complete, remove the unused inport statements.
are correctly lavoked 'Sclution'.
"salutionn uses APTs that are not included in MMggort Statement
Add the cosrespeading otherwise, retine "Function Description' 50 that ronctiorality will require euch APIe. "talutionn uses less than 2 libraries.
statements it these AFIS necessary corplets the furcrionality. +
You thould recine 'Programing Froblen" so that the sunctionality must require the API usage and invoke ChatGPT (GFI-4) in Advanced Data Anslysie
APIe tron Jeast distinct libraries, You inspiration.
"Solution' uses APIS in 'randon' or the zandon functionality. Iricialize the zanson seed ror esch 'TestCases' to control the behavior. "Soiction' contains
cods.
assed on your understanding, soplace the dunny code with the aztusl implementation of each part. "Solution' contains the display tunetionality, suct at () and patplotlib.pypll sypl... If the dunction requires you to krite or shov some data, you shall either return the data or nake the path for tile tunction requires you snow plet, you most return the plot ivia Axe)), It there attribute insida the object, you should mention it in the 'Docstrira' and test it inside 'TastSases" For exampie, the plet conteine the specitic title or lazel names. You should make sure that chose attributes
function
varianie for data vieualization. For exeplee, it
specitic
aither stated 'Deestring' she sunction roquires chocking sone properties and uses 'print
implied
'Docstrina'.
show resuits, you need medity this furction to return taese reruits. The revised function shouls ansigenate these propertiest with the preoxiating return values, thareby facilitating mere varsazlle urilization of the outputa in the of your cods.
Retes Stop3 in the guidelines of the previous stage.
Global constants before 'Frobien Function'.
the globel coastanta
sample inpata the 'Solution', zenove then and wrice your test
input in "Tes:Cases',
It sho glossl ccastants are unused, remove then directly.
Teat cases Inside "SeatCases" only check the range of retorned zesulta or fail to teat in specific way.
39

---

Published as a conference paper at ICLR 2025
assunes you new the returned results are equal to cestain values, For the returnad objects, validate i2 the attributes are equal to cectair velges.
control "Progranning Problen'. Write the tast casea validate
deterniniacie values expecced
Come with expecced oucpats afcer testing.
"TastCases" uses Libraries or AEIs that are not incladed in 'Import Statement',
Add the cocrespeading import statements if these AFIs are necessazy complete the testing, Otherwise, such APIs.
"aaal" containa Lest casea that
vock for "Sulacion".
Repair these test cases or replace than with better cases. "IestCases" does not test all attributes of the returned objezt, where these attrabutes are implied by compltted 'Salation',
Punction Descriputi incrripiion
Add 1inas 2ode test If chose attributes zre not miationed or implied by 'Functica Deseription', try to dascribe then in 1 Punction Description',
actribaces.
"TastCases"
the files
'Bolation',
Boew files are created daring the esecution 'Programning Problen', recessary Lirex of to teat the actribatea of these files in each Leat case.
"IestCases" is wrapped in "run_tests'.
Regerace
Twat cases "TeatCasea" are dupl icalic or used teat sana belavier. Benova them 12 there is a haea overlap, Replace then with nore complex test cases, Make sure that at least included, live test "DeatCases"
Teat to manually desien data utilize ard parcy libraries and websites (e.a. "Fakez" and Y
nissing.
unit mocck vo genecabe oe nock the Lest data, Reles to Stepé in the guidelines of previcus slage f
Lack relucn value.
Finctiona should have clear racuen values to indicate the reault, and these abould be tasted in the 14
Test
Lack
Cozner cases should considared the
In the cases.
Lack of error handling:
You should add necesnaty checka fuz mull inguba, incuremcl data Lypes, valuea out of expwolte ranges (
input Fornst.
$3 Purcher Beglanat ion of Each Isaue
iFunction Yarw
obfuxcated::
The given function should have generic came auch ensure ancnynity, This prevents the user from inferring the funct ton's purpose based on It's name.
Exanple:
Betoze: "set caiouiate_ average inuns):
Afters 'def [(unn))
'Docstring" is unciest, ano-guous, impractionl well aligned with 'Solution's Functions ion's docstring should provide Inguts, oulpuca, and exapless of uxage. description vague or dowan' match the functian's ianf' benavior, st can less to contucior,
concise desoription of ite purpose, expected
Reanple:: le:
Betoze: inexcalculates sonething. sne.
Acters wergalculster the average of a liet of rumbers,
'Solution" does inported not
imperted libreries APIe: the between the prob' en descrigtion for and the solution,
indicates
Exemple:
Before: 'import noth! (but usage 'mata' in the function)
Miter:
Renove 'import math" or choure itts
functica.
'Sa "Salution" uses APTs that are not included ir 'Irport Statement's
Mil extornai librarios sunctions
in the solution should imported beginning the cuns without errore.
script to ensure
Exanpie ee
Is using "squt" rron "nath" library in tae sunction, casure 'tron nath ingort squt is preseat at beyinning.
'Solution" does not use any library AEle:
The proalem should designed in way that requizes the usege et library APIS to solve it, the chailenge intecrating external tools. Exemple:
Ie the proalem is to caloulate the square root, the solution snould leverage the 'nath.sgrt" functica.
'Solution" uses
"rancon', but does
randon
Patemetera";
Whon using randen functionaiities, for zeproductbllsty, it's good practice to silow the user to sec seed.
Suamole:
Bafoze: "sandon. randint (1)
Atter:
'randon, seed seed (seed) randem.condint (1,10
'Solution' ccatains sumny
Placeholder dunny code sheuls replaced with sctual implementation to eneure the sunction as expected.
Example:
Bafores '4 2000: Isplement chis'
Aftez: Actuel implonentarion
zequired loçic.
Unused glchal constants before 'Prolian Fonction":
constants variables
used the solution should ranoved code.
40

---

Published as a conference paper at ICLR 2025
"TeatCases" usea libraries
that are nos included "Impert Beacemen:': Sinilar to the solution, all external libraries or funations used in the test cases should be imported.
"Jestcases' conteins test cases do not Mil test cases thould be alignes with the furction's behavior to ensuze they test the sunction cerrectly.
'Bolution":
11. "aaal does Leat abiributes of rebuzned eajest:
I2 the function retorns an ab jest with multiple actribates ar nothods, the test cases should f-2 validate all of then to ensuze complete coverage, For example, sher plocting data on a graph, you This jaject
label, y-labal, "AawsSulpiet" jees data soiata themselves. You should required in the functionclity.
alizibutes, like these attribuzes it they
12. "TestCases'
tha filas that result in "Solution": the funct ian creates or nodifies files, the text cases should validate thwse files to ensure function woska as expected.
"aset urapped in 'sun_imstx"" The cases the Cunasion zun then should aeparated clarity.
14. Text cases in "Teat Cases" are dupl iupliceted used to best the sene belaviorr suite conciae focused.
Bedundant
should be zenoved
15. Text data used in "asst is nissings
All requized data Zoz teating should be provided genezated enauze the test casea can ran
K EVALUATION SETUP
K.1 INFERENCE
We perform all the model inference A100 GPUs, except for the closed ones. For the closed models, we rely on their official APIs provided in the documents.
K.2 EXECUTION
We conduct the execution mainly on the Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz, composed of 2 sockets, with 18 cores per socket.
K.3 PROMPT TEMPLATE
Please provide self-contained Python script that solves the following problem in markdown code
block:
(prompt) }
Figure 14: Prompt template for models supported by vLLM (Kwon al., 2023).
41

---

Published as a conference paper at ICLR 2025
Please generate self-contained code complete the following problem: (prompt) }
Figure 15: Prompt template for OpenAl/DeepSSeek APIs.
Please generate self-contained code to solve the following problem Python markdown block: (prompt)
Figure 16: Prompt template for Mistral model APIs.
Please generate self-contained code to complete the following problem wrapped in Python mark- down block: (prompt.)
Figure 17: Prompt template for Anthropic model APIs.
42

---

Published as a conference paper at ICLR 2025
L DETAILED BENCHMARKING RESULTS AND ANALYSIS
L.1 DETAILED RESULTS
Table 6: Evaluating LLMs BigCodeBench--Cmm eete. Due the training flaws in StarCoder2 and Granite-Code, we additionally strip the trailing newlines for model inference.
Greedy Decoding Calibrnted
Random Sampling Pazali
Model
Size/ / Checkpoint
Onginal
Poaeit
Iustraciion-tuned I.LMs
GPT-4o (OpenAI, 202
2021--0--3
0.602
0.611
0.557
0.711
GPr---Turbo (OpenAl, 202/bb
2021-01-09
0,582
0.582
0.563
0.699
GPT-4 (Achiam 2023)
0.572
0.407
Tirrr (OpenAI, 2023)
0.504
Clande (Animopic, 2024)
Opus
0.568
0.574
Sonnet
0.534
0.538
Haáku
Mistral (Al, 2024a)
Large
0.379
0.383
0.365
0.539
Small
0,390
0.413
0.398
0.601
DeepSock-Chut (DeepSock-AA,
0.482
Inder 2024)
0.491
6.78
0.432
0.438
0.426
0.624
1.38
0,293
0.296
0.270
0.468
(Roziere al., 2023)
34B
0.351
0.356
0.266
0.518
13B
0.297
0.517
0,247
0.470
7B
0.254
0.257
0.193
0.403
Llama3-instnat 2024)
8B
0.368
0.369
0.338
0.562
Qwen2-Instnat (Rai et al., 2023)
72B
0.539
0.540
0.489
0.682
57B-A14B
0.468
0.399
0.648
0.407
0.21
0.329
Codeeqeenl 5-Chtt (Bai et al., 2023)
7B
0.437
0.443
0.408
0.632
Owenl.5 5hhtt (Bai 2023)
110B
0.432
0.444
0.425
0.590
0.373
32B
0.410
0,420
0.375
0.587
Yi-I.5-Chat (Young et al., 2024)
34B
0.433
0.428
0.411
0.622
9B
0.422
0.424
0.379
0.601
0.290
Mixtral-instrect (Al, 2024c)
Mx228
0.496
0.502
0.45%
0.677
Magiceder-S S DS (Wei al. 2023)
6.7B
0.475
0.476
0.421
0.643
CodeGemma
StarCader2 Caderne22nnnmmee (RigCode, 2024)
13B
0.437
0,411
0.404
0.610
Granite- Ceenlte-Code-tie Inde--nssnner (Mishra et al., 2024)
348
0.438
0.444
0.399
0.613
0.367
0.581
3B
0.315
0.315
0.250
01466
Bose LLMs
CodeLlama (Roziere al., 2023)
34B
0.371
0.297
0.570
13B
0.320
0.243
0.527
0.287
0.207
lamea--hsee Meta, 2024)
70B
0.375
8B
0.288
0.223
0.466
DoepSock- ConpSeck-Ccckr-uue Cuder-buee (Guo ol., 2024)
33B
0.466
0.401
0.661
0.343
L.3
0.223
0.174
0.412
CodeQwenl.5 5 (Bai et al., 2023)
7B
0.156
0.411
0.650
YI-1.5 (Young
348
0.302
0.575
6B
0.274
0.201
0.435
Mistral-huse (AI, 20244)
Mx22B
0.455
0.355
0.633
CodoCiemma (Team
0.295
0.239
0.153
0.375
StarCoder2 (Lozhkov et al., 2024)
15B
0.383
0.332
0.609
0.285
0.232
0.514
Gramite Code (Mistrn al., 2024)
34B
0.385
0.333
0.582
20R
0.258
0.286
0L552
8B
0.356
0.287
0.536
0.177
0.406
43

---

Published as a conference paper at ICLR 2025
Table 7: Evaluating Instruction-tunee LLMs BigCodeBench-Instruee We report the results of greedy decoding, paired with calibrated results. To understand the performance difference between Bi gCodeBench-CCmpl ete and Reoceeench- ruccr, compute (NNT2C C2C) for each model. We do not include the results of Granite 8B & 3B as they constantly have empty outputs.
Creedy Decoding Original
A(N C2C)
Model
Size / Checkpoint
Calibrated
Original
Calibrated
GPT-4o
2024-05-13
0.499
0.511
-0.103
-0.100
GPT-4-Turho
2024-04-09
0.480
0.482
-0102
-0.100
GPT-4
0613
0.459
0.460
-0.025
-0.112
GPT-3.5-Turbo
0125
0.382
0.391
-0.124
-0.115
Claude-3
Opus
0.452
0.455
-0.116
-0.119
Sonnet
0.425
0.427
-0.109
-0.111
Haiku
0.392
0.394
-0.094
-0.107
DeepScck-Chat
V2
0.404
0.404
+0.090
-0.090
DeepSeek-Codee Contrrinstrcct
0.418
0.420
-0.993
-0.091
6.7B
0.353
0.355
-0.079
-0.083
1.3B
0.227
0.228
-0.066
-0.068
CodeLlama-instruct
70B
0.405
0.407
-0.084
-0.089
0.289
0.290
-0.062
13B
0.282
0.285
-0.015
-0.032
7B
0.218
0.219
-0.036
-0.038
LlamaZ-instrust
0.432
0.436
0.107
-0.109
8B
0.317
0.319
-0.051
-0.050
Qnen2-Instrnct
72B
0,382
0.385
-0.157
40.155
57B-A14B
0.357
0.361
-0.108
-0.060
7B
0.291
CodeQwenl.5-CC
7B
0.393
0.396
-0.044
-0.047
Qwen1.5-Chat
1 110B
0.336
0.350
+0.096
-0.094
72B
0.326
0.332
-0.072
-0.071
Yi-1.5-
0.335
0.339
0.098
01.89
9B
0,343
0.345
-0.079
.0.079
6B
0.255
0.256
00.83
-0.83
Mistral
Large
0.296
0.300
+0.083
-0.083
Small
0.318
0.321
-0.072
-0092
Mixiral
8x22B
0.404
0.406
-0.092
-0.096
Magicoder
6.78
0,360
0.362
-0.115
.0.114
CodeGemma-insttuct
7B
0.321
0.323
-0.062
-0.070
StarCoder2-Insttuuet
15B
0.367
0.376
-0.070
-0.075
Granite-Cod Codte-Ccdee-nstrucct
0.358
0.361
-0.080
-0.083
20B
0.337
0.340
-0.084
-0.081
44

---

LLMs are better at computation, eryptography, and general domains We further investigate the perfor- mance among different domains and find that models tend to be more capa- ble the tools in computation, cryp- tography, and general domains. On the other hand, models frequently fail when the tasks involve network tools. Therefore, we encourage the develop- ment of models specializing in such low-performing domains. A few ex- amples of failures in the network do- main can be found in Appendix M Examples 6, 7, and 8. Published as a conference paper at ICLR 2025
L.2 FURTHER ANALYSIS
Bigger models yield better programming performance The performance on BigCodeRench displays signs of scaling laws, where the models with more parameters generally solve more program- ming tasks. While the scaling laws hold for most instruction-tuned and base LLMs, Mistral-Large ap- pears less capable than Mistral-Sman on both BigCodeBench-Comm Instruct, implying that Mistral-Large may be under-fittee fittirn.iiinn... Our findings are consistent with (Yan al., 2024).
and BigCodeBench-
Closed LLMs perform better than open LLMs We notice that most strong LLMs on BigCodeBench are non non-permissive, led by the models from OpenAl and Anthropic. Among the open LLMs, the best model, caumaa-instruct-N0e ranks 5th on both Bi gggoeeeBnchh-Cong ete and Bi BogCodeRench-Inee permissive LLMs are relatively small and, hence, remain in gap from the non-permissive ones.
slightly outperforms Claude-3-Some and
The most
Jack of all trades, master most Figure 7 shows the top for instruction instruction-tumed ranked on Best overall base LLMs like Dpsk- Coder-Basse 6.7B excel in most domains but still fall short in certain ones. We suggest that the domain-specific specialty will likely result from the training data.
DiggodeeBenc--omplett
System
Computation
Visualization
20
General
Time
Lirma--aas-700
Cryptigraphy
Figure 18: Top 5 base LLMs
L.3 PROMPTING TECUNIIQES
Table 8: Calibrated Pass@1 I comparison of plain prompting and zero-shot CoT prompting on both BigCodeBench and BigCodeBench-llaxd.
Split-Subset
GPT-40-0024-05-13 (%)
Gemini-1.-PPro-APPL-551. (%)
Full-Complete
61.1 59.4
57.5 - 55.9
Full-Instruct
51.1 + 49.5
43.8 44.2 (f)
Hard-Complete
29.1 34.5 (t)
31.1 + 27.7
Hard- Hard-Insttrut
25.0 + 23.0
19.6 - 18.2
We provide some preliminary studies the effectiveness of the zero-shot Mlain-of-thought (CoT) (Kojima et al., 2022) on GPT-4o and Gemini-1.5-Pro by appending nLet's think step by step." to the end of the original prompt. From Table 8, we observe that there is no significant advantage to using zero-shot CoT for performing BigCodeBench tasks, indicating that the prompting technique not sufficiently reliable for complex compositional reasoning. Therefore, conclude that prompt design is very tricky and should not recommended for the official evaluation setup. For our prompt design, followed the same approach used in EvalPlus for consistency.
45

---

Published as a conference paper at ICLR 2025
M QUALITATIVE STUDIES
Our benchmark exposes instances where even the strongest models struggle to use APIs, which may be exposed by simple benchmarks like HumanEval. Here, we show some examples with GPT-4o.
Example 1: In the following example, the model fails to use itertools. cycle correctly, producing an infinite loop.
Ezon collections Laport Countar
sandun
from icercools impoct eycle
Conalaans
SLEKENTS MA,, 1BT, 12r, IDr,
(B5 1I rard
def (1))
First
of the 1ist.
Paranetezg:
1. (1ist)s Lixt of elemenbx that the Fonct fon will procexe.
Returns:
counter Teal Tect Tons, Conn.-rnntrrr : A Erequency counter that maps e elenents From the irput List to thwir f- Eeequenciva in the firat 30 wleneraa of the cycled, shuflled list.
Requi rementss collect iona
randon
Examgle:
193 candon, seed (42)
task_
Counterll' 3, (MMs K6r : 3, ratx 3, KM MATE 3, KM 3, ARfE 3, KDT 1ore 313
Hove the tirst elements the end List 1[3s] 1[:3]
Create a cycled, shottled lsst sycled_ 1ist list (eyale(1:)
1..
Example 2: In this example, the model uses the semantics of instanceof incorrectly, Here, isinstance (j, (int, float)) ) does not actually check if j is an int or float, but rather the type (int, float).
inpoct pandas as pá impost
def tunk_func (data)
Analyze dalaset by caloulating average valuea wach and visualizing corcelation e hoatnap.
Faramctors:
Inunpy.areapy areay each rOx soprosants rocord and each columo represents Iocturo
Recurns:
Luple: tuple
an 'Average" represents zean Anwes The plotlolllb Ases ubject showing the healinap of the corselations.
aataPrane: pandas
each zow.
Roquirements:
pandas
punpy
Raises:
Val ValueRrrors IF the Input deta is not array if contal non-numeric date.
Exampla:
530 date allaaray 2, 3, 4, 5, 6, 7, 81, 18, 7, 6, 5, 4, 3, 2, 1113
293 df, Tune
AAvragee l.bo. ceo.stringgee (irinnninneeeeallee)
4.5
..*
acray
not Isiratanc (data, (list, ap.ndarsay)) or not all (iainataane, (llat, opundascay]1 for d
1:
rwise ValuwBeror (*Toput data is
data contains only nuneric arrey")
not Hall (isinetance (b. Has flost() for j in detei:
46

---

Published as a conference paper at ICLR 2025
ValueBrroz Imapus data contains
cuneric data")
Example 3: Here, the model mishandles the case where the directory does not exist with incorrect of os path. dianm..
impost pickic Lnguet
fane
Func lenane,
Serialize an objecct using pickle and overweite the apacified cile with this sarialized data. Before writing, checks if the direatory exists, oreat it nepessary,
Faranetoza:
filename (stel: : the path of the file to be overxritien with seriallzen lidd date. data tol)) The ubject to swrialise and walle Lo the iile.
Reburnes
bool:
Tzue 1I the operation is successful, Jalas othezvise.
Dequirementas
pickle
Examples
300 result task_ funcc (f ("key's Fval
200 pa ini (sesull)
True
ne.
Ley:
Check it the directory exists di ox.peth.di mane (fi Tename)
If pa.path. exists (di nanan) 4 Greate directory it coes not exist
os.nakedi ra tunane)
Open File with opentfilene ffilenane, *dbs)) as files
t Serialige the ooject and write it to tae tile piokle.durp (data, file)
return True
exoept Reception onn as e: (f occurred: (o1")
In this example, the model misunderstands how the itertools s. zip zip_longest method. The arguments must be iterable, but the written function feeds single values.
Example 4:
Inpost nanpy as Ap inport iterteols
task_func :1ac)
Enzipa 1ist of tuples and caloulates the mear of the nunerid values for pasitica.
The Cunet lon accepts a list of tuples, whare each tuple consista alphanunezic values. It unzips the tuples, and calculates the nean of numeric ignores. values at posicion are non acserio, position is set to be na.nan.
posat using nampy, where
IE the provided Luples have diffee number of untries, nixsing values
treated
Paramelweg:
data_1ist (1ist of tuploa): The data process, atructuzed as 1lat of tuples. Each tuple can alphancreric velues.
Becurns:
Hist: A 1ist of nean values for each nunerig posicion lan the tuples. Non-rumeric poxitions
Ignored.
ampty list
inpus 1ist (data_listi enpty.
Iequizenenta:
numpy
itertools
Exangles
(1B) (fet,
Ce', 601)
[nan, 3.0, 4.01 so0 task Fonell M (1, 'a", 25, Ctat, 3, 51, (fet, 1, -2011 [1.0, 2.0, 1.68866566668666677
Hftt
47

---

Published as a conference paper at ICLR 2025
roturn []
nax (t) for in cata_iist)
data_list Flist (itertoola. (tg.otto........ooooe.... fillvalue-0)) for data_
Example 5: In the following example, the model outputs code that has different encoding for password_hhe hash and sacpcctedd__sswwr_________ hash so that even if the passwords are the same, the response will indicate failed one.
task_tunc (date)
This nathod is designed to handle the authentication process in web application context. input in the of distionary "usernana' 'pasavord' kasesd-encedes, S8A-256 bast. mothed
expected
asthentisates
iala
against proderiaed values (for dononstration, purposes, it chesks 14 the usarname is "adnin" and the 'pasexord"). Bacod authentication result,
apozopziate
EITE raspanse.
Farametors:
(dict): dictionary
"pacoword' keys.
Returns:
attpkesponse indicating the lsgin result. WilppeaponeBalllqueet the data invalid.
Rais
KayErzor, Dnioedebecodser.os, binaseii. ValceBsros input dictionary invalid,
Kales:
I2 the authentication succeas, the returned Etiphesponae should contain "Login accossatul, with 400.
authenticat faila, the retucned Hit Hilplesponee should contain "login failed," status
401.
1- the input data inval id Oiee, pasasord is non-base64, mlastnn keysl, the funatiion fon return Hilpheesunneeeeleequee
costaina (Bad Bequest.
Exangless es:
293 Exon d' d'ange.cont Lsguci singings
settings, contiqured:
sett cnn fongirrre)))
259 data
['uaeinen'' "adniaf, "password"; bass64.b6femcodd (haal) ib. 1' passwordd Lemoode (1)
digest decocs 1)
response Funk__unnddetatt
iaponse.staaaie oodde
and "Logān sucoeasful,; resporie content, deccdde ()
False
'adnia', "passwordf; base64 Sbfueetd (bask:l sha256 / sencode
.digest ()) Gecodde
300 response teak Func
293 zesponse.statann atttosssestattsscooo == 401 and 'Login failed." in zeaponse, content, poonteet..ddeee decoda
Requiremente:
djjegoohets hetp
djengu.comt
base64
hashlib
binssci
ne*
try:
Check it the required keys are present it usernemee
data "penswora! in data: KeyError (Hiisiing
ingut
usernane = datal' datal'usernans" usernane"]
hasset deta de ppessorrd))
Secode the bese6d-encodes pasexord trys
password_haae aisedt.asddecoee paeeword__aaeij Beror, raise VolucError involid basesd cacoding")
except
Predef find credentials for cononatration expected_usereg expected axpectes__asexord pasexord_ hash hashlib,shazbil (oassooppssssoed vencode (1) .digest
*adnin'
Authenticats cradencials it usernane expected username paesword. hash expected_passwordd pawssorr_sssseeee return SttpResponse LLegin succesatci.', 2tatu5-2001 else:
SttpResponse tailed.', statcs-401)
except IKeyErret, Onieore retura #tcsBespansaBdRRepeee. Reguest.
ValueErcori as c:
48

---

Published as a conference paper at ICLR 2025
This example showcases failure understand the nuances of the wikipedia library. When page page_title title is not found in Wikipedia, verbatim, the wikipedia page method returns
Example 6:
wwwiiionn.iiiiiiii Dis.mbbigattioorrro...... rather than wi wikipedia excepiaaexccpiiiimmm hww.kceeptooss..perrrr....
import xikipedia
from wordeloud import Mordsloud
malglotiis. pyplot
def Lask
F Funceeagg...eee ting
Create word cloud from the
Wikipadia
Parameters:
pato_citla (acz): The title of the Nikipodia page.
Recurna:
matplet ot Ho.aaees a.ee. Asea: The Axes object of the plosted data. 1a None if chere in no vikigedia page with the Lille givwn Ingub.
Regui renents:
viki pkipe
wordsloud. Rordcloud
natplat b.b.ppp b.b.ypple
Example:
200 taak Func Python (progranning language)") nee
Ery:
Fh the Nikigedia page content
pagn_cunimnt wikipedia.page Lille) .coateat
Generate the word cloud
vordcloud = NoraCloud (Mi (Midth-800, height=40C, cockgroani_co.oorrvitti (etera:eepage_comm
word cloud
513 tig. axe plt ottppottt
ax iishow twordal lood, interpo at in bi linesr")
ex.exia (faEet) 2 Ride the
Show plact Oit.chow(C
except wikipedia.cxcop PogeEzror:
Return Nome 1t the page coes not exisst
return Nome
In this Network example, the error actually originates from the regex. For the first example, instead of www. google com, the regex actually extracts www. google com, which leads to errors down the line.
Example 7:
impost 20
smport socket
task_func (nyatring):
Extracta all URLa fsen given aczing, analyzus each URC Lo extract the donain, and retriwves the IP f-2 address each donain.
ParameLurg:
myString (StE): The atrina from which URLS are extracted. The steing should contain valid URLS f stasting
hitpp.
Returns:
diot: diet Jonary with donsins keys and Uneir respecLivn IP addenasns (IDv4) valuex. If ( be the 12 address will
Ieguliementa::
pbrperse
Rai swx:
galeezus 12 the donein be resulved
Exangle:
299 Lune ("Claok Uhase Liokes bilps/ /tkpp///www.e www.://.ww.ooo.......... bt btupps/ /eews oooy""
l'oww.googl evonn's (192.217,12,11 www.pyyt porgon (151-10" 19...992229
Regular expresston Find URLS
tern reccongi Ieffe beffetbtppe htspstt// ////:///""// (N//8//)))))) ("aa pattenn findall (nySlring!
Diet lonary to slorw donein and TP addreas consin_ip_neg
49

---

Published as a conference paper at ICLR 2025
ueia:
Pazso the URL to extract the donain
parsed_uel wti.ib.parco.urippee.. (url) domain parsedurr.natios
Got the address ot the demain
txys
io_address socket. gethortyyaane (donain) excupt soukel.ggeierrez: io_addresa None
Add and adirwss the diet ionary
joaaiinn ip_adiress
swcuen doguin
Example 8: In this example, the model misunderstands the host naming structures. The issue is in the line response requests get "tthtttps://ipinfee io/ ip htoopppdadddssaaeeeeeeeeeeeeeeee jssnesss///"">>"> json"), where the /json should not be present. This leads to hostname not found error.
sngoott: socket
ingort requestt
del func
..**
resclves
of giver
to sotch geclocation intornaticn from the ipiato.io API, The cunction in robust against various conmnn ercors, sech invalio hostrames, network isaues, or problems with the geolocat fon service.
Faraneters:
bost (str): The host to be resolved.
dict: dictionary contoining the IP address and coolocation information it succasstul.
ValucE:roz: 'host' Nono or caрty string.
ConnectionEEroo: It there 1e problen connecting to the geolocation service,
result task_ tonc google.com' c
>3> "ig_address" in result and "gcoiccation" in result True
335 task_ "ek_"tnn!""
recent call lasti:
ValueE:rozt Host moet be non-enpty string.
Requirements:
socket
requosts
...
host:
raise ValucSrror ("Host must be a nea-ompty string.")
ip_addrese sttbtttgthasss (host)
except cokkt.ggaierros
raise ValoeErroz "Iavalld hostnane.")
Ley:
respense requests. (at (Ethttps///
respense.raiee wtt.coc_oor_tttttee.
geolosation
exeapt requests. Roqusttxceptticcc
raise CcanectionErsor (I" comnecting to the goolocation service: (o)7)
didlessss ipladdreess,
"caolocatioa"; geolocation
Example 9: This example shows an example where there is a simple solution using flatten, but the generated solution does this function and therefore gets the function incorrect.
def taak Bo Func diiiensioon xewd=42)
noreria array instris) given dinension Flat List wlenence the natrix.
Integens
Paraneters:
dinension The dimension square created.
integer.
Returns:
tuples tuple cont containiene siiing
numpy arzay the given cimension
integero
Flat of all blenents in natrix,
50

---

Published as a conference paper at ICLR 2025
Requiranents:
nunpy
itertools
Exampie:
>3> natrix, Clat_list (3n (natrix)
93 15:
172 61 21]
183 87 75:1 pa (flat_1iat)
[52, 15, 72, 21, 83, 87, 751
Sel the sandon sead for reproducibility (sdd
Genezate 20 nampy atray 4ith sandon integeza becween and 100 naciix ap. randon. zandint (1, 192, size= idinensior, dinsnsion)
Platten
flst_list 11st litercools.. wtt.rroolss.chaaaa chw.n. Itorbbe (natrin)]
natrix, flat_list
Ground Pruth
def func (dimension, 2ead=42):
rg.randon, (seed)) Ensure reproducible results
dinens
raise ValoeBrzoz ("hee dimension must be positive lataqez*) natrix randen randint (I, 10", xixes xixeeddieeen idinensior, dinenxion)1 fat__is matelx. fentien)) :ol1ist natrix, flat_list
This is another example where the model chooses to use the wrong function. Here, it uses np mean an instead of np nanmean (which the ground truth solution uses). This leads the failure of a test with incomplete data.
Example 10:
impost pondas 25 pd imgort nanpy ap impost itertesls
tack_tunc (dats__1sc=11''")
2.2), (1B', 3.21, Inet, 3, 4.3), (1dr, 5.41, (eat, 3, 6.5011:
Colculate the mean of nunarical valoes each position accoss tuples in 11st. Non-numeric
ignored,
available data. taples almply ignored.
computed
neans that nissing dasa in
DataFrame with one columee names. 'Mean Value' which contains the mean values for all cuple positionc
indox according to schone: "Position 1' xheze Ir an eapty 1ist is passed, then an eпpty DataFrons is returned.
current position.
Farametors:
data_liat (list Laplus)
containing Luplex nixed data types (atring, ins, flust, sto.).
Decauica to t(tar, 2, 2.11, 1rbt, 2, 3.2), (fet, 3, 4.31, Itd', 4, 5.4), (fef, 5, 6.511
DacaPramo: pandas with
position.
Requizenents:
pandas
numpy
Loerlools
399 df taak_Fune O
>>> print (dZ)
Rean Value
Fasition
Nall
Fosition
Pasition
4.3
[ear, <1r, 2.11, 500 dr Funi (datat
21, rott, 3, 4.3), ([fd"), 5.41, Iffy 5, 6.85)
>>> print (dZ)
Nean Value
Fasition
Fosition
Paxition 2
4.3
list:
return pd. PataPrane (oolumeee Neann Velue(])
Transpose the List tuples to get colunrs
transposed List (itertuols. xip_l Lidat, filloaluenp.nan))
neans []
for col Lranspeseds
non-nurezio values
nuneric, values [x for in txiratance Hat, flosts))
51

---

Published as a conference paper at ICLR 2025
nuneric_valuea
macn_valos sp.naar (numeccc_cccc
clse:
ncans.Eppend value)
Czeate the DataFrame
indax Fositiioo
(mezns))
pd. came (meand,, indexe indwa, volumrae Nean Value")
seturn dr
Ground Touth
def funk_func (dat=_ 1, 2,2), (1b6, 2, 3,21, (naty 3, 4.3), (rdr, 4, 5.01, moty &, 6.5031: uncipped__ale 1ist sttterttola.iip.ooee.
List, fillvaluemp.nan))
For column in stxipped_dttta(:
nunees ve valua = [val foz val in colunn iE ininstanos(vall (val, (inky I1aas)11 if numez valuees nean wtw.eedde...aeeeeedde...... mapamammmn (nmmeci
maan_ wwwpass..peeadaii....... (ap.nan)
dE
pd. Dal.Data raame (nean ve culunnse Mean Value'
Indexe (fPostt Cornat in (nean_valoea))?
soturn
N UNIT TEST DESIGN
HumanEval Test: The HumanEval tests only consider input-output assertions, which only work for simple programs without configuration and environment setup. We use several tests below as an example.
(3t,, d reest!
der (
candidate
0.3)
assert candidate ill.0, 2.0,
5.0, 2,21,
assart sandidate!ll.]] 2.0, 5.8, 4.0, 5.01, 0.951 True
ascert candidzte!ll. I11.0, 2.0,
5.5, 4.0, 5.01, 0.8) - False
candidate (ll.ttell1.,,
0.1)
assort candidate [[1.1, 2.2,
3.4,
5.21, 1.0)
assert candidate I11.1, 2.2, 3.1,
4.2, 5.21, 0.5)
False
BigCodeBench Unit Test: We demonstrate the design Bi BigCodeeRenh unit tests as follows, where we mock various scenarios for the network connection and validate the program behaviours. Compared to the ones in HumanEval and other benchmarks like APPS using input-output assertions, unit tests require great human effort to design and cover various settings.
Requi cument Sellip impost unitiest
from unitt mook Ingort hitp.cllent incost
sngort socket
Text:Cazen (anit Festt
Hook the successful connection and
the responze content
&patch huttppll armroouneett rett__.aoonaecooo ent (aelf, noek_coan)
Test the cortent of the responze. nock_cumn, weturnnveelee result task fune www....e..eeepeeee. www.arteeeeaeee......m sead. retursvveee Expected Contwat?
643, //cantent/ppt
self assert Rquel (result, "Fapected Conten:')
Hook the failed connsct fon and assess the arzor handling Epatch sonket #patch ("atch( httppppliin wtr.iiin.....moou artpsounnciioon det test sal, handshake ercor handl inneling mock conn, nack mock_eooket): san Test handl ing 551. handshake errors. ne* aide_
connect
8StReoo ('831 handshake twiledf) with self, self.asee tesss S8t....rr task, Furm badaal coon', , 443, //test/path"))
We further illustrate how the unit test is constructed for the programming tasks related data visualization. the following example, the function task_funa returns both DataFrame and bar chart Axes object for validation. The DataFrame's Category and Va lue columns
52

---

Published as a conference paper at ICLR 2025
tested to ensure they correctly represent the input data. The Axes object is validated for its title ("Category vs Value") and bar properties, including heights and x-axis labels, using a helper method, is bar. The test cases cover diverse inputs with varying data sizes and values, ensuring the function reliably handles both data transformation and visualization requirements. We note that the test case design for the visualization tasks is similar DS-1000 (Lai al., 2023) but with more detailed validation.
regoont pendee es pd smpost matplctlia.pp import 5es2orn 8he
def wwi.tlinttttc..coo. paire):
nee
Create a Pandes DataFrane tron List or pairs and visuslize the data using a bar chazt. The title of the barplet should be set to "Category ve Value!
Foranetors:
List pairs (Hist tuple)s Rech tuple containes
-str: Catcgory nene.
int: Associated value.
Returns:
tuple:
DataPrane: DataFrame with colunns "Category" "Vales'.. natplotiib cisplaying
categorios ve. volues.
Roquiroments:
natplot
seabarn
Example:
1ist_of_ocire I''Fruatst, 3), ("Vegctabies', 9)]
>3> dt, ax task (tt.roooottts
>>> print (dz)
Value
Vegatables
impoct unitiest
TastCases Fustttast.Tattcasell
"*"Iost cases task_tunc function.""
Bacationethed
daf is_barlax, expocted_vvllsss, oxpectee_ccrtsooriees::
extractedd_vslues
bar.qut_height: for ax.patclaa
extract has heicht
tick.cot_toxt
ax.qet_xtickkabola
extrect catecory label
Fox: actual_valun, expect in sig
assert
vxpoc
("ExgeoLed value [especiee
gat. "valuee
toz astual cattal__aterery, expacted_caterpa in aip!
ratt
expected_cataggriee.
assert
actuual zat egory expected ["Expwol ad pat palagocy (esppoteed but got paactuel__atteeee daf test_izse__ (self):
df, [
(*All 491 ("Casaidy"
("Janio", -44)
(*Randy", -25),
("Joshua", ", 851.
Twoting the DeLaFramm selt Sgnert df (fC"tteggry"" -tttolsst ('Allison", "Dassidy", "damiet, "Randy", Moshus"]
selt assertt l (df "walue"). (49, 72, -74, -25, -85))
Twatiing the plat sitle
self asae (ex.gel_L "Category Value")
self.is hert axeax.
expeoled_ wagrwwggrrreee Lege (fAl'isun" "Cassidy", "Janis", "Bandy, "Joahna"], expeated velueg=[49, -74,
Hore
53

---

Published as a conference paper at ICLR 2025
O COMPARISON TO EXISTING PROGRAMMING BENCHMARKS
Table 9: Correlation coefficients measured by Pearson's r and Spearman's 's against existing bench-
marks.
BigCodeBench-Complee BigCodeBench-Instrree
r
P
MBPP-
0.963
0.937
0,926
0.881
BaturalCodeBench
0.757
0.803
0.913
0.857
HumanHval+ 0.849
0.861
0.864
0.894
LiveCodeBench
0.853
0.898
0.778
0.815
We compare model performances on BiggCodeBenchh-mmmppeeeetee against existing benchmarks like HumanEval+ (Liu et al., 2024), LiveCodeBench (Jain et al., 2024). MBPP+ (Liu et al., 2024), and NaturalCodeBBnch (Python- Enihlss)) (Zhang et al., 2024) used for evaluating the coding capabilities of LLMs. We compute the Pearson and Spearman correlation coefficients for the calibrated model ranks on Biigoodeencc--ompplll marks. From Table 9, we observe strong correlations for both coefficients, particularly with MBPP+ showing the highest correlation with BigCodeBench. The correlation between NaturalCodeBenchh and BigCodeBench-Come Compllete is slightly lower, which is expected NaturalCodeBenchh prompts are more similar to Bi weggoooeecchhheeeee ruct. These results suggest that Bi BigCodeBence is well aligned with the mainstream evaluation trend. However, B Bech measures differ- ent aspects from these benchmarks, some models are expected to have various ranks among them. Additionally, we note that models which have saturated HumanEval like GPT-4o still have room for improvement on BigCodeBench in comparison to human performance.
and BigCodeBentoo ch-Inst ruct against these bench-
P EVALUATION ON CUSS-STRUCTURED INSTRUCTIONS
Although Bi - -encedeeBeec---ntrrrcr-e Instrrct targets the conversational setup, the rule-based prompt structure may not fully reflect how the user prompts look. In real-world scenarios, the user prompts can be less structured and more ambiguous. For instance, in the following block, the user may prompt the model differently, with a less-specified task description (as shown in the second block). The such cases, we preliminarily study several representative models on the rephrased Bi BigCodeBench. Hard, where the NL part was rephrased by Llama-3.1-8B Insttrruuut For the rephrasing prompt, we use "Please rephrase the following programming description in clear and concise plain text. Keep the core meaning intact but use different wording. Write in a casual style from user's perspective, without including any task completion markers. Aninø Programming Descriptionln". We set the temperature to 0.2 and top k as 0.95.
shown in the first text
longer describes the title name and the return type. To simulate
Trair a randon forest class fier so perforn the classi fan fan of the rows in detaframm with reagect to the cul amn of interwat plut Uhe has plot of fealure ingueLance of each column in the decaCrame. should 'Feature Inportance Scose', the ylabel "Feaccres" the title Visuallat fixing Inportant Peatares!.
slabel
Suct the fealure ingostances in descending ordez.
Uae the Zeatore impertances the x-axis and the feacure names on the y-axis.
The
should purput
The candon fores: clamifiee Leained the imput data. nstplot belbib lib.xxe Axess The axes object of the plotted data.
mldennn.mdell RandenPered Clanaai :
You should ld write self-conts cont onde starting with:
From skcee mo enaemble ingort Clasaa ingust swaborn as ans smport matpl ot fopploo plt del taak__ func (df, target_cpium))
want to use randon forest fisxifier to predict the class each row datases based on speci fie
col amm. I alao wans to create bar chart that shevs haw ant wach fealuee is in naking those
predictions. "hhe chart should have title, labels for the and and the features should (
Listed the y forest flass.fius and chast itaelf, Please start with:
order
inportant they
outiput abould include the trained randon f
tron scleasn.ensentlle inport PardonforeatCiaeittier import seaborn as ana
54

---

Published as a conference paper at ICLR 2025
impozt nacpiotll3, pyplot pic def fune (dE, torget_coluna)
Table 10: Performance comparison across models for original BigcCodeBench-Inttreee (Original), rephrased, and the differences.
Original Model
Original
Rephrased
A
Swen2.5-Coodr-32B--neee
27.7
8.8
-18.9
1amma-3.1-7BB Instut--stttt
23.6
11.5
-12.1
NP9--4o-mini22224-77- GPT-44-2022-00-13
23.6
10.1
-13.5
25.0
12.8
-12.2
From Table 10, it evident that Beenn2---ooder322BB-meee exhibits a more significant performance drop compared to other models, despite its stronger performance the original BigCodeBench Hard. While it is true that rephrased instructions may introduce additional ambiguity, the results indicate that some models may still lack sufficient robustness to handle less structured and more casual inputs. This highlights a potential area for future improvement in LLM capabilities, particularly in managing variations in natural language.
BIGCODEBENCH: EVALUATION INFRASTRUCTURE
In this section, document the usage of bigcodebench, the evaluation infrastructure for BigCodeBench. We note that the prototype of bigcodeber.h is based on EvalPlus (Liu al., 2024).
geodebennch, eva luate
model met 1 aana/Wetaa-- 12 11am--333 3.1-8h- -hs--nstruut
split [omplete [[naLruok]]
[full
backend [vl ogenel conthrapiillgege inistral
bigeodebench. insgect --datsset [bigrocebench) --eval-resulte capples-sanitizenn rww..es..tpppp.. [--ia- { place]
R DEVELOPMENT TIMELINE
04/2023 05/2023
Project Initiation.
06/2023 07/2023
Benchmark Construction Data Synthesis.
07/2023 11/2023 Case Generation.
Benchmark Construction Semi-automatic Program Refactoring and Testing
12/2023 04/2024 Tool Development.
Benchmark Construction Human Curation; BigCodeBench Evaluation
04/2024 05/2024 ment.
Benchmark Finalization; Experiment; Analysis; Drafting: Code-Eval Develop-
06/2024
Initial BigCodeBench Release.
55


Reviews:
[{"review_id": "cSH8p3nF1C", "reviewer": "Reviewer_9WoB", "summary": "This paper introduces BigCodeBench, a novel code generation benchmark that improves existing ones like HumanEval by incorporating more diverse function calls and complex instructions. BigCodeBench comprises 1,140 Python code generation tasks, necessitating the use of 139 libraries across 7 domains. The dataset was constructed through a semi-automated framework leveraging collaboration between LLMs and human experts. In this process, LLMs are prompted to generate coding tasks based on sampled code snippets, performing both initial generation and refinement of programs and test cases, while human annotators continuously provide feedback to enhance quality. Extensive experimental evaluations reveal that current LLMs struggle with accurately executing complex instructions and function calls, achieving scores of up to 60%, significantly below the human performance of 97%. These results highlight the ongoing challenges and the need for further advancements in the field of code generation.", "questions": "- The Python libraries used in this study are mostly popular ones, likely present in the pretraining data of most LLMs. Future work could explore how LLMs adapt to newer, domain-specific libraries that may not have been included in their training data.\n- Based on the benchmark results, what potential approaches could improve the performance of existing LLMs in handling complex instructions and function calls? Insights on targeted training techniques or model adjustments would be valuable.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper makes a valuable contribution to code generation benchmarks by incorporating diverse function calls and complex instructions, offering a novel perspective on evaluating LLM capabilities in library and tool use. Results indicate that even top-performing LLMs face challenges with these tasks, highlighting opportunities for further advancements.\n- The paper is well-structured and easy to follow. The benchmark curation framework is technically sound, back with a systematic design and thorough quality checks. The extensive evaluation provides some insights into how current LLMs handle library usage and follow complex instructions.", "weaknesses": "- The technical contribution of the benchmark curation framework appears incremental, as the collaborative approach between LLMs and human is already widely adopted in LLM research.\n- While Python libraries are essential for task solutions in this benchmark, the rapid evolution and versioning of these libraries can introduce compatibility issues. The authors should consider methods to mitigate the effects of version incompatibility in the evaluation process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QANgj6yVaG", "reviewer": "Reviewer_yub5", "summary": "This paper introduces a novel benchmark to evaluate the ability of cutting-edge LLMs in function calling and complex instruction following for code generation.", "questions": "- How to prevent future data contamination? \n- How are test cases constructed to evaluate visualization tasks requiring graph outputs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The proposed benchmark introduces a challenging task that involves diverse function calls as tools and complex instruction-following for code generation. Current state-of-the-art LLMs achieve up to 60% Pass@1, while human performance reaches 97%, providing a clear benchmark for future research to improve on.\n- By evaluating a wide range of both open-source and closed-source LLMs on this benchmark, the study establishes a strong foundation and adds credibility to the proposed framework.", "weaknesses": "- There is a risk of future data contamination [1, 2]. To mitigate this, it may be beneficial to release only the validation set while keeping the test set hidden, similar to practices used in NL2SQL benchmarks [3,4].\n\n- Some domains, such as visualization tasks returning graph outputs, would benefit from examples illustrating how the test cases are constructed for evaluation.\n\n[1] Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039–12050, Bangkok, Thailand. Association for Computational Linguistics.\n\n[2] Jain, Naman, et al. \"Livecodebench: Holistic and contamination free evaluation of large language models for code.\" arXiv preprint arXiv:2403.07974 (2024).\n\n[3] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.\n\n[4] Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, & Yongbin Li (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xdutvK24rW", "reviewer": "Reviewer_pm7H", "summary": "The authors present BigCodeBench, a massive high-quality benchmark to test Large Language Models (LLMs) ability to perform diverse function calls and solve complex code instructions. BigCodeBench comprises 1,140 fine-grained Python programming tasks after proper cleaning and filtering and covers functionality from 139 widely adopted programming libraries. Specifically, the benchmark evaluates two common programming scenarios (1) Code completion based on structured docstrings, and (2) the completion of programming tasks based on natural language instructions. The authors used powerful LLMs to source programming tasks, refactor code, and generate test cases, under constant human supervision. This construction process is a remarkable example of a perfect balance between LLM automation and human curation to ensure data quality and reliability. The authors conduct an extensive evaluation of state-of-the-art LLMs on BigCodeBench. Among others, the results show that LLMs are not yet capable of following complex instructions to use function calls precisely.", "questions": "- How do you distill observations such as \"Interestingly, we observe that instruction-tuned LLMs can omit the essential import statements of the given prompts\", do you rely on static analysis of automatic error logs? are humans also involved in the process?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "### Originality\nThe benchmark proves to be original for the following reasons:\n- It contains diverse code, sourced from widely adopted Python libraries from seven domains. \n- It contains more complex programming tasks than other benchmarks. Specifically, the authors report on task cyclomatic complexity compared to other function-level Python programming benchmarks, and a larger coverage of function calls compared to state-of-the-art function calling benchmarks. \n\n### Quality\n\n#### Benchmark Construction\nThe authors followed best practices to select, clean, and filter the seed data. They also integrate the use of LLMs into the process by keeping humans in the loop when needed. Moreover, code execution in a controlled environment is performed to ensure code quality.\n\n#### Validation\n- The authors extensively evaluate 60 state-of-the-art LLMs on BigCodeBench-Complete (code-completion based on docstrings) and 35 instruction-tuned LLMs on BigCodeBench-Instruct (programming tasks based on natural language instructions). \n\n### Clarity\n- The benchmark construction and evaluation processes are remarkably explained.\n- Limitations of the dataset as well as future work are addressed in detail as part of the appendix.\n\n### Significance\nAs BigCodeBench will be open-sourced it represents a significant contribution to the research community. Moreover, the study reveals interesting observations about the ability of LLMs in complex code-completion tasks, which sets a roadmap for the development of new training datasets to improve these capabilities.", "weaknesses": "Please fix the typo in line 182 (refractor -> refactor) to make even better this amazing work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zAXjEhypk", "reviewer": "Reviewer_TG6s", "summary": "This benchmark paper proposes improved execution-based evaluation for solving complex real-time application like code generation problems. It focuses on both the effective instruction query and the detailed test case setup for unit test sets, curated using human-LLM collaboration.", "questions": "- With respect to line 237, what were the reasons or examples of model failure in the dataset that was improved and captured by the annotators? Subsequently, what steps were taken to overcome it?\n- In case of BigCodeBench-Instruct benchmark, each user query is structured well in terms of sequence and format of instruction for code generation. This might not be the case when it comes to user-like query. Analysis on part of robustness to ambiguity in NL, variation in structured format needs to be further analyzed and presented within the paper.\n- Line 300, 'PL: Programming Language', is not used within context of the table and is unnecessary.\n- Table 1: Tool Statistics, # Call column, format the column to have all '/' inline, like the adjacent columns.\n- Perhaps, Figure 6 can be more visually summarized using scatter plot with grid colored based on size of parameter of LLM. Can be updated, if it helps provide better display of story.\n- A definition of what calibrated scores should be added in description.\n- Table 3 comparison can also be done against other datasets like MBPP Plus dataset, NaturalCodeBench and EvalPlus.\n- Add on analysis of how new benchmark overcomes the issue of data contamination due to the benchmarks being used as training data in several LLM. What rephrasing or variation in query structuring is ensured to tackle this?\n- Add in example of failure cases and categorize to why LLM still fails on new benchmark. \n- How well does an expert perform comparatively?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Quite a thorough analysis and structured approach of generation of benchmark is taken into account. With detailed literature review of past works in the field, the paper also demonstrates its effectiveness as an improved code generation benchmark and evaluation.", "weaknesses": "Apart from tiny bits of improvements needed, the paper is quite detailed in its analysis. However, it lacks the essence of what are the areas where these LLM are failing on the new benchmark, and a compared analysis of how an expert human would perform in the same. So as to note the scope of baseline improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)