NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries (2024)

Abstract: Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NATURALCODEBENCH (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. Noting the extraordinary difficulty in creating testing cases for real-world queries, we also introduce a semi-automated pipeline to enhance the efficiency of test case construction. Comparing with manual solutions, it achieves an efficiency increase of more than 4 times. Our systematic experiments on 39 LLMs find that performance gaps on NCB between models with close HumanEval scores could still be significant, indicating a lack of focus on practical code synthesis scenarios or over-specified optimization on HumanEval. On the other hand, even the bestperforming GPT-4 is still far from satisfying on NCB. The evaluation toolkit and development set are available at https://github. com/THUDM/NaturalCodeBench.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 7907–7928 August 11-16, 2024 ©2024 Association for Computational Linguistics
Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NATURALCODEBENCH (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. Noting the extraordinary difficulty in creating testing cases for real-world queries, we also introduce a semi-automated pipeline to enhance the efficiency of test case construction. Comparing with manual solutions, it achieves an efficiency increase of more than 4 times. Our systematic experiments on 39 LLMs find that performance gaps on NCB between models with close HumanEval scores could still be significant, indicating a lack of focus on practical code synthesis scenarios or over-specified optimization on HumanEval. On the other hand, even the bestperforming GPT-4 is still far from satisfying on NCB. The evaluation toolkit and development set are available at https://github. com/THUDM/NaturalCodeBench.

heading: 1 Introduction
text: Large language models (LLMs) pre-trained on extensive open code repositories (Chen et al., 2021; OpenAI et al., 2023; Li et al., 2023a; Chowdhery et al., 2023) have demonstrated impressive performance on code synthesis and even achieve performance comparable to average human level in coding competitions (Li et al., 2022). Unlike open
*SZ, HZ, XL, and QZ contributed equally. †Work done when SZ and ZQ interned at Zhipu AI.
text generation, which often underscores human preferences as noted by (Ouyang et al., 2022), code synthesis prioritizes accuracy and the fulfillment of user intent, essential for practical production and application.
As a result, evaluating code synthesis presents unique challenges in the era of LLMs. Traditional evaluation metrics by token matching (Papineni et al., 2002; Lin, 2004; Popović, 2015) show a weak correlation with human judgement (Evtikhiev et al., 2023) and overlook functional correctness of the generated code (Eghbali and Pradel, 2023; Tran et al., 2019). Recently, execution-based evaluation has gained increasing popularity, where code generated by models is tested through unit tests to verify its functional correctness. It leads to the development of several benchmarks, including HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), MBXP (Athiwaratkun et al., 2023), CodeContests (Li et al., 2022), and DS-1000 (Lai et al., 2023).
Notwithstanding their commendable reliability and accuracy, these benchmarks fall short to sufficiently capture the wide range of needs and complexity found in real-world engineering applications. They are primarily limited to well-defined coding problems in algorithm, program basics, or data science. For example, as shown in Figure 1, a problem from HumanEval (Chen et al., 2021) tests the implementation of a basic function has_close_elements and takes floatingpoint arguments as inputs. However, in practical applications, user engineering requirements can be much more complex and varied. In Figure 1, we showcase an example adapted from a real user query, where the user asks to read and parse XML files given certain tags. Difficult and costly though it is, curating a benchmark composed of such problems is meaningful for evaluating the real user experience of LLM code synthesis.
Contributions. In light of the challenge, we intro-
7907
duce NATURALCODEBENCH (NCB), a challenging application-driven dataset for code synthesis evaluation. NCB is dedicated to creating a reliable evaluation environment that is more aligned with real-world applications. We leverage an CodeGeeX (Zheng et al., 2023b) online services to collect real and diverse application-related user queries. After filtering and reprocessing, 402 high-quality Python and Java problems are compiled, covering 6 domains including software, front-end, system administration, and artificial intelligence, highlighting practical scenarios. Beyond basic data structures like lists and numbers, the test inputs for NCB problems include versatile file types and other complex structures, making it more challenging.
The challenging nature of NCB necessitates significant human labor in its annotation process To improve construction efficiency, we tailor a semi-automated annotation pipeline to curate highquality, testable, and useful queries with corresponding test cases. Specifically, we employ GPT-4 (OpenAI et al., 2023) to generate reference solutions followed by manual correction. Subsequently, GPT-4, guided by the problem descriptions and reference solutions, generates multiple test cases,
which are also refined with manual correction, for each problem. Consequently, the annotators are only required to correct any errors, substantially reducing the time and manpower required. Comparative experiments reveal that our semi-automated pipeline can quadruple the construction speed of the evaluation framework, as evidenced by tests involving programming experts with or without the pipeline.
Based on NCB, we conduct extensive experiments on a variety range of LLMs, encompassing 39 APIs or open models. The results indicate that although certain LLMs demonstrate comparable performance on established benchmarks like HumanEval, they exhibit significant performance disparities when evaluated using NCB. It suggests that there may be inadequate focus on optimizing LLMs for practical coding applications, or have conducted over-specified optimization on HumanEval-style problems. More importantly, even the best-performing GPT-4 only reaches about a pass rate of 53%, demonstrating a large room for LLMs to improve their coding skills to face realworld coding challenges.
To facilitate community research, we pack up
the whole NCB testing environment into a docker image and make its development set publicly available. To sum up our contributions: • We propose NATURALCODEBENCH, a bench-
mark that aligns with real-world applications, comprising 402 problems in Python and Java across 6 domains. We open source 140 problems (70 Python, 70 Java) as the development set of NCB for research purposes, but keep the 262 problems of the test set closed to avoid contamination. • We introduce a semi-automated pipeline for the construction of code synthesis benchmarks, which significantly reduces time and manpower costs without compromising the quality of test cases. Comparative experiments reveal that our semi-automated pipeline can quadruple the construction speed of the evaluation framework • We systematically benchmark the code generation capabilities of 39 LLMs using NCB. Besides quantitative evaluation, we carry out a deep insight into the present stage of development in LLMs for code generation, and outline potential pathways for future progress.

heading: 2 Benchmark Construction
text: The overview of NCB is shown in Figure 2. The pipeline of constructing NCB consists of four steps: 1) collecting and filtering high-quality problems from online services (Section 2.1) 2) constructing a complete evaluation framework through a semi-automated pipeline (Section 2.2) 3) designing prompts to align different models (Section 2.3) 4) translating all problems and instructions to produce bilingual versions (Section 2.4).

heading: 2.1 Problem Selection
text: Collecting Real-World Problems. To establish a meaningful and practical benchmark, we centered on collecting real-world code problems frequently encountered by users. To achieve this, the seed problems of NCB are cleaned from the queries in coding online services. A part of users have granted permission for their data to be utilized exclusively for research purposes. We have strictly adhered to this directive by collecting only the relevant data from these consenting users and have implemented robust de-identification measures to eliminate any possibility of information leakage. We collect a varied collection of queries, spanning multiple programming languages, problem types, and levels of
complexity. This diversity ensures that our benchmark accurately reflects a broad range of code issues users encountering in practice. We specifically concentrated on queries related to Python and Java, chosen for their widespread use in different domains.
Filtering Testable Problems. While it’s possible to source inexhaustible queries from online services, many of these queries posed by users are either of low value or challenging to test the solution of these queries. For instance, some users may only seek basic clarifications on a built-in function, while others may not clearly articulate their objectives. To sieve out unsuitable queries for our testing, we’ve implemented a two-step filtering process. Initially, we employ GPT-3.5 to filter out low-quality queries, which saves on labour. This is achieved by adding specific criteria in the instruction, instructing GPT-3.5 to abandon those problems that cannot meet all specified requirements. These criteria are as follows: 1) Each query must involve at least one task, where the user requests the model’s assistance in solving one or more problems. 2) Each query should be associated with several inputoutput pairs, ensuring that a given input correspond to a singular, definitive output. 3) The query must not contain any elements of randomness or uncertainty. The specifics of the instruction are detailed in (Appendix A). Following this automated prescreening, we conduct a manual review to further refine the selection, adhering to the outlined criteria. This process yields a final set of 201 unique Python and 201 unique Java problems. It is noteworthy that over 80% of the initial queries failed to meet our stringent requirements.

heading: 2.2 Semi-automated Pipeline
text: In this section, we will introduce our semiautomated pipeline. To generate structurally complex and accurate test cases by GPT-4, it is first necessary to determine the arguments and return values of functions, as well as the names of objects. Therefore, a completely accurate reference solution is required initially. We generate a solution using GPT-4, then manually correct all errors. After this, based on the problem description and the reference solution, we instruct GPT-4 to generate multiple test cases. These are then reviewed by programming experts who correct errors and supplement any deficiencies in the generated test cases.
Generating and Rewriting Reference Solution.
GPT-4 is instructed to generate a solution for each problem in NCB. It is important to note that while GPT-4 is highly capable, it is not infallible. Therefore, each solution generated by GPT-4 is meticulously examined by expert programmers to ensure correctness. In cases where the generated code contains errors, the expert programmers rewrite the code to rectify these issues. This process ensures the quality of the reference solutions. Even though we did not use the reference solution in NCB for evaluation, we provided them to facilitate the generation of test cases and future research.
Build High-Coverage and Corner Evaluation. We employ GPT-4 to generate evaluation codes for each problem. We construct a prompt using 1) the description of the problem for GPT-4 to inspect; 2) the reference solution to demonstrate the names and formats in the code; 3) an instruction to encourage GPT-4 to come up with effective test cases. Specifically, each prompt start with an instruction that ask GPT-4 to produce ten test cases based on the description of problem and the reference solution. Then, we present both the description of problem and its reference solution. We finalize the prompt with an initial segment of the evaluation code to assist GPT-4 in accurately generating the desired code format. Our objective is to harness GPT-4’s advanced comprehension and analytical abilities to learn valid format in the code and essential functionalities of the reference solution to enable the generation of superior test cases that are adept at uncovering latent errors in code.
A complete and effective test should seek to iden-
tify potential bugs at different locations in the code, while also finding inputs that might trigger errors in the code. High coverage ensures that each test case examines more code and branches, thereby facilitating the discovery of concealed errors. Meanwhile, it is often observed that corner values in a problem’s input are most prone to trigger code errors. Consequently, our instruction will cause some of the test cases generated by GPT-4 to have higher coverage, while the other part will be some corner values contained in the problem, so as to obtain more effective test cases.
Subsequently, expert programmers review and correct any test cases with formatting and answer errors. To ensure that the final evaluation framework is error-free.

heading: 2.3 Alignment Between Different Models
text: In contrast to the problem format in HumanEval, the majority of problems in our benchmark are composed in natural language by actual users. Consequently, there is no predetermined naming convention for functions or classes created by models. This divergence can lead to inconsistencies between the names generated by LLMs and those referenced in test cases. To address this issue of name misalignment, we present a representative test case that includes the designated function or class name and its usage within the test. We then instruct the LLMs to adhere to the naming convention specified in the provided test case when generating solutions. It is important to note that the test cases utilized for solution generation are not employed in subsequent testing phases. The details
of the instruction is showed in Appendix A.

heading: 2.4 Building Bilingual Benchmark
text: The majority of the questions we collected from online services are in Chinese, which is not fair for the LLMs that are primarily designed for English. Therefore, we translate all the problems, resulting in both Chinese and English versions.

heading: 3 Dataset Statistics
text: We provide more detailed statistics in Table 2. NCB comprises a total of 402 problems collected from online services, with 201 problems in Python and 201 in Java, spanning across 6 domains: Database, Artificial Intelligence, Data Science, Algorithm and Data Structure, Front-End, Software Engineering, and System Administration. This diversity also leads to complex input data types in NCB, which are classified into 9 categories: number (int/float/boolean), string, list (array), dict, tensor (matrix), data frame (table), plain text file, image, and special format file. The first four are the most common and simplest data types. Since a boolean can be represented by 1 and 0, we consider it as a type of number. Matrix and list are two similar types of data, but they are categorized separately due to differences in their usage scenarios. Due to the current popularity of deep learning, tensor has become a very common data format. Therefore, we have designated a separate category for tensor and have included matrix within this category. The last three are all file types, differentiated by their processing methods. The content of a plain text file is text and can be directly read. Figures require processing of each pixel value. A special format file refers to files that require specific methods for processing, such as PDF and DOCX.
Each problem within the dataset has been carefully curated with a set of test cases to assess the correctness of solutions. On average, there are 9.3 test cases associated with each problem. These cases are strategically designed, with about 60% focused on enhancing statement and branch coverage, and the remaining 40% dedicated to evaluating the robustness of solutions against corner values. The average word count for each problem in the NCB is 78.3.
Compared with Other Benchmark. Table 1 compares NCB to other benchmarks. It is noteworthy that our benchmark offers a substantial supplement to current benchmarks in terms of both problem and
data types. Unlike HumanEval and MBPP, which consist of 96.9% and 89.5% algorithmic and basic programming problems respectively, our benchmark features a more balanced distribution across each domain.
In addition, NCB includes more data types. Furthermore, NCB focuses on assessing the model’s ability to handle multiple file formats, a type of data that is both very commonly used in daily life and relatively challenging to process. We note that the problems involving files have fewer test cases, since GPT-4 still struggles to fully generate various types of file . This is also more challenging for human annotators to design compared to simpler data types.
On the other hand, NCB is also limited by its size due to the high costs of problems collection and the construction of the evaluation framework. We are continuously working on expanding our benchmark.

heading: 4 Experiments

heading: 4.1 Setup
text: We conducted comprehensive evaluations of 39 popular state-of-the-art models. For proprietary models, our focus was on OpenAI’s GPT-4-Turbo0125, GPT-4-Turbo-1106, GPT-4, GPT-3.5-Turbo, Anthropic’s Claude-2, ZhipuAI’s CodeGeeX3. In the case of open-source models, we performed evaluations using the vLLM (Kwon et al., 2023) and FastChat (Zheng et al., 2023a) framework. Our evaluation primarily utilizes pass@k (Chen et al., 2021) as the metric to accurately assess the functional correctness of code generated by these models. For k equal to 1, we employ greedy-search decoding. For random sampling, we demonstrate the best pass@k results of the best-performing models with each LLM family for each k ∈ {10, 50}, where the sampling temperature is set to 0.2 and topp to 0.9.
Our semi-automated pipeline is capable of reducing the time required for benchmark construction without compromising the quality of test cases. This paper primarily focuses on evaluating the efficiency of benchmark construction and the quality of test cases. Specifically, we adopt code coverage (Ivanković et al., 2019), a widely used metric for assessing the effectiveness of testing, as the criterion for evaluating the quality of test cases. We invite five programming experts, each tasked with constructing the same five problems. Initially, we
ask each expert to manually write a standard solution and 5 test cases. Subsequently, for the same problems, they complete the writing of standard solutions and test cases using the semi-automated pipeline. As it is challenging to ensure identical test case coverage, we require that the test cases written under both methods should not have a code coverage of less than 80%. Then, for the sake of convenient comparison, we calculate the scores for each construction method in a straightforward manner, which is outlined as follows:
Score = LineCov.+BranchCov.
T imeCost ∗ 10

heading: 4.2 Results of LLMs
text: Table 3 and Table 6 shows the pass@1 results on the test set and dev set of NCB, respectively. Considering the high consistency of results, we primarily analyze the results on the test set. As expected, OpenAI’s GPT-4 achieves the highest score of 52.8%. The performance of GPT-4-Turbo is very close to that of GPT-4, differing only by 1.3% , with GPT-4-Turbo performing better in Java but showing a larger difference in Python. Among the opensource models, DeepSeek-Coder-33B-Instruct performs the best, reaching a score of 43.0%. However, the 9.8% score gap with GPT-4 remains significant. On the other hand, it surpasses the 40.7% achieved by GPT-3.5, exceeding it by 2.3%. In
summary, the performance of state-of-the-art opensource models is now between GPT-3.5 and GPT-4, yet the majority of open-source models still do not match the performance of GPT-3.5.
When compared to a perfect score of 100%, it is observed that even the best-performing model, GPT-4, still falls significantly short. This is in contrast to its performance in HumanEval, where it has approached 90%.
Comparing the performance of models in Chinese and English versions, it is evident that the vast majority of models perform better in English. This holds true even for the top models, GPT-4 and GPT4-Turbo, which outperform their average scores in Chinese by 1.1% and 3.9%, respectively.
Furthermore, Table 3 systematically presents the performance of various open-source models at different scales. Models smaller than 10B scored between 0.0% and 23.9%, models between 10B and 30B scored between 3.9% and 35.1%, models between 30B and 60B scored between 21.8% and 43.0%, and models larger than 60B scored between 27.9% and 33.2%. It is evident that the scale of the model still has a significant impact on performance. Larger models generally outperform smaller models, indicating that increasing scale can indeed enhance a model’s capabilities. However, this is not to say that scale is everything; more refined data and training strategies can also significantly impact a model’s performance. Some smaller models, such as DeepSeek-Coder-6.7B-Instruct, can outperform those larger than 30B by approximately 2.8% and those larger than 60B by approximately 1.9%.
Table 5 shows the pass@k results of bestperforming LLMs with each LLM family on NCB, where k ∈ {10, 50}. We found that under random sampling, the scores of some models increased significantly. For instance, Codellama-70B-Instruct, unlike its performance on pass@1, clearly outperformed GPT-3.5 on both Pass@10 and Pass@50.
We compared the Python scores on the test set of NCB with the performances of models on HumanEval, as shown in the Figure 1. Most models are located in the upper triangular area of the graph, with many models scoring high on HumanEval but exhibiting relatively lower performance on NCB.

heading: 4.3 Performance mismatch on HumanEval and NCB
text: We show the rank orders of all tested LLMs in Table 3 with regard to HumanEval and NCB, as well as the difference of rank orders. We also plot the corresponding performances on two benchmarks to scatter diagram in Figure 1. Based on the table and figure, we have some interesting findings.
Performances of most LLMs on two benchmarks grow linearly proportional, and the differences of scores’ rank order are around 0. It demonstrates that NCB can indeed reflect the coding abilities of LLMs as HumanEval does in most cases.
However, we observe that some model series, notably the Phi, Deepseek-Chat, and WizardCoder, consistently exhibit a propensity to achieve superior rankings on the HumanEval dataset as opposed to the NCB across various scales, as shown in the Table 3. Additional model families, including CodeGen and Llama-3-Instruct, similarly display the trend, though to a reduced degree.
There might be a few potential hypotheses for the observation. First, as problems in NCB are more difficult and derived from natural user prompts, compared to those in HumanEval, LLMs with poorer generalization and instructionfollowing capabilities tend to perform worse. We find in preliminary experiments that problems in NCB cannot be properly solved by pre-trained base LLMs via mere in-context learning as HumanEval does, which indicates that to solve NCB problems requires stronger alignment and generalizability than HumanEval needs.
Second, it is possible that training sets of some LLMs are over-specifiedly optimized for HumanEval-style problems. On one hand, pretraining data of certain LLMs may be contaminated. As GPT-4 (OpenAI et al., 2023) reported, 25% of HumanEval has been contaminated in their pre-training corpus. On the other hand, instruction fine-tuning dataset may also be polluted. For example, Phi (Li et al., 2023b) reports a considerable amount of synthetic prompts resonating to some test samples in HumanEval. In (Yang et al., 2023b), the authors report leakage unidentifiable
by n-gram overlap when using popular rephrasing techniques to create training sets. The performance discrepancy between HumanEval and NCB in our experiments is also an evidence of the potential contamination.

heading: 4.4 Results of Semi-automated Construction
text: In Table 4, we can observe that the coverage of hand-written test cases is almost identical to that of test cases constructed through a semi-automatic pipeline, yet the time required for the former significantly exceeds the time needed for constructing test cases via the semi-automatic pipeline. Specifically, test cases can be constructed via the semiautomated pipeline in just 40 minutes, whereas manual writing requires 175.9 minutes, a difference of more than 4x. Consequently, the scores obtained for test cases constructed using the semiautomated pipeline are far higher than those for manually written test cases, with an average difference of 37.6. In summary, constructing test cases through the semi-automatic framework can achieve significantly higher efficiency without substantial loss in quality compared to manual writing.

heading: 5 Related Work
text: LLMs for code. Significant advancements in LLMs (Vaswani et al., 2017, Devlin et al., 2019, Brown et al., 2020) are transforming everyday life, particularly in the field of coding, driven by the vast amount of openly available codebases and the push to enhance productivity among developers. Codespecific LLMs have proven their ability to perform various tasks such as code generation (Chen et al., 2021, Iyer et al., 2018, Li et al., 2022), program repair (Jiang et al., 2023b, Wei et al., 2023, Xia et al., 2023, Xia and Zhang, 2022), automated testing (Deng et al., 2023a, Deng et al., 2023b, Liu et al., 2023c, Xia et al., 2024, Yang et al., 2023a), code translation (Roziere et al., 2020, Roziere et al., 2022) and code summarization (Ahmed and Devanbu, 2023, Lu et al., 2021). Notably, prominent LLMs including CODEX (Chen et al., 2021), CodeGen (Nijkamp et al., 2023b), INCODER (Fried et al., 2023), and PolyCoder (Xu et al., 2022) have been developed and rigorously tested, particularly in code generation. This area, often referred to as the ultimate goal in computer science research since the early days of AI in the 1950s, involves the model producing code snippets from natural language explanations of the required functional-
ity. The landscape of code LLMs is currently experiencing a surge, with new models being introduced regularly. This includes both proprietary ones (Moradi Dakhel et al., 2023, OpenAI et al., 2023) and open-source ones (Lin, 2004, Nijkamp et al., 2023b, Touvron et al., 2023, Li et al., 2023a, Anonymous, 2024, Rozière et al., 2024), marking a trend of frequent releases in this domain.
Code Synthesis Benchmarks. As the capabilities of models advance, researchers are developing more challenging and versatile benchmarks for code generation. Initially, the earlier focus was on domain-specific languages (Zelle and Mooney, 1996), while the subsequent effort launched a Textto-SQL benchmark to evaluate the capacity for generating comprehensive SQL programs (Yu et al., 2018). An investigation (Yin et al., 2018) assesses the ability to compose brief yet broadly applicable Python snippets. More recent studies (Hendrycks et al., 2021b, Li et al., 2022) have tested models’ proficiency in solving competitive programming challenges using Python. A leading and extensively researched benchmark in this domain is HumanEval (Chen et al., 2021), which features 164 Python function signatures accompanied by docstrings and corresponding test cases for validating correctness. Additionally, each problem in HumanEval includes a reference solution. The MBPP (Austin et al., 2021) dataset, another Python-centric collection, was developed by having participants contribute 974 programming challenges. Each challenge encompasses a problem description (i.e., docstring), a function signature, and three test cases. There are also benchmarks for other programming languages, such as HumanEval-X (Zheng et al., 2023b) for C++, JavaScript, and Go, CodeContests (Li et al., 2022) for C++ and Java, and MultiPL-E (Cassano et al., 2022), which expands HumanEval and MBPP to 18 languages.
More recent efforts have introduced benchmarks that more closely mirror real-world coding scenarios that require interactive coding. For example, AgentBench (Liu et al., 2023b) introduces interactive tasks regarding unix shell and MySQL. SWEBench (Jimenez et al., 2023) compiles GitHub issues, their associated codebases, and tests, to gauge LLMs’ effectiveness in practical software engineering tasks.

heading: 6 Conclusion
text: We propose NATURALCODEBENCH for evaluating the code generating ability of LLMs. Our benchmark comprises a total of 402 problems selected from coding online services, and it supports automatic evaluation of code generated by LLMs. We have also proposed a semi-automated pipeline for efficiently constructing the entire benchmark, achieving an efficiency gain of more than 4x compared to manual construction. We hope that NCB can provide a fair environment for the comparison between models, and our pipeline can also provide inspiration to other complex tasks or domains where evaluation costs are high.

heading: ACKNOWLEDGEMENT
text: We would like to thank the anonymous reviewers as well as Zhipu AI for covering all GPU and API cost consumed in this study. This work is supported by The National Key Research and Development Program of China 2021YFF1201300Natural Science Foundation of China (NSFC) 62276148 and 62425601. It is also supported by Tsinghua UniversityDepartment of Computer Science and Technology-Siemens Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things (JCIIOT).
Limitations
Here, we discuss several limitations of this work.
To cover more domains. Although our problems are derived from real-world application scenarios, due to the difficulty of constructing accurate and efficient evaluation environments, we are unable to test some types of problems, such as those involving interface creation, web services, etc., which are also common problem types in actual applications. This results in some biases in our evaluation, which may affect the accuracy of the evaluation of certain models. We will leave these issues for future research.
To reduce the cost. The semi-automated pipeline can significantly reduce the time and human resources required to construct an evaluation framework, but the cost of accessing OpenAI’s API remains expensive, and it does not completely eliminate the use of human resources.

heading: B Examples
text: B.1 Examples of Semi-Automated Pipeline
In this section, we present two examples, one each for Python and Java, of semi-automated pipeline
with one test case to illustrate how we construct test cases and rectify errors therein.
Figure 5 shows the Python example. Following the provision of problem description and reference solution, GPT-4 writes the majority of the test case, including the execution procedure and test case input. However, GPT-4 could not guarantee the correctness of each test case, resulting in the generation of erroneous expected outputs. At this point, our programming experts only needed to correct the incorrect expected outputs.
Figure 6 shows the Java exmaple. In this problem, where the input type involves more complex file formats, our semi-automatic pipeline is unable to directly generate the input files corresponding to each test case. Therefore, in this instance, our programming experts need to not only supplement the missing procedures in the test cases but also create an input file for each test case. However, GPT-4 has provided reference content for the input files in the comments, so our programming experts do not need to design the inputs themselves.
B.2 Example Problems
Here, we present an example problem and test cases for each of the 6 domains.
Figure 7 shows a problem of Algorithm and Data Structure, querying the pattern of a sequence transformation and the total number of all transformations.
Figure 8 illustrates an example problem in software engineering, requiring the addition of tags to different titles in a markdown file according to their levels.
Figure 9 presents an example problem in data science, asking to select the row with the highest temperature from the temperature CSV files of each city and write these rows into a new CSV file.
Figure 10 depicts an example problem in frontend development, requiring the replacement of given special tags within a string with specific HTML formats.
Figure 11 shows an example problem in artificial intelligence, requiring the calculation of the distance between each point of two tensors, where the dimension of each tensor is batchsize * n * 3, with the third dimension representing the coordinates of the points.
Figure 12 presents an example problem in system administration, inquiring how to rename all the files within a folder according to a given rule.

heading: C Extra Results
text: Table 6 shows the pass@1 results on the development set of NCB. The results on the development set are essentially consistent with those on the test set, with some changes in the ranking among several models. This is due to differences in the distribution of problems across domains between the development set and the test set.
Table 5 shows the pass@k results of bestperforming LLMs with each LLM family on NCB, where k ∈ {10, 50}. We do not evaluate the performance on pass@k for ErnieBot4, CodeGeeX3, Claude-3, Gemini-1.5-Pro and Llama-3-Instruct due to limitations on the use of API and other resources.

heading: I want to write a python program that rename the files of a folder .

heading: Future Work
text: We will leave these issues for future research. To reduce the cost. The semi-automated pipeline can significantly reduce the time and human resources required to construct an evaluation framework, but the cost of accessing OpenAI’s API remains expensive, and it does not completely eliminate the use of human resources