Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Authors: Jiawei Liu
Chunqiu Steven Xia
Yuyao Wang
LINGMING ZHANG
Venue: NeurIPS 2023 poster (2023)
Keywords: LLM4Code
ChatGPT
Automated Test Generation
TL;DR: We observe that current code benchmarks use weak tests that may affect all LLMs evaluated on them, and address this via automated test generation.

Abstract: Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus – a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Jiawei Liu R.
Chunqiu Steven Xia I+
Yuyao Wang
Lingming Zhang
University of Illinois Urbana- Chaama-Ca
Nanjing University
(jiawei6, chunqiu2, lingming/@illinois edu
yuyao6@outlook.ccm
Abstract
Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus to rigorously benchmark the functional correctness of LLM--ynthesizeed code. EvaiPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus general, we extend the test-cases of the popular HUMANEVAL benchmark by 80x to build HUMANEVAL". Our exten- sive evaluation across 26 popular LLMs (e.g.. GPT-4 and ChatGPT) demonstrates that HUMANEVAL is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k k by up-to 19.3-28.9% We also surprisingly found that test insufficiency both WizardCoder- SrraadCodder-CodeLaamm CodeLlama and Phind-CodeLlama now outperform ChatGPT HUMANEVAL', while none of them could HUMANEVAL. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of L.LMs for code synthesis, but also opens up new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-gennratee code at https: / /gggghtthb.... com/ honn/vallluss// taais/evalluuss to facilitate and accelerate future LLM-for-code research.
code synthesis evaluation framework
lead to mis-ranking For example,
1 Introduction
Automatically generating programs that accurately correspond to user intents is a long-standing challenge in computer science known program synthesis [21]. In the past few decades, classical program synthesis techniques have been developed, including deductive synthesis [19, 39, 62], inductive synthesis [20, 58] and neural-guided synthesis [29]. More recently, with the advent of
Large Language Models [61, 6) (LLMs) and the abundance of open codebase, researchers have been focusing on applying LLMSs for direct code generation. LLMS like CODEX [11] and CodeGen [46]
"Equal contribution. Author ordering is decided by Nigiri.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

14,3,2,8
[,,
[3,2] 2]
c11: 1ist, 12: list]:
[2,3]
23, [4,3,2,8], 4]
comonn_lenentss "'ist(se((L). (ost(eec()..neeeeettonn((((())) common common_elenntss.son
"""Return serted unique comnon elements for two lists""
[2,3,4]
HUMONEVAL inputs
correct
return dist(set(comoneeeeeee
86,8,11, HUNANEEAL" input
8618,1]
[8,1, 6] not sorted!
ChatGPT synthesized code
Figure 1: Exemplary wrong code synthesized by ChatGPT for HUMANEVAL #58.
perform code generation by autoregressively predicting the next token given previous context, in the form of function signature and docstring that denote the desired program functionality. The generated code snippet is then combined with the context to form a complete function that aligns with the user intent. Leveraging both natural language understanding and generative power, LLMs have demonstrated impressive performance in code synthesis [3, 11].
The primary concern when it comes to LLM-generated code is correctness. Because two dramatically different code snippets can be semantically equivalent, classic NLP metrics like BLEU score [50] no longer reliable in the context of program synthesis. Ideally, the correctness of --o-viddeed solutions for any input, but verifying domain-specific problems through methods such translation validation [36, 44, 4] is already challenging enough, let alone building a general verifier with absolute certainty to prove arbitrary problems, including those in code benchmarks. As such, existing code benchmarks (e.g., HUMANEVAL. [11]) heavily rely on manually constructed test-cases to evaluate I.IMM solutions. However, these tests often fall short in capturing all possible scenarios, crafting high-quality tests is laborious. Consequently, argue that current programming benchmarks are inadequate for assessing the actual correctness of LLM-generated code, leading to false confidence in the results. Specifically, we have identified the following common limitations in existing LLM-for-code benchmarks:
would like to formally verify
Insufficient testing. Current programming benchmarks often only include on average less than 10 tests for each coding problem. Furthermore, these tests are relatively too simple to fully explore the functionality of the code or corner cases. Figure shows an incorrect code sample synthesized by ChatGPT [48] return the sorted unique common elements from two lists. At first glance, the function looks correct and computes the desired output when using the base test inputs from HUMANEVAL. However, in the return statement, it incorrectly converts the intermediate list to set which no longer preserves the order of the sorted list. This example shows that a logically flawed solution can still pass all simple tests and be misconsidered correct due to testing inadequacy. Imprecise problem description. The input for code generation includes natural language descriptions in addition the function signature. These task descriptions in existing benchmarks are oftentimes too vague to fully clarify the expected program behaviors. For example, the imput docstring may not specify the expected input domain (e.g., only positive integers) or how the function should handle exceptions. As result, such programming problems can be interpreted differently by LLMs against the actual tests, leading to capable LLMs misjudged as incapable.
These limitations are common across many popular code generation benchmarks [11, 3, 33]. This not only questions the validity of the impressive performance claimed by prior work but also sets a chal- lenge on how properly evaluate the LLM coders. In this paper, aim to address this fundamental evaluation challenge and ask the introspective question: Is the code generated by L.MSS really correct?
Our proposal. In this work, we set out to answer the important question and evaluate the evaluation dataset. Consequently, we build EvalPlus an evaluation framework to improve existing code bench- marks in order to precisely evaluate the functional correctness of I.LM-generated code. At the heart of EvalPlus is an automatic test input generation engine which augments existing code benchmarks by generating interesting test inputs to fully exercise the code solution and check its functional correctness by cross-checking the ground-truth implementation. Specifically. EvalPlus adopts both LLM- and mutation- based [57, 74, 47] methods to automatically generate and diversify additional test inputs. EvalPlus first uses ChatGPT [48] to generate a set of high- high-quality seed inputs that aim to test difficult corner cases and functionalities of the program within the valid input structure. Using these high-quality seed inputs, EvalPlus then performs type-aware mutation to efficiently generate a large number of additional test inputs. These newly generated test inputs are then used to evaluate the M-generrtedd code through differential testing [40] against the ground-truth implementation. Furthermore, option to speed up evaluation, EvalPlus also builds minimal test-suites by only

---

generate complex inputs
generate difficult inputs
ChatGPT
autant
set cover
sample kills Test-suite Reduction
generate corner-case inputs
send inputs
original dataset
EvalPlus dataset
def (runntruth((nnut))
type-aware
groundtruth (input))
mutation
ingut
in in input
input
input
input
seed pool
inputs
new inputs
def Sample_((input))
gt
def sample__d (ánpuut):
def sample_ sanpte_((inuut);
di differential testing
def sanple_ (input):
sampte_00inpus (input):
f(x) = gt(x)?
Rigorously validated LLM samples
LLM samples
Figure 2: Overview of EvalPlus
including the most valuable test-cases, which are selected by running a greedy set cover algorithm to preserve the same code coverage [24], mutation analysis [7] as well as empirical LLM sample killings.
Contribution. Our work revisited and proposed to automatically improve code benchmarks for LLMs:
Study: We are the first to study the test inadequacy problem current programming benchmarks
which can lead to largely over-approximmted functional correctness. Our study also opens up new research direction for precisely and rigorously evaluating LLM-synthesized code.
Approach: We propose EvalPlus an evaluation framework reveal the real correctness of LLM-syythesized code. The test-case generation approach of EvalPlus combines the emerging LLM-based and traditional mutation-based test input generation. It first uses LLM-based strategy to bootstrap the test generator with high-quality seed inputs and then further extends large amounts of inputs via type-aware mutation. We then optionally "distill" the generated tests to much smaller yet almost equivalently effective test-suite via greedy set covering. We also propose to annotate each programming tasks using program contracts to filter out invalid inputs.
Results: EvalPlus extends the popular HUMANEVAL benchmark to create HUMANEVAL*, improving the test-case scale by 80x. Through test-suite reduction, we also produce HU- -ANEVAL which distills HUMANEVAL" tests by 47x while still achieving a similar level testing effectiveness. Our extensive evaluation over 26 popular LLMs surprisingly finds that the pass @ / on the new dataset is up-to 99.3-28.9% (for different ks) lower than the base HUMANEVAL. showing that testing insufficiency can largely affect the result analysis for almost all recent work on LLM-based code generation. Meanwhile, the original HUMANEVAL both of the 34B SeaardCoddr-CoddeLin
[38] and Phind-Codela CodeLlama [52] models are deemed to be no better than ChatGPT, while EUAM"NEVAL" corrected the ranking and shows that the two open-source models are actually better. Additionally, we even found that the ground-truth solutions of HUMANEVAL. be erroneous, further calling into question the quality of code synthesis benchmarks.
2 Approach
Figure 2 shows the overview of EvalPlus. We first take in as input the original dataset containing the ground-truth implementation well the base test inputs. EvalPlus starts with constructing prompt using the original ground-truth, exemplary test inputs as demonstration, and specialized instruction to query ChatGPT and generate a set of high-quality seed inputs. ChatGPT, by following base input formats and inspecting the ground-truie truth solution, can serve vehicle to generate valid yet rigorous test inputs. Starting from these seed inputs, we then perform type-aware mutation to quickly generate numerous new inputs together with seed inputs to extensively evaluate the functional correctness of -enerreee code. We use differential testing [40] as the oracle cross-check the output of the ground-truth and LLM --geeeerrtted solution. As option to speed up evaluation, EvalPlus
3

---

Table 1: List of basic type-aware mutations over input
Type
Mutation
Type
Mutation
Remove/repeat a random item sr(i]
int float
Returns a:
List
Insert/replace x:{i] with Mutate(z(i)
bool
Returns random boolean
Tuple
Returns Tuple (Mutate (List (x)))
NoneType
Returns None
Set
Returns Set (Mutate (List
Remove sub-string
Remove key-value pair k Update - to Mutate(u)
str
Repeat a sub-string 8 Replace with Mutate(s)
Dict
Insert Mutate(k) Mutate(u)
runs set covering to minimize the generated test-suite while preserving the same level of testing effectiveness. As the final output, EvalPlus obtains augmented benchmark using the generated high-quality test inputs to fully evaluate the functional correctness of LLM -uM-syyttesizeed code.
2.1 Automated Test Input Generation
Seed initialization via ChatGPT. EvalPlus first uses ChatGPT to generate set of high-quality seed inputs for later mutation. Following Figure 2, we construct a prompt using (i) the ground-truth solution of the problem for ChatGPT to inspect; (ii) of test inputs as demonstration; and fiii) instruction to encourage ChatGPT to come up with interesting inputs. Specifically, each prompt starts with the ground-truth implementation and then randomly sampled test inputs from the existing dataset. We then finalize the prompt with a selected instruction in Figure 2 and query ChatGPT to produce new inputs. EvalPlus aims to leverage the powerful understanding ability ChatGPT to learn both the valid input formats (e.g., variable types) as well as the desired functionality of the ground-truth solution in order to produce meaningful test inputs reveal bugs in incorrectly synthesized code. Programs can have their own expected input formats, where invalid inputs should not be passed into the function as they can incur undefined behaviors to create false- positivvss in differential testing. such, we filter out any invalid inputs which violate the input precondition required by the ground-truth implementation.
By using ChatGPT
automated generation engine, can generate inputs that are valid even under semantic constraints, For example, a programming problem may require the input to conform specific structure (e.g., a palindrome). Such semantic constraints can be extremely difficult for traditional input generators to satisfy. However, ChatGPT is unsuitable for large amounts of automated test generation due to undesired speed and cost of querying such large model. To address this, we perform type-aware input mutation starting from high- high-quality seed inputs generated by ChatGPT.
Type-aware input mutation. We follow typical mutation-basel fuzzing workflow [74, 57] to continuously create inputs: (i) a corpus of seed inputs from ChatGPT are used initialize the seed pool and bootstrap the generation pipeline; (ii) each time an input (i.e., seed) from the seed pool is randomly selected be mutated a new input (i.e., mutant); and (di)) new inputs that comply with the program contract ($2.3) are added to the seed pool and we start over from (ii) to continue the generation process.
To efficiently create more valid inputs, leverage type-aware mutation [66] in step (ii) which in- spects the data types of the incoming valid seeds and generates new inputs that are structurally similar the seeds. In Table 1 we illustrate the basic mutations used for different types inputs. For simple primitive types such as int and float, the mutation is as simple as iivemenniing/eccrmeennng the value. For compound types and the string type (i.e., str), besides generally removing or repeating existing elements (or sub-strings for str), the elements and sub-strings can be mutated recursively according to their inner types. Such sub-mutants can then be used replace existing items or add new items in finer-grain manner. In addition, to alleviate generating inputs that violate subtle semantic constraints, following [23, 34], we additionally apply an ingredient mechanism to collect appeared data fragments and reuse them during mutation. In short, type-aware input mutation builds on the high-quality seed inputs produced by ChatGPT to generate large amounts of test inputs which we use the final set of extensive test inputs to evaluate LLM-synthesized code.

---

2.2 Test-Suite Reduction
While the large number of newly generated tests in EvalPlus are effective in detecting incorrect code, the test execution can be costly. As an option to more efficiently evaluate LLM-generraed code, we further investigate test-suite reduction strategies [75, 59], which aim to select a subset of the original test-suite while still maintaining the original test effectiveness. To perform test reduction, it is typically assumed that each test can fulfill a set of testing requirements. The problem can then be formalized as reducing the original test-suite T into Tred, such that Vr E R (Et E T, t satisfies r = Et' e Treatt t' satisfies r). In other words, any testing requirement r satisfied by the original test-suite should still be satisfied by the reduced one. Finding such minimal representative subset for given test-suite is equivalent to the set covering problem [17]. To solve this problem effectively, it is crucial to define the testing requirements accurately. In this paper, we focus the following types of requirements:
Code coverage: Code coverage [24] measures the amount of code elements (e.g., statements or branches) executed by each test, and has been widely used in practice to measure test effectiveness. In this strategy, following traditional test-suite reduction [53] leverage the widely used branch coverage as the testing requirement. In other words, the goal of using this metric is to only preserve a minimal subset of tests which can cover the same set of branches as the full tests.
Mutant killings: Coverage measures the extent which the code has been executed; however, a high-coverage test-case is not necessarily effective in finding critical defects in its covered code. Conse- quently, researchers have proposed mutation testing [7] (also known as mutation analysis) to more pre- cisely evaluate test effectiveness. In short, mutation testing applies set of predefined mutation rules (e.g., changing "<" and "<") to the program under test (i.e., the ground-truth truth solutions for this case) to create a large number of artificial buggy programs, each of which is called as mutant and includes exactly one subtle bug seeded. In this way, the ratio of mutation bugs detected by the tests (also called killed) can be used to assess the test effectiveness. In fact, studies have shown that mutation testing can largely outperform code coverage in test effectiveness evaluation [51]. Following prior work [59], we also leverage the set of mutants killed by each test as our testing requirement. Consequently, the goal is to minimize the number of tests while still being able to detect the same set of mutation bugs.
Different LLMs could fail commonly over certain test-cases. Consequently, besides these theoretical metrics, we also use as testing requirement by empirically looking at sample killings, i.e., the set of wrong LLM samples that a test-case can detect and falsify. Of course, for a new LLM under evaluation. we do not have any test execution results for its code samples. Therefore, we only use the execution results for samples generated by other LLMs to evaluate test effectiveness for reduction (i.e., leave-one-out cross validation [22]). As such, we minimize the number of tests while making sure that all incorrect samples synthesized by other models can be detected by the reduced test-suite.
LLM sample killings:
Besides the above three strategies, we also investigate another strategy that merges all three testing requirements for reduction. That is, the goal is to minimize the number of tests while still maintaining the same branch coverage, mutant killing, and incorrect sample detection results.
2.3 Program Input Contracts
The goal of evaluating code synthesis is to check whether the synthesized code accurately reflects the desired user intent. This is done by using several test inputs and comparing the output of the generated code against that of the ground-truth solution. The prior sections demonstrated how to improve the test inputs used to more rigorously evaluate the synthesized code. However, these user intents (cxpressed as natural language docstring) can be too vague for LLMs to follow. As such, LLMs might allow for dif- ferent interpretations of the desired functionality, input formats as well as how to handle corner cases.
To this end, we adopt programming by contract [41] philosophy by systematically annotating function pre-conditions in form of code assertions (e.g., assert n > 0), to ensure the test inputs for the function are well-formed. The benefits of the contracts are two-fold: (i) they can complement the automatic input generation steps to filter out any generated invalid inputs that violate the contracts. Such ill-formed inputs can incur undefined behaviors which are unreasonable to use for evaluating L.LM-synthesized code; and (ii) they can serve as orthogonal descriptors together with the natural language description in the prompt for further clarification.
5

---

Table 2: Overview of EvalPlus-improved benchmarks.
# #Tests
#Tasks
Avg.
Medium
Min.
Max.
HUMANEVAL
9.6
7.0
1
1052
HUMANEVAL"
764.1
982.5
12
1,100
164
HUMANEVAL -MINI
16.1
13.0
110
3 Evaluation
Setup. Our evaluation focuses on using the unbiased version of pass@k [11] to accurately assess the functional correctness of LLM-syntheizzed code. For generalizability, we conducted a comprehensive evaluation over 26 popular and state-of-the-aue thr- LLMs and a wide range of temperature settings. Specifically, following prior work [11, 46], for each model we perform: (i) random sampling to generate 200 program samples for each of the four temperature settings 0.0.2,0.40066...0 0.8)); and (ii) greedy-search decoding. For random sampling, we show the best-performing pass @ for each E (1,10,100) and its corresponding temperature denoted by T For greedy decoding, we only synthesize one deterministic sample for each task and evaluate its pass rate as pass@ I*. By default we evaluate models under both setting (i) and fii), except for the two commercial models due to time and cost constraints: GPT-4 is only evaluated under greedy decoding, and ChatGPT is additionally evaluated on 0.8-temperature random sampling.
While EvalPlus is general, this paper focuses evaluating its effectiveness on HUMANEVAL VAL [11], one of the most widely-used datasets for code generation', HUMANEVAL consists of 164 human- written programming tasks, each of which provides a Python function signature and doestring as the input to the I.I.M. Based on the input, L.MSs complete a solution whose functional correctness is judged by handful of manual test-cases (the first row in Table 2). As such, EvalPlus transforms HUMANEVAL to HUMANEVAL" by adding 80x unique test-cases and fixing incorrect ground-truth solutions in HUMANEVAL. Specifically, for each task, based on around 30 ChatGPT-generated seed inputs which are produced using 3 separate prompts, we run type-aware mutation to generate 1000 additional inputs using one-hour budget. In HUMANEVAL" 83 out of the 164 programming tasks are annotated with hand-crafted contracts. Because EvalPlus requires ground-trui truth solutions to cross-check LLM- -eM--enerrted code, is crucial to ensure the correctness of the mround-truthe However, by inspecting ground-truths in the original HUMANEVAL, we found over 10% of them are incorrectly implemented. Therefore, as another contribution we carefully re-implemented and tested all ground-truthe truths for HUMANEVAL.*. As an option to speed up evaluation, we build HU- MANEVAL* MINI which is minimized from HUMANEVAL. (smaller by 47x) yet preserves similar test effectiveness on the studied models. Lastly, more experimental setups are detailed in Appendix.
Evaluation of I.Mss. Table 3 shows the pass@k when evaluating LLMs using both the base HUMANEVAL. and HUMANEVAI.*. We first observe that across all L.LMs, models sizes and values, using HUMANEVAL", almost all pass@k k results consistently drop compared to using the base HUMANEVAL. Notably, the performance drop is significant with up-to 23.1% (pass@) (²) 19.3% (pass@) 1) 24.9% (pass@) 10) 28.9% (pass @ 100) reduction over the evaluated models. Such performance decrease is not only seen in popular open-source LLMs, such as the widely used CodeGen- 16B [46] (18.5% reduction) well as the emerging CODELLAMA --B [54] (17.6%) and StarCoder [13] (14.1% reduction), but also observed in of-the-arrr commercial ChatGPT (12.6% reduction) and GPT-4 (13.1% reduction) models. Overall, our results overall confirm our hypothesis that the prior evaluation on HUMANEVAL. is not robust enough to detect wrong code synthesized by I.MMs. Not only are these L.MMs widely used for daily programming but they also serve as common reference points for evaluating new code synthesis techniques. As such, evaluating on more robust benchmark such HUMANEVAL* highly recommended in order to draw precise conclusions.
'There are four HUMANEVAL tasks (e.g.. add y)) with over 100 "tests" (i.e.. implemented by cross-checking the ground-truth over random inputs). Without such, the maximum/average number is 26/7.3. 'Top-1 HuggingFacs downloads on April, 2023. https: co/ /attasetssother-cooee ceeeegggeeeaiooo "To date, CodeGen2- 16B is released with unfinished checkpoint [45]. Nonetheless, we show its pass@11 1"

---

Table 3: Evaluating LLMs on HUMANEVAL and HUMANEVAL", All models, except for INCODER, CodeGen2, StarCoder and SantaCoder which perform infilling, use auto-regressive generation. k=1* marks pass@1 1 done with greedy decoding. Tg denotes the optimal pass temperature.
Size
pass@
k=1"
k=1
k=10
k= 100
T
Tio
Tioe
base
88.4
GPT-4 [49]
N/A
+extra
76.2
base
71.3
71.6
90.5
96.2
.2
.8
Phind-CodeLLamma [52]
34B
+extra
67.1
67.0
85.0
92.5
.2
.8
.8
base
73.2
61.6
85.2
94.5
.2
.8
.8
WirardCoder-Codel.lme [38]
34B
+extra
54.5
78.6
88.9
base
73.2
69.4
88.6
94.0
ChatGPT [48]
N/A
+extra
63.4
62.5
82.1
91.1
base
51.8
52.0
82.4
95.0
.2
.8
.8
34B
+extra
89.4
.8
hase
42.7
44.6
77.6
92.7
.4
.8
.8
CODELLAMA [54]
13B
+extra
36.6
37.4
69.4
88.2
.4
.8
.8
base
37.8
39.2
69.1
89.7
.2
.8
.8
7B
+extra
34.5
82,9
base
34.1
32.2
56.7
84.2
.2
.8
StarCoder [13]
15B
+extra
29.3
27.8
50.3
75.4
.2
.8
.8
base
32.9
32.2
56.0
81.5
16B
+extra
27.2
71.4
.6
base
29.3
27.7
46.9
72.7
.2
.6
.8
CodeGen [46]
6B
+extra
25.6
23.6
41.0
64.6
.2
.6
.8
basc
24.4
18.4
39.8
66.8
.2
.8
.8
2B
+extra
20.7
15.1
55.8
.2
base
31.7
32.2
58.5
83.5
.2
.6
.8
CODET5+ [64]
16B
+extra
26.2
27.4
51.1
76.4
.2
.6
.8
base
28.7
28.1
55.2
83.8
.8
.8
MISTRAL [26]
7B
+extra
23.7
76.4
.8
base
19.5
16B*
+extra
16.5
base
18.3
17.9
30.9
50.9
.6
7B
CodeGen2 [45]
+extra
16.5
15.9
45.4
.6
hase
15.9
15.2
23.9
38.6
.2
.4
.8
3B
+extra
12.8
12.9
21.2
34.3
.2
.4
.8
base
11.0
10.2
15.1
24.7
.2
.6
.6
1B
+extra
13.7
21.2
.6
base
16.5
15.3
30.1
54.8
.2
.8
13B
VICUNA [12]
+extra
15.2
13.9
25.8
46.7
.2
.8
.8
base
11.6
10.9
23.8
42.3
.6
7B
+extra
11.0
10.3
20.3
35.0
base
14.6
16.6
29.2
45.4
.6
.8
SantaCoder [2]
1.1B
+extra
12.8
14.2
26.2
40.6
.4
.6
.8
base
15.9
15.6
27.7
45.0
.4
6.7B
INCODER [18]
+extra
12.2
12.4
22.2
38.9
.6
base
12.2
10.0
15.9
25.2
.2
.6
.6
1.3B
+extra
10.4
7.9
13.5
20.7
.2
.6
.4
base
12.2
11.3
17.7
31.8
2
GPT-J [63]
6B
+extra
10.4
15.2
25.9
.6
base
7.9
6.5
11.8
20.7
.2
.6
.6
GPT-NEO [5]
2.7B
+extra
6.7
6.0
9.0
16.8
.2
.6
.6
base
6.1
10.2
17.1
.4
PolyCoder [70]
2.7B
+extra
5.5
5.3
7.9
13.6
.2
.6
.6
hase
2.4
2.7
7.5
15.8
2
.6
.6
StableLM [60]
7B
+extra
2.4
2.6
6.2
11.9
22
.6
.6

---

Table 4: Reduced test-suite for HUMANEVAL." We first show the pass@ I' and average #tests (includ- ing base VUMANEVAL tests) by only doing set covering over each considered metric separately ($2.2).
The Full column then shows the final reduction result by combining all of the three. For reference, the average #tests original HUMANEVAL and HUMANEVAL" are 9.6 and 774.8 respectively (Table 2).
Coverage
Killed mutants
Killed samples
Full
Ref. pass
Size
pass
#tests
pass@] 1"
#tests
pass@1 1*
#tests
pass 1*
#tests
base
+extra
GPT-4
N/A
86.0
11.3
82.9
11.4
78.7
13.8
78.0
16.1
88.4
76.2
ChatGPT
71.3
11.3
69.5
11.4
13.7
65.2
16.0
73.2
63.4
StarCoder
15B
32.9
11.3
32.9
11.4
29.3
13.6
29.3
15.9
34.1
29.3
2B
23.2
11.3
23.8
11.4
21.3
13.2
21.3
15.4
24.4
20.7
CodeGen
6B
28.7
11.3
29.3
11.4
25.6
13.2
25.6
15.4
29.3
25.6
16B
31.7
11.3
31.1
11.4
27.4
13.2
27.4
15.4
32.9
26.8
IB
10.4
11.3
11.0
11.4
9.1
13.8
9.1
16.0
11.0
9.1
3B
15.9
11.3
15.9
11.4
12.8
13.8
12.8
16.0
15.9
12.8
CodeGen2
18.3
11.3
18.3
11.4
16.5
13.8
16.5
16.0
16.5
16B
19.5
11.3
18.9
11.4
16.5
13.8
16.5
16.0
19.5
16.5
7B
11.6
11.3
11.6
11.4
11.0
13.8
11.0
16.1
11.6
10.4
VICUNA
13B
16.5
11.3
16.5
11.4
15.2
13.8
15.2
16.1
17.1
15.2
SantaCoder
1.18
14.6
11.3
14.6
11.4
12.8
13.8
12.8
16.1
14.6
1.38
12.2
11.3
12.2
11.4
10.4
13.6
10.4
16.0
12.2
10.4
INCODER
6.7B
14.6
11.3
14.6
11.4
12.2
13.6
12.2
16.0
15.9
12.2
GPT-J
12.2
11.3
12.2
11.4
10.4
13.8
10.4
16.0
12.2
10.4
GPT-NEO
2.7B
7.3
11.3
7.3
11.4
6.7
13.8
6.7
16.1
7.9
6.7
PolyCoder 2.7B
6.1
11.3
6.1
11.4
5.5
13.8
5.5
16.1
6.1
5.5
StableLM
2.4
11.3
2.4
11.4
2.4
13.8
2.4
2.4
We also show that more rigorous evaluation could yield different or totally contradictory relative results. For example, WizardCoder-Cotell.iee Cama and lhind-Codel.lama on the original HUMANEVAL are evaluated to be no better than ChatGPT in terms of pass@1'. However, HUMANEVAL* demonstrates that the two open-source models can actually outperform the proprietary ChatGPT. Other contrary examples reflected by HUMANEVAL* include that SantaCoder-1B surpasses INCODER-6.7B and VICUNA-7B outperforms INCODER-1.3B. Table further illustrates the distribution of best-performing temperatures over different k values. Our results conforms with prior findings [11] that a lower temperature tends to perform better for smaller k, while higher temperature works better for larger k. We also observe that the optimal temperatures seem to stay fairly consistent before and after using HUMANEVAL" however, slight differences still exist, e.g., best temperature for CodeGen-2B on pass @ 10 becomes 0.2 from 0.8 after using HUMANEVAL" Nonetheless, this motivates future research to look more closely on the effect of temperature with respect the robustness of the evaluation tests, esp. those edge-cases.
Effectiveness of test-suite reduction. Based on HUMANEVAL" which on average obtains 764.1 tests for each programming task (Table 2), our test-suite reducer ($2.2) minimizes to HUMANEVAI.*- MINI which only has 16.1 tests for each task (smaller by 47x). Table 4 performs leave-one-out cross validation to show the pass @ differences over a subset of representative models studied in Table (due to time/space constraints). That for each evaluated LLM we construct the reduced test-suite without considering its own sample kills. The Full column shows that the reduced test-suite can achieve almost the same pass 1* drop as HUMANEVAL* by only using 47x fewer test-cases. Taking a closer look, separately performing set covering over each metric can harness the pass @ 1* of the base HUMANEVAL certain degree. Specifically, the use of empirical LLM sample killings is the most ef- fective, leading to the same effectiveness as the full approach, but also consumes more tests than other theoretical metrics. While using coverage and mutation analysis seems be unnecessary in addition to using sample killings, they still serve the base guarantees for the theoretical test adequacy.
Pass rate distribution. Figure shows for each programming task the overall pass rates on HU- MANEVAL and HUMANEVAL. tests. The pass rate gap between HUMANEVAL. and HUMANEVAL* shows overall HUMANEVAL* can detect solutions that are misidentified by HUMANEVAL. for problems of all levels of difficulties. We also observe that problems HUMANEVAL. are not equal, not only in terms of problem difficulty but also the difficulty of generating counter-examples and
8

---

10a
HUMANEVAL HUMANNEALL
Average Pass Rate (%)
10
100
Problems (Sorted by HUMANEVAL pass rate)
Figure 3: Pass rate distribution. X-axis spans bars for all 164 problems, sorted by the HUMANEVAL pass rate. Y-axis shows the log-scale pass rates averaged by all LLM- ----eeeertted samples.
edge-cases to deeply exercise LLM-generated code. For simple problems such as "adding two numbers" and "length of string" (i.e. problems with top-2 pass rates), it is easy to solve for LLMS and to test manually. While problems dealing with multiple conditions (e.g. "word splitting"), completeness (e.g.. handling negative numbers for "is-prime"), reasoning ability (e.g., "Tribonacei sequence") and efficiency requirements (e.g., "n-th prime Fibonacci number") are the hardest tasks to the evaluated LLMs, positioning future research improve LLMs for conquering such coding skills.
Incorrect "ground-truth" in HUMANEVAL. In addition detecting wrong code from LLMs using EvalPlus, we also found 18 defects (1 (11% of problems) even in the original ground- in HU- MANEVAL, including (i) Unhandled edge-case: five prior ground-truths fail handle corner-case in- puts (e.g., empty list or string); (ii) Bad logic: 10 prior ground-truths incorrectly implement the desired functionality; and (iii) Performance issue: three inefficient implementations lead to slow performance on ceassnabbly-sized inputs. Among those, bad logic (10) is the most serious as the original "ground- truth" does not accurately reflect the user intent. Such defects are detected also through differential test- ing but between our own re-implemented ground-truth and the original ground-truth in HUMANEVAL.
def valid_ date deteedate):
month 3,,3, 2,3,,,,,,112 and day or day
12-31-1999
return False
False
if nonth in [4,6, 6,111) and day day 30: neturn False
12/31/1999
HUMANEVAL' input
A bracket is needed!
is a valid date!
Figure 4:
Exemplary incorrect-logic ground-truth solution in HUMANEVAL (#124)
Figure 4 shows an incorrect ground-truth implementation (validate_ ) from HUMANEVAL classified as having bad logie. The desired task to check if the input date format is correct. We see that in the core logic. the conditions attempt to first check the month condition and then handle the corresponding day conditions. However, this is implemented incorrectly as "and" in Python has higher precedence than "or", leading to the ground-truth function to check if either conditions satisfies instead of the desired both conditions must satisfy. This is exposed via our automatically generated test input of 12-31-1999 where the ground-trut implementation incorrectly labels this as not valid date. Surprisingly this egregious error not exposed by any of the base test inputs in HUMANEVAL, further demonstrating the weakness and limited evaluation power of the original test inputs.
4 Related Work
LLMs for code. The use of LLMs for code has gained traction in recent years, owing the abundance
of open codebase and the need for improving developer efficiency. LLMs have demonstrated state-of-the-art performance on various code-related tasks, including code generation [11, 33. 25]. program repair [69, 27, 68, 65], automated testing [15, 14, 67, 35, 71], code translation [31, 55] and code summarization [1. 37J. In particular, prominent LLMMs including CODEX [11], CodeGen [46). INCODER [18] and PolyCoder [70], have been developed and extensively evaluated for code
'https://docs.
http:///occcs.ph...oeeeeee wttpppptttto/eppppppppppp ort:///eeeerrnnee/ppppssssoosss. htt.srooootttttteeeeeererooooor precedencc

---

generation (widely recognized as the holy grail for computer science research since the inception of AI in the 1950s [21]), where the model generates code snippets based on natural language descriptions (e.g., docstring) of the desired functionalityy
Coding benchmark for LLMs. LLM-based code synthesis is largely evaluated based on functional correctness, which is typically assessed by running test-cases to check the desired outputs. HUMANEVAL [11] is one of the pioneering and most widely studied human-writen benchmarks for LLM-based code synthesis, consisting of 164 pairs of Python function signature with docstring and the associated test-cases for correctness checking. Additionally, each HUMANEVAL problem is also equipped with a reference solution. Another Python-focused dataset, MBPP [3]. is created by crowd-sourcing participants to write in summation 974 programming problems, each of which is comprised of the problem statement (i.e., docstring), the function signature, as well as three test-cases. Beyond Python, there are other benchmarks targeting additional languages such as Spider [73] (SQL), HUMANEVAL-X [76] (C++, Javascript and Go), CodeContests [33] (C++ and Java) and MultiPL-E [9] (extending HUMANEVAL and MBPP to 18 programming languages). More recently, researchers have created a more realistic code synthesis benchmark by collecting GitHub issues along with the corresponding code base together with tests to measure the ability of LLMs to perform real-world software engineering tasks [28]. Our work shows for the first time the test inadequacy problem of widely studied benchmarks and addresses the issue via automatic test generation.
Automated test generation. Automated test generation is a widely used for finding software bugs with automatically generated tests. Black-box test generation such as fuzz testing [43] feeds random inputs (e.g., random bytes) to the system under test (SUT), without knowing its source code. Traditional black-box techniques can mainly be categorized into generation- based [72, 23, 56] and mutation- based [66, 10. 47] ones. White-box approaches provide better-quality test-cases by analyzing the source code of SUT. For instance, symbolic execution [30, 8] breaks the coverage plateaus by solving symbolic path constraints to generate tests targeting deep paths. As mid-point, coverage-gus guided fuzzing [74, 57] (i.e, grey-box) uses the coverage information of SUT as feedback to adjust the input generation and mutation. The discussed traditional methods are inapplicable to generating seman- language. We
tically meaningful inputs for arbitrary problems programmed in dynamically-tyyen address this by using ChatGPT to inspect the ground-truth (i.e., white-box) for initializing interesting seeds, based on which type-aware mutation (i.e., black-box) scales the test inputs to large amount.
5 Conclusion & Future Work
We present EvalPlus - rigorous evaluation framework for program synthesis, driven by automated test generation. EvalPlus combines both LLM- and mutation-based input generation to obtain a diverse set of test inputs for accurately evaluating the correctness of LLLM-generated code. EvalPlus creates HUMANEVAL*, built on top of the popular HUMANEVAL with additional high-quality and automatically generated test inputs. With test-suite reduction, EvalPlus also produces HUMANEVAL* -MINI which is smaller than HUMANEVAL* by 47 x while preserving similar test effectiveness. We extensively evaluate a diverse set of LLMs and show that HUMANEVAL* can identify a significant amount of previously undetected wrong code generated by LLMs, demonstrating its effectiveness to augment programming benchmarks for more accurate evaluation.
Since launched, the EvalPlus PyPI package has been installed by over 6k times in 5 months. We also keep evaluating new models for code and maintain a leaderboard at https: / /ttp///lallls../.ee/e gevaapluu.githull github. bo/leaderboard. .html In the future, we plan to apply EvalPlus to bring better-quality testing for more code benchmarks such as MBPP. Meanwhile. future work can look into how to integrate Evalplus with more formal verification (e.g., Dafny [32]) or validation techniques (e.g.. translation validation [36]) to provide stronger guarantees of the evaluation results when applicable. Additionally, the core test generation technique behind can be even used to remind developers of potential flaws of the accepted LLM -enerrttd code snippets when doing Al pair- paiiraroggnmmnng (e.g.. Copilot [42]).
6 Acknowledgements
This work was partially supported by NSF grants CCF-2131943 and CCF-2141474, as well as Kwai Inc. We thank the reviewers for their invaluable feedback. We further thank Yinlin Deng for providing helpful discussions, as well as Junhao Wang and Songrun Xie for their open-source contributions.
10

---

References
[1] T. Ahmed and P. Devanbu. Few-shot training umss for project-specific code-summariiztion, In IEEE/ACM International Conference Automated Software Engineering, pages 1-5, 2022.
[2] L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra,
A. Gu, M. Dey, al. Santacoder: don't 't reach for the stars! arXiv preprint arXiv:2301.03988 2023.
[3] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Tenry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021.
[4] S. Bang, S. Nam, I. Chun, H. Y. Jhoo, and J. Lee. Smt-based translation validation for machine learning compiler. Computer Aided Verification: 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part II, pages 386-407. Springer, 2022.
[5] Black, L. Gao, Wang, C. Leahy, and Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Besh-Tennsrffow, Mar. 2021. If you use this software, please cite using these metadata.
[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[7] A. Budd. Mutation analysis of program data. Yale University, 1980.
[8] C. Cadar, D. Dunbar, D. R. Engler, et al. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume pages 209-224, 2008.
[9] Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costiee D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. JEEE Transactions on Software Engineering, 2023.
[10] S. K. Cha, M. Woo, and D. Brumley. Program-adaptive mutational fuzzing. In 2015 IEEE Symposium on Security and Privacy, pages 725-741. IEEE, 2015.
[11] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, Evaluating large language models trained on code. arXiv N. Joseph, G. Brockman, 2021.
preprint 2r111:207...00337....
[12] W.-L.. Chiang, 7. L., Z.. Lin, Y. Sheng, Wu, H. Zhang, I.. Zheng, S. Zhuang, Y. Zhuang. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
[13] B. Code. Starcoder. https: / //gtth// cwm/co/c///////. poooeee--tttectccoeeeeee 2023.
[14] Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. Large language models zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In 32nd International Symposium on Software Testing and Analysis (ISSTA), 2023.
[15) Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and I.. Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. In 46th International Conference on Software Engineering (ICSE), 2024.
[16] fauxpilot. Fauxpilot: open-source alternative to github copilot server. 2022.
https:
//github.comt wwt.c/om/maaaa wwtp///auppooo//arrugaoeooo
[17] U. Feige. A threshold of in for approximating set cover. Journal of the ACM (JACM), (5(4):634-652, 1998.
[18] D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023.

---

[19] C. Green. Application of theorem proving to problem solving. In Readings in Artificial Intelligence, pages 202-222. Elsevier. 1981.
[20] S. Gulwani. Automating string processing in spreadsheets using input-output examples. SIGPLAN Not., 46(1):317-33,, jan 2011.
[21] S. Gulwani, O. Polozov, and R. Singh. Program synthesis. Foundations and Trends@ in Programming Languages, 4(1-2):1-119 2017.
[22] Hastie, R. Tibshirani, J. H. Friedman, and H. Friedman. The elements of statistical learning: data mining. inference, and prediction, volume 2. Springer, 2009.
[23] C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In 2/ USENIX Security Sym- posium (USENIX Security 12). pages 145-458, Bellevue, WA, Aug. 2012. USENIX Association.
[24] M. Ivanković. G. Petrović, R. Just, and G. Fraser. Code coverage google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages
2019.
[25] Iyer, I. Konstas, A. Cheung, and Zettlemoyer. Mapping language code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1652-1652, 2018.
[26] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arX1v:2310.06822, 2023.
[27] N. Jiang, K. Liu, T. Lutellier, and L. Tan, Impact of code language models on automated program repair. arXiv preprint 2rXiv:2302.05022, 2023.
[28] C. E. Jimenez, Yang, A. Wettig. S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
[29] A. Kalyan, A. Mohta, O. Polozov, D. Batra, P. Jain, and S. Gulwani. Neural-guided deductive search for real-ting time program synthesis from examples. In International Conference on Learning Representations, 2018.
[30] J. C. King. Symbolic execution and program testing. 19(7):385-38) 394, 1976.
Communications of the ACM,
[31] M.-A. Lachaux, B. Roziere, L. Chanussot, and Lample. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511
[32] K. R. M. Leino. Dafny: An automatic program verifier for functional correctness. In International conference on logic programming artificial intelligence and reasoning, pages 348-370 Springer, 2010.
[33] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, Gimeno, A. Dal Lago, et al. Competition-les level code generation with alphacode. Science, 2022.
(1866624):092211999
[34] J. Liu, Y. Wei, S. Yang. Y. Deng, and L. Zhang. Coverage- guided tensor compiler fuzzing with joint ir-pass mutation. Proceedings of the ACM on Programming Languages, 6(00PSLA1):1-26, Apr. 2022.
[35] Z. Liu, C. Chen, J. Wang, X. Che, Y. Huang, J. Hu, and Q. Wang. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1355-1367. IEEE, 2023.
[36] N. P. Lopes, J. Lee, C.-K. Hur, 7. Liu, and J. Regehr. Alive2: bounded translation validation for llvm. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pages 65-79, 2021.
[37] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al. Codexglue: machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04464 2021.
12

---

[38] Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2300000856,, 2023.
[39] Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):151-165, 1971.
[40] W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100-107, 1998.
[41] B. Meyer. Applying design by contract'. Computer, 25(10):40-51, 1992.
[42] Microsoft. GitHub Copilot Your Al pair programmer. copilot, 2023.
[43] B. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32 44, 1990.
[44] G. C. Necula. Translation validation for an optimizing compiler. Proceedings of the ACM SIGPLAN 2000 conference Programming language design and implementation, pages $3-94, 2000.
[45] E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training Iims on programming and natural languages. arXiv preprint, 2023.
[46] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023.
[47] Ochlert. Violating assumptions with fuzzing. IEEE Security & Privacy, 3(2)58862, 2005.
[48] OpenAL. Chatgpt: Optimizing language models for dialogue. chatgpt/,
2022. 2022.
[49] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[50] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguisties, pages 311-318, 2002.
[51] G. Petrović and M. Ivanković. State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Saftware engineering in practice pages 163- 171- 2018
[52] Phind. Phind/phind--odellaaa------ hugging face. https:///ngginngffe.cc coppii/ Phind-CodeLl ama- -aa.lll-aa----------- 22B-v2, 2023.
[53] G. Rothermel, M. Harrold, J. Von Ronne, and C. Hong. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability, 12(4):219-249, 2002.
[54] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
[55] B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint -xXiv22100.0677 2021.
[56] M. Security. jsfunfuzz. https: htttp:///gituub..cm////t/ati http://wwww...aaaeaeeeerrrttttte
2007.
[57] K. Serebryany. Continuous fuzzing with libfuzzer and addresssanitizer. In 2016 IEEE Cybersecurity Development
(SaeeeenntSeeeeee (SecDev), pages 157-157. IEEE, 2016.
[58] D. Shaw, W. R. Swartout, and C. Green. Inferring lisp programs from examples. IJCAI, volume 75, pages 260-267, 1975.
13

---

[59] A. Shi, A. Gyori. M. Gligoric, A. Zaytsev, and D. Marinov. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pages 246-256, 2014.
[60] Stability-AI. Stablelm: Stability ai language models. httpp://ggichb...ommSaaaaiit
StableLM, 2023.
[61] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[62] R. J. Waldinger and C. Lee. Prow: step toward automatic program writing. In Proceedings of the Ist international joint conference on Artificial intelligence, pages 241-252, 1969.
[63] B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
trans frransformer-jaarr May 2021.
[64] Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 2023.
[65] Y. Wei, C. S. Xia, and L. Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. arXiv preprint (wX.:i33390000000 2023.
[66] D. Winterer, C. Zhang, and Z. Su. On the unusual effectiveness of type-aware operator mutations for testing smt solvers. Proceedings of the ACM on Programming Languages, 4(00PSLA):1225, 2020.
[67] C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models. 46th International Conference on Software Engineering (ICSE), 2024.
[68] C. Xia, Y. Wei, and Zhang. Automated program repair the era of large pre-trained language models. In Proceedings of the 45th International Conference Software Engineering (ICSE 2023). Association for Computing Machinery, 2023.
[69] C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program re- pair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 959-971, 2022.
[70] F. Xu, U. Alon, G. Neubig, and J. Hellendoom. A systematic evaluation of large language models of code. In Proceedings of the ACM SIGPLAN International Symposium on Machine Programming, pages 1-10, 2022.
[71] C. Yang, Y. Deng, R. Lu, J. Yao, J. Liu, R. Jabbarvand, and L. Zhang. White-box compiler fuzzing empowered by large language models, 2023.
[72] X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in e compilers. In Proceedings of the 32nd ACM SIGPLAN Conference Programming Language Design and Implementation, PLDI '11 page 283-294, New York, NY, USA, 2011. Association for Computing Machinery.
[73] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li. J. Li, Q. Yao, Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018
[74] M. Zalewski. American fuzzing lop (afl). https://lcamtuf. wow.dump..... coreeump.cx//fl// cx//ff//2211
[75] L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In 2011 IEEE 22nd International Symposium on Software Reliability Engineering, pages 170-179. IEEE, 2011.
IyternationalSSmposiin
[76] Q Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang. Y. Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303. 2023.
14

---

Table 5: Overview of evaluated models.
Model Name
Sizes
Release Year
Open- Source
CodeGen [46]
2B, 6B, 16B
2022
INCODER [18]
1.3B, 6.7B
2022
PolyCoder [70]
2.7B
2022
SantaCoder [2]
1.1B
2023
Coding
CodeGen2 [45]
1B, 3B, 7B, 16B
2023
StarCoder [13]
15B
2023
CODETS+ [64]
16B
2023
CODELLAMA [54]
7B, 18,,1B,,4BB
2023
WirardCooder-CodeLLanm [38]
34B
2023
Phind-CodeLLama [52]
34B
2023
GPT-J [63]
6B
2021
GPT-NEO [5]
2.7B
2021
General
ChatGPT [48]
N/A
2022
GPT-4 [49]
N/A
2023
VICUNA [12]
7B, 13B
2023
StableLM [60]
7B
2023
MISTRAL [26]
7B
2023
A Detailed Experimental Setup
Evaluation of LLMs. Our goal is to comprehensively evaluate recent and widely used LLMs, both specialized for code generation [46, 70, 18, 2, 52, 38, 64] and general-purpose tasks [49, 48, 12, 60, 63, 5, 26]. Table 5 presents overview of the studied models, with column Sizes reflecting the model sizes billions of parameters, Release Year showing when the LLM is released, and Open-Source marking the models whose weights publicly available. In total, we evaluate 26 of the most representative and popular LLMs with a broad range of configurations fully demonstrate the generalizability of our results.
configurations follow prior work [11, 46). For each model we randomly sample 200 programs and repeat the experiments over temperature ((0.2,0.4,) 0...2.004006..... 4,0.6,0.8)) and greedy decoding with zero temperature. By default, we let each model generate at most 512 new tokens and truncate the produced code with end-of-string (EOS) identifiers suggested in HUMANEVAL [11]. well as those favoured by certain models (e.g.. "<I endoftext |>" and "\n" "). For conversational models (i.e., ChatGPT and GPT-4), we obtain the code fragments by parsing the code blocks (i.e., within ... """"") in the output. We found ChatGPT tends to repeat problem description with detailed explanation, which can consume more than 512 new tokens to complete solution for around 11 % of problems. To align ChatGPT with other models, for tasks with very long problem descriptions, we extend the token limit from 512 to 1024. For model implementation, we run ChatGPT and GPT-4 via OpenAl APIs, and accelerate CodeGen-6B and -16B with NVIDIA FasterTransformer via FauxPilot [16]. All other LLMs are based on the HuggingFace transf ormers library. By default, we follow the official examples of each LLM (e.g.. on HuggingFace model card) to construct their corresponding prompts. Specifically,, the prompts used for ChatGPT, GPT-4, and WizardCoder-Codee CodeLlama is instruction-based i.e., simple instruction is used to wrap the function signature and docstring explicitly encourage the L.MM for code generation.
Our
Test oracles. An L.LM-produee solution regarded to be correct if for all test inputs returns values that match the expected outputs within reasonable run time. We perform exact matching by default. For floating-pote
comparisons, we tolerate absolute differences to the degrees annotated in HUMANEVAL or 10 if not annotated. In original HUMANEVAL, the default timeout is set to three seconds to run the whole test-suite (i.e., all test-cases) for each programming problem. Such setting is neither suitable when having more test-cases run time characteristics. Consequently, we let the timcout for each test-case to be max max(200ms, (200ms, 4 ft)) where taz refers to the execution time of the corresponding ground-truth solution. In other words, expect the A.LM-provide solution to be no slower than the ground-truth by four times or use a base milliselise timeout when 4 x ttt 200ms to avoid variance caused by performance randomness.
reasonable each problem could have its own
15


Reviews:
[{"review_id": "De9AUUNT8S", "reviewer": "Reviewer_yy3N", "summary": "This paper describes an enhanced test dataset and test-driven evaluation of code generation by LLMs. The paper compares different LLMs and a stricter metrics using the enhanced evaluation dataset and shows that these LLMs are about 15% less correct than is reported based on earlier test datasets. ", "questions": "1. Have you tried any A/B experiment or other human evaluation to see how much more useful is this improved test set to the LLM context in a real application? \n2. Have you thought about other LLM tasks which have similar challenges. \n3. As LLMs are constantly improving likely the value of such an enhanced test set diminishes. Have you thought about studying that? ", "limitations": "Limitations - LLMs are used in context of human input. So it is not clear how useful this enhanced data is. It is also not clear if the enhanced test set increases coverage in criticial dimensions. \nLLMs are constantly improving and in the absence of a good A/B test or application context it is hard to tell the usefullness of the system.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Given that LLMs appear to achieve a number of programming tasks but often fail to completely get them right. They fail in unexpected ways and places. Improving the evaluation metrics is critical to use of LLMs in product and commercial contexts. This paper improves the test beds by synthetically creating addtional tests through a framework that uses both LLMs and mutation based techniques. \nThe paper demonstrates that about 13-15% of code generated by typical LLMs that would be qualified as passable according to previous metric is disqualified by this method. Since LLMs are often used for NL->code contexts this problem may arise because of inssufficient tests or because of less specific description in NL of the code to be written. They increase the test suites by 81X and also create a reduced version of it that shrinks it 47x for fast evaluation.\n\nThe test suites are enhanced by augmenting the prompts with ground truth tests and generating higher quality seed data. From these seed inputs, type-aware mutations are created to enhance the test data sets. \n\n\n", "weaknesses": "While the enhanced data set does call out more problems in the LLM-generated code, it is not clear if it is the best it can do. For instance, just by running a static analyser, a syntax checker or some other engine, the issues in the code generated can be found and fixed. The authors should have at least compared one such approach. There is no comparison of another approach that improves the performance of LLMs in context. One of the ways LLMs are often used for code generation is to generate good enough code and then through human interaction or through other static tools rerank, correct or qualify the generated code. \nThe enhanved data set is diefinitely useful but not clear how useful in that it flags about 13-15% of the code. Some human evaluation or A/B test to qualtitatively say how good this is would have been more insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:Have you tried any A/B experiment or other human evaluation ...  in a real application?\n\nIn this work, we focus on improving the evaluation of functional correctness that can be deterministically, objectively and automatically measured through testing and verification techniques. Contrastively, A/B testing or other human evaluation [1] can be useful for understanding the overall usefulness of NLP applications whose correctness is hard to evaluate systematically, but can incur high manual efforts. Please note that we did in fact manually examine the LLM generated samples that had passed the original HumanEval tests but failed on our EvalPlus tests and found that such solutions can contain subtle but important logic errors (e.g., Figure 1). However, due to the sheer number of samples generated (>45,000), manually performing A/B testing or human evaluation on all samples would be infeasible. \n\nThe HumanEval+ benchmark produced by EvalPlus can better reflect the true performance of LLM code synthesis. Not only do we show the average performance drop is around 15%, but even widely used open-source (e.g., CodeGen) and proprietary state-of-the-art LLMs (e.g., ChatGPT/GPT-4) suffer from significant performance decreases (13% to 19%).  As LLMs become more widely used for program synthesis, it is critical to ensure the functional correctness of LLM generated code, with the first step being developing a rigorous benchmark as we did in this work. Furthermore, the input generation proposed in EvalPlus can also be applied to test real-world projects and we hope EvalPlus can inspire more future work in helping developers more rigorously evaluate code for real-world projects. \n\n[1] Zheng, Lianmin, et al. \"Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.\"\n\n> Q2:Have you thought about other LLM tasks which have similar challenges.\n\nThanks for this interesting question! Comprehensively evaluating LLMs is definitely a common challenge in many LLM tasks. In this work, we target program synthesis as it is the holy grail of computer science since the 1950s, with many recent new LLMs including program synthesis as a key evaluation metric. We believe the general idea and approach of EvalPlus can also help improve datasets, such as MathQA and GSM8K in mathematical reasoning tasks.\nMore generally, it is an open research question to apply automated software testing to improve the evaluation of other LLM tasks. For example, consistency checks [2] can automatically evaluate the self-consistency [3] of LLMs, which is a form of “metamorphic testing” -- leveraging multiple inputs and their metamorphic relationship as test oracle. For instance, we can ask an LLM to evaluate two semantically equivalent chess positions to check if the resulting evaluation is also equivalent.\n\n[2] Fluri, Lukasi, et al. \"Evaluating Superhuman Models with Consistency Checks.\" \n\n[3] Wang, Xuezhi, et al. \"Self-Consistency Improves Chain of Thought Reasoning in Language Models.\" \n\n> Q3:As LLMs are constantly improving likely the value of such an enhanced test set diminishes?\n\nIt is true that EvalPlus and any software testing technique is useless when we have perfect LLMs. However, please note that this is the ultimate goal for the community and still requires long-term efforts. In fact, our work also aims to contribute towards such an ambitious goal: EvalPlus helps to better evaluate all code models that are still imperfect. We believe it is extremely important to precisely understand the strengths and limitations of existing code models to make informed decisions in improving them, and hope this initial work can inspire more researchers to join this effort. \nMeanwhile, please also note that while models are getting stronger (e.g., ChatGPT to GPT-4), the performance decrease (i.e., the value of such an enhanced test set) does not necessarily become smaller: according to our experiments, while the pass@1 value of ChatGPT drops by 15.5% when using HumanEval+, it drops by 16.0% for GPT-4.\n> C1:just by running a static analyser, the issues in the code generated can be found and fixed.\n\nThanks for the suggestion. We agree that static code analyzers can detect compile-time (e.g., syntactical error) and simple runtime errors (e.g., wrong arguments). However, code with syntactical errors would already fail the original HumanEval tests, and thus simple syntax checkers cannot help detect any incorrect solutions missed by the original HumanEval. In addition, more advanced static analyzers can only handle very limited types of errors, and are well known to have high false positive rates. For example, Pyright, a state-of-the-art analyzer, can only detect 1 of the 20 incorrect solutions detected by HumanEval+ for GPT-4, and at the same time reported various false positives. As a result, almost all the popular code generation datasets use testing to validate the generated solutions, and EvalPlus also focuses on enhancing tests for such popular datasets.\n\n> C2:no comparison of another approach that improves the performance of LLMs in context. \n\nGreat comment. Please kindly note that our main technique is to rigorously evaluate instead of directly improving LLMs. As a result, we focus on evaluating a large spectrum of LLMs. Given the large number of LLMs studied, we only focus on the default application scenario for each model. Meanwhile, since our main findings hold on all the studied LLMs, our results may very likely further generalize to more LLMs or even LLMs with more advanced in-context learning (e.g., with chain-of-thought or execution feedback). For example, even the recent prompting techniques using execution feedback still rely on the existing tests in the dataset, and may still produce incorrect solutions overfitting to the dataset. Thanks again for the comment, and we can surely add more experiments to further validate this argument. \n"}, {"review_id": "luC0daZVNg", "reviewer": "Reviewer_SGiT", "summary": "This paper introduces a rigorous evaluation framework EvalPlus for program synthesis driven by automated test generation. For automated test generation, this work proposes to combine both LLM-generated tests and mutation-based input generation to largely augment the text inputs for an existing code benchmark of HumanEval. EvalPlus is evaluated with a diverse set of code LLMs and results show that the new augmented benchmark can identify a significant amount of previously undetected wrong code generated by LLMs, leading to a more accurate evaluation. It further introduces a mini-version benchmark reserving almost the same test effectiveness.", "questions": "* Can you explain more on the difference between EvalPlus and AlphaCode/CodeT?\n* I noticed that this work compares with the recently released StarCoder and CodeGen2, but not another SoTA code LLM of CodeT5+ which was released at the same time with better results on HumanEval. Any reasons for not including it?\n", "limitations": "I did not find any discussion on limitations from the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper studies how to evaluate code LLMs more accurately, which has become an important topic in the era of LLMs as coding is one of the key capabilities of LLMs to showcase. The proposed EvalPlus of combining both LLM-generated tests and mutation-based tests is well motivated and technically sound.\n* The paper is well written and easy to follow. The evaluation is very comprehensive (considering most of state-of-the-art LLMs) and provides convincing results to authenticate the effectiveness of the proposed EvalPlus.\n", "weaknesses": "* The biggest weakness of this paper is the lack of discussion and comparison to other related work of AlphaCode and CodeT [1], which share common techniques such as mutation-based input generation and unit tests generated by LLMs. In this regard, the novelty of EvalPlus is relatively limited. Note that in AlphaCode, they have already explored similar mutation-based techniques to largely augment the test cases for program synthesis. Besides, they also explored training a LLM-based test input generation model to generate test cases in a clustering process. For CodeT, they have explored using LLMs to generate test cases, though these generated test cases are used for a different purpose of reranking the generated programs. The authors should discuss and compare with these works to better justify their novelties.\n\n[1] CODET: CODE GENERATION WITH GENERATED TESTS\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:Can you explain more on the difference between EvalPlus and AlphaCode/CodeT?\n\nGreat question. First, we want to clarify that our main contribution is not the input generation technique but rather our generated dataset (HumanEval+) and accompanying rigorous study on recent popular LLMs. In short, we demonstrate for the first time that: 1) existing popular datasets (i.e. HumanEval) used to evaluate almost all recent LLMs for code are not reliable, containing not only insufficient tests but also incorrect groundtruths, 2) such deficiencies in existing datasets can drastically affect the evaluation results of almost all recent LLMs for code, with around 15% decrease in performance when using HumanEval+ compared to base HumanEval. EvalPlus can help researchers more precisely understand the strengths and limitations of existing models, and can in turn help build more powerful models in the near future. \n\nIn terms of the comparison with AlphaCode and CodeT, while indeed they both use LLM-based input generation, the key difference is that both AlphaCode and CodeT only use the LLM generated inputs for clustering in order to determine the ranking of samples for evaluation. In fact, none of them leveraged the LLM-generated inputs to filter out incorrect solutions, since they do not have an exact oracle to ensure the generated tests are correct. In contrast, in EvalPlus, we directly leveraged the generated test inputs to perform differential testing across the groundtruth implementation and LLM-generated solutions to detect any potential incorrect solutions. Furthermore, compared with the simple mutation operators used in AlphaCode to augment the evaluation tests (different from their LLM-based test input generation that is only for clustering), our type-aware mutation approach also includes an additional mutation operator to collect data fragments, from previously generated inputs, and reuse them during later mutation. This allows our mutation strategy to generate more structurally aware inputs that are likely to pass the structural constraints of certain tasks (e.g., need to be palindrome or open/close bracket strings).\n\nThanks again for these great references and we will surely discuss more such related work in our revision.\n\n> Q2:I noticed that this work compares with the recently released StarCoder and CodeGen2, but not another SoTA code LLM of CodeT5+ which was released at the same time with better results on HumanEval. Any reasons for not including it?\n\nAdditional models are not included due to the lack of time and space. After submission we have been improving EvalPlus and adding more outstanding models including CodeT5+. Specifically, here are the pass@1 (i.e. greedy) results for the 2B, 6B and 16B CodeT5+ models. Overall, our main findings still hold on CodeT5+. Note that using 512 as the maximum new token size (i.e., the default setting of our paper) can cause OOM on A6000-50G for certain problems. Consequently we dynamically reduce the new-token size by 0.8 until OOM is overcome.\n\n| Model |  | pass@1 |\n| :---        |    :----:   |          ---: |\n| CodeT5+ 2B      | base      | 25.0  |\n|  | +extra      | 22.0 (-12%)  |\n| CodeT5+ 6B     | base      |29.3 |\n|  | +extra      | 23.8 (-19%)  |\n| CodeT5+ 2B      | base      | 31.7  |\n|  | +extra      | 26.2 (-17%)  |"}, {"review_id": "ooL4RXHIMT", "reviewer": "Reviewer_9v6g", "summary": "The authors propose an evaluation framework for validating the correctness of large language model-generated code. In particular, the framework first utilizes ChatGPT to generate multiple seed inputs, which are then expanded into a large set of inputs through type-aware mutation. In addition, to ensure evaluation efficiency, the input set can be reduced by employing code coverage, mutant killings, and LLM sample killings.", "questions": "- I wonder if GPT-4 can generate more interesting seed inputs than ChatGPT. Explicitly clarifying the employed component in EvalPlus may affect the generality of the solution, I suggest this can be introduced as the implementation or setup information.\n- The authors can elaborate on how to filter out the invalid inputs in Section 2.1.\n- The evaluation does not examine the efficiency of type-aware input mutation, and test-suite reduction.\n- Table 4: “Killed samples” has ambiguity.\n- Section 2: “EvalPlus obtains a augmented benchmark …” -> “an”.\n", "limitations": "The authors do not explicitly identify the potential societal impact or limitations of this work, but they claim that their future work include conducting testing to more code benchmarks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed solution addresses the problem of insufficient testing for LLM-generated code. The expanded dataset can outperformance the original one, while the reduced dataset achieves similar results as the expanded one. \nThe authors evaluate the proposed solution via comparative analysis of 19 large language models, analyze the pass rate distribution of the employed dataset HUMANEVAL, and identify several errors in the ground-truth solutions. \nThe related work section discusses previous research on large language models for code, the coding benchmark, and automated test generation. The authors discuss the research gap between this study and the related work. ", "weaknesses": "The authors do not employ a more capable large language model in the proposed solution, and there is a lack of evaluation of efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:I wonder if GPT-4 can generate more interesting seed inputs than ChatGPT.\n\nPlease note that the EvalPlus input generation component does not rely only on ChatGPT but is general and can be implemented using any other foundational models like GPT-4 or LaMDA. We have also thought about using GPT-4 to generate additionally interesting seed inputs. However, due to the high monetary cost of invoking the GPT-4 API (on 164 different HumanEval tasks will cost more than 600 dollars compared to only 60 dollars using ChatGPT), we choose to only use ChatGPT for our work. In fact, ChatGPT has already shown impressive performance generating high quality seed inputs. Thanks again for the suggestion and we will further use GPT-4 for seed generation (which will very likely lead to even better performance as mentioned by the reviewer) in the future. \n\n> Q2:Elaborate on how to filter out the invalid inputs in Section 2.1.\n\nThis has been elaborated in Section 2.3. In short, we adopt a programming by contract philosophy by systematically adding code assertions as contracts to ensure the test inputs for the function are well-formed. Invalid inputs during generation will trigger assertion failures and thus be detected and dumped by EvalPlus. \n\n> Q3:The evaluation does not examine the efficiency of type-aware input mutation, and test-suite reduction.\n\nThanks for the suggestion and we will emphasize the efficiency results in our next revision.\nPreviously we did not rigorously study the efficiency of input generation and reduction because these are one-time efforts for each dataset (can be reused to evaluate any current or future code models) and can be done reasonably fast. Specifically, EvalPlus is able to generate on average 1100+ valid tests for each of the 164 problems in half an hour (i.e., ~3000 valid tests/hour). The test reduction for all 164 problems in total can also finish in one hour. \n\n> Q4: Table 4: “Killed samples” has ambiguity.\n\nWe apologize for the ambiguous term. “Killed samples'' refers to the number of incorrect LLM generated samples that are “killed” (i.e. failed a test) by a test-suite. Please see Section 2.2 for more detail. We will make the term less ambiguous.   \n"}, {"review_id": "VE4JznRiF3", "reviewer": "Reviewer_VxVU", "summary": "In this paper, the authors introduce EvalPlus, an evaluation framework crafted to assess the code generation of LLMs. Combining LLM and mutation-based techniques, EvalPlus diversifies the generated test inputs, thereby broadening the evaluation spectrum for LLM-produced code. Through comprehensive experimentation, the authors highlight EvalPlus's ability to expose shortcomings in previous LLM-based code evaluations. Additionally, the study brings HumanEval Plus, an enriched dataset built upon the existing HumanEval, offering an abundance of test inputs and a more dependable ground truth for improved reliability.", "questions": "* Given the impressive results demonstrated by EvalPlus within the comparatively simpler context of the HumanEval benchmark, how do you envision expanding this work to more complex environments? Specifically, what strategies or modifications do you plan to implement to ensure EvalPlus is effective in assessing code generated for real-world projects, such as those found in open-source Python libraries? Could you elaborate on potential challenges you anticipate and how you intend to navigate them in this expansion?", "limitations": "* While this study presents a promising step forward with EvalPlus, a limitation lies in the evaluation scenario chosen for testing. Given that the proposed method is intended to be general, using the relatively simple HumanEval dataset for testing may not sufficiently demonstrate the framework's generalization capabilities. To conclusively assert its wide applicability and robustness, it would be beneficial for future research to include tests on more complex datasets or in more challenging real-world contexts, such as open-source Python libraries. This would provide a more comprehensive understanding of EvalPlus's potential in diverse, practical applications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper distinguishes itself through its innovative approach to test input generation. It extends beyond the conventional usage of ChatGPT for creating test inputs, by incorporating a test-mutation technique and a “distill” method. This results in a diverse yet non-redundant range of test inputs, illustrating the authors' commitment to crafting a robust and efficient evaluation framework.\n* This paper is well-written and easy to read.\n* The study stands out for its well-designed experiment, which encompasses a comprehensive selection of contemporary LLMs. ", "weaknesses": "* HumanEval [8], in comparison to real-world coding projects, presents a significantly simpler challenge. It's a benchmark that's even simpler than others, such as APPS [Hendrycks et al.]. Given this context, the effectiveness of EvalPlus when applied to more complex tasks remains uncertain. This indicates a valuable area for enhancement in this study—specifically, future research could delve into assessing how well EvalPlus performs in more complicated, real-world coding scenarios. Such explorations could contribute crucial insights to understanding the scalability and adaptability of EvalPlus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  Q1:how do you envision expanding this work to more complex environments? (strategies, modifications and challenges to ensure EvalPlus is effective in assessing code generated for real-world projects?) \n\nTo begin with, we want to re-emphasise that our main contribution is to show that the existing popular benchmarks for LLM-based code synthesis evaluation (e.g., HumanEval) contain insufficient tests that cannot faithfully evaluate the functional correctness for LLM generated code. As such, the reported performance of LLMs when evaluated on these benchmarks can be inaccurate or exaggerated (compared to their actual performance). For example, this work shows for the first time that almost all recent LLMs for code can be affected by such dataset issues. Furthermore, the paper also helps more precisely understand the strengths/limitations of existing LLMs via automated test generation/reduction, and can in turn help build more powerful LLMs in the near future. \n\nMeanwhile, this is a great question that allows us to discuss potential future work! First, we plan to expand EvalPlus by taking in more knowledge and context from the entire project rather than a single function. For example, developer documentation can be directly used as input to help ChatGPT synthesize interesting inputs that not only invokes a single function but also multiple functions or API sequences. Second, one key challenge is that unlike the benchmarks that we seek to improve in this paper, real-world projects do not have the exact groundtruth solution. In order to address this challenge, we can instead apply partial test oracles such as 1) crashes: to discover rare inputs that trigger crashes in developer code (e.g,. segmentation faults), 2) differential testing: to evaluate developer code on two different setups (i.e. CPU vs GPU for deep learning programs) to discover bugs, and 3) LLM-based oracle generation: we can even leverage the generative and code understanding power of LLMs themselves to generate the oracle.These partial oracles may not give us the same guarantees as using the reference groundtruth solution, but EvalPlus can still leverage these oracles to assist developers in more complex real-world systems. \n"}, {"review_id": "eM7hmB9aKm", "reviewer": "Reviewer_jsgr", "summary": "This paper leveraged large language model (LLM) based and mutation-based strategies to generate high-quality test cases for the popular dataset HumanEval. The extended dataset HumanEval+ provides a better code generation benchmark for assessing the performance of LLMs such as ChatGPT and GPT4. Experimental results showed that compared to the HumanEval dataset, the extended one can detect more issues of the generated code, where the generation performance of 19 LLMs is reduced by 13.6-15.3% on average in terms of pass@k. To save the testing time, this paper also provided a refined set of test cases with fewer numbers but with the same code coverage, killed mutants, and sample killings.\n\n", "questions": "How is the quality of the seed inputs generated by ChatGPT? Are these inputs correct? How do you ensure the quality of the generated seed inputs? and how many seed inputs did the ChatGPT generate? What are the prompts used? Will the quality of seed inputs affect your approach significantly?", "limitations": " The authors addressed the limitations in the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Proposing a test case generation method for LLMs, which combines the LLM-based and mutation-based strategies.\n\n+ Evaluating 19 LLMs on the extended datasets and showing the overestimation of the original dataset.\n\n+ Providing a test-suite reduction for quick evaluation.\n", "weaknesses": "- Some technical details, such as the quality of seed inputs, are unclear, which affect the soundness of the proposed approach.\n\n- There are some unclear claims/overclaims.\n\n- Lack of comparisons with other test case generation methods and related work.\n\nThe claimed contribution indicates “that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis”. Can the extended dataset proposed by this paper reflect the “true performance”? Similarly, the paper said that the existing tests “often fall short in capturing all possible scenarios”, leading to “false confidence” in the results. Can your test cases “capturing all possible scenarios”? Furthermore, this paper claimed that the proposed benchmark can “precisely evaluate the functional correctness of LLM-generated code” by generating “interesting test inputs”. It is unclear to me how “precise” can the evaluation provide and what are the “interesting test inputs”. Please clarify them.\n\nEvalPlus “first uses ChatGPT to generate a set of high-quality seed inputs for later mutation”. The quality of the seed inputs is not verified. Are these inputs correct? How do you ensure the quality of the generated seed inputs? and how many seed inputs did the ChatGPT generate? What are the prompts used?\n\nThe type-aware input mutation “randomly” selected some inputs from the generated seed pool as the inputs of mutants. This pipeline did not clearly describe how randomness affects the result of the input mutation and the quality of the evaluation.\n\nThe proposed approach “adopted a programming by contract philosophy by systematically adding code assertions as contracts (e.g., assert n > 0) to ensure the test inputs for the function are well-formed”. The context did not tell how to define the assertions specifically, how to ensure the correctness of the definitions, and which docstrings need clarification with these assertions. Will it incur a lot of manual effort?\n\nFor the evaluation, the paper did not compare the method with other baseline methods. The paper says that traditional automated test generation methods are inapplicable to generating semantically meaningful inputs. Why is that so? More descriptions are needed here. Otherwise, it is difficult to determine the advantages of the proposed method over the traditional methods.\n\nRecently, there is also a related work on applying ChatGPT prompts to code generation, the authors also discussed the quality/correctness of the generated code: C. Liu et al., Improving ChatGPT Prompt for Code Generation, https://arxiv.org/abs/2305.08360.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:How is the quality of the seed inputs generated by ChatGPT and will the quality of seed inputs affect your approach significantly?\n\nThe quality of seed inputs and the effect it has on mutation-based test generation and fuzzing has been well-studied in prior work [5, 6]. Similarly, the quality of seed inputs will also affect our approach significantly, and that is exactly why we are leveraging ChatGPT to generate high-quality seed inputs. We found that compared to the original HumanEval test, by adding the 30 high-quality seeds generated by ChatGPT, we can already improve coverage from 96.7% to 98.4%, and finally decrease the average pass@k by 11% (full EvalPlus is around 14%). This demonstrates the quality of the seed inputs generated by ChatGPT.\n\n[5] Rebert, Alexandre, Sang Kil Cha, Thanassis Avgerinos, Jonathan Foote, David Warren, Gustavo Grieco, and David Brumley. \"Optimizing seed selection for fuzzing.\"\n\n[6] Pailoor, Shankara, Andrew Aday, and Suman Jana. \"MoonShine: Optimizing OS fuzzer seed selection with trace distillation.\"\n\n> Q2:Are these inputs correct?\n\nThe raw inputs generated by ChatGPT are not guaranteed to be correct. However, as mentioned in Section 2.3, EvalPlus uses program input contracts to filter out ill-formed inputs and the ones satisfying contract conditions are bound to be correct. \n\n> Q3:how many seed inputs did the ChatGPT generate?\n\nPlease kindly see Section 3 for the discussion. For each programming task, we generate 30 seed inputs using ChatGPT.\n\n> Q4:What are the prompts used?\n\nWe apologize for not including the detailed prompt in our supplementary material (we did show an overview of the prompt in Figure 2), and will include the detailed prompt in the appendix of the next revision. Below is our detailed prompt used for seed generation.\n```\nHere is a function that we want to test:\n[[FUNCTION]]\nThese are some example inputs used to test the function:\n[[EXAMPLES]]\n[[INSTRUCTION]]\n```\nSpecifically, the meanings of these macros are:\n[[FUNCTION]]: The ground-truth solution of the programming task\n[[EXAMPLES]]: (At most) 5 inputs randomly selected from original HumanEval\n[[INSTRUCTION]]: We randomly select one of the following instructions:\n“Please generate complex inputs to test the function”\n“Please generate corner case inputs to test the function”\n“Please generate difficult inputs to test the function.”\n\n> C1: Lack of comparisons with other test case generation methods and related work.\n\nWe first want to clarify that our main contribution is not our test generation techniques but instead is our generated dataset and accompanying rigorous evaluation study on recent popular LLMs, which demonstrates for the first time that prior popular benchmarks are insufficient to evaluate functional correctness of LLM generated programs. Regarding the comparison with prior test case generation methods, there has been a ton of work on automatically generating tests for programs [7, 8] (see Section 4 for more detail). These prior work mainly focused on either generating new inputs (generation-based) or mutating existing seeds (mutation-based) to create new test cases. Unfortunately, discussed traditional methods are mostly inapplicable to generating semantically meaningful inputs for arbitrary problems programmed in a dynamically-typed language. We address this by using ChatGPT to inspect the ground-truth (i.e., white-box) for initializing interesting seeds, based on which type-aware mutation (i.e., black-box) scales the test inputs to a large amount. Of course, we fully agree with the reviewer that our work may inspire more test generation techniques for this important domain. We will also work towards adding more discussion and comparison with previous test case generation methods in the next revision. \n\n[7] Fraser, Gordon, and Andrea Arcuri. \"Evosuite: automatic test suite generation for object-oriented software.\"\n\n[8] Serebryany, Kosta. \"Continuous fuzzing with libfuzzer and addresssanitizer.\"\n"}]

Author rebuttal:
We thank all the reviewers for their insightful comments and suggestions to improve the paper! We address the main questions (labeled as Q) and concerns (labeled as C) in the response to individual reviewers below. Furthermore, we will also revise the paper accordingly to address all other minor suggestions and comments. 

Please kindly let us know if there is any misunderstanding of the questions, and we are very happy to further communicate with all the reviewers during the reviewer-author discussion period (Aug 10-16).

Meta-review: The reviewers found this paper to be a valuable contribution. The rebuttal addressed several concerns.
Decision: Accept (poster)