Title: Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: LLM4Code

ChatGPT

Automated Test Generation
Abstract: Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus – a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.

Is Your Code Generated by ChatGPT Really Correct?

Abstract

Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus - a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HUMANEVAL benchmark by 80 × to build HUMANEVAL + . Our extensive evaluation across 26 popular LLMs ( e.g., GPT-4 and ChatGPT) demonstrates that HUMANEVAL + is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@ k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HUMANEVAL + , while none of them could on HUMANEVAL. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.

1 Introduction

Automatically generating programs that accurately correspond to user intents is a long-standing challenge in computer science known as program synthesis [21]. In the past few decades, classical program synthesis techniques have been developed, including deductive synthesis [19, 39, 62], inductive synthesis [20, 58] and neural-guided synthesis [29]. More recently, with the advent of Large Language Models [61, 6] (LLMs) and the abundance of open codebase, researchers have been focusing on applying LLMs for direct code generation. LLMs like CODEX [11] and CodeGen [46]

def common(l1: list, l2: list):

"""Return sorted unique common elements for two lists"""

common_elements = list(set(l1).intersection(set(l2)))

common_elements.sort()

return list(set(common_elements))

[5,3,2,8], [3,2]

[4,3,2,8], [3,2,4]

[4,3,2,8], []

[]

[2,3]

[2,3,4]

[6,8,1], [6,8,1]

[8,1,6]

HUMANEVAL

inputs

HUMANEVAL

+

input

not sorted!

ChatGPT synthesized code

correct

perform code generation by autoregressively predicting the next token given previous context, in the form of function signature and docstring that denote the desired program functionality. The generated code snippet is then combined with the context to form a complete function that aligns with the user intent. Leveraging both natural language understanding and generative power, LLMs have demonstrated impressive performance in code synthesis [3, 11].

The primary concern when it comes to LLM-generated code is correctness. Because two dramatically different code snippets can be semantically equivalent, classic NLP metrics like BLEU score [50] are no longer reliable in the context of program synthesis. Ideally, we would like to formally verify the correctness of LLM-provided solutions for any input, but verifying domain-specific problems through methods such as translation validation [36, 44, 4] is already challenging enough, let alone building a general verifier with absolute certainty to prove arbitrary problems, including those in code benchmarks. As such, existing code benchmarks ( e.g., HUMANEVAL [11]) heavily rely on manually constructed test-cases to evaluate LLM solutions. However, these tests often fall short in capturing all possible scenarios, as crafting high-quality tests is laborious. Consequently, we argue that current programming benchmarks are inadequate for assessing the actual correctness of LLM-generated code, leading to false confidence in the results. Specifically, we have identified the following common limitations in existing LLM-for-code benchmarks:

Insufficient testing. Current programming benchmarks often only include on average less than 10 tests for each coding problem. Furthermore, these tests are relatively too simple to fully explore the functionality of the code or corner cases. Figure 1 shows an incorrect code sample synthesized by ChatGPT [48] to return the sorted unique common elements from two lists. At first glance, the function looks correct and computes the desired output when using the base test inputs from HUMANEVAL. However, in the return statement, it incorrectly converts the intermediate list to a set which no longer preserves the order of the sorted list. This example shows that a logically flawed solution can still pass all simple tests and be misconsidered as correct due to testing inadequacy.

Imprecise problem description. The input for code generation includes natural language descriptions in addition to the function signature. These task descriptions in existing benchmarks are oftentimes too vague to fully clarify the expected program behaviors. For example, the input docstring may not specify the expected input domain ( e.g., only positive integers) or how the function should handle exceptions. As a result, such programming problems can be interpreted differently by LLMs against the actual tests, leading to capable LLMs misjudged as incapable.

These limitations are common across many popular code generation benchmarks [11, 3, 33]. This not only questions the validity of the impressive performance claimed by prior work but also sets a challenge on how to properly evaluate the LLM coders. In this paper, we aim to address this fundamental evaluation challenge and ask the introspective question: Is the code generated by LLMs really correct?

Our proposal. In this work, we set out to answer the important question and evaluate the evaluation dataset. Consequently, we build EvalPlus - an evaluation framework to improve existing code benchmarks in order to precisely evaluate the functional correctness of LLM-generated code. At the heart of EvalPlus is an automatic test input generation engine which augments existing code benchmarks by generating interesting test inputs to fully exercise the code solution and check its functional correctness by cross-checking the ground-truth implementation. Specifically, EvalPlus adopts both LLM- and mutation-based [57, 74, 47] methods to automatically generate and diversify additional test inputs. EvalPlus first uses ChatGPT [48] to generate a set of high-quality seed inputs that aim to test difficult corner cases and functionalities of the program within the valid input structure. Using these high-quality seed inputs, EvalPlus then performs type-aware mutation to efficiently generate a large number of additional test inputs. These newly generated test inputs are then used to evaluate the LLM-generated code through differential testing [40] against the ground-truth implementation. Furthermore, as an option to speed up evaluation, EvalPlus also builds minimal test-suites by only

def sample_56(input):

...

def sample_2(input):

...

def sample_1(input):

...

input

def groundtruth(input):

...

input

input

base inputs

original dataset

seed pool

ChatGPT

seed inputs

type-aware

mutation

def sample_0(input):

...

LLM samples

differential

testing

def sample_11(input):

...

Rigorously validated

LLM samples

pass

generate corner-case inputs

generate difficult inputs

generate complex inputs

new inputs

def groundtruth(input):

...

EvalPlus dataset

new input

x

gt

f

f

(

x

) =

gt

(

x

)?

input

input

input

input

input

Test-suite Reduction

coverage

mutant kills

sample kills

set

cover

including the most valuable test-cases, which are selected by running a greedy set cover algorithm to preserve the same code coverage [24], mutation analysis [7] as well as empirical LLM sample killings.

Contribution. Our work revisited and proposed to automatically improve code benchmarks for LLMs:

Study: We are the first to study the test inadequacy problem in current programming benchmarks which can lead to largely over-approximated functional correctness. Our study also opens up a new research direction for precisely and rigorously evaluating LLM-synthesized code.

Approach: We propose EvalPlus - an evaluation framework to reveal the real correctness of LLM-synthesized code. The test-case generation approach of EvalPlus combines the emerging LLM-based and traditional mutation-based test input generation. It first uses LLM-based strategy to bootstrap the test generator with high-quality seed inputs and then further extends large amounts of inputs via type-aware mutation. We then optionally 'distill' the generated tests to a much smaller yet almost equivalently effective test-suite via greedy set covering. We also propose to annotate each programming tasks using program contracts to filter out invalid inputs.

Results: EvalPlus extends the popular HUMANEVAL benchmark to create HUMANEVAL + , improving the test-case scale by 80 × . Through test-suite reduction, we also produce HUMANEVAL + -MINI which distills HUMANEVAL + tests by 47 × while still achieving a similar level of testing effectiveness. Our extensive evaluation over 26 popular LLMs surprisingly finds that the pass@ k on the new dataset is up-to 19.3-28.9% (for different k s) lower than the base HUMANEVAL, showing that testing insufficiency can largely affect the result analysis for almost all recent work on LLM-based code generation. Meanwhile, on the original HUMANEVAL both of the 34B WizardCoder-CodeLlama [38] and Phind-CodeLlama [52] models are deemed to be no better than ChatGPT, while HUMANEVAL + corrected the ranking and shows that the two open-source models are actually better. Additionally, we even found that the ground-truth solutions of HUMANEVAL can be erroneous, further calling into question the quality of code synthesis benchmarks.

2 Approach

Figure 2 shows the overview of EvalPlus . We first take in as input the original dataset containing the ground-truth implementation as well as the base test inputs. EvalPlus starts with constructing a prompt using the original ground-truth, exemplary test inputs as demonstration, and a specialized instruction to query ChatGPT and generate a set of high-quality seed inputs. ChatGPT, by following base input formats and inspecting the ground-truth solution, can serve as a vehicle to generate valid yet rigorous test inputs. Starting from these seed inputs, we then perform type-aware mutation to quickly generate numerous new inputs together with seed inputs to extensively evaluate the functional correctness of LLM-generated code. We use differential testing [40] as the oracle to cross-check the output of the ground-truth and LLM-generated solution. As an option to speed up evaluation, EvalPlus

runs set covering to minimize the generated test-suite while preserving the same level of testing effectiveness. As the final output, EvalPlus obtains a augmented benchmark using the generated high-quality test inputs to fully evaluate the functional correctness of LLM-synthesized code.

2.1 Automated Test Input Generation

Seed initialization via ChatGPT. EvalPlus first uses ChatGPT to generate a set of high-quality seed inputs for later mutation. Following Figure 2, we construct a prompt using (i) the ground-truth solution of the problem for ChatGPT to inspect; (ii) a set of test inputs as demonstration; and (iii) an instruction to encourage ChatGPT to come up with interesting inputs. Specifically, each prompt starts with the ground-truth implementation and then randomly sampled test inputs from the existing dataset. We then finalize the prompt with a selected instruction in Figure 2 and query ChatGPT to produce new inputs. EvalPlus aims to leverage the powerful understanding ability of ChatGPT to learn both the valid input formats ( e.g., variable types) as well as the desired functionality of the ground-truth solution in order to produce meaningful test inputs to reveal bugs in incorrectly synthesized code. Programs can have their own expected input formats, where invalid inputs should not be passed into the function as they can incur undefined behaviors to create false-positives in differential testing. As such, we filter out any invalid inputs which violate the input precondition required by the ground-truth implementation.

By using ChatGPT as an automated generation engine, we can generate inputs that are valid even under semantic constraints. For example, a programming problem may require the input to conform to a specific structure ( e.g., a palindrome). Such semantic constraints can be extremely difficult for traditional input generators to satisfy. However, ChatGPT is unsuitable for large amounts of automated test generation due to undesired speed and cost of querying such a large model. To address this, we perform type-aware input mutation starting from high-quality seed inputs generated by ChatGPT.

Type-aware input mutation. We follow a typical mutation-based fuzzing workflow [74, 57] to continuously create inputs: (i) a corpus of seed inputs from ChatGPT are used to initialize the seed pool and bootstrap the generation pipeline; (ii) each time an input ( i.e., seed) from the seed pool is randomly selected to be mutated to a new input ( i.e., mutant); and (iii) new inputs that comply with the program contract (§2.3) are added to the seed pool and we start over from (ii) to continue the generation process.

To efficiently create more valid inputs, we leverage type-aware mutation [66] in step (ii) which inspects the data types of the incoming valid seeds and generates new inputs that are structurally similar to the seeds. In Table 1 we illustrate the basic mutations used for different types of inputs. For simple primitive types such as int and float , the mutation is as simple as incrementing/decrementing the value. For compound types and the string type ( i.e., str ), besides generally removing or repeating existing elements (or sub-strings for str ), the elements and sub-strings can be mutated recursively according to their inner types. Such sub-mutants can then be used to replace existing items or add new items in a finer-grain manner. In addition, to alleviate generating inputs that violate subtle semantic constraints, following [23, 34], we additionally apply an ingredient mechanism to collect appeared data fragments and reuse them during mutation. In short, type-aware input mutation builds on the high-quality seed inputs produced by ChatGPT to generate large amounts of test inputs which we use as the final set of extensive test inputs to evaluate LLM-synthesized code.

2.2 Test-Suite Reduction

While the large number of newly generated tests in EvalPlus are effective in detecting incorrect code, the test execution can be costly. As an option to more efficiently evaluate LLM-generated code, we further investigate test-suite reduction strategies [75, 59], which aim to select a subset of the original test-suite while still maintaining the original test effectiveness. To perform test reduction, it is typically assumed that each test can fulfill a set of testing requirements . The problem can then be formalized as reducing the original test-suite T into T red , such that ∀ r ∈R ( ∃ t ∈T , t satisfies r = ⇒ ∃ t ′ ∈ T red , t ′ satisfies r ) . In other words, any testing requirement r satisfied by the original test-suite should still be satisfied by the reduced one. Finding such minimal representative subset for a given test-suite is equivalent to the set covering problem [17]. To solve this problem effectively, it is crucial to define the testing requirements accurately. In this paper, we focus on the following types of requirements:

Code coverage : Code coverage [24] measures the amount of code elements ( e.g., statements or branches) executed by each test, and has been widely used in practice to measure test effectiveness. In this strategy, following traditional test-suite reduction [53] we leverage the widely used branch coverage as the testing requirement. In other words, the goal of using this metric is to only preserve a minimal subset of tests which can cover the same set of branches as the full tests.

Mutant killings : Coverage measures the extent to which the code has been executed; however, a high-coverage test-case is not necessarily effective in finding critical defects in its covered code. Consequently, researchers have proposed mutation testing [7] (also known as mutation analysis ) to more precisely evaluate test effectiveness. In short, mutation testing applies a set of predefined mutation rules ( e.g., changing ' < ' and ' ≤ ') to the program under test ( i.e., the ground-truth solutions for this case) to create a large number of artificial buggy programs, each of which is called as a mutant and includes exactly one subtle bug seeded. In this way, the ratio of mutation bugs detected by the tests (also called killed ) can be used to assess the test effectiveness. In fact, studies have shown that mutation testing can largely outperform code coverage in test effectiveness evaluation [51]. Following prior work [59], we also leverage the set of mutants killed by each test as our testing requirement. Consequently, the goal is to minimize the number of tests while still being able to detect the same set of mutation bugs.

LLMsamplekillings : Different LLMs could fail commonly over certain test-cases. Consequently, besides these theoretical metrics, we also use as a testing requirement by empirically looking at sample killings , i.e., the set of wrong LLM samples that a test-case can detect and falsify. Of course, for a new LLM under evaluation, we do not have any test execution results for its code samples. Therefore, we only use the execution results for samples generated by other LLMs to evaluate test effectiveness for reduction ( i.e., leave-one-out cross validation [22]). As such, we minimize the number of tests while making sure that all incorrect samples synthesized by other models can be detected by the reduced test-suite.

Besides the above three strategies, we also investigate another strategy that merges all three testing requirements for reduction. That is, the goal is to minimize the number of tests while still maintaining the same branch coverage, mutant killing, and incorrect sample detection results.

2.3 Program Input Contracts

The goal of evaluating code synthesis is to check whether the synthesized code accurately reflects the desired user intent. This is done by using several test inputs and comparing the output of the generated code against that of the ground-truth solution. The prior sections demonstrated how to improve the test inputs used to more rigorously evaluate the synthesized code. However, these user intents (expressed as natural language docstring) can be too vague for LLMs to follow. As such, LLMs might allow for different interpretations of the desired functionality, input formats as well as how to handle corner cases.

To this end, we adopt a programming by contract [41] philosophy by systematically annotating function pre-conditions in form of code assertions ( e.g., assert n > 0 ), to ensure the test inputs for the function are well-formed. The benefits of the contracts are two-fold: (i) they can complement the automatic input generation steps to filter out any generated invalid inputs that violate the contracts. Such ill-formed inputs can incur undefined behaviors which are unreasonable to use for evaluating LLM-synthesized code; and (ii) they can serve as orthogonal descriptors together with the natural language description in the prompt for further clarification.

3 Evaluation

Setup. Our evaluation focuses on using the unbiased version of pass@ k [11] to accurately assess the functional correctness of LLM-synthesized code. For generalizability, we conducted a comprehensive evaluation over 26 popular and state-of-the-art LLMs and a wide range of temperature settings. Specifically, following prior work [11, 46], for each model we perform: (i) random sampling to generate 200 program samples for each of the four temperature settings ( { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } ); and (ii) greedy-search decoding. For random sampling, we show the best-performing pass@ k for each k ∈{ 1 , 10 , 100 } and its corresponding temperature denoted by T ∗ k . For greedy decoding, we only synthesize one deterministic sample for each task and evaluate its pass rate as pass@ 1 ⋆ . By default we evaluate models under both setting (i) and (ii) , except for the two commercial models due to time and cost constraints: GPT-4 is only evaluated under greedy decoding, and ChatGPT is additionally evaluated on 0 . 8 -temperature random sampling.

While EvalPlus is general, this paper focuses on evaluating its effectiveness on HUMANEVAL [11], one of the most widely-used datasets for code generation 3 . HUMANEVAL consists of 164 humanwritten programming tasks, each of which provides a Python function signature and a docstring as the input to the LLM. Based on the input, LLMs complete a solution whose functional correctness is judged by a handful of manual test-cases (the first row in Table 2). As such, EvalPlus transforms HUMANEVAL to HUMANEVAL + by adding 80 × unique test-cases and fixing incorrect ground-truth solutions in HUMANEVAL. Specifically, for each task, based on around 30 ChatGPT-generated seed inputs which are produced using 3 separate prompts, we run type-aware mutation to generate 1000 additional inputs using one-hour budget. In HUMANEVAL + , 83 out of the 164 programming tasks are annotated with hand-crafted contracts. Because EvalPlus requires ground-truth solutions to cross-check LLM-generated code, it is crucial to ensure the correctness of the ground-truths. However, by inspecting ground-truths in the original HUMANEVAL, we found over 10% of them are incorrectly implemented. Therefore, as another contribution we carefully re-implemented and tested all ground-truths for HUMANEVAL + . As an option to speed up evaluation, we build HUMANEVAL + -MINI which is minimized from HUMANEVAL + (smaller by 47 × ) yet preserves similar test effectiveness on the studied models. Lastly, more experimental setups are detailed in Appendix.

Evaluation of LLMs. Table 3 shows the pass@ k when evaluating LLMs using both the base HUMANEVAL and HUMANEVAL + . We first observe that across all LLMs, models sizes and k values, using HUMANEVAL + , almost all pass@ k results consistently drop compared to using the base HUMANEVAL. Notably, the performance drop is significant with up-to 23.1% (pass@ 1 ⋆ ) / 19.3% (pass@ 1 ) / 24.9% (pass@ 10 ) / 28.9% (pass@ 100 ) reduction over the evaluated models. Such performance decrease is not only seen in popular open-source LLMs, such as the widely used CodeGen-16B [46] ( 18 . 5% reduction) as well as the emerging CODELLAMA-34B [54] ( 17 . 6% ) and StarCoder [13] ( 14 . 1% reduction), but also observed in state-of-the-art commercial ChatGPT ( 12 . 6% reduction) and GPT-4 ( 13 . 1% reduction) models. Overall, our results overall confirm our hypothesis that the prior evaluation on HUMANEVAL is not robust enough to detect wrong code synthesized by LLMs. Not only are these LLMs widely used for daily programming but they also serve as common reference points for evaluating new code synthesis techniques. As such, evaluating on a more robust benchmark such as HUMANEVAL + is highly recommended in order to draw precise conclusions.

We also show that a more rigorous evaluation could yield different or totally contradictory relative results. For example, WizardCoder-CodeLlama and Phind-CodeLlama on the original HUMANEVAL are evaluated to be no better than ChatGPT in terms of pass@ 1 ⋆ . However, HUMANEVAL + demonstrates that the two open-source models can actually outperform the proprietary ChatGPT. Other contrary examples reflected by HUMANEVAL + include that SantaCoder-1B surpasses INCODER-6.7B and VICUNA-7B outperforms INCODER-1.3B. Table 3 further illustrates the distribution of best-performing temperatures over different k values. Our results conforms with prior findings [11] that a lower temperature tends to perform better for smaller k , while a higher temperature works better for larger k . We also observe that the optimal temperatures seem to stay fairly consistent before and after using HUMANEVAL + ; however, slight differences still exist, e.g., best temperature for CodeGen-2B on pass@ 10 becomes 0.2 from 0.8 after using HUMANEVAL + . Nonetheless, this motivates future research to look more closely on the effect of temperature with respect to the robustness of the evaluation tests, esp. those edge-cases.

Effectiveness of test-suite reduction. Based on HUMANEVAL + which on average obtains 764.1 tests for each programming task (Table 2), our test-suite reducer (§2.2) minimizes it to HUMANEVAL + -MINI which only has 16.1 tests for each task (smaller by 47 × ). Table 4 performs leave-one-out cross validation to show the pass@ 1 ⋆ differences over a subset of representative models studied in Table 3 (due to time/space constraints). That is, for each evaluated LLM we construct the reduced test-suite without considering its own sample kills. The Full column shows that the reduced test-suite can achieve almost the same pass@ 1 ⋆ drop as HUMANEVAL + by only using 47 × fewer test-cases. Taking a closer look, separately performing set covering over each metric can harness the pass@ 1 ⋆ of the base HUMANEVAL to certain degree. Specifically, the use of empirical LLM sample killings is the most effective, leading to the same effectiveness as the full approach, but also consumes more tests than other theoretical metrics. While using coverage and mutation analysis seems to be unnecessary in addition to using sample killings, they still serve as the base guarantees for the theoretical test adequacy.

Pass rate distribution. Figure 3 shows for each programming task the overall pass rates on HUMANEVAL and HUMANEVAL + tests. The pass rate gap between HUMANEVAL and HUMANEVAL + shows overall HUMANEVAL + can detect solutions that are misidentified by HUMANEVAL for problems of all levels of difficulties. We also observe that problems in HUMANEVAL are not equal, not only in terms of problem difficulty but also the difficulty of generating counter-examples and

10

0

10

1

10

2

Average Pass Rate (%)

HumanEval

HumanEval

+

Problems (Sorted by HumanEval pass rate)

edge-cases to deeply exercise LLM-generated code. For simple problems such as 'adding two numbers' and 'length of a string' ( i.e., problems with top-2 pass rates), it is easy to solve for LLMs and to test manually. While problems dealing with multiple conditions ( e.g., 'word splitting'), completeness ( e.g., handling negative numbers for 'is-prime') , reasoning ability ( e.g., 'Tribonacci sequence') and efficiency requirements ( e.g., 'n-th prime Fibonacci number') are the hardest tasks to the evaluated LLMs, positioning future research to improve LLMs for conquering such coding skills.

Incorrect 'ground-truth' in HUMANEVAL. In addition to detecting wrong code from LLMs using EvalPlus , we also found 18 defects (11% of problems) even in the original ground-truth in HUMANEVAL, including (i) Unhandled edge-case : five prior ground-truths fail to handle corner-case inputs ( e.g., empty list or string); (ii) Bad logic : 10 prior ground-truths incorrectly implement the desired functionality; and (iii) Performance issue : three inefficient implementations lead to slow performance on reasonably-sized inputs. Among those, bad logic (10) is the most serious as the original 'groundtruth' does not accurately reflect the user intent. Such defects are detected also through differential testing but between our own re-implemented ground-truth and the original ground-truth in HUMANEVAL.

def valid_date(date):

...

if month in [1,3,5,7,8,10,12] and day < 1 or day > 31:

return False

if month in [4,6,9,11] and day < 1 or day > 30:

return False

...

12-31-1999

False

HUMANEVAL

+

input

12/31/1999

is a valid date!

A bracket is needed!

Figure 4 shows an incorrect ground-truth implementation ( validate_date ) from HUMANEVAL classified as having bad logic. The desired task is to check if the input date format is correct. We see that in the core logic, the conditions attempt to first check the month condition and then handle the corresponding day conditions. However, this is implemented incorrectly as ' and ' in Python 5 has higher precedence than ' or ', leading to the ground-truth function to check if either conditions satisfies instead of the desired both conditions must satisfy. This is exposed via our automatically generated test input of 12-31-1999 where the ground-truth implementation incorrectly labels this as not a valid date. Surprisingly this egregious error is not exposed by any of the base test inputs in HUMANEVAL, further demonstrating the weakness and limited evaluation power of the original test inputs.

4 Related Work

LLMsforcode . The use of LLMs for code has gained traction in recent years, owing to the abundance of open codebase and the need for improving developer efficiency. LLMs have demonstrated state-of-the-art performance on various code-related tasks, including code generation [11, 33, 25], program repair [69, 27, 68, 65], automated testing [15, 14, 67, 35, 71], code translation [31, 55] and code summarization [1, 37]. In particular, prominent LLMs including CODEX [11], CodeGen [46], INCODER [18] and PolyCoder [70], have been developed and extensively evaluated for code

generation (widely recognized as the holy grail for computer science research since the inception of AI in the 1950s [21]), where the model generates code snippets based on natural language descriptions ( e.g., docstring) of the desired functionality.

Coding benchmark for LLMs. LLM-based code synthesis is largely evaluated based on functional correctness, which is typically assessed by running test-cases to check the desired outputs. HUMANEVAL [11] is one of the pioneering and most widely studied human-written benchmarks for LLM-based code synthesis, consisting of 164 pairs of Python function signature with docstring and the associated test-cases for correctness checking. Additionally, each HUMANEVAL problem is also equipped with a reference solution. Another Python-focused dataset, MBPP [3], is created by crowd-sourcing participants to write in summation 974 programming problems, each of which is comprised of the problem statement ( i.e., docstring), the function signature, as well as three test-cases. Beyond Python, there are other benchmarks targeting additional languages such as Spider [73] (SQL), HUMANEVAL-X [76] (C++, Javascript and Go), CodeContests [33] (C++ and Java) and MultiPL-E [9] (extending HUMANEVAL and MBPP to 18 programming languages). More recently, researchers have created a more realistic code synthesis benchmark by collecting GitHub issues along with the corresponding code base together with tests to measure the ability of LLMs to perform real-world software engineering tasks [28]. Our work shows for the first time the test inadequacy problem of widely studied benchmarks and addresses the issue via automatic test generation.

Automatedtestgeneration . Automated test generation is a widely used for finding software bugs with automatically generated tests. Black-box test generation such as fuzz testing [43] feeds random inputs ( e.g., random bytes) to the system under test (SUT), without knowing its source code. Traditional black-box techniques can mainly be categorized into generation-based [72, 23, 56] and mutationbased [66, 10, 47] ones. White-box approaches provide better-quality test-cases by analyzing the source code of SUT. For instance, symbolic execution [30, 8] breaks the coverage plateaus by solving symbolic path constraints to generate tests targeting deep paths. As a mid-point, coverage-guided fuzzing [74, 57] ( i.e., grey-box) uses the coverage information of SUT as feedback to adjust the input generation and mutation. The discussed traditional methods are inapplicable to generating semantically meaningful inputs for arbitrary problems programmed in a dynamically-typed language. We address this by using ChatGPT to inspect the ground-truth ( i.e., white-box) for initializing interesting seeds, based on which type-aware mutation ( i.e., black-box) scales the test inputs to a large amount.

5 Conclusion & Future Work

We present EvalPlus - a rigorous evaluation framework for program synthesis, driven by automated test generation. EvalPlus combines both LLM- and mutation-based input generation to obtain a diverse set of test inputs for accurately evaluating the correctness of LLM-generated code. EvalPlus creates HUMANEVAL + , built on top of the popular HUMANEVAL with additional high-quality and automatically generated test inputs. With test-suite reduction, EvalPlus also produces HUMANEVAL + -MINI which is smaller than HUMANEVAL + by 47 × while preserving similar test effectiveness. We extensively evaluate a diverse set of LLMs and show that HUMANEVAL + can identify a significant amount of previously undetected wrong code generated by LLMs, demonstrating its effectiveness to augment programming benchmarks for more accurate evaluation.

Since launched, the EvalPlus PyPI package has been installed by over 6k times in 5 months. We also keep evaluating new models for code and maintain a leaderboard at https://evalplus.github. io/leaderboard.html . In the future, we plan to apply EvalPlus to bring better-quality testing for more code benchmarks such as MBPP. Meanwhile. future work can look into how to integrate EvalPlus with more formal verification ( e.g., Dafny [32]) or validation techniques ( e.g., translation validation [36]) to provide stronger guarantees of the evaluation results when applicable. Additionally, the core test generation technique behind can be even used to remind developers of potential flaws of the accepted LLM-generated code snippets when doing AI pair-programming ( e.g., Copilot [42]).

References

T. Ahmed and P. Devanbu. Few-shot training llms for project-specific code-summarization. In 37th IEEE/ACM International Conference on Automated Software Engineering , pages 1-5, 2022.

L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988 , 2023.

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021.

S. Bang, S. Nam, I. Chun, H. Y. Jhoo, and J. Lee. Smt-based translation validation for machine learning compiler. In Computer Aided Verification: 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part II , pages 386-407. Springer, 2022.

S. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. If you use this software, please cite it using these metadata.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

T. A. Budd. Mutation analysis of program test data . Yale University, 1980.

C. Cadar, D. Dunbar, D. R. Engler, et al. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI , volume 8, pages 209-224, 2008.

F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering , 2023.

S. K. Cha, M. Woo, and D. Brumley. Program-adaptive mutational fuzzing. In 2015 IEEE Symposium on Security and Privacy , pages 725-741. IEEE, 2015.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

B. Code. Starcoder. https://github.com/bigcode-project/starcoder , 2023.

Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In 32nd International Symposium on Software Testing and Analysis (ISSTA) , 2023.

Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. In 46th International Conference on Software Engineering (ICSE) , 2024.

fauxpilot. Fauxpilot: an open-source alternative to github copilot server. https: //github.com/fauxpilot/fauxpilot , 2022.

U. Feige. A threshold of ln n for approximating set cover. Journal of the ACM (JACM) , 45(4):634-652, 1998.

D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations , 2023.

C. Green. Application of theorem proving to problem solving. In Readings in Artificial Intelligence , pages 202-222. Elsevier, 1981.

S. Gulwani. Automating string processing in spreadsheets using input-output examples. SIGPLAN Not. , 46(1):317-330, jan 2011.

S. Gulwani, O. Polozov, and R. Singh. Program synthesis. Foundations and Trends® in Programming Languages , 4(1-2):1-119, 2017.

T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In 21st USENIX Security Symposium (USENIX Security 12) , pages 445-458, Bellevue, WA, Aug. 2012. USENIX Association.

M. Ivankovi´ c, G. Petrovi´ c, R. Just, and G. Fraser. Code coverage at google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages 955-963, 2019.

S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 1643-1652, 2018.

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

N. Jiang, K. Liu, T. Lutellier, and L. Tan. Impact of code language models on automated program repair. arXiv preprint arXiv:2302.05020 , 2023.

C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 , 2023.

A. Kalyan, A. Mohta, O. Polozov, D. Batra, P. Jain, and S. Gulwani. Neural-guided deductive search for real-time program synthesis from examples. In International Conference on Learning Representations , 2018.

J. C. King. Symbolic execution and program testing. Communications of the ACM , 19(7):385-394, 1976.

M.-A. Lachaux, B. Roziere, L. Chanussot, and G. Lample. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511 , 2020.

K. R. M. Leino. Dafny: An automatic program verifier for functional correctness. In International conference on logic for programming artificial intelligence and reasoning , pages 348-370. Springer, 2010.

Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science , 378(6624):1092-1097, 2022.

J. Liu, Y . Wei, S. Yang, Y . Deng, and L. Zhang. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation. Proceedings of the ACM on Programming Languages , 6(OOPSLA1):1-26, Apr. 2022.

Z. Liu, C. Chen, J. Wang, X. Che, Y. Huang, J. Hu, and Q. Wang. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 1355-1367. IEEE, 2023.

N. P. Lopes, J. Lee, C.-K. Hur, Z. Liu, and J. Regehr. Alive2: bounded translation validation for llvm. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation , pages 65-79, 2021.

S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 , 2021.

Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568 , 2023.

Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Communications of the ACM , 14(3):151-165, 1971.

W. M. McKeeman. Differential testing for software. Digital Technical Journal , 10(1):100-107, 1998.

B. Meyer. Applying'design by contract'. Computer , 25(10):40-51, 1992.

Microsoft. GitHub Copilot - Your AI pair programmer. https://github.com/features/ copilot , 2023.

B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM , 33(12):32-44, 1990.

G. C. Necula. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation , pages 83-94, 2000.

E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint , 2023.

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations , 2023.

P. Oehlert. Violating assumptions with fuzzing. IEEE Security & Privacy , 3(2):58-62, 2005.

OpenAI. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/ chatgpt/ , 2022.

OpenAI. Gpt-4 technical report. ArXiv , abs/2303.08774, 2023.

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318, 2002.

G. Petrovi´ c and M. Ivankovi´ c. State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice , pages 163-171, 2018.

Phind. Phind/phind-codellama-34b-v2 · hugging face. https://huggingface.co/Phind/ Phind-CodeLlama-34B-v2 , 2023.

G. Rothermel, M. J. Harrold, J. Von Ronne, and C. Hong. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability , 12(4):219-249, 2002.

B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 , 2023.

B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773 , 2021.

M. Security. jsfunfuzz. https://github.com/MozillaSecurity/funfuzz , 2007.

K. Serebryany. Continuous fuzzing with libfuzzer and addresssanitizer. In 2016 IEEE Cybersecurity Development (SecDev) , pages 157-157. IEEE, 2016.

D. E. Shaw, W. R. Swartout, and C. C. Green. Inferring lisp programs from examples. In IJCAI , volume 75, pages 260-267, 1975.

A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering , pages 246-256, 2014.

Stability-AI. Stablelm: Stability ai language models. https://github.com/Stability-AI/ StableLM , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

R. J. Waldinger and R. C. Lee. Prow: A step toward automatic program writing. In Proceedings of the 1st international joint conference on Artificial intelligence , pages 241-252, 1969.

B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax , May 2021.

Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922 , 2023.

Y. Wei, C. S. Xia, and L. Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. arXiv preprint arXiv:2309.00608 , 2023.

D. Winterer, C. Zhang, and Z. Su. On the unusual effectiveness of type-aware operator mutations for testing smt solvers. Proceedings of the ACM on Programming Languages , 4(OOPSLA):1-25, 2020.

C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models. In 46th International Conference on Software Engineering (ICSE) , 2024.

C. S. Xia, Y. Wei, and L. Zhang. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery , 2023.

C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages 959-971, 2022.

F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , pages 1-10, 2022.

C. Yang, Y. Deng, R. Lu, J. Yao, J. Liu, R. Jabbarvand, and L. Zhang. White-box compiler fuzzing empowered by large language models, 2023.

X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in c compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation , PLDI '11, page 283-294, New York, NY, USA, 2011. Association for Computing Machinery.

T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018.

M. Zalewski. American fuzzing lop (afl). https://lcamtuf.coredump.cx/afl/ , 2018.

L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In 2011 IEEE 22nd International Symposium on Software Reliability Engineering , pages 170-179. IEEE, 2011.

Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 , 2023.

A Detailed Experimental Setup

Evaluation of LLMs. Our goal is to comprehensively evaluate recent and widely used LLMs, both specialized for code generation [46, 70, 18, 2, 52, 38, 64] and general-purpose tasks [49, 48, 12, 60, 63, 5, 26]. Table 5 presents an overview of the studied models, with column Sizes reflecting the model sizes in billions of parameters, Release Year showing when the LLM is released, and Open-Source marking the models whose weights are publicly available. In total, we evaluate 26 of the most representative and popular LLMs with a broad range of configurations to fully demonstrate the generalizability of our results.

Our hyper-parameter configurations follow prior work [11, 46]. For each model we randomly sample 200 programs and repeat the experiments over temperature ( { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } ) and greedy decoding with zero temperature. By default, we let each model generate at most 512 new tokens and truncate the produced code with end-of-string (EOS) identifiers suggested in HUMANEVAL [11], as well as those favoured by certain models ( e.g., ' 
 ' and ' \n``` '). For conversational models ( i.e., ChatGPT and GPT-4), we obtain the code fragments by parsing the code blocks ( i.e., within ' ``` ') in the output. We found ChatGPT tends to repeat problem description with detailed explanation, which can consume more than 512 new tokens to complete a solution for around 11% of problems. To align ChatGPT with other models, for tasks with very long problem descriptions, we extend the token limit from 512 to 1024. For model implementation, we run ChatGPT and GPT-4 via OpenAI APIs, and accelerate CodeGen-6B and -16B with NVIDIA FasterTransformer via FauxPilot [16]. All other LLMs are based on the HuggingFace transformers library. By default, we follow the official examples of each LLM ( e.g., on HuggingFace model card) to construct their corresponding prompts. Specifically, the prompts used for ChatGPT, GPT-4, and WizardCoder-CodeLlama is instruction-based, i.e., a simple instruction is used to wrap the function signature and docstring to explicitly encourage the LLM for code generation.

Test oracles. An LLM-produced solution is regarded to be correct if for all test inputs it returns values that match the expected outputs within a reasonable run time. We perform exact matching by default. For floating-point comparisons, we tolerate absolute differences to the degrees annotated in HUMANEVAL or 10 -6 if not annotated. In original HUMANEVAL, the default timeout is set to three seconds to run the whole test-suite ( i.e., all test-cases) for each programming problem. Such a setting is neither suitable when having more test-cases nor reasonable as each problem could have its own run time characteristics. Consequently, we let the timeout for each test-case to be max(200ms , 4 × t gt ) where t gt refers to the execution time of the corresponding ground-truth solution. In other words, we expect the LLM-provided solution to be no slower than the ground-truth by four times or use a base 200-millisecond timeout when 4 × t gt < 200ms to avoid variance caused by performance randomness.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leveraged large language model (LLM) based and mutation-based strategies to generate high-quality test cases for the popular dataset HumanEval. The extended dataset HumanEval+ provides a better code generation benchmark for assessing the performance of LLMs such as ChatGPT and GPT4. Experimental results showed that compared to the HumanEval dataset, the extended one can detect more issues of the generated code, where the generation performance of 19 LLMs is reduced by 13.6-15.3% on average in terms of pass@k. To save the testing time, this paper also provided a refined set of test cases with fewer numbers but with the same code coverage, killed mutants, and sample killings.\n\nSTRENGTHS:\n+ Proposing a test case generation method for LLMs, which combines the LLM-based and mutation-based strategies.\n\n+ Evaluating 19 LLMs on the extended datasets and showing the overestimation of the original dataset.\n\n+ Providing a test-suite reduction for quick evaluation.\n\nWEAKNESSES:\n- Some technical details, such as the quality of seed inputs, are unclear, which affect the soundness of the proposed approach.\n\n- There are some unclear claims/overclaims.\n\n- Lack of comparisons with other test case generation methods and related work.\n\nThe claimed contribution indicates “that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis”. Can the extended dataset proposed by this paper reflect the “true performance”? Similarly, the paper said that the existing tests “often fall short in capturing all possible scenarios”, leading to “false confidence” in the results. Can your test cases “capturing all possible scenarios”? Furthermore, this paper claimed that the proposed benchmark can “precisely evaluate the functional correctness of LLM-generated code” by generating “interesting test inputs”. It is unclear to me how “precise” can the evaluation provide and what are the “interesting test inputs”. Please clarify them.\n\nEvalPlus “first uses ChatGPT to generate a set of high-quality seed inputs for later mutation”. The quality of the seed inputs is not verified. Are these inputs correct? How do you ensure the quality of the generated seed inputs? and how many seed inputs did the ChatGPT generate? What are the prompts used?\n\nThe type-aware input mutation “randomly” selected some inputs from the generated seed pool as the inputs of mutants. This pipeline did not clearly describe how randomness affects the result of the input mutation and the quality of the evaluation.\n\nThe proposed approach “adopted a programming by contract philosophy by systematically adding code assertions as contracts (e.g., assert n > 0) to ensure the test inputs for the function are well-formed”. The context did not tell how to define the assertions specifically, how to ensure the correctness of the definitions, and which docstrings need clarification with these assertions. Will it incur a lot of manual effort?\n\nFor the evaluation, the paper did not compare the method with other baseline methods. The paper says that traditional automated test generation methods are inapplicable to generating semantically meaningful inputs. Why is that so? More descriptions are needed here. Otherwise, it is difficult to determine the advantages of the proposed method over the traditional methods.\n\nRecently, there is also a related work on applying ChatGPT prompts to code generation, the authors also discussed the quality/correctness of the generated code: C. Liu et al., Improving ChatGPT Prompt for Code Generation, https://arxiv.org/abs/2305.08360.\n\nQUESTIONS:\nHow is the quality of the seed inputs generated by ChatGPT? Are these inputs correct? How do you ensure the quality of the generated seed inputs? and how many seed inputs did the ChatGPT generate? What are the prompts used? Will the quality of seed inputs affect your approach significantly?\n\nLIMITATIONS:\nThe authors addressed the limitations in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce EvalPlus, an evaluation framework crafted to assess the code generation of LLMs. Combining LLM and mutation-based techniques, EvalPlus diversifies the generated test inputs, thereby broadening the evaluation spectrum for LLM-produced code. Through comprehensive experimentation, the authors highlight EvalPlus's ability to expose shortcomings in previous LLM-based code evaluations. Additionally, the study brings HumanEval Plus, an enriched dataset built upon the existing HumanEval, offering an abundance of test inputs and a more dependable ground truth for improved reliability.\n\nSTRENGTHS:\n* This paper distinguishes itself through its innovative approach to test input generation. It extends beyond the conventional usage of ChatGPT for creating test inputs, by incorporating a test-mutation technique and a “distill” method. This results in a diverse yet non-redundant range of test inputs, illustrating the authors' commitment to crafting a robust and efficient evaluation framework.\n* This paper is well-written and easy to read.\n* The study stands out for its well-designed experiment, which encompasses a comprehensive selection of contemporary LLMs.\n\nWEAKNESSES:\n* HumanEval [8], in comparison to real-world coding projects, presents a significantly simpler challenge. It's a benchmark that's even simpler than others, such as APPS [Hendrycks et al.]. Given this context, the effectiveness of EvalPlus when applied to more complex tasks remains uncertain. This indicates a valuable area for enhancement in this study—specifically, future research could delve into assessing how well EvalPlus performs in more complicated, real-world coding scenarios. Such explorations could contribute crucial insights to understanding the scalability and adaptability of EvalPlus.\n\nQUESTIONS:\n* Given the impressive results demonstrated by EvalPlus within the comparatively simpler context of the HumanEval benchmark, how do you envision expanding this work to more complex environments? Specifically, what strategies or modifications do you plan to implement to ensure EvalPlus is effective in assessing code generated for real-world projects, such as those found in open-source Python libraries? Could you elaborate on potential challenges you anticipate and how you intend to navigate them in this expansion?\n\nLIMITATIONS:\n* While this study presents a promising step forward with EvalPlus, a limitation lies in the evaluation scenario chosen for testing. Given that the proposed method is intended to be general, using the relatively simple HumanEval dataset for testing may not sufficiently demonstrate the framework's generalization capabilities. To conclusively assert its wide applicability and robustness, it would be beneficial for future research to include tests on more complex datasets or in more challenging real-world contexts, such as open-source Python libraries. This would provide a more comprehensive understanding of EvalPlus's potential in diverse, practical applications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an evaluation framework for validating the correctness of large language model-generated code. In particular, the framework first utilizes ChatGPT to generate multiple seed inputs, which are then expanded into a large set of inputs through type-aware mutation. In addition, to ensure evaluation efficiency, the input set can be reduced by employing code coverage, mutant killings, and LLM sample killings.\n\nSTRENGTHS:\nThe proposed solution addresses the problem of insufficient testing for LLM-generated code. The expanded dataset can outperformance the original one, while the reduced dataset achieves similar results as the expanded one. \nThe authors evaluate the proposed solution via comparative analysis of 19 large language models, analyze the pass rate distribution of the employed dataset HUMANEVAL, and identify several errors in the ground-truth solutions. \nThe related work section discusses previous research on large language models for code, the coding benchmark, and automated test generation. The authors discuss the research gap between this study and the related work.\n\nWEAKNESSES:\nThe authors do not employ a more capable large language model in the proposed solution, and there is a lack of evaluation of efficiency.\n\nQUESTIONS:\n- I wonder if GPT-4 can generate more interesting seed inputs than ChatGPT. Explicitly clarifying the employed component in EvalPlus may affect the generality of the solution, I suggest this can be introduced as the implementation or setup information.\n- The authors can elaborate on how to filter out the invalid inputs in Section 2.1.\n- The evaluation does not examine the efficiency of type-aware input mutation, and test-suite reduction.\n- Table 4: “Killed samples” has ambiguity.\n- Section 2: “EvalPlus obtains a augmented benchmark …” -> “an”.\n\nLIMITATIONS:\nThe authors do not explicitly identify the potential societal impact or limitations of this work, but they claim that their future work include conducting testing to more code benchmarks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a rigorous evaluation framework EvalPlus for program synthesis driven by automated test generation. For automated test generation, this work proposes to combine both LLM-generated tests and mutation-based input generation to largely augment the text inputs for an existing code benchmark of HumanEval. EvalPlus is evaluated with a diverse set of code LLMs and results show that the new augmented benchmark can identify a significant amount of previously undetected wrong code generated by LLMs, leading to a more accurate evaluation. It further introduces a mini-version benchmark reserving almost the same test effectiveness.\n\nSTRENGTHS:\n* The paper studies how to evaluate code LLMs more accurately, which has become an important topic in the era of LLMs as coding is one of the key capabilities of LLMs to showcase. The proposed EvalPlus of combining both LLM-generated tests and mutation-based tests is well motivated and technically sound.\n* The paper is well written and easy to follow. The evaluation is very comprehensive (considering most of state-of-the-art LLMs) and provides convincing results to authenticate the effectiveness of the proposed EvalPlus.\n\nWEAKNESSES:\n* The biggest weakness of this paper is the lack of discussion and comparison to other related work of AlphaCode and CodeT [1], which share common techniques such as mutation-based input generation and unit tests generated by LLMs. In this regard, the novelty of EvalPlus is relatively limited. Note that in AlphaCode, they have already explored similar mutation-based techniques to largely augment the test cases for program synthesis. Besides, they also explored training a LLM-based test input generation model to generate test cases in a clustering process. For CodeT, they have explored using LLMs to generate test cases, though these generated test cases are used for a different purpose of reranking the generated programs. The authors should discuss and compare with these works to better justify their novelties.\n\n[1] CODET: CODE GENERATION WITH GENERATED TESTS\n\nQUESTIONS:\n* Can you explain more on the difference between EvalPlus and AlphaCode/CodeT?\n* I noticed that this work compares with the recently released StarCoder and CodeGen2, but not another SoTA code LLM of CodeT5+ which was released at the same time with better results on HumanEval. Any reasons for not including it?\n\nLIMITATIONS:\nI did not find any discussion on limitations from the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes an enhanced test dataset and test-driven evaluation of code generation by LLMs. The paper compares different LLMs and a stricter metrics using the enhanced evaluation dataset and shows that these LLMs are about 15% less correct than is reported based on earlier test datasets.\n\nSTRENGTHS:\nGiven that LLMs appear to achieve a number of programming tasks but often fail to completely get them right. They fail in unexpected ways and places. Improving the evaluation metrics is critical to use of LLMs in product and commercial contexts. This paper improves the test beds by synthetically creating addtional tests through a framework that uses both LLMs and mutation based techniques. \nThe paper demonstrates that about 13-15% of code generated by typical LLMs that would be qualified as passable according to previous metric is disqualified by this method. Since LLMs are often used for NL->code contexts this problem may arise because of inssufficient tests or because of less specific description in NL of the code to be written. They increase the test suites by 81X and also create a reduced version of it that shrinks it 47x for fast evaluation.\n\nThe test suites are enhanced by augmenting the prompts with ground truth tests and generating higher quality seed data. From these seed inputs, type-aware mutations are created to enhance the test data sets.\n\nWEAKNESSES:\nWhile the enhanced data set does call out more problems in the LLM-generated code, it is not clear if it is the best it can do. For instance, just by running a static analyser, a syntax checker or some other engine, the issues in the code generated can be found and fixed. The authors should have at least compared one such approach. There is no comparison of another approach that improves the performance of LLMs in context. One of the ways LLMs are often used for code generation is to generate good enough code and then through human interaction or through other static tools rerank, correct or qualify the generated code. \nThe enhanved data set is diefinitely useful but not clear how useful in that it flags about 13-15% of the code. Some human evaluation or A/B test to qualtitatively say how good this is would have been more insightful.\n\nQUESTIONS:\n1. Have you tried any A/B experiment or other human evaluation to see how much more useful is this improved test set to the LLM context in a real application? \n2. Have you thought about other LLM tasks which have similar challenges. \n3. As LLMs are constantly improving likely the value of such an enhanced test set diminishes. Have you thought about studying that?\n\nLIMITATIONS:\nLimitations - LLMs are used in context of human input. So it is not clear how useful this enhanced data is. It is also not clear if the enhanced test set increases coverage in criticial dimensions. \nLLMs are constantly improving and in the absence of a good A/B test or application context it is hard to tell the usefullness of the system.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}