SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiawei Liu $^{1*}$  Chunqiu Steven Xia $^{1*}$  Yuyao Wang $^{2}$  Lingming Zhang $^{1}$
University of Illinois Urbana-Champaign 1 Nanjing University
{jiawei6, chunqiu2, lingming}@illinois.edu yuyao6@outlook.com
Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus – a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HUMANEVAL benchmark by  $80 \times$  to build HUMANEVAL+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HUMANEVAL+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HUMANEVAL+, while none of them could on HUMANEVAL. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.
Automatically generating programs that accurately correspond to user intents is a long-standing challenge in computer science known as program synthesis [21]. In the past few decades, classical program synthesis techniques have been developed, including deductive synthesis [19, 39, 62], inductive synthesis [20, 58] and neural-guided synthesis [29]. More recently, with the advent of Large Language Models [61, 6] (LLMs) and the abundance of open codebase, researchers have been focusing on applying LLMs for direct code generation. LLMs like CODEX [11] and CodeGen [46]
*Equal contribution. Author ordering is decided by Nigiri.
Figure 1: Exemplary wrong code synthesized by ChatGPT for HUMANEVAL #58.
perform code generation by autoregressively predicting the next token given previous context, in the form of function signature and docstring that denote the desired program functionality. The generated code snippet is then combined with the context to form a complete function that aligns with the user intent. Leveraging both natural language understanding and generative power, LLMs have demonstrated impressive performance in code synthesis [3, 11].
The primary concern when it comes to LLM-generated code is correctness. Because two dramatically different code snippets can be semantically equivalent, classic NLP metrics like BLEU score [50] are no longer reliable in the context of program synthesis. Ideally, we would like to formally verify the correctness of LLM-provided solutions for any input, but verifying domain-specific problems through methods such as translation validation [36, 44, 4] is already challenging enough, let alone building a general verifier with absolute certainty to prove arbitrary problems, including those in code benchmarks. As such, existing code benchmarks (e.g., HUMANEVAL [11]) heavily rely on manually constructed test-cases to evaluate LLM solutions. However, these tests often fall short in capturing all possible scenarios, as crafting high-quality tests is laborious. Consequently, we argue that current programming benchmarks are inadequate for assessing the actual correctness of LLM-generated code, leading to false confidence in the results. Specifically, we have identified the following common limitations in existing LLM-for-code benchmarks:
- Insufficient testing. Current programming benchmarks often only include on average less than 10 tests for each coding problem. Furthermore, these tests are relatively too simple to fully explore the functionality of the code or corner cases. Figure 1 shows an incorrect code sample synthesized by ChatGPT [48] to return the sorted unique common elements from two lists. At first glance, the function looks correct and computes the desired output when using the base test inputs from HUMANETVAL. However, in the return statement, it incorrectly converts the intermediate list to a set which no longer preserves the order of the sorted list. This example shows that a logically flawed solution can still pass all simple tests and be misconsidered as correct due to testing inadequacy.
- Imprecise problem description. The input for code generation includes natural language descriptions in addition to the function signature. These task descriptions in existing benchmarks are oftentimes too vague to fully clarify the expected program behaviors. For example, the input docstring may not specify the expected input domain (e.g., only positive integers) or how the function should handle exceptions. As a result, such programming problems can be interpreted differently by LLMs against the actual tests, leading to capable LLMs misjudged as incapable.
These limitations are common across many popular code generation benchmarks [11, 3, 33]. This not only questions the validity of the impressive performance claimed by prior work but also sets a challenge on how to properly evaluate the LLM coders. In this paper, we aim to address this fundamental evaluation challenge and ask the introspective question: Is the code generated by LLMs really correct?
Our proposal. In this work, we set out to answer the important question and evaluate the evaluation dataset. Consequently, we build EvalPlus – an evaluation framework to improve existing code benchmarks in order to precisely evaluate the functional correctness of LLM-generated code. At the heart of EvalPlus is an automatic test input generation engine which augments existing code benchmarks by generating interesting test inputs to fully exercise the code solution and check its functional correctness by cross-checking the ground-truth implementation. Specifically, EvalPlus adopts both LLM- and mutation-based [57, 74, 47] methods to automatically generate and diversify additional test inputs. EvalPlus first uses ChatGPT [48] to generate a set of high-quality seed inputs that aim to test difficult corner cases and functionalities of the program within the valid input structure. Using these high-quality seed inputs, EvalPlus then performs type-aware mutation to efficiently generate a large number of additional test inputs. These newly generated test inputs are then used to evaluate the LLM-generated code through differential testing [40] against the ground-truth implementation. Furthermore, as an option to speed up evaluation, EvalPlus also builds minimal test-suites by only
Figure 2: Overview of EvalPlus
including the most valuable test-cases, which are selected by running a greedy set cover algorithm to preserve the same code coverage [24], mutation analysis [7] as well as empirical LLM sample killings.
Contribution. Our work revisited and proposed to automatically improve code benchmarks for LLMs:
- Study: We are the first to study the test inadequacy problem in current programming benchmarks which can lead to largely over-approximated functional correctness. Our study also opens up a new research direction for precisely and rigorously evaluating LLM-synthesized code.
- Approach: We propose EvalPlus – an evaluation framework to reveal the real correctness of LLM-synthesized code. The test-case generation approach of EvalPlus combines the emerging LLM-based and traditional mutation-based test input generation. It first uses LLM-based strategy to bootstrap the test generator with high-quality seed inputs and then further extends large amounts of inputs via type-aware mutation. We then optionally “distill” the generated tests to a much smaller yet almost equivalently effective test-suite via greedy set covering. We also propose to annotate each programming tasks using program contracts to filter out invalid inputs.
- Results: EvalPlus extends the popular HUMANEVAL benchmark to create HUMANEVAL $^+$ , improving the test-case scale by  $80 \times$ . Through test-suite reduction, we also produce HUMANEVAL $^+$ -MINI which distills HUMANEVAL $^+$  tests by  $47 \times$  while still achieving a similar level of testing effectiveness. Our extensive evaluation over 26 popular LLMs surprisingly finds that the pass@ $k$  on the new dataset is up-to  $19.3 - 28.9\%$  (for different  $ks$ ) lower than the base HUMANEVAL, showing that testing insufficiency can largely affect the result analysis for almost all recent work on LLM-based code generation. Meanwhile, on the original HUMANEVAL both of the 34B WizardCoder-CodeLlama [38] and Phind-CodeLlama [52] models are deemed to be no better than ChatGPT, while HUMANEVAL $^+$  corrected the ranking and shows that the two open-source models are actually better. Additionally, we even found that the ground-truth solutions of HUMANEVAL can be erroneous, further calling into question the quality of code synthesis benchmarks.
Figure 2 shows the overview of EvalPlus. We first take in as input the original dataset containing the ground-truth implementation as well as the base test inputs. EvalPlus starts with constructing a prompt using the original ground-truth, exemplary test inputs as demonstration, and a specialized instruction to query ChatGPT and generate a set of high-quality seed inputs. ChatGPT, by following base input formats and inspecting the ground-truth solution, can serve as a vehicle to generate valid yet rigorous test inputs. Starting from these seed inputs, we then perform type-aware mutation to quickly generate numerous new inputs together with seed inputs to extensively evaluate the functional correctness of LLM-generated code. We use differential testing [40] as the oracle to cross-check the output of the ground-truth and LLM-generated solution. As an option to speed up evaluation, EvalPlus
Table 1: List of basic type-aware mutations over input  $x$  .
Type	Mutation	Type	Mutation
int | float	Returns x±1	List	Remove/repeat a random item x[i]
Insert/replace x[i] with Mutate(x[i])
bool	Returns a random boolean	Tuple	Returns Tuple(Mutate(List(x)))
NONE	Returns None	Set	Returns Set(Mutate(List(x)))
str	Remove a sub-string s	Dict	Remove a key-value pair k→v
Repeat a sub-string s	Update k→v to k→Mutate(v)
Replace s with Mutate(s)	Insert Mutate(k)→Mutate(v)
runs set covering to minimize the generated test-suite while preserving the same level of testing effectiveness. As the final output, EvalPlus obtains a augmented benchmark using the generated high-quality test inputs to fully evaluate the functional correctness of LLM-synthesized code.
2.1 Automated Test Input Generation
Seed initialization via ChatGPT. EvalPlus first uses ChatGPT to generate a set of high-quality seed inputs for later mutation. Following Figure 2, we construct a prompt using  $(i)$  the ground-truth solution of the problem for ChatGPT to inspect;  $(ii)$  a set of test inputs as demonstration; and  $(iii)$  an instruction to encourage ChatGPT to come up with interesting inputs. Specifically, each prompt starts with the ground-truth implementation and then randomly sampled test inputs from the existing dataset. We then finalize the prompt with a selected instruction in Figure 2 and query ChatGPT to produce new inputs. EvalPlus aims to leverage the powerful understanding ability of ChatGPT to learn both the valid input formats (e.g., variable types) as well as the desired functionality of the ground-truth solution in order to produce meaningful test inputs to reveal bugs in incorrectly synthesized code. Programs can have their own expected input formats, where invalid inputs should not be passed into the function as they can incur undefined behaviors to create false-positives in differential testing. As such, we filter out any invalid inputs which violate the input precondition required by the ground-truth implementation.
By using ChatGPT as an automated generation engine, we can generate inputs that are valid even under semantic constraints. For example, a programming problem may require the input to conform to a specific structure (e.g., a palindrome). Such semantic constraints can be extremely difficult for traditional input generators to satisfy. However, ChatGPT is unsuitable for large amounts of automated test generation due to undesired speed and cost of querying such a large model. To address this, we perform type-aware input mutation starting from high-quality seed inputs generated by ChatGPT.
Type-aware input mutation. We follow a typical mutation-based fuzzing workflow [74, 57] to continuously create inputs: (i) a corpus of seed inputs from ChatGPT are used to initialize the seed pool and bootstrap the generation pipeline; (ii) each time an input (i.e., seed) from the seed pool is randomly selected to be mutated to a new input (i.e., mutant); and (iii) new inputs that comply with the program contract ( $\S 2.3$ ) are added to the seed pool and we start over from (ii) to continue the generation process.
To efficiently create more valid inputs, we leverage type-aware mutation [66] in step (ii) which inspects the data types of the incoming valid seeds and generates new inputs that are structurally similar to the seeds. In Table 1 we illustrate the basic mutations used for different types of inputs. For simple primitive types such as int and float, the mutation is as simple as incrementing/decrementing the value. For compound types and the string type (i.e., str), besides generally removing or repeating existing elements (or sub-strings for str), the elements and sub-strings can be mutated recursively according to their inner types. Such sub-mutants can then be used to replace existing items or add new items in a finer-grain manner. In addition, to alleviate generating inputs that violate subtle semantic constraints, following [23, 34], we additionally apply an ingredient mechanism to collect appeared data fragments and reuse them during mutation. In short, type-aware input mutation builds on the high-quality seed inputs produced by ChatGPT to generate large amounts of test inputs which we use as the final set of extensive test inputs to evaluate LLM-synthesized code.
2.2 Test-Suite Reduction
While the large number of newly generated tests in EvalPlus are effective in detecting incorrect code, the test execution can be costly. As an option to more efficiently evaluate LLM-generated code, we further investigate test-suite reduction strategies [75, 59], which aim to select a subset of the original test-suite while still maintaining the original test effectiveness. To perform test reduction, it is typically assumed that each test can fulfill a set of testing requirements. The problem can then be formalized as reducing the original test-suite  $\mathcal{T}$  into  $\mathcal{T}_{red}$ , such that  $\forall r\in \mathcal{R}$  ( $\exists t\in \mathcal{T}$ ,  $t$  satisfies  $r\Rightarrow \exists t'\in \mathcal{T}_{red},t'$  satisfies  $r$ ). In other words, any testing requirement  $r$  satisfied by the original test-suite should still be satisfied by the reduced one. Finding such minimal representative subset for a given test-suite is equivalent to the set covering problem [17]. To solve this problem effectively, it is crucial to define the testing requirements accurately. In this paper, we focus on the following types of requirements:
Code coverage: Code coverage [24] measures the amount of code elements (e.g., statements or branches) executed by each test, and has been widely used in practice to measure test effectiveness. In this strategy, following traditional test-suite reduction [53] we leverage the widely used branch coverage as the testing requirement. In other words, the goal of using this metric is to only preserve a minimal subset of tests which can cover the same set of branches as the full tests.
Mutant killings: Coverage measures the extent to which the code has been executed; however, a high-coverage test-case is not necessarily effective in finding critical defects in its covered code. Consequently, researchers have proposed mutation testing [7] (also known as mutation analysis) to more precisely evaluate test effectiveness. In short, mutation testing applies a set of predefined mutation rules (e.g., changing “<” and “≤”) to the program under test (i.e., the ground-truth solutions for this case) to create a large number of artificial buggy programs, each of which is called as a mutant and includes exactly one subtle bug seeded. In this way, the ratio of mutation bugs detected by the tests (also called killed) can be used to assess the test effectiveness. In fact, studies have shown that mutation testing can largely outperform code coverage in test effectiveness evaluation [51]. Following prior work [59], we also leverage the set of mutants killed by each test as our testing requirement. Consequently, the goal is to minimize the number of tests while still being able to detect the same set of mutation bugs.
LLM sample killings: Different LLMs could fail commonly over certain test-cases. Consequently, besides these theoretical metrics, we also use as a testing requirement by empirically looking at sample killings, i.e., the set of wrong LLM samples that a test-case can detect and falsify. Of course, for a new LLM under evaluation, we do not have any test execution results for its code samples. Therefore, we only use the execution results for samples generated by other LLMs to evaluate test effectiveness for reduction (i.e., leave-one-out cross validation [22]). As such, we minimize the number of tests while making sure that all incorrect samples synthesized by other models can be detected by the reduced test-suite.
Besides the above three strategies, we also investigate another strategy that merges all three testing requirements for reduction. That is, the goal is to minimize the number of tests while still maintaining the same branch coverage, mutant killing, and incorrect sample detection results.
2.3 Program Input Contracts
The goal of evaluating code synthesis is to check whether the synthesized code accurately reflects the desired user intent. This is done by using several test inputs and comparing the output of the generated code against that of the ground-truth solution. The prior sections demonstrated how to improve the test inputs used to more rigorously evaluate the synthesized code. However, these user intents (expressed as natural language docstring) can be too vague for LLMs to follow. As such, LLMs might allow for different interpretations of the desired functionality, input formats as well as how to handle corner cases.
To this end, we adopt a programming by contract [41] philosophy by systematically annotating function pre-conditions in form of code assertions (e.g., assert n > 0), to ensure the test inputs for the function are well-formed. The benefits of the contracts are two-fold: (i) they can complement the automatic input generation steps to filter out any generated invalid inputs that violate the contracts. Such ill-formed inputs can incur undefined behaviors which are unreasonable to use for evaluating LLM-synthesized code; and (ii) they can serve as orthogonal descriptors together with the natural language description in the prompt for further clarification.
Table 2: Overview of EvalPlus-improved benchmarks.
#Tests	#Tasks
Avg.	Medium	Min.	Max.
HUMANEVAL	9.6	7.0	1	1052
HUMANEVAL+	764.1	982.5	12	1,100	164
HUMANEVAL-MINI	16.1	13.0	5	110
Setup. Our evaluation focuses on using the unbiased version of  $\text{pass} @ k$  [11] to accurately assess the functional correctness of LLM-synthesized code. For generalizability, we conducted a comprehensive evaluation over 26 popular and state-of-the-art LLMs and a wide range of temperature settings. Specifically, following prior work [11, 46], for each model we perform: (i) random sampling to generate 200 program samples for each of the four temperature settings ( $\{0.2, 0.4, 0.6, 0.8\}$ ); and (ii) greedy-search decoding. For random sampling, we show the best-performing  $\text{pass} @ k$  for each  $k \in \{1, 10, 100\}$  and its corresponding temperature denoted by  $T_k^*$ . For greedy decoding, we only synthesize one deterministic sample for each task and evaluate its pass rate as  $\text{pass} @ 1^*$ . By default we evaluate models under both setting (i) and (ii), except for the two commercial models due to time and cost constraints: GPT-4 is only evaluated under greedy decoding, and ChatGPT is additionally evaluated on 0.8-temperature random sampling.
While EvalPlus is general, this paper focuses on evaluating its effectiveness on HUMANEVAL [11], one of the most widely-used datasets for code generation<sup>3</sup>. HUMANEVAL consists of 164 human-written programming tasks, each of which provides a Python function signature and a docstring as the input to the LLM. Based on the input, LLMs complete a solution whose functional correctness is judged by a handful of manual test-cases (the first row in Table 2). As such, EvalPlus transforms HUMANEVAL to HUMANEVAL<sup>+</sup> by adding  $80 \times$  unique test-cases and fixing incorrect ground-truth solutions in HUMANEVAL. Specifically, for each task, based on around 30 ChatGPT-generated seed inputs which are produced using 3 separate prompts, we run type-aware mutation to generate 1000 additional inputs using one-hour budget. In HUMANEVAL<sup>+</sup>, 83 out of the 164 programming tasks are annotated with hand-crafted contracts. Because EvalPlus requires ground-truth solutions to cross-check LLM-generated code, it is crucial to ensure the correctness of the ground-truths. However, by inspecting ground-truths in the original HUMANEVAL, we found over  $10\%$  of them are incorrectly implemented. Therefore, as another contribution we carefully re-implemented and tested all ground-truths for HUMANEVAL<sup>+</sup>. As an option to speed up evaluation, we build HUMANEVAL<sup>+</sup>-MINI which is minimized from HUMANEVAL<sup>+</sup> (smaller by  $47 \times$ ) yet preserves similar test effectiveness on the studied models. Lastly, more experimental setups are detailed in Appendix.
Evaluation of LLMs. Table 3 shows the pass@k when evaluating LLMs using both the base HUMANEVAL and HUMANEVAL+. We first observe that across all LLMs, models sizes and k values, using HUMANEVAL+, almost all pass@k results consistently drop compared to using the base HUMANEVAL. Notably, the performance drop is significant with up-to  $23.1\%$  (pass@1*) /  $19.3\%$  (pass@1) /  $24.9\%$  (pass@10) /  $28.9\%$  (pass@100) reduction over the evaluated models. Such performance decrease is not only seen in popular open-source LLMs, such as the widely used CodeGen-16B [46] (18.5% reduction) as well as the emerging CODELLAMA-34B [54] (17.6%) and StarCoder [13] (14.1% reduction), but also observed in state-of-the-art commercial ChatGPT (12.6% reduction) and GPT-4 (13.1% reduction) models. Overall, our results overall confirm our hypothesis that the prior evaluation on HUMANEVAL is not robust enough to detect wrong code synthesized by LLMs. Not only are these LLMs widely used for daily programming but they also serve as common reference points for evaluating new code synthesis techniques. As such, evaluating on a more robust benchmark such as HUMANEVAL+ is highly recommended in order to draw precise conclusions.
There are four HUMANEVAL tasks (e.g., add(x, y)) with over 100 "tests" (i.e., implemented by cross-checking the ground-truth over random inputs). Without such, the maximum/average number is 26/7.3.