SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>3</sup>Top-1 HuggingFace downloads on April, 2023. https://hf.co/datasets?other=code-generation
4To date, CodeGen2-16B is released with an unfinished checkpoint [45]. Nonetheless, we show its pass@1*.
Table 3: Evaluating LLMs on HUMANETVAL and HUMANETVAL+. All models, except for INCODER, CodeGen2, StarCoder and SantaCoder which perform infilling, use auto-regressive generation.  $k = 1^{\star}$  marks pass@1 done with greedy decoding.  $T_{k}^{*}$  denotes the optimal pass@k temperature.
Size	pass@k	k=1*	k=1	k=10	k=100	T1*	T10*	T100*
GPT-4 [49]	N/A	base	88.4
+extra	76.2
Phind-CodeLlama [52]	34B	base	71.3	71.6	90.5	96.2	.2	.8	.8
+extra	67.1	67.0	85.0	92.5	.2	.8	.8
WizardCoder-CodeLlama [38]	34B	base	73.2	61.6	85.2	94.5	.2	.8	.8
+extra	64.6	54.5	78.6	88.9	.2	.8	.8
ChatGPT [48]	N/A	base	73.2	69.4	88.6	94.0
+extra	63.4	62.5	82.1	91.1
CODELLAMA [54]	34B	base	51.8	52.0	82.4	95.0	.2	.8	.8
+extra	42.7	43.1	73.7	89.4	.2	.8	.8
13B	base	42.7	44.6	77.6	92.7	.4	.8	.8
+extra	36.6	37.4	69.4	88.2	.4	.8	.8
7B	base	37.8	39.2	69.1	89.7	.2	.8	.8
+extra	34.1	34.5	61.4	82.9	.2	.8	.8
StarCoder [13]	15B	base	34.1	32.2	56.7	84.2	.2	.8	.8
+extra	29.3	27.8	50.3	75.4	.2	.8	.8
CodeGen [46]	16B	base	32.9	32.2	56.0	81.5	.2	.6	.8
+extra	26.8	27.2	48.4	71.4	.2	.6	.8
6B	base	29.3	27.7	46.9	72.7	.2	.6	.8
+extra	25.6	23.6	41.0	64.6	.2	.6	.8
2B	base	24.4	18.4	39.8	66.8	.2	.8	.8
+extra	20.7	15.1	34.8	55.8	.2	.2	.8
CODET5+ [64]	16B	base	31.7	32.2	58.5	83.5	.2	.6	.8
+extra	26.2	27.4	51.1	76.4	.2	.6	.8
MISTRAL [26]	7B	base	28.7	28.1	55.2	83.8	.2	.8	.8
+extra	23.8	23.7	48.5	76.4	.2	.8	.8
CodeGen2 [45]	16B4	base	19.5
+extra	16.5
7B	base	18.3	17.9	30.9	50.9	.2	.6	.8
+extra	16.5	15.9	27.1	45.4	.2	.6	.8
3B	base	15.9	15.2	23.9	38.6	.2	.4	.8
+extra	12.8	12.9	21.2	34.3	.2	.4	.8
1B	base	11.0	10.2	15.1	24.7	.2	.6	.6
+extra	9.1	8.7	13.7	21.2	.2	.6	.6
VICUNA [12]	13B	base	16.5	15.3	30.1	54.8	.2	.8	.8
+extra	15.2	13.9	25.8	46.7	.2	.8	.8
7B	base	11.6	10.9	23.8	42.3	.2	.6	.6
+extra	11.0	10.3	20.3	35.0	.2	.6	.6
SantaCoder [2]	1.1B	base	14.6	16.6	29.2	45.4	.4	.6	.8
+extra	12.8	14.2	26.2	40.6	.4	.6	.8
INCODER [18]	6.7B	base	15.9	15.6	27.7	45.0	.2	.4	.6
+extra	12.2	12.4	22.2	38.9	.2	.6	.6
1.3B	base	12.2	10.0	15.9	25.2	.2	.6	.6
+extra	10.4	7.9	13.5	20.7	.2	.6	.4
GPT-J [63]	6B	base	12.2	11.3	17.7	31.8	.2	.6	.6
+extra	10.4	9.5	15.2	25.9	.2	.6	.6
GPT-NEO [5]	2.7B	base	7.9	6.5	11.8	20.7	.2	.6	.6
+extra	6.7	6.0	9.0	16.8	.2	.6	.6
PolyCoder [70]	2.7B	base	6.1	5.9	10.2	17.1	.2	.4	.6
+extra	5.5	5.3	7.9	13.6	.2	.6	.6
StableLM [60]	7B	base	2.4	2.7	7.5	15.8	.2	.6	.6
+extra	2.4	2.6	6.2	11.9	.2	.6	.6
Table 4: Reduced test-suite for HUMANEVAL+. We first show the pass@1* and average #tests (including base HUMANEVAL tests) by only doing set covering over each considered metric separately (\$2.2). The Full column then shows the final reduction result by combining all of the three. For reference, the average #tests of original HUMANEVAL and HUMANEVAL+ are 9.6 and 774.8 respectively (Table 2).
Size	Coverage	Killed mutants	Killed samples	Full	Ref. pass@1*
pass@1*	#tests	pass@1*	#tests	pass@1*	#tests	pass@1*	#tests	base	+extra
GPT-4	N/A	86.0	11.3	82.9	11.4	78.7	13.8	78.0	16.1	88.4	76.2
ChatGPT	N/A	71.3	11.3	69.5	11.4	65.2	13.7	65.2	16.0	73.2	63.4
StarCoder	15B	32.9	11.3	32.9	11.4	29.3	13.6	29.3	15.9	34.1	29.3
CodeGen	2B	23.2	11.3	23.8	11.4	21.3	13.2	21.3	15.4	24.4	20.7
6B	28.7	11.3	29.3	11.4	25.6	13.2	25.6	15.4	29.3	25.6
16B	31.7	11.3	31.1	11.4	27.4	13.2	27.4	15.4	32.9	26.8
CodeGen2	1B	10.4	11.3	11.0	11.4	9.1	13.8	9.1	16.0	11.0	9.1
3B	15.9	11.3	15.9	11.4	12.8	13.8	12.8	16.0	15.9	12.8
7B	18.3	11.3	18.3	11.4	16.5	13.8	16.5	16.0	18.3	16.5
16B	19.5	11.3	18.9	11.4	16.5	13.8	16.5	16.0	19.5	16.5
VICUNA	7B	11.6	11.3	11.6	11.4	11.0	13.8	11.0	16.1	11.6	10.4
13B	16.5	11.3	16.5	11.4	15.2	13.8	15.2	16.1	17.1	15.2
SantaCoder	1.1B	14.6	11.3	14.6	11.4	12.8	13.8	12.8	16.1	14.6	12.8
INCoder	1.3B	12.2	11.3	12.2	11.4	10.4	13.6	10.4	16.0	12.2	10.4
6.7B	14.6	11.3	14.6	11.4	12.2	13.6	12.2	16.0	15.9	12.2
GPT-J	6B	12.2	11.3	12.2	11.4	10.4	13.8	10.4	16.0	12.2	10.4
GPT-NEO	2.7B	7.3	11.3	7.3	11.4	6.7	13.8	6.7	16.1	7.9	6.7
PolyCoder	2.7B	6.1	11.3	6.1	11.4	5.5	13.8	5.5	16.1	6.1	5.5
StableLM	7B	2.4	11.3	2.4	11.4	2.4	13.8	2.4	16.1	2.4	2.4
We also show that a more rigorous evaluation could yield different or totally contradictory relative results. For example, WizardCoder-CodeLlama and Phind-CodeLlama on the original HUMANEVAL are evaluated to be no better than ChatGPT in terms of pass@1*. However, HUMANEVAL+ demonstrates that the two open-source models can actually outperform the proprietary ChatGPT. Other contrary examples reflected by HUMANEVAL+ include that SantaCoder-1B surpasses INCODER-6.7B and VICUNA-7B outperforms INCODER-1.3B. Table 3 further illustrates the distribution of best-performing temperatures over different  $k$  values. Our results conform with prior findings [11] that a lower temperature tends to perform better for smaller  $k$ , while a higher temperature works better for larger  $k$ . We also observe that the optimal temperatures seem to stay fairly consistent before and after using HUMANEVAL+; however, slight differences still exist, e.g., best temperature for CodeGen-2B on pass@10 becomes 0.2 from 0.8 after using HUMANEVAL+. Nonetheless, this motivates future research to look more closely on the effect of temperature with respect to the robustness of the evaluation tests, esp. those edge-cases.
Effectiveness of test-suite reduction. Based on HUMANEVAL $^{+}$  which on average obtains 764.1 tests for each programming task (Table 2), our test-suite reducer ( $\S 2.2$ ) minimizes it to HUMANEVAL $^{+}$ -MINI which only has 16.1 tests for each task (smaller by  $47 \times$ ). Table 4 performs leave-one-out cross validation to show the pass@1\* differences over a subset of representative models studied in Table 3 (due to time/space constraints). That is, for each evaluated LLM we construct the reduced test-suite without considering its own sample kills. The Full column shows that the reduced test-suite can achieve almost the same pass@1\* drop as HUMANEVAL $^{+}$  by only using  $47 \times$  fewer test-cases. Taking a closer look, separately performing set covering over each metric can harness the pass@1\* of the base HUMANEVAL to certain degree. Specifically, the use of empirical LLM sample killings is the most effective, leading to the same effectiveness as the full approach, but also consumes more tests than other theoretical metrics. While using coverage and mutation analysis seems to be unnecessary in addition to using sample killings, they still serve as the base guarantees for the theoretical test adequacy.
Pass rate distribution. Figure 3 shows for each programming task the overall pass rates on HUMANEVAL and HUMANEVAL+ tests. The pass rate gap between HUMANEVAL and HUMANEVAL+ shows overall HUMANEVAL+ can detect solutions that are misidentified by HUMANEVAL for problems of all levels of difficulties. We also observe that problems in HUMANEVAL are not equal, not only in terms of problem difficulty but also the difficulty of generating counter-examples and
Figure 3: Pass rate distribution. X-axis spans bars for all 164 problems, sorted by the HUMANEVAL pass rate. Y-axis shows the log-scale pass rates averaged by all LLM-generated samples.
edge-cases to deeply exercise LLM-generated code. For simple problems such as "adding two numbers" and "length of a string" (i.e., problems with top-2 pass rates), it is easy to solve for LLMs and to test manually. While problems dealing with multiple conditions (e.g., "word splitting"), completeness (e.g., handling negative numbers for "is-prime"), reasoning ability (e.g., "Tribonacci sequence") and efficiency requirements (e.g., "n-th prime Fibonacci number") are the hardest tasks to the evaluated LLMs, positioning future research to improve LLMs for conquering such coding skills.
Incorrect "ground-truth" in HUMANEVAL. In addition to detecting wrong code from LLMs using EvalPlus, we also found 18 defects (11% of problems) even in the original ground-truth in HUMANEVAL, including (i) Unhandled edge-case: five prior ground-truths fail to handle corner-case inputs (e.g., empty list or string); (ii) Bad logic: 10 prior ground-truths incorrectly implement the desired functionality; and (iii) Performance issue: three inefficient implementations lead to slow performance on reasonably-sized inputs. Among those, bad logic (10) is the most serious as the original "ground-truth" does not accurately reflect the user intent. Such defects are detected also through differential testing but between our own re-implemented ground-truth and the original ground-truth in HUMANEVAL.
Figure 4: Exemplary incorrect-logic ground-truth solution in HUMANEVAL (#124)
Figure 4 shows an incorrect ground-truth implementation (validate_date) from HUMANEVAL classified as having bad logic. The desired task is to check if the input date format is correct. We see that in the core logic, the conditions attempt to first check the month condition and then handle the corresponding day conditions. However, this is implemented incorrectly as "and" in Python<sup>5</sup> has higher precedence than "or", leading to the ground-truth function to check if either conditions satisfy instead of the desired both conditions must satisfy. This is exposed via our automatically generated test input of 12-31-1999 where the ground-truth implementation incorrectly labels this as not a valid date. Surprisingly this egregious error is not exposed by any of the base test inputs in HUMANEVAL, further demonstrating the weakness and limited evaluation power of the original test inputs.
LLMs for code. The use of LLMs for code has gained traction in recent years, owing to the abundance of open codebase and the need for improving developer efficiency. LLMs have demonstrated state-of-the-art performance on various code-related tasks, including code generation [11, 33, 25], program repair [69, 27, 68, 65], automated testing [15, 14, 67, 35, 71], code translation [31, 55] and code summarization [1, 37]. In particular, prominent LLMs including CODEX [11], CodeGen [46], INCODER [18] and PolyCoder [70], have been developed and extensively evaluated for code
<sup>5</sup>https://docs.python.org/3/reference/expressions.html#operator-precedence
generation (widely recognized as the holy grail for computer science research since the inception of AI in the 1950s [21]), where the model generates code snippets based on natural language descriptions (e.g., docstring) of the desired functionality.
Coding benchmark for LLMs. LLM-based code synthesis is largely evaluated based on functional correctness, which is typically assessed by running test-cases to check the desired outputs. HUMANEVAL [11] is one of the pioneering and most widely studied human-written benchmarks for LLM-based code synthesis, consisting of 164 pairs of Python function signature with docstring and the associated test-cases for correctness checking. Additionally, each HUMANEVAL problem is also equipped with a reference solution. Another Python-focused dataset, MBPP [3], is created by crowd-sourcing participants to write in summation 974 programming problems, each of which is comprised of the problem statement (i.e., docstring), the function signature, as well as three test-cases. Beyond Python, there are other benchmarks targeting additional languages such as Spider [73] (SQL), HUMANEVAL-X [76] (C++, Javascript and Go), CodeContests [33] (C++ and Java) and MultiPL-E [9] (extending HUMANEVAL and MBPP to 18 programming languages). More recently, researchers have created a more realistic code synthesis benchmark by collecting GitHub issues along with the corresponding code base together with tests to measure the ability of LLMs to perform real-world software engineering tasks [28]. Our work shows for the first time the test inadequacy problem of widely studied benchmarks and addresses the issue via automatic test generation.
Automated test generation. Automated test generation is a widely used for finding software bugs with automatically generated tests. Black-box test generation such as fuzz testing [43] feeds random inputs (e.g., random bytes) to the system under test (SUT), without knowing its source code. Traditional black-box techniques can mainly be categorized into generation-based [72, 23, 56] and mutation-based [66, 10, 47] ones. White-box approaches provide better-quality test-cases by analyzing the source code of SUT. For instance, symbolic execution [30, 8] breaks the coverage plateaus by solving symbolic path constraints to generate tests targeting deep paths. As a mid-point, coverage-guided fuzzing [74, 57] (i.e., grey-box) uses the coverage information of SUT as feedback to adjust the input generation and mutation. The discussed traditional methods are inapplicable to generating semantically meaningful inputs for arbitrary problems programmed in a dynamically-typed language. We address this by using ChatGPT to inspect the ground-truth (i.e., white-box) for initializing interesting seeds, based on which type-aware mutation (i.e., black-box) scales the test inputs to a large amount.
We present EvalPlus – a rigorous evaluation framework for program synthesis, driven by automated test generation. EvalPlus combines both LLM- and mutation-based input generation to obtain a diverse set of test inputs for accurately evaluating the correctness of LLM-generated code. EvalPlus creates HUMANEVAL $^+$ , built on top of the popular HUMANEVAL with additional high-quality and automatically generated test inputs. With test-suite reduction, EvalPlus also produces HUMANEVAL $^+$ -MINI which is smaller than HUMANEVAL $^+$  by  $47\times$  while preserving similar test effectiveness. We extensively evaluate a diverse set of LLMs and show that HUMANEVAL $^+$  can identify a significant amount of previously undetected wrong code generated by LLMs, demonstrating its effectiveness to augment programming benchmarks for more accurate evaluation.
Since launched, the EvalPlus PyPI package has been installed by over 6k times in 5 months. We also keep evaluating new models for code and maintain a leaderboard at https://evalplus.github.io/leaderboard.html. In the future, we plan to apply EvalPlus to bring better-quality testing for more code benchmarks such as MBPP. Meanwhile, future work can look into how to integrate EvalPlus with more formal verification (e.g., Dafny [32]) or validation techniques (e.g., translation validation [36]) to provide stronger guarantees of the evaluation results when applicable. Additionally, the core test generation technique behind can be even used to remind developers of potential flaws of the accepted LLM-generated code snippets when doing AI pair-programming (e.g., Copilot [42]).
This work was partially supported by NSF grants CCF-2131943 and CCF-2141474, as well as Kwai Inc. We thank the reviewers for their invaluable feedback. We further thank Yinlin Deng for providing helpful discussions, as well as Junhao Wang and Songrun Xie for their open-source contributions.
[38] Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023.
[39] Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):151-165, 1971.
[40] W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100-107, 1998.
[41] B. Meyer. Applying'sign by contract'. Computer, 25(10):40-51, 1992.
[42] Microsoft. GitHub Copilot - Your AI pair programmer. https://github.com/features/copilot, 2023.
[43] B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32-44, 1990.
[44] G. C. Necula. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation, pages 83-94, 2000.
[45] E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint, 2023.
[46] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023.
[47] P. Oehlert. Violating assumptions with fuzzing. IEEE Security & Privacy, 3(2):58-62, 2005.
[48] OpenAI. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/chatgpt/, 2022.
[49] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[50] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[51] G. Petrović and M. Ivanković. State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice, pages 163–171, 2018.
[52] Phind. Phind/phind-codellama-34b-v2·hugging face. https://huggingface.co/Phind/Phind-CodeLlama-34B-v2, 2023.
[53] G. Rothermel, M. J. Harrold, J. Von Ronne, and C. Hong. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability, 12(4):219-249, 2002.
[54] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
[55] B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773, 2021.
[56] M. Security. jsfunfuzz. https://github.com/MozillaSecurity/funfuzz, 2007.
[57] K. Serebryany. Continuous fuzzing with libfuzzer and addresssanitizer. In 2016 IEEE Cybersecurity Development (SecDev), pages 157-157. IEEE, 2016.
[58] D. E. Shaw, W. R. Swartout, and C. C. Green. Inferring lisp programs from examples. In *IJCAI*, volume 75, pages 260–267, 1975.
Table 5: Overview of evaluated models.
Model Name	Sizes	Release Year	Open-Source
Coding	CodeGen [46]	2B, 6B, 16B	2022	✓
INCODER [18]	1.3B, 6.7B	2022	✓
PolyCoder [70]	2.7B	2022	✓
SantaCoder [2]	1.1B	2023	✓
CodeGen2 [45]	1B, 3B, 7B, 16B	2023	✓
StarCoder [13]	15B	2023	✓
CODET5+ [64]	16B	2023	✓
CODELLAMA [54]	7B,13B,34B	2023	✓
WizardCoder-CodeLlama [38]	34B	2023	✓
Phind-CodeLlama [52]	34B	2023	✓
General	GPT-J [63]	6B	2021	✓
GPT-NEO [5]	2.7B	2021	✓
ChatGPT [48]	N/A	2022
GPT-4 [49]	N/A	2023
VICUNA [12]	7B, 13B	2023	✓
StableLM [60]	7B	2023	✓
MISTRAL [26]	7B	2023	✓
A Detailed Experimental Setup
Evaluation of LLMs. Our goal is to comprehensively evaluate recent and widely used LLMs, both specialized for code generation [46, 70, 18, 2, 52, 38, 64] and general-purpose tasks [49, 48, 12, 60, 63, 5, 26]. Table 5 presents an overview of the studied models, with column Sizes reflecting the model sizes in billions of parameters, Release Year showing when the LLM is released, and Open-Source marking the models whose weights are publicly available. In total, we evaluate 26 of the most representative and popular LLMs with a broad range of configurations to fully demonstrate the generalizability of our results.
Our hyper-parameter configurations follow prior work [11, 46]. For each model we randomly sample 200 programs and repeat the experiments over temperature (\{0.2,0.4,0.6,0.8\}) and greedy decoding with zero temperature. By default, we let each model generate at most 512 new tokens and truncate the produced code with end-of-string (EOS) identifiers suggested in HUMANEVAL [11], as well as those favoured by certain models (e.g., “
” and “\n` ``”). For conversational models (i.e., ChatGPT and GPT-4), we obtain the code fragments by parsing the code blocks (i.e., within “` ``”) in the output. We found ChatGPT tends to repeat problem description with detailed explanation, which can consume more than 512 new tokens to complete a solution for around  $11\%$  of problems. To align ChatGPT with other models, for tasks with very long problem descriptions, we extend the token limit from 512 to 1024. For model implementation, we run ChatGPT and GPT-4 via OpenAI APIs, and accelerate CodeGen-6B and -16B with NVIDIA FasterTransformer via FauxPilot [16]. All other LLMs are based on the HuggingFace transformers library. By default, we follow the official examples of each LLM (e.g., on HuggingFace model card) to construct their corresponding prompts. Specifically, the prompts used for ChatGPT, GPT-4, and WizardCoder-CodeLlama is instruction-based, i.e., a simple instruction is used to wrap the function signature and docstring to explicitly encourage the LLM for code generation.
Test oracles. An LLM-produced solution is regarded to be correct if for all test inputs it returns values that match the expected outputs within a reasonable run time. We perform exact matching by default. For floating-point comparisons, we tolerate absolute differences to the degrees annotated in HUMANEVAL or  $10^{-6}$  if not annotated. In original HUMANEVAL, the default timeout is set to three seconds to run the whole test-suite (i.e., all test-cases) for each programming problem. Such a setting is neither suitable when having more test-cases nor reasonable as each problem could have its own run time characteristics. Consequently, we let the timeout for each test-case to be  $\max(200\mathrm{ms}, 4 \times t_{gt})$  where  $t_{gt}$  refers to the execution time of the corresponding ground-truth solution. In other words, we expect the LLM-provided solution to be no slower than the ground-truth by four times or use a base 200-milliseconds second timeout when  $4 \times t_{gt} < 200\mathrm{ms}$  to avoid variance caused by performance randomness.