Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM
Authors: Chunqiu Steven Xia
Yinlin Deng
LINGMING ZHANG
Venue: COLM (2024)
Keywords: Program Synthesis, Evaluation, Dataset, LLM4Code
TL;DR: EvoEval – a program synthesis benchmark suite created by evolving existing benchmarks into different targeted domains for a comprehensive evaluation of LLM coding abilities.

Abstract: Large language models (LLMs) have become the go-to choice for code generation tasks, with an exponential increase in the training, development, and usage of LLMs specifically for code generation. To evaluate the ability of LLMs on code, both academic and industry practitioners rely on popular handcrafted benchmarks. However, prior benchmarks contain only a very limited set of problems, both in quantity and variety. Further, due to popularity and age, many benchmarks are prone to data leakage where example solutions can be readily found on the web and thus potentially in training data. Such limitations inevitably lead us to inquire: Is the leaderboard performance on existing benchmarks reliable and comprehensive enough to measure the program synthesis ability of LLMs? To address this, we introduce EvoEval – a program synthesis benchmark suite created by evolving existing benchmarks into different targeted domains for a comprehensive evaluation of LLM coding abilities. Our study on 51 LLMs shows that compared to the high performance obtained on standard benchmarks like HumanEval, there is a significant drop in performance (on average 39.4%) when using EvoEval. Additionally, the decrease in performance can range from 19.6% to 47.7%, leading to drastic ranking changes amongst LLMs and showing potential overfitting of existing benchmarks. Furthermore, we showcase various insights including the brittleness of instruction-following models when encountering rewording or subtle changes as well as the importance of learning problem composition and decomposition. EvoEval not only provides comprehensive benchmarks, but can be used to further evolve arbitrary problems to keep up with advances and the ever-changing landscape of LLMs for code. We have open-sourced our benchmarks, tools, and all LLM-generated code at https://github.com/evo-eval/evoeval.

Published as a conference paper at COLM 2024
Top Leaderboard Ranking = Top Coding Proficiency, Always? EVOEVAL: Evolving Coding Benchmarks via LLM
Chunqiu Steven Xia*
Yinlin Deng"
Lingming Zhang
University of Illinois Urbana-Champaign I
(chunqiu2, yinlind2, Singmigg/1lllinot1
Abstract
Large language models (LLMs) have become the go-to choice for code generation tasks, with an exponential increase in the training, development, and usage of LLMSs specifically for code generation. To evaluate the ability of LLMs code, both academic and industry practitioners rely on popular handcrafted benchmarks. However, prior benchmarks contain only a very limited set of problems, both in quantity and variety. Further, due to popularity and age, many benchmarks are prone to data leakage where example solutions can be readily found on the web and thus potentially in training data. Such limitations inevitably lead us to inquire: Is the leaderboard performance on existing benchmarks reliable and comprehensioe enough to measure the program synthesis ability of LLMs? To address this, we introduce EvOEVAL- program synthesis benchmark suite created by evolving existing benchmarks into different targeted domains for comprehensive evaluation of LLM coding abilities. Our study on 57 LLMs shows that compared to the high performance obtained on standard benchmarks like HUMANEVAL, there is significant drop in performance (on average 38.1%) when using EvoEvAL. Additionally, the decrease in performance can range from 19.6% to 47.7%, leading to drastic ranking changes amongst LLMs and showing potential overfitting of existing benchmarks. Furthermore, we showcase various insights including the brittleness of enstruction-oolloiing models when encountering rewording subtle changes well the importance of learning problem composition and decomposition. EvOEVAL not only provides comprehensive benchmarks, but can be used further evolve arbitrary problems to keep up with advances and the ever-changing landscape of LLMs for code. We have open-sourced our benchmarks, tools, and all LLM-generated code at at/https://ghh t/gttbbb./ com/ com/evo- htt//vooevvll/vvee eva-eevoealllllllleeeeeeeoaoaaaa
Introduction
Program synthesis (Gulwani et al. 2017) regarded as the only-grail in the field of computer science. Recently, large language models (LLMs) have become the default choice for pro- gram synthesis due to its code reasoning capabilities acquired through training on code repositories. Popular LLMs like GPT-4 (OpenAl 2023), Claude-3.5 (Anthropic 2024b), and Gemini [Team et al. 2023) have shown tremendous success in aiding developers on a wide range of coding tasks (Chen et al. 2021 Xia & Zhang 2023 Deng et al. 2023b). Furthermore, researchers and practitioners have designed code LLMs (e.g., DeepSeek Coder (Guo et al., 2024), CodeLlama (Rozière al. 2023), and StarCoder (Li al. 2023 using a variety of train- ing methods designed specifically for the code domain to improve LLM code understanding.
Coding benchmarks like HUMANEVAL (Chen et al., 2021) and MBPP (Austin al., 2021) have been handcrafted to evaluate the program synthesis task of turning natural language descriptions (e.g,, docstrings) into code snippets. These code benchmarks measure
* "Contributed equally with author ordering decided by Nigiri
1

---

Published as a conference paper at COLM 2024
cer slintran: anrd
rellesire thas rvlar Ef the nen wert cafte Anstasee caraact, esplace ajos
eutte hastien cest siic takes sorten peareenting
retuon Tarle centolalag
Creatiie Frokiww
sore meaddh
too artut of plarety, serted
Frokie
SURREE coeengts sogether
anite Puaition eoeetin must anien
Aude charges
za the string. feurls this cos
EaT plaeut, rest, cacetpurett plawezt: tame Flanet name and slawtz letem Intraun amproratiin the locatse betwes serted stweet and crstt stewet
the and of the glons sare.
pridiem thet
Buzill mozillery writale
Contter Fridlim
soett, countaa) He stvmn word. Macacaat Subcle Prebten
chack (oa
uach vaut. scriag. kitam the dictiesrs. *41
Tao Uhe Pribles
Figure 1: Exemplar problems generated in EvOEVAL starting from a HUMANEVAL HUMANEYAl.problem.
functional correctness by evaluating LLM-generated solutions against set of limited predefined tests. Recent work (Liu et al. 2023) has further included augmented tests to rigorously evaluate the functional correctness of LLM generated code. However, apart from test inadequacy, existing popular code synthesis benchmarks have the following limitations:
Limited amount and variety of problems. Code benchmarks are mainly constructed by human annotators manually. Due to the high manual effort required, they only contain limited amount of problems (e.g., only 164 problems in HUMANEVAL). Such a low amount of problems is not sufficient to fully measure the complete spectrum of program synthesis capability of LLMs. Additionally, prior benchmarks include mostly self-contained prob- lems that lack variety in both types and domains, where the final evaluation output only shows the percentage of problems solved. While they provide a baseline overview of the coding abilities, LLM builders and users cannot gain deeper insights to exactly which problem types or coding scenarios the particular LLM may excel in or struggle with. Prone to data leakage and training dataset composition. Popular benchmarks like HU- MANEVAL and MBPP were released almost 4 years ago, with example solutions available in third-party open-source repositories. In fact, recent work (Riddell et al., 2024) has shown that there is substantial overlap between benchmark solutions and open-source training corpuses. Furthermore, the problems within these benchmarks are often simple derivatives of common coding problems. While recent LLMs have been climbing the leaderboard by achieving higher pass@1 scores (often with minimal difference between the next best model), it is unclear whether high scores achieved by LLMs are truly due to their learned coding capability or instead obtained via memorizing benchmark solutions.
As more LLMs are being constructed, trained, and used especially for code, the insufficient evaluation benchmarks raise a critical question: Is leaderboard performance on existing benchmarks reliable and comprehensive enough to measure the program synthesis ability of LLMs?
Our work. To address the limitation of existing benchmarks, we introduce EvOEVAL- a set of program synthesis benchmarks created by evolving existing problems. The key idea behind EVOEVAL is to use LLMs to automatically transform existing problems into targeted domains, enabling more comprehensive evaluations. Unlike prior benchmark construction approaches, which either obtain problems from open-source repositories (posing data leakage risks) or require manual construction (resulting in high manual effort and limited diversity), EVOEVAL leverages LLMs with targeted transformations to synthesize new coding problems. Specifically, we design five such transformations: Difficult, Creative, Subtle, Combine, and Tool Use. We then prompt GPT-4 to independently transform any existing problem in previous benchmarks into a new problem within the targeted domain.
Figure shows concrete example of EvOEVAL starting with initial problem in HU- MANEVAL- vowels count to count the number of vowels. We first observe the transforma- tion to more difficult problem by asking GPT-4 to add additional requirements. This new
2

---

Published as a conference paper at COLM 2024
problem contains a separate custom vowel list that makes the overall program logic more complex. We can craft a more creative problem of create_alias that still uses concepts like vowels and consonants but involves much more innovative and unusual problem descrip- tion. We can also make subtle changes to the problem where we only count the lowercase vowels to test if the LLM is simply memorizing the benchmark. We can additionally com- bine concepts from multiple problems together. In the example, we use another problem bf to create a new problem that returns the vowels in each planet sorted based on the orbiting order. Furthermore, we can test LLMs' ability to utilize helper functions (commonplace in real-world code repositories) to solve more complex problems. Again, we reuse the concepts of vowels from the initial problem. However, instead of directly solving the problem, the LLM can use the provided check vowel helper function to simplify the solution.
Together, these transformed benchmarks are designed to introduce more challenging problems and assess different aspects of LLMs' code understanding and synthesis abilities. In EvOEVAL, we additionally use GPT-4 to generate the ground truth solution to each problem as well as rigorous test cases to evaluate the functional correctness of LLM-syythesized code. Finally, we manually check each generated problem and ground truth to ensure problem clarity and correctness. EVOEVAL serves as a way to further evolve existing benchmarks into more complex and well-suited problems for evaluation in order to keep up with the ever-growing LLM research. Our work makes the following contributions:
Benchmarks: We present EvOEVAL- a set of program synthesis benchmarks created by evolving HUMANEVAL problems. EvOEVAL includes 828 problems across 7 benchmarks, equipped with ground truth solutions and test cases to evaluate functional correctness. Approach: We propose a complete pipeline to generate new coding problems for benchmarking by evolving existing problems through targeted transformations via LLMs. Furthermore, our pipeline reduces manual checking effort by automatically refining problem inconsistencies, generating ground truth, and producing test cases.
Study: We conduct a comprehensive study on 57 LLMs. We found that compared to the high performance on prior benchmarks, LLMs significantly drop in accuracy (average 38.1% %) on EvOEVAL. Additionally, this drop is not uniform across LLMs (from 19.6% to 47.7%), leading to drastic ranking changes. We further demonstrate that certain LLMs cannot keep up their high performance when evaluated on more challenging tasks or problems in different domains, highlighting the possibility of overfitting to existing benchmarks. Moreover, we observe that instruction-folooingg LLMs are sensitive to rephrasing subtle changes in the problem description. They also struggle with utilizing already provided auxiliary functions. We further demonstrate that current LLMs fail to effectively compose multiple general coding concepts to solve more complex variants, or address subproblems decomposed from problems they previously solved.
2 Approach
Targeted problem transformation. We first prompt a powerful LLM to evolve an existing problem into new one using a transformation prompt. Each transformation prompt aims to transform the existing problem in a specific manner. We define two different transformation types: semantic-altering cchanges the semantic meaning of the problem and semantic-preservingg - modifies the description while keeping the same semantic meaning.
Problem refinement & ground truth generation. The initial evolved problem produced by the LLM may include inconsistencies like incorrect examples. For coding benchmarks, such mistakes can lead to inaccurate evaluation. As such, we introduce a refinement pipeline to iteratively rephrase and refine the problem as needed. We first query the LLM to obtain possible solution and test inputs for the initial problem. We then evaluate the test inputs on the solution to derive the expected outputs. Next, we instruct the LLM to refine the problem by adding or fixing the example test cases in the docstring using the computed test inputs/ / /otputs, and then regenerate a solution. We then check the new solution on the test inputs produces the same outputs as the previous solution. The intuition is that since the refined problem should only include minimal changes, the solution output should then remain the same in the absence of any inconsistencies. As such, if we observe
3

---

Published as a conference paper at COLM 2024
differences between the two solution outputs, we ask the LLM to further revise and fix any inconsistencies and repeat the process, If both solutions agree on outputs, we return the new problem description, solution, and test cases for functional evaluation.
Manual examination test augmentation. For each transformed problem, carefully ex- amine and adjust any final faults to ensure each problem and ground truth are correctly spec- ified and implemented. We further generate additional tests using an LLM-based test aug- mentation technique (Liu et al. 2023). Finally, we produce EvOEVAL, a comprehensive code synthesis benchmark suite containing diverse problems to evaluate LLM coding capability across various domains. Details like transformation prompts are presented in Appendix D
EvOEvAL Benchmarks & Evaluation Methodology
EvOEVAL uses HUMANEVAL problems 828 problems across 7 different benchmarks (5 semantic-altering and 2 semaanic-prssevvng)) For the semantic-altering benchmarks, we generate 100 problems each using different seed problems from HUMANEVAL. For the semantic prreantiipreeerving benchmarks, we transform all 164 problems in HUMANEVAL as we reuse the original ground truths, requiring less validation.
probless as seeds and GPT-4 as the foundation LM to produce
DIFFICULT: Increase complexity by adding constraints, replacing commonly used re- quirements to less common ones, or introducing additional steps to the original problem.
CReATIVE: Produce a more creative problem using stories or narratives.
SUBTLE: Make subtle change such as inverting or replacing requirement.
COMBINE: Combine two problems by using concepts from both problems.
TOOL.USE: Produce main problem and helper functions. Each helper function is fully implemented and provides hints or useful functionality for solving the main problem.
VERBOSE: Reword the original docstring to be more verbose with descriptive language
CONCISE: Reword the original docstring to be more concise using concise language.
Evaluation setup: Each LLM generated sample is executed against the test cases and evaluated using differential testing (McKeeman) 1998) - comparing against the ground truth results to measure functional correctness. We focus on greedy decoding and denote this as pass@1.
We evaluate 57 LLMs (Appendix C), including both proprietary and open-source models. Further, we classify the LLMs as either base or instruction-follooing and discuss the effect of model variants.
Models:
To produce the code solution using each LLM, we provide specific input prompt: For base LLMs, we let the LLM autocomplete the solution given the function header and docstring. For instruction-followinn LLMs, we use the recommended instruction and ask the LLM to generate a complete solution for the problem.
Input format:
4 Results
4.1 LLM Synthesis & Evaluation on EVOEVAL
EvoEVAL produces more complex and challenging benchmarks for program synthesis. Table [ shows the pass@1 performance along with the ranking of LLMs on each of the semantic-altering EvoEVAL benchmarks with the average pass@1 and ranking all benchmarks in the last columns First, compared the success rate on HUMANEVAL, when evaluated on EvOEVAL, all LLMs consistently perform worse. For example, the tate-of-the-are GPT-4o, GPT-4 and Claude-3.5 models solve close to 85% of all HUMANEVAL problems but fall almost below 55% pass@1 when evaluated on the DIFFICULT problems. On average, across all benchmarks, the performance of LLMs decreased by 38.1% (DIFFICULI: 56.6%, CREATIVE: 48.2%, SUBTLE: 5.0%, COMBINE: 74.7%, and TOOL.USE: 6.1%). Additionally, this drop is not uniform across all LLMs and can range from 19.6% to 47.7%.
We evaluated all 57 LLMs, however, omitted some LLMss in Table for space reasons.

---

Published as a conference paper at COLM 2024
Table 1: pass@1 and ranking results (* indicates tie) on the semantically-alternng EVOEVAL and HUMANEVAL benchmarks (including HUMANBVAL+ scores in the parenthesis). denotes instruction-folllwii LLMs.
HUMANEVAL
DIFRCULLT #CREATIVE
WSUBTLE
CCMMBinE WTooL..Ust WEvoEvAL
Size
pass81 rank passi) rank passo1 rank passö) rank passö) rank passe1 rank passoi rank
GPT-4o8
NA 86.0 (61.7)
51.0
64.0
2
80.0
4
51.0
-3
72.0
1
67.3
2
Gurbo-
NA 8A.5 (80.5)
*2
50.0
"6
61.0
3
82.0
45.0
69.0
65.1
4
GPT-49
NA (76.2)
*6
52.0
-2
66.0
76.0
6
53.0
68.0
66.2
GPT-3.5-Tubbl
76.8 (69.5)
"9
33.0
-19
42.0
*13
70.0
33.0
64.0
*10
53.1
12
Claude-3.59
NA 83.5 (78.0)
*2
56.0
60.0
4
82.0
'1
37.0
68.0
"6
67.8
Claude-39
NA 82.3 (25.0)
"6
50.0
'6
53.0
7
81.0
3
42.0
7
69.0
P3
62.9
7
Claude-3-haiku
NA 74.4 (66.5)
*13
40.0
47.0
*11
65.0
'16
25.0
'12
61.0
'16
52.1
13
Claude-29
NA 66.5 (62.2)
*24
29.0
23
'13
64.0
'19
19.0
20
57.0
122
46.2
21
Gemini-1 1.5-ro
NA 83.5 (76.8)
*2
52.0
12
55.0
55
78.0
5
43.0
69.0 00
63.4
Gemini- -.0-ro
NA 62.2 (56.7)
27
37.0
116
40.0
18
53.0
'27
23.0
$15
57.0
'22
45.4
23
PaLM-22
NA 40.2 (36.6)
44
18.0
*38
22.0
39
36.0
'48
3.0
+45
46.0
135
27.5
43
DS Coder-v2-Insti 236b 82.9 (78.7)
5
52.0
22
55.0
55
75.0
7
51.0
13
70.0
2
64.3
5
33b 78.0 (73.2)
8
47.0
47.0
*11
67.0
*11
31.0
'10
66.0
8
56.0
8
DS Coder-Inst
6.7b 74.4 (69.5)
*13
40.0
*12
37.0
*19
61.0
'23
18.0
221
51.0
30
46.9
20
1.3b 63.4 (60.4)
*31
227
+47
34.9
30
33b 50.6 (42.7)
32
26.0
26
23.0
136
47.0
132
11.0
'31
63.0
*13
36.8
29
DS Coder
6.7b 45.1 (38.4)
*37
21.0
Y32
24.0
*33
47.0
*32
5.0
'41
55.0
225
32.9
35
1.3b 29.9 (26.2)
51
6.0
'54
19.0
*41
27.0
55
0.0
57
40.0
46
20.3
51
DS Coder-
7b 68.9 (63.4)
*21
37.0
116
37.0
'19
66.0
'14
24.0
14
60.0
*18
48.8
16
DS Coder-11
7b 42.1 (34.8)
*41
21.0
132
34.0
'23
43.0
137
4.0
143
54.0
+27
33.0
34
Llama 33
70b 75.0 (68.9)
*11
10
49.0
73.0
8
34.0
62.0
55.8
co Llama I3-Inst
70b 73.8 (71.3)
*15
41.0
11
52.0
70.0
31.0
110
64.0
*10
55.3
10
70b 66.5 (59.8)
*24
31.0
22
41.0
*16
65.0
*16
18.0
'2.1
65.0
9
47.7
18
oCooeelama-Inst
34b 51.8 (43.9)
22.0
130
27.0
29
43.0
'37
134
47 0
133
33.3
33
13b 48.8 (42.7)
35
21.0
+32
25.0
*31
46.0
35
8.0
'37
54.0
227
33.8
32
7b 43.3 (39.0)
+43
*43
44.0
+39
27.9
70b 60.4 (52.4)
29
25.0
27
29.0
*26
49.0
$29
14.0
'27
63.0
*13
40.1
28
coCodellama
34b 52.4 (43.3)
30
15.0
43
24.0
*33
47.0
132
11.0
*31
44.0
139
32.2
36
13b 42.7 (36.6)
40
18.0
*38
24.0
*33
38.0
+45
6.0
40
48.0
'31
29.4
39
7b (36.6)
44.0
139
25.6
44
WizardCoder
34b 61.6 (54.3)
28
24.0
28
32.0
55.0
26
17.0
'24
55.0
+25
40.8
26
WizandCoder-1.11
33b 73.8 (69.5)
*15
48.0
8
48.0
10
66.0
'14
20.0
19
64.0
*10
53.3
11
XwinCoderi
34b 68.9 (62.2)
*21
33.0
'19
42.0
*13
67.0
*11
15.0
26
60.0
*18
47.7
19
34b 70.7 (66.5)
19
22.0
130
35.0
22
63.0
21
25.0
*12
58.0
21
45.6
22
Code Millenials
34b (69.5)
35.0
18
*16
65.0
*16
17.0
<24
56.0
24
47.9
17
Speechless-CL.
34b 75.0 (69.5)
*11
38.0
15
37.0
$19
64.0
$19
23.0
115
59.0
20
49.3
15
Magicoder-s:125
6.7b 76.8 (70.7)
"9
40.0
*12
34.0
'2
67.0
*11
21.0
*17
61.0
*16
50.0
14
Msgicoder--CLL
7b 70.1 (65.9)
20
27.0
25
26.0
30
58.0
25
11.0
*31
52.0
29
40.7
27
15b 45.1 (36.0)
*37
16.0
141
19.0
*41
41.0
'41
5.0
14
48.0
131
29.0
41
StarCuder2
7b 34.8 (31.1)
*66
12.0
+45
17.0
45
38.0
+45
2.0
*51
46.0
135
25.0
46
3b 31.1 (26.2)
50
8.0
'51
14.0
*48
31.0
150
2.0
151
35.0
51
20.2
52
StarCoder
15b 34.8 (30.5)
*46
44.0
+39
23.5
47
8x7b 42.1 (38.4)
*41
21.0
132
18.0
'43
41.0
141
9.0
'34
45.0
137
29.3
40
OpenChat
7b 71.3 (66.5)
18
33.0
'19
29.0
'26
62.0
22
14.0
'27
43.0
44
42.1
24
Gemma-Ines
7b 28.0 (23.2)
*54
6.0
'54
10.0
*54
29.0
54
2.0
151
31.0
53
17.7
54
31.7 (25.0)
49
12.0
*45
13.0
150
40.0
'43
2.0
151
39.0
+47
23.0
48
Gemmaa
2b 22.0 (17.1)
57
2.0
57
6.0
57
24.0
57
2.0
151
21.0
56
12.8
57
Phi-2
2.7b 50.0 (45.1)
*33
18.0
138
23.0
*36
49.0
$29
14.0
'27
37.0
30
31.8
38
72b 67 (61.6)
23
28.0
24
28.0
28
61.0
223
21.0
'17
47.5
133
42.0
25
Qwen-1
14b 50.0 (45.7)
*33
20.0
*36
23.0
*36
48.0
31
18.0
221
44.0
+39
33.8
31
7b 42.12216 (37.8)
*41
16.0
141
13.0
150
43.0
+37
7.0
39
32.0
25.5
45
5

---

Published as a conference paper at COLM 2024
1.7
6-18-
wid
derss Arats
of
1.3
4.9%
Deffiest: parso
Creasive paaa0l
Cono2te 233902
Teol 2ex pasag:
Figure 2: HUMANEVAL+ EVOEVAL pass@1. Red identity line shows equivalent perfor- mance. We cluster the LLMs into: purple region aligned performance on HUMANEVAL vs. EVOEVAL and blue region over performance on HUMANEVAL vs. EVOEVAL.
LLMs struggle on EvOEVAL benchmarks compare to the high performance achieved on HUMANEVAL. One surprising finding comes from SUBTLF, where the average performance of LLMs drops by 22.5% on the same 100 problems even though only small changes are made to the original problems and the difficulty remains roughly the same. Appendix Fig- ure 21 presents an example problem and failing solution. Furthermore, we can also identify LLMs which struggle heavily on specific types of problems compared to their relative perfor- EVOEVAL scores. As we saw before, the significant portions of the models tend to be worse on EvOEVAL than HUMANEVAL (i.e., purple shaded region). However, there are LLMs that have a much higher HUMANEVAL score compared to their performance on EVOEVAL (i.e., blue shaded region). This implies potential data leakage of popular benchmarks where LLM performances are ar- tificially inflated but do not translate to more difficult or other program synthesis problems.
mance on HUMANEVAL Figure Showws scatter plot of HUMANEVAL+
shows
Significant ranking changes of LLMs on EVOEVAL. Compared to HUMANEVAL where top models all perform similarly, we observe drastic differences in ranking changes on EVOEVAL. We observe that while the relative difference between the top 10 models HUMANEVAL is around 10,, the difference on EvOEVAL on average is over 20%. Due to such saturation, existing benchmarks may not reliably rank the program synthesis ability of each model. For example, while Claude-3.5 and GPT-4-Turbo are tied for second on HUMANEVAL, they both excel at different types of problems: Claude-3.5 performs best difficult and combine problems, while GPT-4-Turbo is better with tool use and creative tasks. Furthermore, while GPT-4o achieves the top HUMANEVAL and HUMANEVAL+ score, it falls off compared to the base GPT-4 variant where it is worse on DIFFICULT, CREATIVE and COMBINE problems. Such evaluation cannot be gained through naively reporting existing coding benchmark performance. Overall, by evolving the original benchmark into more difficult and diverse problems of different types, EVOEVAL can provide a more holistic evaluation and ranking of the coding ability of LLMs.
EvOEVAL can be used to comprehensively compare multiple mod- els. In Figure 3, while both WizardCoder-1.11 and Phind-CodeLlamme 2 have similar HUMANEVAL scores, they perform drastically dif- ferently across EVOEVAL benchmarks. WizardCoder-1.11 is better on DIFFICuLT and CREATIVE while Phind-CodeLlama-2 is better on COMBINE problems. This can be explained through the train- ing dataset used in each LLM: WizardCoder-1.11 uses an evolv- ing dataset by generating more complex problems whereas Phind- CodeLlama-22 is fine-tuned on high quality programming problems that seems to boost the ability to solve programs which combines multiple programming concepts. Different from just reporting a singular pass@k score, EVOEVAL also allows detailed analysis
Figure 3: Radar graph
Note that SUBTLE only contains 100 problems, and the pass@1 score on these 100 seed HUMANEVAL problems is higher compared to the full 164 problems. Therefore, this back-to-back performance drop is much higher than the performance drop from full HUMANEVAL SUBTLE (5.0%) mentioned above.

---

Published as a conference paper at COLM 2024
Figure 4: HUMANEVAL pass@1 with relative decrease or increase on VERBOSE and CONCISE.
Table 2: Results COMBINE and COMBINE-NAIVE. HUMANEVAL is categorized into pass both, one and none, depending on the success on the two parent problems used for combination. COMBINE (Solved) and COMBINE-NAIVE (Solved) then show the distribution of solved problems that came from the previous categories. Composition Percentage is the % of pass both problems the LLM can still solve when combined.
HUMANEVAL.
COMBINE (Solved)
Composition Percentage
Size
pass both pass pass
pass both pass pass
@ GPT-4o
NA
80
20
61.2%
* GPT-I-Turbul
NA
79
19
2
38
6
1
48.1%
GPTT-199
93
@ GPT-3.5-Turbo
NA
65
34
24
9
0
36.9%
Claude-3.5
NA
81
18
8
60.5%
Claude-3
NA
81
19
35
7
43.2%
Gemini-1.55 p.
NA
86
13
3
46.5%
DS CSdee--v2-Inde
236b
83
56.6%
HUMANEVAL
COMBINE-NAIVE (Solved)
+ GPT-4o
NA
881
185
589
50
66.9%
GITT-4Turbuul
NA
863
195
16
407
61
47.2%
* GPT-443
NA
1018
55
768
7
75.4%
GPT-3.5-Turbbll
NA
799
Claude-3.5
NA
861
203
10
710
93
82.5%
Claude-33
NA
796
268
96
45.1%
Gemini-1.55pron
NA
798
267
19
595
148
75.5%
er Co
236b
805
252
598
95
74.3%
across different dimensions of coding capability to identify particular domains or types of coding questions an LLM struggles with or excels in.
Instruction-following LLMs are sensitive to subtle changes or rephrasing in problem docstrings. Figure shows the HUMANEVAL score (bar) and the relative performance drop or improvement (arrows) on VERBOSE and CONCISE. We observe that almost all instruction- following LLMs drop in performance (average 3.4% and 4.0% decrease on VERBOSE and CONCISE respectively) when evaluated on the two semantic-preserving dataset compared to the original HUMANEVAL. This is drastically different from the base variants, where we even observe performance improvements (average 0.5% and 2.1% increase on VERBOSE and CONCISE respectively). VERBOSE and CONCISE do not change the semantic meaning of the original problem; they simply reword in either a more verbose or concise manner. Prior work Deng et al. (2023a) has shown that by rephrasing the original problem description, one can further boost LLM performance, and we observe the similar phenomenon here mostly only for con-instructioonfollrre
4
models. Additionally, even SUBTLE, where only small changes are applied, on average, instruction-folloving L.MSs drops by 7.4% whereas base models only decrease by less than 1%. These findings across LLM types show that while instruction-turee LLMs are expected to align better with detailed instructions, they fail to distinguish between these rephrasing or subtle changes in docstring, indicating potential memorization or contamination of prior evaluation benchmarks.
4.2 Problem Composition & Decomposition
Composing problems. The ability to compose different known concepts to solve new problems is known compositional gemeralization (Keysers et al.) 2020). This skill is essential for code synthesis, especially for complex problems in real-world programs. However, measuring compositional generaiizttion in L.M presents fundamental challenge since
7

---

Published as a conference paper at COLM 2024
Table 3: Results on DECOMPOSE. HUMANEVAL shows the pass/fail breakdown of the 50 seed HUMANEVAL problems. DECOMPOSE categorized into pass both, one and none, based on if the LLM can solve both subproblems. Decomp. % and Recomp. % are the % of originally passing and failing problems for which the LLM can solve both subproblems respectively.
Size
HUMANEVAL
DECOMPOSE
Decump.
Recomp.
HUMANEVAL pass
HUMANBVAL fail
%
%
passs
pass both
pass one
pass none pass buth pass one
pass nune
GPT-4o
NA
26
14
0
63.4%
55.6%
GPT-o-Turb
NA
39
29
9
1
74.4%
36.4%
GPT-49
NA
47
37
10
0
78.7%
0.0%
NA
13
64.7%
Claude-3.5
NA
38
12
25
9
0
63.8%
25.0%
Claude-3m
NA
26
11
66.7%
54.5%
Gemini-1.5-prof
NA
41
27
13
1
65.9%
55.6%
y DD Coder-v2-Inst
236b
31
7
81.6%
50.0%
requires controlling the relationship between training and test distributions (Shi et al. 2024). While it is not easy to control the pre-training data of LLMs, have more control in the testing phase. Hence, we focus on program concepts that have been demonstrated to fall within the capabilities of an LLM, and explore whether this proficiency extends to the combination of program concepts. As such, we start by taking deeper look at the COMBINE problems evolved from combining previous HUMANEVAL problems.
First half of Table shows the COMBINE dataset results of the top LLMs. We observe that almost all problems solved came from the pass both category, which is intuitive as do not expect LLMs to solve problem composed of subproblems that it cannot already solve. However, the composition percentage quite low, as only a few LLMs are able achieve greater than half. This demonstrates that while state-of-the-ar LLMs can achieve a high pass rate on simple programming tasks, they still struggle with composing these known concepts to address more complex problems.
Composing problems naively. Since COMBINE prob- lems are not guaranteed to contain no additional new concepts, build simplified dataset for sequential composition. Let A and B be two separate problems with input(s) for A, aim to create new prob- lem C with the same inputs where the solution can be written (x)). To accomplish this, the new prob- lem combines docstrings for A and В sequentially. However, simple concatenation of docstringsleads to
add(k: int, int):
aad two numbers and y*.
Froblan
def digits(n):
"Siven a positive integer n, return the product of the add digits.
Return B all digits are even,
Problon
dof add digits(x: int, y: int): unnarrst, add two nuebers and
unclear descriptions. As such, for each problem in HUMANEVAL, we manually create two separate vari- ants based on which order the problem may come in the new docstring. how naive combination problem is constructed with the manual sequential instruction highlighted in red. Using these modified problem doestrings, we build sequential combination dataset COMBINE-NAIVE, containing 1074 problems by randomly combining problems filtering for input output matching (i.e., the type of (xx) should equal to the type of y in Byy)).
Next, given the resulting positive integer
return the product of the odd digits. Return alt digits
*..
an example of
Problen t
Figure 5: COMBINE-NAIVE problem
The latter half of Table ssoows the results on COMBINE-NAIVE following the same setup as COMBINE. We observe that while the composition percentage on the naive dataset improves significantly compared to the evolved COMBINE dataset, it still fails to reach near perfection, with the best LLM being able only solve 80% of prior pass both problems. While existing LLM training or inference paradigms for code focus on obtaining high quality datasets boosted with instruction-tuiing our result shows that existing LLMs still struggle with the concept of problem composition to tackle more complex problems.
Decomposing problems. We also evaluate problem decomposition decomposing p larger problems into multiple subproblems. We start by selecting 50 HUMANEvAL problems and then follow our approach in Section 2 to decompose each original problem into two smaller subproblems, creating 100 problems in our DECOMPOSE benchmark. Table shows the
8

---

Published as a conference paper at COLM 2024
0.6
Instruct
Base
passgl
0.4
0.2
95-33b
05-76-91.5
05-6.7b
BS -b.3b
CL-78b
CL-30b
CL-13b
CL-7b
gemss-18 -b
nistral-26
Figure 6: pass@1 from TOOL.USE-MAIN.ON (darker bar) to TOOL_USE (lighter bar).
results of selected LLMs DECOMPOSE, We first observe that similar to the composition percentage in the COMBINE and COMBINE-NAIVE problems, LLMs do not achieve a high decomposition percentage. Since current LLMs are trained to recover seen outputs in their training data, and when used for program synthesis, they cannot generalize the concepts from training data. This is demonstrated by not being able solve smaller subproblems obtained from solved more difficult parent problems. Conversely, LLMs can sometimes solve both subproblems even when the parent problem not solved (i.e., recomposition percentage), showing room for improvement with techniques like planning (Jiang et al. 2023b and least-to-most prompting (Zhou et al. 2022).
4.3 Tool Use Problems
We analyze the USE benchmark, which contains helper functions. We further con- struct the TOoL. USE-MAIN ONLY benchmark, which contains the same set of problem as TOOL_USE, except that the input to the LLM does not include any helpers. We observe that compared to without any helper functions (average 29.8%), LLMs on average improve by 80.1% when provided with helper functions, This is expected as helper functions reduce the work required to solve the more complex problem. However, this improvement not uni- form: the average improvement when given the auxiliary functions for instruction-following models is only 59.2% compared the base LLMs' improvement of 122.0%.
In Figure 6 we observe that without the helpers, the instruction-followin models signif- icantly outperform their base LLMs. However, once the helpers are provided, this gap is drastically decreased, with cases even where the base models outperform their instruction- following counterparts. As real-world coding involves understanding, using, and then reusing existing functions across different places in the repository, being able to success- fully leverage auxiliary methods is key. Current instruction-folloiin fine-tuned with data consisting of self-contained code snippets without the interaction and learning of function usages. This is further exacerbated by prior benchmarks, which mostly use self-contained functions, thus cannot test the tool-using capability of LLMs.
LLMs are generally
Related Work
Large language models for code. Starting with the general development LLMs for general purpose tasks, developers have applied LLMs to perform code-related tasks by further training LLMs using code snippets from open-source repositories. Such LLMs include
CODEX (Chen et al.) 2021), CodeT5 (Wang et al., 2021), CodeGen (Nijkamp et al. 2023), InCoder (Fried et al. 2023), CodeLlama (Rozière et al. 2023), StarCoder (Li et al., 2023
Lozhkov al. 2024), DeepSeek Coder (Guo et al. 2024), etc. More recently, researchers have applied instruction tononnntttnnggnoo methods to train code-specific LLMs that are well-versed in following instructions. Examples of such LLMs include CodeLlama-Ine (Rozière al. 2023 and DeepSeek Coder-Instruie (Guo et al., 2024). WizardCoder (Luo et al., 2023) instruction- tunes the model using Evol-Instruct to create more complex instructions. Magicoder (Wei) et al. 2023) develops OSS-Instruct by synthesizing high quality instruction data from open- source code snippets. Ipennodeenterrpeeeer (Zheng al. 2024) leverages execution feedback for instruction-tuing in order to better support multi-turn code generation and refinement.
Program synthesis benchmarking. HUMANEVAL (Chen et al. 2022 and MBPP (Austin et al., 2021 are two of the most widely-used handcrafted code generation benchmarks complete with test cases. Building these popular benchmarks, additional variants have been

---

Published as a conference paper at COLM 2024
crafted including: EVALPLUS (Liu et al. 2023) which improves the two benchmarks with more complete test cases; HUMANEVAL-> (Zheng et al.
2023 which extends HUMANEVAL
to C++, JavaScript and Go; MultiPL-E (assano et al. 2023 which further extends both HUMANEVAL. and and MBPP to 18 languages. Similarly, other benchmarks have been developed for specific domains: DS-1000 (Lai et al., (023) and Arcade (Yin et al. 2022) for data science APIs; CodeContests (Li et al. 2022), APPS [Hendrycks et al., 2021), and LiveCodeBench (Jain al. 2024 for programming contests, and SWE-Bench (Jimenez al. 2024) for software engineering tasks. Different from prior benchmarks which require handcraft problems from scratch-high manual effort or scrape open-source repositories or coding contest websites leading to unavoidable data leakage, EvOEVAL directly uses LLMs to evolve existing benchmark problems to create new complex evaluation problems. Furthermore, contrasting with the narrow scope of prior benchmarks (often focusing on a single type or problem, Le., coding contests), EvOEVAL utilizes targeted transformation to evolve problems into different domains, allowing for more holistic evaluation of program synthesis using LLMs.
Conclusion
We present EvOÉVAL- a set of program synthesis benchmarks created by evolving existing problems into different target domains for a holistic and comprehensive evaluation of LLM program synthesis ability. Our results on 57 LLMs show drastic drops in performance (average 38.1%) when evaluated on EvOEVAL. Additionally, we observe significant ranking differences compared to prior leaderboards, indicating potential dataset overfitting on existing benchmarks. We provide additional insights, including the brittleness of instruction- following LLMs as well as the limited compositional generalization abilities of LLMs.
7 Acknowledgment
We thank Owen Colegrove for his help on starting this project and providing valuable feedback throughout, Jiawei Liu for providing helpful discussions and Yifeng Ding for his help in running experiments. This work was partially supported by NSF grant CCF-2131943 and Kwai Inc. This project is supported, in part, by funding from Two Sigma Investments, LP Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Two Sigma Investments, LP.
References
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arXiv preprint ar arxi:22305.11 104003 2023.
Anthropic. Introducing claude 2.1. 2023.
Anthropic. Introducing the next generation of claude. -laude-3-famill 2024a.
Anthropic. Introducing claude 3.5 sonnet.
https://ww..nntrooic.cco///
claude 1e1eia-35555 10-5-tsonnt///om 2024b.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et Qwen technical report. arxiv preprint arXiv:2309.16669, 2023a.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Xiaodong Deng Kai Dang, Yang Fan, Wenbin Ge, Fei Huang, Binyuan Hui, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Tianyu Liu,
10

---

Published as a conference paper at COLM 2024
Keming Lu, Jianxin Ma, Rui Men, Na Ni, Xingzhang Ren, Xuancheng Ren, Zhou San, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Jin Xu, An Yang, Jian Yang, Kexin Yang, Shusheng Yang, Yang Yao, Jianwei Zhang Bowen Yu, Yichang Zhang, Zhenru Zhang, Bo Zheng, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Introducing qwen1.5. httll:.//tttt:///t htttps///wwenll. tanlll..gtthul github httblog//wwen.../ooooo//o// 5/ 2023b.
BudEcosystem. Code millenials 34b.
URL [nttp:://hugingffaee co/ (https: hwwh/hggggggg ho...eecoccbbdeoossstttttte
http:/ssstttemmoooemmmlleeaaaaaa code-millemials-34b))
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arv preprint arXiv:2107.00333 003774, 2021.
Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. Rephrase and respond: Let large language models ask better questions for themselves, 2023a.
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In 32nd International Symposium on Software Testing and Analysis (ISSTA), 2023b.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arviv preprint X20::2207:21783, 2024.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL/ http:/reww....ettfoooooreaaaaaaa
Google. Our next-generation model: Gemini 1.5. https://blog. 2024.
httpg//sss////ooooooooooooaaaaar goggle/techmooogg//alll
Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. Program synthesis. Foundations and Trends in Programming Languages, 4(1-2):1-119 2017. ISSN 2325-1107. doi: 10.1561/ 2500000010. URL http:://xx/.o.iiit doi org/ 10.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming rise of code intelligence. ar preprint axX:::2401.14196 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurlPS, 2021.
Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural information processing systems, 15, 2002.
HuggingFace. Hugging face, 2022.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. ar arviv preprint, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ar Xiv preprint ar arXiv:2310.06825, 2023a.
11

---

Published as a conference paper at COLM 2024
Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. Self-planning code generation with large language model. arxio preprint urXio:2303.06683 2023b.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations, 2020. URL https::/
net/forumm SygodddNKKyKrrrrr
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319-18345. PMLR, 2023.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you!, 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 2022. URL https://www http://wwwwsscen...... www.sce.eecccoo......o.oo wwwpsiii......oooooooooooooooooc abs 10. 1126/ science. abq1158
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code gen- eration. In Thirty--eventth Conference on Neural Information Processing Systems, 2023. URL https://openreevex. hwtp:ewwwwwwwwneeeeeeeeeeeeeeeee htt://orummiidde wtw.mmmddd1g1xx00000
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dinitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack V2: The next generation, 2024.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint ar Xix::2330 00066008888...8 2023.
William M McKeeman. Differential testing for software. Digital Technical Journal, 10(1):
100-107, 1998.
Meta. Introducing Ilama 3.1: Our most capable models to date. https://ai meta. com/ blog/meta-11as 1aapttttallaaa------ 3/ 2024.
Microsoft Research.
Phi-2: The surprising power of small lan- https:
guage models.
com/en wm-us/researcc/bllo. 2023.
-1-22the-- htt.:eeeeerpprsiiiioorrrrrrraraa
12

---

Published as a conference paper at COLM 2024
Mistral AI team. Mixtral of experts high quality sparse mixture-of-expers: https 2023.
//mistral.. att////ss/mitttaaaooooeeoooeeepe
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Represenlations, 2023.
URL httptttt////peerrrrreee
OpenAl. Chatgpt: Optimizing language models for dialogue. https://oponai.com/bun chatgpt/ 2022.
OpenAl. Gpt-4 technical report. ArXio, ab52000084 2023.
OpenAL. Hello gpt-4o. httpss//ppeeaa.ccoo////eee/lllls
2024.
phind team. Beating gpt-4 on humaneval with fine-tuned codellama-34b https://ww/
phind.com/blog.. bttpc///ooooooooeeaaaaaaaaaaaaaa gpt4 2023.
Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, and Nathan Cooper. Stable code 3b. URL
sttabe-code-33h
Martin Riddell, Ansong Ni, and Arman Cohan. Quantifying contamination in evaluating code generation capabilities of language models. arXiv preprint arXi::2403.04811, 2024.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. Xiu preprint or Xio:2308 2023.
Kensen Shi, Joey Hong, Yinlin Deng, Pengcheng Yin, Manzil Zaheer, and Charles Sutton. Exedec: Execution decomposition for compositional generalization in neural program synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrevier httpa//wwwwaaaaeeooooooooo//////
Jiangwen Su. Code millenials 34b.
URL
[https://hugginggaac. co/
muzggyy/ppeedilesssceeellmmmmmmp
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, (ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ar arXiv preprint ar XrX::22322.1185,, 2023..
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
Xwin-IM Team. Xwin-lm. https:://gtthubb.mm///tti//////
2023.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Open- chat: Advancing open-source language models with mixed-quality data. ar preprint arXio:2309.11235 2023.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware uni- fied pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696-8708, 2021.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang, Magicoder: Source code all you need. arviv preprint Xrx::22322.0210,, 2023.
Chunqiu Steven Xia and Lingming Zhang. Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. arXiv preprint arXio:2304.00385, 2023.
13

---

Published as a conference paper at COLM 2024
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, and Charles Sutton. Natural language to code generation in interactive data science notebooks. 2022.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.1756. 2023.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. Xrviv preprint ar wtx::24422.16666 2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint ar Xito:2205.16655, 2022.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2. Breaking the barrier of closed-soure models in code intelligence. arXiv preprint preprint Arii::2406.119900000 11931, 2024.
14

---

Published as a conference paper at COLM 2024
A Reproducibility Statement
We provide the detailed setup and steps to reproduce our results in the README file of our supplementary material. Furthermore, we have prepared a public repository, package, and dataset release to support open-source usage (these are not linked here to preserve anonymity). Here describe our computation setup and estimated runtime to reproduce the full experimental result of EvOEvAL.
Compute. We perform all our evaluation on an Ubuntu 22.04 linux machine with 96 cores (192 threads), 512GB RAM, 8 NVIDIA RTX A6000 GPUs with 50GB VRAM each. For LLMs that can only be accessible via endpoints, we use the provided commercial endpoints and do not require local GPU resources for those models.
Time. Due to the cost of running LLMs, to completely regenerate all 57 LLMs' solutions for all benchmarks we estimate it will take several days depending on available GPU resource. However, we provided the raw LLM samples generated by us in the supplementary material. To perform evaluation on the LLM samples and regenerate our reported results, we estimate it will take around 1 hour to complete.
B EVOEVAL Benchmarks
Table 4: EVOEVAL and HUMANEVAL benchmark statistics. Note:
the number in bracket shows the number of testcases in the augmented HUMANEVAL+ benchmarks, in EVOEVAL, they are directly reused in SUBTLE, VERBOSE and CONCISE due the similarity.
original
semantic-altering
semantic pomaatic-ressevvigg
HUMANEVAL
DIFACCLTT
CREATIVE
SUBTLE
COMBINE
TOOL_SE
VERBOSE
CONCISE
of problems
164
100
100
100
100
100
164
164
Avg. problem len.
450.6
749.4
982.1
40688
860.4
1224.6
450.
450.6
Avg. test cases
9.6 (764.1)
49,8
43.1
10.3
(745.4)
51.8
51.3
9.6 (764.))
9.6 (764.1)
creative
cuatte difficatt corbiee
(a) CREATIVE TOOL USE (b) DIFIUL,,
(c) VERBOSE & CONCISE Figure 7: 2 dimensional t-SNE visualization of EvOEVAL benchmarks.
Problems in EvOEVAL consist mainly of self-contained functions, except for TOOL USE that includes helper functions specifically designed to test the tool using capability of LLMs. Each problem uses a docstring to illustrate the problem specification, along with test cases and ground truth to evaluate the functional correctness. Table 4 shows the statistics of the benchmarks in EvOEVAL. In total, EvOEVAL includes 828 problems across 7 different datasets (5 semantic-altering and 2 semantic-preserrinng
Figure 7 shows the embedding visualization using t-SNE (Hinton & Roweis 2002) by projecting high-dimension representation of the problems docstrings in both EVOEVAL and HUMANEVAL into the 2D plane. We produce this embedding using seatt--medding--33 3-laage model from OpenAl with t-SNE perplexity=50 and iter=1000. First, we see that CREATIVE and TOOL_UsE drastically change the embedding distribution compared to the original dataset. The arrow in Figure 7 shows one example of the shift in distribution from the original problem to creative one. Next, we see that SUBTLE, DIFFICULT and COMBINE largely retain the same distribution as the original problems. This due to the high parity across these problem descriptions where SUBTLE only applies subtle changes and DIFFICULT adds additional complex constraints while keeping the main problem descriptions largely the same. Specifically, for COMBINE, we can see from example arrow in Figure 7b the
15

---

Published as a conference paper at COLM 2024
combined problem shifts the embedding for both of the original problems. Finally, we observe that for VERBOSE and CONCISE, the embeddings almost perfectly match the original problem, reflecting their semaatic-preserving nature. In Appendix E we present example problems for each benchmark in EVOEVAL.
B.1 Potential Bias in Performance
Using GPT-4 to transform the original HUMANEVAL, problems into new domains may intro- duce potential biases. For instance, LLMs trained on data generated by GPT-4 might unfairly benefit from our benchmark, their training data could be similar to the transformed problems. We note here that we manually examine all problems transformed by GPT-4 (including each ground truth solution) and adjust any faults incorrect solutions, such, the problem descriptions are not all purely generated by GPT-4. Moreover, our method of evolving existing problems to new domains is similar to existing instruction-tuning methods (e.g., Evo-Instruct (Luo et al. 2023), ) which may also favor these LLMs trained with similar evolved datasets. Another potential bias also comes from the usage of GPT-4, as it might create problems that reflect its own bias or patterns, resulting in a lack of diversity. In future work, we hope to apply additional state-of-the-art LLMs to perform our transformation to improve problem diversity.
C Evaluation LLMs
C.1 Evaluated LLMs
Table and 6 shows the overview of the 57 LLMs we evaluated in our work. For any LLMs that provide their open-source weights, we directly obtain them from HuggingFace model hub (HugginFaace For any close-sourced LLMs, we directly access their model endpoints using their providers.
C.2 Detailed Evaluation Setup
LLM generation. As mentioned Section g we report the pass@1 score for each LLM on our dataset generated using greedy decoding (i.e., sampling with temperature 0). For each LLM, we provide specific input prompt depending on the model type. For base LLMs (i.e., not instruction-folloving variants), we use only the function headers as input. For instructionffolloiing we make the best effort to follow examples provided by each model maker the exact instruction and format to use at the time of writing. Specifically, for instruction-following LLMs, we ask the model to return the code snippet wrapped by code CREATIVE problem.
blocks (i.e., "*) Figure 8 shows an example input for GPT-4
Furthermore, we also provide a custom sanitization script adopted from EVALPLUS LLu et al.) 2023) which parses the raw LLM outputs for code block parsing (e.g., removing indicators for instruction-folloiing models) and end-of-string identifiers (e.g., removing tokens like 
). Each model generated output is passed into the sanitization script and the evaluation occurs on the sanitized outputs.
Oracle. To evaluate the functional correctness of each LLM synthesized solution, we use differential testing by comparing the model output with the ground truth output on a set of test inputs. We build our evaluation framework on top of the EVALPLUS evaluation script used for HUMANEVAL and HUMANEVAL+ benchmark which evaluates multiple problems and solutions in parallel for efficiency. For each test case, we perform exact matching check if the output is within absolute difference threshold of 10 if the output is floating point type. We additionally implement our evaluation script by recursively checking the type and performing the appropriate comparison (e-g., dictionary outputs are first length checked for equivalence and then matching is done for each value and key). Furthermore, we also implement custom oracles for specific problems where there could be multiple solutions or
For certain LLMss we may use the V.LM inference library for more efficient generation
16

---

Published as a conference paper at COLM 2024
Table 5: Detailed overview of evaluated models. Model ID indicates either the API endpoint name or huggingface model name used the particular model. Available Weights indicate whether the model is evaluated by accessing close-sourced API endpoint or ran locally with provided weights. Note: denotes instruction-following LLMs
Size Model ID
Available Weights
GPPT-oo [pPp-A)))OppnAA 2024
8pt-o-2002400--3
@GPT-4-uubo OpenA
2025
gpt-40022-preview
&GPT- (OpenAl 2023
NA 8pt-4-0613
x
@GPT-35-Turbo OpenAl 2022
NA gpt-3.5-turbo-0225
X
Claude-3.5
Anthropic
2024h
NA chaade335xxomee222240000
Claude-3
Anthropic
2024a
clnade-3uuuux22222222
Claude-S-haiku Anthropic
2024a
NA caudee3-halluu2224400077
Claude-2
Anthropic 2023
NA claude-2.1
X
Gemini-1.5 1-ppii Google
2024
NA gemini-l.5-pro
Gemini
etll
2023
NA gemini-L.0-pru
Anilet PaLMM
2023
text-bison-001
Coder-v2- Inst
Zhu
2024
236b
DeepSeek-Cooder-2--0744
33h
code 33b-instruct
Coder-Inst
Guo al
2024
6.7b deepseek-ai/ codpseck -bbinnstuuct
1.3b deepseek-al/ -aaprreeeeeoddeee--------e
33b deepseek-ai httpaa/ddddeppeooooooon----baa
ar Coder etal 2024
6.7b deepseek-al/ seepaackka///epppseeeooddrrebbbb
1 1.3b deepseek-ai/ Bbrpre/oole/eeppcccooooooobbbbb
Coder-115-Inst.
Gto
2024
7b deepseek-al htapa//////eeeeeeeooooooo////e/r
Coder-1.5
Guo
2024
7b deepseek-ai http:ackka//apppaeeeccooo///eeas
09 Llama -3. 1-Inst Meta
2024
70b meta-llama netb-lanna/Meeellamm-33-000----0
00 Llaaa -3-1nst
Dubey al
2024
70b
Instructt
70b
codellama Dagllamaa/CudeLLlmmaa200 Baaaell-lam------sttttttttt
oa CodeLllama-Inst Rozière al.
2023
34h
codellama /aaee/Cooeeeeaaaaaaaaaaaaa CaaarCooeeuuum-------ntttthhhhhh
codellama lama 30w.amm------mmaaee--
7b
70b
codellama codellama/Codelllmeee Paaiaa--0---PPth------
34b
codellama Maa-eelllam------ttttaaeaae
oa Codellama Ruzière et
2023
13b cudellama/ LCudeLlama 13b---11------------
7h
codellama/ Beooll lama- 7b-Python-hf
WizardCoder al 2023
34b WizardLM/ Waardd/M//aaandCCooey-yo-------
WizardCoder Luncettl 2023
33b Wiradd.M/WiiandCdeerres
XwinCuder Team
Xwin:LM//XwinCClee:3BBB
Phind-CodeLlamm-2 phind team
2023
34b Phina//PhnndCooellaaaa-4422
Code Millenials
BudEctsystem
34b budeossstemmccddeellllinl----a
Speechless-C1.c
Su
34b uukuggy//peeellssseeddddmmm--222
Magicuder-s- DS
ae
2023
6.7b bee-iicc/Maggiosller/55555-7777
Magicoder-s-CL
ael
2023
7b lse-utue/Maglcodders-L////
15b biggode//starcoder2--15
StarCoder2 Lozhkov al
2024
7b bigcode//tarcoode22//
3b biguodl//tanccdder-Bbb
StarCoder Liet
2023
15b bigcode/starcoder
14 Iixtral-Inst
Mistral AT team
2023
8x7b mistralai Saaraaaii/iiittallee//tsssstttt
/
M Mistral-Insttvi12
jiagg
2023a
7b mistralai mistralai/Misttul
7
Mistral-Instt
fiangetal
2023a
mistalai/MMittal/-lnnsnnuttt
M Mistral
[ang et
2023
7b mistralal/MMistall7BBMa
OpenChati Wang et
2023
7b openchat/opemmchaa3555000
stable-code
ettaaparauuctal.
3b stabliityai/stabblccddd-bb
Gemma-Ing Inst
Teamet
2024
7b google/ gaggl///mmmm--b--ttt
7b google/ google/gemma-76
gemma-7b
Gemma Team et 2024
2b google/ gemma-2b
17

---

Published as a conference paper at COLM 2024
Table 6: Detailed overview of evaluated models continued. Model ID indicates either the API endpoint name or huggingface model name used for the particular model. Available Weights indicate whether the model is evaluated by accessing close-sourced API endpoint or ran locally with provided weights. Note: denotes instruction-following LLMs
Size ModelID
Available Weights
Phi-2 Microsofft Research
microsoft/phi:
Qwen/Qwenl Beten//weel.3-7210 3-72B-Chat
Qwen-1.5
at.
2023b
14b Qwen/Qwen
7b /eee///eelnl111 15-14--Chhh
14b Qwen/ Owen/Qwen-lkee Chhhhtt
Qwen Bai al. 2023a
Swen/Qwen-7B-Chal
Please complete the following code snippet.
def transform, caw.aaarrr_aaaascccccaaaaa str) str:
unn
You have an canvas containing either '#" (representing wall),
) representing
an empty apace), ?p (representing the point which painter starts). The
+ painter
can move horizontally on the canvas and paints all empty spaces he encounters with '.' without crossing or hitting the walls.
The task is keeping wall configuration and unaccessible spaces same. the canvas contains 4 no painter 'P",
updated canvas with all the accessible spaces painted,
return the canvas as it is. If there more than one 'p the number of painted space divides the empty spaces evenly, return IIvvaldd canvas'.
Examples:
>>> transform_canval ('p---a)
*Pa*** " #-----
>>> transform_ canvas ('-- #-----P--- -------
'Invalid canvas'
transform_ canffarrmccacaeeeCcc (>=---=-------
#*4P44t---**
>>> Saansform_canvan
#- P---P #p
Invalid canvas' unn
Figure Example input prompt for GPT-4
simple tolerance exact matching cannot fully guarantee correctness. Additionally, also use timeout as another evaluation method. Our setting again follows EVALPLUS default setup where the timeout per problem is defined as T max (aaxx x fgt) with default values of Tmex 1000ms, f= and fgt defined as the measured ground truth solution time to produce the correct output. All timeout related factors can be adjusted to account for variance on different underlying machine and hardware.
D Transformation Prompts
Here we provide the exact targeted transformation prompts used to evolve existing bench- mark problems. Figure 9 10
15 and show the prompt for DIFFICULI, CREATIVE, SUBTLE, COMBINE, TOOL USE, VERBOSE, CONCISE and DECOMPOSE respec- tively.
11
12
13
14
18

---

Published as a conference paper at COLM 2024
Here example coding problem:
(problem}
Please increase difficulty of the given coding problem
You can increase the difficulty using the following method:
Add new constraints and requirements to the original problem, adding
approximately 10 additional words. Replace a commonly used requirement in the programming task with a less common
4) more specific one.
Add more reasoning steps.
Return the new problem in the same format as the example problem (i.e., funcction
+ header docstring''! **)
Figure 9: Prompt for DIFFICULT
Here is an example coding problem:
{problem}
Please generate a more creative coding problem.
You should avoid common programming concepts and instead focus creating
+ problem that is interesting and fun to solve.
Return the new problem the same format
example problem (i.e., "fuunction
+ header docstring"
Figure 10: Prompt for CREATIVE
Please add subtle and simple change to the given problem.
You change the problem using, but not limited to, the following methods:
Add one new requirement to the original problem, such as "Return the list
+ ascending order", "Return the 1ist in ascending alphabetical order" and + Return unique elements only".
Invert requirement of the original problem; instance, reverse the instruction "from shortest to longest" to "from longest to shortest", reverse "maximum" to "minimum", or reverse "the first" to "the last".
Replace requirement with another similar but different one; for example, if the original problem requires the values to sorted, change it to keeping the original order.
Replace constants; for instance, replace with
Please only apply a minor change, ensuring that the new problem remains logical. + Return the new problem in the same format as the original problem (**
Below is the question:
(problem}
Figure 11: Prompt for SUBTLE
19

---

Published as a conference paper at COLM 2024
Here example problems:
Example problem 1:
(problem_1) 1}
Example problem 2:
(problem_2) 2}
Please create a new problem that combines problem 1 with problem 2 in a logical way.
+ The new problem should seamlessly integrate the concepts from the two previous examples into novel context, and require solution that exercises the understanding of the concepts from both problems. It does not need to take all the inputs from both problems.
An incorrect way would be for the problem to simply return the answers of the two problems separately. Another incorrect method would be pass all inputs from both problems but some them are not used to compute the output.
Return the problem in the format the example problems in the docstring concise, ensure problem remains simple clear.
Keep
Figure 12: Prompt for COMBINE
Here example coding problem:
{problem}
Please come up with new problem which uses helper functions to solve the problem.
The new problem should contain the following:
first: one more helper functions second: the main problem description consist the function header and docstring
The main problem description should not refer to the helper function(s) in any way.
The helper function(s) should implement simple parsing checking logic.
To solve the main problem, one should also use additional complex logic than just + calling the helper function(s).
Avoid problems simple math concepta such as prime, palindrome, anagrame, factorial
Avoid concepts like emails, string parsing- based problems
Please return the full implementation the helper function (s) the main problem description (not the implementation) in the same format as the example problem "....."*")
Figure 13: Prompt for TooL__SS
Figure and 18 show the refinement and I/O extraction fixing prompt used in EvOEVAL.
The refinement prompt is used to refine the original generated problem when inconsistency detected (see Section 2). The extraction prompt is used to initially obtain set of testcases from the problem docstring used for self-consistency evaluation. We further use an I/O fixing prompt (also in Figure (8) fix any examples in the docstring which do not contain the right output (as computed by the ground truth generated by GPT-4).
20

---

Published as a conference paper at COLM 2024
Below is coding problem
(problem)
Make the docstring more verbose and detailed but preserve the semantic meaning Ensure the function name, input argument names, and example input/output are the
() same
Return the transformed problem in the format the original problem (i.e., function header docstring)
Figure 14: Prompt for VERBOSE
Below is coding problem
(problem}
Make the docstring shorter and more concise but preserve the semantic meaning Ensure the function name, input argument names, and example input/output are the same
Return the transformed problem in the same format as the original problem (i.e.,
+ function header docstring)
Figure 15: Prompt for VERBOSE
Below is complex coding problem
{problem}
Please decompose the above into 2 smaller sub problems Return the two modified problems in the same format the initial problem (i.e., "ffunction header docstring"
Figure 16: Prompt for DECOMPOSE
Below is coding problem
{problem}
Ensure logical coherence the given problem. Improve the docstring's clarity and conciseness. Fix missing or helpful imports.
Include example input/output if absent.
Return the modified problem the same format the example problem (i.e., function header docstring"
Figure 17: Refinement prompt E Example Problems in EvoEvAL
Here we demonstrate a few example problems across the benchmarks in EVOEVAL and corresponding GPT-4 solution which cannot solve the problem. Figure 19 24. and show such example for the EvOEVAL DIFFICULT, CREATIVE, SUBTLE, C COMBINE, TOOL_ SE VERBOSE and CONCISE respectively.
20
21
22 23
21

---

Published as a conference paper at COLM 2024
Here function header with docatring:
(problem)
Please extract the example input argument and expected output from the docstring.
If there are no example input and output please provide new ones.
Return each pair of imput and output as assertions in this format:
assert (fumction_namm) (ffthe ((ftteefirat__nnnpee htt.rrrrtrtttptteeeaaa..... )) }}
{(the_ ftttee.fraat.outuueeeraaaa
assert (function_nanie (((the_see) seccheeeseenn__eeeeeeeeeee_eeeee (umput_exampee)) {{
+ uue_sscond_outputteeeee
Here is problem with docstring:
(problem}
Some example inputs and outputs in the docstring may be wrong. Please correct them according to provided correct assertions belou, and ensure that correct example inputs and outputa (assertions)
Return the revised problem in the format the original problem (i.e., any + helper functions function header docstring"
Figure 18: Input extraction and fixing prompts
def common (11: list, 12: list, int): nn.
Return the n longest strings, sorted by increasing length that are common in two
4 lists.
However, in the of tie, prioritize the string that appears first listi.
>>> common ( ["apple", "banana", "cherry", "date", "elderberry"], ["banana", 4 date", "fig", "grape", "elderberry"), 3)
["banana", "date" "elderberry"]
common(["cate "dog", "elephant", "fox", "goat"), ["dog", "fox", "cat", "
<) horse", "iguana", "goat"], 3)
["cat", "dog", "goat"]
>>> common ( ["hello", "goodbye", "yes", "no", "maybe", "absolutely", "never"], [" no" "maybe", "absolutely", "always", "sometimes"), 1) ["absolutely"]
common_ common_atrings= = [string for string in 11 if string in 12]
sort common_stao www.stttrnnsseeeeeee sort (key=len) # does not adjust for tie sorting scenario.
return common_stao caannonnstriinnaeeee [-n:]
Figure 19: GPT-4 failing solution problem EvoEvAL/58 in DIFFICULT
22

---

Published as a conference paper at COLM 2024
def wttp:oordddaliddaooooeeee list, words: list):
Consider grid representing square crossword puzzle. Each cell grid < contains single character.
Determine if all the words from provided list of words can be found in the grid. 7
Words can found left-to-right top-to-bottom down (in a column), and bottom-to-top (in a column).
row), right-to-leff (in a row)
We assume that words only found in contiguous cells and words do not 'wrap' around grid.
Note: the grid is square (N x N) and all words are lovercase.
Parameters:
grid: list of list char where each char lowercase letter.
words: list of str where each str is a lovercase word.
Returns:
boolean
True all words found the grid, False otherwise.
>>> crossword crossword_vallddaoorr valiidttor ta', 't), ['d,, 'o', 'g'J, Ddd, 'o', 'p']], [' + cat', 'dog', 'pop'l)
False
crossword validator(([P'', ta', Pt, Pa', 'o', 'g',, Pdd, 'o', 'p'1l, P 4 cdd', 'do'', 'pod'])
True
n len (grid)
for word in words:
found False
for in range(n):
# does correctly check if word be found sublist of column/ grid
if ", ('ioiiggrdd(()) word or " jo [::-1]) == word:
found True break
", (".jon(([grid])]] for j (n)]) word or " ".join([[rrid] [i]
4 for in range (::-]] word:
found True break
if not found:
return False
return True
Figure 20: GPT-4 failing solution on problem EvOEvAL/21 in CREATIVE
23

---

Published as a conference paper at COLM 2024
def nowels_counte
"""Write function vowels_count which takes a string representing word as input and returns the number of vowels in the string. Vowels in this
are 'a', 'e', '1', 'o', 'u'. Here, 'y' is also vouel, but only when is the end of the given word. Only count the lovercase vowels.
Example:
vowels_count( ("abcdee)
>>> vowels_count (wl.cttnnnnn ""YACED")
0
vouels 'aeiou'
count
for char in s:
if char in vowels:
count
miss empty string case leading crash if s[-1] 'y':
count += 1 return count
Figure 21: GPT-4 failing solution on problem EvoEVAL/64 in SUBTLE
24

---

Published as a conference paper at COLM 2024
sawe_ def grade with same_ chars (grades: (gge_ list, s0: str, s1: str):
A teacher has special grading system where she assigns her students letter
+ grades
based their GPA according the following table:
GPA
Letter grade
4.0
A+
> 3.7
A
3.3
A-
3.0
> 2.7
B
> 2.3
B-
2.0
C+
1.7
1.3
> 1.0
D+
> 0.7
D
0.0
D-
0.0
However, this year she adds an extra condition: the students can only receive a ④ grade
title (s1) have the characters. The if their (s0) and the
4 function
receives list of GPAs and two strings (s0 and s1) and returns list of grades for the students that pass the condition, an empty list if student passes.
Example:
grade_with_ writh_same_cheeeeeeeeeeeeeeeeeee chars ([4.0, 3, 1.7, 2, 3.5], 'abod', 'deba') =n> [ 'B', C 4 -', 'C', 'A-']
grade_with_same chars [[4.0, 3, 1.7, 3.5], 'aaabod', 'ddcbaa) ==> ['A+', B 'C-", 'C², 'A-']
grade_with__ae sacih__smm_hhhaae [[4.0, 3, 1.7, 2, 3.5], 'abcd', 'dcbae') ==>
# does not correctly check characters are same if ss0ted(s0) != sorted(s1):
[]
grading_eyee system (4.0: 'A+,, 3.7: 'A', 3.3: 'A-', 3.0: 'B+', 2.7: "B', 2.3: 'B-',
+ 2.0: 'C+,, 1.7: ' 1.3: '0-', 1.0: 'D+', 0.7: 'D', 0.0: "D-,, 0.0: 'B²}
grades_assigned []
for gpa grades:
for key in sortee(gradinggeesttee..k
reverse=True):
if gpa >- key:
leades__ssigned.appee brrgonddrrdinggystte.o (key])
break return grades_asaignet
Figure 22: GPT-4 failing solution on problem EvOEvAL/28 in COMBINE
25

---

Published as a conference paper at COLM 2024
def is_ w.w.aaldd...pppr.... (passpoort str) -> bool: (passport:
Checks whether given string is valid passport number.
A valid passport number is defined as having exactly 9 alphanumeric characters.
>>> is_ (e__vall..paspor((112355177
>>> is_wallid_passport (PP123455789)
False
>>>
(PP12345777)))
False
u..
if 1en (passpoot)) != 9: return False
if not passport. return False return True
from typing import List, Tuple
def poocess_paspportddttaaaoa.
str) -> List [sur, str]]:
Parse string containing passport data and return list of tuples.
Each tuple contains the paseport number and name of the passport holder.
The passport data string is in the format of "PassportNunber:be Name! separated by lines.
Only process valid passport data. valid passport number is defined having exactly alphanumeric characters.
>>> wwtpppppsssssssapppp11 [('P12345678', ' 'Jhnn Doe')]
Doe boe\nxx234567: Jane Doe')
perocess_passpon sart_ddtaa'PPP (111122455558 Doe n 1234567228 Jane Doe') [(PP12345678', John Doe')]
valid_passports= []
lines data.split((Mm.)
line in lines:
(passport, name) line shnesppli((:") # directly splits without checking for
4 validity
if is_ pas.ppoopppssott (passport):
halid_passports.appp append (passport, name))
return valid_passports
Figure 23: GPT-4 failing solution on problem EvOEvAL/23 in TOOL_USE
26

---

Published as a conference paper at COLM 2024
def starts_ _ne__onee___eee__e_ ends ends(n):
nnn
This function, given a user-provided positive integer 'n', calculates and
+ returns the total count of n-digit positive integers where appears either the start the end of the number.
For instance, the input is 2, the function will count all the two-digit
4 numbers that either start or end with 1, like 10, 11, 21, 31, 91, and < 12, 13, 19.
It is important to note that this function only accepts positive integers. a
+ non non-integer or non-positive integer is provided, the function may not behave as expected.
Args:
n int
A positive integer that representa the number of digits in the numbers to be
4 counted.
Returns:
The count of the numbers of n-digit positive integers that start or end with int
1.
Examples:
The function can be called in the following way: starts_one one (2)
This example call would return the count of two-digit numbers where appears + the start or at the end.
if == 1: return else:
# incorrectly adds 2 the final answer return 18 10 ** (n - 2)
Figure 24: GPT-4 failing solution on problem HUMANEVAL/83 in VERBOSE
def Aircular_shift@e shift):
"""Performs circular right shift on the digits of as per shift value and 4 returns the result in string format.
Returns reversed digits shift exceeds number of digits.
>>> circular_shift(12,, 1)
"21"
>>> dircular_shift(t (12, 2)
"12"
u..
x = str(x))
does not check shift exceeds string length and return reversed digits shift shift len(x)return x[-shift:] x[:-shift]
Figure 25: GPT-4 failing solution on problem HUMANEVAL/65 in CONCISE
27


Reviews:
[{"review_id": "ncvUsEsOJ9", "reviewer": "Reviewer_Y48c", "summary": "This paper presents EvoEval, an evaluation benchmark for code generation tasks. The method converts existing problems into different tasks, with 7 different types of modifications.\nThe EvoEval benchmarks are challenge and the performance of many LLMs has dropped using the benchmarks. The paper further points out some interesting trends, such as the instruction-following LLMs are very sensitive even to subtle changes.\nOverall, this paper identifies and attempts to solve a valuable problem: the contamination and potential overfit of LLMs on existing benchmarks. Theoretically, the method can be further used to create new benchmarks in the future.\n\n*Writing*\n\nDue to the relative complex setup and multiple types of “evol” methods, some explanations of the methods are a bit hard to follow. For example, the paper explains “COMBINE-NAÏVE” clearly, yet how “COMBINE”is done is a bit unclear.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful question and recognition that our benchmark suite can be valuable to the community!\n\n> Unclear “COMBINE” dataset\n\nFor our combined dataset, we first randomly select two HumanEval problems to combine together. In order to select problems that make sense to combine, we apply a simple heuristic\nto combine only problems of the same type together categorized based on the type of\ninput arguments in the original problem. Our prompt used for the combined dataset is in Figure 12 (in Appendix D). We will work towards clarifying the benchmark creation process in more detail in the updated version of the paper.\n\n>models that are trained on similar evolution methods as a different form of overfitting? \n\nThanks for this great question! We believe that models which are trained on evolving into more difficult problems like WizardCoder indeed contain a different form of overfitting. That is to say they overfit to the type of evolved problems they are trained on. For example, in our analysis, we found that while WizardCoder is better on more DIFFICULT problems, they perform worse on the COMBINE dataset since they are not trained to solve problems that combine multiple programming concepts.There actually motivated us to create a composite evolving dataset with different strategies for more comprehensive evaluation of code LLMs. We thank the reviewer again for this question and will add more discussion in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The benchmark presented by the paper is valuable to the community. Evaluation is one of the key components for advancement in AI.\n- The paper presents interesting finding of the current landscape of the models, by performing experiments 51 different variations of LLMs. These trends, such as the ranking change, variations related to subtle changes, point out some potential weakness of the language models.\n- The experiment is comprehensive and provide valuable insights", "reasons_to_reject": "No particular reasons to reject.", "questions_to_authors": "As mentioned in the text, models such as WizardCoder performs more robustly since it is trained on Evolved data. Would you consider models that are trained on similar evolution methods as a different form of overfitting?", "extra_scores": {}}, {"review_id": "CryT8ysOI3", "reviewer": "Reviewer_ihPF", "summary": "The current research piece introduces EvoEval, a new benchmark constructed by “evolving” (i.e. modifying) existing problems from HumanEval on different aspects, making them more difficult, more creative in their description, changing the problems in a subtle way, combining different problems into one or make the models to have (or produce) a helper function to solve the problem. In addition, two extra evolutions were presented, rephrasing problem descriptions, while preserving their semantics, by evolving in a more concise or in a more verbose way. \nProblems were created using HumanEval as a starting point and prompting GPT-4 for their modification. \nThe resulting benchmark contains 828 problems across 7 different benchmarks. The benchmark was evaluated on 51 open and proprietary LLMs with coding capabilities.\nFinding results interesting and new. While all the LLMs drop in performance, some of the best performing models on HumanEval drop significantly on EvoEval. Interestingly, some models have a significant drop in performance on subtle changes on their description. \nThe overall assessment of this work is good. This new benchmark provides new insights to the scientific community in regards to code synthesis while introducing a new way to assess the quality of such models.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments! Please kindly let us know in case you have any questions later in the discussion period.\n\n> Icons from Figure 5 are pixelated.\n\nThanks for catching this! We will ensure it is fixed for the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strong work.\n\nClear, and easy to read.\n\nA new benchmark bringing new insights to the scientific community.\n\nProper and extensive evaluation, always being compared with previous benchmarks, HumanEvan in this case.", "reasons_to_reject": "No reasons to reject", "questions_to_authors": "Icons from Figure 5 are pixelated", "extra_scores": {}}, {"review_id": "YFfAwOQNk2", "reviewer": "Reviewer_AGkd", "summary": "This paper introduces a program synthesis benchmark called EVOEVAL for a comprehensive evaluation of LLM coding abilities. It aims to address the reliability limitation of previous benchmarks due to their potential data leakage issues. The proposed benchmark is created by evolving existing benchmarks into different targeted domains. In evaluation, it comprehensively considers 51 different LLMs across all benchmarks in EVOEVAL and the experiment results provide insightful and interesting findings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer’s great comments! Please kindly let us know in case you have any questions later in the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper effectively tackles the shortcomings of prior code generation benchmarks by introducing a more comprehensive and robust benchmark to mitigate the potential data leakage problem. By doing so, it significantly improves the evaluation of LLMs' code generation capabilities across diverse aspects.  I would anticipate this benchmark will be widely adopted by the community.\n* The experiments conducted are thorough and offer insightful findings. \n* This paper is very well-presented.", "reasons_to_reject": "N/a", "questions_to_authors": "N/a", "extra_scores": {}}, {"review_id": "8Xrcps7cpt", "reviewer": "Reviewer_brBL", "summary": "The paper introduces EvoEval, a novel benchmark suite for evaluating the coding abilities of large language models (LLMs) that addresses the limitations of existing benchmarks by evolving them into more varied and targeted domains. Through empirical analysis of 51 LLMs, the authors demonstrate that LLMs perform significantly worse on EvoEval compared to traditional benchmarks like HumanEval, with an average performance drop of 39.4%. This decrease, ranging from 19.6% to 47.7%, suggests potential overfitting to current benchmarks and results in significant changes in LLM rankings. Additionally, the paper highlights the brittleness of LLMs to changes in problem wording and underscores the importance of problem composition skills in programming tasks. EvoEval not only broadens the scope of evaluation but also provides a framework for continually adapting benchmarks in response to advancements in LLM capabilities.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very detailed comments and suggestions on our work! \n\n> beneficial for the paper to clearly indicate which models have even been trained on GPT-4-generated data, perhaps using specific symbols or annotations.\n\nThanks for this great suggestion to further improve the paper! We will definitely add this to the final version along with the additional discussion of potential biases. \n\n> Typo: \"DeepSeeker\" -> \"DeepSeek-Coder\"\n\nThanks for catching this and we will fix it in the next revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a critical need by highlighting that existing code generation benchmarks, such as HumanEval and MBPP, suffer from limited scale and diversity and are susceptible to data contamination.\n\n2. The methodology is innovative; it creatively uses existing benchmarks and large language models (LLMs) to generate new, more diverse problems from those already in the benchmarks.\n\n3. Figure 2 offers key insights by comparing LLM performance on HumanEval+ and EvoEval, revealing that LLMs fine-tuned for instruction-following may perform well on HumanEval+ due to data leakage, yet underperform on EvoEval.\n\n4. The evaluation is thorough, assessing 51 LLMs across the new benchmark, providing a robust experimental section. The analysis of the results is detailed and insightful.\n\n5. The appendix of the paper is detailed, providing comprehensive supplementary material that supports the findings and methodology.", "reasons_to_reject": "1. The EvoEval benchmark leverages GPT-4 for problem creation, which is also used in instruct-finetuning datasets for some evaluated LLMs. This could lead to biased performance estimates for these models, and may also skew the results that some key findings in this paper is based on. It would be beneficial for the paper to clearly indicate which models have even been trained on GPT-4-generated data, perhaps using specific symbols or annotations.\n\n2. Typo: \"DeepSeeker\" -> \"DeepSeek-Coder\"", "questions_to_authors": "N/A", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces EvoEval, a novel benchmark designed to evaluate the coding abilities of large language models (LLMs) by addressing the limitations of existing benchmarks. EvoEval modifies existing problems from HumanEval across seven different dimensions to create more challenging and diverse tasks. The benchmark consists of 828 problems evaluated across 51 open and proprietary LLMs. The study reveals significant performance drops for LLMs on EvoEval compared to traditional benchmarks, highlighting issues of potential overfitting and sensitivity to subtle changes in problem descriptions. The results provide valuable insights into the robustness and reliability of current LLMs.

This paper addresses a critical need in the field of code generation by introducing EvoEval, a benchmark that mitigates the limitations of existing benchmarks such as HumanEval. EvoEval enhances evaluation by evolving existing problems into diverse and targeted domains, revealing significant performance drops and new insights into the brittleness of LLMs. The paper is well-written and presents a thorough experimental evaluation of 51 LLMs, providing a robust analysis of the results. While the methodology is innovative and the findings are valuable, the paper could benefit from addressing potential biases introduced by using GPT-4 for problem creation, especially given its role in training some of the evaluated LLMs. Despite this minor concern, the paper's contributions are substantial, and it is recommended for acceptance.
Decision: Accept