NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness
Authors: Manav Singhal
Tushar Aggarwal
Abhijeet Awasthi
Nagarajan Natarajan
Aditya Kanade
Venue: COLM (2024)
Keywords: Code LMs, Non-Functional Requirements, Code Comprehension, Evaluation
TL;DR: Comprehensive evaluation of code language models on real-world code editing scenarios and metrics beyond functional correctness.

Abstract: Existing evaluation benchmarks of language models of code (code LMs) focus almost exclusively on whether the LMs can generate functionally-correct code. In real-world software engineering, developers think beyond functional correctness. They have requirements on "how'' a functionality should be implemented to meet overall system design objectives like efficiency, security, and maintainability. They would also trust the code LMs more if the LMs demonstrate robust understanding of such requirements.

We propose a new benchmark NoFunEval to evaluate code LMs on non-functional requirements and simple classification instances for both functional and non-functional requirements. We propose a prompting method, Coding Concepts (CoCo), as a way for a developer to communicate the domain knowledge to the LMs. We conduct an extensive evaluation of twenty-two code LMs. Our finding is that they generally falter when tested on our benchmark, hinting at fundamental blindspots in their training setups. Surprisingly, even the classification accuracy on functional-correctness instances derived from the popular HumanEval benchmark is low, calling in question the depth of their comprehension and the source of their success in generating functionally-correct code in the first place. We release our benchmark and evaluation scripts publicly at https://aka.ms/NoFunEval.

Published as a conference paper at COLM 2024
NoFun Eval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness
Manav Singhal, Tushar Aggarwal,; Abhijeet Awasthi, Nagarajan Natarajan, Aditya Kanade
nanavsinghaal117@gma11 com, (t-tuaggarwal, abawasthi ( hagarajjj,aaaaaaeaa tag @emirrmootttccmmmm Microsoft Research India
Abstract
Existing evaluation benchmarks of language models of code (code LMs) focus almost exclusively on whether the LMs can generate functionally- correct code. In real-world software engineering, developers think beyond functional correctness. They have requirements on "how" a functionality should be implemented to meet overall system design objectives like effi- ciency, security, and maintainability. They would also trust the code LMs more if the LMs demonstrate robust understanding of such requirements.
We propose a new benchmark NoFunEval to evaluate code LMs on non- functional requirements and simple classification instances for both func- tional and non-functional requirements. We propose a prompting method, Coding Concepts (CoCo), as way for developer to communicate the do- main knowledge to the LMs. We conduct an extensive evaluation of 27 code LMs. Our finding is that LMs generally falter when tested on benchmark, hinting at fundamental blindspots in their training setups. Surprisingly, even the classification accuracy on stances derived from the popular HumanEval benchmark is low, calling in question the depth of their comprehension and the source of their success in generating functionally-corree code in the first place. We release our benchmark and evaluation scripts publicly at //aka akaa ms ms/Nsss/o /soooonuvee
in-
Introduction
There has been dazzling progress in the development of newer and more capable language
models (LMs) of code (Chen et al. 2021 Austin et al. 2021 Fried et al.] 2022 Nijkamp
al. 2023 Li et al. Wang et 2013 OpenAl 2023b Luo et al. 2023 Muenmighoo
ct al. 2023) ). Simultane faneouay, the community has been actively designing benchmarks
(Hendrycks et al. 2021 Chen al. 2021 Austin et al. 2021 Puri et al. 2021 Li al. 2022a
et. 2023b) with emphasis on generating code for given problem specification of whal functionality to achieve, e.g., writing a Python function to sort an array.
This is but narrow slice of application of LMs in software engineering pipelines where the tasks are often not straight-forward Developers must consider the overall requirements of the system (e.g., an Android application) to which the code belongs. So, a problem instance in the real-world would be closer to editing Java code to optimize for resource usage on a low-memory Android device than to generating functionally- non-functional requirementt guide the design decisions and constrain how the functionality may be realized (Landes & Studer) 1995), and play central role in real-world software engineering (Chung et al. 2012).. The premise of our work is that while satisfying functional requirements ("what" to implement) is necessary, it is not sufficient.
sort. Such
In this work, we forefront the above-identified significant gap in the current evaluation suites, and introduce a complementary benchmark NoFunEval in first attempt to bridge the gap. We identify a set of five broad non-functional requirements: latency, resource utilization, runtime efficiency, maintainability, and security. We construct code-editing problem in- stances spanning these non-functional requirements and refer to them as NoFunEdit. As LMs
"equal contribution

---

Published as a conference paper at COLM 2024
Functional corrocincss:" "What""
Dappterd
Geparation Ecit
Compenheciin HumanEvalCiassity
OPER
HumanEval/ HumanEvailii
OPTA
0974
Edit NoFunEdkt
Comarahenalan NoFunCiassilyy
Non-functional requinemnts: "How"
Our contr bution: NoFunEval
7g
Safurbip
(a) Composition of NoFunEval (b) Functional correctness
(c) Non-functional requirements
Figure 1: (a) NoFunEval contributes edit and comprehension tasks, NoFun and NoFun- Classify, for non-functional requirements, and complements HumanEval and HumanEvalFix with comprehension task HumanEval Elasssssf...... (b)-(c): Performance of LMs on NoFunEval, HumanEval, and HumanEvalFix benchmarks (metrics, full results in $ For consistency, in plot (c), we include only those instances with a binary evaluation oracle.
are finding increasing use in code generation and editing, it is imperative to test whether they have robust comprehension of the requirements and code semantics. We therefore propose classification instances for both functional-correctness,, derived from the HumanEvalFiix dataset (Muennighoff et al. 2023), and non-functional requirements, from NoFunEdit.
Figure Taa shows the three distinct subsets, NoFunEdit, NoFunClassify and HumanEvalClassife of our NoFunEval benchmark and how they complement existing generation and edit benchmarks HumanEval (Chen et al. 2021) and HumanEvalFix (Muennighoff al. 2023 focused on functional correctness. Our benchmark consists of 958 problem instances in multiple programming languages sourced from public repositories and existing datasets.
Two key challenges in our benchmark design are: (1) how do we convey notions like latency and efficiency that tend to be relative unlike functional correctness that is absolute? Simply describing the requirement in the prompt often fails because LMs may lack the necessary do- main knowledge, unlike in the case of functional correctness where the problem description is usually sufficient. To this end, we design new prompting strategy Coding Concepts (CoCo) which allows developer to succinctly communicate actionable domain knowledge LMs; (2) how do we evaluate the output of LMs? We employ combination of functional (i/o specification) and non-functional (static analysis tools, execution time) oracles. In addition, we consider DiffBLEU (Bairi et al., 2023), , metric defined on code diffs to capture the closeness of predicted edits with the ground-truth edits.
We present comprehensive evaluation of 27 code LMs. A key takeaway of our work, be- sides the benchmark itself, is that existing code LMs (spanning different training strategies, instruction tuning paradigms, and model sizes) falter when we test them on requirements beyond functional correctness. This is highlighted in Figures [Tb by (1) their surpris- ingly low performance on classification (yellow) compared to generation (green) and editing (blue) tasks in both functional and non-functional requirements, and (2) the generally low performance on tasks related non-functional requirements compared relatively higher per- formances on tasks related to functional correctness.
Contributions: To the best of our knowledge, no prior work comprehensively evaluates code language models for multiple non-functional requirements in the context of code- editing and code-comprehension tasks. Our study is the first to observe that most code LMs typically fail to " "discriminate" between buggy code and correct code, despite their ability to "generate" the correct fixes for the buggy code. In summary, we (1) identify the almost exclusive focus on functional correctness in existing benchmarks of code LMs; (2) prepare benchmark to evaluate non- funccionciolll requirements and comprehension ability code LMs; (3) extensively evaluate 27 code LMs and find there is much room to improve comprehension of requirements and code semantics; (4) release our benchmark and evaluation scripts at aka.ms/ Kta.ms/NoounFval
2

---

Published as a conference paper at COLM 2024
NoFunEval
Requirements (Metric): Lanency, Rescurealutiiiation (DMBLEUU) Rumtime Efficlency (CFU-Time), Mointenance, Securit (CodeQL) Languages::Pthon Python, jave, JavaScripp, Kotlin, Ino
Taskss: NoFunEdit, NoFunClasiify, Ex
NoFunEdit
NoFunClassity
ese Instructione
Betow an instustion recerioo a tock The instust temotted so 1ist lnyeense ansiaauss pars
edy instnatiio::
Ruwoitetio janr progbame ostimmze
oyaneean fet of pristlle sngen anditinn
Orly tres ei the two cande aniispate fax
and mprove thurmanna ussige.
lowar tramaayy sorege.
White the anti woude and other textin the
response.
jooug prnggereeeaallln exemireed veste
(+) What spoethelllnnn librrr prognamming or duta structaro desige oonnopp,, or empfemment repuired
[nd] What sproniiu fumatiion skero foum tiiingg lomeer the repparet of chungebi
Codo Bi <Target Codee
Code-A: A: <Source Codeo
<Source Codte:
What speciiii liverr, progiannningge deiia strurlare haree in uorsuep,
Plccse oetect code entopet frorr Code Code with lower memory unilization,
pregeemee detived censtreet in the onde to Not usE reetooe imptoment the requred changett
edl Response:
Prompt
Reaporiss: Coca-
Jova
Lovernte What spectitl funation parent clasa ovvenntee and implementi the class interest
enB instruction: catampiie
Dring
tentual direstyption kime implementt
order
ene Response: <ecempiee
White that speccal complote the roquest
Rueitlet la glvar juva programm zu and mprrvv the memory usage.
instructinn
Instussionn Resenoo thon program opoinite and teprove the moneyy
<Targee
anti coode....
ail Ourospto
HumanEvalClassify
lavo
actioon:oon::
Altnna Cun. (000 559
Ennyene twrn code xnippate conts bug.
Respremtt:
1-Shot Prompt
Proggam:
Code-A A: <Bugar Code with probrion
ede instruction: caramppis
oniCrenlegt
Code: sCaronisal Code with problem caseso
platement capes-
ede Thought: <acemple>
Coding Concepts (CoCo) Prompt
nee Response: <ecemple>
smppett
edle Instruction:
Rewitte java program optimize and mprrvv the memory usoge.
public vold and m iepassnCreette
Curcee
Write the entil code.......
Fiyping the order code and lubul abowe
<Sounce Codeo
Chain-ot-Thouage (CoT) Prompt
Model Input
edt Thunight:
Target Output
Figure 2: Overview of the NoFunEval benchmark. NoFunEval consists of three subtasks, NoFunEdit, NoFunClassify, HumanEvalClassify, spanning multiple programming languages. NoFunEdit ($2.1) involves editing given source code per a user-specified non-functional requirement (c.g., improving memory usage). We design four prompting techniques 2.2 for eliciting LMs to perform the required editing, ranging from minimal task-related infor- mation ("Base") to guiding with high-level hints ("Coding Concepts"). NoFunClassify ($ 2.3
involves distinguishing between two code snippets based on non-functional property (e.g,, selecting the code with lower memory utilization). We construct it by reformulating problems in NoFunEdit. Similarly, we construct HumanEvalClassify (6 2.4) by reformulating HumanEvalFix (Muennighoff et al. 2003), which involves distinguishing two code snippets based on their functional correctness (Le.., bug detection).
2 The NoFunEval Benchmark
Our NoFunEval benchmark (Figure 2 comprises one edit and two classification tasks. In this section, we describe these tasks, the design of the datasets, and the evaluation metrics (summarized Tables A1 and A2 in Appendix A.1
2.1 NoFunEdit
As shown in the LHS of Figure 2 each problem instance in NoFunEdit consists of an instruc- tion specifying the non-functional requirement for code editing, prompting strategy, and the source code that forms the input to the LM, along with the ground-truid code as the desired output. We consider five non-functional coding aspects: (1) Latency: Optimizing code for response times in applications, (2) Resource Utilization: Optimizing for resource utilization like memory, energy or bandwidth, (3) Runtime Efficiency: Improving algorith- mic runtime complexity of code, (4) Maintainability: Enhancing code readability and style as per the best programming practices, and (5) Security: Resolving security vulnerabilities in code. Prior works have studied some of these aspects in isolation (S5). In contrast,
3

---

Published as a conference paper at COLM 2024
NoFunEdit attempts to unify these tasks under a general framework of editing the code to satisfy non-functional requirements. Below, we describe the design of NoFunEdit in detail.
Latency and Resource Utilization: Real-world software systems are often optimized for latency (e.g,, network delays) and resource utilization (e.g,, memory, energy) on edge devices. We turn to open-source Android applications that have commits optimizing latency and resource utilization. We derive such examples from Callan et al. (2022) and Moura et al. (2015), where they mine GitHub commits involving non-tunctional aspects of Android applications. The mining process involves keyword-based filters, manual selection, and learned classifiers, resulting in 931 examples covering latency (execution time and frame rate) and resource utilization (memory, bandwidth, and energy) properties, For our benchmark, we retain only commits from repositories with permissive licenses and targeting a single file. From the commits, we extract the pre-commit code as the input and the post-commit code as the target output. Based on the typical context size of 8192 tokens in many code-LMs, and the prompt lengths in our benchmark, we further filter out instances where the input source code length exceeds 3K tokens (using the StarCoder tokenizer). This results in a total of 114 examples, from which we manually discard 11 examples that do not conform to the commit message. Of the remaining 103, we reserve 5 examples for prompt construction (.2..) The resulting 98 examples are then grouped based on their non-functional property - Tatency (47) and resource utilization (51). For evaluation, directly comparing latency or resource utilization is challenging, as examples vary in their target platforms (devices) and run time requirements (OS). So, we use the DiffBLEU score (Bairi et al. 2023) as our evaluation metric. DifBLEU is designed to capture the similarity of edits made by the LM w.r.t. the target edits.
Runtime Efficiency: For assessing the ability to optimize code runtime, we derive examples from the Python test split of the PIE dataset (Madaan et al.) 2023), which comprises 1K pairs of slow-code and fast-code for problems in the CodeNet challenge (Puri et al.) 2021). We retain functio example pairs with at least two test cases (to ensure functional correctness). Further, we ensure (1) that each selected pair represents unique CodeNet problem (to mitigate biases); (2) the target code is statistically significantly faster than the slower code in each selected pair, and (3) manually verify if the target edit is indeed reasonable edit that can explain the observed speedup. This filtering results in 113 examples. We choose one example from the validation split Madaan et al. (2023) for prompts (S (.2). For evaluation, we measure the average runtimes of the model-edited code the target code over the test cases and over 25 repeated runs, and report the relative speedup. If the model-edited code is functionally incorrect or slower than the original code, we discard the model edits and assume the original code as output (i.e,, a speedup of 1). We conduct experiments on Azure VM[C16]
Maintainability: We derive examples corresponding to various maintainabilit-relatedd issues from Sahu et al. (2024), which was in turn sourced from real-world git reposito- ries (Raychev al. 2016). We select 29 static checks that inspect code maintainability using CodeQL (Cod), a widely-used static analysis tool. We sample 5 examples per check (from thousands) from (aau et al. (2024) for coverage, diversity, and economy. We ensure each selected code has token length less than 3K (as above). This results in total of 145 instances, each with at least one maintainability-rllate
issue flagged, and for which we manually write valid target code. For evaluation, we test LM-edited outputs using the CodeQL tool. If CodeQL flags a warning related to the issue of interest, the LM output is considered a failure. On the other hand, absence of CodeQL warnings does not imply that the code is necessarily improved. LM could simply output empty code or delete offending lines of code, which suppresses warnings. So, to reward LM edits that are closer to the ground-truth edits, we weigh the binary CodeQL success / failure scores with the continuous DiffBLEU scores. We denote this metric by DiffBLEU x CodeQL.
Security: For evaluating an LM's ability to fix security vulnerabilities, we repurpose the Pearce al. 2202) dataset which covers 18 out of the top 25 Common Weakness Errors (CWE) 2021) scenarios. We use upto 2 generations from GitHub Copilot per CWE in the dataset with at least one security issue flagged by CodeQL. This results in total 41 examples across 13 CWEs. We manually write the reference code that addresses the flagged vulnerabilities. For evaluation, we use the DiffBLEUx CodeQL metric as above.
4

---

Published as a conference paper at COLM 2024
Overall, the raw datasets from where we derive NoFunEdit have many more examples. However, to ensure the reliability of the evaluation, we only retained the examples that were strictly consistent with the non-functional requirements. Further, prioritized inclusion of examples that are diverse and cover more types of non-functional requirements than having greater number of similar examples.
Use of DiffBLEU for Evaluation: Our benchmark partly derives examples from files of end- to-end applications designed for diverse platforms like mobile devices, web applications, etc. Thus, setting up execution environments for such tasks is not scalable and expensive to run for the benchmark users. We therefore use the DiffBLEU (Bairi et al., 2023) metric, designed to specifically compare the "edits" generated by the model with the ground truth edits. DiffBLEU addresses the limitations of prior surface-level metrics like Code-BLEEU (Ren et al. (2020) that do not explicitly focus on the edits made to the input file. We also observe high correlation between DiffBLEU and execution-based metrics like accuracy as well as static-analysis based CodeQL scores ($ 4.5 Thus, DiffBLEU serves as a reasonable light- weight alternative to heavy-weight execution or static-analysis based oracles. Moreover, oracles can also be imperfect. For instance, test-cases the CodeQL static checks though expert-designed, may not cover all the corner cases. Thus, we augment our oracles for Security and Maintainability by combining them with DiffBLEU scores, as described earlier.
2.2 Crafting LM Prompts for NoFunEdit
We design four types of prompts to elicit editing abilities in code-LMs (Figure 2 These vary from a minimalist specification of task requirements ("Base") to more comprehensive speci- fication leveraging domain expertise ("Coding Concepts") described below. Examples of all the prompt templates are given in Appendix A.2
For each non-functional requirement in NoFunEdit, write simple instruc- tion that conveys the requirement at a high level. The (Base) example in Figure@isowe how we specify the requirement of optimizing the resource (memory) utilization. Depending on the problem instance, resource utilization prompt specifies one of memory, bandwidth, or energy. Similarly, the prompt for improving latency uses the keywords frame-rate or execution time; and for runtime efficiency, execution time. The prompts for maintainability and security requirements utilize the title of CodeQL warnings flagged for the input code or the common weakness enumeration (CWE) respectively.
Base Prompt:
1-Shot Prompt:
We expand on the Base prompt above, which zero-shot, to include an example that shows how to implement the desired non-functional requirement. In particular, we give a pair of the original source code and the edited source code, as illus- trated in the 1-Shot prompt of Figure 2 Considering very limited data available for each non-functional requirement, limited context lengths supported by various LMs, and each example containing code from entire file, we do not explore multi-shot prompts.
Chain-of-TT Thhought (CoT) Prompt: Combining reasoning with few-shot examples via chain- of-thought has led to improved results across many tasks (Wei et al. 2022). This is partic- ularly appealing for code rewrite tasks that require multiple levels of reasoning under- standing the implementation of the functionality (i.e., the what); the nature of the issue (e.g., memory leak); the source of the issue (i.e., localization); how the issue can be tackled (i.e., the exact code edit necessary). Considering these requirements, we manually augment each example in 1-Shot prompts with explanations (thoughts) eliciting such reasoning steps. As shown in the CoT prompt in Figure the LM first generates a thought by attending to the example in the prompt, and then outputs its response conditioned on the reasoning steps in the generated thought.
thought-augmente
CoT prompts rely on the LM's ability to generate reason- ing steps for the required edits. This could potentially lead to poor judgements in terms of code localization and resolution, depending on the LM's generative abilities as well as its domain knowledge about the non-functional requirement. Often, developers have some idea of what they expect in the edits and can provide the domain knowledge they possess with respect to their codebase, such as what libraries to import for optimizing the code, or which part of code requires the edits, etc. To this end, we propose simple and fairly general
Coding Concepts (CoCo) Prompt:
5

---

Published as a conference paper at COLM 2024
prompting strategy - Coding Concepts (CoCo), that gives the LM hints the programming concepts to use for the task. As shown in the middle pane of Figure 2 we first provide legend of concepts and their descriptions. Then, for the problem instance, we give candidate values for the applicable concepts, serving as directions to the LM for how to implement the edits. The LM has to figure out how to compose the concepts to accomplish the desired task. To ensure high quality, we manually create the CoCo prompts for each problem instance, while also ensuring that hints in these prompts do not reveal the actual edits.
2.3 NoFunClassify
To study comprehension of non-functional requirements in code LMs, we repurpose the NoFunEdit task into a code-clasiffication task called NoFunClassify; it involves distinguishing between two code snippets based on non-functional property (e.g., selecting the code snippet with lower memory usage). The upper right of Figure shows an example. We use standard accuracy as the evaluation metric for this task. However, be agnostic to the ordering of code snippets in the prompt, we prompt the code LM using both the orderings separately. An LM is considered correct on the example only if both the orderings result in correct outputs. While NoFunEdit tests for code editing abilities, NoFun CoFasifyy tests only for code comprehension; thus, one would anticipate NoFunClassify to be easier than NoFunEdit.
2.4
HumanEvalClaassffy
Similar to NoFunClassify, we design HumanEvalClassiff, but to test the comprehension of functional correctness in code LMs. We construct HumanEvalClassify by repurposing the Python split of HumanEvalFixDDoss dataset from Muennighoff et al. (2023), which we refer to simply as HumanEvalFix. HumanEvalFix contains functionally incorrect and correct pairs of code, where the incorrect code obtained by introducing synthetic bugs in the original examples of HumanEval dataset. To convert HumanEvalFix into a code comprehension task, we prompt LMs to select the incorrect code from the two code snippets (as in the bottom right of Figure? 2. We use the same evaluation methodology as for NoFunClassify (5 2.3
3 Experimental Details
We evaluate multiple open and closed-weight code LMs: GPT-4 (OpenAI 2023b) GPT-3.5- Turbo (OpenAI, 2023a), WizardCoder (Luo et al. 2023), StarCoder (Li et al. 2023b), Codel- lama (Roziere et al. 2023), Mistral (Jiang et al. 2023). DeepSeekCoder (Guo et af. 2024),
CodeGemma [CodeGemma 2024), and Llama3 (Dubey et al. 2024) model tamilies. The open-weight models were downloaded from Huggingtace the model sizes vary from 1B to 70B parameters. We cover a total of 27 LMs in our study.
Generating LM Outputs: To decode LM outputs for NoFunEdit and HumanEvalFix, we use two sampling mechanisms: (1) sample 20 generations (per problem instance) with temperature of 0.8, and (2) greedy sampling with temperature 0. We utilize top-k (Fan et al. 2018) and nucleus sampling (Holtzman et al., 2019), with the default values of p 0.95 and 50. All the LMs in our evaluation support context size of 8192 tokens. For the instances corresponding to runtime efficiency and security, which are of relatively shorter length, we restrict the maximum number of sampled tokens to 1200, as in Madaan et al. 2003),, and to 1500 for CoT prompting to accommodate thoughts. For NoFunClassify and fuuaanvvlCClassff we do greedy sampling to generate output labels. We utilize the vLLM library (Kwon et al.
2023) for generating LM outputs for tasks in NoFunEval. For EvalanEvllFix, we use BigCode's evaluation harness (Ben Allal et al., 2022).
Evaluating LM Outputs: We design evaluation metrics specific to each non-functional requirement as detailed in Section 2 summarized in Table For NoFunEdit, since we sample n = 20 candidate outputs per input, we report expected scores using the score@k, n function (Agrawal et al. 2023), a generalization of the pass@k, и function (Chen et al. 2021 that accounts for continuous metrics like DiffBLEU or average speed-up, in addition to discrete metrics. We primarily use score@1, 20 for reporting our observations in Section Similarly, for HumanEvalFix, we report pass@1, 20 scores. For NoFunClassify and HumanEval- Classify, we report classification accuracy.
6

---

Published as a conference paper at COLM 2024
Models
Latency
Resource Util.
Runtime Efficiency
Maintainability
Security
FvaliEi
Min
Min
Max
Min
Bose Prompt
GPT-3.S-Turbu
12.2
$1.3
10.8
29.9 tece
1.302
1.774 Fel
21.0
40.3
39.3
x
72.3
GPT-
14.2
35,9
10.7
26.3
1.303
2.380
53.2
51.1
61.5
61.3 1e0/
90.2
StarCoder-1B 1B
1,9
4.5
1.3
4.8
1004
1009
13"
2,2
1.3
14.4
7.7
WardColkrBB 10
0.2
5.6
0,9 Get
3.99
1002 1
1008
1.7
2,9
7.7a
26.7
19.6
35
3.4
7.6 (x77
8.0
1.033
1.056 Fets
4.5"
6.1
16.5
47.8
40.9
WeardCedir1111
16.1 Sei
5,7 1
15.3 oe
1031
1.183 Sei
6.8"
13,0 M4n
25.6"
51.7 x
Mistral-7B
1
12.0 Cxtx
14
9.3 0.
1010 "
1.182 Setx
4.6"
21.8
42.5
28.5
Mistral-E--rrtt
9.5 tg
1.118 Sei
10.9
CodaLlama-78
10.1 Fita
1,079 Cetl
2.7
46.1
CodiLiima 7B lest
3.7
12.0
1.3
9.1
1.029
1.100
4.8
12.3
19.3*
45.9
20.6
Lodel Lima- 15
3.9
9.5
3.3
8.5
1,051
1.224
3.7"
7.2
14.6
46.6
33.1
Lodal Lame- 1:5 hat
3.6
13.9 Keis
4.0%
11.6
1.337 1
1 1,259
3.5 F.
13.7
22.7*
48.8
30.5
Code 1mm3-14
1.1
3.39
8.2
1.0443
1,512 SKC
7,8
15,8 Goto
23.2
52.7 5
55.9
34g-1sst
2
20.2 Cei
3,9
11.3 Cec
1.052
1.510 Sas
9.7 15
23,4 Vite
24.5
51.7
Phin LodoL (m
1,7
5
21.5 Cacs
1.148
2.155 CAca
15.7
38,8 Cote
33.0
59.1 1
12.7
28.9
11.0
23,
1.76
1,421
11.2
19,8
73.6
0,5
1.022
1.128
7.0
11.7
22.09
53.6
41.5
Liema-3 3-000----5--00 (05-399)
13.2"
34,6
10.7*
27.6
1.172'
2,835
27.1
34.4 MM
37.2"
57.9
81.2
CodeGemma-22 a 20
0.0-3
7.1
2,3
7,6
1.002
1000
2.29
3,3
3.9
47.0
22.7
Loa 7B
0
9.1
1,3
6. 6,3
1006 sat
1.115
5.6"
12.0 uie
28.5"
54.3
11.7
CodeGommaaa lost
3.6
21.0 Cits
2,9-a
12.8
10011
1612
6,5
21,7 Got
$2.4"
67.2 s4
72.4
Dec:eediCoder1.333
2.1
4.7 190
3.0 1
1.07 1
1016 Saa
12a
3.3 Gat
6.4
24.2 tar
16.4
Doc:beelodd71111 1:30-100
cics
cocs
1064
1.184 cics
3.79 Gr
32.7 cide
Uve Coderr--
13.8
14.6
1.150
1,404 Fei
531
12.5 9.m
51.3
Loca5 Lodar Inst
8.6
22.0
7.5
19.6
1.413
1,810
16:1
29.7
32,9
53.7
73.3
3.6
19.7
5.2
16.2
1.521
1.524
10.6
19.7
29.7
52.4
61.6
D8635 Coaeer 3305 Inst
10.7
28.7 0ac
8.7
20.8
1.548
2.269
18.7
32.2
29.5
47.3
81.0
Ground-truth Score
100
100
37
100
100
100
Table 1: Performance of code LMs on the NoFunEdit dataset by different non-functional requirements ($/4.1
4.2) LMs from the same family or sharing the same base model are grouped together. For brevity, we only report the performance of the worst (Min) and the best (Max) performing prompt, with prompt type in the superscript abbreviated 1-Shot (1S), Shaain-of-Thoughe (CoT), Coding Concepts (CoCo). The numbers correspond to the metrics discussed in Section 2
Base (B),
(higher is better; highest in bold; second-highess underlined). We present results for all the prompts in Appendix, Figure A.4 Additionally, we report results on HumanEvalFix for studying its difficulty relative to NoFunEdit.
4 Evaluation Results
4.1 Summary of Results: Two Key Takeaways
Tables 1 and 2 provide performance of all the LMs over code-editing (NoFunEdit) and code-classification tasks (NoFunClassify, HumanEvalClaasifty respectively. We also report the performance of these LMs on HumanEvalFix for reference in Table Overall, we find:
(1) Code LMs struggle to edit code for satisfying non-functional requirements (inferred from Table 1. This is evidenced by the gap between the scores received by them compared to the ground-truth score (the last row in Table 1. This suggests that the LMs have weak understanding of the non-functional requirements of code. Note that since the metrics vary as per the requirements, we cannot compare performances across requirements directly. (2) Code LMs fail to sufficiently comprehend code they can otherwise synthesize or edit (inferred from Table 2. The accuracy of the LMs over NoFunClassify ranges from 0-42% and 0-95.7% for HumanEvalClassify, continuing the trend that non-functional requirements pose a challenge. GPT-4, the best performing model overall, performs much worse on three of the five non-functional tasks; on the other hand, it achieves 95.7% classification accuracy over functional tasks (i.e., (ulasSyvllllassffy.. To our surprise, we find that given an incorrect and corresponding correct code snippet, many code LMs fail to distinguish between the two, but can successfully edit and fix the incorrect code snippet. For instance, DeepSeekCooder-3BBBInss fixes bugs in HumanEvalFix 81% % of times (Table but can distinguish between the incorrect and the correct code only 20.7% of times in HumanEvalClassin (Table We observe similar trends across the LMs except GPT-4 (Figure A.10
7

---

Published as a conference paper at COLM 2024
4.2 How Well do LMs Perform on NoFunEdit?
Larger instruction-tuned models and CoCo prompts offer superior performance. First, in accordance with the scaling laws (Kaplan et al., 2020 Hoffmann al., 2022), we ob- serve that larger models are consistently better than their smaller variants. Second, our proposed CoCo prompt often the highest scoring prompting strategy (99 out of 27 x 5=15 cases). Figure A.7 in Appendix (A.ipreesss an example where GPT-4 utilizes hints in the CoCo prompt to arrive at the correct output, while all other prompts including CoT lead to undesired code edits. Third, instruction-tuning in general helps improve the model performance across all the prompts. For instance, compare the scores of StarCoder- 15. 5B (row 5) with that of its instruction-tuner version WizardCoder-15.5B (row 6). We also note that CoCo prompting strategy may lead to much larger improvements in instruction-tunee models compared to their base-variants- owing to the ability of the former models to follow the instructions encoding the domain knowledge in the CoCo prompt. For instance, on four out of five requirements, CoCo offers greater improvements over the Base prompt for DeepSeekCoder 33B-Inst, when compared to DeepSeekCoder-333 33B.
Zero-shot base prompts outperform 1-shot prompts in several cases. Initially, we an- ticipated higher performance using 1-Shot prompts given the in-context learning ability of LMs. Counter-intuittvell, zero-shot base prompts offer superior results compared to 1-Shot prompts in several cases. A notable exception is the Security task, where 1-Shot prompts usually emerge the winner. This is not surprising given that there are at most one or two test instances per security vulnerability, and the 1-Shot example (derived from official CWE or CodeQL web pages) often captures the nature of the required edits. For the other non-functional requirement,, while we could improve the choice of the example we pick for 1-Shot with additional efforts, we hypothesize that conditioning on 1-Shot examples may introduce unintended biases thereby restricting LMs to generalize beyond the provided example. Figure in Appendix peeroddi/..rrrovides a supporting anecdote. We expect multi-shot prompts with diverse examples to overcome this limitation. However, as discussed in Section 2.2 we could not explore multi-shot prompts due to limited data, and limited context lengths in LMs to support multiple file-sized prompts.
Non-functional improvements may come at the cost of functional correctness. For Run- time Efficiency tasks, we observe that LMs like GPT-4, GPT-3.5-Turee and Llama-3-70B-Ins yield code with significant runtime improvements. However, we also find that they often make edits that compromise on functional correctness while they im- prove the runtime (recall that in such cases, we simply ignore the suggested edits, and retain the input program as mentioned in Section 2.1). Figure AigureA.66in in Appendix A.5 shows one such output obtained from GPT-4, This observation again points to the lack of fundamental understanding, e.g., the non-negotiable nature of functional-correcte non-functional improvements, in code LMs. Appendix presents more detailed obser- vations comparing run-time improvements with drops in execution accuracy (functional correctness).
34B,
while aiming for
4.3
How Well do LMs Perform on Classification Tasks?
Table 2 shows the results for classification tasks NoFunClassify (6 2.3 and HumanEvalClassify (S 2.4 We discover some counter-intuitive trends: (1) No model is consistently the best across all the tasks. For instance, GPT-4 is the best model only for Maintainability,, Security, and Bug Detection tasks, and Llama-3-70B-In on the remaining tasks of identifying programs with better Latency (+15.8%), Resource poanutiiiiingoggrraaaaa
considerably outperforms GPT-4
Utilization (+16.8%), and Run'Time Efficiency (39.0%). (2) Larger models or instruction- tuned variants may not be better than the corresponding smaller or base variants. For instance, CodeLlama-13B outperforms CodeLlama-34B by 14.3% on the task of identifying code snippets with lower latency, and DeepSeekCoder-6 6B-st outperforms DeepSeekCoder 33B-Inst by 5.5% on HumanEvalClassiffy Comparing between base models and their instruction- tuned variants, we notice that CodeLlama-344 outperforms CodeLlama-34B-Ins by 16.7% on identifying code snippets with lower resource utilization. Similarly, DeepSeekCoder-33B outperforms its instruct version by 22.4% on Latency examples. Worse performance of instruct models could be attributed to the lack of task diversity in instruction-tuning datasets.
8

---

Published as a conference paper at COLM 2024
Models
I Latency
Resource Util.
Run'Tim Etfiiency
Maintain- ability
Security
NoFun- Classify
HumanEy- alClassity
GPT-3.5-Turbo
22.4
9.3
12.4
26.8
13.4
28.0
GP
20.4
18.5
15.9
63.4
92.7
42.0
93.7
StarCoder
0.0
0.0
0.8
0.0
0.0
0.0
StarCcder 15.58
2
1.9
3.9
0.0
0.0
2.2
2.4
20.4
5.6
0.0
1.8
Mistral-78
11.1
7.3
3.0
Mistral-75-Inst
8.2
0.0
17.8
9.8
10.3
23,8
Lodil (8-1m)))
4.1
3.7
12.2
0.0
LodeLiama-130 118
20.4
14.8
2.4
1,8
0.0
LodeLLBma-JJW 315
6.1
20.4
39
4.1
4.9
16.4
6.1
4.1
3.7
48.8
25.4
8.5
Phind Lodrl 5m3
18.4
16.7
22.9
18.6
46.3
22.4
34.1
WizardLoder- P433
10.2
9.3
47.5
15.9
31.7
25.0
15.9
3-98-nss
[lama-377 7U8--Int
36.2
35.3
54.9
16.6
29.3
33.5
69.5
CodeGemma 28
4.3
5.9
13.3
4.1
12.2
7.8
12.8
0.0
6.7
Lodetammma 78 lest
10.6
33.3
26.5
11
22
19.4
54.3
DeeSSekCaden-1.99
0.0
2.4
Lader L. Inst
14.3
5.6
16.9
3.1
JeepSiekt adim 78
0.0
0.0
0.8
0.0
D.O
0.2
12.2
DetpSeeRLoder 0.78
3.7
Lespzzak o0er-11
30.6
14.8
2.4
12.8
DespSeekLoder 335 Inst
8.2
3.7
2.5
2.4
20.7
Average Maximum
6.8
30.6
20.4
55.1
63.4
92.7
42.0
95.7
Table 2: Accuracy of LMs (highest; second-highest) on the 5 non-functional requirements, full NoFunClassify (micro-averaged over the requirements), and HumanEvalClassifyy
4.4 How do Comprehension Abilities Compare with Edit Abilities?
From Figures T(b) and cc) we find that LMs are relatively more accurate in code editing compared to the corresponding code comprehension tasks. Notably, this observation holds not only for non-functional requirements but also for functional correctness. For instance, DeepSeekCoder-33B
correctly edit 81% of buggy code in the HumanEvalFix dataset, but it can discriminate between a buggy and the corresponding correct code only 20.7% of the times. Figure A.9 in Appendix shows example. In Figure/A.10) of appendix[A.6 we present performance breakdown for all LMs on classification (HumanEvalClassiffy and the corresponding edit instances (HumanEvalFis)) A glaring observation from Figure A.10 is that all the open-weight LMs except Llama-3- 3-ma---700B 7-BB--Inss invariably fail on getting both the classification and the corresponding edit instance right (red); especially, LMs get the classifi- cation instance wrong when they get the edit instance right in significant number of cases (teal). These observations hint towards poor discriminative abilities generative language models (West et al. 2023).
4.5 How Well does DiffBLEU Correlate with Execution-based or Static-analysis Oracles?
To understand the utility of the DiffBEEU metric and its corre- lation with execution-based or static-analysis based oracles, we measure the Pearson coefficients between DiffBLEU and accu- racy or CodeQL scores across all the models.
Prompt
Pearson Coefficient
Maintainability
Security
(1) For the subset of NoFunEdit tasks where CodeQL metric applies, the results are shown in Table We observe high correlation between Diff- BLEU and CodeQL scores for Maintainability (Pearson=0.90) and Security subsets (Pea))on=0.719) averaged across different types prompts. The correlation in the Security subset rela- tively lower, possibly due to the open-ended nature of the tasks that admit multiple ways of fixing the vulnerabilities. (2) Simi- larly, for HumanEvalFix (not shown in Table 3 where (execution)
Base
0.9646
074011
1-Shot CoT
0,8239
07211
CoCo
0.9677
0.638
Average
0.9080
0.790
Table 3: Correlation be- tween DifBLEU and Cod- eQL scores.

---

Published as a conference paper at COLM 2024
accuracy metric applies, we observe high Pearson coefficient of 0.978 between DiffBLEU and accuracy. Thus, DiffBLEU offers a reliable and light-weight alternative to heavy-weight execution static-analysis based oracles.
Related Work
Code Generation:
Prior work on evaluating code LMs has largely focused on generating functionally correct code for tasks like basic or algorithmic problem solving (Chen et al.
2021 Austin et al. 2021 Hendrycks
al. 2021 Li et al. 2022a), data science (Lai et al. 2022), lext-to-SQL Yu et 2018: et al. 2023a), etc. HumanEval (Chen et al. 2021 serves as a widely used dataset for evaluating generative ability of LMs. HumanEval has been extended to Multipl-E (Cassano et al. 2023) for multilingual evaluation in eighteen pro- gramming languages; to Hvlan (Liu et al. 2023a) with more test-cases for robustness; to InstructHlumnnEva (CodePPrroo 2023) for instruction-following ability.
Code Editing: The majority of soltware engineering workflows involve code-editing tasks like bug fixing (Gupta et al.] 2017: Muennighoff et al., 2023), performance opti- mizations (Madaan et al. 2023) Garg et al. (0022,, improving readability and maintain-
ability (Al Madi 2022: Wadhwa et al., 2023) Jain al.) 2023; Loriot et al., 2022). ) code
migration (Bairi et al. 2023), securit-rrlatted edits (Perry et al. 2022 lony et al. 2023 Pearce
et al.) 2022 He & Vechev [2023] Bhatt et al.) 2023 Zhuo al. 2024), etc. More recent works
like SWE-bench (Jimenez et al. 2024) and Repo oder (Zhagg et al. (0233 focus on repository-
level coding tasks. SWE-bench (Jmenez et al. 2024) requires a model to generate patches
(that can edit multiple files) to resolve issues like bug fixes in large Python repositories;
RepoCoder (Zhang et al. 2023) constructs a code completion dataset using repository-level context under different scenarios like line, API invocation, and method body completion. CodeReviewer (Li et al. 2022b) curates a dataset of real-world code changes and associated reviewer comments aimed at automating code review related activities like change quality estimation, comment generation, and code refinement. Prior work has studied specific non-functional requirements in isolation. In contrast, with NoFunEval, we attempt pt to unify these requirements under a general framework of code-editing, using file-level context.
Code Comprehension: Prior works have considered code-comprehension tasks like clone detection (Svajlenko et al., 2014 Lu et al. 2021), , defect detection (Zhou et al. [2019] Li et al.
2021 Lu et al. 2021), code explanation Muennighoff et al. 2023: Leinonen et al. (002))
and Question Question-Answering over code (Liu & Wan 2021 Lee et al. 2022 Sahu et al. 2024).
Evaluating code LMs on NoFunClassify and HumanEvalllassiin allowed us to directly compare and contrast their comprehension abilities with the corresponding edit abilities.
Conclusions
Code LMs are assuming an important role in the art and craft of software engineering and they should be evaluated on scenarios that matter to practitioners. We focus on two of these in this paper: ability to improve code as per non-functional requirements and ability to comprehend the relation between requirements and code semantics, Our extensive experiments found that the LMs falter on both these counts. With the rapid progress in the field, benchmarks can quickly become irrelevant. A remedy is to continuously improve the benchmarks. Our future goal is to keep extending NoFunEval by adding more languages, labeled examples spanning different requirements, and evaluation harnesses.
7 Reproducibility
We. have open sourced the code and datasets for reproducibility of our experiments. We acknowledge that some of the code in our benchmark might have been seen by the LMs during pre-training. However, the LMs are unlikely to have seen the prompts constructed by us paired with the expected code revisions during pre-training. We thus observe generally poor performance on the NoFunEval Benchmark across all the LMs. Note that we do not use the actual commit messages as instructions in the prompts of our benchmark.
10

---

Published as a conference paper at COLM 2024
References
CodeQL. fttp:://codeel.gittii com
Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram K. Rajamani. Guiding language models of code with global context using monitors, 2023.
Naser Al Madi. How readable model-generated code? examining readability and visual inspection of github copilot. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1-5, 2022.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis with Large Language Models, August 2021. URL pstp::/arxiv.orge. abs/ arXiv:2108.07732 [cs].
2108.07732
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. Codeplan: Repository- level coding using Iims and planning, 2023.
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models. https:: tiub 2022.
Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksandar Straumann, Gabriel Synnaeve, Varun Vontimitta, Spencer Whitman, and Joshua Saxe. Purple llama cyberseceval: A secure coding benchmark for language models, 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason I'hang, and others. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint ar arXiv:2204.06745, 2022.
James Callan, Oliver Krauss, Justyna Petke, and Federica Sarro. How do android developers improve non-functional properties of software? Empir. Softw. Eng., 27(5):113, 2025))211120002.0.0001222 doi: 1007
10.1007/s10664-022-101333 URL battpp://doi.orgge 101100000000010000000001-----111
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and poly- glot approach to benchmarking neural code generation. IEEÉ Transactions on Software Engineering, 49(7):3675-369) 3691, 2023. doi:
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and others. Evaluating large language models trained on code. Xiv preprint arXiv:2107.03374 2021.
Lawrence Chung, Brian A Nixon, Eric Yu, and John Mylopoulos. Non-functional requirements in software engineering, volume Springer Science & Business Media, 2012.
Team CodeGemma. Codegemma: Open code models based on gemma. arxio preprint
arXiv:2406.11409, 2024.
CodeParrot.
http:://huggingaae..c/aaasssss codepparoo//nsstrcchhaaaaaall 2023. [Accessed 01-02-2024].
Instructhumaneval dataset.
CWE. The cwe top 25, 2021. URL htt.:t/pp..crree.aeeeeee///////p cwe_top25. html http:22ttttttfceetoop_______
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint 2xx::2447.22788, 2024.
11

---

Published as a conference paper at COLM 2024
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Amuul Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, 2018.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. ar arXiv preprint ar erxio22044005999 2022.
Spandan Garg, Roshanak Zilouchian Moghaddam, Colin B Clement, Neel Sundaresan, and Chen Wu. Deepperf: A deep learning-basee approach for improving software performance. arXiv preprint arXiv:2206. 13611, 2022.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence, 2024.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. Deepfix: fixing common llanguage errors by deep learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 1345-1351, 2017.
Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communicaaionn Security, pp. 1865-1879, 2023.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arv preprint ar Xr1::2203.15556, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
Huggingface. Huggingface. httpss//huugginffacee.co/ooo
Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen, and Ion Stoica. Lim-assisted code cleaning for training accurate code generators, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint ar arXiv:2310.06825 2023.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. Xr preprini ar Xiv::2000 008661,,..6,,,,, 2020.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. ar Xiv preprint ar arXin:2309.061180 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: natural and reliable benchmark for data science code generation, 2022.
12

---

Published as a conference paper at COLM 2024
Dieter Landes and Rudi Studer. The treatment of non-functional requirements in mike. In European software engineering conference, pp. 294-306, Springer, 1995.
Changyoon Lee, Yeon Seonwoo, and Alice Oh. CSIQA: A dataset for assisting code-based question answering in an introductory programming course. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hummn Language Technologies, pp. 2026-2040) 2022.
Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. Comparing code explanations created by students and large language models. arXiv preprint ar (rXiv:2304.03988, 2023.
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. Can I1m already serve database interface? big bench for large-scale database grounded text-to-sqls. ar Xiv preprint arXit:2305.03111, 2023a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! Xiv preprint ar Xiv:2305.00167 2023b.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Suther- land Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-leve code generation with AlphaCode. Science, 378(6624) 2022a. doi: 10.1126 (0.1126/science.ll science.abql 158. URL https: /ttp://wwsseeeerroogoooo//////// http://///cciinncaagb5551.
httpa/ccccnncc..re/ooaa/aaaaa//a 1126/
science. science.abg11588 h
Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jiin, Yawei Zhu, and Zhaoxuan Chen. Sysevr: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions 2021.
on Dependable and Secure Computing, 19(4):2244-22
Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. Automating code review activities by large-scale pre-training, 2022b. URL (9095)
ar203bs/2203
Chenxiao Liu and Xiaojun Wan. Codeqa: A question answering dataset for source code com- the Association for Computational Linguistics: EMNLP. Association for Computational Linguistics, 2021.
prehension. In Findings the
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation. Xiv preprint ar arXiv:2305.01210, 2023a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation. Xiv preprint ar arXio:2305.01210, 2023b.
Benjamin Loriot, Fernanda Madeiral, and Martin Monperrus. Styler: learning format- ting conventions to repair checkstyle violations. Empirical Software Engineering, 27 (6), aug 2022. doi: 10.1007 10.1007/s10664--2222 /100664-021--00077-0 2Fs10664-021-100 1010000
URL
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint drXiv:2102.04664, 2021.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint Xit::206.00856. 2023.
13

---

Published as a conference paper at COLM 2024
Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. Learning performance- improving code edits. ari preprint (rXi::2322.07867, 2023.
Irineu Moura, Gustavo Pinto, Felipe Ebert, and Fernando Castor. Mining energy-aware commits. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, pp. 56-67, 2015.
doi: 10.1109/ 20R.200..13.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models, 2023.
NC16. Standard NC16as T4 Azure VM.
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Code- gen2: Lessons for training Ilms on programming and natural languages. arv preprint #rXio:2305.02309, 2023.
OpenAI. Gpt-3.5 turbo. [Accessed 2023-03-15).
2023a.
OpenAL. Gpt-4 technical report, 2023b. [Accessed 2023-03-15].
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot's code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 754-768. IEEE, 2022.
Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with ai assistants? arXiv preprint ar 2312v2211.00662,, 2022.
Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. Project codenet: A large-scale for code dataset for learning diversity of coding tasks. arxiv preprint ar Stir:2205.126555 1035, 2021.
Veselin Raychev, Pavol Bielik, and Martin T. Vechev. Probabilistic model for code with decision trees. Proceedings of the 2016 ACM SIGPLAN International Conference on Object- Oriented Programminng, Systems, Languages, and Applications, 2016. URL https://api
semanticscholar.c org/CorpusID 2658344
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: method for automatic evaluation of code synthesis. Xiviv preprint ar arXio:2009.1029) 2020.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. ar Xrv preprint ar 1tkit:2338.1295555 2023.
Surya Prakash Sahu, Madhurima Mandal, Shikhar Bharadwaj, Aditya Kanade, Petros Maniatis, and Shirish Shevade. CodeQueries: Dataset of Semantic Queries over Code. In 17th Innovations in Software Engineering Conference. ACM, 2024.
Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards big data curated benchmark of inter-project code clones. 2014 JEEE International Conference on Software Maintenance and Evolution, pp. 476-480, IEEE, 2014.
Catherine Tony, Markus Mutas, Nicolas Díaz Ferreyra, and Riccardo Scandariato. Llm- seceval: dataset of natural language prompts for security evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), 2023. doi: 10.5281 /2tp//ooooooo.55663.
Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Natural language processing with transformers. O'Reilly Media, Inc.", 2022.
14

---

Published as a conference paper at COLM 2024
Nalin Wadhwa, Jui Pradhan, Atharv Sonwane, Surya Prakash Sahu, Nagarajan Natarajan, Aditya Kanade, Suresh Parthasarathy, and Sriram Rajamani. Frustrated with code quality issues? Iims can help!, 2023.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 billion parameter autoregressive language model, 2021.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. Xiv preprint arXio:2305.07922, 2023.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Adoances in Neural Information Processing Systems, 35:24824-2444 222277,,77,, 2022.
Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, et al. The generative ai paradox:" what can create, it may not understand". ar arx preprint Xrit:2331.00055 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018. Association for Computational Linguistics.
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian- Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2471-2484, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v1/20BB /t0p////11223.mllppppp ma1n..11. URL htttps::/aclaatthoollo wwwpttthooooooooo////000 emnlp-main
151
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 32, 2019.
Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameter-efficiet instruction tuning code large language models. Xiv preprint arXi::2401.0073. 2024.
15

---

Published as a conference paper at COLM 2024
A
Appendix
A.1 Summary of Datasets in NoFunEval as Described in Section
Task Type
Requirement
Examples
Eval Metric
Latency
47
DifÍBLEU
Resource Utilization
DiffBLEU
NoFunEdit
Runtime Efficiency
113
Average Speed-up
Maintainability
145
DifiBLEU XCdeeQ
Security
41
HifBLEUXCodee)
NoFunClassify
All the above
397
Accuracy
HumanEvaiClassity
Correctness
164
Accuracy
Table A1: Summary of the datasets comprising the NoFunEval benchmark ($
Task Type
Language
Examples
Python
277
Java
NoFunFdit and NoFunClassify
C
28
Kotlin
Ino
Javascript XML, Javascript, C++,, Assembly, Objective C
(each) (each)
HumanEvalClassify
Python
164
Table A2: Number of examples from different programming languages present in the NoFunEval benchmark ($
A.2 Prompt Templates
Base Prompt. Non-functional requirement Resource Utill (Bandwidthh)
Below
resporse thot approprianety
mmmolete reguect.
#*# Instruction: Resniče the given Auza program to optimize and improve the bandwiath usage White the entire code and no other test in the response:
Java
paceage oeg, Fivudd fdloadd
pul Reyocti extende Bedauiiiondeen
pelusne untd getteer 4pp
striing cectpath getstrinnsk steiigg deare_path) dean;
Barferachhuatttrea getis ne suffenddrruttttwwalll. (waerse
Fita 25te(destaath); (xxstt())
gatit. closs(;;
Response
Figure A.1: An example Base Prompt for improving bandwidth usage in code for an Android application 2.2..
16

---

Published as a conference paper at COLM 2024
1 1-Shot/CoT Prompt Non-functional requirement Maintainability
Polowris is an distruction that deserbees a tast Write response that aparapoicttl completrr the request
вee Instruction Alonte the gien Puttzn program to avord the 3inquateed cost ponerata" (odtey warning White the entrre code and no other toxt to the response.
for File in Files:
Lines (fax))
i.res]
ylels header, sody
nee Thought The (unction headeend
Stoptterati the underiying iterotor
this, onil showld amapped handie Inspiteattion exoeptions.
ePe Response
def separa (eseeeeedde(eless)))
Lines ster(fste)
beater reat(Llarn))
Expty file Juait
continee lirea]
4dd Instruciian (eewille the given Pytiim progrem in swail the Linoundeizwdinn gevemboy " wonning While the viltee cade und to cllire (azl to the rmpaamnee
Zopall sourty codl
nee Thought
Figure A.2: An example 1-Shot / Shain-of-Thouugt prompt template for fixing maintain- ability issue ("Unguarded next in generator") as flagged by CodeQL. The underlined texts instantiated based the example. The shaded text denotes the reasoning we include for the corresponding Chain-of Thoughh prompt (S 2.2).
CoCo Frompt. Non-functional requirement: Security
/ooo desenders While crepppaoe thet oppegriae
request
Rewnte the given goshon progrom to svoid the Deseriolaction of Vntuvated Dota Codeol worning White the enive code and no other text.
#** Instruction:
d** Concepts:
Jru
(hint)
sofe. lood Ensteod of yom' funccion:
Civen Program:
rhoa Flace copare Floor, angartt, redtrert
Lepors
tecart tyt
Lecort pen2
KRS Flaskt rare
Bag rostec (epl")
dat aol():
ver seri_perlodd repucat. ergaust.errrg.te grpkt "sylooe'', (and)
puni. paodoae sict yerd seuddyand poyd)
nume gunl.aylood well afeett "eaae1) arroe"
return
Responee
Figure A.3: An example CoCo prompt template for fixing a security issue ("Deserialization of Untrusted Data") as flagged by CodeQL (...) The underlined texts are instantiated based on the example.
Figure A.1 provides an example base prompt used for prompting models with the high-level non-functional requirement to be achieved in editing the code. Figure highlights the template used for prompting models with a single example along with a thought relevant for the non-functional requirement being redses..d. Figure A.3 highlights the template used for prompting models providing them the coding concepts needed for handling the non-functional requirement in the example shown (S 2.2
A.3 Performance of Different Prompts on NoFunEdit
Figure A.4 provides summary of all the absolute values of the evaluation numbers obtained across models and prompts. The darker shades indicating better performance on that specific task 4.2
17

---

Published as a conference paper at COLM 2024
Latency
Resource Viillcttio
141
13:
13.
11.3
10.
Lana TED
141
18.3
114
131
174
153
37
83
154
1.3
103
Wax Maarrrerr
Prompts
Effickescy
Heintoisaiiity
1.30
25.
25.4
21.3
1883
121
181
1.06
1.34
1.01
1.73
1307
1.33
DRILEU CHEAGE 01,70
157
53
1.27
1.39
187
1.00
111
1338
207
148
1113
1:21
1320
1.38
1230
1.00
133
Fesmen
(opues))
MM
162
1:44
0..
23.3
35.2
SHoneess
471
13:0
16-7
213
Prompts
Figure A.4: Performance of code LMs on the NoFunEdit dataset by different non-functional requirements, for the four prompts ($ 4.2).
A.4 Additional Experiments on Classification Tasks
Observing embarrassingly lower performance on classification compared to the correspond- ing edit tasks (6 4.44, led us to try out more prompts to reduce the dependence of key our observations on a specific prompt format. Thus, we re-ran classification experiments with different prompt format. In the main paper, we report numbers for the prompt that asks the model to select from "Code-A" or "Code-B" (Figure 2 R.H.S.). As an alternate, we tried prompting the model to output "Yes" or "No" using the prompt format shown in Figure This prompt takes two code snippets "Code-A" and "Code-B" as input, claims hypothesis (e.g. Code-A is faster than Code-B), and asks the LLM to predict whether the hypothesis is correct not by answering in "Yes" or "No". Table A3
numbers for NoFun Classiff
18

---

Published as a conference paper at COLM 2024
Models
Latency
Resource Utilization
RunTTme Efficiency
Maintainability
Security
NoFunClassify
HumanBvalClassify
(Bug Detection)
GPT-3. 5-Turbo
9.3
15.0
24.4
12.8
39.6
GPT-4
4.1
3.7
0.9
49
95.1
28.9
90.9
StarCoder-1B
2.0
1.9
1.4
0.0
0.0
Wizzare ooser 18
0.0
0.D
5.1
0.7
2.4
2,0
1,8
0.0
2.8
0.0
0.6
Wizaros.oder-1 15.38
0.0
0.0
0.0
8.3
0.0
3.0
0.0
Mistral-75-Inst
4.1
9.3
4.2
6.9
7.3
3.0
CodeLloma-7B
0.0
0.7
0.0
0.0
Lodel Jama- Inst
2.0
1.9
1.7
0.0
0.0
1.0
0.6
LodeLloma-193 138
18.4
9,3
4.2
3.4
0.0
5.8
0.0
Codel Jama- In5
7.4
0.0
6.1
Codel Jama 145
2.0
0.0
0.0
0.0
0.0
0.2
4,3
2.4
Phind LodeLiama 34B
6.1
7.4
10.2
18.6
9.8
12.4
9.8
Wizard(_ooder119-5
2.0
0.0
7.6
0.0
1.8
Coder- 38
8.2
5.6
0.0
0.0
0.0
1.7
18
0.0
0.0
0.0
0,0
1.5
0.0
JespbeexLaddem ader-b.78
4.1
0.0
0.0
0.7
0.0
0.0
Theppeent.adeen Ladee---ddee---- 78--nst
0.0
0.0
0.0
0.7
0.0
0.3
1.8
DespbeokL_dder3350
DespSeeLLader3383 338-
0.0
3.7
7.6
7.6
2.4
5.7
32.9
Avg Max
18.4
9,3
15.3
49
93.1
28.9
90.9
Table A3: Accuracy of LMs (highest in bold; second-highen underlined) on the five non- functional requirements, entire NoFunClassify (micro-averaged over the five requirements), and HumanEvalClassify, for the alternate "Yes-No" prompts.
and HumanEvalClassify for this prompt, corresponding Table Overall, that "Yes" or "No" prompt in Figure leads to even lower performance for both NoFunClassify and HumanEvalClassify.
Classification Prompt Template
Below is an instruction that describes a task. Write response that appropriately completes the request.
### Instruction:
Only one of the two code snippets has lower memory usage.
Code-A: <Source Code>
Code-B: <Target Code>
Does Code-A have lower memory usage than Code-B9 B? Please answer just "Yes" or "No"
### Response:
Figure A.5: Prompt Template for alternate classification prompt in "Yes" "No" format.
19

---

Published as a conference paper at COLM 2024
A.5 Trade-off in Non-functional Improvements and Functional Correctness 4.2
Model
Average SpeedUp @1,20
Execution Accuracy @1,20
UPT-3.5-Turbb
Base Prompt
1.302
69.2
One Shot
1.303
79.5
Chain of Thought
1.583
67.6
Coding Concepts
1.774
48.0
GPT-4
Base Prompt
1.303
67.8
One Shot
1.839
69.3
Chain of Thought
2.294
66.0
Coding Concepts
2.380
59,2
WizardCoder-155.
Base Prompt
1.031
65.0
One Shot
1.060
68.1
Chain of Thought
1.107
27.6
Coding Concepts
1.183
54.8
CodeLlama- 138-Inst
Base Prompt
1.055
68.8
One Shot
1.037
67.0
Chain of Thought
1.058
45.6
Coding Concepts
1.259
55.9
Biind-CoeeLlama-4BB
Base Prompt
1.148
53.1
One Shot
1.276
67.3
Chain of Thoughh
1.447
37.7
Coding Concepts
2.155
52.7
WizardCoder- PYB34B
Base Prompt
1.076
26.9
One Shot
1.162
25.5
Chain of Thought
1.238
12.7
Coding Concepts
1.421
24.4
DeepSeckCoder-6 6.7-
Base Prompt
1.573
58.3
One Shot
1.475
63.8
Chain of Thought
1.413
45.4
Coding Concepts
1.810
61.7
DeepSeekCCader-33B-B
Base Prompt
1.548
64.4
One Shot
1.566
69.1
Chain of Thought
1.611
53.4
Coding Concepts
2.269
63.8
Table A4: Trade-off in Non-functional improvements (speed-up) and functional correctness (execution accuracy).
As discussed in Section 4.2 edits made by the model to satisfy non-functional requirements may come at the cost of functional correctness. In Table we present results from few models which compromise the functional correctness of the input code an in attempt to improve the runtime. Recall from Section 2.1 that the input code is always functionally correct (i.e., execution accuracy of 100%). Figure|A.s presents example where GPT-4 generates an output functionally different from the input code while attempting to its improve runtime.
20

---

Published as a conference paper at COLM 2024
Base Prompt
Model Output
Below is an instruction that describes task. Write response that appropriately completes the request.
### Instruction:
Rewrite the given python program to optimize and improve the speed of execution. Write the other the response.
entire code
import numpy as np
python
import numpy as пр
A np. ap.arraa(listmmapiinn input input().spli())) split())))
(val((n) np. app.arayylissttttareppp (Imptmannimttttmmiiintt, input().split(())
print(np.minnnp.
(ma www.oolnoop...oo log2 (A))))
while(all A A/2 ans +- 1
pannt
AN" Response:
python
Figure A.6: In an attempt to improve the execution time, GPT-4 makes the output code functionally different from the input code, resulting in test-case violations. When all the elements of the array are odd, the model-generated code (RHS) would print non-zero number, while the original code (LHS) would print 0.
21

---

Published as a conference paper at COLM 2024
A.6 Anecdotal Examples
Prompt
Truth/CoCo Output
pockage ocn. Srowson Hightning;
Below 15 an irstruction that deseribes task. The instruction contains concapta formatied concept baysords separated by realine.
puelic closs Wistersaatatoeeeenn tutencs SQLtbclperkeipe"
kayaord pairs
puelsc synchrontzed unta delete(Sering unl)
Mrring a gebii st treforl
discriptions
[in] spectric
imploment
change(s)-
specific library, programeing
public synchrentzed vold deletellssooy (String unnl)
prograesr--oe defined
sbatabase delete( (TABLE_HESTORN KEYLURL 25 non unt 333
implement required changets)
Sening]]
spacific librany,
defince construct REMOVE seplesent required
USE
[Smplement] specttic turction imolerent class intereat
(owerr de] what specific function the class overnide inpliment in the class of intarest
Output
[Hint] Full or partial textusl description of the idea imploment write resporse that sopropristely completus the request.
breaser. It includes nethots for croating, readirg, g. updating, and deleting sistory icees. The expcution tim can be seproved optieizinn the dacabasa quartes. for exceolo, the aet.asttineroditeee retrieves all stens free the dstabase and then selects the last hundred itert. This can be optimized by Liniting the quary to reture only the last hundred itens Similarly, the gebnisatoyy (esssut nethod retrienes all item froe the catabase to court then. this can be optinited by using the squ couRi function. the gollodnn optixized code includes these reprovesents.
Thought: The given zava class is responsible for handling the history database
and Instruction:
Rownibe ristory deletion code to Ieprove exccution time. Mrite the entire code and no other text in the response.
method
see concepts:
Stieie ar3) funs
del cteet storrtteenttttiigg 16) Durctios
Lieslerecti deletedd stooetttooyyee the. /Stein url) Dunction
Given program:
Jave
package scr. ser.browser- Lighting::
Resparse:
public ristoryaccaseeenddarr
sq.iteopemmlper
parkagg acr. benoser Hightning:
public syrchronized delete(string
public mistoryatabasseandiia extends squftuqpurmilper
Strang n pesisstoryytexnor)))
public synchronized delete(steing url)
String peovlstoryiten(urb) Seletettssteyttee((])..
public syrchrontzed eeleteiittrrytteeittrnngg 15) del.torTABLE HISTORY, <EY_I0
string[] String.valuenne valuuor(i)))))
public synchronized vold delete--rtoryItennee Dostabare.eeeetoftee del.eeddeettt.e........ HOSTORY, KEY stringl] SString veranng.
public int penidstaryttesccartt
string countquery "SELECT FRDY TABLE HISTOR))
curson curson #bscabso.rrw werbboorwwnuerrrccownterrr null];
returr curson. cursen.getcoun((); getcount();
tursor.closedg
TABLE_HISTORY
public int getrasteryyteacComtt
String counsquery "SZLECT counmeey FROM
Curson cursor abatabose. wwstabsse..atteerrr............e www.ecyycoouttuuuyyyy null); cursen-invetooaseee
int court cunson protnteR0y
hasponse:
curase curasr.cloied ccbou())> return cursor.jetcount((;
Javo
Figure A.7: An example where GPT-4 utilizes hints from the CoCo prompt to successfully produce the ground truth code whereas the CoT and other prompts lead to unnecessary code edits (highlighted in green).
22

---

Published as a conference paper at COLM 2024
1-Shot Prompt
Ground Truth
N = int int(eval(input())
[ ] (N+1)
Below is instruction that describes task along with an example. Write a response that appropriately completes the request.
L[0] = 2 L[1]
seu Instruction:
Rewrite the given python progran to optimize and improve the speed of execution. write the entire
for i in range N+1): L[i] = L[i-2] L[i-1]
"'yyhhon =nn((eva((nppu(()) ans 18*TN 2*9**N + 8**N
print (L[[t))
print((ans((1071944))
Base-Prompt Output
#е# Response:
def fibonacci(n):
python iit(eeva((nnutt)))
b = 2, 1 range(n): a, b = b, b return int int(inut()) print(fibonacci
MOD 18**947
ans (pow(10, N, MOD) 2" pow(9, N, NOD)
pow(8, N, (O00 print(ans)
#е# Instruction:
Rewrite the given python progran optinize
improve the speed of execution. Write the entire
code and other text in the response.
''python
1-Shot Output
(in((e)al(nnuutt)))
int (input()) [0, 1]
L [[ 1690098
L[e] =
[[1] =
for range(2, N+1):
L[1] L[1-2] L[1-1]
for in range (2, N+1): [pppdd + L[-1])
print((L(N])) ...
Response:
python
print(L[N]]
Figure A.8: Augmenting the Base prompt with 1-Shot example (1-Shot prompt) often leads to worse performance. The figure shows example output from the GPT-4 model. Here, the Base prompt results the correct output, even better than the ground truth (more memory efficient), however, using 1-Shot prompt (base prompt augmented with 1-Shot example) results in a functionally incorrect output, suggesting that model might learn unintended edit patterns from the 1-Shot example irrelevant to the non-functional requirement.
23

---

Published as a conference paper at COLM 2024
Problem Statement
You're given a list of deposit and withdrawal operations on bank account that starts with zero balance. Your task detect at any point the balance of account falls below zero, and at that point function should return True. Otherwise, it should return False. below zero([1, 2, 3])
False
(eroo_, -4, 5])
True
Bugny
Model Output
from typing import List
from typing import List
def below wwwl:w/eeo zow.rrtatop::::ons: List[int]) -> bool: balance
def below below_zero (operrattt:: List[int]] -> bool: balance
for op in operations: balance += op if balance return True
for op operations: balance += if balance e
return True
return False
return False
Figure A.9: An example where DeepSeekCooerr33BB... successfully fixes the buggy code, but fails to distinguish between the buggy and the fixed code.
24

---

Published as a conference paper at COLM 2024
GPT-3.5-Turbo
GPT-4
Starcoder-1B
WizardCoder--1B
Starcoder-158
WizardCoder-15B
Mistral-7B
Mistral-7B-Inst
CodeLlama-78
Codellama-7B-Inst
CodeLlama-138
Codellama-13B-Inst
CodeLlama-34B
Codellamaa344--ist
Phind-Codeellmm--448
WizardCoder-Py-33B
Llama-3-8B-Inst
Llama-3-70B-ins
CodeGeema-7B-Inst
CodeGemma-28
CodeGemma-7B
Deepseek-1.3B
Deepseek-1.3B-Inst
Deepseek-6.78
Deepseek-6.7B-Inst
Deepseek-33B
Deepseek- -3-eekk-33B-Ins- Inst
Accuracy
Both Correct
Correct classification
Correct edit
Both Wrong
Figure A.10: Comparing code-editing (HumanEvalFiix) and the corresponding code- classification (HumanEvalClassitt) performance of LMs ($ 4.4. LMs fail invariably at getting both the classification and edit instance correct (red color). For a significant number of instances where LMs get the editing right, they fail on the classification instance (teal color).
25


Reviews:
[{"review_id": "UaQZhr7Q63", "reviewer": "Reviewer_ddLg", "summary": "This paper discusses the limitations of current benchmarks for evaluating code LLMs, which primarily focus on functional correctness and use test cases to evaluate the correctness of the generated codes.  The authors propose a new benchmark called NoFunEval to conduct more code evaluations. They evaluate twenty-two code LLMs and find that they struggle on the new benchmark, indicating deficiencies in their training. Overall, this paper is well-written and clear.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review. We respond to the reviewer’s comments and questions below.\n\n>My main concern is that there are some code benchmarks, that also aim to evaluate the coding ability of LLMs at the Non-functional level, such as SWE-bench[1], CoderReviewer[2], and RepoCoder[3]...\n\nThank you for pointing us to these benchmarks. As we discuss below, the objectives of these benchmarks and our work are different. We will nevertheless include this discussion in the paper.\n\nSWE-Bench and RepoCoder focus on repository-level coding tasks. SWE-Bench requires a model to generate pull-requests to resolve GitHub issues from Python repositories. These issues are primarily bug fixes, and the correctness is validated using test cases. RepoCoder constructs a dataset for code completion within repository-level context under different scenarios like line, API invocation and method body completion. These are evaluated using edit similarity and tests. Our aim in NoFunEval is distinct. We focus on evaluating whether a model can edit a given code snippet to improve a non-functional property. Due to a variety of requirements, we use corresponding oracles (like static analysis or execution, as applicable) for evaluation.\n\nCodeReviewer curates a dataset of real-world code changes and associated reviewer comments. It aims at automating code review related activities like change quality estimation, comment generation and code refinement. They measure classification accuracy and code similarity. In our work, we focus on the task of improving code as per a non-functional requirement and evaluate different LMs and prompting techniques.\n\n>The link can not be opened.\n\nWe have intentionally elided the URL on page-2 to preserve anonymity. However, we have submitted our code, dataset, and results as a supplementary resource along with our submission. The URL will point to a public open-source repository in the final version of our paper.\n\n>The statistics of the dataset on different program languages should be shown.\n\nThank you for this suggestion. The dataset primarily consists of Python, Java, and C. In addition, there are few instances in latency and resource utilization from other languages such as Kotlin, Javascript, Scala, Ino, Assembly, C++, and ObjectiveC. We will update our paper with the statistics as suggested by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper builds a new dataset to better evaluate the coding ability of LLMs.\n2. This paper conducts lots of experiments on different LLMs.\n3. This paper is well-motivated, well-written and clear.", "reasons_to_reject": "1. My main concern is that there are some code benchmarks, that also aim to evaluate the coding ability of LLMs at the Non-functional level, such as SWE-bench[1], CoderReviewer[2], and RepoCoder[3]. In this case, this paper should discuss the differences between the NoFunEval and existing benchmarks, such as the new evaluation metrics, evaluation on different dimensions, and code optimization.\n2. The link can not be opened.\n3. The statistics of the dataset on different program languages should be shown.\n\n\n[1] Can Language Models Resolve Real-world Github Issues?\n[2] CodeReviewer: Pre-Training for Automating Code Review Activities.\n[3] RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mSMfTMSUGE", "reviewer": "Reviewer_Qahr", "summary": "The authors propose and construct an evaluation suite for code language models to test their performance with respect to non-functional requirements. The authors claim to release the required code and data to the public, providing a valuable benchmark that, as the authors show, at least existing models do not perform well on.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review. We respond to the reviewer’s comments and questions below.\n\n>Authors restrict evaluation to a small number of model providers (both closed and open weights). Given authors finding that GPT-4 does not perform well on the proposed benchmark, comparison to other closed-weight models such as Claude or Gemini would have been interesting.\n\nThank you for this suggestion. As presented in the paper, we have already evaluated 22 leading open and closed source code LMs of different sizes. We have since then evaluated newer open-source models like the Gemma family from Google, Llama family from Meta and DBRX from Databricks which we add below. The evaluation of these and other models is an ongoing process. We share some preliminary results below (the best prompts for the corresponding configurations are stated in parentheses). We will try to add results from these models along with Claude and Gemini in the revised version.\n\n| Models| Latency | Resource Utilization | Runtime Efficiency | Maintainability | Security |\n|-|-|-|-|-|-|\n| Meta Llama-3-8B              | 6.3 (CoCo) | 4.4 (CoCo) | 1.062 (CoCo) | 11.7 (CoCo) | 53.6 (CoT) |\n| Meta Llama-3-70B-Instruct    | 34.6 (CoCo) | 27.6 (CoCo) | 2.335 (CoCo) | 44.4 (CoCo) | 57.9 (1S) |\n| Google CodeGemma-2b          | 7.1 (Base) | 7.6 (1S) | In progress | 3.3 (CoCo) | 47.0 (CoT) |\n| Google CodeGemma-7b          | 9.1 (CoCo) | 6.1 (CoCo) | 1.115 (CoCo) | 12.0 (CoCo) | 54.3 (1S) |\n| Google CodeGemma-1.1-7b-it   | 21.0 (CoCo) | 12.8 (Base) | In progress | 21.7 (CoCo) | 67.2 (1S) |\n| Databricks DBRX-Base         | 16.0 (CoCo) | 6.7 (CoCo) | In progress | 32.4 (CoCo) | 59.2 (CoCo) | \n\nWe highlight that the results on these newer models are consistent with the main observation in our paper that most models struggle to satisfy non-functional requirements, while scale in terms of model parameters and hints in the form of CoCo prompt are generally helpful.\n\n>Please include samples from every task and prompt type in the appendix\n\nThank you for this suggestion. We will add samples from each task with examples of the different types of prompts in the appendix in the revised version. We will also be releasing the code, and all the data including the prompts publicly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Authors formalize an important task (non-functional requirement related performance) and construct a benchmark dataset for it\n- Authors release code and dataset\n- Authors show that existing models do not perform well on the task", "reasons_to_reject": "Authors restrict evaluation to a small number of model providers (both closed and open weights). Given authors finding that GPT-4 does not perform well on the proposed benchmark, comparison to other closed-weight models such as Claude or Gemini would have been interesting.", "questions_to_authors": "- Please include samples from every task and prompt type in the appendix", "extra_scores": {}}, {"review_id": "jkLpxRPEgt", "reviewer": "Reviewer_mw5q", "summary": "The paper introduces a new benchmark, NoFunEval, to evaluate language models of code (code LMs) based on their ability to handle non-functional requirements such as efficiency, security, and maintainability, instead of simply focusing on functional correctness. The paper proposes a prompting method called Coding Concepts (CoCo) to help communication between the developers and code LMs. Twenty-two code LMs are tested using this new benchmark revealing significant shortcomings in their performance and indicating fundamental flaws in their training setups.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We  thank the reviewer for the insightful review. We respond to the reviewer’s comments and questions below.\n\n>The CoCo prompting needs expert-provided hints which is expensive to acquire in real scenarios.\\\n\\\n>Given various problems in the dataset, how would you efficiently generate the \"concepts\" for each problem? Are the concepts different from problem to problem?\n\nThe purpose of the CoCo prompt is to allow a developer to communicate domain knowledge (e.g., about relevant libraries/APIs) that a model may lack. We agree that providing such hints incurs some extra effort; however, as Reviewer- Mb9C also notes, providing such hints should be trivial, particularly in the settings where the developer is familiar with the codebase they are editing. Note that at inference time, the developer has to provide hints specific to only the particular issue they are working on. \n\nOur preliminary experiments with GPT-4 suggested that we could generate these hints automatically from raw data (e.g., by summarizing the ground-truth diff) for the purposes of benchmark creation. However, in the interest of generating a high-quality dataset, we chose to manually curate CoCo prompts for each problem, mainly providing clues around what libraries to use, and what parts of code require an edit, without revealing the actual edits.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The non-functional requirements are important to code LM in real applications. The experiments are extensive to show the potential direction.", "reasons_to_reject": "The CoCo prompting needs expert-provided hints which is expensive to acquire in real scenarios.", "questions_to_authors": "Given various problems in the dataset, how would you efficiently generate the \"concepts\" for each problem? Are the concepts different from problem to problem?", "extra_scores": {}}, {"review_id": "LfhL3zmFuz", "reviewer": "Reviewer_Mb9C", "summary": "In this paper, the authors introduced a new benchmark called NoFunEval, focusing on code generations against other non-functional qualities of codes such as security, efficiency, etc. In addition, the authors also provide a new prompting technique called Coding Concepts (CoCo) to prompt the models with the specification of the non-functional requirements.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review. We respond to the reviewer’s comments and questions below.\n\n> The evaluation metrics are not ideal in some cases...\n\nWe use DiffBLEU exclusively for 2/5 requirements as noted by the reviewer. For others, we also perform execution- or static analysis-based evaluations.\n\nExecution-based evaluation is feasible for benchmarks like HumanEval or PIE comprising standalone functions. Our benchmark derives examples from end-to-end applications from diverse platforms like mobile devices and web applications. Setting up execution is difficult across multiple environments and is expensive to run for the benchmark users. Compared to evaluation of functional correctness using tests, measuring improvements to non-functional properties requires real-world workloads which is challenging, e.g., simulating network traffic to measure latency or replaying user requests to measure memory utilization.\n\nWe hope to address these limitations in future and will discuss this in our paper. Given the high correlations between DiffBLEU and static-analysis oracles, we consider DiffBLEU as a cheaper yet reliable alternative.\n\n> The overall number of samples in each task is quite small...more general languages or coding scenarios.\n\nWe agree that more examples will be helpful. But our benchmark has 397 examples, which is much higher than code-editing datasets like HumanEvalFix (164) or CanItEdit (105). Our benchmark is already multi-lingual, primarily Python, Java, and C.\n\nThe raw datasets we derive NoFunEval from have orders of magnitude more examples. However, to ensure quality, we retained examples clearly consistent with non-functional requirement (e.g., filter out examples with unrelated edits - Sec 2.1). We prioritized inclusion of diverse examples covering more requirements than more but similar examples. We evaluate four different prompting strategies further adding to diversity. \n\nWe aim to maintain and augment NoFunEval as a live benchmark.\n\n> The prompting technique CoCo...Are they extracted from data sources or manually written by humans?\n\nIn preliminary experiments with GPT-4, we could generate CoCo prompts automatically from raw data (e.g., by summarizing ground-truth diff). Though possible to use it for benchmark creation, to ensure quality, we manually curated CoCo prompts, providing clues around what libraries to use, and what parts of code require an edit, without revealing actual edits. We will discuss this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The benchmark is a timely direction to address the arising research of code generation and the need to evaluate diverse qualities of codes beyond functional correctness. I found the proposed list of non-functional requirements quite comprehensive, including the code latency, resource utilization, runtime efficiency, maintainability, and security. \n-Quite comprehensive evaluation with many experimental results from strong base code LLMs, including StarCoder, WizardCoder, Mistral, CodeLLama, etc. The results are fully described and analyzed with interesting insights e.g. generation vs. classification performance.", "reasons_to_reject": "- The evaluation metrics are not ideal in some cases. For instance, in latency and resource utilization, the performance is evaluated by only DiffBLEU against the ground-truth code. It would be better to have a metric that actually measures the impacts of the generated code e.g. by real runtime, or utilized resources, even in a sandbox/ independent execution environment.\n- The overall number of samples in each task is quite small, ranging from 47 to 145 samples only. Therefore, it would be quite hard to generalize the results to more general languages or coding scenarios. \n- The prompting technique CoCo is rather trivial. Using the list of detailed hints/ guidelines, the models are expected to improve in their generation outputs. However, I am uncertain how these guidelines were obtained. If it was not explained in the paper, please describe how these hints/ guidelines were obtained. Are they extracted from data sources or manually written by humans?", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new benchmark dataset for code generation, specifically on non-functional requirement related performance. This appears to be an important area in evaluating code generation models that deserves more attention. In addition, the paper also provides a comprehensive experimental study, with more results from the rebuttal. Some concerns from the reviewers, such as the exact prompting technique and more justification of the evaluation metrics, are legit and should be address properly in the next version of the paper.
Decision: Accept