NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness

Authors: manav  singhal
tushar  aggarwal
abhijeet  awasthi
nagarajan  natarajan
aditya  kanade
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 07:08:05

Abstract: Existing evaluation benchmarks of language models of code (code LMs) focus almost exclusively on whether the LMs can generate functionally-correct code. In real-world software engineering, developers think beyond functional correctness. They have requirements on "how'' a functionality should be implemented to meet overall system design objectives like efficiency, security, and maintainability. They would also trust the code LMs more if the LMs demonstrate robust understanding of such requirements.

We propose a new benchmark NoFunEval to evaluate code LMs on non-functional requirements and simple classification instances for both functional and non-functional requirements. We propose a prompting method, Coding Concepts (CoCo), as a way for a developer to communicate the domain knowledge to the LMs. We conduct an extensive evaluation of twenty-two code LMs. Our finding is that they generally falter when tested on our benchmark, hinting at fundamental blindspots in their training setups. Surprisingly, even the classification accuracy on functional-correctness instances derived from the popular HumanEval benchmark is low, calling in question the depth of their comprehension and the source of their success in generating functionally-correct code in the first place. We release our benchmark and evaluation scripts publicly at https://aka.ms/NoFunEval.

Main paper: 
 introduction: 
 There has been dazzling progress in the development of newer and more capable language models (LMs) of code (Chen et al., 2021; Austin et al., 2021; Fried et al., 2022; Nijkamp et al., 2023; Li et al., 2023b; Wang et al., 2023; OpenAI, 2023b; Luo et al., 2023; Muennighoff et al., 2023) . Simultaneously, the community has been actively designing benchmarks (Hendrycks et al., 2021; Chen et al., 2021; Austin et al., 2021; Puri et al., 2021; Li et al., 2022a; Liu et al., 2023b) with emphasis on generating code for a given problem specification of what functionality to achieve, e.g., writing a Python function to sort an array. This is but a narrow slice of application of LMs in software engineering pipelines where the tasks are often not as straight-forward. Developers must consider the overall requirements of the system (e.g., an Android application) to which the code belongs. So, a problem instance in the real-world would be closer to editing Java code to optimize for resource usage on a low-memory Android device than to generating a functionally-correct sort. Such non-functional requirements guide the design decisions and constrain how the functionality may be realized (Landes & Studer, 1995) , and play a central role in real-world software engineering (Chung et al., 2012) . The premise of our work is that while satisfying functional requirements ("what" to implement) is necessary, it is not sufficient.In this work, we forefront the above-identified significant gap in the current evaluation suites, and introduce a complementary benchmark NoFunEval in a first attempt to bridge the gap. We identify a set of five broad non-functional requirements: latency, resource utilization, runtime efficiency, maintainability, and security. We construct code-editing problem instances spanning these non-functional requirements and refer to them as NoFunEdit. As LMs are finding increasing use in code generation and editing, it is imperative to test whether they have robust comprehension of the requirements and code semantics. We therefore propose classification instances for both functional-correctness, derived from the HumanEvalFix dataset (Muennighoff et al., 2023) , and non-functional requirements, from NoFunEdit.Figure 1 (a) shows the three distinct subsets, NoFunEdit, NoFunClassify and HumanEvalClassify, of our NoFunEval benchmark and how they complement existing generation and edit benchmarks HumanEval (Chen et al., 2021) and HumanEvalFix (Muennighoff et al., 2023) focused on functional correctness. Our benchmark consists of 958 problem instances in multiple programming languages sourced from public repositories and existing datasets.Two key challenges in our benchmark design are: (1) how do we convey notions like latency and efficiency that tend to be relative unlike functional correctness that is absolute? Simply describing the requirement in the prompt often fails because LMs may lack the necessary domain knowledge, unlike in the case of functional correctness where the problem description is usually sufficient. To this end, we design a new prompting strategy Coding Concepts (CoCo) which allows a developer to succinctly communicate actionable domain knowledge to LMs;(2) how do we evaluate the output of LMs? We employ a combination of functional (i/o specification) and non-functional (static analysis tools, execution time) oracles. In addition, we consider a DiffBLEU (Bairi et al., 2023) , a metric defined on code diffs to capture the closeness of predicted edits with the ground-truth edits.We present a comprehensive evaluation of 27 code LMs. A key takeaway of our work, besides the benchmark itself, is that existing code LMs (spanning different training strategies, instruction tuning paradigms, and model sizes) falter when we test them on requirements beyond functional correctness. This is highlighted in Figures 1(b )-1(c) by ( 1) their surprisingly low performance on classification (yellow) compared to generation (green) and editing (blue) tasks in both functional and non-functional requirements, and (2) the generally low performance on tasks related to non-functional requirements compared to relatively higher performances on tasks related to functional correctness.To the best of our knowledge, no prior work comprehensively evaluates code language models for multiple non-functional requirements in the context of codeediting and code-comprehension tasks. Our study is the first to observe that most code LMs typically fail to "discriminate" between buggy code and correct code, despite their ability to "generate" the correct fixes for the buggy code. In summary, we (1) identify the almost exclusive focus on functional correctness in existing benchmarks of code LMs;(2) prepare a benchmark to evaluate non-functional requirements and comprehension ability of code LMs;(3) extensively evaluate 27 code LMs and find there is much room to improve comprehension of requirements and code semantics; (4) release our benchmark and evaluation scripts at aka.ms/NoFunEval. NoFunEdit ( § 2.1) involves editing a given source code as per a user-specified non-functional requirement (e.g., improving memory usage). We design four prompting techniques ( § 2.2) for eliciting LMs to perform the required editing, ranging from minimal task-related information ("Base") to guiding with high-level hints ("Coding Concepts"). NoFunClassify ( § 2.3) involves distinguishing between two code snippets based on a non-functional property (e.g., selecting the code with lower memory utilization). We construct it by reformulating problems in NoFunEdit. Similarly, we construct HumanEvalClassify ( § 2.4) by reformulating HumanEvalFix (Muennighoff et al., 2023) , which involves distinguishing two code snippets based on their functional correctness (i.e., bug detection). 
the nofuneval benchmark: 
 Our NoFunEval benchmark (Figure 2 ) comprises one edit and two classification tasks. In this section, we describe these tasks, the design of the datasets, and the evaluation metrics (summarized in Tables A1 and A2 in Appendix A.1).As shown in the LHS of Figure 2 , each problem instance in NoFunEdit consists of an instruction specifying the non-functional requirement for code editing, a prompting strategy, and the source code that forms the input to the LM, along with the ground-truth code as the desired output. We consider five non-functional coding aspects: (1) Latency: Optimizing code for response times in applications, (2) Resource Utilization: Optimizing for resource utilization like memory, energy or bandwidth, (3) Runtime Efficiency: Improving algorithmic runtime complexity of code, (4) Maintainability: Enhancing code readability and style as per the best programming practices, and (5) Security: Resolving security vulnerabilities in code. Prior works have studied some of these aspects in isolation ( § 5). In contrast, NoFunEdit attempts to unify these tasks under a general framework of editing the code to satisfy non-functional requirements. Below, we describe the design of NoFunEdit in detail. (Madaan et al., 2023) , which comprises 1K pairs of slow-code and fast-code for problems in the CodeNet challenge (Puri et al., 2021) . We retain functionally-correct example pairs with at least two test cases (to ensure functional correctness). Further, we ensure (1) that each selected pair represents a unique CodeNet problem (to mitigate biases); (2) the target code is statistically significantly faster than the slower code in each selected pair, and (3) manually verify if the target edit is indeed a reasonable edit that can explain the observed speedup. This filtering results in 113 examples. We choose one example from the validation split of Madaan et al. (2023) for prompts ( § 2.2).For evaluation, we measure the average runtimes of the model-edited code and the target code over the test cases and over 25 repeated runs, and report the relative speedup. If the model-edited code is functionally incorrect or slower than the original code, we discard the model edits and assume the original code as output (i.e., a speedup of 1). We conduct experiments on Azure VM NC16 .We derive examples corresponding to various maintainability-related issues from Sahu et al. (2024) , which was in turn sourced from real-world git repositories (Raychev et al., 2016) . We select 29 static checks that inspect code maintainability using CodeQL (Cod), a widely-used static analysis tool. We sample 5 examples per check (from thousands) from Sahu et al. ( 2024) for coverage, diversity, and economy. We ensure each selected code has token length less than 3K (as above). This results in a total of 145 instances, each with at least one maintainability-related issue flagged, and for which we manually write a valid target code. For evaluation, we test LM-edited outputs using the CodeQL tool.If CodeQL flags a warning related to the issue of interest, the LM output is considered a failure. On the other hand, absence of CodeQL warnings does not imply that the code is necessarily improved. LM could simply output empty code or delete offending lines of code, which suppresses warnings. So, to reward LM edits that are closer to the ground-truth edits, we weigh the binary CodeQL success/failure scores with the continuous DiffBLEU scores. We denote this metric by DiffBLEU×CodeQL.For evaluating an LM's ability to fix security vulnerabilities, we repurpose the Pearce et al. (2022) dataset which covers 18 out of the top 25 Common Weakness Errors (CWE, 2021) scenarios. We use upto 2 generations from GitHub Copilot per CWE in the dataset with at least one security issue flagged by CodeQL. This results in total 41 examples across 13 CWEs. We manually write the reference code that addresses the flagged vulnerabilities.For evaluation, we use the DiffBLEU×CodeQL metric as above.Overall, the raw datasets from where we derive NoFunEdit have many more examples. However, to ensure the reliability of the evaluation, we only retained the examples that were strictly consistent with the non-functional requirements. Further, we prioritized inclusion of examples that are diverse and cover more types of non-functional requirements than having a greater number of similar examples.Use of DiffBLEU for Evaluation: Our benchmark partly derives examples from files of endto-end applications designed for diverse platforms like mobile devices, web applications, etc. Thus, setting up execution environments for such tasks is not scalable and expensive to run for the benchmark users. We therefore use the DiffBLEU (Bairi et al., 2023) metric, designed to specifically compare the "edits" generated by the model with the ground truth edits. DiffBLEU addresses the limitations of prior surface-level metrics like Code-BLEU (Ren et al., 2020) that do not explicitly focus on the edits made to the input file. We also observe a high correlation between DiffBLEU and execution-based metrics like accuracy as well as static-analysis based CodeQL scores ( § 4.5). Thus, DiffBLEU serves as a reasonable lightweight alternative to heavy-weight execution or static-analysis based oracles. Moreover, oracles can also be imperfect. For instance, test-cases or the CodeQL static checks though expert-designed, may not cover all the corner cases. Thus, we augment our oracles for Security and Maintainability by combining them with DiffBLEU scores, as described earlier.We design four types of prompts to elicit editing abilities in code-LMs (Figure 2 ). These vary from a minimalist specification of task requirements ("Base") to more comprehensive specification leveraging domain expertise ("Coding Concepts") as described below. Examples of all the prompt templates are given in Appendix A.2.Base Prompt: For each non-functional requirement in NoFunEdit, we write a simple instruction that conveys the requirement at a high level. The (Base) example in Figure 2 shows how we specify the requirement of optimizing the resource (memory) utilization. Depending on the problem instance, resource utilization prompt specifies one of memory, bandwidth, or energy. Similarly, the prompt for improving latency uses the keywords frame-rate or execution time; and for runtime efficiency, execution time. The prompts for maintainability and security requirements utilize the title of CodeQL warnings flagged for the input code or the common weakness enumeration (CWE) respectively.We expand on the Base prompt above, which is zero-shot, to include an example that shows how to implement the desired non-functional requirement. In particular, we give a pair of the original source code and the edited source code, as illustrated in the 1-Shot prompt of Figure 2 . Considering very limited data available for each non-functional requirement, limited context lengths supported by various LMs, and each example containing code from entire file, we do not explore multi-shot prompts.Chain-of-Thought (CoT) Prompt: Combining reasoning with few-shot examples via chainof-thought has led to improved results across many tasks (Wei et al., 2022) . This is particularly appealing for code rewrite tasks that require multiple levels of reasoning -understanding the implementation of the functionality (i.e., the what); the nature of the issue (e.g., memory leak); the source of the issue (i.e., localization); how the issue can be tackled (i.e., the exact code edit necessary). Considering these requirements, we manually augment each example in 1-Shot prompts with explanations (thoughts) eliciting such reasoning steps. As shown in the CoT prompt in Figure 2 , the LM first generates a thought by attending to the thought-augmented example in the prompt, and then outputs its response conditioned on the reasoning steps in the generated thought.Coding Concepts (CoCo) Prompt: CoT prompts rely on the LM's ability to generate reasoning steps for the required edits. This could potentially lead to poor judgements in terms of code localization and resolution, depending on the LM's generative abilities as well as its domain knowledge about the non-functional requirement. Often, developers have some idea of what they expect in the edits and can provide the domain knowledge they possess with respect to their codebase, such as what libraries to import for optimizing the code, or which part of code requires the edits, etc. To this end, we propose a simple and fairly general prompting strategy -Coding Concepts (CoCo), that gives the LM hints on the programming concepts to use for the task. As shown in the middle pane of Figure 2 , we first provide a legend of concepts and their descriptions. Then, for the problem instance, we give candidate values for the applicable concepts, serving as directions to the LM for how to implement the edits. The LM has to figure out how to compose the concepts to accomplish the desired task.To ensure high quality, we manually create the CoCo prompts for each problem instance, while also ensuring that hints in these prompts do not reveal the actual edits.To study comprehension of non-functional requirements in code LMs, we repurpose the NoFunEdit task into a code-classification task called NoFunClassify; it involves distinguishing between two code snippets based on a non-functional property (e.g., selecting the code snippet with lower memory usage). The upper right of Figure 2 shows an example. We use standard accuracy as the evaluation metric for this task. However, to be agnostic to the ordering of code snippets in the prompt, we prompt the code LM using both the orderings separately. An LM is considered correct on the example only if both the orderings result in correct outputs. While NoFunEdit tests for code editing abilities, NoFunClassify tests only for code comprehension; thus, one would anticipate NoFunClassify to be easier than NoFunEdit.Similar to NoFunClassify, we design HumanEvalClassify, but to test the comprehension of functional correctness in code LMs. We construct HumanEvalClassify by repurposing the Python split of HumanEvalFixDocs dataset from Muennighoff et al. (2023) , which we refer to simply as HumanEvalFix. HumanEvalFix contains functionally incorrect and correct pairs of code, where the incorrect code is obtained by introducing synthetic bugs in the original examples of HumanEval dataset. To convert HumanEvalFix into a code comprehension task, we prompt LMs to select the incorrect code from the two code snippets (as in the bottom right of Figure 2 ). We use the same evaluation methodology as for NoFunClassify ( § 2.3). 
experimental details: 
 We evaluate multiple open and closed-weight code LMs: GPT-4 (OpenAI, 2023b), GPT-3.5-Turbo (OpenAI, 2023a), WizardCoder (Luo et al., 2023) , StarCoder (Li et al., 2023b)  and  (2) greedy sampling with temperature 0. We utilize top-k (Fan et al., 2018) and nucleus sampling (Holtzman et al., 2019) , with the default values of p = 0.95 and k = 50. All the LMs in our evaluation support context size of 8192 tokens. For the instances corresponding to runtime efficiency and security, which are of relatively shorter length, we restrict the maximum number of sampled tokens to 1200, as in Madaan et al. (2023) , and to 1500 for CoT prompting to accommodate thoughts. For NoFunClassify and HumanEvalClassify, we do greedy sampling to generate output labels. We utilize the vLLM library (Kwon et al., 2023) for generating LM outputs for tasks in NoFunEval. For HumanEvalFix, we use BigCode's evaluation harness (Ben Allal et al., 2022) .Evaluating LM Outputs: We design evaluation metrics specific to each non-functional requirement as detailed in Section 2 and summarized in Table A1 . For NoFunEdit, since we sample n = 20 candidate outputs per input, we report expected scores using the score@k, n function (Agrawal et al., 2023) , a generalization of the pass@k, n function (Chen et al., 2021) , that accounts for continuous metrics like DiffBLEU or average speed-up, in addition to discrete metrics. We primarily use score@1, 20 for reporting our observations in Section 4.Similarly, for HumanEvalFix, we report pass@1, 20 scores. For NoFunClassify and HumanEval-Classify, we report classification accuracy.  
summary of results: two key takeaways: 
 Tables 1 and 2 provide performance of all the LMs over code-editing (NoFunEdit) and code-classification tasks (NoFunClassify, HumanEvalClassify) respectively. We also report the performance of these LMs on HumanEvalFix for reference in Table 1 . Overall, we find:(1) Code LMs struggle to edit code for satisfying non-functional requirements (inferred from Table 1 ). This is evidenced by the gap between the scores received by them compared to the ground-truth score (the last row in Table 1 ). This suggests that the LMs have weak understanding of the non-functional requirements of code. Note that since the metrics vary as per the requirements, we cannot compare performances across requirements directly.(2) Code LMs fail to sufficiently comprehend code they can otherwise synthesize or edit (inferred from Table 2 ). The accuracy of the LMs over NoFunClassify ranges from 0-42% and 0-95.7% for HumanEvalClassify, continuing the trend that non-functional requirements pose a challenge. GPT-4, the best performing model overall, performs much worse on three of the five non-functional tasks; on the other hand, it achieves 95.7% classification accuracy over functional tasks (i.e., HumanEvalClassify). To our surprise, we find that given an incorrect and a corresponding correct code snippet, many code LMs fail to distinguish between the two, but can successfully edit and fix the incorrect code snippet. For instance, DeepSeekCoder-33B-Inst fixes bugs in HumanEvalFix 81% of times (Table 1 ), but can distinguish between the incorrect and the correct code only 20.7% of times in HumanEvalClassify (Table 2 ). We observe similar trends across the LMs except GPT-4 (Figure A.10).Larger instruction-tuned models and CoCo prompts offer superior performance. First, in accordance with the scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) , we observe that larger models are consistently better than their smaller variants. Second, our proposed CoCo prompt is often the highest scoring prompting strategy (99 out of 27×5=135 cases). Figure A.7 in Appendix A.6 presents an example where GPT-4 utilizes hints in the CoCo prompt to arrive at the correct output, while all other prompts including CoT lead to undesired code edits. Third, instruction-tuning in general helps improve the model performance across all the prompts. For instance, compare the scores of StarCoder-15.5B (row 5) with that of its instruction-tuned version WizardCoder-15.5B (row 6). We also note that CoCo prompting strategy may lead to much larger improvements in instruction-tuned models compared to their base-variants -owing to the ability of the former models to follow the instructions encoding the domain knowledge in the CoCo prompt. For instance, on four out of five requirements, CoCo offers greater improvements over the Base prompt for DeepSeekCoder-33B-Inst, when compared to DeepSeekCoder-33B.Zero-shot base prompts outperform 1-shot prompts in several cases. Initially, we anticipated higher performance using 1-Shot prompts given the in-context learning ability of LMs. Counter-intuitively, zero-shot base prompts offer superior results compared to 1-Shot prompts in several cases. A notable exception is the Security task, where 1-Shot prompts usually emerge the winner. This is not surprising given that there are at most one or two test instances per security vulnerability, and the 1-Shot example (derived from official CWE or CodeQL web pages) often captures the nature of the required edits. For the other non-functional requirements, while we could improve the choice of the example we pick for 1-Shot with additional efforts, we hypothesize that conditioning on 1-Shot examples may introduce unintended biases thereby restricting LMs to generalize beyond the provided example. Figure A.8 in Appendix A.6 provides a supporting anecdote. We expect multi-shot prompts with diverse examples to overcome this limitation. However, as discussed in Section 2.2, we could not explore multi-shot prompts due to limited data, and limited context lengths in LMs to support multiple file-sized prompts.Non-functional improvements may come at the cost of functional correctness. For Runtime Efficiency tasks, we observe that LMs like GPT-4, GPT-3.5-Turbo, Phind-CodeLlama-34B, and Llama-3-70B-Inst yield code with significant runtime improvements. However, we also find that they often make edits that compromise on functional correctness while they improve the runtime (recall that in such cases, we simply ignore the suggested edits, and retain the input program as mentioned in Section 2.1). Figure A.6 in Appendix A.5 shows one such output obtained from GPT-4. This observation again points to the lack of fundamental understanding, e.g., the non-negotiable nature of functional-correctness while aiming for non-functional improvements, in code LMs. Appendix A.5 presents more detailed observations comparing run-time improvements with drops in execution accuracy (functional correctness).Table 2 shows the results for classification tasks NoFunClassify ( § 2.3) and HumanEvalClassify ( § 2.4). We discover some counter-intuitive trends: (1) No model is consistently the best across all the tasks. For instance, GPT-4 is the best model only for Maintainability, Security, and Bug Detection tasks, and Llama-3-70B-Inst considerably outperforms GPT-4 on the remaining tasks of identifying programs with better Latency (+15.8%), Resource Utilization (+16.8%), and RunTime Efficiency (39.0%).(2) Larger models or instructiontuned variants may not be better than the corresponding smaller or base variants. For instance, CodeLlama-13B outperforms CodeLlama-34B by 14.3% on the task of identifying code snippets with lower latency, and DeepSeekCoder-6.7B-Inst outperforms DeepSeekCoder-33B-Inst by 5.5% on HumanEvalClassify. Comparing between base models and their instructiontuned variants, we notice that CodeLlama-34B outperforms CodeLlama-34B-Inst by 16.7% on identifying code snippets with lower resource utilization. Similarly, DeepSeekCoder-33B outperforms its instruct version by 22.4% on Latency examples. Worse performance of instruct models could be attributed to the lack of task diversity in instruction-tuning datasets. Table 2 : Accuracy of LMs (highest; second-highest) on the 5 non-functional requirements, full NoFunClassify (micro-averaged over the 5 requirements), and HumanEvalClassify.From Figures 1(b ) and 1(c), we find that LMs are relatively more accurate in code editing compared to the corresponding code comprehension tasks. Notably, this observation holds not only for non-functional requirements but also for functional correctness. For instance, DeepSeekCoder-33B-Inst can correctly edit 81% of buggy code in the HumanEvalFix dataset, but it can discriminate between a buggy and the corresponding correct code only 20.7% of the times. (teal) . These observations hint towards poor discriminative abilities in generative language models (West et al., 2023) . To understand the utility of the DiffBLEU metric and its correlation with execution-based or static-analysis based oracles, we measure the Pearson coefficients between DiffBLEU and accuracy or CodeQL scores across all the models. (1) For the subset of NoFunEdit tasks where CodeQL metric applies, the results are shown in Table 3 . We observe a high correlation between Diff-BLEU and CodeQL scores for Maintainability (Pearson=0.908) and Security subsets (Pearson=0.719) averaged across different types of prompts. The correlation in the Security subset is relatively lower, possibly due to the open-ended nature of the tasks that admit multiple ways of fixing the vulnerabilities. (2) Similarly, for HumanEvalFix (not shown in Table 3 ), where (execution) accuracy metric applies, we observe a high Pearson coefficient of 0.978 between DiffBLEU and accuracy. Thus, DiffBLEU offers a reliable and light-weight alternative to heavy-weight execution or static-analysis based oracles.Base One-shot CoT CoCSpeedUps@1,20  
related work 6 conclusions: 
 Code LMs are assuming an important role in the art and craft of software engineering and they should be evaluated on scenarios that matter to practitioners. We focus on two of these in this paper: ability to improve code as per non-functional requirements and ability to comprehend the relation between requirements and code semantics. Our extensive experiments found that the LMs falter on both these counts. With the rapid progress in the field, benchmarks can quickly become irrelevant. A remedy is to continuously improve the benchmarks. Our future goal is to keep extending NoFunEval by adding more languages, labeled examples spanning different requirements, and evaluation harnesses. 
reproducibility: 
 We have open sourced the code and datasets for reproducibility of our experiments. We acknowledge that some of the code in our benchmark might have been seen by the LMs during pre-training. However, the LMs are unlikely to have seen the prompts constructed by us paired with the expected code revisions during pre-training. We thus observe generally poor performance on the NoFunEval Benchmark across all the LMs. Note that we do not use the actual commit messages as instructions in the prompts of our benchmark. Scores@1,20 6.63.7 3.0 3.9 3.92.9 3.5 0.93.94.5 1.3 3.3 
 
 Appendix: Observing embarrassingly lower performance on classification compared to the corresponding edit tasks ( § 4.4), led us to try out more prompts to reduce the dependence of key our observations on a specific prompt format. Thus, we re-ran classification experiments with a different prompt format. In the main paper, we report numbers for the prompt that asks the model to select from "Code-A" or "Code-B" (Figure 2 , R.H.S.). As an alternate, we tried prompting the model to output "Yes" or "No" using the prompt format shown in Figure A.5. This prompt takes two code snippets "Code-A" and "Code-B" as input, claims a hypothesis (e.g. Code-A is faster than Code-B), and asks the LLM to predict whether the hypothesis is correct or not by answering in "Yes" or "No". As discussed in Section 4.2, edits made by the model to satisfy non-functional requirements may come at the cost of functional correctness. In Table A4 , we present results from a few models which compromise the functional correctness of the input code an in attempt to improve the runtime. Recall from Section 2.1, that the input code is always functionally correct (i.e., execution accuracy of 100%). Figure A.6, presents an example where GPT-4 generates an output functionally different from the input code while attempting to its improve runtime.  


Decision: This paper proposes a new benchmark dataset for code generation, specifically on non-functional requirement related performance. This appears to be an important area in evaluating code generation models that deserves more attention. In addition, the paper also provides a comprehensive experimental study, with more results from the rebuttal. Some concerns from the reviewers, such as the exact prompting technique and more justification of the evaluation metrics, are legit and should be address properly in the next version of the paper.

Reviews: review: paper_summary: paper_summary: This paper discusses the limitations of current benchmarks for evaluating code LLMs, which primarily focus on functional correctness and use test cases to evaluate the correctness of the generated codes.  The authors propose a new benchmark called NoFunEval to conduct more code evaluations. They evaluate twenty-two code LLMs and find that they struggle on the new benchmark, indicating deficiencies in their training. Overall, this paper is well-written and clear.
strength_weakness: strength_weakness: 1. My main concern is that there are some code benchmarks, that also aim to evaluate the coding ability of LLMs at the Non-functional level, such as SWE-bench[1], CoderReviewer[2], and RepoCoder[3]. In this case, this paper should discuss the differences between the NoFunEval and existing benchmarks, such as the new evaluation metrics, evaluation on different dimensions, and code optimization.
2. The link can not be opened.
3. The statistics of the dataset on different program languages should be shown.


[1] Can Language Models Resolve Real-world Github Issues?
[2] CodeReviewer: Pre-Training for Automating Code Review Activities.
[3] RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation.
paper_summary: paper_summary: The authors propose and construct an evaluation suite for code language models to test their performance with respect to non-functional requirements. The authors claim to release the required code and data to the public, providing a valuable benchmark that, as the authors show, at least existing models do not perform well on.
strength_weakness: strength_weakness: Authors restrict evaluation to a small number of model providers (both closed and open weights). Given authors finding that GPT-4 does not perform well on the proposed benchmark, comparison to other closed-weight models such as Claude or Gemini would have been interesting.
questions: questions: - Please include samples from every task and prompt type in the appendix
paper_summary: paper_summary: The paper introduces a new benchmark, NoFunEval, to evaluate language models of code (code LMs) based on their ability to handle non-functional requirements such as efficiency, security, and maintainability, instead of simply focusing on functional correctness. The paper proposes a prompting method called Coding Concepts (CoCo) to help communication between the developers and code LMs. Twenty-two code LMs are tested using this new benchmark revealing significant shortcomings in their performance and indicating fundamental flaws in their training setups.
strength_weakness: strength_weakness: The CoCo prompting needs expert-provided hints which is expensive to acquire in real scenarios.
questions: questions: Given various problems in the dataset, how would you efficiently generate the "concepts" for each problem? Are the concepts different from problem to problem?
paper_summary: paper_summary: In this paper, the authors introduced a new benchmark called NoFunEval, focusing on code generations against other non-functional qualities of codes such as security, efficiency, etc. In addition, the authors also provide a new prompting technique called Coding Concepts (CoCo) to prompt the models with the specification of the non-functional requirements.
strength_weakness: strength_weakness: - The evaluation metrics are not ideal in some cases. For instance, in latency and resource utilization, the performance is evaluated by only DiffBLEU against the ground-truth code. It would be better to have a metric that actually measures the impacts of the generated code e.g. by real runtime, or utilized resources, even in a sandbox/ independent execution environment.
- The overall number of samples in each task is quite small, ranging from 47 to 145 samples only. Therefore, it would be quite hard to generalize the results to more general languages or coding scenarios. 
- The prompting technique CoCo is rather trivial. Using the list of detailed hints/ guidelines, the models are expected to improve in their generation outputs. However, I am uncertain how these guidelines were obtained. If it was not explained in the paper, please describe how these hints/ guidelines were obtained. Are they extracted from data sources or manually written by humans?

Comments: title: Thank you for acknowledging our response. We would like to add a few clarifications.
Response to Authors
comment: Thank you for the follow up. We will incorporate our response in the revised version and clearly articulate the significant differences in the objectives of these benchmarks compared to ours.
Thank you for acknowledging our response. We would like to clarify that:
 
- In Appendix A5, we report a high correlation between DiffBLEU and Execution accuracy for HumanEvalFix dataset, in addition to correlation with CodeQL scores for Security and Maintainability. While prior work has shown BLEU to correlate poorly with functional correctness in HumanEval and MBPP benchmarks, it's important to note that unlike BLEU, DiffBLEU is specifically designed for code-edits, and that it correlates strongly with Execution accuracy for HumanEvalFix  (Pearson coefficient 0.978). We will move these details from the appendix to the main paper for more clarity in the revised version.
 
- Providing additional hints via prompts like CoCo can be useful in settings where users are already familiar with the codebase they are editing. Our main objective in evaluating CoCo prompts is to understand how well different LMs can respond to additional hints.
Even though the authors have claimed that the work is different with other benchmarks. But, I think these benchmarks also show the contribution to the No Function Eval. This paper should carefully revise to avoid the confusing points.
I appreciate the authors for responding to my comments. While I noted the novelty of the proposed benchmark to test various aspects of code generation, I am still not fully convinced due to the two concerns:
-  The evaluation method (BLEU has been shown to correlate quite badly with functional correctness e.g. in HumanEval and MBPP papers; in Appendix A.5, the reported correlation of DiffBLEU and CodeQL scores is only for Maintainability and Security tasks); and
-  The benefits of the proposed CoCo prompting technique (which requires human experts to provide detailed coding concepts; this is not scalable as the concepts are quite fine-grained and require a deep understanding of the code context)