Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models

Authors: Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun
Primary area: datasets and benchmarks
Keywords: Code Generation, Multidimension, Benchmark, LLM Evaluation

Abstract:
In recent years, researchers have proposed numerous benchmarks to evaluate the impressive coding capabilities of large language models (LLMs). However, current benchmarks primarily assess the accuracy of LLM-generated code, while neglecting other critical dimensions that also significantly impact code quality in real-world development. Moreover, relying exclusively on correctness as the guiding metric renders LLMs vulnerable to data contamination. Therefore, this paper proposes the **RACE** benchmark, which comprehensively evaluates the quality of code generated by LLMs across 4 dimensions: Readability, mAintainability, Correctness, and Efficiency. Specifically, considering the demand-dependent nature of dimensions beyond correctness, we design various types of user requirements for each dimension to assess the model's ability to generate correct code that also meets user demands. We analyze 28 representative LLMs based on RACE and find that: 1) current correctness-centric benchmarks fail to capture the multifaceted requirements of code in real-world scenarios, while RACE provides a comprehensive evaluation that reveals the defects of LLMs across multiple dimensions; 2) the RACE benchmark serves as an effective tool for resisting the risk of data contamination; 3) even the most advanced code LLMs still encounter significant challenges in customized requirements involving complex instructions; 4) most LLMs exhibit an inherent preference for specific coding style. These findings highlight the need for a multidimensional evaluation of code LLMs, emphasizing metrics beyond correctness for real-world applications. Future efforts should aim to develop novel learning algorithms to enhance code generation under varied constraints and improve coverage and usability for diverse user needs.

Decision: reject
Reviewer scores: 3, 6, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730606737862

thread_type:
official_review

<thread_object>
creator: reviewer_ucc2

<content>
summary:
This paper aims to develop code generation benchmark that focuses on objectives beyond correctness.
Specifically, this paper develops the RACE benchmark to cover readability, maintainability, correctness, and efficiency.
Additionally, this paper studies the performance of a comprehensive set of LLMs.

strengths:
+ This work focuses on evaluating code generation of LLMs which is an important problem
+ Studywise this work evaluates a comprehensive set of LLMs and covers some recent LLMs

weaknesses:
- There have been various published evaluation papers studying the aspect beyond code correctness [1,2,3]. For example, [1,2] covers the additional dimensions introduced in this paper, i.e., efficiency/maintainability/readability. Other non-correctness dimensions such as self-consistency and security have also been studied. Unfortunately, these prior papers seem to be ignored by this paper.
- There is little new technical design/novelty in the evaluation pipeline except for adding customized requirements.
- Overall this paper also did not bring new coding tasks but rather reused existing benchmarks.
- The metric design for readability and maintainability is debatable as these are subjective dimensions where the corresponding metrics should be prompt-defined.
- It seems the efficiency metric relies on execution runtime, whose value can be platform-dependent and impacted by system noises. Prior work [2,4] have proposed solution metrics to address this issue.
- This work did not talk about how to ablate correctness impact as different LLMs can have different sets of correct solutions.

[1] NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness. COLM 2024.
[2] Evaluating Language Models for Efficient Code Generation. COLM 2024.
[3] Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain. ICLR 2024
[4] Learning Performance Improving Code Edits. ICLR 2024.

questions:
- What's the novelty and contribution of this work considering the prior closely related work [1-4]
- When evaluating non-functional dimensions, how does this work deal with the associated correctness when computing the metrics
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608788210

thread_type:
official_review

<thread_object>
creator: reviewer_sbvp

<content>
summary:
The paper proposes RACE, a new benchmark evaluating 4 dimensions of code generation: Readability, mAintainability, Correctness, and Efficiency. The authors claim that existing benchmarks primarily address code accuracy, neglecting the other aspects. For each dimension, the authors designed various types of user requirements to assess the model's ability to generate correct code that also meets user demands. A study of 28 LLMs using the RACE reveals that: 1) correctness-focused benchmarks are insufficient for real-world code requirements, while RACE offers a comprehensive evaluation; 2) RACE helps mitigate data contamination risks; 3) even advanced code LLMs struggle with complex, customized instructions; and 4) most LLMs show biases toward specific coding styles.

strengths:
- In addition to the correctness, the paper evaluates multiple dimensions of code generation, including readability, maintainability, and efficiency, which most existing benchmarks are lack of. This is a good motivation.
- The provided artifact is well documented and seems to be reproducible. The authors also did a wide range of evaluations, involving 28 LLMs, open or closed, with different sizes.
- The authors showed that RACE is more robust to contaminations. It's also interesting that readability seems to be a critical
indicator of overall code quality.

weaknesses:
1. The 4 dimensions chosen for constructing RACE are not convincing. While the authors claimed that the philosophy of the design comes from the demands for code quality in software engineering, it is unclear why these dimensions they are eventually selected.
2. The data source is quite limited, only covering 4 Python datasets. These datasets are more focused on self-contained code, limiting the scope of the paper. Also, if RACE is multi-dimensional, why not treat multi-language coding as another dimension?
3. The core contribution of the paper is weak. Actually, many high-quality correctness and code efficiency benchmarks exist in the wild. For correctness, besides HumanEval+, MBPP+, LeetCode, and ClassEval as mentioned in the paper, there are BigCodeBench, LiveCodeBench, MultiPL-E, etc. For efficiency, there are PIE and EvalPerf. Then the paper only contributed to readability and maintainability, which are much less important than correctness and efficiency.

questions:
1. See the weaknesses mentioned above.
2. Can the authors justify the 4 selected dimensions? There are many other important aspects like security, documentation, modularity, robustness... What's the reason for not considering them?
3. How does RACE add value compared to various existing coding benchmarks, such as those discussed in "Weakness 3"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687930417

thread_type:
official_review

<thread_object>
creator: reviewer_jz3b

<content>
summary:
Prior coding benchmarks primarily assess the accuracy of generated code - whether it follows natural language instructions and produces a correct answer. This paper introduces a new benchmark called RACE, which evaluates additional aspects beyond functionality, namely readability, maintainability, and efficiency. For example, each problem description adds additional requirements like “use camel/snake case for function names”, “each line is less than 60 characters long and each function is less than 20 lines long”, “add comments for each line in each function”, “use two additional sub-functions”, “use for/while”, “make sure that the time/space complexity is X”. Evaluation results show that most LLMs cannot follow such customized requirements, and the best model o1-mini has an overall score of only 63.5.

strengths:
- The paper highlights an important problem. Readability, maintainability, and efficiency are important dimensions of code quality, yet understudied in prior benchmarks.

- The authors build their new benchmark RACE by adding additional requirements on existing, widely-adopted benchmarks (HumanEval+, MBPP+, ClassEval, LeetCode), and provide a comprehensive evaluation on 28 representative LLMs.

weaknesses:
The motivation of this paper is somewhat unclear and unconvincing. In the introduction, the authors emphasize that code readability, maintainability, and efficiency are crucial properties for deciding code quality, and that existing benchmarks fail to distinguish between high-quality code from merely correct code. This is suggesting that RACE is designed to address this challenge by assessing these readability/maintainability/efficiency dimensions. However, RACE actually does not directly measure these qualities (except for the MI metric); rather, it measures the instruction-following ability, such as “use camel case for function names” or “use two additional sub-functions”. I agree with the authors that (1) readability/maintainability dimensions are hard to capture with a single metric like accuracy, and (2) some dimensions are demand-dependent. However, it seems to me that to tackle these challenges, more diverse metrics are needed to quantify and approximate different dimensions. For instance, tools like CodeQL can analyze source code for security issues, code smell [a] is often used for maintainability, and efficiency can be assessed with time efficiency and space efficiency metrics.

The authors seem to address a different problem - whether LLMs can follow coding style instructions. However, the styles studied are often superficial (e.g., naming conventions, lines of function, comment density). Consequently, a high RACE score might indicate that an LLM is better at understanding the ~7 types of code style instructions or at following specific coding styles, rather than reflecting actual code quality. For example, the model can easily boost its score by writing dummy code comments or dummy sub-functions. 

Additionally, some test settings appear unnatural. Requiring the model to solve any problem using 1, 2, or 3 functions, for instance, may be impractical: some problems are too complex for a single function, while 3 functions may be too many for simple problems. Furthermore, as discussed in the paper, some LLMs may have an inherent preference bias towards specific styles (e.g., camel-case v.s. snake-case), and they cannot follow specific style-related instructions well, probably because of lack of instruction-tuning data in this domain. I wonder if better prompting, such as chain-of-thought or simply giving a few examples for camel-case naming, would significantly increase the IF accuracy of LLMs.

Human assessment is often essential for code quality studies, as it is very challenging to automatically evaluate source code quality. It would be beneficial to conduct a user study to verify whether RACE aligns with human preference, i.e., whether higher scores on RACE correlates with perceived better readability and quality for developers.

Lastly, I find the experimental setup of section 4.3 unconvincing. Using NL-to-code datasets like HumanEval+ as the contaminated dataset and using Magicoder-OSS-Instruct as the clean dataset may be problematic. Fine-tuning a model on such a small, domani-specific dataset (like HumanEval and MBPP) for 10 epochs would not only cause serious memorization issues, but also potentially impair the general instruction-following capabilities. This is because the instructions in HumanEval-like benchmarks usually only contain functionality requirements, and lack diversity (e.g., no coding style requirements). On the other hand, instruction-tuning datasets like Magicoder-OSS-Instruct include a wide range of instructions and potentially cover some scenarios similar to RACE. A more realistic contamination setup might involve mixing the correctness benchmark problems with instruction tuning datasets.


[a] Refactoring: Improving the Design of Existing Code.

questions:
1. Could you please clarify the motivation of this paper?
2. Could you please justify the setup for section 4.3 and address my comments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706048987

thread_type:
official_review

<thread_object>
creator: reviewer_sfyx

<content>
summary:
The paper introduces the RACE benchmark for evaluating code generation by large language models (LLMs) across four dimensions: Readability, Maintainability, Correctness, and Efficiency. 
By mimicking real-world software quality requirements, the benchmark is designed to address the limitations of current existing, correctness-focused benchmarks. 
For each dimension of evaluation, the authors provide practical metrics with specific user demands.
Via empirical evaluation of 28 representative LLMs using RACE, the authors provide insights into code generation capabilities and validate the robustness of multidimensional evaluation as an effective tool in resisting the risk of data contamination, identifying areas where LLMs struggle beyond correctness, e.g., coding style and complex instructions.

strengths:
* The RACE benchmark introduces a novel approach to code quality evaluation, reflecting real-world demands more closely (e.g., including user-specific requirements within each dimension) and advocates for more robust, practical benchmarks
* The multidimensional evaluation is sound, with well-defined metrics for each dimension, e.g., the use of established software quality metrics like the Maintainability Index is well-justified. 
* The paper conducts an extensive evaluation across a wide range of models, including both open-source and closed-source models, to provide a comprehensive view of current LLM capabilities and limitations. 
* The results support the main claims well, showing that models generally underperform in dimensions beyond correctness. The insight and highlight of several critical issues are useful, such as LLMs' biases toward certain coding styles, and performance degradation under complex requirements.
* The paper is well presented with a reasonable level of technical detail for each metric.

weaknesses:
- The claim that RACE mitigates data contamination is somewhat speculative without further investigation or deeper examination: While RACE may lessen overfitting by constraining diverse, customized requirements, the inherent data contamination issues with large code datasets remain complex.
- While the benchmark is thorough, the study lacks concrete examples and case studies to compare failed or succeeded in each dimension
- The paper focuses on code generation models using Python-specific tasks. Expanding RACE to other languages (e.g., JavaScript, Java) would allow the benchmark to better generalize and expand the applicability.

questions:
1. Regarding this statement "If these correctness-based benchmarks serve as guiding indicators and correctness alone becomes the sole criterion for driving LLM development, these models might end up memorizing the exact solutions from the training data instead of understanding the underlying principles or patterns. This overfitting implies the model may reproduce code that is highly similar to the training data during inference, leading to data leakage."? (Line 73), can you elaborate on why focusing on correctness leads to overfitting? and data leakage? assuming you provide large enough data for training and evaluation?
Any reference for these statements?

2. For the metrics:
- Could the authors provide a rationale for using MI as the primary metric for maintainability, given its known limitations? Would combining MI with additional maintainability metrics (e.g., code cohesion) offer a more comprehensive assessment?
- For the Normalized Index (NI) and proposed metrics, have you unit tested its consistency with provided complexity (or ground truth metrics)? And  How sensitive are the Normalized Indices (NIT, NIS) to minor improvements in time and space efficiency?

3. For the decline in performance with complex instructions, an alternative hypothesis could be that models lack generalization to uncommon or contradictory requirements, rather than simply not understanding multiple demands. Have the authors tested model tuning with various prompt instructions that might mitigate this issue?

4. For an alternative hypothesis, have you considered if the data contamination robustness observed could also stem from inherent limitations in LLMs’ instruction-following capabilities? This could validate whether the robustness is an incidental outcome or  a direct effect of the multidimensional approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730746325376

thread_type:
official_review

<thread_object>
creator: reviewer_arws

<content>
summary:
The paper introduces the RACE benchmark, a novel multi-dimensional evaluation framework for assessing the quality of code generated by large language models (LLMs). Unlike traditional benchmarks that focus solely on code correctness, RACE evaluates LLMs across four dimensions: Readability, mAintainability, Correctness, and Efficiency. The authors develop specific metrics and customized requirements for each dimension, conducting a comprehensive evaluation of 28 representative LLMs. Their findings reveal significant limitations in current correctness-centric benchmarks and highlight the necessity for a holistic evaluation approach to guide LLM development towards generating high-quality, user-compliant code in real-world scenarios.

strengths:
The paper addresses a critical gap in LLM evaluation by moving beyond correctness to include essential dimensions like readability, maintainability, and efficiency. The RACE benchmark is a good contribution, offering a comprehensive, demand-dependent evaluation framework that aligns closely with real-world software development needs. The empirical evaluation of a wide range of LLMs provides valuable insights into the capabilities and limitations of existing models. The paper’s methodology, which includes both static analysis and runtime monitoring, ensures robust and objective assessment, enhancing the credibility and applicability of its findings.

weaknesses:
Although the benchmark addresses critical dimensions, the paper falls short in validating its framework through real-world case studies or industry-specific applications. Real-world validation would significantly enhance the credibility and practical relevance of the RACE benchmark. Incorporating case studies or pilot projects in industry settings could provide deeper insights into its applicability and effectiveness across diverse environments. Specifically, the paper should focus on the repository-level evaluation, as generating and executing code at this level offers a more comprehensive and practical measure of dimensions like maintainability and efficiency. For instance, the use of the Maintainability Index (MI), as defined by Coleman et al., is somewhat misplaced in this context. MI was designed for evaluating entire software systems rather than simple coding tasks typical in competitive coding benchmarks.

Regarding code efficiency, the motivation behind the Normalized Index (NI) metric is insufficiently explained. The authors should refer to established methodologies, such as those in [5][6], to justify the metric and outline a robust data collection pipeline with appropriate metrics for assessing code efficiency. This would provide a more grounded and insightful evaluation framework, ensuring that the NI metric accurately reflects real-world efficiency demands.


[1] RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems, https://arxiv.org/abs/2306.03091

[2] DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories, https://arxiv.org/abs/2405.19856

[3] CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion, https://arxiv.org/abs/2310.11248

[4] On the Impacts of Contexts on Repository-Level Code Generation, https://arxiv.org/abs/2406.11927v3

[5] Learning Performance-Improving Code Edits, https://arxiv.org/abs/2302.07867

[6] PerfCurator: Curating a large-scale dataset of performance bug-related commits from public repositories, https://arxiv.org/abs/2406.11731

questions:
1) Have you considered validating the RACE benchmark through real-world case studies or pilot projects in industry settings? If so, what industries or types of projects do you envision as ideal test beds?

2) Given that the Maintainability Index (MI) was originally designed for entire software systems, how do you justify its application to individual code snippets or competitive coding benchmarks? Do you plan to develop or adapt other metrics that might be more appropriate for this scope?

3) The results indicate that LLMs struggle with complex, multi-faceted instructions. Have you explored specific training techniques or architectural adjustments that could improve these capabilities? What future directions do you see for enhancing LLMs’ instruction-following skills?

4) Since the evaluation depends heavily on user-defined requirements, what strategies do you recommend for minimizing bias or ensuring consistency in requirement selection? Would a standardized set of requirements or guidelines be feasible?
</content>
</thread_object>

</thread>


Metareview: