Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
Authors: Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun


Reviewer paper summaries: This paper addresses a major limitation of popular code generation benchmarks - fixating on the functional correctness of generated code (usually based on execution-based evaluation) instead of a holistic focus on a variety of dimensions of code quality like readability, maintainability, and efficiency. They address this issue by creating a Python coding benchmark called "RACE" from existing datasets like MBPP+, HumanEval+, LeetCode, and ClassEval for code generation with given user requirements related to the additional dimensions of code quality (e.g. a specific time complexity when it comes to efficiency). They propose a variety of automated metrics based on static and runtime analysis of code to operationalize the dimensions (e.g. normalized index and maintainability index for efficiency and maintainability). They use their proposed RACE benchmark to evaluate 16 representative LLMs of code and the results show that current LLMs can't satisfy user requirements along the proposed dimensions which are required in software engineering, readability of code is a critical indicator of overall quality, and LLMs exhibit idiosyncrasies/preferences when it comes o coding style.
Summary of strengths: - The paper addressed an important challenge in the field of code generation - evaluation of code quality beyond correctness which is relevant for practical applications like software engineering and could capture notions like code helpfulness which are more important for user satisfaction from code generation models compared to functional correctness (https://arxiv.org/abs/2210.16494).
- The paper proposes a user requirement-based paradigm of code generation to allow for the measurement of each dimension of code quality which represents a more realistic scenario of code LLM usage.
- The paper proposes automated metrics for measuring each of the dimensions which makes the evaluation paradigm scalable.
- The authors are careful enough to remove any existing user requirement instructions from the datasets used to build the benchmark to avoid any accidental conflicts in the requirements during the addition of dimension-specific requirement instructions.
- The visualization in Figure is quite nice and a great way to visualize the various dimensions making it a great tool for model selection based on user requirements.
- The findings of the paper are quite interesting, especially the existence of coding preferences in LLMs.
Summary of weaknesses: **Weaknesses:**
- **There are several concerns associated with the maintainability index**: https://avandeursen.com/2014/08/29/think-twice-before-using-the-maintainability-index/ so I would recommend the authors to reconsider its usage since it's not very scientifically motivated.
- Code readability dimensions/indexes are quite subjective. I could argue that restricting code lengths could mean writing code with uninformative variables or hard-to-understand one-liners that make the code more compact but are less straightforward to follow. An argument related to display issues is not sufficient because they could be countered using formatting tools and linters that can prettify the code. Similarly number of code comments and the level of detail are also not very straightforward. For example, line-level comments can make code less maintainable because any code change would require changing a bunch of comments to ensure the faithfulness of comments to the code.  Additionally, the exact heuristics used are not mentioned which makes it hard to judge the usefulness of the scores.
-  As hinted in my previous criticism the dimensions of code quality are not completely independent from each other. Maintainability and readability could have a decent amount of overlap. 
- Since the metrics rely heavily on run-time and static analysis they will need significant re-implementation to deal with programming languages other than Python like Java, Javascript -and C++ which are more common in software engineering.
- Additionally, the metrics related to efficiency and correctness that use run-time analysis are limited by code that can be executed. Code that requires downloading external libraries, setting up an environment, access to external data, sandboxing, etc. might be challenging to evaluate using their framework.
- The evaluation method would increase the evaluation time and cost since more LLM calls are needed to generate code for specified instructions which also needs to be executed in addition to the code generated without user requirements.
- The evaluation dataset for code efficiency with 49 LeetCode problems seems a bit small.

**Questions for the authors:**
- In section 4.4 (Preference bias in code LLMs): "This issue may be more pronounced in programming languages like C++ and Java, where there is no standardized code style." - don't you think it could end up being the opposite because the LLMs might come across a variety of coding styles in the training data. I speculate the limitations in Python stem from the ossification of coding style among Python programmers. 

I'm sorry if any of these questions are addressed by a part of the paper that I might've missed.
Comments, suggestions and typos: .
Overall assessment scores: 4
Metareview score: 3
Metareview: This paper critiques current code generation benchmarks for focusing primarily on functional correctness. It emphasizes the need to evaluate other aspects of code quality: readability, maintainability, and efficiency. The authors introduce a new benchmark called "RACE," created from existing datasets, which incorporates these additional evaluation criteria. They propose simple statistical metrics to measure each factor, such as the Maintainability Index and a new Normalized Index. The authors analyze these metrics and test how user instructions influence generated code. Experimental results show that while some LLMs perform well, they struggle to meet all user requirements related to the proposed quality dimensions. The findings indicate that code readability is crucial for overall quality and reveal biases in coding style among various LLMs.
Summary of reasons to publish: 1. The paper introduces a user requirement-based paradigm for measuring various dimensions of code quality.
2. It proposes automated metrics for each dimension, enhancing the scalability of the evaluation paradigm.
3. The evaluation criteria add value to benchmarking code-generating LLMs, addressing often overlooked aspects like readability.