On the importance of assessing helpfulness of code generated with LLMs
Authors: Axel Bendavid, Orna Raz, Eitan Farchi


Reviewer paper summaries: This paper identifies issues with currently popular ways of evaluating code generation using functional correctness and execution-based evaluation. It proposes the usefulness of code as a complementary dimension that could potentially recognize LLM code which doesn't produce correct results but could be used by programmers with minor adjustments. The paper gathers annotations from a single annotator for this scale of usefulness on the HumanEval dataset and computes correlations of various evaluation metrics (reference-based and reference-free) including execution-based evaluation with the human-annotated usefulness values. The current results seem to suggest that usefulness correlates the most with execution evaluation but the correlation isn't very strong suggesting the need for the development of automated metrics for evaluating usefulness. Although the paper serves as a good motivation for the issue it highlights it is not very scientific (in the opinion of this reviewer) since the reliability of the annotation scale isn't tested (agreement between multiple annotators isn't evaluated). Thus we don't know how subjective the scale is and if the trends would be reproduced if the annotations are done by a different annotator.
Summary of strengths: - The paper identifies a pertinent issue in code generation evaluation of capturing aspects other than functional correctness, like helpfulness
- The paper identifies some limitations of execution-based evaluation especially for hard-to-execute code
- The paper elicits human annotations for the usefulness of code and does extensive evaluation of various existing code generation metrics
- The results demonstrated the need for capturing the usefulness of code and being less strict than execution based correctness to recognize utility of code
Summary of weaknesses: **Overall weaknesses:** The paper feels very incomplete despite being a short paper.

**Weaknesses by section:**
1. Introduction - 
- Just quoting Andrew Ng on the need to improve evaluation feels insufficient. Authors should summarize some reasons that Andrew Ng or others have laid out to motivate better evaluation. 
- No citations for the "LLM as a Judge (LLMaaJ)" paradigm.
- No acknowledgment of research that tries to solve the issue of excitability for complex code repos (e.g. CodeScore: https://arxiv.org/pdf/2301.09043 that learns execution, CodeBenchGen: https://arxiv.org/abs/2404.00566 that can create executable benchmarks from repository code and REPOEXEC: https://arxiv.org/html/2406.11927v2 (a repository level code execution benchmark)). I think REPOEXEC was after the ARR deadline but CodeBenchGen and CodeScore aren't.
- Also no mention of this paper (https://arxiv.org/abs/2210.16494) on aligning code generation evaluation with human values along with functional correctness. They had a notion of usefulness based on the number of edits from a ground truth solution (how much effort is required to get a correct solution). It would be interesting to compare that with your 4-value Likert-like scale. 
- _a lightweight statistically valid approach to determine the best evaluation metrics for specific tasks and domains, demonstrated through experimental results_ - only HumanEval Python data is used so it is effectively just one kind of domain and task 
- Future work should ideally be a separate section. Also various "quality dimensions" are mentioned like usefulness, conciseness, and consistency but only usefulness is defined. This makes it seem like you haven't defined/developed these aspects.

2. Method - 
- 2.2: no motivation for why it is important for the means of randomly sampled subsets to be normally distributed. It seems the authors are conflating the fact that the means of samples greater than n (n = 30) would follow a normal distribution with the fact that the sample mean would be representative of the population mean (which seems more reasonable for representativeness?). I'm not sure what the intent was with the CLT-based representative sample selection strategy. 
- 2.2: _to turn the record set into a more representative set of actual usage of the solution_ the writing is a bit ambiguous as to whether you did this or not. But I'm assuming you didn't as point 2 acknowledges that external libraries aren't a part of HumanEval. 

3. Experiment - 
- The HumanEval dataset is already pretty small so I wonder why you sampled a representative set.
- **Using a single annotator is a major limitation**. We don't know if your scale is reliable and reproducible (will two different people with the same set of instructions about the scale assign the same scores). This in itself in my opinion makes the paper too incomplete to be accepted. I would also take any conclusions with a huge grain of salt.
- CodeScore can function as the alternative metric that you are searching for (as mentioned in Table 1)
Comments, suggestions and typos: Suggestions:
For the list within section 2 (Method), the list items after "Our methodology comprises of four steps:" have too much item space between them.

Grammar:
In the introduction "First (our current work), have a good estimate" -> "First (our current work), **has** a good estimate"
Overall assessment scores: 1