MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

## Abstract

Abstract Recent advancements in large language models (LLMs) have greatly improved code generation, specifically at the function level. For instance, GPT-4 has achieved an 88.4% pass rate on HumanEval. However, this draws into question the adequacy of existing benchmarks in thoroughly assessing function-level code generation capabilities. Our study analyzed two common benchmarks, HumanEval and MBPP, and found that these might not thoroughly evaluate LLMs’ code generation capacities due to limitations in quality, difficulty, and granularity. To resolve this, we introduce the Mostly Hard Python Problems (MHPP) dataset, consisting of 140 unique human-curated problems. By focusing on the combination of natural language and code reasoning, MHPP gauges LLMs’ abilities to comprehend specifications and restrictions, engage in multi-step reasoning, and apply coding knowledge effectively. Initial evaluations of 22 LLMs using MHPP showed many high-performing models on HumanEval failed to achieve similar success on MHPP. Moreover, MHPP highlighted various previously undiscovered limitations within various LLMs, leading us to believe that it could pave the way for a better understanding of LLMs’ capabilities and limitations. Dataset and code are available at https://github.com/SparksofAGI/MHPP .

## 1 Introduction

Large language models (LLMs) have recently driven striking performance improvements across various tasks (Ouyang et al., ; Touvron et al., ; OpenAI, ). Recent models such as Code Llama (Rozière et al., ), and GPT-4 (OpenAI, ), have been successful in demonstrating their efficacy in code-related tasks from program repair (Haque et al., ; Jin et al., ) to automated testing (Lemieux et al., ; Schäfer et al., ). LLMs are utilized to develop innovative tools aimed at aiding programmers to write code more efficiently (Chen et al., ).

Code generation is a key area for evaluating LLMs’ capabilities. Code generation broadly spans converting natural language prompts into executable code, not limited to predefined templates such as function signatures and docstrings. This process can range from pure text descriptions to complete code generation, emphasizing the versatility and adaptability required for LLMs.
Our focus is on Function-Level Code Generation.
An example is illustrated in LABEL:fig:enter-label.
It emphasizes the translation of natural language into functional code, underlining natural language comprehension’s importance for creating accurate programming constructs.
Benchmarks like HumanEval (Chen et al., ) and MBPP (Austin et al., ) stand out in assessing these models, showcasing LLMs’ strengths in code generation through their understanding of natural language and generation abilities. For instance, GPT-4 impresses with an 88.4% pass rate on the first try, and the open-source WizardCoder achieves a 79.9% pass rate on the HumanEval benchmark. This suggests that current models can rival or even surpass human performance in code generation tasks.

However, on existing benchmarks, performance differences between models are insignificant - all achieve high pass rates. We thus raise two concerns: 1) Basic datasets lack discriminative power to distinguish model capabilities, making it difficult to assess their relative strengths and weaknesses. 2) High overall pass rates on existing tasks alone cannot determine if models have truly mastered functional programming competency and encoding skills to address diverse challenges.
To answer these questions, we conducted detailed experiments with the strongest code models on the market, including closed-source models like GPT-4, GPT-3.5, and open-source models like WizardCoder and DeepSeek Coder, using the HumanEval and MBPP benchmarks.
Results are displayed in LABEL:fig:error.
Our error analysis revealed that different models make similar mistakes on the same problems, highlighting corresponding challenges.

Through an extensive manual analysis, we identified 7 main challenges in code generation tasks, leading to the introduction of the Mostly Hard Python Problems (MHPP) dataset. MHPP consists of 140 unique, manually created Python programming problems, each supplemented by unit tests.
MHPP focuses on comprehensively evaluating LLMs’ abilities to tackle various challenges in code generation.
This includes handling variance in natural language inputs, understanding newly defined contexts, demonstrating commonsense, dealing with edge cases, following complex instructions, using mathematical and algorithmic knowledge, and showing familiarity with coding principles. It is important to note that each challenge within MHPP necessitates different degrees of natural language comprehension and code reasoning abilities.

We extensively evaluated 22 LLMs on MHPP, revealing many previously undiscovered limitations and different weaknesses across models when addressing various challenges involved in code generation tasks. Notably, the models struggled the most with challenges that required advanced algorithmic reasoning. Our comprehensive experiments demonstrate that MHPP can effectively test model performance against diverse code generation challenges. We hope MHPP can serve as a stepping stone for a better understanding of LLM capabilities and limitations to advance code generation. particularly in the domain of algorithmic reasoning.

## 2 Dataset Analysis

In this section, we carry out a comprehensive manual analysis of two standard benchmarks: MBPP and HumanEval along multiple axes.
Our findings indicate that these benchmarks may not fully assess LLMs’ code generation capacities due to LLMs’ rapid development.

### 2.1 MBPP

The analysis of the MBPP test set revealed three main issues. Firstly, data contamination was identified as a significant problem. Through manual inspection, we found that many instances appeared on the open-access websites, such as GeeksforGeeks(^1^11[https://www.geeksforgeeks.org/](https://www.geeksforgeeks.org/)).
To further investigate this issue, we calculated the contamination rate using the tool (Li, ), 65.4% of instances in the test set were found to be contaminated.
For more details refer to [Appendix B](#A2).
This issue may be attributed to the annotation process of MBPP, which allows crowd workers to use internet references without implementing measures to filter out questions collected directly from websites. The presence of contaminated data enables models to “cheat” by memorizing test data rather than demonstrating genuine generalization, thus distorting model comparisons and undermining the reliability of benchmarks (Jacovi et al., ; Sainz et al., ).

Additionally, upon conducting an error analysis based on the best-performing model (e.g. GPT-4), we found that 18.82% of errors identified were attributed to the quality of the test instances in MBPP. Specifically, these errors were categorized into two types: 10.59% of the errors were associated with unclear problem descriptions, while 8.23% were caused by instances lacking necessary constraints or containing incorrect test cases. A more detailed analysis, along with specific cases, can be found in Appendix [H](#A8). Lastly, the problems within MBPP primarily center around basic code generation, covering tasks that entail simple arithmetic or standard library usage. The length of the natural language descriptions averages about 15.7 words per sentence, with predominantly unbalanced types, wherein 77% were related to mathematical and list as shown in Figure [3](#S2.F3). The imbalance in problem types and difficulty levels may not thoroughly assess the capabilities of LLMs, particularly given the rapid development of LLMs.

Figure: (a) Challenges in HumanEval
Refer to caption: /html/2405.11430/assets/x1.png

### 2.2 HumanEval

We conduct an extensive error analysis for 4 best-reported LLMs on HumanEval, including GPT-4 (OpenAI, ), GPT-3.5 (OpenAI, ), DeepSeekCoder (DeepSeekAI, ) and WizardCoder (Luo et al., ) as depicted in Figure LABEL:fig:error.
We analyze the errors made by LLMs on HumanEval and categorize the code generation challenges that led to these mistakes into 7 types:

Distraction: The description is lengthy and contains redundant information. To address this challenge, LLMs need to extract essential information needed for accurate code generation.

Redefinition: The description introduces new concepts or operational rules, presenting a counterfactual scenario with corresponding explanations. LLMs need to comprehend this newly introduced context for accurate code generation.

Shortcut: This challenge requires LLMs’ unconventional thinking, solving such problems often involves concise solutions derived from logical reasoning, lateral thinking, and a grasp of knowledge including mathematics and game theory.

Commonsense: Understanding the problem relies on commonsense knowledge not explicitly explained in the description. Commonsense involves universally understood facts for humans, such as temporal, spatial, and geometric knowledge. LLMs need a solid grasp of commonsense to interpret the context and then generate code.

Cornercase: This challenge demands thorough thinking of the problem, paying close attention to implicit boundary conditions that could affect the outcome. LLMs need to consider all the corner cases for correct code generation.

Complexity: The description contains multiple constraints or requires executing multiple steps to reach a solution. This complexity necessitates advanced logical reasoning or complex instruction following capabilities for code generation.

Codesense: This challenge requires a deep understanding of the Python language and broader programming knowledge, including familiarity with specific Python packages and the parameters needed for function calls.

In addition to seven identified challenges, we incorporated a Basic category in HumanEval that necessitates elementary programming abilities, such as string manipulation or arithmetic operations. Our analysis reveals an imbalance in HumanEval’s challenge and problem type distribution, with Basic and Codesense problems comprising 17.7% and 20.1% respectively, as depicted in [Figure 3a](#S2.F3.sf1) and further illustrated in Figure [3](#S2.F3). Codesense, demanding minimal Python proficiency, along with Basic, exhibits significantly lower error rates compared to other categories. To sum up, both MBPP and HumanEval face challenges concerning data contamination, quality, distribution, and difficulty levels, potentially affecting the reliability of benchmarking processes and the precise evaluation of LLMs’ code generation capabilities.

## 3 Benchmark Construction

To delve deeper into the capabilities and limitations of LLMs beyond the basic code generation capabilities identified by MBPP and HumanEval, we have created a unique code generation benchmark Mostly Hard Python Problems (MHPP). This benchmark comprises expert-curated problems tailored specifically for the seven challenges we identified in code generation. Note that using HumanEval as a starting point may limit the coverage of problem types and error patterns.
Therefore, we actively sought to generalize the problem types and address more realistic and challenging error patterns in the creation of MHPP. We refer readers to Appendix  [C](#A3). Our annotation team includes 10 members, all of whom possess either a master’s or a Ph.D. degree in computer science. To ensure the quality of our dataset, three members serve as meta-annotators. Based on the seven challenges, the annotators were tasked with defining the problem statement for each challenge, creating a single, self-contained Python function to solve the given problem, and developing test cases to validate the semantic correctness of the function, as detailed in Section [3.1](#S3.SS1). Additionally, the annotators were required to provide a ground-truth solution that successfully passed all the proposed test cases.

Table: Table 1: Detailed statistics of MHPP. Avg. Input Words represents the average number of words contained in the input, Avg. Code Lines means the average number of lines in code solutions and Avg. Tests represents the average number of test cases per problem. Reasoning level indicates the level of reasoning difficulties for solving the specific challenge.

In defining the problems, annotators were instructed to formulate descriptions clear and detailed enough to allow for the translation of these descriptions into code by a human, without further clarification. To maintain the originality and integrity of MHPP, annotators were strictly prohibited from directly copying problems from publicly accessible websites, or employing simple modifications to existing problems, such as synonym replacements or paraphrasing, as outlined in Section [3.2](#S3.SS2).

### 3.1 Challenge-Specific Annotation

We provide guidelines catered to the diverse requirements of annotating different challenges.

Distraction: Annotators are required to create elaborate natural language descriptions that incorporate redundant information. These descriptions should exceed 200 words and introduce distractions.

Redefinition: Annotators are required to introduce new concepts or operational rules, effectively creating counterfactual scenarios. Each problem should introduce more than one new concept along with comprehensive explanations.

Shortcut: Annotators are required to craft problems that permit concise solutions by lateral thinking, or applying knowledge from mathematics and game theory.

Commonsense: Annotators are required to construct problems that are grounded in foundational commonsense concepts. These problems should not include explicit explanations of the involved commonsense principles, and more than one concept should be featured.

Cornercase: Annotators are required to write problems with solutions that need to consider more than 1 corner case.

Complexity: Annotators are required to develop problems that have more than 3 operational steps or hops of reasoning. An example would be a problem that necessitates sorting a list, extracting maximum and minimum elements, and then calculating the difference between these elements.

Codesense: Annotators are required to craft problems that necessitate the utilization of more than 1 specific Python package, both internal and external, such as RE and Numpy.

### 3.2 Quality Assurance

To ensure the quality of MHPP, we initiated a comprehensive two-phase quality assurance process. Our primary goal in the first phase is to eliminate any risk of data contamination that may arise from the inclusion of problems that have previously appeared on open-access websites. To achieve this, we tasked meta-reviewers with meticulously searching the Internet to ensure none of the problems selected were already publicly available. Additionally, we employed a contamination detector (Li, ), to confirm a 0% contamination rate, resulting in the exclusion of 6 problems identified at this stage. We then asked the annotators to annotate another 6 problems until all of the problems met the requirements. Progressing to the second phase, our focus shifted towards ensuring that each problem rigorously meets the specific criteria for the respective challenges. This entailed a detailed review of every aspect of the problem, including the natural language description, the reference solution, and the test cases, conducted by a panel of three meta-annotators. To guarantee consistency and accuracy, we adopted an iterative approach wherein annotators were tasked with addressing and rectifying any issues flagged by the meta-reviewers until unanimous approval was obtained.

### 3.3 Dataset Statistics

Detailed statistics of MHPP are outlined in Table [1](#S3.T1). The total number of our dataset is 140 and each challenge category contains 20 questions. A significant observation is that the average problem in MHPP contains 150.2 words and the corresponding solutions span across 12.2 lines of code. This indicates a considerable increase in verbosity and code complexity when compared to benchmarks such as MBPP and HumanEval. Furthermore, MHPP surpasses these benchmarks in the number of test cases, with an average of 13.5 test cases per problem—higher than MBPP’s 3.0 and HumanEval’s 7.2. Further comparisons can be found in Appendix [A](#A1). Crucially, the design of MHPP specifically addresses more nuanced challenges and diverse context formats, a distinction not observed in other datasets. For instance, challenges categorized under the Distraction and Complex categories are marked by significantly longer descriptions, posing unique challenges in context comprehension. Conversely, problems falling under the Shortcut class feature notably fewer lines of code in their solutions, highlighting challenges in achieving concise problem solutions.

As detailed in Table [1](#S3.T1), our analysis of the top 5 distribution of problem types underscores the unparalleled diversity in MHPP, in contrast to MBPP and HumanEval where three types predominantly emerge. This diversity extends to the varied problem types observed across different challenges; for example, while dynamic programming is a prevalent theme in the Complex category, it appears less frequently in the Redefinition and Cornercase categories, showcasing the diverse range of challenges encapsulated within MHPP.

MHPP spans a wide range of complexity levels, testing the reasoning capabilities of LLMs to varying degrees. Commonsense and Codesense challenges involve basic logical operations, such as identifying concepts and patterns, applying factual and programming knowledge, and drawing simple inferences. Distraction, Redefinition, and Cornercase challenges demand complex cognitive processes. These include analyzing the docstring, evaluating the context, and forming conclusions based on multiple conditions. Shortcut and Complex challenges necessitate even more advanced reasoning, involving abstract thinking, critical analysis, and optimization under various constraints. In essence, MHPP provides a spectrum of complexity, testing LLMs’ ability to perform natural language and algorithmic reasoning at different levels.

## 4 Experiment

**Table 2: LLMs’ performance on MHPP in terms of pass@1 and pass@5 scores. We pinpoint top performers in open-source LLMs based on pass@1 and pass@5 scores. The best models are highlighted in bold, while those in second place are underscored, including ties. The performance of LLMs on MHPP using greedy-search decoding can be seen in Appendix [E](#A5).**
| Model | Distraction | Redefinition | Shortcut | Commonsense | Cornercase | Complex | Codesense | Total |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $k$=1 | $k$=5 | $k$=1 | $k$=5 | $k$=1 | $k$=5 | $k$=1 | $k$=5 | $k$=1 | $k$=5 | $k$=1 | $k$=5 | $k$=1 | $k$=5 | $k$=1 | $k$=5 |
| Closed-Source LLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4-1106-Preview | 45.0 | 58.3 | 61.6 | 73.4 | 32.4 | 49.3 | 55.4 | 78.2 | 44.3 | 61.9 | 37.5 | 64.2 | 58.2 | 74.6 | 47.8 | 65.7 |
| GPT-3.5-Turbo | 28.2 | 38.8 | 33.7 | 45.8 | 28.6 | 47.2 | 37.6 | 57.9 | 25.3 | 39.2 | 15.9 | 29.1 | 44.5 | 63.5 | 30.6 | 45.9 |
| Open-Source LLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Phi 1-1.3B | 3.0 | 9.5 | 15.7 | 25.7 | 8.8 | 15.5 | 11.2 | 18.2 | 8.3 | 18.1 | 6.9 | 11.3 | 25.7 | 34.8 | 11.4 | 19.0 |
| Phi 1.5-1.3B | 0.9 | 3.7 | 5.5 | 12.5 | 2.9 | 8.9 | 9.0 | 17.8 | 5.4 | 11.2 | 0.6 | 2.8 | 13.9 | 24.5 | 5.5 | 11.6 |
| Phi 2-2.7B | 7.1 | 16.3 | 9.7 | 22.3 | 4.6 | 14.5 | 13.9 | 27.0 | 8.2 | 15.5 | 1.3 | 5.5 | 16.2 | 33.2 | 8.7 | 19.2 |
| Mistral-7B Ins | 3.0 | 11.1 | 7.5 | 16.8 | 2.8 | 9.5 | 7.1 | 14.4 | 9.0 | 14.7 | 0.8 | 3.4 | 13.0 | 26.3 | 6.2 | 13.7 |
| LLaMA2 Chat-7b | 0.0 | 0.0 | 2.8 | 7.9 | 0.6 | 2.5 | 1.8 | 5.5 | 6.8 | 9.5 | 0.3 | 1.5 | 5.5 | 9.7 | 2.6 | 5.2 |
| LLaMA2 Chat-13b | 0.0 | 0.0 | 1.0 | 3.9 | 0.7 | 2.7 | 5.0 | 10.2 | 9.5 | 10.5 | 0.3 | 1.4 | 8.3 | 11.6 | 3.5 | 5.8 |
| CodeLLaMA-7B | 1.4 | 6.0 | 2.6 | 10.4 | 0.7 | 3.3 | 7.0 | 18.3 | 5.7 | 11.9 | 0.3 | 1.3 | 6.5 | 15.8 | 3.4 | 9.6 |
| CodeLLaMA-13B | 2.2 | 9.2 | 3.1 | 12.3 | 1.6 | 7.0 | 6.0 | 17.7 | 5.6 | 14.8 | 0.8 | 3.7 | 7.7 | 19.4 | 3.9 | 12.0 |
| CodeLLaMA-34B | 7.1 | 20.7 | 15.4 | 34.2 | 3.8 | 13.8 | 21.2 | 43.4 | 10.5 | 23.7 | 1.5 | 6.6 | 15.0 | 33.7 | 10.6 | 25.1 |
| CodeLLaMA-7B Ins | 8.2 | 20.1 | 9.7 | 17.8 | 8.5 | 15.0 | 13.2 | 23.5 | 14.7 | 19.1 | 2.2 | 7.2 | 14.2 | 25.7 | 10.1 | 18.4 |
| CodeLLaMA-13B Ins | 11.0 | 22.2 | 11.9 | 17.2 | 3.8 | 9.9 | 15.3 | 26.1 | 13.1 | 18.9 | 1.1 | 4.7 | 19.4 | 26.4 | 10.8 | 17.9 |
| CodeLLaMA-34B Ins | 6.0 | 15.5 | 11.7 | 17.3 | 7.2 | 11.3 | 23.0 | 35.8 | 9.3 | 14.4 | 0.8 | 3.5 | 17.0 | 23.7 | 10.7 | 17.4 |
| CodeLLaMA-7B Py | 1.2 | 5.5 | 3.0 | 11.8 | 0.6 | 2.6 | 6.9 | 18.6 | 5.4 | 11.8 | 0.4 | 2.0 | 4.9 | 12.0 | 3.2 | 9.2 |
| CodeLLaMA-13B Py | 1.9 | 7.4 | 6.4 | 18.5 | 1.2 | 5.2 | 9.9 | 23.5 | 6.3 | 12.9 | 0.6 | 2.9 | 9.8 | 22.1 | 5.2 | 13.2 |
| CodeLLaMA-34B Py | 8.0 | 24.2 | 13.6 | 32.4 | 2.5 | 10.5 | 19.6 | 43.4 | 9.9 | 20.6 | 2.0 | 9.0 | 14.2 | 32.6 | 10.0 | 24.7 |
| DeepSeek-1.3B | 11.2 | 22.9 | 12.5 | 24.9 | 7.3 | 18.2 | 21.8 | 36.7 | 11.7 | 19.3 | 0.7 | 3.3 | 15.2 | 33.2 | 11.5 | 22.7 |
| DeepSeek-6.7B | 17.9 | 33.6 | 29.6 | 43.2 | 15.5 | 30.8 | 36.5 | 60.8 | 22.7 | 37.1 | 8.3 | 25.0 | 36.8 | 60.1 | 23.9 | 41.5 |
| DeepSeek-33B | 26.8 | 39.2 | 35.4 | 52.9 | 20.6 | 38.6 | 51.8 | 69.3 | 31.3 | 45.9 | 14.6 | 34.7 | 39.2 | 64.3 | 31.4 | 49.3 |
| WizardCoder-33B | 25.6 | 39.9 | 34.5 | 53.7 | 19.8 | 36.3 | 48.5 | 67.5 | 30.6 | 44.0 | 14.5 | 37.7 | 39.8 | 62.0 | 30.5 | 48.7 |
| WizardCoder-34B Py | 12.6 | 28.3 | 21.4 | 38.2 | 7.1 | 16.9 | 20.2 | 41.1 | 11.9 | 25.6 | 3.3 | 14.0 | 23.8 | 45.1 | 14.4 | 29.9 |

### 4.1 Setup

Following prior works (Chen et al., ; Nijkamp et al., ), code generation is conducted under the setting of greedy-search and sampling decoding with a temperature of 0.7, which are evaluated with unbiased versions of pass@1 and pass@5 scores, respectively.
We examined 22 LLMs on MHPP to demonstrate a comprehensive study, including the open-sourced LLMs such as CodeLLaMA (Rozière et al., ), DeepSeek (DeepSeekAI, ) and WizardCoder (Luo et al., ). GPT-4 (OpenAI, ) and its predecessor are also evaluated. Each model is prompted with “Write a Python function according to the function name and the problem description in the docstring below. [function definition with docstring]”, while all finetuned LLMs are equipped with the additional instruction template used during their specific finetuning.
To carry out an in-depth investigation of LLMs’ capability of code generation and the effectiveness of MHPP, three research questions are naturally raised:

RQ1 How do open-sourced coding models compare to proprietary models like GPT-3/GPT-4 in their ability to generate high-quality code?ration performance? ([Section 4.2](#S4.SS2))

RQ2 What weaknesses do even the most advanced models still exhibit? ([Section 4.3](#S4.SS3))

RQ3 How well does performance on MHPP correlate with performance on the existing HumanEval benchmark for evaluating code generation capabilities? ([Section 4.4](#S4.SS4))

### 4.2 Main Results

Open-sourced LLMs are impressive, however still fall short of the performance compared with GPT-4. Table [2](#S4.T2) show that there is a large gap between the performance of open-sourced coding models and all the other baselines, while the open-sourced LLMs exhibit varying quality on the response of MHPP. Specifically, despite the introduction of supervised instruction tuning and the adaptation of Python coding, most CodeLLaMA models are still inadequate to generate acceptable responses, as Phi-1 and Phi-2 demonstrate comparable performance with nearly 30x smaller sizes. However, the Phi series does not deliver the benefits of scaling up the model size, as there is little correlation between the model size and the ability of code generation. In contrast, significant improvements accompany the up-scaling of the model size on the DeepSeek and WizardCoder series. Surprisingly, DeepSeek-Instruct 33B and WizardCoder-33B-v1.1 achieve 31.4 and 30.5 pass@1 scores, respectively, which is better than or on par with the GPT-3.5 (30.6), demonstrating their potential to shape the future of open-sourced coding LLMs. However, although their performance is comparable to GPT-3.5, a huge performance gap between them and GPT-4 still exists. With a 47.8 pass@1 score, GPT-4 leaves it without any competitors in the field of code generation. This suggests that the development of open-sourced coding LLMs still faces considerable challenges and requires further exploration. Furthermore, we extend our research beyond Python by translating MHPP’s problems and test cases into Java and C++. The results of GPT-4’s performance on these languages are in Appendix [D](#A4). To help the community further improve performance on fine-grained code generation tasks, we have devised a set of potential strategies tailored to each category of challenges presented in MHPP, as detailed in Appendix [F](#A6).

### 4.3 Results on Different Types of Challenges

Challenges in MHPP are still hard even for top-performance LLMs. Despite the impressive performance compared with all the other baselines, GPT models’ error rates are still unignorable. Figure LABEL:fig:mhpp-error-type shows that MHPP struck GPT models on all fronts, as GPT-3.5 performed poorly on every subject of MHPP, getting $84.1\%$ error rate in the worst category, and over $55\%$ even in the best one. Besides, although GPT-4 outperforms its predecessor on all subjects, there is still a long way from being able to fully master problems in MHPP, especially for shortcut questions, which happened to be among the least represented in HumanEval. The experimental results indicate that MHPP provides a comprehensive overview of the LLMs’ capability to generate codes.

### 4.4 Correlation between MHPP and HumanEval

MHPP is highly correlated with the HumanEval, but more challenging and representative. Following the CRUXEval (Gu et al., ), we present the correlation between the HumanEval and MHPP in Figure LABEL:fig:correlation. GPT-4 is beyond all the disputes the finest LLMs in code generation, followed by DeepSeek, GPT-3.5 and WizardCoder. It is worth noting that a bunch of model families benefit from scaling up the model size, especially the DeepSeek with an incredible performance-to-size ratio. Most of these findings are strongly correlated with the results on HumanEval. However, when comparing the CodeLLaMA-python with the CodeLLaMA-instruct, it is unexpected that the instruction-finetuned version outperforms the python-variants on MHPP, while contradicting the results on HumanEval. By inspecting all the responses generated from these two series, we attribute this phenomenon to the poor generalization capability of the python-variants, as over half of the generated responses are almost the same as the instruction-finetuned CodeLLaMA, yet the variable names of each function didn’t match those in the MHPP but were the same as in the training corpus, see Figure [7](#A10.F7) in Appendix [H](#A8). Consequently, MHPP is highly correlated with the HumanEval for the most part. However, MHPP can reflect the model’s performance more truthfully in complex scenarios.

## 5 Analysis

### 5.1 Performance Sensitivity to Input Example

Our exploration of the GPT-4’s performance sensitivity revealed two significant results/: (a) model performance is insensitive to the number of examples (if it has), but the performance is significantly low when there are no examples. (b) the variability introduced by different input examples affects synthesis performance.

Number of Examples in Docstring: We constructed zero to four-shot settings. Without any demonstrations, only language specifications in the docstring led to the weakest performance whose pass@1 equals 46.4%, it has a huge gap between that in the one-shot scenario (pass@1 is 54.3%). Interestingly, we noted a peak in performance at two examples (pass@1 is 55.0%), indicating that a moderate number of examples might strike the best balance for guiding the model without risking overfitting (Austin et al., ), which can degrade performance, for example, pass@1 are 52.9% and 51.4% when given three or four examples in prompt, respectively.

Different Input Example: We evaluated the GPT-4 model’s performance sensitivity to the nature of the single input examples used. As evident from our analysis, the specific input examples significantly influence the model’s problem-solving success rate, averaging around 50.7% with a variability of ±3.6%. The substantial fluctuation in performance based on the input example choice suggests that the model’s reasoning ability can be highly context-dependent.

Figure: (a) Error in the Commonsense challenge.
Refer to caption: /html/2405.11430/assets/x4.png

### 5.2 Case Review

In this section, we reviewed the GPT-4’s failures to see if, for a particular problem, the model indeed failed to solve it due to the specific challenge we set for the problem. Two examples are shown in Figure [6](#S5.F6), we refer the reader to Appendix [J](#A10) for more whole examples. From these examples, the rationality of the challenge classification can also be confirmed.

Figure [6a](#S5.F6.sf1) shows one problem with “Commonsense” as its challenge and model’s solution. More specifically, this problem concerns the model’s understanding of space or orientation. Only people who are walking toward each other will meet, yet the model mistakenly believes it also needs to calculate for people moving in opposite directions. This indicates that the model lacks real-world spatial concepts.

The problem in Figure [6b](#S5.F6.sf2) addresses the challenge of multiple constraints - “Complex”. At the position marked pale blue, the model knows it should use index 3 to retrieve the fourth number from a Python array. However, for those parts marked by the color pink, even though the model claims in the comments that it will operate on the fourth number, it still uses 4 as the index. Therefore, as the number of constraints increases, the model commits errors that would not occur under fewer constraints.

## 6 Related Work

### 6.1 LLMs for Code

The burgeoning interest in LLMs for code has coincided with the profusion of openly available code repositories and the pressing need to enhance the productivity of software developers. Initial models predominantly focused on code generation tasks have included CodeT5 (Wang et al., ), AlphaCode (Li et al., ), CodeGen (Nijkamp et al., ), InCoder (Fried et al., ), StarCoder (Li et al., ), SantaCoder (Allal et al., ) and DeepSeek Coder (DeepSeekAI, ), all of which were trained on code. Contrastingly, models such as Codex (Chen et al., ) and CodeLLaMA (Rozière et al., ) represent a subsequent stride, having been fine-tuned from foundation models (Brown et al., ; Touvron et al., ). The evolution continued as LLMs leveraged instruction-like datasets derived from GPT (OpenAI, ; ) for fine-tuning. Among these, WizardCoder (Luo et al., ), Phi-1 (Gunasekar et al., ), and Phi-1.5 (Li et al., ) are notable examples. Across various coding applications, these code LLMs have set new standards of excellence, showcasing their prowess in domains including program repair (Haque et al., ; Jiang et al., ), automated testing (Lemieux et al., ; Deng et al., ), code translation (Rozière et al., ; Ahmad et al., ), type prediction (Mir et al., ; Wei et al., ), and code summarization (Hasan et al., ; Ahmed & Devanbu, ).

### 6.2 Code Generation Benchmarks

Code generation (Chen et al., ; Austin et al., ) has emerged as a vital domain for evaluating LLMs, where models generate code snippets based on natural language descriptions, often given in the form of docstrings. Creating datasets for this task is challenging, leading most efforts to source natural language and code pairs from the Internet (Hendrycks et al., ; Li et al., ; Chandel et al., ; Jain et al., ; Shinn et al., ) or use distant supervision (Agashe et al., ). For instance, APPS (Hendrycks et al., ) compiles questions from open-access coding portals like Codeforces and Kattis, covering a wide difficulty range. Similarly, CodeContests (Li et al., ) and LeetcodeHard (Shinn et al., ) draw problems from specific platforms, enriching the diversity and challenge of datasets. However, the training of LLMs on vast repositories, including GitHub, poses a risk of including solutions to these problems, thereby emphasizing the importance of hand-written sets like HumanEval (Chen et al., ) and MBPP (Austin et al., ) for accurate benchmarks. These datasets, based entirely on human-written questions, are pivotal despite their focus on simpler functions, highlighting a need for advancing benchmarks to match the growing capabilities of LLMs. More code generation benchmarks are discussed in Appendix [A](#A1)

## 7 Conclusion

In this work, we construct the MHPP benchmark comprising 140 unique, manually created Python problems. The prime focus of MHPP is the semantic grounding of code generation, effectively measuring LLMs’ competence in comprehending detailed specifications and restrictions in natural language descriptions, undertaking complex reasoning, and employing code knowledge to facilitate the desired functionality. Upon applying MHPP, we observe that the most powerful LLM still struggles on this challenging benchmark. We hope MHPP can shed light on understanding the capabilities and limitations of LLMs for code generation and form a foundation for further improvements. Though MHPP offers valuable insights into code generation, it’s important to acknowledge its limitations in terms of data size and potential bias, which are provided in Appendix [G](#A7)

## Appendix A Related Works for Other Code Generation Taks

Recent works try to improve HumanEval and MBPP from different perspectives. For example, HumanEval+ (Liu et al., ) enhances the HumanEval with improved test cases, remedying the issue of mistakenly accepted faulty solutions. Meanwhile, ReCode (Wang et al., ) takes a different approach by altering function names and docstrings within the HumanEval structure. Expanding the scope beyond Python, HumanEval-X (Zheng et al., ), MultiPLe (Cassano et al., ), and MBXP (Athiwaratkun et al., ) extend the HumanEval and MBPP benchmarks to incorporate a variety of programming languages. The universe of code generation benchmarks widens further when we consider the specialized needs of data science. DS-1000 (Lai et al., ), ARCADE (Yin et al., ), NumpyEval (Zan et al., ), and PandasEval (Jain et al., ) focus on the generation of code within this context. Beyond mere code creation, there are benchmarks like APIBench (Patil et al., ), MTPB (Nijkamp et al., ), RepoBench (Liu et al., ), ODEX (Wang et al., ), SWE-Bench (Jimenez et al., ), GoogleCodeRepo (Shrivastava et al., ), RepoEval (Zhang et al., ), and Cocomic-Data (Ding et al., ), which ratchet up the complexity by evaluating a model’s prowess in utilizing APIs or completing broader software engineering tasks. Table [3](#A0.T3) shows comparisons among MHPP and several representative benchmarks.

## Appendix B Data Contamination

Following the official guideline of the contamination detector (^2^22[https://github.com/liyucheng09/Contamination_Detector/tree/master](https://github.com/liyucheng09/Contamination_Detector/tree/master)), we extract only the question stems from MBPP and use Bing Search to find related content online. When matches are discovered, they are evaluated based on token-level similarity. This evaluation helps determine how similar the test sample is to online content, assisting in identifying potential contamination. We set a threshold of 0.7, meaning a match is considered contaminated if the similarity exceeds 0.7.

## Appendix C Generalization Beyond Challenge of HumanEval

Using HumanEval as a starting point may limit the coverage of problem types and error patterns. Therefore, we actively sought to generalize the problem types and address more realistic and challenging error patterns in the creation of MHPP. We provide how we generalize from different challenges as follows:

Distraction: there is only one problem in which there are some short sentences that are irrelevant to solving the problem, but we design more subtypes of this challenge, for example, we add a lot of background information to the problem to evaluate the model’s ability to accurately filter out redundant information and focus on core functionalities, some problems have more than 500 words (indeed, the context is not as long as those in SWE-bench (Jimenez et al., ) or other repo-level benchmarks, but we do find that many strong models have extremely low performances on these benchmarks, such and Claude2 (4.8%) and GPT4 (1.74%) on SWE-bench, currently there are still many models have small context window like 4096 tokens, we think it’s still necessary to have a in-between benchmark to distinguish models’ ability). We also inserted tables or misleading/ambiguous descriptions into the problem. These are all points beyond which using HumanEval can evaluated.

Redefinition: in HumanEval there are always equations defined in problems or some redefinition of concepts in the real world, we generalize subtypes by adding more counterfactual concepts, to challenge the model’s ability to focus on current context but not the common sense it learned in the pre-training.

Shortcut: compared to those in HumanEval which can only be classified as arithmetic or brainstorming tricks, we not only keep original subtypes but also make it more general and comprehensive to be math algorithms or even gaming theory problems.

Commonsense: there are merely problems with simple common sense like the alphabet or cars. We make this situation more general, by adding problems relevant to temporal or spatial concepts, and academic knowledge like chemistry problems, optical problems, physics problems, etc.

Cornercase: there are only several problems in HumanEval contain the requirement of branches to handle simple corner cases (like dealing with the case when the input is 0), we further generalize the subtypes to be more practical cases as well as those that have hidden requirements (for example, a model must know requirements of forming a triangle before judging a triangle whether is isosceles), there are more real-world scenarios like this which are important in real-world programming tasks.

Complexity: there are also different subtypes from that in HumanEval, such as combining multiple simple logic units, focusing on numbers of control flow statements, dynamic programming relevant problems that are more abstract in complexity, and problems requiring models to have planning ability.

Codesense: we can barely say that the questions in HumanEval assess function calls, as the required function calls are either too few or too basic. We further extend it to more libraries that can be used in real-world programming tasks, for example, like the scientific computing library Numpy, or the calendar library that could be used in actual development. Additionally, the number of calls in one problem is more than that in HumanEval.

## Appendix D JAVA and C++ Results on MHPP

**Table 4: GPT-4’s pass@1 performance on MHPP across different languages.**
|  | Distraction | Redefinition | Shortcut | Commonsense | Cornercase | Complexity | Codesense | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Python | 35.0 | 65.0 | 40.0 | 70.0 | 55.0 | 55.0 | 55.0 | 53.6 |
| Java | 20.0 | 35.0 | 20.0 | 45.0 | 20.0 | 20.0 | 15.0 | 25.0 |
| C++ | 45.0 | 30.0 | 10.0 | 40.0 | 25.0 | 25.0 | 20.0 | 27.9 |

We have translated the MHPP’s problems and test cases into Java and C++ and tested the GPT-4 model’s performance in these languages.
The results, as depicted in the newly introduced Table [4](#A4.T4), reveal that the model’s performance in Python significantly surpasses that of Java and C++, with pass@1 rates of 25.00% and 27.86% respectively. This disparity suggests that the model has been more comprehensively trained in Python. Interestingly, we noticed a more pronounced performance drop from Python to other languages in our dataset compared to other function-level code generation datasets, such as from HumanEval (Chen et al., ) to HumanEval-X (Zheng et al., ). We hypothesize that this could be attributed to the increased difficulty level of the problems, making it more challenging for LLMs to solve them in languages other than Python. Upon closer examination of the data across different categories, we found that the model exhibits a stronger performance in “Commonsense” problems, but struggles with “Shortcut” problems. This indicates that the model has a better understanding of common sense concepts compared to abstract mathematical algorithms.

## Appendix E LLMs’ Performance on MHPP Using Greedy Search Decoding

**Table 5: The performance of LLMs on MHPP using greedy decoding. We identify the top-performing models among open-source LLMs. Models achieving the highest performance are emphasized in bold, while those in second place, including tied results, are underscored.**
| Model | Distraction | Redefinition | Shortcut | Commonsense | Cornercase | Complex | Codesense | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Closed-Source LLMs |  |  |  |  |  |  |  |  |
| GPT-4-1106-Preview | 35.0 | 65.0 | 40.0 | 70.0 | 55.0 | 55.0 | 55.0 | 53.6 |
| GPT-3.5-Turbo | 25.0 | 40.0 | 25.0 | 30.0 | 15.0 | 15.0 | 45.0 | 27.9 |
| Open-Source LLMs |  |  |  |  |  |  |  |  |
| Phi 1-1.3B | 10.0 | 15.0 | 10.0 | 10.0 | 10.0 | 5.0 | 30 | 12.9 |
| Phi 1.5-1.3B | 0.0 | 10.0 | 5.0 | 15.0 | 10.0 | 0.0 | 20.0 | 8.6 |
| Phi 2-2.7B | 10.0 | 15.0 | 15.0 | 25.0 | 10.0 | 0.0 | 20.0 | 13.6 |
| Mistral-7B Ins | 15.0 | 0.0 | 0.0 | 25.0 | 0.0 | 15.0 | 0.0 | 7.9 |
| LLaMA2 Chat-7b | 0.0 | 0.0 | 0.0 | 5.0 | 10.0 | 0.0 | 10.0 | 2.9 |
| LLaMA2 Chat-13b | 0.0 | 0.0 | 0.0 | 0.0 | 10.0 | 0.0 | 5.0 | 2.1 |
| CodeLLaMA-7B | 0.0 | 0.0 | 0.0 | 10.0 | 10.0 | 0.0 | 5.0 | 3.6 |
| CodeLLaMA-13B | 0.0 | 5.0 | 0.0 | 10.0 | 10.0 | 0.0 | 10.0 | 5.0 |
| CodeLLaMA-34B | 10.0 | 15.0 | 5.0 | 30.0 | 10.0 | 0.0 | 30.0 | 14.3 |
| CodeLLaMA-7B Ins | 5.0 | 5.0 | 5.0 | 15.0 | 15.0 | 0.0 | 15.0 | 8.6 |
| CodeLLaMA-13B Ins | 20.0 | 15.0 | 10.0 | 15.0 | 15.0 | 10.0 | 20.0 | 15.0 |
| CodeLLaMA-34B Ins | 20.0 | 25.0 | 10.0 | 25.0 | 5.0 | 5.0 | 25.0 | 16.4 |
| CodeLLaMA-7B Py | 0.0 | 5.0 | 0.0 | 10.0 | 5.0 | 0.0 | 5.0 | 3.6 |
| CodeLLaMA-13B Py | 0.0 | 5.0 | 0.0 | 10.0 | 10.0 | 0.0 | 10.0 | 5.0 |
| CodeLLaMA-34B Py | 10.0 | 10.0 | 0.0 | 25.0 | 10.0 | 5.0 | 5.0 | 9.3 |
| DeepSeek-1.3B | 5.0 | 10.0 | 5.0 | 30.0 | 20.0 | 0.0 | 10.0 | 11.4 |
| DeepSeek-6.7B | 15.0 | 35.0 | 10.0 | 30.0 | 20.0 | 10.0 | 40.0 | 22.9 |
| DeepSeek-33B | 30.0 | 40.0 | 20.0 | 60.0 | 45.0 | 30.0 | 45.0 | 38.6 |
| WizardCoder-1B | 0.0 | 0.0 | 0.0 | 5.0 | 10.0 | 5.0 | 15.0 | 5.0 |
| WizardCoder-7B | 5.0 | 25.0 | 5.0 | 20.0 | 20.0 | 0.0 | 25.0 | 14.3 |
| WizardCoder-33B | 20.0 | 30.0 | 15.0 | 65.0 | 35.0 | 15.0 | 40.0 | 31.4 |
| WizardCoder-34B Py | 15.0 | 35.0 | 10.0 | 40.0 | 15.0 | 10.0 | 35.0 | 22.9 |

## Appendix F Potential Strategies for Improving LLMs on MHPP

Based on the experimental results of various LLMs on MHPP. We propose potential strategies for overcoming the challenges of MHPP. We have devised a set of strategies tailored to each category of challenges as follows:

Distraction: To tackle this challenge, we propose incorporating controlled noise into the training data and designing tasks that require the model to identify the genuine development intent and generate corresponding code.

Redefinition: We recommend enhancing the model’s exposure to knowledge-based data. This will improve its ability to comprehend concepts within questions. For new or contradictory definitions, we suggest refining the model’s in-context learning to prioritize the given context over general world knowledge. Techniques like symbol tuning could be beneficial for this purpose.

Shortcut: To address this, we propose augmenting the training data with more mathematical and logical reasoning tasks to help the model recognize patterns.

Commonsense: We recommend incorporating more relevant knowledge data. However, it’s crucial to avoid overfitting. Models can benefit from interacting with real-world data, such as world models and multimodal data, including images, to enhance their understanding of spatial concepts.

Cornercase: We suggest training models with more real-world code data, especially data rich in corner cases, to strengthen this capability. Using non-code data with many corner cases and extremes can also enhance the model’s robustness and accuracy during training.

Complexity: It’s beneficial to construct longer training data with more logical units, teaching the model to handle intricate logic. Strategies like curriculum learning can help models gradually master complex reasoning.

Codesense: We recommend providing rich programming language materials, such as official documentation and open-source libraries.

Furthermore, we suggest leveraging interpreters’ execution feedback to enhance the language model for the latter categories. For instance, rich test cases with execution feedback can make it easier to identify missing logic and correct generated code in Cornercase challenges. For Complexity challenges, feedback can help break down problems into smaller, more manageable tasks for improved accuracy. For Codesense challenges, error messages from code libraries can guide the model in understanding how to correctly use a library or function, leading to accurate solutions.

We believe that a well-designed dataset like MHPP can provide insights to guide strategies for improving model capabilities. By categorizing problems based on specific coding abilities, MHPP not only benchmarks models but also highlights areas for improvement. For example, if a model performs poorly on “code reasoning” problems, it suggests that incorporating more coding knowledge into the training data could help boost its capabilities in that area.

## Appendix G Limitations of MHPP

Data Size: The MHPP dataset indeed has a smaller scale compared to automatically generated datasets. This characteristic is intrinsic to hand-written datasets like HumanEval, to which MHPP is similar in terms of scale. While the dataset’s size enables a detailed analysis, we acknowledge that it could potentially restrict the diversity and representativeness of the data, thereby limiting the model’s ability to generalize to larger, more diverse codebases.

Potential Bias: The focus of MHPP on function-level code generation might introduce certain biases due to the annotation process primarily targeting challenges encountered during the writing of functions. This emphasis may result in a bias towards specific types of errors or difficulties, which might not comprehensively represent the wide array of challenges encountered in real-world coding practices. We recognize the importance of acknowledging these potential biases in the dataset collection procedure.

These limitations highlight the need for further research to develop strategies for effectively scaling up hand-written datasets while maintaining annotation quality. Extending the scope of the dataset beyond the function level to capture the broader context of code generation tasks is also important. By addressing these limitations, future code generation datasets can provide a more comprehensive picture of real-world software development challenges, ultimately leading to the development of more robust and versatile code generation models.

## Appendix H Error Analysis on MBPP

Upon analyzing GPT-4 errors in the MBPP benchmark, several critical issues have been identified. Text highlighted in red indicates the specific areas where the model makes mistakes or the error patterns appear. These issues encompass a range of deficiencies, including the absence of explicit return format specifications, the presence of ambiguous requirements, and inconsistencies between the parameters specified in function definitions and those utilized in test codes.

Example 1: No specification for the return format: The question does not declare that a specific string like ’Found a match!’ or ’Not matched!’ should be returned when indicating a match or not. The generated code will not address this issue at all.

Example 2: The question is ambiguous, for example, it is unclear whether the term ’non-repeated’ should retain or not retain duplicate elements, but the question does not provide any example to clarify this.

Example 3: Missing conditions regarding parameters: it is unclear and does not explain what N represents as a parameter.

Example 4: Incorrect function name in the test code (missing “r” in “arrange”): a normal language model should generate the correct function name and should not have this error in the test code.

Example 5: The question does not specify the return format: it is unclear that two elements need to be returned.

Example 6: Copying the question missed a requirement: “Given an array of n integers. The problem is to find the maximum length of the sub-sequence with the difference between adjacent elements as either 0 or 1.”

Example 7: Missing original question formula images, etc.

Example 8: The number of parameters in the function does not match those in the test code.

Example 9: The definition from the question is missing.

## Appendix I Error Analysis on HumanEval

Example 1 - Distraction: The first paragraph of the problem talks a lot about background information that is not very relevant to solving the problem.

Example 2 - Redefinition: This problem typically defines or redefines a new concept called Tribonacci sequence.

Example 3 - Shortcut: A shortcut to this problem does exist (number of 1s equals to 18 * (10 ** (n - 2)) when n is larger or equals to 2), by using a formula, this problem can be more easily solved.

Example 4 - Commonsense: The problem requires the model to understand the concept of collisions and spatial concepts.

Example 5 - Cornercase: The problem has a corner case which is that the numbers are an empty list, the solution is expected to have a single control branch to handle this case.

Example 6 - Complex: There are many constraints in this problem.

Example 7 - Codesense: The model needs to know the knowledge of binary operators.

## Appendix J Error Analysis on MHPP

Example 1 - Distraction: By introducing a table in the question to distract the model’s focus, the strategy was indeed effective, leading the model to produce a series of table-based problems, completely deviating from solving the original question properly.

Example 2 - Redefinition: The model did not grasp the concept of redefinition; it misunderstood that the balance factor is only applicable if the total weight is even.

Example 3 - Shortcut: It resulted in a timeout due to not knowing the shortcut.

Example 4 - Commonsense:
There is a complete lack of understanding of spatial awareness for LLMs; those moving left from the right side won’t meet those moving right from the left side.

Example 5 - Cornercase: LLMs truly did not check for the boundary condition of being a triangle.

Example 6 - Complex: The model knows that it should use index 3 to retrieve the fourth number from a Python array in early lines. However, even though the model claims in the comments that it will operate on the fourth number, it still uses 4 as the index. As the number of constraints increases, the model commits errors that would not occur under fewer constraints.

Example 7 - Codesense: Despite understanding ternary, the model mistakenly used a binary function, indicating a weakness in utilizing external functions.

Figure: Figure 7: The variable name in the generated function didn’t match that in the MHPP but was the same as in the training corpus.
Refer to caption: /html/2405.11430/assets/x6.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - While not essential, including results for GPT-o1-preview could further underscore the point that current LLMs have yet to fully master function-level code generation, particularly if its performance is lower.
soundness: 3
strengths: - A key question about whether current LLMs have mastered function-level code generation, and the detailed breakdown of 7 challenge types effectively motivate the need for this benchmark.
summary: This paper presents a function-level code generation benchmark comprising 7 distinct problem types. With state-of-the-art models like GPT-4o achieving a Pass@1 of 51.1, this benchmark offers a valuable resource for advancing future work in the community.
weaknesses: - The benchmark includes 210 problems, which, while comparable to HumanEval’s 164, may be insufficient for broader generalizability. Note that recent benchmarks, like BigCodeBench [1], offer over 1K problems.

- Current code generation benchmarks including this work are vulnerable to future data contamination as the test set is often public. To mitigate this, splitting the benchmark into validation and hidden test sets, with evaluations on the test set limited to submissions, may be advisable, following NL2SQL [2, 3]. 

[1] Zhuo, Terry Yue, et al. "Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions." arXiv preprint arXiv:2406.15877 (2024).

[2] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.

[3] Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, & Yongbin Li (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: * Can the authors clarify the confidence intervals constructed differentiating the types of noise observed? Additionally, it might be useful to measure the noise in evaluating the performance of 30/40 samples using a bootstrapping-based approach (sample n problems out of 210 and measuring noise).
soundness: 3
strengths: * **Benchmark and Error Analysis.** The authors analyze existing benchmarks MBPP and HumanEval identifying mistakes and contamination. They also produce a manual categorization of mistakes made on HumanEval
* **New benchmark.** The authors provide a manually curated benchmark of 210 problems focusing on mistakes identified on 
HumanEval. This provides confidence in the quality of the benchmark problem statements
* **Qualitative Analysis.** The authors provide insights into model failures through detailed qualitative analysis on existing and proposed benchmarks.
summary: The paper introduces a new dataset called Mostly Hard Python Problems (MHPP), consisting of 210 unique, human-curated Python programming problems designed to evaluate LLMs' abilities across seven manually identified challenges. Next, it presents experiments on 26 LLMs providing empirical insights.
weaknesses: * **Benchmark Size.** The benchmark only consists of 210 benchmark problems. The benchmark size is limited and puts concerns over empirical findings in question. This is an even more serious concern for problem for the category-wise results presented where the number of benchmark samples would be around 30-40 (Tables 2, 3 and Figure 4) -- potentially increasing noise levels to over 10/15% making the results unreliable.

* **Confidence Intervals.** As I understand, section 5.1 computes the confidence intervals for noise introduced due to temperature sampling. This should be properly clarified since it does not account for variability due to the limited size of the benchmark which is a considerably bigger source of noise. This is also a known issue with HumanEval (see [1]).

* **Related Work.** The paper does not clarify or distinguish from many programming benchmarks released in the past year including but not limited to xCodeEval, CodeScope, RepoEval, RepoBench, ClassEval, LiveCodeBench, EvoEval, BigCodeBench. 

MHPP is particularly similar to EvoEval [2] which identifies overlapping sets of problem types `Difficult, Creative, Subtle, Combine, and Tool Use` against `Redefinition, Distraction, Shortcut, CommonSense, Cornercase, Complexity and Codesense` studied in this work. [2] additionally proposed an approach to automatically curate problems from existing benchmarks -- however at the cost of not providing sufficient guarantees on problems - a strength of manual curation performed by the authors. The related works should be discussed and differences clarified appropriately 

* **Novelty.** While collected through manual curation, the paper does not challenge existing empirical performance trends (beyond the reduced performance on the more challenging benchmark). However, this should not diminish the usefulness of the benchmark problems considering saturation in HumanEval.

[1] https://github.com/crux-eval/eval-arena

[2] Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM. COLM 2024

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: It would be very useful for the authors to more explicitly situate this benchmark in terms of existing work and what it uniquely brings to the table.
soundness: 2
strengths: 1. The benchmark is human-created and (for the moment) is unlikely to be a part of any pre-training corpora
2. The authors show that the problems are challenging enough to leave some headroom, even for the SOTA models
summary: The paper proposes a new coding benchmark that is intended to be more challenging for Code LMs than MBPP and HumanEval. It also undertakes an analysis of the standard failure modes of models on existing benchmarks as well as leakages of existing benchmarks in the pre-training data.
weaknesses: Overall, I do not see the point of this benchmark in terms of bringing something to the field that is not already out there:

1. ~14 tests on average per sample makes it better than HumanEval and MBPP but is still far outmatched by benchmarks such as EvalPlus [1]
2. In terms of being a challenging test for CodeLMs, due to limitations in chosen domains, library usage and question difficulty, it is, on average, well outdone by existing benchmarks like BigCodeBench [2], ClassEval [3] and SWE-Bench [4].
3. It is also a Python-native benchmark, which leaves out medium and low-resource programming languages that existing benchmarks like McEval [5], Multipl-e [6] or MxEval [7] cover.

[1] https://arxiv.org/abs/2305.01210
[2] https://arxiv.org/abs/2406.15877
[3] https://arxiv.org/abs/2308.01861
[4] https://arxiv.org/abs/2310.06770
[5] https://arxiv.org/abs/2406.07436
[6] https://arxiv.org/abs/2208.08227
[7] https://arxiv.org/abs/2406.07436

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * How to ensure high diversity and comprehensive coverage of fully human-crafted benchmarks?
* Could you provide a summary of the key insights derived from the experimental results?
* Are there any potential approaches to enhance LLMs to better handle these challenges uncovered by MHPP?
soundness: 3
strengths: Overall, it is a good attempt to address the limitations of existing code generation benchmarks with more finegrained and complex evalution. Extensive evaluation show that this increase in complexity helps reveal the performance limits of advanced LLMs, which perform well on simpler benchmarks but struggle with MHPP.
summary: This paper introduces MHPP, a new benchmark for code generation that seeks to address the limitations of existing benchmarks like HumanEval and MBPP, which have become too simple for current high-performing language models such as GPT-4. MHPP contains 210 human-curated Python problems, designed in a similar code completion style to HumanEval but with enhanced diversity and difficulty across various problem types (categorized as 7 challenges). Evaluation results on 26 LLMs reveal that models performing well on HumanEval do not achieve comparable results on MHPP, highlighting the increased complexity of this benchmark. Additional analysis also examines the types of errors made by models like GPT-4 and confirms a correlation in performance trends with HumanEval.
weaknesses: - **Incremental Contribution**: While this paper makes a commendable attempt to address the limitations of prior code generation benchmarks by offering a more fine-grained evaluation, its novelty and technical contributions are relatively incremental. It aligns with a line of research that introduces increasingly challenging benchmarks to highlight the limitations of LLMs without substantially advancing our understanding of how to overcome these challenges. Additionally, the insights drawn from the results are somewhat **limited**, especially considering that other more complex code generation benchmarks, such as LiveCodeBench, already exist. To make it more solid work, the authors should provide more insights from the results and how they guige targeted improvements to model architectures or training. For example, they can analyze patterns in the errors to gain insights into fundamental limitations of current LLM approaches.
- **Unsystematic Problem Curation**: The problem categories and challenges are curated manually, which may affect consistency, coverage, and diversity. The lack of an automated, systematic process for problem generation could limit the scalability and reproducibility of this benchmark, making it harder to ensure high quality and comprehensive coverage across problem types.

Meta-review: The paper introduces MHPP, a dataset of 210 unique, human-curated Python programming challenges aimed at assessing LLM performance across seven challenge types. These challenges types are inspired by manual inspections from existing benchmarks HumanEval and MBPP. The work reports experimental results from 26 LLMs, demonstrating the challeges of MHPP.

While all reviewers agreed the dataset can be valuable, the work lacks a convincing statement of its novelty. Particularly (1) the comparison with many existing benchmarks such as EvolEval, CruxEval is unclear (2) The dataset size is small, and the data collection process relies heavily on human efforts, leaving scaling the dataset size a challenge.

Decision: Reject