Hard400:A Bilingual Code Generation Evaluation Benchmark for Large Language Models
Authors: Qin Zhang, Song Zhang, Yitong Li, Han Liu, Meng Fang


Reviewer paper summaries: The authors have compiled a code evaluation benchmark consisting of 422 hard-difficulty problems from the Leetcode platform. They have provided bilingual docstrings (English and Chinese) for the problems and finetuned a set of LLMs on a train-test split. The analysis shows that the benchmark is indeed more challenging than the existing ones. They also provide some insights about LLMs' performance variation on English vs Chinese docstrings.
Summary of strengths: 01. Current LLMs have almost cracked the existing Code Synthesis Benchmarks (HumanEval, MBPP, etc.), so more challenging benchmarks are needed in this field of study. Hard400 could be a decent contribution towards that aspect.

02. Almost all works only focus on coding tasks with English prompts and/or docstrings. The inclusion of Chinese is a solid contribution as well.

03. The authors have hand-picked models that are properly suited for such experimentations, having a good distribution of models that are pretrained and/or finetuned in English and/or Chinese. This could help answer some good research questions.
Summary of weaknesses: 01. As per the 'terms of services' of the platform (https://leetcode.com/terms/); LeetCode does not allow scrapping their problems from the website and redistributing as such. The ethics guidelines are needed to be thoroughly checked for this work.

02. The solutions for each task that they provide were also collected from the platform's forum section. these were selected based on the number of stars and simplicity (as mentioned in Lines 277-280). However, it was not mentioned what 'simplicity' entails in this case.

03. A bigger concern is whether it is ethically allowed to collect other users' solutions from the platform and redistribute them.

04. Table 3 is missing most of the SOTA LLMs (MagiCoder, GPT4, Claude 3.5 etc. ).

05. Table 4 has some models with both finetuned and non-finetuned results. Firstly, why? Secondly, nothing is mentioned about how the non-finetuned models were evaluated. Zero-Shot prompting? Few Shot? CoT?

05. No error analysis or insights. Why and where the models are failing?
Comments, suggestions and typos: 01. Line 32 - You meant Hard400, right?
02. Line 30,213,214,215,216,217, 219,221,222,287,288 - WhiteSpace issue.
03. Line 206-229 - Redundant paragraph.
Overall assessment scores: 2