McEval: Massively Multilingual Code Evaluation

Authors: Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, JinKe, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, Noah Wang, Boyang Wang, Xianjie Wu, Bing Wang, Tongliang Li, Liqun Yang, Sufeng Duan, Zhaoxiang Zhang, Zhoujun Li
Primary area: datasets and benchmarks
Keywords: Benchmark, Code Intelligence, Multilingual, Large Language Model, Multilingual Multitask Code Evaluation

Abstract:
Code large language models (LLMs) have shown remarkable advances in code understanding, completion, and generation tasks. Programming benchmarks, comprised of a selection of code challenges and corresponding test cases, serve as a standard to evaluate the capability of different LLMs in such tasks. However, most existing benchmarks primarily focus on Python and are still restricted to a limited number of languages, where other languages are translated from the Python samples degrading the data diversity. To further facilitate the research of code LLMs, we propose a massively multilingual code benchmark covering 40 programming languages (McEval) with 16K test samples, which substantially pushes the limits of code LLMs in multilingual scenarios. The benchmark contains challenging code completion, understanding, and generation evaluation tasks with finely curated massively multilingual instruction corpora McEval-Instruct. In addition, we introduce an effective multilingual coder mCoder trained on McEval-Instruct to support multilingual programming language generation. Extensive experimental results on McEval show that there is still a difficult journey between open-source models and closed-source LLMs in numerous languages. The instruction corpora and evaluation benchmark are available at https://github.com/MCEVAL/McEval.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729755461673

thread_type:
official_review

<thread_object>
creator: reviewer_3qms

<content>
summary:
This paper presents MCEval,  a massively multilingual code benchmark covering 40 programming languages with 16K test samples. MCEval provides a new evaluation benchmark in multilingual scenarios. In addition, the authors also present a new instruction dataset MCEVAL-INSTRUCT for multilingual code LLMs. The experiments results suggest that the benchmark MCEval could measure the gap between open-source and closed-source models.

strengths:
1. This paper is easy to follow. The figures presented are clear and easy to understand.
2. The benchmark MCEval is comprehensive, covering 40 programming languages with 16K test samples.

weaknesses:
1. The contribution of MCoder is unclear.
The contribution of this paper is comprise of two parts, a multilingual evaluation benchmark MCEval and a multilingual code LLM MCoder (trained on an instruction corpora MCEVAL-INSTRUCT). However, the performance of MCoder (using gpt-4 to generate data) are still behind the previous sota code LLMs like magicoder who uses gpt3.5 to generate instruction data.  The authors claim that MCoder is used as a strong baseline for MCEVAL(line 073), but why others don't use a better model (Magicoder) but MCoder as their baseline?

2. The method of constructing dataset is lack of novelty.
The biggest contribution of this paper seems the number of programming languages included in their dataset. However, the method of constructing such their dataset is using LLM to generate and refine from the code snippets, which has been used in many previous works such as bigcodebench(https://arxiv.org/pdf/2406.15877).

questions:
1. What impact will the Unbalance on Different Languages (in section 5) bring to the evaluation process of code LLMs？
2. What kinds of languages will help code LLMs to generate better? What will do harm?  These problems maybe more insightful for multilingual code evaluation of coding LLMs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730031725721

thread_type:
official_review

<thread_object>
creator: reviewer_kkun

<content>
summary:
This paper introduces MCEVAL, a massively multilingual code evaluation benchmark covering 40 programming languages with 16,000 test samples. It aims to advance the evaluation of code LLMs in multilingual scenarios. The benchmark includes tasks for code generation, completion, and explanation. Additionally, the authors present MCODER, a multilingual coder trained on the MCEVAL-INSTRUCT corpora, demonstrating its effectiveness in multilingual programming language generation. In its experiments, this paper highlights the gap between open-source and closed-source models and provides a comprehensive framework for evaluating and improving code LLMs across diverse languages.

strengths:
1. This is truly a BIG project. The authors have made significant contributions to the field of multilingual code generation.

weaknesses:
1. Although this work makes a significant contribution to benchmarking the multilingual code generation capabilities of LLMs, its main contribution is labor-intensive, with limited technical contributions or insights (MCEVAL-INSTRUCT looks similar to MagiCoder by generating instruction data from code snippets).
2. Is it meaningful to benchmark code LLMs on 40 languages? Can the author elaborate on why it is important to measure the generation capabilities of code LLMs in 40 languages simultaneously?
3. Based on the experimental results, MCEVAL and MultiPL-E have similar measurement capabilities, which diminishes the significance of MCEVAL compared to MultiPL-E.
4. It is said that "three volunteers are employed to evaluate the correctness of the benchmark (> 90% accuracy) and correct the errors" but no details are provided. Can the author explain how to ensure that the entire benchmark has a 90% accuracy rate? How was the "90%" quantified (we need ground truth to calculate the 90% right)?

questions:
1. Table 2 shows that MCEVAL has 16,031 questions, but in reality, there are only 2,007 completely independent programming problems. I think that comparing the benchmarks based on independent programming problems would better illustrate the quality of each benchmark, as a benchmark can increase its number of questions by "creating new tasks", but it cannot do the same to generate new independent programming problems.
2. The fonts in the leaderboard tables are too small. 
3. The programming tasks differ between languages, making it hard to directly compare a model's capabilities across different languages.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730541285867

thread_type:
official_review

<thread_object>
creator: reviewer_1wxp

<content>
summary:
The MCEVAL benchmark introduces a challenging, multilingual evaluation framework for code LLMs, covering 40 programming languages with 16K test samples. Alongside this, MCEVAL-INSTRUCT - a curated multilingual instruction corpora, and MCODER—a multilingual coder trained on MCEVAL-INSTRUCT are introduced.

strengths:
(1) Extensive study and effort has been spent in generating and documenting this code benchmark.

(2) While HumanEval and MBPP are the popular code programming benchmarks today, Code LM community needs additional multilingual programming benchmark. Hence, this paper addresses a problem statement that is of demand today.

weaknesses:
GPT models, like, gpt-4-1106-preview, has been used to generate the problem description for code instruction corpora. That could have influenced GPT models to lead in the benchmarks with significant performance margins over other models.

questions:
Is there any study done to validate the accuracy or correctness of  MCEVAL benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693363452

thread_type:
official_review

<thread_object>
creator: reviewer_npvu

<content>
summary:
The authors propose a new code benchmark called McEval. The proposed benchmark has 40 programming languages with 16k samples. The benchmark spans code generation, code completion and code understanding tasks. The authors also introduce mCoder trained on the McEval-Instruct corpora.

strengths:
The paper is well-written and is very easy to read and follow. The paper is also a significant leap towards better evaluation for code models. Current models are mostly evaluated on smaller benchmarks like HumanEvalPack or MBPP which have very few languages (<10) and this limits the ability to effectively evaluate code models.

The authors' work is quite significant since it introduces a new benchmark with 40 programming languages which is more than 4x of any previous benchmarks. The paper also presents an instruction corpora.

I overall like the paper and the amount of experiments done by the authors.

weaknesses:
The instruction corpora is generated using GPT-4 and thus is commercially unusable. I understand that collecting good human annotated instructions is a significant undertaking and might be infeasible.

questions:
1. Is the dataset being released?
2. Have you evaluated [Granite code models](https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330), it would be nice to have them in the report since there are very few SOTA code models available under Apache 2.0 license?
3. Are the human annotators fairly compensated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731946949949

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors

comment:
We sincerely thank all reviewers for their time and constructive feedback. In response to these valuable suggestions, we have made the following revisions to the paper, highlighted in yellow for clarity. We list these changes below.
1. We updated Table 2 (Page 3) by adding the number of data for each task to provide a more objective comparison, as suggested by Reviewer kkUn.  
2. We tested the Granite Code Models[1] (ranging from 3B to 34B) and added the results to Table 3 (Page 6), as suggested by Reviewer npVu. 
3. As suggested by Reviewers 1Wxp and kkUn, we revised Appendix A.2.3 QUALITY CONTROL (Page 18), providing a clearer description to address concerns raised by reviewers about the data annotation process.  
4. We redraw some model results as bar charts (Figures 14, 15, 16, 17, and 18, Pages 22–24), which are temporarily placed in the appendix. This aims to address Reviewer kkUn’s concerns about the small font size in the experiment result tables and provide a more intuitive comparison for readers. Due to time constraints, we will continue to adjust the number of models in the figures and their placement in the main text.  
5. As noted by Reviewer kkUn and 3QMs, our synthetic data generation method shares similarities with Magicoder[2] and the concurrent work BigCodeBench[3]. We have added a description of Magicoder and BigCodeBench in the "Detailed Related Work" section of the Appendix (Page 26) and summarized the key differences: our approach extends to a broader range of programming languages.

We sincerely appreciate the positive feedback on McEval and are dedicated to further advancing research in multilingual programming. We remain committed to continuously maintaining and improving the benchmark to support ongoing research and development.

**Reference** 

[1] Mishra M.,et al. "Granite code models: A family of open foundation models for code intelligence". arXiv 2024.

[2] Wei Y.,et al. "Magicoder: Empowering code generation with oss-instruct". ICML 2024.

[3] Zhuo T Y.,et al. "Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions". arXiv 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
> The authors propose a new code benchmark called McEval. The proposed benchmark has 40 programming languages with 16k samples. The benchmark spans code generation, code completion and code understanding tasks. The authors also introduce mCoder trained on the McEval-Instruct corpora.

The reviewers all accept the paper. Code generation benchmark are severely underserved beyond Python (and a few other popular languages). This benchmark seems to be of relevant scope and size to help the whole field of code generation measure progress.