McEval: Massively Multilingual Code Evaluation
Authors: name: Linzheng Chai; name: Shukai Liu; name: Jian Yang; name: Yuwei Yin; name: Ke Jin; name: Jiaheng Liu; name: Tao Sun; name: Ge Zhang; name: Changyu Ren; name: Hongcheng Guo; name: Zekun Wang; name: Boyang Wang; name: Xianjie Wu; name: Bing Wang; name: Tongliang Li; name: Liqun Yang; name: Sufeng Duan; name: Zhoujun Li
Published: 2024-06-11 16:45:17

Abstract: Code large language models (LLMs) have shown remarkable advances in code
understanding, completion, and generation tasks. Programming benchmarks,
comprised of a selection of code challenges and corresponding test cases, serve
as a standard to evaluate the capability of different LLMs in such tasks.
However, most existing benchmarks primarily focus on Python and are still
restricted to a limited number of languages, where other languages are
translated from the Python samples (e.g. MultiPL-E) degrading the data
diversity. To further facilitate the research of code LLMs, we propose a
massively multilingual code benchmark covering 40 programming languages
(McEval) with 16K test samples, which substantially pushes the limits of code
LLMs in multilingual scenarios. The benchmark contains challenging code
completion, understanding, and generation evaluation tasks with finely curated
massively multilingual instruction corpora McEval-Instruct. In addition, we
introduce an effective multilingual coder mCoder trained on McEval-Instruct to
support multilingual programming language generation. Extensive experimental
results on McEval show that there is still a difficult journey between
open-source models and closed-source LLMs (e.g. GPT-series models) in numerous
languages. The instruction corpora, evaluation benchmark, and leaderboard are
available at https://mceval.github.io/.