M2rc-Eval: Massively Multilingual Repository-level Code Completion
  Evaluation
Authors: name: Jiaheng Liu; name: Ken Deng; name: Congnan Liu; name: Jian Yang; name: Shukai Liu; name: He Zhu; name: Peng Zhao; name: Linzheng Chai; name: Yanan Wu; name: Ke Jin; name: Ge Zhang; name: Zekun Wang; name: Guoan Zhang; name: Bangyu Xiang; name: Wenbo Su; name: Bo Zheng
Published: 2024-10-28 15:58:41

Abstract: Repository-level code completion has drawn great attention in software
engineering, and several benchmark datasets have been introduced. However,
existing repository-level code completion benchmarks usually focus on a limited
number of languages (<5), which cannot evaluate the general code intelligence
abilities across different languages for existing code Large Language Models
(LLMs). Besides, the existing benchmarks usually report overall average scores
of different languages, where the fine-grained abilities in different
completion scenarios are ignored. Therefore, to facilitate the research of code
LLMs in multilingual scenarios, we propose a massively multilingual
repository-level code completion benchmark covering 18 programming languages
(called M2RC-EVAL), and two types of fine-grained annotations (i.e.,
bucket-level and semantic-level) on different completion scenarios are
provided, where we obtain these annotations based on the parsed abstract syntax
tree. Moreover, we also curate a massively multilingual instruction corpora
M2RC- INSTRUCT dataset to improve the repository-level code completion
abilities of existing code LLMs. Comprehensive experimental results demonstrate
the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.