McEval: Massively Multilingual Code Evaluation
Authors: Linzheng Chai
Shukai Liu
Jian Yang
Yuwei Yin
JinKe
Jiaheng Liu
Tao Sun
Ge Zhang
Changyu Ren
Hongcheng Guo
Noah Wang
Boyang Wang
Xianjie Wu
Bing Wang
Tongliang Li
Liqun Yang
Sufeng Duan
Zhaoxiang Zhang
Zhoujun Li
Venue: ICLR 2025 Poster (2025)
Keywords: Benchmark
Code Intelligence
Multilingual
Large Language Model
Multilingual Multitask Code Evaluation
Primary area: datasets and benchmarks
TL;DR: To facilitate the development of code LLMs, we introduce a complete framework that includes the multilingual code instruction corpora, multilingual coder (mCoder), and multilingual code evaluation benchmark.

Abstract: Code large language models (LLMs) have shown remarkable advances in code understanding, completion, and generation tasks. Programming benchmarks, comprised of a selection of code challenges and corresponding test cases, serve as a standard to evaluate the capability of different LLMs in such tasks. However, most existing benchmarks primarily focus on Python and are still restricted to a limited number of languages, where other languages are translated from the Python samples degrading the data diversity. To further facilitate the research of code LLMs, we propose a massively multilingual code benchmark covering 40 programming languages (McEval) with 16K test samples, which substantially pushes the limits of code LLMs in multilingual scenarios. The benchmark contains challenging code completion, understanding, and generation evaluation tasks with finely curated massively multilingual instruction corpora McEval-Instruct. In addition, we introduce an effective multilingual coder mCoder trained on McEval-Instruct to support multilingual programming language generation. Extensive experimental results on McEval show that there is still a difficult journey between open-source models and closed-source LLMs in numerous languages. The instruction corpora and evaluation benchmark are available at https://github.com/MCEVAL/McEval.

Published as a conference paper at ICLR 2025
McEVAL: MASSIVELY MULTILINGUAL CODE EVALUA- TION
Linzheng Chai', Shukai Liu' Jian Yang' "i Yuwei Yin', Ke Jin', Jiaheng Liu', Tao Sun', Ge Zhang'. Changyu Ren'. Hongcheng Guo', Zekun Wang'. Boyang Wang', Xianjie Wu', Bing Wang'. Tongliang Li', Liqun Yang'. Sufeng Duan', Zhoujun Li' 'CCSE, Beihang University, UUniversity of British Columbia, 'University of Waterloo *Beijing Information Science and Technology University, SShanghai Jiao Tong University
ABSTRACT
Code large language models (LLMs) have shown remarkable advances in code understanding, completion, and generation tasks. Programming benchmarks, com- prised of selection of code challenges and corresponding test cases, serve a standard to evaluate the capability of different LLMs in such tasks. However, most existing benchmarks primarily focus on Python and are still restricted a limited number of languages, where other languages are translated from the Python samples degrading the data diversity. To further facilitate the research of code LLMs, we propose a massively multilingual code benchmark covering 40 programming languages (MCEVAL) with 16K samples, which substan- tially pushes the limits of code LLMs in multilingual scenarios. The benchmark contains challenging code completion, understanding, and generation evaluation tasks with finely curated massively multilingual instruction corpora McEVAL- INSTRUCT. In addition, we introduce an effective multilingual coder MCODER trained on McEVAL-INSTRUCH
support multilingual programming language generation. Extensive experimental results on McEVAL show that there is still a difficult journey between open- sver-s models and closed-source LLMs in numer- ous languages. The instruction corpora and evaluation benchmark are available at https:: https:://gttuu:
INTRODUCTION
Large language models (LLMs) designed for code, such as Codex (Chen et al., 202T). CodeGen (Nijkamp et al.)
CH4
X
2023), Code Llama Rozière et al. 2023), DeepSeek- Coder (Guo et al. 2024), and CodeQwen (Hui et al. (024) excel at code understanding, completion, and generation tasks.
Jub
Code LLMS with a large number of parameters (e.g. 7B, 13B, or larger) pre-trained large-scale code databases with self-supervised autoregressive objectives, followed by instruction tuning (Ouyang et al./ (2022) for aligning to human preferences and downstream code- related tasks. Most code benchmarks (Chen et al., 2021]] Austin et al.] 2021 Athiwaratkun et al.) 2023) are intro- duced evaluate the performance of code LLMs by as- sessing their ability generate executable code based on the problem descriptions. The assessments aim to gauge the capacity of the models to understand and generate code effectively, thereby contributing to facilitating and streamlining the programming process for developers. The execution bassed method executes generated code against test cases to measure the
Codl Coaphii
Cae
Pythos
4012
(41
Figure/ 1: McEVAL comprised three
tasks: code generation, code completion, and code explanation.
Equal contribution.
Corresponding Author.

---

Published as a conference paper at ICLR 2025
success rate. Due to the difficulty of creating the problem and its corresponding solution (requiring specialized programming staff), the development of evaluation benchmarks is limited within Python, with few other languages being translated from Python. Therefore, the community desperately needs a massively multilingual programming benchmark (not from HumanEval or MBPP) comprised of instruction corpora and evaluation
comprehensively facilitate and evaluate the generation, completion, and understanding capability of LLMs.
To facilitate the development of code LLMs, we introduce a complete framework that includes the multilingual code instruction corpora, multilingual coder (MCODER), and multilingual code evaluation benchmark. First, we propose McEVAL, the first massively multilingual code evaluation benchmark (from human handwriting) covering 40 languages (I6K samples in total), encompassing multilingual code generation, multilingual code explanation, and multilingual code completion tasks. Then, we create a massively multilingual instruction corpora MCEVAI-INSTRRIC" of 40 languages. We initially select and refine high-quality code snippets from various programming languages (PLs) using an LLM. The LLM then generates clear and self-contained instructional content, including problem descriptions and corresponding solutions, based on the refined snippets. To ensure consistency and enhance learning across languages, introduce cross-lingust code transfer, adapting instructional content to different PLs while increasing sample complexity. Based on open-source models and McEVAL-IINSRRN the transferability of LLMS among different PLs.
MCODER is used ströng baseline explore
The contributions summarized as follows:
(1) We propose McEVAL with enough test samples (16K), a true massively multilingual multitask code evaluation benchmark (not from HumanEval or MBPP) covering 40 languages, encompassing multilingual code generation, multilingual code explanation, and multilingual code completion tasks. (2) We introduce McEVAL-INSTREN
the massively multilingual code instruction corpora covering from the multilingual code snippet from 40 languages. Based on MCEVAL-INSTRUCT, an effective multilingual coder MCODER is used as a strong baseline for MCEVAL. (3) We systematically evaluate the understanding and generation capabilities of 20+ models on our created MCEVAT. and create leaderboard to evaluate them on 40 programming languages dynamically. Notably, extensive experiments suggest that comprehen- sive multilingual multitask evaluation can realistically measure the gap between open-source (c.g. DeepSeekCoder and CodeQwenl.5) .5) and dosed-source models (e.g. GPT-3.5 and GPT-4).
MULTILINGUAL CODE EVALUATION: McEVAL
2.1 DATASET STATISTICS
Table 1: McEVAL dataset statistics.
The created MCEVAL is comprised of three key code-related tasks covering 40 programming languages, including multi- lingual code generation, multilingual code explanation, and multilingual code completion tasks. The multilingual code generation and explanation tasks separately contain 2K sam- ples, where each language has nearly 50 samples. The code completion task can be decomposed intomulti-line comple- tion (3K samples), single-line completion (3K samples), span completion (4K samples), and span completion (light) (2K
Statisties
Value
Questions
Code Generation
2, 007
Code Explanation
2. 2,007
Code Completion
12, 017
Single-Line
2, 2.998
Multi-Line
2.998
Spea
4,014
Spaan
007
samples) Bavarian et al. 2022).
Total Test Cases
10, 086
Difficulty Level
In Table we display the number of questions, test cases, and difficulty levels corresponding to the three tasks in McEVAL and the number of questions in the four sub-tasks of the completion task. Moreover, we counted the token length of the prompt and solutions. (The tokens are calculated based on the Llama-3 tokenizer.) Among these tasks, the span completion (light) task is similar in form to the span completion task. However, in the span completion (light) task, each problem is paired with all the corresponding code, making it a balanced version of the span completion task (fewer samples for fast inference and the same test size of
-aay
1,221
Medium
401
Hard
385
Length
Prompt
maximum length
793 tokens
minimum length
16 tokens
- avg lengih
173.8 tokens
Solution(Oatuut)
maximum length
666 takens
minimum length
4 tokens
avg length
120.9 tokens
2

---

Published as a conference paper at ICLR 2025
Prorpe Lergth
Oorpet mickkee)
Tex Carae
50x
Cotbedlege Consonn
Dyeetogg vorting Vound Bessi
Figure 2: Data statistics of the McEVAL benchmark involving 40 programming languages.
each programming language). The results of span completion (light) can better reflect the differences model performance across different languages.
Figure 2 plots the length of the prompt, solution(output),, and the number of test cases of each programming language. In Table We compared MCEVAL with other multilingual benchmarks. is noteworthy that our benchmark provides a significant supplement to current benchmarks in terms of both the variety of programming languages and the number questions.
Table 2: Comparison between MCEVAL and other multilingual code benchmarks. The number of each of the three tasks (Generation, Explanation, and Completion).
Benchmarkk
Multi Task
#Langunges
Data source
#Questions
MuliPL-E Cassano et al 2023
Translate
-3,000
MBXP Athrwaraikun el al 2023
Translate
12,425
HumanEval X Thongg 2022h
Hand-Wrirten
820
HumanEval-Xi
Peng Peng ct a. 2024
x
12
Hand-Written
22,080
McEVAL
40
Hand-Wrine
16.031 (2007/2000//20117))
2.2
HUMAN ANNOTATION & QUALITY CONTROL.
To create the massively multilingual code evaluation benchmark McEVAL, the annotation of mul- tilingual code samples is conducted utilizing comprehensive and systematic human annotation procedure, underpinned by rigorously defined guidelines to ensure accuracy and consistency, Initially, 10 software developers in computer science recruited as multilingual programming annotators with proven proficiency in the respective programming languages. Following detailed training session on the annotation protocol, which emphasizes the importance of context, syntactical correctness, and semantic fidelity across languages, annotators are tasked with creating problem definitions and the corresponding solution. The annotators should follow: (1) Provide clear and self-ccontiined problem definition, answer the question with any tools, and design the test cases to evaluate the correctness of the code. (2) Classify them into multiple difficulties (Easy/Middle/Hard)) based on algorithmic complexity and functionality. Each sample is independently annotated by at least two annotators to minimize subjective bias and errors. Discrepancies between annotators are resolved through consensus adjudication by senior annotator. Finally, three volunteers employed to evaluate the correctness the benchmark (> 90% accuracy) and correct the errors, (See Appendix A.2 for more details).
2.3 EVALUATION TASKS
Multilingual Code Generation. Given the k-th programming language La E [Li) where K = 40 is the number of programming languages, we provide the problem description qra and examples test cases eº as the input for code LLMs M to generate the corresponding code Lx, We obtain the sampled code result from the code generation distribution P(a4* Iq_*, M from code I.MM M, and then feed the test cases into the generated code, where the generated outputs by code should equal the expected outputs. The process can be described as:
Lh [[P(aa Le 1ath Lu; M); ))::a EN)
(1)
3

---

Published as a conference paper at ICLR 2025
Instruction: Language Python
Instruction: Language
Instraction: Langsage
Write a Python function 'daf www.n.prratoo.... then
inter
sepaeiie westpereeestte strinng
code imploenttation.
sobve the problem:
Wecoitriege
pastoer urtegen wan that the Catuatte hormette serder
let wot cosdraeessttiee wectCh
fectorial tawer.
pivm
ler corestt Appriistt wi
strrig stringg irew))
(1rO: Shreger rop mesenting the tooe valae (5
for In sores chrieg.chensc
Pacariu:
ectrateting facterti" of
Argat
(tet): pasttiv trteger
thrashodd
depth
let: The wutteen Ieteer for aetch the nonte
corvent Jayc
string
precen resarttin
Gale:
4.4 OiL 1907,
Reference Solution
resalt
fi11 (NASK) Insteaple Sinas code say
si ot ALAA 144 laks Trituation wollaiis
(decstring) of the Rast code English
langaage
Solution
mast we diaraiters.
wille Sun-cu
(ocal proce aer peael3
Reference Solution
sdle "her_aaaa
Cases
ER
Text Cases
def
sstart (iI Auu araiii Mi0u tie cas
Irour this fusetier stringg costatatng
SW MoN)
fattedd fil Jiuu www.eregeenttt (00 "hat ase
and return
It to caparate theee group tnto cspanate ctringe
assar Cunoeest (epant13)
Separata
balarced (each open braca
proporly closol) and not neated sithis each other
princiall CAll
Ignone any quaces
48912
fan the tut fveetirr
retur's
(I) Code Generation
(2) Code Explain
(3) Code Completion
Figure 3: Examples of multilingual code generation, explanation, and completion.
where I(()) is the indicator function by executing the generated code with the given test cases wh when the generated code a passes all test cases, the evaluation result r = else r 0.
Multilingual Code Explanation. To evaluate the understanding capability of code LLMs, we Lo--Natural-Langee
and NetturalLannuageee tooCe))) since the text-similar metrics (e.g. BLEU (Papineni et al. 2002) ) are hindered by the n--rram text matching and can not produce accurate score. We first prompt the code L.MSs generate the natural language description th based on the code a and then we force the model to restore the original code based on tha The sampled code from P(a"a L. ( MM); is used to evaluate the understanding capability as:
adopt two-pass generation
r = I(P(( Ja bx (a MM) u**)
where I(() is used to check the correctness of the generated code by running the code with test cases.
Multilingual Code Completion. Another important scenario is code completion, where the code LLM produces the middle code att based on the prefix code af" and suffix code snippet a!*,, Hence, we concatenate a ahe and af* as the complete code for evaluation
(( Es la .a "*)
(3)
where ate, att, and a are concatenated as the complete code to be executed with test cases utt.
3 MCODER
3.1
MCEVAL-INSTRUCT
Collection from Code Snippet. For programming language Lt (L+ E (L,}KK 1) and K is the number of programming languages), consider an existing code snippet e E Dér, we prompt the LLM to select the high-quality code and refine the code to a self-contained code snippet by using the prompt "I Code Snippet//Determinnn
its educational value for student whose goal is to learn basic coding concepts. AntnIf the answer is 'YES'. Please refine the code with clear variable definitions, comments, and docstring.". Then, we obtain the multilingual refined code snippets. (More details can be found in Appendix A.3)
4

---

Published as a conference paper at ICLR 2025
Step1 Code Collection
Stepl: Code Selactin and Refierment
Steph Code Coepora intructie
Prompt
Prompt
designing highquaalty (laggage question
stadzst goul loas
given erigpet.
codryg concppes
Thhe queetion and snower muse ho comp eaely setl- contanee and skow.
sii Guslllines
the saoves is "SS"", Pleas refine te sole clear varibiie defasiiios. ceerenents, and docanin
The diffisslly of the code car taken soop funher and doorting dena tes the problem descrpttion Ake Code uxppes
L
Strpli Maxeidly Muhiliggal Doirenice
Code
(cods]
Complctias
Prompt
( Queaivoll
McFvall
Code Explais
Codeliana
Quaction and onowerof
Langsage See);
Stepe Cooe-lingaa Feafonommers
Koilin
the Crensd Solation
@(Ereated Solatinn)
HteephenbCCoder
comten))
GO
CodeQwes
Rust
glwn queston regenee ts cratte new queston lasswer of language ltpi)
Pascal
OO
Steph: Inemaction Turning
print: [mums]))
LEM
R
HTML
consolo log couns. reducal (aa.b)-aa))))) b)- 0012
Scala
MeCaler
begini
Scala
Python
std
Multilngual Instruction Corpora
Figure 4: The framework of MCODER. We first create MCEVAL- covering 40 languages from code snippets to fine-tune MCODER. 20+ existing LLMs and MCODER are then evaluated on McEVAL comprised of multilingual code generation, explanation, and completion.
Instruction Corpora Generation. To construct comprehensive massively multilingual code instruction corpora (D K 1im1- we prompt the LLMs
create problem description g+t and the corresponding solution a² by drawing inspiration from the refined code snippet cht We use LLM to generate instruction dataset by using the prompt "You are an expert in programming, especially in designing high-quality language question and answer based on the given code snippet.\nlv An'n #### Guidelines: 'The question and answer must be completely self-contained and clear. *In The difficulty of the code can be taken step further and the docstring describes the problem description. \n ### Given Code snippet: codeAn ### Created Question Created Question'n ### Created Solution'n Created Solution" in Figure
Cross-lingual Code Transfer. Since the created instruction samples of different programming languages focus on different aspects of coding, we adopt the cross-lingual code transfer min- imize the gap among multiple languages. Given the instruction dataset D of language L., we randomly sample a pair (qtr, (at),a"+) and force the LLM to modify them to another language L, with more complex sample (g (a, " ala--s))) In this way, struction corpora 1Db.-4, Vi AjA1 (DI==b)) }i & JAA S i,j S K) as the multilingual instruction corpora MCEVAL-INSTRUCT (DLEK 1K mc]km1 covering 40 programming languages.
get the derived in- i,j K). Finally, combine {DL Ik=1 and
3.2 MULTILINGUAL CODE INSTRUCTION TUNING
The training objective Lall of the multilingual instruction fine-tuning can be described as:
K
Call Σ Fe [og P(a (gg :M) MD]
(4)
where giz and ah are the code-related question and answer from the dataset D4a of language Lx. respectively. K is the number of programming languages.
EXPERIMENTS
4.1 EXPERIMENT SETUP
Code LLMs. We evaluate 30 + models with sizes ranging from 7B to 236B parameters, including
general/code LLMs, opwn/closed-souusee models, and base/instruction models. For general models, evaluate GPT series (Brown al. 2020 OpenAl 2023), Qwen1.5 1.5 (Bai al. 2023). Llama3 (AI
5

---

Published as a conference paper at ICLR 2025
Table 3: Pass@1 (%) scores of different code LLMs for multilingual code generation tasks on MCEVAL. "Avgalt" represents the average scores of all code languages.
2011
10ng
Seriics
37a
10.0
1000
700
301
668
4133
360
K2.6
11.8
1000
774
1800
15.0
46.3
350
403
540
56.8
16.9
14.30
54.3
1148
668
7400
303
21.0
9-48
7760
740
320
24.8
15.0
240
154
248
$2.0
14.8
20.4
12.0
18.0
213
140
35A
1600
255
36.
1600
194
546
70.0
36.0
1356
1824
12.0
140
158
540
163
364
60.
768
300
10.5
18.0
403
116
200
174
340
310
1333
9-3
136
250
198
158
740
0-1
145
138
480
300
233
153
360
369
176
100
340
213
264
1554
2.
13.
580
560
153
330
309
103
253
160
403
16.
220
143
200
510
180
9-3
Colobil sua
18
244
246
360
3.0
140
360
220
225
1669
340
18
14.
344
300
360
123
120
140
-20
120
1055
1969
1969
4.0
76
14.8
546
7000
56.0
24.8
174
1869
400
411
21.8
75.0
34.3
16.8
10.0
303
18.0
66.8
06.0
10.0
50.0
06.6
11.2
1944
15.00
31.8
460
18.0
62.5
263
15.8
368
54.0
7605
3400
64.0
1546
14.0
500
23.8
19.3
64.5
313
24.0
125
150
663
14.
140
1485
1869
345
61.3
36.
256
180
46.3
360
565
11.5
546
520
31.8
405
196
240
542
Kload V
303
120
300
160
26.5
186
428
140
180
300
Coll AK
06.
234
330
360
144
200
360
413
360
4.0
31.
408
160
160
363
511
106
340
679
11.
231
1943
703
Cold
213
363
2.0
213
26
11.
438
180
217
Cold
123
380
128
128
380
900
560
303
308
368
1881
198
Cold
203
115
118
130
250
113
146
140
133
512
145
130
2-5
114
226
134
253
U
413
186
251
109
140
115
141
140
0-
013
364
254
360
368
168
126
240
813
238
164
146
360
8-3
368
218
164
180
6.2
2024).
Phi-3 (Abdin et al.
2024 and Yi (Young et al. 2024). For code models, we test Code-
Qwen (Hui al. (024), CeepSeekCoder (Guo et al. (024). CodeI.lama (Rozière et al., (023))
OCTOCODER (Muennighoff et al., 2023), CodeShell (Xie et al. 2024). MagiCoder (Wei et al.
2023), WizardCoder (Luo et al. 2023). Codegemma (Gemma Team 2024) and Granite (Mishra et al.
2024. Furthermore, we further line-tune MCODER based on CodeQwen1.5 and DeepSeek to
explore the language transfer capabilities of code LLMs.
Evaluation Metrics. We assess the models by executing the code and evaluating it using the @ass@1 I metric. @ass@1 is widely recognized measure in machine learning, particularly for code generation, as it gauges the model's accuracy in producing correct solutions on the first attempt.
Instruction Corpora. The resulting dataset, McEVAAL-INSTRUCT (1 10KK samples), is comprised of created question-answer pairs and open-source collection (Wei et al. 2023). We apply data decontamination before training our MCODER. Following Li et al. (2023); Wei
20233) we adopt et al.. 2021), ). et al., 2023). MBPP(Austin et al. 2021). For supervised one-tunnng (SFT), we utilize CodeQwen- 1.5 as the foundational code LLMs. Specifically, we select all Python data from comprising 50K training samples, for MCODER-Python training.
the N-gram exact match decontamination method with MCEVAL, Eval
Optimization & Evaluation. Our MCODER based on CodeQwen 1.5 are trained for 2 epochs with a cosine scheduler, starting a learning rate of 2c-5 (3% warmup steps). We use AdamW (Loshchilov & Hutter, 2017) as the optimizer and batch size of 512 (max length 4096). We adopt the greedy Pass@I I (%) ) metric (Kulal et al., 2019 Chen et al. 202T) for evaluations. For closed-source LLMs, the answers are generated by the official API. For code explanation, we prompt the LLM to describe the code and then restore the descriptions to the original code. (Details can be found Appendix A.6).
4.2 MAIN RESULTS
Multilingual Code Generation. Table shows the Pass@1 results of various models on McEVAL for multilingual code generation task. The results reveal significant disparity between closed-source
6

---

Published as a conference paper at ICLR 2025
Table 4: Pass@ I (%) scores of different models for multilingual code completion tasks on McE
"Avgal" represents the average scores of all code languages.
virt
C-4
C
Celor
Tae
Elo-
Eioe-
Erter:
Evens
Fr
Draon
Blocc
Ene
30a
the
30.8
744
106
143
1441
$4.5
10.4
2014
805
617
N4
1511
195
308
1988
22.8
3448
81.8
794
1533
96.
183
964
011
350
403
18.3
196
11.4
144
312
14.5
145
18.3
21.3
111
135.
734
442
961
411
06.0
710
0010
43.0
11108
63.0
1631
354
112
62.8
4100
263
2344
18.8
41.3
11.4
61.0
9611
750
12.5
8553
K9A
254
1HA
303
12.5
96.5
246
36.
18.3
54.2
CI be (1%
64.3
$1.8
71.
311
13.
3x2
65.
544
13.8
410
4744
223
313
912
08.3
16.3
19K2
948
94.
Devel Lifee
18.3
$16
27.6
54.5
Yi Chal
(758
06.3
250
23.4
6238
130
503
Soc
C
0-4
19
C+9
1et
Eis
Els:
Edee
Forn
F4
Tall
114
35
30.
79A
963
64X
311
147
1113
1983
303
111.
610
310
196
32.8
11.
Crobd
133
3068
16.0
35.8
13.0
51.5
21.4
13.00
21.4
14.0
940
11.8
51.0
5059
10.5
14.4
16.1
52.0
11.5
150
1766
10.0
$4.55
21.4
11.6
1811
756
$2.8
1131
16-1
103
111
106
15A
11.8
16.1
10.0
21.6
312
18.1
7801
05.5
142
be carice
(142
01.3
15.0
5.3
11.
12.5
12.1
71.
14.1
18.0
16.3
15:1
184
30.40
1944
21.
134
110
224
114
14.5
15.4
3448
324
3633
154
36.5
CookOhoa'' S-llut
(34)
11.3
35
470
102
04.3
901
10.5
16.3
40.2
40.5
$6.51
Sper
Sic
C
C-4
C.99
16t
Eio-
Exs:
Edee:
Furm
F4
0reen
1e4
15
30.
00.0
67M
840
740
45.
7503
2548
300
10.3
604
91.5
6551
1848)
10.
12.0
300
41.3
18.0
01.0
111
14.0
353
16.1
110
113
12.8
200
154
813
240
14.0
13.3
1400
21.8
23.6
21.8
11.0
2644
23.0
00.3
11.0
35.9
478
158
53.8
20.0
15.8
76.9
766
550
$60
18.1
1948
25.3
$1.8
84.0
160
12.
7961
155
18.3
8118
3161
1533
3011
54.0
2101
1811
18.8
34.3
142
14.0
21.3
184
11.3
240
1941
7-33
73.8
3113
14.
04.0
09.3
08.1
22.8
11.3
13.0
2500
11.
388
123
301
264
144
11.0
20.5
1344
14.0
23.3
3515
13.8
23.
880
378
K1.8
5530
146
322
2-3
18.0
11.3
$5.3
(964
11.4
159
754
133
550
544
546
776
513
12.8
340
144
04.8
43.3
56.11
Compecii
Sac
0-4
04
C.9
Cafoa
164
Ein
El:
Limi
Turm
16
66
Cree5
104
15
304
Kile
CFT- 230
16.6
198
B48
101
596
66.7
1403
7566
3000
13.8
153
16.
19.00
9600
16.8
1625
10.00
1840
36.8
14.0
754
1611
300
142
05.3
30.0
08.0
06.3
31.8
196
10.0
311
1340
1440
Dolld
133
11.8
12.
31.
213
148
2400
12.0
11.5
14.5
14.0
$1.8
34.
168
15.0
13.9
766
18.8
36.3
$1.8
36.9
181
128
15.3
1868
758
1028
14.9
36.0
1840
18-
7801
30.2
142
700.
Areu
CFF be OM1
504
349
100
734
11.3
505
104
163
404
413
7340
23.5
340
044
04.8
11.3
2044
14.8
12.3
14.0
(864
128
11.
200
213
254
18.0
25.3
Yo Che
1036
1968
30
1820
168
122
10.0
12.5
164
142
36.3
(766
096
310
340
754
13.
120
544
166
12.0
15.1
12.8
560
224
434
11.
55
-fate-of-thh-aat models and open- models across nearly all programming languages. Notably, GPT-4o and GPT-4 Turbo lead the benchmark with substantial performance margins over other models. The McEVAL apart from previous benchmarks (such as HumanEval), where various open models have achieved comparable or superior performance. The results indicate that MCODER
7

---

Published as a conference paper at ICLR 2025
Table 5: Pass@ I (%) scores of different models for multilingual code explanation tasks on McEVAL "Avgall" represents the average scores of all code languages.
Witted
20%
City
E3*e
Elong
Ferm
Greere
15:4
36.3
15A
154
624
34.
$1.3
7508
$4.0
184
214
764
62.3
50.3
$1.0
13.0
11.0
7460
(75.)
33.9
244
21.
61.8
1214
21.0
6123
61.3
144
11.0
754
413
228
320
31.8
2314
303
36.3
184
8.3
194
140
16.8
16.4
12.0
180
16.9
51.9
1711
811
11.
776
74.8
1443
1500
1643
71.
1.00
$6.8
61.
794)
$6.0
40
15%
140
910
18.0
541
1841
14.0
86.8
1629
44
84.8
6118
1401
Mices Vou Martec
10
$2.8
$2.9
560
666
1068
6.3
16.5
646
61.8
013
66.0
560
076
66.3
58.9
44-
16.3
80.3
25.3
Cole
11mm
140
Ruce
Fr1l
118
Faam
110.m
340g
6.41
5.6mm
16611
1.48
044
1.61
4rg.
16.
74.0
134
264
78.
61.8
64.3
02.
00.8
76.0
78.0
845
75.0
71.0
51.5
648
62.8
730
76.0
63.3
65.
628
61
188
18.0
14.0
3.50
14.0
10.0
57.0
08.1
6220
11.0
42.
168
813
170
140
181
11.0
10.0
19.5
363
60
220
354
363
320
240
11.0
56.
22.6
180
3-8
5809
280
530
013
36.3
12.4
100
40
170
315
12.
382
120
31.
11.
180
360
13.
Cnleegeen Chal
14.8
03.3
60
1831
114
515
613
5601
140
540
36.3
15.0
123
0.8
164
620
61.3
23.3
1841
K23
MCSemR Cou Metacc
0228
50
348
302
380
313
exhibits clear improvement over the base model in nearly all the studied programming languages. It is noteworthy that MCODER, despite being trained with very limited multilingual data, still outperforms other large language models (LLMs) of similar or even larger sizes.
Multilingual Code Explanation. Table 5 displays the Pass@1 results for multilingual code expla- nation tasks. The results show that GPT models still significantly outperform open-source models in the code explanation task. For markup languages (Json and Markdown), the complexity of the code structure makes it diffieult to describe accurately in natural language, resulting in generally poorer performance. Code LLMs need instruction-following capabilities for such complex structures.
Multilingual Code Completion. The completion tasks consist of single-line completion, multi-line completion, span completion, and span completion (light). As shown in Table 4 the Pass @ results for multilingual code completion tasks indicate that GPT-4 Turbo still achieves the best performance. Additionally, since this task is relatively easier compared to code generation, some open-source models perform comparably to GPT-4 Turbo in certain programming languages.
5 FURTHER ANALYSIS
Programming Classification. In Figure 5] we categorize the programming languages of McEVAL into 5 programming paradigms and 11 application scenarios and summarize the performance of code LLMs on the code generation task in Figuue 6 It can be observed that code LLMs generally perform better in object-oriented and multi-paradigm programming languages (high-resource languages), while perform worse in functional and procedural programming languages (low-resource Languages). In areas like web development and scientific computing, the gap between open- and closed- source models is narrowing. However, for application scenarios, there is still substantial gap between open-source models and the closed-soures source GPT-4 series in low-resource languages related to scripting, mobile development, and educational research. MCODER performs superior over multiple same-size models and even some larger open- models.
Unbalance on Different Languages. We compare the results of several open-source models on the MultiPL-E multilingual benchmark with corresponding languages on McEVAL. We obtained scores for 11 programming languages (including Python, Java, JavaScript. C++, PHP, Rust, Swift, R, Lua, Racket, Julia) from the BigCode leaderboard]" As shown in Figure 1), due to the sim- plicity of Python language tasks in this dataset, many models exhibit significant score discrepancies between the two benchmarks. Figure 2] highlights a majority of models within the blue circle, indicating that the current state-of-the art performance of most models primarily lies in high-resource languages like Python, while their proficiency in low-resoures languages awaits further exploration and enhancement. By examining Figure 7(2) and (3), becomes evident that all L.MMs demonstrate consistent multilingual capabilities between MultiPL.-E and MCEVAL.
https::/huggiingfaee co/ http//pacee/bbbee htt.:e////.iioddd///i/eeee/eeeee model
8

---

Published as a conference paper at ICLR 2025
Programming Paradigm
(2) Application Scenurio
Tu
Kendin
Nome
MAR
Torpen
Prccated
Cuilia
RUN
Vied
Clip
Ricka
Bise
Hasion
Club
Figure 5: Classification of McEVAL. The programming languages in MCEVAL can be categorized into 5 programming paradigms and 11 application scenarios.
Prooddumi
Mobile Davelopmont
027
Scrpps
Cross Platform
(192 STabo
Markup Language
Object Oriented
Scripls
Automatiion
Agp loatin
Besktop
OcilOvaa L.S
Latrat
Edocation
Web
Mode:
Frontend
Content
Web Backund
Funchonal
Mulúglee Poridign
Generalá Systom&SSfbrrn
Computing
Prygamming Panalggn Calegories
Applicstion Calagories
Figure 6: The performance of models in code completion tasks under different categories.
Pythonn-Pthon
Multilinuul-PPan
Figure 7: Unbalanced performance on different languages across MultiPL-E and our MCEVAL.
Cross-lingual Transfer. We fine-tune the CodeQwen-1.5 model using Python-only data in McEVAL-INSTRUCT and compare with MCODER. In In/Figure 8) CodeQwen-1.5 performs well in most high-resource languages, but CodeQwen without alignment exhibits unsatisfactory results in some low-resource languages due to the inability to follow instructions. As such, with fine-tuning us- ing only Python data, 1.te-oee--.5-Pythhee improves significantly across most languages. It shows that the CodeQwen foundation model already possesses strong coding capabilities but lacks adequate mnstructionnfolloviii skills. Therefore, fine-tuning with Python-only data can still effectively transfer -orluction-followinn abilities to other languages, resulting in superior multilingual performance.
Difficulty of McEVAL. Based on algorithmic complexity, we classify MCEVAL into three levels (Fasy/Mediun/Harc)) In Figure 9 we conduct statistical analysis CodeQwen- 1.5-Cha's perfor- mance code generation tasks across various languages. For most languages, the code LLM can answer the majority of easy questions but struggles with medium and hard ones.
6 RELATED WORK
For the field of soft engineering, code LLMs (Feng et al. 2020
Chen al. 2021
Scao al. 2022
Li al., 2022 Allal et al. 2023 Fried et al., [022] Wang et al.
2021 Zheng et al.
2024 Guo et al.
2024) pre-trained on billions of code snippets, such as StarCoder [Li al. 2023
Lozhkov al.

---

Published as a conference paper at ICLR 2025
600
Figure 8: Cross-lingual transferability of LLMs among different languages. We fine-tune CodeQwen- 1.5 using Python data in McEVAL-INSTRUCT and OSS-Instruce to create Male@wen--..--ythhee
Enow Tipeterne
Figure 9: CodeQwen- 1.5-Chat performance on McEVAL. for problems of different difficulty levels.
2024.. CodeLlama (Rozière al., 2023), DeepSeekCoder (Guo al. 2024). and Code- Qwen (Hui
et al. 2024). The development and refinement of code LLMs have been pivotal in automating
software development tasks, and supporting code senecatioonraaalaatooaaammmtnnii
Many benchmarks Yu et al. 2024 Yin et al. 2023 Khan et al. 2023 Orlanski et al. 2023 Jain
al. 2024) have been woven accurately assess code quality, functionality, and efficiency, such as
EvaanEval (Chen et al. 2021 D, MBPP (Austin et al., 2021). their upgraded version EvalPlus (Liu
al., 2023b). Studies have explored a variety of approaches, ranging from static evaluation using text matching to dynamic methods that involve code execution under controlled environment. The current benchmarks support code LLMs to evaluate a series of different types of tasks, such as
code understanding, function calling Zhuo et al. (2024), code repair (Lin et al. [017]] Tian et al.)
2024; Jimenez et al. 2023: ZZang et al., 2023] Prenner & Robbes, 2023 He et al.) [022 code translation (Yan al.
(2023). Some works focus the multilingual scenarios (Wang al. 2023
Athiwaratkun et al.
[023] Peng et al. 2024 Zheng et al. (2023b) by extending the Python-only
HumanEval/MBPP benchmark (e.g. MultiPL-E (Cassano et al. 2002)),, which challenged by the number of the languages.
CONCLUSION
In this work, we push a significant advancement in the assessment of code LLMs by proposing the first massively multilingual code evaluation benchmark (McEVAL) by involving an annotation and verification process conducted by professional developers, which spans 40 programming languages and helps comprehensively tackle various tasks, including code generation, explanation, and comple- tion. The multilingual SFT on created instruction corpora the proficiency of LLMs in multiple coding languages. Systematic evaluations of existing code LLMs on McEVAL illuminate the performance disparities among open-source and closed-source models. Extensive multilingual multitask assessment on McEVAL provides realistic and comprehensive measurement code LLMs, marking leap forward for developers utilizing AI techniques to understand and generate code effectively across wide spectrum of programming languages.
further emphasizes
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 62276017, 62406033, U1636211, 61672081), and the State Key Laboratory of Complex Critical Software Environment (Grant No. SKI.CCSE--0247X-188)
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint 2rXiv:2404.142111 2024. URL http::t/aaxxx...ooo
abs 2404
Meta AI. Introducing meta llama 3: The most capable openly available Ilm to date.
https:
7/ai. /aaimeta.... meta. cww/blog/meeeeeeeeccoocooooooooe wtt////taaaieee 1atttaaalamaa333 3/, apr 2024.
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. SantaCoder: Don't t reach for the stars! arXiv preprint arXiv:2301.03990 2023, URL
abs/ / 22012331.0300 03988
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, and Ramesh Nallapati. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL http:tttt////poeee..t.i het/Eorummid==077eee 6An8
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021. URL battps://artt orq/ abs 2108 07732
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijic Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, abs/2309.16609, 2023. URL httph: org abs ..6099.66669
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models fill in the middle. arXiv preprint arXiv:2207.14255 2022. UtL /ttptpps//aaxxi..om arxiv. org/ abs/ 2207. 2207.14255 14255
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting programming: A query language for large language models. Proc. ACM Progrum. Lang., (PPLD)); 2023. doi: 10.1145/3591300.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020. URL
www.nngn...uurrrie www.nnu.ri.... wwtprrippppppppppe.epeee /20/ap///000//////// / ott.://cc....................22. Abst .ttl
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 2023. URL
ieee. orgg wttgabbbttac/////////////////cct
/occumen/1011331
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian Man-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htt.hhttpss//pppnneeeet
net /dd d= rw 68 9c
11

---

Published as a conference paper at ICLR 2025
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, arXiv preprint 2iiv:22007003374 abs/2107.033777 2021. URL https://arxiv org/ abs 21 2107. 03374
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprins
arXiv:2211.12588. 2ttp/22111125522 2022. doi: 10.48550ARXIV.2211..1 1.1221.......... URL https
2t1p://vvvvv........... 12588
Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Kim M. Hazelwood, Gabriel Synnaeve, and Hugh Leather. Large language models for compiler optimization. arXiv preprint arXii:2309.070000 abs020900022 2023, doi: 10.48550/ARXV..2300.. 07062.. URL https: http://oooo...0.0.00/// 40550////.0......00.... 2309. 07062
Gautier Dagan, Frank Keller, and Alex Lascarides. Dynamic planning with LLM. arXiv preprint arXiv:2308 063911 abs/aabss.oooooo 063911, 2023. doi: 10448550/ARXIV.2308. 063391. URL https 7/doi. 48550/ www.:wwww.......... 06391
Peng Di, Jianguo Li, Hang Yu, Wei Jiang, Wenting Cai, Yang Cao, Chaoyu Chen, Dajun Chen, Hongwei Chen, Liang Chen, Gang Fan, Jie Gong, Zi Gong, Wen Hu, Tingting Guo, Zhichao Lei, Ting Li. Zheng Li. Ming Liang, Cong Liao, Bingchang Liu. Jiachen Liu, Zhiwei Liu, Shaojun Lu, Min Shen, Guangpei Wang, Huan Wang. Zhi Wang. Zhaogui Xu, Jiawei Yang, Qing Ye, Gehao Zhang, Yu Zhang, Zelin Zhao, Xunjin Zheng, Hailian Zhou, Lifu Zhu, and Xianying Zhu. Codefuse-13b: A pretrained multi-lingual code large language model. arXiv preprint arXiv:2310.062666, abs/2310.06266, 2023. doi: 10.45550/RXXI.231000066 //doi. 48550/455553.0 http://wwwww.......... 06266
URL https
Yihong Dong, Xue Jiang. Zhi Jin, and Ge Li. Self-collahoration code generation via chatgpt. arXiv preprint arXiv:2300. 075500, abs/2304.07550, 2023.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou Codebert: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 1536-1547 1547, Online, November 2020. Association for Computational Linguistics. doi: 10. 0188886331122200 http:///w/1220o.onnnngneeeeee findings-emnll....... 1399 URL https:/7
ac moolanhhooog.org/2000 wtt.g//222...innnnn. findings emn em..p.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 abs/2204.05999 2022. URL /tttts:://aatt
org/abs/ 2204 05999
Google Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08200 2024. URL httpt:tt///sstttt wtggabs/22033 08295
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming the rise of code intelligence. arXiv preprint 2rXiv:2401.14196 2024. httthttt::/aaat org/abs/ /401 14196
Jingxuan Hc, Luca Beurer- Beurer-Kellner, and Martin Vechev. On distribution shift in learning bug detectors. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and
12

---

Published as a conference paper at ICLR 2025
Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 8559 8580. PMLR, 17-23 Jul 2022. /tRptttts.//rrreednnng.... pre.s/..peesseeee..ee..ee http:///gs...mmprpreee//////////
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 122866.6.. 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint WxXhv:2433009973 2024. URL
btttps:/arxiv..oe wwtp:///g//abb//b//b/// /tt/:/bb//22233993
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint
arXiv:2310.06770 2023, URL/tttss://aaxii. wtt.://ggg.aa.....o..a.. 06770
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint arXiv:2303.03004, 2023, URL
https://arxiv> prg/ 03004
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pscudocode to code. Advances in Neural Information Processing Systems, 32, 2019. URL https::/p/pooceeddngg neurips wwtpaperpp.....///// 2atperr////9/hhhhh 9/hash/
html
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. URLhttps:
https: //cl://ll... www. orq/
3613165
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding. Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06 abs/2305.06113111 2023. doi: 1448550/axXiv.2305.066666
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schritwieserr, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. arXiv preprint arXi::2203.078 abs/2203.07814, 2022. URL wttp:///////////////////
https::/aaxii>: http://axxivvvx.x/aaa/// http://b//223033
Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. Quixbugs: multi-lingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN international conference systems, programming, languages, and applications: software for humanity, pp. 55-56, 2017. URL https: htop:://d //ll. acm. org/doi/ bbs//o..115/111599...
13

---

Published as a conference paper at ICLR 2025
Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haibo Hu, Xiaohong Zhang, and Meng Yan. Improving chatgpt prompt for code generation. arXiv preprint arXiv:2305.08 abs/2305.08360, 2023a. URLhttts::/// /ttplp:::///riv....//o wwwaaaaivvv....... org/ aataaabs////// 2305. 08360
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint urXiv:2305.0122 abs/2305.012I1 2023b. CRLhhttss://aaee org. /
01210
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:171100500, 2017.
051011
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang. Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, al. Starcoder and the stack v2: The next generation. arXiv preprint 2412,,3300..917,, 2024. URL 2402. 19173
org. /bs/
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. WizardCoder: Empowering code large language models with evol-instruct. arXiv preprint wrXiv22306.08568 2023. Ctthttps:: / /awxiv..vv.. http//rx.i....rgaba.aaaaaaaa/aaa abs/ 2306. 08568
Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza So- ria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, al. Gran- ite code models: A family of open foundation models for code intelligence. arXiv preprint arXiv:2405.04324. 2024.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. OctoPack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124, Abs/2308.07124 2023. URL
petpss//arxiin wwwp:/iwii.or...oooaaoeeeeeeeeee
org. hwtg://b//3330 07124
Fionn Murtagh and Pedro Contreras. Algorithms for hierarchical clustering: an overview. Wiley Interdiseiplinary Reviews: Data Mining and Knowledge Discovery, 2(1):86 2012. doi:
10.1002/WIDM.53 URLLattps:///oo. orq 1002 53
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. CodeGen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:/ts::///ott Utt.ptttss://openrer.eee net.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 2023, URL httthttt///aaaae org/aba/2303 08774
Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishabh Singh, and Michele Catasta. Measuring the impact of programming language distribution. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett e,ds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 26619-26645, PML.., 23- 29 Jul 2023. URL BoR[/attpp::/pppceddngg mmw.eeiiiiimm.i........mmmmmmmrm mlr wrwssppssppee..... pre.eeeeeeecceaaa.......e.see.ee 22tteaaa/22222rr11snk2333......
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744 Curran Associates, Inc., 2022. URL httpp//p///pppcceeiieeeeeeouuuup
files http://appppeiieeeeeaprr//////// hat.ae/eeeccctttaaaa3aSSaaaaaaaa
14

---

Published as a conference paper at ICLR 2025
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation, In Proceedings of the 40th Anmual Meeting of the Association
for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA. pp. 311-318 ACL, 2002. http://///aaaaaahllaaocc doi: 10.3115/1073083.10000 1073135. URL httpp///eee///eaaaaaoooooooooo/.
1040/
Qiwei Peng. Yekun Chai, and Xuhong Li. Humaneval-xi: A multilingual code generation benchmark for cross-lingual natural language generalization. arXiv preprini arXiv:2402.16694 2024. URL
org/ abs / 2402. 16694
Julian Aron Prenner and Romain Robbes. Runbugrun an executable dataset for automated program repair. arXiv preprint arXiv:2304.01102, 2023. URL bttps://axxiv.rqq wttpr///ab///a0/
02
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In CHI 21: CHI Conference on Human Factors Computing Systems, Virtual Event / Yokohama Japan, May 8-13, 2021, Extended Abstracts, pp. 314:1-31477, ACM, 2021. doi: httppp/ppe//ool..oo//11..1..1110 11450....1413310 1111111///1113133 3451760
10.1145/3411173.3455
URL https::pp...... http:/oi..oort/
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 0tti::233..29990 12950, 2023. URL httttts/p//axxiii
org/ abs aa////aaa/2/0/ 2t1.212......0.00
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter access multilingual language model. arXiv preprint arXiv:2211.05 2022. URL
httts:://axiv..cr wwtp://ggg/aaaa////o/e// wwtp///1..............ee.ee
Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Debugbench: Evaluating debugging capability of large language models. arXiv preprint
arXiv:2401.04621, 2024. UtLt:tts:://aaa
org/ abs/ 401. 2401.046211
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(1)), 2008 URL https::: ://::://wwwwww jm wwg.pppprss///oooo ume
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained
encoder-decoder models for code understanding and generation. arXiv preprint Kiv:2200,00089, 2021. URL https:tss: wawpvaaaaavvv/aa///a//aa.eaaaa org/ abs / 2tt.//11....0 00859
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution bxased evaluation for open-domain code generation. Findings of the Association for Computational Linguistics: EMNIP 2023, pp. 1271-1290, 2023. BtR.stts.... http:///wwwww..ooa/////a/oo///
10481
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need, arXiv preprini arXivv2312..0//20, ab//2312..2223 2023. doi: 10.48550/ARXIV. 2312.02120. HRLLhttps:/
Rui Xie, Zhengran Zeng, Zhuohao Yu, Chang Gao, Shikun Zhang, and Wei Ye. Codeshell technical report. arXiv preprint 1rXiv:2403.15747, 2024. URL aattps://arxisn
15747
Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. CodeTransOcean: A comprehensive multilingual benchmark for code translation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5067-5089, Singapore, December 2023. Association for Computational Linguistics. doi: 18.1 1/2023. findings-emnlp.3377 URL https::/aclantthllege org/20223333 Findings lp.lpp...3. 337]
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural language to code generation in interactive data science notebooks. In Anna Rogers, Jordan Draber,, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 126-173, Toronto,
15

---

Published as a conference paper at ICLR 2025
Canada, July 2023. Association for Computational Linguisties. doi: 10.18663ww1/202333o-lonngg settps://aclatthoogge.. org/ 202.2 1onq.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, al. Yi: Open foundation models by Ol.ai. arXiv preprint
arXiv:2403.04652, 2024. URL htts:t /ttppps://axxivei
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxi- ang Wang, and Tao Xie. Codereval: benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Soft- ware Engineering, pp. 1-12, 2024. URL https: acm. org. httpg//////bba/....111111 3597503 3623316
Daoguang Zan, Ailun Yu, Bo Shen, Jiaxin Zhang, Taihong Chen, Bing Geng, Bei Chen, Jichuan Ji, Yafen Yao, Yongji Wang, and Qianxiang Wang. Can programming languages boost each other via instruction tuning? arXiv preprint arXiv:2308. 16824, abs/2308. 16824, 2023. URL
wttp 233/ 16824
Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang. Bowen Yu, Weisong Sun, and Zhenyu Chen. critical review of large language model on software engineering: An example from chatgpt and automated program repair. arXiv preprini arXiv:2310.08889, 2023. URL https
//axxiv.oxq/aba/23100 08880
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang. Yufei Xue, Zihan Wang, Lei Shen, Andi Wang. Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprini arXiv:2303. /7568,
abs/2303.17568. 2023a. doi: 10448550ARRXVV2300.. 48550/arXiv. 2303......33.1111 17568
URL
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang. Lei Shen, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations humaneval-x. arXiv preprint arXiv:2303. /7568, 2023b. URL htthttpss:////// //axxiv
axg/abs/2303 17568
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreeer:: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024. URL https::ttp/.... http:://xxvvvvvoov aas///b///// 24022242220 14658
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint (xXi::2406.15877 2024.
16

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 LIMITATIONS
Cover more languages and tasks Our work focuses on evaluating models on multilingual pro- gramming tasks, currently supporting assessments in 40 programming languages. Many models claim to support over 80 programming languages, so our work can continue to expand the range of programming languages, the number of test cases, and types of tasks to provide a more comprehensive evaluation of the models.
Research on cross-lingual transfer We do not delve deeply into the cross-lingual transfer capabili- ties of models, leaving room for exploring a wider variety models and models of different sizes. In the future, will explore the programming language transferability scaling law of the code LLMs with different sizes.
A.2 DATA ANNOTATION
A.2.1 HUMAN ANNOTATION
To create the massively multilingual code evaluation benehmark McEVAL, the annotation of mul- tilingual code samples is conducted utilizing comprehensive and systematic human annotation procedure, underpinned by rigorously defined guidelines ensure accuracy and consistency.
We recruited annotators with backgrounds in computer software development from universities. During recruitment, we evaluated their programming and system operation skills to ensure they could handle tasks like question writing, code static analysis, and unit testing. Each annotator was assigned tasks based on their proficiency in specific programming languages.
Initially, 10 software developers in computer science are recruited as multilingual programming annotators with proven proficiency in the respective programming languages.
Following a detailed training session on the annotation protocol, annotators are tasked with creating problem definitions and the corresponding solution.
The guidelines for our annotation training session primarily cover the following aspects:
Standardized Format:
We provide annotation example for 40 programming languages. Annotators are required to adhere to this standardized format when annotating data.
Accessibility: The reference data for our annotations is sourced from materials available under permissive licenses, allowing unrestricted use and redistribution for research purposes.
We provide annotators with detailed guidelines on the difficulty clas- sification for each language. Annotators must strictly follow these guidelines to label problems according to their respective difficulty levels based on algorithmic complexity and functionality.
Difficulty Level:
Selr-C :ar-Contaiined: Annotators are required thoroughly review their annotated problems to ensure that the problem descriptions include all necessary information for solving them without ambiguity. The provided example inputs and outputs must be correct, the reference answers must execute correctly, and the test cases written should comprehensively evaluate the accuracy of the functions.
A.2.2 REFERENCE DATA
We use reference data (The annotators only draw the inspiration from the reference website, and create the question manually.) to draft questions and solutions with the help of GPT-4-Turbo. The draft questions initially may contain numerous issues, such as self-contained errors, inconsistent difficulty levels, overly simple unit tests, and incorrect unit tests. To address these issues, annotators revised and tested the draft questions ensure that all questions and code were accurate and that the unit tests would pass. These reference data came from the following websites: For the algorithmic types of questions, we refer to the following websites:
17

---

Published as a conference paper at ICLR 2025
http://wwww.doepp.com/
https//www.luogu.com.oo
http://www.cooccaddmm.coom
https//wwww.odewwrs.coom//
For markup language and design-type questions, we refer to the following websites:
https//wwww.uuoobb.cmm//
https//wwww...schoob..comm
A.2.3 QUALITY CONTROL
We adopt a dual-pass system to ensure the quality of our benehmark McEVAL. First, one annotator labels the code snippets and their corresponding unit tests. Then, another annotator independently reviews these annotations, verifying their correctness, code accuracy, and unit test integrity. Following this, three senior annotators evaluate the overall unit test pass rate of the annotated dataset. the pass rate exceeds 90%, the senior annotators proceed to perform additional reviews and corrections. Otherwise, the data returned to the annotators for refinement. Finally, the canonical solution (labeled by the annotator) passes all corresponding test cases to ensure the correctness of each created problem (100% pass rate). This rigorous process ensures the creation of a high-quality, multilingual programming benchmark that supports in-depth analysis and understanding of code examples across diverse programming languages.
A.2.4 ANNOTATION COSTS
We paid all the annotators the equivalent of $6 per question and provided them with a comfortable working environment, free meals, and souvenirs. We also provided the computer equipment and GPT-4 interface required for labeling. We labeled about 2,000 questions total and employed them to check the quality of the questions/answers, and the total cost was about $12,000 in US dollars. The annotators checked the derived tasks, including multilingual code explanation and code completion.
A.3 DETAILED ABOUT MCEVAL-INSTRUCT
Here describe in detail the specific methods and processes of code sampling and quality control in the process of constructing MCEEAA-IINSRRUCT.
Code Sampling
(1) We crawled a large amount of code data from GitHub and preprocessed the data according the StarCoder processing flow. Then, we can get high-quality code dataset.
(2) For each programming language, randomly sampled the data to build original dataset containing 40 languages, where the instances of each language are sampled to the same number of samples.
(3) In addition, in the refined stage, we further used GPT-4 to refine the code snippets to the clearer and more standardized code (the code with doestring, code comments, and clear variables) and verified the refined code snippet through static code analysis and unit testing methods to ensure the correctness and quality of the code.
Quality Control To check the correctness of the generated code from the GPT-4, static analysis and unit testing to ensure the accuracy of generated data as much as possible:
We use static code analysis tools and Abstract Syntax Tree (AST) parsing to ensure the correctness of the code.
Code Correctness:
Automated Testing: We leverage LLM to automatically generate unit tests, filtering out code that fails to pass these tests.
Harmful Content Filtering: We filter out code snippets containing harmful information using keyeord-based filters.
18

---

Published as a conference paper at ICLR 2025
A.4 EXAMPLES IN McEVAL
Figure 10 three examples of multilingual generation.
In Figure 11 we show three examples of multilingual explanation.
In Figure 12 we display three examples of multilingual explanation. The three examples from left to right correspond to the span completion task, the single-line completion task, and the multi-line completion task.
Instruction: Language isp
Instruction: Longuage
Instruction: Language R
Comerr Lis pertion "pliran cendee gist watter
Muig ad comana LUNe, compure Ew fillosing taac:
Wettee foortiie Teegrst tncrusiing
data/lat.caa
that Iuaction taoss lust passtivn cocagen
Nillew:
this sucction inchu ug jid in taak sactar trbegers return
possible
Bania: 05132000 1200-800.75
that thete dight whne and iot pirlas
Heng Fetoa 600UCTE) :158192500
the
ana meatee la Kangrat memter 1255 45)
songhing 4131/083 12 100-1011175
cntetral withou tha cleg their wrker The lenghs of
coeate La wraeetaaaaaa (Ek 93)
Segr stariiin 15%h 19:
"5000"
Comnaee
Reference Solution
1he entr
Sortua Craate agsit (naiban))
the firet sate Ccocastanuting the first excand cilurre reaber,
Reference Solution
[cale Caent sie alies Langg ovecablecaa))
Ploue tio full 23
caluere thre derettes
Lengest Lec wer.eaannnnn saarauane (eectorrengaeee)
(radice L100044 di Ecppattrntt latring acc 23) str-
vaten ""131
Reference Solution
2010 aat
beriie string malkarg)
jeliteei Cooeroiioaa wate "string 10 puanatouli "chrog b-40))
of (oequncel)) sell) sepurcsc) Ms 1a6)]) > segaes)
Test Cases
Wefhe coeate Lange memter- E1
Cases
sarchin))
Test Cases
"Sede" y
Cestert (etringg corate--argeet-voe
Cxlrttge Erevdte Longrt witer 14 127 90)
56))
check shallefree texe-rre)
802 wttpggaetttioorrrnnnnnoooooo 7,
stapi tespifetttagg
Feaction))
Cenort Cytrting fenette longsttttite (19 2331
23. 14, mi)
Estringe (cnaster gest suiterr (4.0) 400)
*404b4ey 313
wwwttcrrete-arger-toooe number)
Figure 10: Examples of multilingual generation. The data mainly consists of an instruction part (including function name, function description, and function call cases), reference solution, and a test cases part. Left. Shows an example of the Lisp language. Middle. Shows a file processing programming task in AWK language. During the evaluation, the corresponding file processing result by the generated code will be compared with the reference answer. Right. Shows an example of the language.
Instruction: Language Kotlin
Instruction: Language
Instruction: Language ITTML
midertaersissii sttaraaterr Di05 dissouu-dit, tres
Ful tahrteetran: THth: bocien
(1 in aell men)
reture falze
rotare the
2uutertemen 33 (1avieect) patns aijtarrry 2D11
pairs
-ndirme ent, Lottte
guage description (deattiigg teglish
towide whene tarpure Amown vee strrect
Reference Explanation
Waddd a conciss autural languge cescription teacttrin) 1IM Englts estec west see churacters.
Reference Explanation
Give nay wigetive lnteger nas, rapeatadly ald all ita digits realt rigit.
give canter and the pair cirters by A lie pair catined as simers wtere the
Reference Explanation
Exmegt::
Taola wits beo rowt and the columa, with colame rams and orl The taste contett tudes its. coraazeding ostumn maneery
ECh, 1h, 13
Decausn La the rextt.
stace only ons dight,
eal.
[EL, 10, V0, Ob, 50, (3%, 200, (8), H31, (40, 481, (99) 60), (%, 193
Figure 11: Examples of multilingual explanation. The data mainly consists of an instruction part (including complete function), a reference Explanation. Left. Shows an example of the Kotlin language. Middle. Shows an example of the Lua language. Right. Shows an example of the HTMI. language.
19

---

Published as a conference paper at ICLR 2025
Iastraction: Language C++
Instruction: Rust
Instruction: Language Shell
Sclor za saplansien of Shult code and stoep
Felte explomation CP code txcepletee
inslouretstin
DDostring:
are Eive them rntrpen is tu r fied usse occar wastly ande.
arr
You Lus depeazt vitrersal
the remker pets Paa be baught with pnaunt soney.
lask
peant
The TeaecTic tiees integsts, and by iepat meryy acatiaste,
et. thet pount casciuee alouls retare True. Ottervses
Jaan part
extru dhad
that
Inconslaca
late Code:
corviceeretinn wew.erteeeeeeeeeS boett
for euanple, of aud and buly the faxcttie wiTl reture
6f
hellenn traei))
Encerslate powd)
fallas
Plaaze <1Ll the (WAa) andespte tines of code say be
local www.aoreee torta too
rasted octs and unitee the compltts facction
Reference Solution
Moone (NOS itattt lanp pade
le pereCauerar www.sre........o........ (ND beelc
Reference Solution
for aperec2aro balorce ballace 101 retum bratt
(8 Loulate gevill
actras Lite Lirt-k. Us of
returs C1
tutel Luoedd 10 (uoo))
<3
falon;
local persuc chotta 3ian price.per.pee)
Stas
P
Test Cases
2R
Test Cases
nal tects
cxtiztutt, macupeeatl
#bbac]
reautoniicate bati aetuaeet_oo po1 5.31
assem tha cereieed
faduet:
7, Sensurty 11
oith tatsels
resutbrer
mated's
Cent
seaer (ealon seratved
treete
-00
traocted
If Bresalt"
Sreattt's cest
resteen
antten (eet aera 129900000,, assent Coat ritarie (19800067, Seccaane, 311
-00
If "Seasate -s a. I Lisaa Teat fasted: Eepecsed
-40
resuttee late seno 31 at
Text fastad: Expected
droweli
Figure 12: Examples of multilingual completion. The data mainly consists of an instruction part (including a incomplete function reference complete code solution and test cases. Left. Shows an span completion example of the C++ language. Middle. Shows an single line completion example of the Rust language. Right. Shows an multiple line completion example of the Shell language.
I Instruction Language Json
Rerference Solution
Creaze 250V represents collection kebsites, each key detalls: nane URL. sites objects.
"sites": 1
object reprosents website "Coogle" www.goooee.c.e"
"romsel
"ur1%: "Wek, google.
"Google"
object describes website "waibo" with www.weebo.comm
This structure afficiently organizes these popular websitts their most essential lidntifierss.
www.waebbb.ooo
S
Test
"sites":
"situs"s ts [
furi":
"Niwe wwwglo..ooooooooooooooooooooooo. con")
"namp"" "Google".
"nam"" "Gongle
match
"url": Twak "www.goog www.ggogggg.o... come
"urd": Vaw www..wwww.......ooooo.oo conb"
Trume": "Maiboo",
"nare": "Metbo"
Turi": "wwn www.weeboo.oom wetbo.coot
All subcomponents match exactly, test passes
Figure 13: Examples of Markup language (Json) generation task evaluation.
A.5 EVALUATION
For programming languages other than markup languages, we use an execution- based correctness metric by running the code with the provided test cases. For markup languages, use the Exact
20

---

Published as a conference paper at ICLR 2025
Table Runtime environments for different programming languages.
Language
Runtime Environments
AWK
GNU bash, version 4.4.20( --relcasc
C
gee (Ubuntu 7.5.0-3ubumtul 18.04) 7.5.0
dotnet 8.0. 1001
CPP
g++ (Ubuntu 7.5.0-3ubuntul (8.04) 7.5.0
CoffeeSeript
CoffceSeript version 1.12.7
Common Lisp
SBCL 1.4.5.dcbian
Dart
Dart SDK version: 3.3.1 (stable)
Elixir
clixir 1.3.3
Emacs Lisp
GNU Emacs 25.2.2
Erlang
Erlang/OTP [erts-9.2]
F##
dotnet 8.0.100
Fortran
GNU Fortran (Ubuntu 7.5.0-3ubuntul 18.04) 7.5.0
go version go1.18.4 18.4 linux/amd64
Groovy
Groovy Version: 4.0.16 JVM: 17.0.9 Vendor: Oracle Corporation OS: Linux
HTML
Haskell
The Glorious Glasgow Haskell Compilation System, version 9.4.7
Json
Java
javac 11.0.19
JavaScript
Node.js v16.14.0
Julia
julia v1.9.4
Kotlin
kotlinc-jvm 1.9.21 (JRE 17.0.9+11-LTS-201)
Lua
Lua 5.4.6 Copyright (C) 1994-2023 Lua.org, PUC-Rio
Markdown
PHP
PHP 7.2.24-Oubuntuff 10..tt....8....... 18.04.17 (cli) (built: Feb 23 2023 13:29:25) (NTS)
Pascal
Free Pascal Compiler version 3.2.2 [2021/05/16] for x86_
Perl
perl 5, version 26, subversion (v5.26.1) 1) built for x86_64-hi 64---44-lnuxx--
PowerShell
PowerShell 7.4.0
Python
Python 3.8.12
R version 3.4.4
Racket
Racket v6.11
Ruby
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86 (266664-linu-gmuu)
Rust
ruste 1.74.0 (79c9716c3) 2023-11-13)
Scala
Scala code runner version 3.3.1 Copyright 2002-2023, LAMP/EPEL
Scheme
Racket v6.11
Shell
GNU bash, version 4.4.20(1) --reeaase (x86 64 64-pe-linux-gnu Swift version 5.9.2 (.wift-...22RELEEA..
Swift
Tel
telsh 8.6.11
TypeSeript
Isc Version 5.3.3
VimSeript
VIM Vi IMproved 9.0 (2022 Jun 28, compiled Dee 20 2023 18:57:50)
Visual Basic
dotnct 8.0.100
Match metric for evaluation. Taking Json an example, we parse all subcomponents in Json. If the model result is exactly the same as the subcomponent of the reference solution, the model generation result considered correct. An example of Markup language (Json) is shown in Figure 13
We adopt the greedy Pass @1 1 (%) metric Kulal et al. 2019 Chen et al. 2021) for our evaluations. For closed-source models, we generate answers through the official API service. For open-source models. we prioritize using vLLM [ Kwon et al. 2023) for faster inference if the model is supported by vLLM. Otherwise, we perform inference with the Distributed Data Parallel (DDP) module from PyTorch. For the code generation and code completion tasks, we extract the functional part of the code from the model outputs and combine with corresponding test cases form compilable and executable code. For the code explanation task, we adopt two-pass generation approach Loddetto-aaturl-Laanne and Natural-Lam Natural-Language-tu Lagguage-t--Codee The extraction and execution process for this task consistent with the previous two tasks. We conduct all evaluations in a Docker environment. Detailed information on the code compilation and execution environment are displayed in Table 6) We have uploaded the Docker image to docker hub to facilitate the reproduction of results and the evaluation of new models.
21

---

Published as a conference paper at ICLR 2025
Qwen225-Cuules 7B-Itatuut
Magiodde-SS-B S-D------------1
Co (Our)
Coddicana-TBB
Opentoder
Instract
CodeOweet 15:7
Code instract
0
Average
Clisp
Coffee
Elisp
Elixir
Enlang
Ginowy Haskell
Html
18
Java
Json
Fulia
Kotiin
List
РHН
Pascal
Port
Power
Python
0-
Racket
Ruby
Rust
Scala
Scheme
Shell
Swift
1el
Viml.
MD
Figure 14: Pass@1 1 (%) scores of different code LLMs (<10B) for multilingual code generation tasks on McEVAL. "AVG" represents the average scores of all code languages.
Qwen? S-Cudder-22H--neeee 128i-nnrrcct
Clodel Inmas 54ft-nemme
DS Entte
DBs-Cuda-V1-33--lee
50
Average
AWK
C++
Co
Clisp
Coffee
Dart
Elisp
Elixir
Errlong
F#
Fortran
Go
0
Groovy
Haskell
Html
Java
Json
Julia
Kotlin
Lua
PHPP
Pascal
Perl
Power
Python
Racket
Ruby
Scala
Scheme
Shell
Swift
TS
VB
Viml.
MD
Figure 15: @ass@1 (%) scores of different code LLMs (1OB 40B) for multilingual code generation tasks on McEVAL. "AVG" represents the average scores of all code languages.
A.6 OPTIMIZATION DETAILS
All MCODER models are fine-tuned using 8 NVIDIA A800-80GB GPUs. The models trained for 2 epochs with a cosine scheduler, starting at learning rate 2e-5 and incorporating a 3% warmup phase. Training model takes about 5 hours. We used AdamW (Loshchilov & Hutter 2017) as the optimizer and a batch size of 512 with a sequence truncation length of 4096. We use Py Torch''s 's Fully Sharded Data Parallel (FSDP) perform distributed training of the model, and use gradient checkpointing technology and gradient accumulation to save memory and achieve training with larger batch size.
A.7 EXTRA RESULTS
A.8 PROGRAMMING CLASSIFICATION
As shown in Table and Table 8 we comprehensively display the code generation performance of the models we tested across various programming paradigms and application scenarios.
22

---

Published as a conference paper at ICLR 2025
ol-preview
cluude-333
D6-Cudes-V2 In-ttrruc
ol--niai
31240620
got
ppt mini 07:18
0
Avergge
Clisp
Coffee
Elisp
Elixir
Erlang
Groovy
Haskell
Huml
Jancs
Json
Julia
Kotlin
Lue
PHP
Pascal
Porl
Power
Python
0
Racket
Ruby
Rust
Scala
Schemme
Shell
Swift
Tal
Viml.
Figure 16: Pass (%) scores of different code LLMs (Closed Source & 200B+) for multilingual code generation tasks MCEVAL. "AVG" represents the average scores of all code languages.
22--0-3124405-5
got-3 100-5-5-rr--200125
Magicoder
g1-4-att--233066
DS-Cadaa-V14e TB-ssiist
CelleQwall---B--C
Codelllume- 13B-aattmc
50-
0-
Average
Clisp
Elisp
Erlang
50
Groovy Haskell
Huml
J5
Java
Json
Julia
Kotlin
Lina
PEIP
Paseal
Perf
Power
Python
0
Daxs.
Racket
Ruby
Scala
Schome
Sbell
Swift
Tel
VimL
Figure 17: Pass@1 1 (%) scores of different code LLMs for multilingual code explain tasks on MCEVAL. "AVG" represents the average scores of all code languages.
A.9 MCODER RESULT
show some extra MCODER @ass@1 @1 1 (%) results multilingual code generation tasks. We evaluate the base models CodeQwen- 1,5 and DeepsSeek- Boreeekk-Corr--le 1.5 respectively. In addition to CodeQwen-1 also selected DeepSeek- Codeke--oorrr--eee 5--b-bbee the base model for fine-tuning.
In Table
A.10 PARALLEL QUESTIONS ACROSS LANGUAGES PROGRAMMING GRAMMAR
Due the large number of languages, most language annotations, we follow the characteristics of the language and perform independent annotations. For example, structured languages such as Markdown and HTML need independent annotations. For some similar languages, such as Typescript and Javascript, use parallel annotation on some data.
difficult ensure parallel problem annotation. For
we analyzed the programming languages in the MCEVAL from the represen- tation perspective. We used CodeBERT (Feng et al. (220)) to extract code representations from code snippets in MCEVAL. These representations were visualized using t-SNE (Van der Maaten & Hinton 2008) and hierarchical clustering (Murtagh & Contreras 2012) methods. The figure clearly shows that languages with similar syntax have closely related representations. For example, other functional programming languages similar to Common Lisp. as well as C, C++, Java, and scripting languages, exhibit high grammar similarity.
As shown in Figure 19
23

---

Published as a conference paper at ICLR 2025
TSS D5-Cadee-11- Micharaa
Codrt Totteaa
Codatjonn
1000
Awerage
AWK
Coffee
Flop
Friling
100
Grounyy
Haskell
Huni
Julin
Kodiin
PUP
Paeasal
Power
50
Racket
Ruby
Rust
Scala
Schreme
Shall
Swill
15
Tel
VB3 VinL.
MD
(a) Single-line Completion
1.00
Ascrage AWK
Clisp
Coffee
Durt
Elsp
Elisir
Briung
Forzan
30
Grouvy
Haskell
Huml
Kollin
PHP
30
Backet
Kuby
Rusi
Scala
Schemne
Shell
Switt
15
Tel.
V8
VimL
MD
(b) Multi-line Completion
Anerage
AWK
C
C+f
C#
Cliep
Coffee
Dart
Flisp
Elissr
Firling
Fortran
Gn
01
Gooony
Haskell
Huml
Kodia
Pydhoa
50
Racket
Ruby
Rist:
Seala
Scheme
Sheell
Swift
15
Tel
VB
VimL
MD
(c) Span Completion
0-
Anarage
AWX
C++
Clisp
Culloc
Dust
Elle
Elisic
Edlang
Forman
Gio
0-
Geonty
Homl
18
Julia
Kattin
PHP
Python
50-
Ir.
Radket
Raby
Rust
Scala
Scheme
Shell
Swilt
18
Tel
Vis
VinL
MDD
(d) Span Completion Light
Figure 18: Pass @ 1 (%) scores of different models for multilingual code completion tasks on MCEVAL. "Avg" represents the average scores of all code languages.
24

---

Published as a conference paper at ICLR 2025
Table 7: Pass @ ((%) results of code generation performance of across various programming paradigms
Method
Procedural
Object Oriented
Muliiple Paradigms
Functional
Markup Language
GPT-4o (240517)
58.0
79.8
65.9
67.0
46.0
GPT-4 Turbo (231106)
78.7
65.2
59.3
GPT-3.5-Turbo (240125)
66.8
$7.6
44.3
39.3
Codegemma-7b-it
19.3
46.6
34.0
16.3
34.0
CodeL 13--u-------tsuu
21.3
32.0
27.0
32.3
28.0
CodeLlama-334--mmee
27.3
33.6
28.0
30.0
30.7
CodeLlama-7b
20.3
28.1
23.4
26.7
30.7
Cod=Qwen-115-7--CH
57.3
46.3
41.0
37.3
Codeshell-7h-ccat
16.0
24.1
25.7
14.0
34.7
Bodestral-22B-v0)
40.0
67.6
54.1
39.7
40.7
DeepSeckCoders SadbeockCCor------ittteeeee
52.7
62.8
56.3
$2.0
34.7
DeepSeciCooleer155-77--
51.8
41.0
40.0
Magicoder-S-DS-6.7B
45.7
58.5
49.4
49.0
32.0
Llama-3-8B-Instee
27.3
44.7
38.0
32.0
33.3
Nxcodo-CQ-7B-orpo
40.7
54.9
45.5
41.3
36.7
OCTOCODER
20.7
28.9
21.9
25.0
25.3
OpennCodellmeerpeeeerDDSSS. DS-6.7B
57.7
42.0
Phi-3-mediimm-4-iinsstet
32.3
43.1
36.6
26.7
35.3
Beenl..-72B-hhr
38.3
37.2
36.2
29.3
39,3
WizarCCoder- 15--rr--5-VV111 15.-V1.0
19.0
31.6
34.2
24.0
6.7
WizardCoder- Pytdde---t
43.9
38.2
33.7
MCODER
41.3
57.3
47.4
42.3
43.3
Table 8: Pass @ I(%) results of code generation performance of across various application scenarios
Method
Mobile
Desktop
Frontend
Backend
Scientifcc
General
Content
Education
Scripes
Editoe
GPT-4o (230517)
$4.0
68.3
75.0
66.7
64.6
71.6
$7.6
46.0
72.7
65.7
$2.0
(PT-4
74.0
46.7
50.0
GPT:3.5 (240125)
71.0
Codegemma--bb..
45.0
40.4
43.0
34.7
37.7
21.6
24.8
34.0
22.0
29.7
13.0
colo-I 13b
30.0
15.4
39.
34,7
28.0
23.2
34,8
28.0
24.0
27.7
13,0
Codel.lma-- Insmmmcttt
33.0
13.3
38,0
27.2
24.0
32.8
30.7
26.7
19.0
24.0
12.5
37,0
29.3
22.7
20.8
29.2
30.7
19.3
27.0
14,0
CodeQuen-11 1.5-c
55.0
442
$9.0
56.7
48.7
47.5
46.8
37.3
42.7
40.0
20,0
Codtshell-7hhchhe
23.0
144
26.0
40.7
26.1
17.2
21.2
34.7
13.3
22.7
8.0
Codestral-222
CeeySeekCoder--33
Conter-TD-iinstrett
50.0
$7.0
54.0
34.7
$6.7
DespSeekCoder. Sonder--.5.700
1.5-7b-insttr
40.0
42.3
$9.0
62.0
$2.7
40.8
50.0
40.0
34.7
46.0
22.0
Magicoder-S S-DS-6.78
49.0
43.3
64.0
611.7
50.4
49.6
$2.4
32.0
48.7
49,7
24.0
Hama---8R- Instrnct
30.8
48.0
Nscode-CO--BBorpp
46.8
OCTOCODER
22.0
20.2
33.0
28.7
21.8
16.4
27.2
25.3
16.0
29.0
14.0)
OpenC Inde Intepprter--1--4.
47.0
42.3
64.9
58.0
45.9
47.6
46.4
42.0
43.3
44.0
24.00
Phi- Plii-3-medium- Rk-instruct
Qwenl 5-728-Chat
29.8
43.0
44.7
30.4
38.0
39.3
32.7
21,0
WizardCoder Coder-15R-V7 V5R-
28.0
24.0
36.0
48.0
37.1
29.2
27.2
6.7
20.7
26.3
9.01
WicardCokkr-Pya Pyikkr-Pythom334BB
42.0
28.8
46.0
44.2
32.8
38.0
36.0
32.7
32.3
19,0
We selected training data from several languages in MCEVAL-INSTRUCT, which exhibit significant grammatical differences (approximately 10K samples of Python and 1K samples for other languages) and fine-tuned the model. The results are shown in Table 10
When trained using only Python data, the performance on Python and AWK improved. However, this led to the scores for TypeScript and JavaScript dropping to O. Upon inspection, we found that generated code for these two languages contained syntax errors (Less data may lead to instability in model training).
When training mixture of several languages, Python performance decreased slightly compared to using only Python data, while Scheme performance improved significantly. Furthermore, the syntax generation for TypeScript and JavaScript returned normal (even without adding JavaScript data, as TypeScript and JavaScript share similar syntax). However, there was no significant improvement compared to the base model.
Thus, fine-tuning multilingual code models presents significant challenges. Similar languages can provide mutual benefits, while languages with greater differences may negatively impact performance.
25

---

Published as a conference paper at ICLR 2025
Table Additional MCODER @ass@ (%) results on multilingual code generation tasks. "Avgali represents the average @ass@1 scores across all programming languages in the MCEVAL. Here, MCODER-DS indicates that the fine-tuned base model is DeepSeekCoder- 1.cdler---55-111 1.5-7b-base
Matked
Sizz
.AXK
C
C+-
C#
Clsp
Datec
Dan
Elip
Eloe
triue
Forna
26
Go
Greory
Hasl
Jua
15
Jice
Jula
32.
910
SL.
15:1
11.1
41.
36.3
4b
33
31.
16
16.0
400
463
41.5
26.
360
2.5
14.
14.0
0.0
400
46.3
44.0
52.3
24
80
41.2
328
313
50.0
MCHuER
315
72,8
26.0
31.
364
48.0
Natked
Kocas
PHP
Vieee
Pytka
Racha
Raby
181
Ark.
Cordee how
12.
30.0
360
34.0
36.
42.9
54.1
24,8
30
16.0
76.6
41.0
20.0
18.3
13.0
64.0
23.0
31.
26.0
46,0
46.0
240
95.0
96.9
36.0
64.0
18.0
400
62.
480
48,1
12.
300
42.0
35.0
32.0
$4.0
64.11
60.1
36.0
68.0
63.0
56.0
44.0
46.3
64.
360
5339
59.0
310
467
Shall
Lisg
Tel
Common Laop
Ped
Razket
Scheme
Shall
CPP
ExpeScript JantSertpe
Java Groog
Representation visualization based on I-SNE
(2) Representation visualization based on Hierarchccl Cluster
Figure 19: Analysis from the representation perspective McEVAL. Languages with similar syntax have closely related representations
Table 10: Preliminary explorations the impact of finetuning across different languages model performance.
Setting
Python
Scheme
TypeScript
JavaScript
AWK
1 base
38.0
34.0
52.0
52.0
38.0
Python + PythontkSchemmc&TyyyScciittee
48.0
12.0
0.0
0.0
40.0
44.0
38.0
50.0
48.0
42.0
A.11 DETAILED RELATED WORK
Code Large Language Model. In recent years, numerous large language models (LLMs) have been developed specifically for code-related tasks. For the field of soft engineering, code LLMs (Feng
al. 2020; Chen al. 202T, Scaa al. 2022
Li
2022
Allal al.) 2023 Fried al.
2022 Wang et al. 2021 Zheng et al.) 2024
Guo et al., 2024)
pre-trained on billions of code
snippets, such as StarCoder (Li et al. 2023 Lozhkov et al. (024). CodeLlama (Rozière al.
2023), DeepSeekCoder (Guo et al. 2024). and Code-Qwen (Bai et al. (202)). The development and refinement of code I.LMs have been pivotal in automating software development tasks, providing code suggestions, and supporting code generation/translatio
To improve the performance of code generation, researchers used optimized prompts (Liu et al. 2023a Reynolds & McDonell 2021 Zan et al. 2023 Beurer-Kellner et al. 2023). bring test cases (Chen et al. 2023) and collaborative roles (Dong et al. 2023). There are also some related studies on using large language models for other code tasks, such as dynamic programming (Dagan et al. 2023). compiler optimization (Cummins et al. 2023). multi multi-lingual prompts (Di et al. 2023), and Program of Thoughts (Chen 2022).
26

---

Published as a conference paper at ICLR 2025
Code Evaluation. In the domain of code evaluation, rich tapestry of benchmarks (Zheng et al. 2023b Yu et al. 22224] Yin et al. [022] Peng et al. 2024; Khan et al. 2023; Orlanski et al. 2023) has been woven to address the challenges of accurately assessing code quality, functionality, and efficiency, such as HvmanFval (Chen et al.
2021), MBPP (Austin et al.] 2021), their upgraded version EvalPlus (Liu et al.] 2023b). Studies have explored variety of approaches, ranging from static analysis techniques (c.g. exact match (EM) and edit similarity (ES)), which examine code without executing it, dynamic methods that involve code execution in controlled environments (e.g. Pass @ k). The current benchmarks support code models to evaluate series of different types of tasks, such code understanding, function calling (Zhuo et al. 2024), code repair (Lin al. 2017]
Tian et al. 2024 Jimenez et al. 2023 Zhang et al. 2023 Prenner & Robbes 2023 He et al. 2022
code translation Yan et al. 2023). Recently, many works Wei et al. (2023): Zhuo et al. (2024) have
leveraged LLMS to construct large-scale evaluation datasets and instruction-
corpora, further enhancing the evaluation and performance code models. In our work, we used a similar approach to
construct instruction dataset and proposed the Cross-lingual Code Transfer method to expand the
number of languages to 40. Some recent works pay attention to the multilingual scenarios (Cassano
et al. 2023 Wang et al. [203] AAthiwaratkun al., [023] [Zheng al.) [203aa] Peng et al. [2024]]
Zheng et al.
2023b) by extending the existing python-only HumanEval or MBPP benchmark, such as
MultiPL-E
Cassano al., 2023) and MBXP (Athiwaratkun et al. (023), which is challenged by the
number of the covering languages and data leaking problem (Li et al. 2023 Jain et al. 2024)
27


Reviews:
[{"review_id": "Kqzr0t5pbf", "reviewer": "Reviewer_npVu", "summary": "The authors propose a new code benchmark called McEval. The proposed benchmark has 40 programming languages with 16k samples. The benchmark spans code generation, code completion and code understanding tasks. The authors also introduce mCoder trained on the McEval-Instruct corpora.", "questions": "1. Is the dataset being released?\n2. Have you evaluated [Granite code models](https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330), it would be nice to have them in the report since there are very few SOTA code models available under Apache 2.0 license?\n3. Are the human annotators fairly compensated?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written and is very easy to read and follow. The paper is also a significant leap towards better evaluation for code models. Current models are mostly evaluated on smaller benchmarks like HumanEvalPack or MBPP which have very few languages (<10) and this limits the ability to effectively evaluate code models.\n\nThe authors' work is quite significant since it introduces a new benchmark with 40 programming languages which is more than 4x of any previous benchmarks. The paper also presents an instruction corpora.\n\nI overall like the paper and the amount of experiments done by the authors.", "weaknesses": "The instruction corpora is generated using GPT-4 and thus is commercially unusable. I understand that collecting good human annotated instructions is a significant undertaking and might be infeasible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "diLp5ZkLXK", "reviewer": "Reviewer_1Wxp", "summary": "The MCEVAL benchmark introduces a challenging, multilingual evaluation framework for code LLMs, covering 40 programming languages with 16K test samples. Alongside this, MCEVAL-INSTRUCT - a curated multilingual instruction corpora, and MCODER—a multilingual coder trained on MCEVAL-INSTRUCT are introduced.", "questions": "Is there any study done to validate the accuracy or correctness of  MCEVAL benchmarks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(1) Extensive study and effort has been spent in generating and documenting this code benchmark.\n\n(2) While HumanEval and MBPP are the popular code programming benchmarks today, Code LM community needs additional multilingual programming benchmark. Hence, this paper addresses a problem statement that is of demand today.", "weaknesses": "GPT models, like, gpt-4-1106-preview, has been used to generate the problem description for code instruction corpora. That could have influenced GPT models to lead in the benchmarks with significant performance margins over other models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n2mno68bmX", "reviewer": "Reviewer_kkUn", "summary": "This paper introduces MCEVAL, a massively multilingual code evaluation benchmark covering 40 programming languages with 16,000 test samples. It aims to advance the evaluation of code LLMs in multilingual scenarios. The benchmark includes tasks for code generation, completion, and explanation. Additionally, the authors present MCODER, a multilingual coder trained on the MCEVAL-INSTRUCT corpora, demonstrating its effectiveness in multilingual programming language generation. In its experiments, this paper highlights the gap between open-source and closed-source models and provides a comprehensive framework for evaluating and improving code LLMs across diverse languages.", "questions": "1. Table 2 shows that MCEVAL has 16,031 questions, but in reality, there are only 2,007 completely independent programming problems. I think that comparing the benchmarks based on independent programming problems would better illustrate the quality of each benchmark, as a benchmark can increase its number of questions by \"creating new tasks\", but it cannot do the same to generate new independent programming problems.\n2. The fonts in the leaderboard tables are too small. \n3. The programming tasks differ between languages, making it hard to directly compare a model's capabilities across different languages.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This is truly a BIG project. The authors have made significant contributions to the field of multilingual code generation.", "weaknesses": "1. Although this work makes a significant contribution to benchmarking the multilingual code generation capabilities of LLMs, its main contribution is labor-intensive, with limited technical contributions or insights (MCEVAL-INSTRUCT looks similar to MagiCoder by generating instruction data from code snippets).\n2. Is it meaningful to benchmark code LLMs on 40 languages? Can the author elaborate on why it is important to measure the generation capabilities of code LLMs in 40 languages simultaneously?\n3. Based on the experimental results, MCEVAL and MultiPL-E have similar measurement capabilities, which diminishes the significance of MCEVAL compared to MultiPL-E.\n4. It is said that \"three volunteers are employed to evaluate the correctness of the benchmark (> 90% accuracy) and correct the errors\" but no details are provided. Can the author explain how to ensure that the entire benchmark has a 90% accuracy rate? How was the \"90%\" quantified (we need ground truth to calculate the 90% right)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "doZfI9LUGw", "reviewer": "Reviewer_3QMs", "summary": "This paper presents MCEval,  a massively multilingual code benchmark covering 40 programming languages with 16K test samples. MCEval provides a new evaluation benchmark in multilingual scenarios. In addition, the authors also present a new instruction dataset MCEVAL-INSTRUCT for multilingual code LLMs. The experiments results suggest that the benchmark MCEval could measure the gap between open-source and closed-source models.", "questions": "1. What impact will the Unbalance on Different Languages (in section 5) bring to the evaluation process of code LLMs？\n2. What kinds of languages will help code LLMs to generate better? What will do harm?  These problems maybe more insightful for multilingual code evaluation of coding LLMs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper is easy to follow. The figures presented are clear and easy to understand.\n2. The benchmark MCEval is comprehensive, covering 40 programming languages with 16K test samples.", "weaknesses": "1. The contribution of MCoder is unclear.\nThe contribution of this paper is comprise of two parts, a multilingual evaluation benchmark MCEval and a multilingual code LLM MCoder (trained on an instruction corpora MCEVAL-INSTRUCT). However, the performance of MCoder (using gpt-4 to generate data) are still behind the previous sota code LLMs like magicoder who uses gpt3.5 to generate instruction data.  The authors claim that MCoder is used as a strong baseline for MCEVAL(line 073), but why others don't use a better model (Magicoder) but MCoder as their baseline?\n\n2. The method of constructing dataset is lack of novelty.\nThe biggest contribution of this paper seems the number of programming languages included in their dataset. However, the method of constructing such their dataset is using LLM to generate and refine from the code snippets, which has been used in many previous works such as bigcodebench(https://arxiv.org/pdf/2406.15877).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)