Multi-lingual Evaluation of Code Generation Models
Authors: Ben Athiwaratkun
Sanjay Krishna Gouda
Zijian Wang
Xiaopeng Li
Yuchen Tian
Ming Tan
Wasi Uddin Ahmad
Shiqi Wang
Qing Sun
Mingyue Shang
Sujan Kumar Gonugondla
Hantian Ding
Varun Kumar
Nathan Fulton
Arash Farahani
Siddhartha Jain
Robert Giaquinto
Haifeng Qian
Murali Krishna Ramanathan
Ramesh Nallapati
Baishakhi Ray
Parminder Bhatia
Sudipta Sengupta
Dan Roth
Bing Xiang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: code generation
execution-based evaluation
test-based evaluation
language models
multi-lingual code generation benchmark
code insertion
code summarization
robustness for code
code translation
zero-shot code translation
multi-lingual
mono-lingual
language models.

Abstract: We present two new benchmarks, MBXP and Multilingual HumanEval, designed to evaluate code completion models in over 10 programming languages. These datasets are generated using a conversion framework that transpiles prompts and test cases from the original MBPP and HumanEval datasets into the corresponding data in the target language. By using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of  few-shot prompting to teach the model new languages, and zero-shot translation abilities. In addition, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks.

Published as a conference paper at ICLR 2023
MULTI-LINGUAL EVALUATION OF CODE GENERATION MODELS
Ben Athivaratkur Sanjay Krishna Gouda', Zijian Wang', Xiaopeng Li Yuchen Tian', Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Glaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, Bing Xiang
AWS Al Labs
ABSTRACT
benchmarks, MBXP and Multilingual HumanEval, designed to evaluate code generation models in over 10 programming languages. These datasets are generated using a conversion framework that transpiles prompts and test cases from the original MBPP and HumanEval datasets into the corresponding data the target language. By using these benchmarks, we are able to assess the performance of code generation models in multi- multi-lingual fashion, and discovered generalization ability of language models on oft-of-domain languages, advantages of multi-ling lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities. In addition, we use our code generation model to perform large-scale bootstrapping obtain syn- thetic canonical solutions in several languages, which can be used for other code- related evaluations such code insertion, robustness, summarization tasks.
We present
INTRODUCTION
Code completion by machine-learnin models has great potential to improve developer productivity (Barke et al.] 2022). This line of research has seen tremendous progress with several models recently proposed such Codex (Chen et al. 2021), CodeGen (Nijkamp et al. 2022). PaLM (Chowdhery et al. 2022), BLOOM (Mitchell et al. 2022). and InCoder (Fried et al. 2022).
One key component for code generation research is how to evaluate such program synthesis abil- ities. In the literature, two primary evaluation approaches emerged, namely, the match-based and the execution-bass
evaluations. For both approaches, each problem contains prompt which model uses as input to generate candidate body of code. The match-based evaluation compares the candidate code against reference source code using n-gram metrics such as BLEU, whereas the execution based evaluation executes the candidate code against test cases and calculates success rate. The execution- based evaluation has benefits over the n-gram evaluation in that it permits solu- tions that are functionally correct but might not be equivalent to the reference solution in terms of the exact implementation. Since the release of datasets such as HumanEval (Chen et al. 202I) or MBPP (Austin et al.) 2021), the community has been widely adopting the execution recutioo-bbseed approach as a primary tool to evaluate program generation capabilities. However, creating execution-based evaluation datasets is time-consuming since it requires careful construction of test cases check the correctness of the code's functionality. Such difficulty leads to limited available of execution- based evaluation data. For instance, date, many exccution- baased datasets contain only problems in Python.
*Corresponding authors (oonoth:, akgouda, zi jwan, xisopel, tlayuche, bww.nggia)ggg) 8ama 'We release the data and evaluation ende at https: thu. thub. com ama zon research mbxp-exec-ee execeeveev

---

Published as a conference paper at ICLR 2023
MEXP
Exacution
(8) Other Evaluation
Original Data
Languages
based Evalustion
Synthesie
Solution
transiaiion
[E-1]
code insertion
[E-2]
summanication
(F-3)
robustress
[E-4]
prompt
prompt
SOUICO
prompt
prompt
prompit
ectracted
perturbed
prompt
(A)
regenerate
source
context
suammary
solution
Language Conversion
solution
Model
solution
solution
test
Framework
test
X
keep
prompt
target
context
Model
Figure 1: Benchmark Construction.
In this work, we propose scalable framework for dataset convertion from Python many different languages. While translating code from one language to another is typically a non-trivial task, it possible convert existing execution-basee datasets to another language by transforming only prompts and test statements (see Figure D part A and Figure 2 That is, the purpose of evaluat- ing function completion ability, we do not need the canonical solution since it is not used during evaluation. The function signature prompts and test cases of basic programming problems involve sufficiently simple data structures that can be analyzed to synthesize dataset new languages. With- out having to translate the generic function body of code to another language, the conversion process becomes possible via a rule- based transpiler.
The result of such conversion two benchmarks, MBXI and Multilingual HumanEval, which are derived from the original Python dataset MBPP (Austin et al., 2021) and HumanEval (Chen et al.) 202T We provide the evaluation data in many languages besides the original Python, namely, Java, JavaScript, TypeScript, Go, Ruby, Kotlin, PHP, C#t, Scala, C++, Swift, and Perl, with plans for more language expansion in the future. Along with these datasets, we also release code package to perform execution in all supported languages. In addition, our conversion framework is easily ex- tensible and allows us obtain the multi-lingual version of other existing datasets such as MathQA (Schubotz et al.) 22019). In the main paper, we provide results and analyses mostly on MBXP where the results on Multilinuual Human and MathQA can also be found in Appendix [
Our benchmarks also support other code complction tasks such as code insertion or translation in many languages. This extension is made possible by performing large-scale bootstrapping to syn- thetize solutions (Section O.I.1I The result of our dataset conversion framework and the solution synthesis process is, to date, the first multi-lingual execution- with canonical solutions, which can be adapted for many code-related evaluations. In this paper, we process MBXP for multiple use cases, namely, for zero-shot translation t-MBXP, prompt robustness I-MBXP, code insertion i-MBXP, and the summarization s-MBXP.
evaluation benchmark equipped
Overall, the constructed datasets provides us new opportunitics to explore many facets of code gen- eration abilities. In this work, we conduct large scale evaluation where we train models of various sizes spanning three orders of magnitude (from ~ 100M to ~ 10B parameters) in both multi-lingual and mono-lingual settings. We analyze results from hundreds of thousands of code generation sam- ples to investigate the models' code generation abilities with respect to in-domain versus out-of- domain languages, the effectiveness of few-shot prompting, zero-shot translation abilities, robust- ness to prompt perturbation, code summarization, and code insertion.
2 FINDING HIGHLIGHTS
We provide the highlights of out findings below.
1. Given the same model size, a multi-lingual model often outperforms the best of mono- lingual models trained with equivalent training resources, especially when the models are sufficiently large. This observation indicates that it is beneficial train single model on all programming languages, and provided that the model size has enough capacity, the performance will be better than the best of monolingual models.
2. Language models are able generate code with correct syntax and pass unit tests in pro- gramming languages they are not intentionally trained on. We hypothesize that the data "spillover" effect, where code in one language present in other languages through code
MBXP stands for Most Basic X(Pychon//ava/GG//Rbyyy etc.) Programming Problems
2

---

Published as a conference paper at ICLR 2023
Funciton signature and docstring
def binomial, cectmnaal_cceffeee : s*e Python function find binomial coefficient. >>> binomial_coeff
class Binomialcooff
/Aa Write Java function to find binonial
efficient.
BinomialCoeff. binomialcoeff (5,
10
*/ public
binomialCoaff (int D, int X)
Model
if (n k) return 0g
Function body
int res 1;
return
k==0 =
For (int i
k: 1ee)
(n 1):
retura
binomial cooff (m-1,
res /= 17
binomial coeff binomial_coeff (n-1,k
return zes;
class Main
public static vold main (String[) azga) chrowa
Test cases
assert binomial www.ffofff........ (5,2) -- 10
Exception {
binomial coeff (4,3)
int x0 BinomialCoeff BicossstCCe....ooossssssoeeeeee www.omiilnooefffecc.e.. (5, 2);
assert binomial cooff (3,2) =
if (1x0 aixalliu {
throw java. lang Exceptioon ('Ecceptionn test case did not paes. x0 x0)
Figure 2: Conversion formatted MBPP (Python) MBJP (Java).
comments co-occurrences. Such amount of spillover data are enough for large language models to learn different languages that are embedded within the main language.
3. The occurrences of multi-lingual data in natural data also explains the superior performance multi-lingual over mono-lingual models. That is, the multi-lingual model can perform better on language A since can pick up and combine all knowledge of language A from the training data in languages A, B, C, etc. in the multi-lingual setting.
4. Few-shot prompting the model has not seen, significantly improving out-of-domain code generation abilities. Through error analysis, few-shot prompting helps reduce compilation or parsing errors that are the major sources of errors when it comes to programming language the model is not familiar with.
effectively help teach provide knowledge
new language
5. Language models have zero-shot code translation abilities; that is, even though they are not specifically trained to perform translation, they able reference code in one language to improve code generation in another language. Problems that are difficult can become much easier with access another language's solution. This observation holds for mono-lingual as well as multi-lingual models.
6. Multi-lingual models are also more robust to prompt perturbation and better at summarizing code.
CONVERSION OF EXECUTION-BASER EVALUATION DATASETS
In this section, we provide high-level details on the data conversion process. Figure 2 illustrates the mapping of the original Python prompt, consisting of function signature and a docstring, to an equivalent prompt Java (which we call a target prompt). The target prompt is a valid code including function signature from which the model can use to complete the function body. In the case of Java or typed languages, constructing the target prompt requires inferring input and output types. We perform such type inference by parsing the original test cases, taking into account heterogeneous data types. For instance, the first argument includes values of types int and float, we deduce it to have the most general type of all types encountered. The converted prompt also needs to work in harmony with the converted test cases. For instance, the Java test case in Figure refers to the defined class Binomin al alCoo Coe ff and the defined method binomia alCoeff in the converted prompt with appropriate function call based on the defined argument list. For more details including data validation and generated solutions via bootstrapping, Appendix 0
3

---

Published as a conference paper at ICLR 2023
4 MULTI-LINGUAL EVALUATION OF CODE GENERATION MODELS
From the previous section, we have established framework to perform dataset conversion, from which we obtain a collection of execution-bane evaluation datasets in 10+ programming languages. These evaluation datasets contain rich information in the prompts including natural language de- scription as well as appropriate function signatures that help steer model to generate code a particular language. Most importantly, they also contain test cases in the respective language that can be used to check code correctness, which is applicable for most types of evaluation in MBXP+. This section describes the training, evaluation setup, and findings from each evaluation task.
4.1 DATA AND MODELS
For the purpose of this work, collected training data in three primary programming languages, namely, Python, Java, and JavaScript, containing permissively licensed code data from GitHub. Fol- lowing Chen et al. (202T); Nijkamp al. (2022), we perform filtering, deduplication, and remove data that contains a significant amount of non-English text or is not parsable with respect to that lan- guage's syntax parser. We also ensure the original MBPP and HvaanEval datasets are included in data. After all the post processing steps, our dataset contains 101 GB Python, 177 GB Java, and 216 GB JavaScript data.
We use decoder-only transformers as the model architecture and train the models via next-token prediction loss (Vaswani et al., 2017; Brown al. 2020). We design our training compare multi- lingual versus mono-lingual settings by using the same compute budget for each language in both cases. In particular, we train mono-lingual models on 210 billion tokens with their respective lan- guages (Python, Java, and JavaScript) and train multi-lingual models 210 billion tokens from each language, with 630 billion tokens total. To study effects of model sizes, we train models of various number of parameters, namely, 125M, 672M, 2.7B and 13B. For the synthetic canonical solution process, we use a separate 13B multi- lingull model which we refer to as the 13B* model.
4.2 EXECUTION- BASED FUNCTION COMPLETION
We use pass@k scores (Kulal et al., 2019) with the unbiased estimate presented in (Chen et al. 2021 the metries for evaluation, where each task is considered successful if any of the k samples are correct. We generate up until the end of the function, such as end of indented function block for Python until the closing curly brace for PHP Go, for example (see Appendix C.2 for end of scope details). We refer to an evaluation language that the model is not specifically trained on as out-of-domain with respect to that model. Otherwise, the language is considered in-domain. For instance, Java is out-of domann for Python-only model and PHP is out-of-domain for our multi- lingual model trained on Python, Java, and JavaScript.
4.2.1 ACCURACY VS. SAMPLING BUDGET
Overall, we observe sigmoid-libe relationships between pass@k and sampling budget & across all datasets in MBXP where the performance increases smoothly & increases (Figure and Appendix F22) This trend is consistent with the original MBPP and HumanEval which are manually-annotee This sigmoid-like performance with respect to sampling budget indicates that problems vary in terms of difficulty, where certain problems require many more attempts to get them right. We do not find degenerate case in any evaluation language where all problems
either trivial to solve (pass@k saturated near 100%), or impossible (pass@k all zeros). The consistency of the observed performance trend across all programming languages in the MBXP benchmark provides reassurance regarding the benchmark's 's applicability as a muulti-lingua evaluation tool for assessing model's capabilities at different levels.
4.2.2 GENERALIZATION TO OUT-OF-DOMAIN LANGUAGES
As demonstrated in Figure 3 our model can achieve non-zero pass@k scores for languages. We emphasize that models not specifically trained out-of-doomin languages since we filter languages based on file extensions and verify that the data have correct syntax with respect to each language (refer to Section 4.1. However, we hypothesize that cross-language data
4

---

Published as a conference paper at ICLR 2023
Accurscy vs. k for 13B Multi-Lingual Model
In-domain
MBPP (Python)
MBBIP (Iwaa)
MBIISP (JavaSeript)
paccOk (%)
N0
30
out-of-domain)
MBPHP (PHP)
MBREIP (Ruby)
MBKP (Katlin)
paccolk (%)
10
130
100
29
189
Figure 3: pass@k versus sampling budget & for various datasets across MBXP. We observe generalization behavior where the model can write valid code languages not trained on, as indicated by the non-zero execution scores on out-of-domain evaluation. Model performance also tends to be sigmoid-like; that is, when the performance is on the lower end such as in the out-of-domain case, the curve breaks out upward, similar to the earlier part of the sigmoid function. The behavior also applies for models of other sizes as well as mono-lingual models (not shown in this figure).
spillover are quite typical, since there can be data related to other languages mentioned in code comments, natural texts, or intentionally embedded in cross-lingual code projects. Examples of such projects are Django Flask, where JavaScript pieces of code can be embedded in Python files for web development, or mixed use of Java and Python code in projects such as Jython. We provide further discussion of types and examples of cross-lingual data occurrences in Appendix
In Figure 4a we also observe that the out-of-domain scores are not symmetric for a given language pair; i.e., Python models perform well Java but Java models have negligible performance on Python. The data spillover hypothesis supports this observation where it is likely that there are many languages embedded in, e.g. Python files, whereas not many languages are embedded in Java files. We provide further analyses related to data spillover hypothesis in Section 4.2.3
4.2.3 MULTI-LINGUAL VERSUS MONO-LINGUAL MODELS
Figure 4a shows plot of passOkk scores versus model sizes for multi- and mono-lingual mod- els. where we observe approximate log-linear relationships similar to those found in the literature (Chowdhery et al. 2022, Chen et al. 202T Nijkamp et al. 2022 Austin et al. 2021 Li et al. 2022a). For small model sizes, we see that muuli-lingual models can perform slightly sub-par or on-par to mono- mono-lingual models. For instance, size 125M and 672M, mono-lingual models out- perform multi-lingual models in some evaluation languages such as Python and Ruby. However, reach certain size such 2.7B or 13B parameters, large multi-lingual model begins to outperform the best of mono-lingual models in all evaluation languages. The performance gains of multi-lingue over mono-lingual models are particularly significant for oft-of-domain languages such as MBPHP and also noticeable for in-domain ones such as MBJSP and MBJP.
We observe that for MBPP, the mono-lingual Java and JavaScript models obtain close to zero pass@k, suggesting that the amount of spillover Python code in Java or JavaScript training data is likely low. This finding coincides with the Python and multi-lingua, models achieving near identical MBPP scores in Figure 4al suggesting that both Python and multi-lingual models observed similar amount of Python code during training. This evidence is consistent with the previous observation that there is little Python code in Java or JavaScript training data.
In contrast, for the JavaScript evaluation (MBJSP) shown Figure 4a| each of the mono-lingual models obtain reasonable pass@k scores, suggesting that the spillover of JavaScript code is prevalent (at least in Python and Java data). This finding also explains why the multi-lingual model performs significantly better to the JS model on JS evaluation (MBJSP), as the multi-lingual model learn JS knowledge from other sources, while the mono-lingual JS model's source of knowledge more limited.
5

---

Published as a conference paper at ICLR 2023
(Python)
(Java)
(JavoScriot)
hoz 201g
MBPHP (PHP)
MBRBP (Ruby)
MBKP (Kocli)
Indoroois
Model
pazsokk (%)
31
Mutt--lngull
Pylken
31
JaadSriipe
125
27b
132
125m
2.70
125
2.bb
Model Size
Model Size
Model Size
(a) pass@10 vs. model size for multi- and mono-lingual models
(b) Validation losses per language
Figure 4: (a) We observe log-linear relationships between model sizes and scores, with multi-lingual models outperforming mono-lingual ones. This trend persists across all evaluation datasets in MBXP, including out-of-domain languages such as PHP, Ruby, and Kotlin. Interestingly, the performance of MBRBP (Ruby) breaks out of this log- linear trend, the multi-lingual 13B model performs significantly better than the extrap- olated performance would suggest. (b) Despite having higher validation losses for each in-domain language compared to their mono-lingue counterparts, multi-lingual models consistently outperform cono-lingual models in all evaluation datasets in MBXP
On languages such as PHP, Ruby, Kotlin which are outside of the core training data (Python, Java, JS), multi-lingual models are also more capable of learning such languages, as demonstrated in Figure [4a] Overall, the performance in the multi-lingual setting tends to improve more rapidly as they are able to draw knowledge from many sources at once, observed by higher slopes in the plots (Figure4al
Interestingly, note that even though the multi-lingual models perform better during evaluation, the validation losses per language for multi-lingual models are higher than those of mono-lingual models (See Figure 4b). We provide further discussion validation losses in Appendix N.2
4.3 ZERO-SHOT CODE TRANSLATION
Our dataset conversion framework yields parallel data in many different languages. These parallel datasets provide valuable resource for studying the translation abilities of code generation models, as we can evaluate how well the models generate code any other supported language using the canonical solutions in our source language. For this study, we prepend the function in a source language to the beginning of the function-completion prompt of the target language (Figure 5 We function completion with augmented information, where we provide
can also think of this setup a reference solution in another language. Therefore, we also refer to the usual function completion setup the non-translation setting.
Zero-shot translation abilities Figure 6a showcases the ability of language models to perform translation by using reference solutions in a different language aid in function completion. Ex- amples in Figures 5 and 8 illustrate how the models are able to produce code that retains the same underlying logic as the reference solution in Python, while conforming to the syntax of the target language, such as PHP (e.g., using $ before variable names) Specifically, the generated code in the translation mode mimics the content of the reference solution, including the same loop and control flow structures, shown in the upper part of Figure g Additionally, the generated code cab exhibit similar semantics, such as sorting by the summation, as illustrated the lower part of Figure
Interestingly, our analysis in Figure 7c suggests that the translation setting enable the models to solve problems that are otherwise difficult without the aid of reference solutions. For instance, on the MathQA dataset, which requires complex reasoning but has solutions with simple arithmetic syntax, our models are able to translate to a different language with near- ceareeerfect accuracy, achieving almost 1 100% pass@ 100 scores (see Appendix D.1).

---

Published as a conference paper at ICLR 2023
Solution in source language (Python)
Socas Parrrcapti
zrite rrhen leagth
meet 2.3
ep1l)
caast metom d1
Prompt in target longuage (Pt-P)
/ee ocfce ne conetice
anberringe
pop Ssacisbetziige (12211 8)
Function completion translation mode
Reettee centionnneendd tel
Function completion
witoou tranelation
Insust 0/
16 .*) cosson by frem
(ep10)
($4
errey
(Bsw): #scatzion (ta) feltz 12
do
Bares and (843663) 1or3
eu [atrlinnde)
(Bocm 37
(atp__tpe_attt_e (6.,
1000000:
Scound:
Figure 5: Demonstration of prompt construction in the translation setting where we prepend the source language's solution. In this translation, the generated code retains similar logic as the reference solution, but has the correct syntax of the target language.
Indomns
Source
adh-o-domain
Baneliee
Milt-lnguu
Pytnon JewaBor
pass (9%)
pass (1)
Sesaa (16)
JeveBorge
MEJEP MBPHP MBRBP
MBPP
MBJP. MBJSP MEPHP MERBP
MEPP MEJP MB.SP MEPHP
(a) Translation
(b) Few-shot prompting
(c) Effects of source languages
Figure 6: (a) Translation setting: Significant gain in function completion ability when using reference solutions in Python. (b) Few-shot prompting: Improvement on out-of- domain evaluation due few-shot prompting, where the examples help guide the model to generate more correct code in the given language.
(c) Translation with different source lan- guages: Reference solutions in different source languages have unequal effects on trans- lation performance. Note: The dots in (a) and (b) indicate the bascline without few-shot or translation. All results are from the 13B multi-lingual model.
Unequal effects of different source languages We find that different source languages can in- teract quite differently with each target language. For instance, JavaScript yields better translation performance as a source language compared to Python, when evaluated on datasets such as Kotlin (MBKP) or Ruby (MBRBP). Languages that are too close in terms of syntax can confuse the model when used together in the translation setting. For instance, Python as a source language for transla- tion to Ruby can sometimes lead the model to generate code in Python, which is undesirable. For each evaluation language, the best source language is fairly consistent, relatively consistent across models. We discuss the language compatibility with respect to translation further in Appendix H.]
4.4 FEW-SHOT PROMPTING
Few-shot prompting is a technique that can provide additional information to a language model, which can help it perform better on specific tasks (Brown et al., 2020). In our experiment, we construct few-shot prompts consisting of three correct functions from the respective MBXP dataset (see Appendix[G.2) for prompt format). We observe consistent improvement in execution accuracies, especially for oft-of-domain evaluations, shown in Figure 6b
One possible explanation for this improvement is that the few-shot prompt can help disambiguate programming languages, which is most beneficial in out-of-domain evaluations when the models

---

Published as a conference paper at ICLR 2023
non-assertion error
MBKP with few-shot prompting
MBKP with JavaScript translation source
100
baseline di shot
cod baseline
1.02
cod- fow shot
0.75
0.75
20
salve rateo
Few-shot
salve rate
Translation
fraction of
0.50
with
0.50
with without
with out
0.22
0.2
20
0.02
0.00
Lask ids ratiied rate
Lask ids ranked ralle
(a) Non-assertion errors
(b) Few-shot solve rate
(c) Translation solve rate
Figure 7: (a): Few-shot prompts results in lower non-assertion (compile, parsing, syntax) errors on out-of-domain (ood) evaluation but has no effect on in-domain (id). (b) and (c): For each task, we show a fraction of generations that pass the tests over the total number of samples (solve rate), where the task indices are ranked to show increasing difficulty. In the translation setting, tasks that are previously difficult (low solve rate for the baseline) can become easily solvable, demonstrating that models can leverage reference solutions in the source language to solve hard tasks. In contrast, the solve rates with few-shot prompting do not deviate as much per task from the baseline solve rate.
reference solution
eample generation non-transiation mode
in translation
Fina aum for 161 913 61
52 G12
51 maxc 01)
(2F1
Smax 51-41
0,201 zeborn (-1)
8
51
(2*62
Sotucn
192 2151]]
Python
PHР
netronn
[-1 1
PHP
sunthyllows
Int)
reoult
sousod IM,
n.gez (x) .reducs At )
-2
catura n. sorteday
Rey un)
roturn reault
ver
sunilyGll (ooll 5 Twee (o Tob (o
[ (1t. 3
Python
mapandaddddd
it..
Kottn
Figure 8: Code generation model can use the style and content of a reference solution in the translation setting generate a correct solution in different language.
not familiar with the target language. For example, in MBRBP evaluation (Ruby), the Ruby function signature can be very similar to that of Python, which can lead to confusion and the model generating Python code without the few-shot prompt. The error analysis in Figure 77a demonstratess that compilation, syntax, or parsing errors (non-assertion errors) drop significantly due to the few- shot prompts.
The improvement due to few-shot prompts also applies to other datasets such as MathQA (Appendix D.I. These observations suggest that soft prompts obtained via prompt tuning or its variants (Lester et al. 2021 Liu et al. 2b2 a Li & Liang (021) could further help condition models to perform better in out-of-domain or searce programming languages.
4.5 ROBUSTNESS EVALUATION: R-MBXP
We evaluate the robustness of models across r-MBXP datasets perturbed by common transforma- tions in NI--Augmenter (Dhole et al.) (02T), standard collection of data augmentations for ro- bustness evaluation on text. Our experiments show that multi-lingual models are more robust on average, with less percentage of performance drops (7.80% vs 9.39% for multi- and mono-lingual models) and higher pass@1 scores across most perturbed datasets compared to mono-lingual models. For more details and other interesting observations robustness, refer readers Appendix D As the first code- cede-generation robustness benchmark, we encourage researchers to further investigate robustness evaluation metrics, data augmentations, adversarial attacks, and defenses based our released datasets.
4.6 CODE INSERTION: 1-MBXP
We introduce i-MBXP, an insertion-based variant of our MBXP benchmark, which the first execution-based multi-lingual code insertion benchmark. Each data sample consists of left and right
8

---

Published as a conference paper at ICLR 2023
contexts where we split the original function signature and the canonical solution into left context, right context, and ground truth insertion code. Code insertion is evaluated in an execution- basee manner by using the same test statements in MBXP. We benchmark using the publicly available insertion-based model, InCoder (Fried et al. 2022).
Both models show that incorporating right context can significantly boost performance compared to using only the left context. In InCoder, we observed 23.2%, 14.4%, and 37.6% relative im- provements on Python, JavaScript, and Java respectively compared to the case without right context. Ablation studies on the performance versus the number of right context lines show positive cor- relation, indicating the models' abilities to incorporate partial right context information improve prediction.
We provide fürther details on dataset construction and results in Appendix K This work demon- strates the usefulness of our MBXP benchmark for code insertion and highlights the need for further research in execution-ber based multi-lingual code insertion evaluation.
4.7 CODE SUMMARIZATION: S-MBXP
We evaluate the ability of models to perform code summarization, where along with its solution as the prompt, with the natural language description in the docstring removed. Based on this prompt, we induce the model to generate the description of the code's functionality. Our results show that, in both zero-shot and few-shot settings, multi-lingual models generally out- perform monnollnggual models, consistent with the performance trends observed in other evaluation tasks discussed in Section 4.2.3] In the few-shot case, we observe noticeable improvements com- pared to the zero-shot setting, with more significant improvement on larger models. We provide examples and detailed results in Appendix
function signature
5 RELATED WORK
Many other evaluation datasets can be considered for the conversion to multi- multi-lingual counterparts such APPS (Hendrycks et al. (222)) and CodeContest (Li et al. 2022a). These datasets in its original forms are execution-bass based datasets containing challenging algorithmic competition prob- lems and tests that sunguage-agnostie but can be converted to Python and many other languages. Existing benchmarks for code generation are primarily either match-based or focused mostly on Python, if not language-agnostic. Our work fills a gap the literature by providing a multi-lingual code evaluation framework that includes synthetic solutions, handles datasets beyond HvmanEval (e.g., MBPP and MathQA), and investigates various types of code generation abilities. Concur- rent work by Cassaaoo et al. (2022) converts prompts and test cases of HumanEval into multiple languages. Recent work by Orlanski et al. (2023) presents BabelCode, a framework for execution- based evaluation, and investigates the effectiveness of balancing the distribution of languages in training dataset. Together, these works provide a valuable resource for researchers to evaluate multi- lingual code generation. We provide further discussion of related work in Appendix B
6 DISCUSSION
Our release of these datasets is significant contribution to the field of code generation research, pro- viding researchers with a valuable resource evaluate various aspects of code generation abilities. The findings from our evaluations have shed light on interesting areas such as multi- vs mono lono-lingual models, out-of-domain performance, zero-shot translation abilities, and lingullingual code insertion, all of which hold potential for advancing the state-of-the-art in code generation.
Our observations suggest that large multi-lingual models are more effective than multiple mono- lingual models in code generation tasks, benefiting from the data spillover across languages. The success of our multi-lingual models in out-of-domain evaluations and robustness testing demon- strates their potential to generalize to new languages and tasks. However, to comprehensively eval- uate the complexities of real-world software development tasks, it may be necessary to include additional togguage-speciff evaluations where appropriate. Overall, our datasets provide solid foundation for future research to explore and enhance various aspects of code generation, with the potential to lead to significant advancements in the field.

---

Published as a conference paper at ICLR 2023
REFERENCES
Karan Aggarwal, Mohammad Salameh, and Abram Hindle. Using machine translation for con- verting python 2 to python 3 code. Technical report, PeerJ PrePrints, 2015. URL https: 7/peerj wttp:/oppppp.ss///////ooooopp
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 2655-26663 2021a.
Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai- Wei Chang. Avatar: A parallel corpus for ayta-pyhon program translation. arXiv preprint arXiv:2108. 11590, 2021b.
Miltiadis Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In 2013 10th Working Conference on Mining Software Repositories (MSR), pp. 207-216. IEEE, 2013.
Jacob Austin, Augustus Odena, Maxwell I. Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Pro- gram synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https:
7/arxiv. www. hatgg//bb/21000 2208. 07732
Shraddha Barke, Michael B. James, and Nadia Polikarpova. Grounded copilot: How program- mers interact with aedeggenerating models, 2022. URL settp:://arxl...o/r/bbbsss
15000
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Horbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ticc Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL 977449974441 (at.aaa..................626e-e.
CC wtteeiis..cpppppeeeee pttpp/ppp/2///aaah
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming Ho Yee, Yangtian 7i, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. A scalable and extensible approach to benchmarking n12code for 18 programming languages, 2022. URL /ttts://arxxee httpp://axxi...cc//bb//2222/ wtt//org///ab////o 08227
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CORR, abs/2107.03374, 2021. URL https: http///axvvvv//// org/ /0t//201..033333
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program transla- tion. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. desa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018 URL ht / http:///ww..aooo.ooooo/o///////a GT5S11558ee337709226444555583883
pdf
10

---

Published as a conference paper at ICLR 2023
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language model- ing with pathways. CoRR, abs/2204.02311. 2022. doi: 11485500axXiw.2244.2333 URL
https://dbl.ozg//30 www.:oi.ogg..... 48550/ www.a.aivv.... 220. 02311
Colin Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, and Alexey Svyatkovskiy. Long-range modeling of source code files with eWASH: Extended window access by syntax hierarchy. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 4713-4722, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2000 387. URL http:pss/acaaanchhoooyyooo
- 387
Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Ma- hamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, al. NI-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint arXiv:21 2021.
Zhiyu Fan, Xiang Gao, Abhik Roychoudhury, and Shin Hwei Tan. Improving automatically gener- ated code from codex via automated program repair. arXiv preprint vrXiv22255.0583 2022.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng. Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. CoRR, abs/2002.08155. 08155, 2020. URL
08155
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and symthesis. CoRR, abs/2204.05999, 2022. doi: 10445500aaxx...2200059999 URL https:
1/doi.org// 48550 48550/ 2204 05999
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy. Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. In ICLR, 2021.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Joaquin Vanschoren and Sai-Kit Yeung (eds.),
Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NewrIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htttt::://ppeeee/////////
net//oorum21d=zyyGyyre
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv: 190.00999 2019.
11

---

Published as a conference paper at ICLR 2023
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of 6 for deep learning training. arXiv preprint arXiv:1905.12322. 2019.
Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reftections on Programming & Software, pp. 173-184, 2014. URL
https://doi 0tte//....115//116626666663 266114
Sumith Kulal, Panupong Pasupar, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search- based pseudocode to code. CoRR, 040999 2019. URL http:
7/arxiv. //arxiv.org/abart
aeriiioorgaaai atrgaabs//99/ 04908
Maric-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample. Un- supervised translation of programming languages. In Advances in Neural Infor- mation Processing Systems, volume 33, pp. 20601-20611 Curran Associates, Inc., neur ips cc/papprr/2220/illl
2020a. URL https://proceediii Paper. Paperr.dddppppp.pp pdf
Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages, 2020b. URL http:p:://.a.iiooog/abbb////////
03511
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 1021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 3045- 3059. Association for Computational Linguistics, 2021. doi: 10. 1221166631/22211.co www.emmmmiiima4ee-11111 URL https:
org/ 10 /tt.:////oo...ogg/..../3///////1 emnlp ma ma.in.2..
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Pa- pers), Virtual Event, August 1-6, 2021, pp. 4582-4597. Association for Computational Linguis- tics, 2021. doi: 10.18653/v1/2022 waw.-llog...5.... URL 2021 aal-acl--l00 long. 353.
10
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autumme Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, 2022a. URL http://aaavvvv/aa org. /btp//abb/222....11
Zhenhao Li and Lucia Specia. Improving neural machine translation robustness via data augmenta- tion: Beyond back translation. arXiv preprint arXiv:1910.0300 2019.
Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Shuai Wang, and Cuiyun Gao. Cctest: Testing and repairing code completion systems. arXiv preprint arXiv:2208. 082889 2022b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/2110.07602, 2021a. URL http:ppp:///rxii.oo//aaaaaaaa
Xiao Liu, Yanan Zheng. Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. CoRR, abs/2103.10385, 2021b. URL http:tpp://ax..w./g///b... 10385
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2023
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2102.04664 2021.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11); 39-41, 1995.
Margaret Mitchell, Giada Pistilli, Yacine Jernite, Ezinwanne Ozoani, Marissa Gerchick, Nazneen Rajani, Sasha Luccioni, Irene Solaiman, Maraim Masoud, Somaich Nikpoor, Carlos Muñoz Fer- randis, Stas Bekman, Christopher Akiki, Danish Contractor, David Lansky, Angelina McMillan- Major, Tristan Thrush, Suzana Ilić, Gérard Dupont, Shayne Longpre, Manan Dey, Stella Bider- man, Douwe Kiela, Emi Baylor, Teven Le Scao, Aaron Gokaslan, Julien Launay, and Niklas Muennighoff. Bloom, 2022. URL BELLEtt::////
co http::/ww....eeiccccccnn/ee/e//a
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine trans- lation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 651-654, 2013. URL https: /oii org/
wttp:/...........0.............. 1ttp:/09111452491111111
2494584
Erik Nijkamp, Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. CoRR, abs/2203.13374 2022. doi: 1044850WarXx..2223.147777 URL
2203.1344
Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishah Singh, and Michele Catasta. Measuring the impact of programming lan- guage distribution. CoRR, abs/2302.01973 2023, doi: 1044850Waxii.2222.01997 URL
https://doi.orq//
48550/ arxiv.
Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint arXiv:2201. /127, 2022.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. ACM SIGPLAN Notices, pp. 731-747, 2016.
Moritz Schubotz, Philipp Scharpf, Kaushal Dudhat, Yash Nagar, Felix Hamborg, and Bela Gipp. Introducing mathqa math-aware question answering system. CORR, abs/1907.01642, 2019.
URL /estt://arrxe org http:/bbs1190..00. 01642
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida Wang. Natural lan- guage code translation with execution. arXiv preprint 211::2204411454,, 2022.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. CoRR, htt//199..08050 2019.
08053
Amane Sugiyama and Naoki Yoshinaga. Data augmentation using back-translation for context- aware neural machine translation. In Proceedings of the Fourth Workshop on Discourse in Ma- chine Translation (DiscoMT 2019), pp. 35 2019.
Mare Szafraniec, Baptiste Roziere, Hugh Leather Francois Charton, Patrick Labatut, and Gabriel Synnaeve. Code translation with compiler representations. arXiv preprint arXiv:2207.03574 2022.
13

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.
URL http::/axxxiv....o.mm org/ abs/17bs...../ 03762
Yue Wang, Weishi Wang. Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. CoRR, abs/2109.0088 00000999
2021. [RLL:ttts//arxiiv.ooom ozg abs hbtp///s../21.///////// 2109.00859
Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank Xu, and Graham Neubig. Mconala: bench- mark for code generation from multiple natural languages. arXiv preprint arXiv:2203.08388 2022.
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Mining Software Repositories, MSR, pp. 476-486. ACM, 2018. doi: https://doi.orr/ wt..oorgg////...... 145/ 3196398.3 3196408.
Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. Coditt5: Pre- training for source code and natural language editing. arXiv preprint vrxiv:2208.05444 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022b. URL https:
abs/2205. 01068
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipimeni,, and Chandan K Reddy. Xlcost: A benchmark dataset for cross-lingual code intelligence. arXiv preprint arXiv:2206.00 284444,206.08474, 2022.
14

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
Extended Discussion
17
A.1
Implication of findings
17
A.2
Implication of Evaluation Data Scale
17
A.3
Possibilities of true generalization
17
A.4
Potential proxy for general coding capabilities
17
A.5
Limitations
17
Generation lendency versus generation ability
B
Other Related Work
18
C
Evaluation Setup
20
C.I Sample Generation
20
Stopping Criteria
20
Code Execution
D Evaluation Results Additional Datasets
21
D.i Multi-lingual MathQA
21
D.2 Multi lingull HumanEval
23
E
Language "Spillover"" in Training Data
24
E.1
Types of Cross-Language Data Spillover
24
24 _...............................
F.2 Example Embedded JavaScript Python files
E.3
Example 2: Java and Python integration Jython
25
F
Execution-Based Function Completion Results
26
Fl
Performance Trend with Respect to Model Size
................................
26
Comprehensive Sampling Results
................................
27
G
Few-Shot Prompting
31
Evaluation Results
31
2
Qualitative Examples
31
Translation
35
Translation Results from Various Language Sources
35
Comparing translation performance multi lingual and mono mono-lingual models
H.3
Generated Translation Examples
40
Analysis: Effects of few-shot and translation prompts
42
1.1
Test case error versus non non-assertion emror
42
Solve rate per problem due to few-shot prompting and translation
42
Robustness Evaluation: r-MPXP
44
J.1
Dataset Preparation and Evaluation Sctup
44
1
Evaluation Results
44
J.3
Qualitative Examples
45
Code Insertion: 1-MBXP
47
K.
Dataset Preparation
47
Evaluation Setup
47
Evaluation Results
47
15

---

Published as a conference paper at ICLR 2023
K.4
Qualitative examples for 1-MBXP
47
L Code Summarization: s-MBXP
51
I.1 Dataset Preparation and Eva uation Sctup
51
Evaluation Results
................................
51
Qualitative Examples
51
M Evaluating Public Models
56
Training
60
N.1 Model architecture and training details
60
N.2 Observations on validation losses versus performance
60
Dataset Conversion Framework
0.1 Language Conversion Prompts and Test Cases
62
Potential Use of Transcoder for Dataset Construction
64
P
Synthetic Canonical Solutions
65
P.I Multi-stage data bootstrapping
65
P.2 Discussion: Ground truth assumptions lest cases
65
Quality Check Converted Datasets
R
Datasets
68
R.
MBXP
68
R.2
Multi-lingual Human
77
R.3
Multi-lingun MathQA
77
16

---

Published as a conference paper at ICLR 2023
A EXTENDED DISCUSSION
A.1 IMPLICATION OF FINDINGS
From our findings, it is clear that a large multi-lingual model compared to multiple mono-lingual is better choice if we to consider deploying code generation models. This is due to the data spillover from each language source which reinforces the knowledge of the model in the multi- lingual training. However, such model needs to be of sufficient size capture all the available knowledge. For our controlled setting. model sizes 2.7B and above begin to clearly outperform all mono-lingual models. It is possible that as the number of languages in the training set increase, the required size for the multi-lingual model to be superior to individual mono-lingual models can increase.
A.2 IMPLICATION OF EVALUATION DATA AT SCALE
Our parallel datasets provide valuable resource for studying the translation abilities of code genera- tion models. By leveraging the canonical solutions in our source language, we evaluate how well the models generate code in any other supported language. This opens up a range of research ques- tions, such as how well the models generalize across languages, what factors contribute successful or unsuccessful translations, and how different modeling strategies affect translation performance.
A.3 POSSIBILITIES TRUE GENERALIZATION
Out-of-domain evaluations from our controlled experiments reveal interesting behavior of how code in multiple languages present themselves in natural data. We hypothesize that the oft-on-domain code generation abilities are mainly due to the data spillover. However, we believe it is also possible that a true generalization plays a role where the model is able to complete code in a new language that is not in the training data at all. To test this, can design new language which avoids the complication of data spillover in the any training dataset. We can use our framework to construct the evaluation set in such language and use it to evaluate the existing models. However, we note that such new language likely are similar to existing languages in the training set in some aspects. For instance, the control flows (if clause), loops, variable declaration, dictionaries potentially might not differ much from each component of existing languages. Even with the new language constructed, the boundary between evaluating a true generalization versus generalization between data spillover can be somewhat unclear.
objects such as lists or
A.4 POTENTIAL PROXY FOR GENERAL CODING CAPARILITIES
MBXP and other code completion benchmarks such as HumanEval measure the general understand- ing of basic tasks from natural language description with function signature and the model's ability to complete such tasks. Given the description of these problems in natural language and function signature where a competent human coder can complete, this benchmark helps measure if a code generation model perform such tasks. The scores of these evaluations can be useful proxy for overall code capabilities they correlate with the performance on all coding-related tasks. We believe that such correlation is possible or likely the case if the models not trained to adapt to a specific distribution of evaluation datasets. By using these evaluations as proxies of general cod- ing abilities, we implicitly accept the premise that zero-shot evaluation on a slice of all possible problems (the slice being MBXP, for instance) is an unbiased proxy to measure overall model's 's ca- pabilities in each language. Hence, in this paper, we particularly avoid finetuning even though results in the literature demonstrate increased performance so that the results established can be less biased towards specific kinds of coding problems and can better reflect models' true coding capabilities.
A.5 LIMITATIONS
The proposed conversion framework is well suited for basic programming problems that are appli- cable to wide set of programming languages. While the original MBPP dataset is meant for basic programming problems, some tasks can be more appropriate for certain languages than others. For instance, string manipulation problems can be naturally encountered in languages such as Python
17

---

Published as a conference paper at ICLR 2023
or PHP more than C++. By design, conversion "naively" assumes that problem is relevant to the target language which might not be true for all problems in a given dataset. That is, the scores obtained from MBXP benchmark might not align with the distribution of natural usage in different languages equally.
In addition, the programming problems in MBXP do not cover language-specife functionalities; for instance, there are no specific questions related to web development for JavaScript, or memory allocation for C++. Therefore, it be unclear how the conclusion from MBXP benchmark trans- fers to coding performance in the wild given the complexity of real-world software development. The test conversion we support are calue-oriented which do not cover all possible types of testing. The value-oriented test performs assertion by checking if the values match. If the assertion process more complex such as in deep integration tests with specific code packages or APIs, the con- version process is not applicable. In fact, we explicitly define the types of Python objects that we support converting from Appendix [ We suggest that it can be beneficial complement MBXP evaluation with other language-specific evaluation, if available.
A.6 GENERATION TENDENCY VERSUS GENERATION ABILITY
The prompts in our benchmark heavily guide the model to generate in a particular language. For example, when a prompt contains functioo ion method name (. the model is highly encouraged to generate code that has such syntax, in this case PHP, but not Python where the function signature would have started with def method_need name (. In that sense, this benehmark measures the ability of a model to conform to the guided prompt and the completion ability based on the function sig- nature that has already started, and not necessarily the tendency to generate in particular languages. Note that without explicit guidance or special tags, models can generate code in any languages, especially multi-lingual models, which makes fair evaluation of code completion harder since we might not penalize correct code that is correct but in different language. Our prompt format helps isolate evaluation of the generation ability in a desired language from the generation tendency. This is contrast free-form prompt style in datasets like the original MBPP, APPs, or CodeContests where the model generates its own function signature. However, in our case, if the evaluation is out- of-domain, it is still possible that with explicit guidance of function signature, the model can still generate in similar yet different language, as in the case of confusion between Ruby and Python with similar function signature syntax.
We also observe that while this benchmark is about generic understanding of basic programming tasks and does not particular attempt to measure the knowledge of model in terms of specific syntax in the desired language, we observe that language-specing for instance, the usage of list... select in Ruby, or nums fi felter in Kotlin to select elements of list, instead of generic for loop. We provide sample generations for all converted languages in Section R.I
syntax usage typically emerge,
B OTHER RELATED WORK
Language models for code generation is a rising domain in recent years. CodeBERT (Feng al.) 2020) is the first BERT-based language model trained on code. GraphCode- BERT Guo et al. (021) improves upon CodeBERT by leveraging AST and data flow. CodeTS Wang al. 2021a) pretrained encoder-decoder based generative lan- guage models for code. More recently, various work have been proposed to use large language models for code generation. Codex (Chen et al., 202T) was pretrained on Python on top of GPT-3, resulting in up to 175B parameters code generation models. CodeGen (Nijkamp et al. 2022) was pretrained on multiple programming languages and optimized for conversational code generation with up to 16B parameters. InCoder (Fried et al. (002), along with CoditT5 Zhang et al. (2022a) on a similar line of research, is able to perform program synthesis (via left-to-right generation) as well as editing (via infilling) with up to 6.7B parameters. Further, researchers also found that generic (natural) language models are also able to perform code completion to a certain degree, e.g., PaLM
Code Generation Models
al. 2021) and PLBART Ahmad Ahmad
(Chowdhery et al. 2022) and BLOOM (Mitchell et al., (022).
In addition, researchers proposed various ways of improving code generation models. For example, Poesia al. (2022) propose Target Similarity Tuning for code retrieval augmentation and Con-
18

---

Published as a conference paper at ICLR 2023
strained Semantic Decoding to improve code generation by constraining the output to set of valid programs in the target language. (Shi et al. (2022) introduce execution assultbased minimum Bayes risk decoding that improves choosing single correct program from among a generated set. Another line of work is to "repair" the generated code by language models, e.g., (Fan et al. 2022 [i et al.
20226)
Our work is model-agnostic and complimentary to all the above works that serves as a testbed of code generation.
Code Completion Resources Many code completion evaluation benchmarks have been proposed recently, but they differ in style and focus. [u al. (202T) composed a token and line completion datasets in Java and Python based on existing benchmarks (Allamanis & Sutton 2013; Raychev
(et al. 2016). Clement et al. (2021) presented a method generation dataset in Python based on CoreSearchNet (Husain et al., 2019). All these datasets are primarily collected from open-source projects or GitHub and focus on match-based evaluation (using n-gram metrics). In contrast to these efforts, recent works promote unit tests-based execution evaluation to assess the functional correctness of ML-based code completion techniques.
In this line of work, Austin et al. 22021) introduced the MBPP dataset focusing basic program- ming problems where the prompt format consists of a natural language description and assert state- ments in Python. HumanEval (Chen al. 202T focuses on more challenging algorithmic problems with prompts containing function signatures in Python along with docstring descriptions of the prob- lems, including test case descriptions. APPS (Hendrycks
al. 202T) and CodeContest (Li al.) 2022a) contain sunguage-agnoste problems in algorithmic competition style and tend to be very challenging. Both datasets expect solutions (complete programs, unlike functions in other datasets) in any language that uses standard input and output to consume and return values. The output is compared directly with the expected values without test cases written in any particular language to test for correctness. In contrast, HvmanEval and MBPP use test statements written directly in Python. We show all the dataset formats for comparison in Section R.
We find that the HvmanEval format aligns best with how programmers would write in typical coding environment; therefore, we use this format for our converted MBXP benchmark. We also convert the original Python MBPP dataset to be of this format well for comparison consistency. Our benchmark, MBXP, is the first execution- based function completion benchmark available in multiple languages for all (or most) tasks in parallel.
Code Translation Resources Several works in the literature have developed parallel corpus to facilitate source code translation. Earlier works (gguyen et al., [013): Karaivanov et al. (014)) fo- cused on building semi- semi-automatic tools to find similar functions in Java and C# from open source projects. Subsequent works used libraries and transcompilers to construct parallel corpora in Python 2 and Python 3 (Aggarwal et al. (2015) and CoffeeScript and JavaScript (Chen et al.] 2018). Among the recent works, Lachaux et al. (2020a) collected a corpus of parallel functions in Java, Python, and C++ from Geeks forkef ks and provided unit tests for execution-based evaluation. Very cently. Szafrannee et al. (2022) extended the dataset in Go and Rust languages. On a similar line, Zhu et (2022) introduce new dataset which is parallel across 7 programming languages snippet level and program level based on Geeks Geeeks data. Another work (Ahmad et al.) 2021b) aggregated a comparatively larger parallel corpus in Java and Python by collecting program- ming problem solutions from several sources. Different from the prior works, our proposed dataset, MBXP, covers a wide range of languages with unit tests facilitate the evaluation of functional accuracy of ML-based code translation models.
both
Multi-linguas Evaluation of Code Generation Models Wang et al. (2022) proposed MCoNaL.a multi multi-lingual version of CoNaLa Yin et al. (2018) in various natural languages. This is orthogonal to our work that extends multi hangii-iiguulityy on programming languages. Similar approaches could be applied to MBXP expand the dataset to multiple natural languages and we leave it as one of the future directions.
19

---

Published as a conference paper at ICLR 2023
C EVALUATION SETUP
C.I SAMPLE GENERATION
We use nucleus sampling with p 0.95 (Holtzman et 2020). For all experiments, limit the input length to be 1792 and generate up to 256 tokens. If the context exceeds 1792 tokens, we perform truncation from the left. Note that the truncation can happen more often in the case of few-shot prompting or translation settings.
C.2 STOPPING CRITERIA
We categorize languages into groups where each group has the same stopping criteria.
Curly-brace style with standalone function: JavaScript, TypeScript, Go, Kotlin, PHP, C++, Rust. We truncate right after the closing character.
Curly-brace style with function wrapped under class: Java, C#. We truncate right after the closing and add \n) to close the higher-level class wrapper. This slightly different from letting the model generate a closing for the wrapper class. We find that if we do let the model generate a closing on its own, it can go on to generate another function, which technically should not harm the evaluation, but it can cause the generation to be too long and can hit the maximum token limit. Therefore, we find that it is fair and more efficient to close out the class right away after the current function generation.
Other keywords: 'end' for Ruby
Note that it is possible to extend these stopping criteria to include multi-functin function evaluation, where the first function can refer to other functions that follow. However, it is out of scope for this current paper.
C.3 CODE EXECUTION
We adapted the human-eva repository by OpenAl which provides multi-thread execution-basee evaluation framework in Python along with unbiased pass@tk calculation. Our adaptation supports exccution in all languages in MBXP where we use Python's subprocess to execute native com- mand in each respective language. For instance, The test statements for each language are such that exceptions are thrown the test cases fail. Each task can also fail due to improper generated code that does not parse compile. We capture the failure or success of each execution via exit code as well as standard error message for further analysis.
execute with node file. js for JavaScript.
Ihtpps: http:/:///ghhbbbbococc..c. http::/gggghhhccccco///aa//aaaaa
20

---

Published as a conference paper at ICLR 2023
D EVALUATION RESULTS ON ADDITIONAL DATASETS
D.1 MULTI-LINGUAL MATHQA
We evaluate our 13B multi-lingual and mono-lingual models on MathQA datasets in different pro- gramming languages. The original MathQA dataset was formatted to the Python version by [Austin ct al. (002)) after which we use our conversion framework to obtain the corresponding data in dif- Terent languages for analyses. We do not leverage the training and validation sets of MathQA to finetune our models. The purpose of this evaluation to investigate the generalization capability of
models complex context, which requires mathematical reasoning. Similar to Section [4.4 and
we compare the models with respect to adding few-shot prompts, conducting canonical solution translation, well as the normal function completion.
Table I: Evaluating pass 100 execution scores (%) on multi-lingual MathQA using sam-
pling with temperature=0.8
Mode
Model
Param. Size
MathQA-Java MathQA-Pythhn
MathQA-JS
Multi
672M
N/A
91.66
94.21
Translate
Multi
13B
N/A
96.33
98.08
Mono
13B
N/A
94.31
96.49
Multi
672M
15.61
13.54
13.54
Few-shot
Multi
13B
21.50
26.21
24.96
Mono
13B
22.78
15.29
19.33
Multi
13B
13.43
18.05
10.67
Normal
Mono
13B
20.23
14.86
10.78
Specifically, for the few-shot setup, we prepend the first 4 examples in the MathQA training data with their canonical solutions. For MathQA-Python, the canonical solutions given; we manually adapt the Python solutions to other languages for these four examples. For solution translation, we evaluate the models on Java and JavaScript with the Python canonical solutions given in the context. The mono-lingual models are only evaluated on the MathQA dataset of the same language.
Following findings summarized below based on Table I
Both multi- munti-linguull and monolingual models perform very well for solution translation (>94%), probably because the solutions required for solving MathQA problems are usually simple mathematical calculations. Converting them to different languages are straightfor- wards, python solutions are provided.
Prepending few-shot examples achieves better performances than normal predictions for both multi-lingual and monolingual models. As illustrated in the MathQA example in Section R.3 the context are significantly different from the training corpus. Involving few examples from MathQA domain in the context does help alleviate the domain divergence.
The multi-lingual models do not consistently outperform the monolingual counterparts. This may be due to the nature of MathQA problems that emphasizes the models' mathe- matical reasoning capability.
Below, we present correct prediction and a failed case for MathQA-Java examples under the trans- lation mode. The case below demonstrates that the model understands that exponentiation in Python (a+ and is able to translate to Math. pow (a, b).
Problem: MathCA/2 (Correes prediction)
Python prompt-canonica. Solution
problem
what will be the difference between simple and compound interest at 14 pez anaun on of 1000 after years n0 14.0 ml 1000.0 - 4.2
nl 1000.0
21

---

Published as a conference paper at ICLR 2023
12
t1 t3 1.0
13
t2 n1
14
t3 n2 t2
15
:4 tiv.min (:2, 3)
t5
17
=6 tb n1
18
answer
19
return апвиет
Java prompt Iranslaticn
Import 1ava.1o.*;
import java.lang.
import jeva.util.Ap
import jova.meth.
class Problam
10
/xy
11
what will be the difference between simple and compound interest az 24
sum of rs 1000 after 4 years ? no 14.0 nl 1000.0 n2 4.0
12
13
public static double problem()
15
// m1 1000.0
16
4.0
17
double n0 14.0;
18
double ni 1000.0;
doubla n2 4.0;
20
double
100.0;
22
double tl to 1.3;
23
double t2 nl t0;
24
double t3 n2 t2;
25
double t4 Math.pasitl, (n2, 5)1;
t4;
27
double L5 LS nl: double answer
29
30
return answerr
12
Illustrated by the failed example below, despite the good overall performance. the model some- times fails to translate mathematical built-in functions from Python to Java (eg. max in Python vs. Math.max in Java). Additionally, math.log in Python can take second argument for logarithmic base, while Math.log in Java specifically means natural logarithm, taking only one argument. The translation model ignores this difference.
Problem: MatHCA/1822 (Nrong prediction) Python caampitcanonica. solution
problem()
Find the least number of complete years in which a sum money put out & compound interest will be than double of itself ? no 45.0
import
no 45-00
na 100.0
10
tl moth. (max (1e-5, 2.01,
11
E2 E2 1.0
=3 loth. (magtmax (1e-5, t2), 2)
13
E1 t1 L3
15
answer = floortto
16
return answer
Java promps*translation result
Inport java.lo.*;
import jeva.leng.e/
import jeva.util.
import java.meth.
asa Problem
22

---

Published as a conference paper at ICLR 2023
I1
find the least number of complete years in which aum of money out 45 $ componnd Intereat will be more than double of itself 45.0
12
13
public static double problem()
double no 45.0;
15
double to n0 103.0;
ti Math.log (ne-, 2.0), 21 argument.
17
// Errors cannot find symbol "nax". Also, math, in Python can take second argument foz logaritamic base, while Moth. Java specitically means natural logarithm, taking only
18
double t2 t0 1.0;
double t3 Math. (max
2) Error
20
double t4 t1 / t3; deuble t5 Int anawer (int) cltor (t.);
1.9;
22
23
zeturn anawess
25
D.2 MULTI-LINGUAL HUMANEVAL
We present the results multi-lingual HumanEval in Section M using models well pub- licly available models. We find that the results on few-shot prompting and translation are generally consistent with MBXP Details on multi- multi-lingual HumanEval dataset preparation be found in Section R.2
23

---

Published as a conference paper at ICLR 2023
E LANGUAGE "SPILLOVER" IN TRAINING DATA
Our evaluation indicates that code generation models typically have out-of-domain generalization
performance (see Section 4.2p. We hypothesize that it due the effect of data spillover that quite common especially in cross-lingual code projects where each file can have multiple languages present. In this section, we provide discussion and examples such cross cooss-lingual code occurrences.
E.1 TYPES OF CROSS-LANGIUS DATA SPILLOVER
We provide discussion types of data observed for code in the wild where multiple languages can co-occur. In particular, there are four categories:
1. Source code from two programming languages occurring in the same file via explicit lan- guage embedding mechanism other than "putting code in strings". There are actually two categories "deep" and "shallow" embeddings of the guest language into the host language. A good example of this in Python https: //nuu////////e// cds github. io/ python numba/ / 055 cuda/] which uses python syntax but does not have the seman- ties of the corresponding python program.
2. Source code from two programming languages occurring in the same file, where the "guest language" is included in the "host language" via the host language's string type. Most web code will fit in this category, but also stuff like code generators (e.g.
codegen//Cxxreessionn scala)
3. Source code from two programming languages occurring in the same project, but always separate files. This is another potential source of cross-lingual data, but it does apply to the models trained in our paper since we filter languages per file, not per project.
4. Combinations of programming languages via a Foreign Function Interface, where the host language does explicitly any source code from the source language but does, e.g., refer to identifiers or function names in compiled bytecode.
E.2 EXAMPLE 1: EMBEDDED JAVASCRIPT IN PYTHON FILES
wtt.ll..ooobrrrr//eeeeeeee The example below taken from http::/hhhhb...ooooooooorrere//t master htwprr/scppptssc.c make ewcodinggjss.p..po.oooeeee shon shows JavaScript written in Python strings
throughout the code file make baakke__encdinggjs....
daserihed in file available az decode specific encoding
Javascript script
http://unicode.orgg// http///AN////////////// TttPP///////////////// /
import import re import jeon
import brlltb.reques
10
11 line_:e compile(" 10x1A 20 9) (0x[A 9]+)**, ze.M)
12
Lmp1 "http:/ heLtt:://unieee wwtp://bic///ae/aaaaa..... http://////.EEE///E///////AAAAA
14 encoding input ("Lazoding name:
15 raq urllib. request segpest.urlegg. urlepen
(encoding, upper
16 data = req. read decode ("ascii*)
17
18 soct_dir path. dirname (os. (os. ditnane (_ile___
19 libs_dit aa path, join troot_dir, "ww*", "src", *libs")
filenane us oz.path. join flibs_dirt E"encoding (encoding lower 111-)s))
21 with open (filename, "u", sucoding="uz--a") as out: out. write (*var Ltable (*)
23
for Line in data. ("An"):
25
mo line_re.match (line)
if key, velue = mo grogro out.e (key,, (value or -1;,
26
28
out write("] out,erite ("var decoding soodding_taboo table [],r
encod ng table ()5n")
24

---

Published as a conference paper at ICLR 2023
out, out. panr (var 0, Saabl..enngin lem;
211
31 vor value _tblee + 1]
32
(value 1-- null)
33
encoding_table (value] _tabbl[l]
34
bbcoding_.ablel_tabbeiii _table[i
16
Smoduls sacodinggtablee decoding_cable)
38
(**)
A simple search query on Github can reveal multiple other examples.
E.3
EXAMPLE 2: JAVA AND PYTHON INTEGRATION A$ JYTHON
This example is taken from
https://jython: www.tteeooocs.ooeeee///////
which shows a combination of Java and Python code in cross-lingual project Jython.
aythonAnddavaatteeraaiii
Irem ozg. www.tyyttoooo........
import CostCalentatorlype
class CostCalculator C
object):
"1 Calculator Utility o1
def init _nnit__selltt (8e1f): print 'Initializing' pass
10
def calculateCost (self, salePrice, tex):
return
(salesrice tax)
12
13
package org, wyt.oo.boo...ioe.....oooo.ooo. boo...iteeraaeee........
14
15
public interfece Costüalculaterippe
17
public double calculatecost (double salePrice, double tax);
18
19
20
import jeva.io. www.coppoiii.i.n.
22
import java.util, ponauuil.looee www.utll.oogggng.... Lovelng
java.util, logging. import
24
26
public class Main
27
28
public static void (Strinti args)
29
dythondojectTacter factory jythondbjeetPaetory tet.ettooryyetttttaneeellel
31
costcslmulstoryypa costcale = icosttalcuiatorrype factory.creataabjeee:
32
costcaleulator hataaatoooyypec.aaaa... "GostCaleulatee
33
System. ptt.itittll (alatCale
wtw.aaaa......laeeeeeeeeee.eeeee calculateCost (25.96, .07)
34
36
25

---

Published as a conference paper at ICLR 2023
F EXECUTION-BASED FUNCTION COMPLETION RESULTS
F.1 PERFORMANCE TREND WITH RESPECT TO MODEL SIZE
Figure 9 shows pass@1, I, pass 10, and pass@ 100 for many evaluation datasets in MBXP. We can observe that the trends for passOk for different & are consistent, but simply different in terms of scale for scores. That is, the observation that multi-lingual models begin to clearly outperform mono-lingual models when the model size becomes sufficiently large holds for any k.
MBPP (Python)
MBJP (Java)
MBJSP (JavaScript)
pass (%)
MBPHP (PHP)
MBRBP (Ruby)
MBKP (Kotlin)
pass (%)
out augeef-dumii
Multi-ingual
Python
JasoSeript
125m
672m
27b
136
175m
67hm
135
175m
072ms
2.7h
135
Model Size
Model Size
Model Size
(a) Temperature 0.2 and
MBPP (Python)
MBJP (Java)
MSJSP (JavaScript)
10
pass0k (%)
20
MAPHP (PHP)
MBRBP (Ruby)
MRKP (Kotlin)
20
passik (%)
in-domiin
Model
15
out-of-damain
Mutti-inuual
Python
20-
10
10
Scrie
072m
672m
Model Size
Model Size
Model Size
(b) Temperature 0.6 and k = 10 (as in main paper).
MBPP (Python)
MBJP (Java)
MBJSP (JavaScript)
pass@kk (%)
(0)
60
(51
MBPHP (PHP)
MBRBP (Ruby)
MBKP (Kotlin)
in-domain aub-of-dgroin
Model
passikk (%)
20
Muht--iguul
Python
40
20
2.7b
Model Size
Model Size
Model Size
(c) Temperature 0.8 and & 100.
Figure Performance versus model size
26

---

Published as a conference paper at ICLR 2023
F.2
COMPREHENSIVE SAMPLING RESULTS
Comparing pass@k k scores of 125m Monolingual and Multi-lingual Models
Pyteer
Tytne
Tytes
pews
pasigk
pas-2kk
pass-gt
11s pass@33 passgg100 MBPP (Python)
possag1
pess-ff posag10 aessg300 pass@100
paseg1
pessegs
passe330 poss-0100
MBJSP (JevaScffp))
MB.P(Jlevi)
Mersinged
Mulilinpa
Pyter
Pytice
Tythis
passgi
1 passgao
possgg
peosgg00
pasag1 preeete
paseg3.0 pessigr00
MBPHP
MBTSP (TypeBerlot)
MBRBP (Ruby)
50
Mulliingul
40
Jevidag
30
pess-gie
20
pasegt
paosgg
1 10 pasegga parag tn0 MEKP (Katli)
pwodg
paougg
paoegg1 MBCSP (CH)
pavegg000 pase 100
pasgg1
paange
paeage
parageo paoaug100
MBGP (Go)
30
pase g
p
pa paoeigg0 pareg MBCPP (CPP)
poodd
pRoueg11 pooeeea paee 100
paoog]
paavEg10 parvag prosugy
MBSCP (Scola)
MB SWP (Swit)
Pytee
paeagt
pasegg30 paradg (Port)
Figure 10: @ trends for 125M monlingual and multi-lingual models for in-domain and of languages.
27

---

Published as a conference paper at ICLR 2023
Comparing pass@k scores of 672m Monolingual and Multi-linuaal Models
Jere
40
paseBK
poss
parm
22 panegg30 paradg
paoodg!
panaiio peouiato pareRE30 paen
paoagg
pasaaeo
paasde pasagg30 paoagg/00
MBPP (Python)
MEJSP (JavaScrpp))
MB.P((lave)
Pytno
Pyttur
40
pasagi
20
MBPHP pass@go
possi
poss-0100
(TypeSeript)
MBRBP (Ruby)
Meriingsl
Mtati peai-iraall
Pythee
Alridorg:
passugk
parrakk
passga
pass
10 pass@a0 paseg 100
possgg
pessigg
pew-gr10 MBCSP (CH))
sessg000 paseg110
paseg1
proeee
peeeg paseg pessig1000
MBKP (Kalin)
MBGP (Goo
Maraingal
Maliiiguil
80
dlridiong
passgk
passagN
paoegg
10 pareggt pareg MECPP (CPP)
pwwdg
pooegg00 paregtt 100
panag1
paeeges
pasagg10 panngg 1o MBW (Swit)
MBSCP (Scala)
Pyiver
40
pooege
pawgs
paseiga0 parsdg
MBPLP (Port)
Figure 11: @ trends for 672M monlingual and multi-lingus models for in-domain and oft-of languages.
28

---

Published as a conference paper at ICLR 2023
Comparing pass@k scores 2.7b Monolingual and Multi-lingual Models
Pyteer
Pypee
Tytts
paroak
paxagk
passgk
40
passid
pase221
(Python)
(JavaSoript)
MB.P(Java)
Miraingust
Mullirppl Pyteee
60
Jevilange
paregk
pacsgg
passggk-
passeg1
ga pazagao passgg 100 MBPHP (PHP)
pase421
p4ss-25
pesrg210 seaaggo0
pasa $2000
paseg21
pasagg
pased210 10 passag30 pasegg100
MBTSP (TypeSerrbt)
MBRBP (Ruby)
Alra
Jocs
paswik
20
paosag!
pwrogg
10 pasegggt panegg
pooogg
paosugg
pare 100
paaag1
paosges
pa pazegg30 peooiggt0
мEKP (Kotlin)
MBCSP (CH)
MBGP (Goo
Mersingal
Mali-irui Pythce
Prrner
paroo @k
pareo
parse
40
passga0
MBCPP (CPP)
MBSCP (Scall)
(Swit)
60
Jur doorp
pasag!
pass-S
p15 passiD0 100 MBPLP (Perl)
Figure 12: pass @k trends for 2.7B monlingual and multi-lingual models for in-domain and ofm-ai- languages.
29

---

Published as a conference paper at ICLR 2023
Comparing pass@k scores 13b Monolingual and Mult-ingual Models
Pyteer
typhe
Tytos
paraak
passgk
40
passid
pase2g1
(Python)
(JavaSoript)
MB.P(Java)
Miraingus
Mullirppl Pythee
Airidors
Jevilange
paneak
pacsgge
parsgkk
parseg1
ga pazagao passgg 100 MBPHP (PHP)
pase421
seaaggo0 pasa 10000
paseg21
pasagg
paseg210 10 passag seseg2100
MBTSP (TypeSeribt)
MBRBP (Ruby)
80
Jlra
penwgk
parsgg
10 paseggt paregg
pawodg
paosuggs
powoagt paveggo00 pare
paoeg1
paosges
pa pasegg paoougg00 MBGP (Goo
мEKP (Kotlin)
MBCSP (CH)
Mersingal Prrner
Multliguil
Pythce
00
pare @k
parse
40
passga0
MBCPP
(Scala)
MB.SWP (Swift)
Jur ddorp
40
20
0
pasa@1
pass-SS
$15 passiD00 poss MBPLP (Perl)
Figure 13: pass@k trends for 13B monlingual and multi-lingual models for in-domain and out-of-domain languages.
30

---

Published as a conference paper at ICLR 2023
G FEW-SHOT PROMPTING
In this section, present more detailed results few-shot prompting with prompt consisting of three correct functions from the respective MBXP dataset. The few-shot prompts are selected from three correct samples for each language. We note that this gives an automatic performance gain of roughly 0.3% since there are R 1000 cases for each evaluation language. However, this is quite small compared to the gains observed. We do not tune on the few-shot examples selected; that is, these examples are chosen once and fixed for all usage. It is possible that this can be further tuned, such as the case of prompt engincering the literature.
G.1 EVALUATION RESULTS
we observe that the performance gain is quite clear especially for out-of-domain languages (pass with temperature 0.2). In some cases, there large performance boosts for mono-lingual models evaluated on out-of-domain languages. For instance, with few-shot prompting, the pass @1 of the 13B Python model evaluated MBJP increases from 5.7% 10.3%. Similarly, the pass I of the 13B multi-lingual model increases from 5.9% to 12.2% with few-shot prompting.
From Figure 14
G.2 QUALITATIVE EXAMPLES
We demonstrate the few-shot prompts for select languages. Each of these prompts precede the function completion prompt for each evaluation.
G.2.1 PYTHON FEW-SHOT PROMPT
def find_char_ Lon_chhr__ooggtxr
*n*
Write function to find all words which are at least characters long in string by using regex.
>35 lon
back to stream")
['Please', 'move', "back', , 'stream")
>>> Hind_char_lome lanr__lnn((" Eco and Toch!) ["Jing", *Tech"]
chad_char_ Thingai
Zone 31)
['Shingai', "wolv', troad', [Xone"]
10
11
import
12
return Findsll (blW(4, 18b", text)
14
16
def square_numa (numa):
17
18
Write function find squares of individual elements in liss using lambda function.
19
>>> square_nume nums ([1, 2, 3, 4, 5, 6, 7, 9, 9, 201)
16, 36, 49, 64, 81, 100]
21
ce5 squace_nums (1,0,20,
([100, 900])
23
>32 square_nums ([12,15))
24
([144,225])
26
return 1ist (map (1 ambda x: x nums)
28
cest_duplicate (arraynums)
30
31
Write a function to find whether a given array of integers contains any duplicate element. >>> test_duplicated (1 3,55)
33
False
Lest__dppiicate(1111222
411)
35
Tzue
>>> (e2_
37
Tzue
38
if (arrayyumm))) len iset (arraynums)
40
return False
42
zeturn Trve
31

---

Published as a conference paper at ICLR 2023
Few shot pars@t1 with temperature 0.2
Murt Inguol
Pythan
Java
(PIP) (TynRRooph)
(Ge)
(CPP)
Model Size: 122:
2.8
6.0
(PHP)
(Ge)
(CPP)
Model Sizs: 672M
16
(PP) (TywwBorp))
(C#)
(Ge)
(CPP)
Model Slee: 2.78
30
29
(PHP)
(TygwBBorpt)
(Ge)
(CPP)
(Scar)
Modal Slza:
138
Figure 14: Performance difference due few-shot prompting (pass@1 with temperature 0.2).
32

---

Published as a conference paper at ICLR 2023
G.2.2
JAVASCRIPT FEW-SHOT PROMPT
Write Javascript Function identify prime numbers.
> PiNotPrime (2)
false
iskottrime (10)
true
PakotPrime (35)
true
10
funct Ton 1sNotPrime (n1
11
For (1et 1 2; ny 11
12
if (n 8 mew Feture truey
13
14
15
return false;
17
18
19
20
Write a javasoript funct lon to remove all digits from a Hist of scrings. ["'worda", "3lettera", "4digits"))
22
["words", "letters*, "digits"]
zemove [["2Baan", "12Jan", "lidan"I)
24
(*Jon", "Jen", "Jan"]
25
zemove (["oonder'', "wonder2", "wonder3"])
("wander*, "wander", "wonder")
27 /
Tunct lon renove (List) ! return List.map (item => stem.replace///dk/g (1));
29
30
31
32
/+-
Write Javasoript function datermine whether all the numbers different from each other are not.
LestDistinct ([1, 91)
35
true
36
tastDistinst ([2, 4,
911
37
false
38
> tastDiatinst ((1, 2, 3])
40
w/
function tes:Distinct (data) {
42
return data. every (item, index) => return data. ndexod titem) index;
44
));
45
G.2.3
PHP FEW-SHOT PROMPT
<?php
/.-
You are an expert PHP programmer, and here is your teak.
Mrite Lunctlon recursion
php recursiveListSum 2, [3, 41, [5, 611)
php securalvet.atssum ([3, 10, [15, 14), (19, 41]]1
10
recursiveLietsum 20, [30, 401, (50, 6011) 21.
11
x/
13
functi ion recursivelliersmm
$5um
15
0; Foreach ($datatist as Sitem! {
$1
16
17
if (is_arrey (Sitem))
18
$sum t- recurss.veriatttuun[SSteee)
20
$sum += $iremy
22
23
return Saum;
25
27
You are expert P&P programmer, and here is your task.
function to sum items the given tonary.
33

---

Published as a conference paper at ICLR 2023
phg retuznSum [*a" 100, 200, "c" 3001)
30 600
phg
returnSum ["a" 25, "b" =x 18, "t" => 451)
32
E8
33
php
zeturnSum ["a" => 36, *b" w> 39, "c" m> 491)
124
35
w/
SneurrSuumSdict.
17
Saum
38
Foreach ($dict as Skey => Svalue)
39
Saun Svalue?
40
return Saum;
42
44
45
You are an expert PMP programmer, and here is your Lesk.
46
Write function to find squares of individus? elements list using lambda function. php squareSums ([1, 2, 3, 4, 5, 6, 7, 9, 9, 10]) 49, 87, 100]
47
4, 9, 16, 25, 69 phg aquazeNums ([10, 20, 301)
[130, 400, 900]
51
aquazeNums ([12, 151) [144, 225]
52
+
54 funct ion aquareNums ($numa)
Sequares
56
foresch (Snuns Snuml {
$squares[1
Smum;
58
59
return Saquares;
61
63
6/8>
G.2.4 RUBY FEW-SHOT PROMPT
24
expert Ruby programmer, 3 8 krite a. Ruby function to remove all digits from a list of strings. remove I ["4words", "Bletters", "Acigits")) 3 a> ("worda", "I "letters", "digits")
is your
lzbs remove removel["2ee [""8Can"" *12Jan", "]lJan"]]
["Jan", "Jan", "Jan"]
9 irb> zemove [ [[Monder1", "wonder2", "Honder3*])
["wonder", "wonder", "wondez"]
10 def remove (list)
recurn 1ist.map [word]] word.gaub (/\6t/, (1)
12 end
13
14 #
15 You are an expert Ruby programmer, and here is youz Leak. a Ruby function zenove even
given list.
17 $ irb> Sanove_evan:ll 3, 5, 211
18 [1, 3, 5]
19 irbs remove_even 15, 6, 7J)
20 [5, 7]
irb> [emove_even:[L,
3,
22 => [2, 3]
remove_even (1)
24 return r.refect [[x] x 2 == 2)
25
26
27 #4
expart Ruby programmer, and here is your task. 29 $ Krite a Ruby funct to "ind the minimum of two numbers.
You
mininum 2)
31 9 =>
32 lrb> mininum (-5, -41
33
34
1zb> mininum (0, 0)
36 def minimum (a, 51
37 recurn
38
end
34

---

Published as a conference paper at ICLR 2023
H TRANSLATION
H.1 TRANSLATION RESULTS FROM VARIOUS LANGUAGE SOURCES
In this section, show translation results using (1) multi-lingual and mono-lingual models of various scales and (2) three different languages as source solutions (Python, Java, JavaScript). We note that the canonical solutions from Java and JavaScript are from the data bootstrapping using separately trained model, as detailed in Section P For tasks that we do not have solutions for, we do not prepend anything to the usual target-language prompt.
While the training data can potentially consist of translation-like data which allow the model to per- form zero-shot translation, such volume to be low. In addition, the model is not trained specifically on certain languages such Kotlin or PHP for multi-lingual models, even on Java for Python-only models, for instance.
do know the volume of such translation-related data and suspect
Figures 15 illustrate the zero-shot translation results for multi-lingual, Python, JavaScript and Java mono-lingual models respectively. In most settings. we observe improvements due to zero-shot translation over the baseline.
Out-of-domain evaluation languages benefit more from translation We can observe consistent performance gains due to translation as opposed to without using reference solution. The perfor- mance gain is drastic in certain cases. For example, for Ruby, the 13B multi-lingual model obtains 5.9% pass&1 I in the normal mode and 15.9% in the translation mode with JavaScript as a source language, or for PHP, the performance improvement is from 19.1% 46.5% with Java as source language.
Effects of language compatibility affinity for zero-shot translation Based the trends of performance gains from the translation settings, we observe that different source languages have unequal effects as reference solutions. For instance, based on the multi-lingual 672M and 13B mod- els, Java is the source language that yields the highest performance for MBPHP, whereas JavaScript seems the best for MBRBP and MBKP. These compatibility trends can change slightly but are roughly consistent. For instance, for MBISP, JavaScript is the best source language for the 13B JavaScript and Java monolingual models, whereas Python the best source language for MBJSP in many other settings. However, for MBKP and MBRBP, JavaScript consistently is the best source language across all model types. We summarize the best model types for each evaluation set below in Table We observe that it is not necessarily the source languages that closest in syntax that the best source language, since has potential to confuse the models during translation and lead the model generate in incoorect syntax.
Table 2: Source language that yields the best zero-shot translation scores for each evalua- tion language
Model Type
Evaluation Dataset
Multi-lingual
Python
JavaScript
Java
MBPP
None or Java
Java
Java
Python
MBJP
Python
Python
None Python
Python
MBISP
Python or Java
Python
Java
Java
MBPHP
Java
Python
Java
Java JavaScript
MHRBP
JavaScript
JavaScript
JavaScript
JavaScript
MBKP
JavaScript
JavaScript or Python
JavaScript
JavaScript
We provide some examples in Section
Mono-lingual versus multi-lingual models For mono-lingual models, we observe large perfor- mance boost, partly due to mono-lingual models not performing well for baseline start with.
Trends with respect to model sizes Larger models typically perform better, as observed in the normal code completion case and also in the translation case well.
35

---

Published as a conference paper at ICLR 2023
Model knowledge of source language target language is likely the case that the knowl- edge of the target language is more important than the source language for translation performance. We note that Python model obtains high scores with translation on MBJSP with Python source (13.8% + 30.7%). JavaScript model also obtains high scores with translation on MBJSP with Python source, with better performance compared to Python model, which is in part due to better baseline performance to start with (23.3% + 32.8%).
Trareiaion
lingual
Offerett
Pythen
123
Souts kmpugge
Sourse
presee 190
USFP (PP0))
NSJUPP Mode
MOSHHP 130
HEREP
Mgm
NSUP (iW0))
SCP Model
MERS
130
Zere-Sout Translaiinn with Diflree Transbsionn Bourcos Jawan cript Modills
mitabannnn Defurent Sources Java Modes
Dourme
Dourme
(S)
(161 Boood
150 130aot
(1840)
(Rabe)
(A08))
(Ruby)
Figure 15: Zero-Shot Translation with Translation Sources from Different Languages: Multi lingual Models
36

---

Published as a conference paper at ICLR 2023
H.2
COMPARING TRANSLATION PERFORMANCE MULTI-LINGUAL AND MONO-LINGUAL MODELS
Shat pass@1
Python
Model
Mut-ingusl
paraa
Java
MBPP (Python)
(PHP))
(Ruey)
(Katlin)
parngg
15
MBPP [Python))
(Ruby)
27.A
20
No1 AM
MBFP (Python)
(Ruey)
(Kalin)
Modal
40
20
(Python]
BlavaSorpti Model
(Ruby]
Figure 16: Translation performance compared to baseline (dot) for multi- and mono- lingual models, with Python source language.
37

---

Published as a conference paper at ICLR 2023
Zero-Shal Translation pass@t 1 with Translation Sources from JavaScript
Model Muti-ingual
Python
MBPP (Pylhon)
MELP (J
MRJSP (1 (lavaSerpp))
MBPHP (PHP)
MARBP (Ruey)
MBKP (Kallie)
Model Size:
125
MBPP (Python)
ME.P (Java)
MEJSP (1 Saropp
MEPHP (PHP)
MEREP (Ruey]
MEKP (Kalin)
Model Siee: 872M
30
25
20
1.5
10
MBPP (Python)
MBLP (Jare))
MBUSP (1 (avipt))
MEPHP (PHP)
MBRBP (Ruey)
MBKP (Kalli))
Model Size:
2.78
50
113
MBPP (Pythen)
ME.P (Jave)
MEJSP (1 SaraSp
MEPHP (PHP)
MBRBP (Ruey]
MEKP (Katin)
Model Sizw: 138
Figure 17: Translation performance compared baseline (dot) for multi- and mono- lingual models, with JavaScript as a source language.
38

---

Published as a conference paper at ICLR 2023
Shot pass@1
Modal
12
Baswine
Mut-ingusl
Python
Java
MRLP (Javo]
MEKP (Kotin))
[Python]
(Ruay)
pawgt
15
MBPP (Python)
MELP (Jeva)
MEUIP (Jansllorpt)
MBPHP
MBRBP (Ruey)
MEKP (Katin))
30
20
MRPP (Python)
(PHP)
(Ruey)
(Katln))
XVA
10
MBPP
(Ruey)
(Kulin)
Figure 18: Translation performance compared to baseline (dot) for multi- and mono- lingual models, with Java source language.
39

---

Published as a conference paper at ICLR 2023
H.3
GENERATED TRANSLATION EXAMPLES
H.3.1
MBPHP (PHP)
def a)
write python function find the of repeated elements in array, 61
>>>
Dum
>>> Sim [1,2,3,1,3])
>>> Sum [1,1,2],3)
11
12
return
arr count
13
14
<Pphp
15
16 ++
expert programmer, here is your toak. Write PAP function to find the sun of repested elements in given arzay.
18
19 php findSum ( 2, 3, 1, 1, 4, 5, 6),
20
21
php
LindBum [1, 2, 3, 1, 11. 5)
22
23
php findSum ([1, 1, 21,
25 +/
function
($ars, Scount array_count laless (Sarr);
27
28
return suray_sun (array_____l filtez (8arz, function ($value) use (Scounz) { return Scount [$value] 1;
30
1));
Find_Praduct n):
Welte pythos Lunction rind the product of non non- repeated elementa given array. >>> Productt (112)
Find_Product (11,2,3,1,114
Find_prodset (1,,1,4,5,6) 6)
120
10
12 acr.)0:L()
prod
14
for i in range 1;
15
(arr[: - 1- [1)) prod prod acr
16
17
return prod;
18
19
<?php
21 '..
22
are an expert PHP programmer, and here is your taak.
23 Write PIP Iunction to Sind the product of non zepeated elements in given array.
24 php findProduct (11, 1, 31, 4)
26
php findProduct (11, 2, 3, 1, 11, 5)
28
php FindProdset (11, 1, 4, 5, 6), 5) 123
29
30
x/
31
function findProduct (Sarr, $m)
arrry perray_unqque (Sarm);
33
Sprod - 1;
foreach ($2rr as Skey Svalue) {
35
in (Skey = 0 11 Sualue 1- $ar> 111 $prod Sprod Svalle;
36
37
38
return sprod;
40
H.3.2 MBRBP (RUBY)
40

---

Published as a conference paper at ICLR 2023
Write javascrigt function to find the element cocurring odd number of times.
get0dssbeuereence (11, 2, 3, 1, 2, 3, 11,
1
4, 4, 21, 13) getoddberurrencee ([L, 2, 3, 2, 3, 2, 31,
getousocaurrence ([2, 3, 5, 4, b, 2, 4, 3 b1
21
-/
10
Tunct lon getoduoccurzenne (azz, acrSize)
11
12
for i in sange10, arr_aize);
14
Ioz in range (0, arr_size) arr(i] arr(]]: count +- count $ 0: asr[i]
15
16
17
18
19
return
21
23 You expert Ruby programmer, here is taak.
24 Write & Ruby function to Sind the element occurting odd number of simes.
irb> get_ood__occureene ([1, 2, 3, 2, 11, 7)
26 =>
irbs gei_ondd_oocurrenee (11,
31,
28 => 3
30 get_ood__occurrene (12,
21, 131
31
32 def get_odd__ot cecurrenno arr_siza]
33 cccurrences Rash.new
0...art_size
35
occurrences [arr[i]] ||=
eccurrences (arr [i]] end 37
38
resui:
nil
cocurrences.co
do Ik, if $ 2 1- 0 b6 result nil
40
42
end
44
result
45
H.3.3 MBKP (KOTLIN)
def sort_matrix(M::
Write function romB.
given matrix in ascending order accerding the of its
(I1 31,
51, 1,
[[i, 1, 11, [1, 2, 31, [2, 511
51]
>>> sort_matrix
2, 3), [-2, 4, -5], O1 -1, 111)
I[-2, 4, -5,, [1, -1, 11, [], 31]
16, (2,1,411
[12, 1, 41, [6, 4, 31, [5, 91]
10
result
keynsum)
12
return recult
13
14 /ve
15
You are an expert Kotlin programmes, and here is your task.
16
write function sort given matrix in ascending order according the of its
17
sortxatrix
31, [2,
51, [L, 211)
18 [11, 1, 1), [1, 2, 3), [2,
4, 5]]
19 aortMatrix 2, 31, [-2, 4, -51, 11, -2, 111)
20 [1-2, 4, -], [1, -1, 11, [1, 311
21
>>> sostxatrix 8, 91, [6, 4, 31, 12, 411)
[12, 4), (6. 4, 3],
8, 9]]
23 /
cortMatrix(m 1isr<List<<ne>>) Micc<liet<<nn>>>>>> return m.oorteyBy I it.
25
26
41

---

Published as a conference paper at ICLR 2023
I ANALYSIS: EFFECTS OF FEW-SHOT AND TRANSLATION PROMPTS
Below, we show an extended version of the analysis in main text (Figure 7 where we demonstrate the differences in qualitative behaviors of few-shot prompting versus translation settings and their effects on helping the usual function completion task. The differences in these two modes are simply the prompts the precede the function completion prompt. That is, the few-shot setting uses 3 examples in the corresponding language and the translation mode uses the solution from the same problem in a different language. The translation setting helps the model solve difficult tasks that very difficult to solve without reference solution (Figure 21) whereas the the few-shot prompting helps condition the model to generate code properly in that respective syntax (See Section
1.1 TEST CASE ERROR VERSUS AON-ASSERTION ERROR
We categorize the failure of each generation code sample into two main categories: assertion or test- based errors versus non-assertion errors, which consist of all other errors such as compile, parsing, runtime error not related to test cases. We use the results from temperature 0.2 with 30 samples for each problem and calculate the fraction of non-assertion errors over the number of all samples.
The results in Figure 19 show that the few-shot prompting results in lower non-assertion errors for out-of-domain languages, indicating that few-shot prompts help models generate code with more precise syntax in each language. In contrast, there is little effect even evaluated the in-domain languages, since the models already are fluent in these languages where the additional signals from the few-shot prompts do not help further. For the translation case, interesting we observe higher non-assertion errors on in-domain evaluation.
MBPP dataset
MBJP dataset
MBJSP dataset
1.00
1000
Iron aer-asooriiin error (%)
boseline
non assertion error (%)
80
fere-hhot
50
80
translatee
nath asseriion
40
20
20
20
125M
872M
2.78
138
125M
672M
2.78
13B
125M
872M
2.78
138
models
models
models
MBPHP dalaset
MBRBP dalasol
MBKP dalasel
100
non assertion entor (1%)
non sonttnn errar (%)
non osssention (%))
80
60
80
60
40
40
20
20
20
125M
672M
2.78
138
125M
G72M
2.78
13B
123M
672M
2.78
138
models
models
models
Figure 19: The percentage of test case non-assertion error out of all the error cases on different in-domain (upper) and out-of-domain (lower) datasets.
1.2 SOLVE RATE PER PROBLEM DUE TO FEW-SHOT PROMPTING AND TRANSLATION
We perform sampling generate 100 samples with temperature 0.8. For each task id, we calculate the fraction of number of code samples that pass the test over a total of 100 samples. We repeat this experiment for the function completion baseline, few-shot prompting, and translation settings. In Figure 21 we sort the task ids by the solve rate of the function completion baseline, which indicates the difficulty of various tasks each dataset. We observe differences in how the solve rates for few- translation settings accumulate. For the few-shot case, the accumulation of the solve rates per task revolve near the baseline solve rate, indicating that the difficulty of the problem given the sho- prompts do not deviate much from the difficulty in the baseline case. However, in the translation case, some tasks ids that correspond to low baseline solve rate have much higher solve rate in the translation case, sometimes with perfect rate 1.0.
shot prompting
42

---

Published as a conference paper at ICLR 2023
MBPP dataset
MBJP dataset
MBJSP dataset
100
baseline for-shot
rosertion eror (56)
assertion error (%)
Iesertion erTO1 (55)
transfafle
60
60
40
40
125M
models
models
models
MBPHP dataset
MBRBP dataset
MBKP dataset
asserion error (%)
60
assertion anror (%)
asserriion error
60
80
80
40
80
123M
672M
125M
138
123M
2.78
models
models
modele
Figure 20: The percentage of test case assertion error out of all the enror cases on different in-domain (upper) and out-of-domain (lower) datasets.
MBKP with fespsshot prompting
MBKP with Python transtation MBKP with javaScript translation
NBKP franstation
Few-shot
Translation
Translation
Transiation with whzz
5.00
without
witout
wttlcat
Leak la ramee by solve rat
Lash by to
to ranfed sall Tale
lask
(a) MBKP (Kotlin)
MEPH-P with prompting
MBPHP with Python transtation
MEPHP with Sovtt transtation
HEPHP transtation
Few-ssiit
Translation
without
withiut
withoat
120
cadc da rembee by sabv rate
task bakk/derae ranked ss rate
ics rarled cale rato
task of splve rate
(b) MBPHP (PHP)
HBRSP with
MBRBP transtation
MEREP with javaS
Translation
Transiation
4.5
1000 1nd wore GRAATTE
(c) MBRBP (Ruby)
Figure 21: For each task, we show fraction of generations that pass the tests over the total number of samples (solve rate), where the task indices are ranked to show increasing difficulty. In the translation setting, tasks that are previously difficult (low solve rate for the baseline) can become easily solvable, demonstrating that models can leverage reference solutions in the source language solve hard tasks. In contrast, the solve rates with few- shot prompting do not deviate as much from the baseline solve rate. Translation from different language sources exhibit similar trends where the source solution can help solve hard tasks where we observe unequal effects from different source languages.
Figure also also demonstrates consistent effects of source languages. For language such as Java, we observe that it can help solve many of the hard problems for MBPHP (PHP) evaluation, based the high concentration of points around solve rate 1.0. This analysis also complements Section H which demonstrates unequal effects of source languages.
43

---

Published as a conference paper at ICLR 2023
J ROBUSTNESS EVALUATION: R-MBXP
J.1 DATASET PREPARATION AND EVALUATION SETUP
Robustness is important indicator of the reliability of code generation models in practice. Here
we provide robustness benchmark for all the trained models across MBXP datasets. Specifically, consider three natural data augmentations (I) Paraphrase by Back Translation (Li & Special
2019 Sugiyama& & Yoshinaga 2019) (e.g.. "create a function" to "write one function") (2) Character
Case Change ("Create A FunctioN") and (3) Synonym Substitutions (Miller 1995) ("generate
function") as basic transformations to perturb the docstrings in prompts. We use the default settings and implementations of these three transformations from NI-AAugmentr
standard collection of data augmentations for robustness evaluation on text (Dhole et al.] (021). )). We select these three transformations since they can mostly maintain the naturalness for the tasks of code generations based on our observations. We then measure the average pass@ I with greedy decoding for all the models on datasets perturbed by each transformation. Here multi-lingual models are trained with multiple languages including Python, Java, Javascript (JS) while mono- mono-lingual models are trained with each language individually. To simplify the comparisons, we call Ruby, PHP, and Kotlin as out-of-domain datasets for all the evaluated models while Python, Java, JS as in-domain datasets.
J.2 EVALUATION RESULTS
We present the detailed results in Figure 22 and summarize several interesting observations below. (1) The percentages pass@) drops on perturbed datasets over regular ones are consistent across different sizes of the models. In specific, the average pass@ @ over all the datasets drops from 2.26 to 2.07 for 125M models (8.56% drop), 6.40 to 5.87 for 672M models (8.25% drop), 9.20 to 8.33 for 2.7B models (9.40% drop), and 12.63 to 11.62 (8.02% drop). (2) For in-domain datasets, multi- lingual models have less percentage of performance drops compared to mono-lingun models under perturbations. On average, pass of multi- lungtll models drops from 21.36 19.73 (7.63% drop) while pass @ 1 of mono-lingual models drops from 16.90 to 15.24 (9.81% drop).
(3) For out-of- domain datasets, multi-lingual models also have less percentage of performance drops compared to lono-lingual models under perturbations. On average, pass @ of multi-lingual models drops from 6.78 6.24 (7.98% drop) while pass@) @ I of mono-lingual models drops from 2.72 to 2.47 (8.97% drop).
MEJSP
Sif
Mode's
Mode's
Modes
MEPHP detasa!
MERBP detesal
MBKP delase!
pall (NC
pithos
Puisi?s
molf ihy Inw WIM
PP
P
2.5
13
124
1724
170
100
125
C22
230
100
120
02:0
270
130
Figure 22: Pass measured on nominal datasets (N) and perturbed datasets (P) across different model sizes for each dataset.
44

---

Published as a conference paper at ICLR 2023
J.3 QUALITATIVE EXAMPLES
In this subsection, we provide qualitative examples to illustrate the three types of perturbations applied to the datasets. A successful and failure MBPP sample completions before and after each type of perturbation are provided based on the code completion results by 672M_ pythhnn model on MBPP dataset. We performed similar perturbations for all the other datasets for robustness evaluation (quantitative results shown in Section J2
MBPP Example 1 for Back Translation Paraphrasing
[672M [612M_python python complete fon on normal prompt def Left_Rotate (n, d)
(passed: True)
..*
Weite python function to left rotate the aits of given number. Left_Rotate (15,2)
64
>>> (100
40
>>> (99t_))
792
11
Begin
completion
13
return n << d
code completion ###
15
16
(6122_python completion perturbed prompt, (passed: False)
def Left_Rotate (n,d): 18
..*
20
Write Python function to rotate the bite of given numbar to the left.
21
>>> (ott_Rotate(16,2)
22
61
23
>>>
25
555 Reft_Rotatee99:2
792
27
28
Begin of code completion #44 return >> d (n (2 <c d)
30
#48 Bad of code completion #4#
MBPP Example 2 for Character Case Changing
[672N_python completion on normal prompt (passed: Tzue) def text_match_string (text):
Write function that matches word the beginning of string. python*)
>>>
matched!') >>> hert__atck_ssriiie (tt_ssttiinn"""ther""e ("pythoa*)
("Found match!'i
cas pext_matci_strrng]" lang") ('Not matched!') *n*
10
12 #4 Begin of code completion #44
return "Not natched!* is stattsttatt ") "Found match!"
14 #t End of code completicn #+t
15
16 [672x_python completion perturbed prempt]; (passed: False)
17 daf text_matcis cort_match_atring(ttan (text
18
...
19
Welle function ion that matches a word at the BeginKing of strinG. python")
text_matck_strinn ('Not matched!')
21
22
c53 mext_matci_sttrnng (ttritn"""pyhher ("pythea")
('Found match!']
24
>>> hntt_match_sttrnnne lang") matched!")
26
segin
completion
28
return "Not matchod!*
29
End code completion #4#
MBPP Example 3 for Synonym Substitutions
[672X_ python completion en normal prompt] (passed: True)
daf calck_value (diet, =):
45

---

Published as a conference paper at ICLR 2023
are same dictionary.
>>> vallek wa
((' ('Ciorra Vega" 12, x "Xlden Controll': 12, "Kierra Genpryl: 12, (Pierre Cox
': 11)
Palce
>>> check_vaiue ((' ("eirrra Vege": 12, "Alden Cantrell'; 12, "Kierra Benzzy": 12, "Fierre Cox
': 12)
True
>>> check_ (( ('Cierra Vega' : 12, AAlden Contrell'; 12, 'Kierra Genczy': 12,
1: (2))),5)
10
Falae
11
12
*+ Begin of code completion #ff
for key in dict:
14
If dict [key] [= n:
15
reture
16
return True
17
Bad code completion #4
18
19
[672M] python completion an parturbed prompt, (paased: Folse)
eheck_value (diet, n):
21
Write function check
like dictionary.
23
>>> check_value (a (l'Clerra Vega/ 12, "Alden Cantrell's 12, 'Kierra Genozyf: 12, 'Plerze Cox
": 12),10)
Valse
25
>>> check check_value (ollee(('e (('Cerr Cierra Vega" : 12, "Alden Contrell'; 12, 'Kierra Genpryt: : 12, 'pierre Cox 12),
26
Tzue
check_valse (('Cierra vega! 12, 'Alden Cantrell'; 'Kierra Genczy': 12, "pierre
1: 12),5)
28
Faise
10
#4 Regin of code completion #d#
For key dict:
32
if type (dict (key]) !- int:
reture
return True 34
35
# End code completion #+#
46

---

Published as a conference paper at ICLR 2023
K CODE INSERTION: I-MBXP
K.1 DATASET PREPARATION
Assume there lines of code in canonical solution, skip problems with < 2 and randomly mask out m = [1, consecutive lines for remaining problems. For each problem, we run multiple times (say, 1 if < 5, if 12, otherwise 3) to generate variants and remove duplicate masks. We report data statistice in Table 3
Table 3: Data statistics for i-MBxP. We report total number of problems and the number of problems as function of the number of insertion lines.
total
0
1
4
i-MBPP
1489
1
922
246
138
69
28
23
i-MBJI
1813
19
805
328
217
172
119
114
39
i-MBJSP
1521
38
739
266
159
138
71
27
K.2 EVALUATION SETUP
We evaluate InCoder-6.7B model (Fried et al.] 2022 pretrained on 159 GB of code, 52 GB in Python and 107 GB in other 28 languages, and 57 GB of text content from StackOverflow. We do greedy search and report averaged execution accuracy over all problems. We feed a sequence right <Mask: to the model as prompt, where left denotes the right context and <Mask: 0> denotes sentinel token of Incoder model. We apply two stopping criteria: (I) <EOM> is generated and (2) any token from the predefined list [" ("Ancaas "\n#", "miff] is generated for Python or braces that close the function scope for Java and Javascript. After that, we match generated tokens with right context and remove duplicated tokens. We compare it against left-right (L-R) baseline where we feed left to the model as prompt and follow the same stopping criteria.
left <Mask:
"Andef
K.3 EVALUATION RESULTS
show that right context can significantly boost performance across all languages. We also studied the effect of the number of lines right context and observed increasing accuracy add more lines of right context (in Table which is intuitive since more context is beneficial. Furthermore, qualitative examples (K.4) show that models are able to leverage right context to fill in the blank. In example example 1, given index and in the right context, InCoder model fill in two inner loops L15-L... Otherwise, fails to do so. In example 2, given add operator the right context, the InCoder model can mimic the behavior in L33-L38. Otherwise, model might generate irrelevant operator remove. In example 3, given result result(char) model generate result cchar + 1. Otherwise, the model will replace with count.Str (chharr))c which results in wrong outputs.
Results in Table 4
1 in the right context, InCoder
Table Pass@1 accuracy on code insertion datasets: i-MbXP
Model
i-MBPP
i-MBJSP
i-MJJP
L-R 30.1
48.65
41.7
Insertion
37.07
55.68
57.41
Table 5: Pass 1 vs the number of lines of right context.
dataset 1-MBPP 30.1 32.1
3
ALL
35.6
36.4
37.07
K.4 QUALITATIVE EXAMPLES FOR 1-MBXP
The example below shows that the model able to use the right context information and generate appropriate insertion code that consistent with the right context. In contrast, the left-to-right
47

---

Published as a conference paper at ICLR 2023
approach does not make of the right context which leads to very different implementation that would not be consistent with the right context.
Example 1: Python insertion mode
def Finc_trrplet__azee aaapplet__rrryylee (A, arr_size, sum)
sen
Write a Function to Find it chere is a triplet In the areay whose sum is equal. to given value.
>>> Find_tripled #ind_:riplet_arreyyll
4, 45, 6, 10, 81, 6, 221
(4, 10, 8) >>> Lind_errpiet__erraylll
91, 6, 24) inn__rrppeet_arra(()) 3, 4, {1, 3, 5)
(12, 3, 9)
*еn
11
in zange( 0, arr arr_atze-2): #t begin of insertion #+#
13
14
for 1 in rangel i-1, arr arr_size-1): for ranget 1+1, arr_sizel:
16
*4+ end of insection ##4
if Alil A'3l A[k] sum: return A(3),A(3),A(k)
18
return True
Folse
Example 1: Python left-to-right mode
begin of left right #44
15 A[i]
A[i+1] Ali-2]
rocurn (A[3), A(i+1", A(:+2))
return None left-right 649
#44
Example Java insertion mode
Import java.io..)
import Java.leng.A;
Import java.util.Aj
CountVariable
/As
write function to iterate over elements repeating each as many times as its count.
0, -2)
10
["p", "p", "p", "p", "e", *c*]
11
A. bounttariisbl.ccunttaiii. ["e", "p", "E", "p", "p", "p" "p", "p", "p", "p", 'e", taty "a", "a", "gtr "a"y "aty q", "q", "q", "q", "q", "q", "g", "g" fg", "g", fr "y", "y", "y", "y", sy", "y", "y", Tp", "y", "y", *a", fet, rar, nat, fot, nst, fer, nery s", *a", ta", ta", *3", *a", *3", *3", *97, ret, "a", "a", "a", "3")
1, 2, 3) ["q", "y", "y", *3", *g", *g*]
13
15, 12, 23)
15
16
18
public static List<String> countVariable (int &, int by int e, ins d) your code tist<Scring> finalResuls a new Arraytiste>> List<Steings countabletist new crray Suuntablllist.dd(""""
20
22
23
countabletiat add ("d(")))) countablelist..ddd countsbluuist..cd((""")
25
27
int countFor = a;
28
Ioz (int index index countFory indext11
29
30
begin insertion
32
34
countPor b; for (int index index countFory index
35
36
37
insertion
39
countFor = c)
Fount indextt!
48

---

Published as a conference paper at ICLR 2023
fint///eett...dccccnneeeleeeeree (
42
43
46
countPor
45
foz (int index index countrory indext+]
46
Sinalsosult.odf (wwuntablet.ist.......oo.. (3)1;
47
}
49
return fing Result;
50
Example 2: left-to-right mode
#4 begin of left-right #it
countableList. remove ge
countPor by for (int index index countFor: indextt1
addileesutt.addccc. (ccttaaaeecc get (0)
countableLiatt.
(ccuntableiist.cc
countPor -;
10
for (int index 0; index countFor; index++]
11
CinalResult add add abbbleee.......e............ (0));
12
countablelist. reneve (ccuntablelist. (0)
13
14
countFor
15
for (int index D; Index < countFor; indextti
finalResult.c (cduttt (ccunt ableliett get (0)); countablelist.
17
remevee remeve (ccuntabeeiist.ger (0)
18
finelkesult;
20
22
#4 end of left right #4#
Example 3: JavaSeript insertion mode
Write function to find all anagrans of string in given 11st of scriinga using lmmoda function.
anagrantamoda || ["boda", "aboe", "cboa", "obea", "adob"), "abod") ("ooda", "obda", "adob"]
anagranLenbda ["recitals", pythoo"), "articles") ("recitaia"]
anagramtamoda [[" keep", abadef", xyz"), peek*)
[* keep"!
+/
10
Tunct anagramtambda (texts,
11
const countFunc (st:) =>
result
13
for (let i = 0; i lengthh i++) char str[i];
15
LE (cesult[chat]] {
16
rohult +=
else result[char] 1;
18
20
return result;
22
23
const countStr = countFunciatry,
return
(word) let result 11i londhtength; i++) (
25
(let let char word[1];
27
28
#+ begin of insertion 4*#
30
LE (seault [seault[char]] result[shar] 1; #+ end of Insertion trt
32
elsa result[char]
34
35
36
37
return Ob'ect keykeyyyaaaana (ccuntStri every (char) =>
countSte result (chaz)
39
))
));
49

---

Published as a conference paper at ICLR 2023
Example JavaScript left-to-right mode
244 begin left-right ###
1f (sesult result [char] countstr(char);
elsa {
result/cheze countStr(char)/
object (zesult) rreduce (fa, b) ((a, b) => a + b, C)
(count5Ur)
});
10
end left-right #4#
50

---

Published as a conference paper at ICLR 2023
L CODE SUMMARIZATION, S-MBXP
L.1 DATASET PREPARATION AND EVALUATION SETUP
Here we re-purpose the original MBXP datasets for code summarization task. We remove the natural language description from the original prompt and use the function signature and the canonical solu- tion as the model input for code summarization. To induce the model to generation natural language in comments, we design two types of prompt in zero-shot and few-shot setting, respectively.
Zero-shot Evaluation. In this setting, we append "The above code writes in the format of code comment after the original code prompt. For example, in Python, the appended sequence is "# The above code writes a See examples different languages in Section L.3
Few-shot Evaluation. In this setting, we select three code-summan pairs and prepend them before the original prompt. Examples are shown Section L.3
To evaluate the code summarization performance, we use smoothed BLEU score as the metrics following the setting in CodeXGLUE (Lu et al. 202T) that compare the generated outputs with the groundtruth docstrings. In MBXP datasets, the summarizations are short paragraphs with one two sentences which makes smoothed BLEU score a suitable metrics (Feng et al. 2020).
L.2 EVALUATION RESULTS
Experimental results are shown in Figure 23 covering Python, JavaScript and Java. Overall, we found that performances are improved along with the increasing of the model size. For example, the BLEU-4 scores on Python language in 13B, 672M and 125M models are 6.07, 5.59, 3.20 under zero- shot settings, and 34. 10, 24.72, 20.75 under few-shot settings. We also noticed that multi-lingual models achieve better performances compared with monolingual models trained on individual lan- guages. An interesting observation is that though the monolingual models are trained on a specific language, they can generalize to other languages well when few-shot examples are provided. From the table we can also notice that the improvements brought by few-shot settings are more significant on larger models. Comparing the multi-lingual models and monolingual models under few-shot settings, found that the multi-lingual models are more robust the few-shot examples while monolingual models in smaller sizes show unstable performances.
MBPP delsset
MEJSP dalasel
MBJP dalseet
partipoott
3
1784
17M5
2.34
120
RDM
2:20
180
479M
2.20
19
Models
Figure 23: Code summarization evaluation in BLEU scores for all models.
L.3 QUALITATIVE EXAMPLES
Zero-shot Prompt Examples. Here we list some examples in Python, JavaScript and Java to show the zero-shot prompts, generation from models and their ground-truth.
Example Pythen-0
begin of prompt
2x)
discriminent = (y+*2) - 14xxaz)
diseriminant.
return ("Twe "alutiona",diaariminne
elif discriminent 0:
return (*one solution", discriminant)
elif discriminent < 0:
10
return (*no zeal solution", discriminant)
11
The above code writes
12
and prompt
51

---

Published as a conference paper at ICLR 2023
13
14
Croundtruth: Write function to calculate the digoriminant volue.
15
16
Generation: Weite "functien" that returne the value of the discriminant quedratis
equation.
18
Example Python-1
begin prompt
20
from cal lectiontnnn ions import Counter
21
def add_dict (21,
22
add_dict Counter Ceunter
25
returr add
The above code writes
25
end of prompt
27
Cuounstruth: Write Function to conbine two dictionarlen adding values for common keys. Generation: Write *add "add dies" function that adds two dictionaries together.
28
Example Java-0
begin of prompt
Import java.1o.*;
import java.lang.e;
import jeva.util.n)
Changalittaoorma
10
public statie Steing changeDateFo:nat (String dt)
11
String[] tokens st.sppit("" ");
12
return takens [2] + tckens[1] tokens [0];
14
15 // abova
16
end of prompt
17
18 // Groundtruth: Weite function convert date of yyvy-mm-dd fermat to dé-mm-yVVy format.
19
// Ceneration: Write .txt file with the date in the format "ad-mn-yvyy"
Example SavaSeript-O
----- begin of pronot
Sunction swaplist (newlist)
let first newlist
let last newlist [nwwlist. 11; newlist newlist newlist(newwl:dt (n.wlist.lengtt. - 1] first;
return newlist;
10 // The above code writes a
prompt
12
13 Groundtruth:
javascript
intershango first
1ist.
14
// Generation: Mrite "swap" Iuaction that awaрs the first and last elements of 1ist.
Few-shot Prompt Examples. The following examples show the few-shot prompts in different languages.
Example Python-0
begin of prompt
aummarize the functionality of the code
Code
import
def find_char_ (text):
return (ro. findall (r*\bw 18b", text)] Summary: Xrite a Function to rind all words whi ch are least 4 charaezers long in string by uaing zegex.
10
#4+
11
Code
Einé_Rotations/ttte cmp str + str
13
len (stx)
15
For 1. in rangell,n 11:
16
substring tmpli: i(n] if (str substring) reture
18
return
20
Summory: Krite a Fython function to find the minimum number of retations required to get the sama string.
52

---

Published as a conference paper at ICLR 2023
22
Code
23
def equare_nume (numa):
24
square_nums 1ist (map (lambda
nums))
25
return square_nums
Summory: Write function find aquares of individsal slemants in list using lambda Function.
28
Code
29
def check_tuples (Leat_ R):
30
all fele in For ele in test_tuple)
31
return (res)
Summary:
33
end of prompt
35
Crounctruth: Write function to check if the given tuple contains only k elementa,
36
Generation: Weite a Sunction to check if tuple is a subset of 1ist.
Example Java 0
begin prompt
// summarize the Sunctionality of the code
// Code
Import java.io.*;
import java.lang.
import jeva.util.*;
10
11
class PindChariong !
12
13
public statle tist:String>> rinechartiong iString text)
Siste0Lrings result
Arraytiatex
15
(String str : text.split( "))
17
If (str. (str.))ngth() >= 4)
18
result.add
20
result;
22
1n// Summory: Write function find all words which are chorasters long
string by using rege
24 x.
///
26
// Code
import java.io.1;
28
import java.leng.1/
Import java.util.43
30
31
FindRotations
33
public static int findkotations (String stx) {
35
String tmp str str; int str.length for (int 1 = 1; <= n; 1-+)
37
38
String substring Lmp, substring niz if (stz. (subsrrinng)))
40
42
43
44
return
45
47 // Summary: Write javascript function to find the minimum number of retations required to
get same string
48
49
///
50
// Code
51 Inport java..o.*;
import java.leng.Ap
53
import java.utál.A]
55
56
class SquareNums
58
public ststio List<Integer> squarettums (List<Crt numa)
60
Mist<Tntgger> squaretis: new Arragtilatstnteger221
53

---

Published as a conference paper at ICLR 2023
(intager num : nums] squaretist add (numenam);
63
65
return squarelist;
66
68
// Summary: Write function to find squarea of individual elementa in a 11at usiin lambda
69
///
70
// Code
71
72
import java.io.*;
Import java. 12ng.A
74
Import java.util.a/
75
76
77
class RecurGed
78
79
public static int recurGed (int a, int 51 (
Int low Math.min bl: İnt high - (a, b);
81
83
return high;
86
1 eise if (low -- 21
86
recurGodtlow, higa low);
88
90 // Summary:
91
end of prompt
92
93 // Groundtruth: Write a function to find the greatest common divisor (ged) of two Integers by
using recursion.
94 // Generation: Write functicn to find the grestest common divisor of two numbers.
Example Lavascript-0
// begin of prompt functionslity
// Coda
Tunctionn rinoChartong (text]
let regex /Abwe14, /bb/g
(regex);
// Summary: Rzite function to find words which least string by using zegex.
long
12 // Code
13 function (antat (
14
let
15
let - str.iength
17
for (let i 1, i <= i++] {
18
substring emp.subbtring if (etr substring) { return
19
20
22
24
25
// Summary: Write javasoript Function to rind the minimum number of rotationa required to get the same string.
26 ///
// Code
28 Function squareliums (nums) code
30
let aquare_nume pun nums mamtmnctp// (num)
31
return num ** 2;
32 });
33
return aquare_numa;
35 // Summary: write function to find squares of individual elements in list using lambda
function.
36
///
37
// Code
38
function EindMin low, hight
39
while (low hight ("
(high lewl
41
if (arr [mid] arr(high)) I 1;
54

---

Published as a conference paper at ICLR 2023
44
if (arr (righh)))) 1
high mid;
46
47
return ari[high];
49
// Summary:
51
erd of prompt
52
53 // Groundtruth: Weite javascript function find the minimum element in sorted and
rotated arrey.
44 // Goneration: Write function find the minimum element in 11sz using lamada function.
55

---

Published as a conference paper at ICLR 2023
M EVALUATING PUBLIC MODELS
We used MBXP to evaluate several public models such as OPT (Zhang et al. 2022b), BLOOM (Mitchell et al., 2022), and CodeGen (Nijkamp et al. 2022). We use pass@] evaluate these models, where we generate the samples using greedy decoding. We generate 256 tokens per example and truncate the output to one function for evaluations. The trends we observe with public models are aligned with those observed with our models. In general, we observe a log-linear performance gain with model sizes, across all model families, and better execution accuracy in-domain languages.
Among the general large-language models we observe that BLOOM models outperform OPT mod- els (See Table (). This can be attributed the fact that 13.4% of the pretraining data used for BLOOM models is code, while OPT does not train on code specifically. BLOOM's pretraining data includes code in PHP, Java, Python, Javascript, and Ruby among others, making them all in-domain languages. This explains relatively similar performance across all languages barring Kotlin and Ruby.
CodeGen models trained in three stages, first is text pretraining, which is followed by code pretraining and python-only training. Here pretraining code data includes code in Python, Java, Javascript, C++, and Go. The CodeGen-multi refers to the models at the end of the code pretraining stage without the python-only training, while the CodeGen-mone is models at the end all three training stages.
Experiments with CodeGen models show similar performance trends as our models listed in Section 4.2
Specifically, observe that large models show better than log-linear performance out-
of-dmmi languages (Kotlin, Java, and PHP). Interestingly, when compared with CodeGien-mult,, CodeGen-mono 16B models show 6%. and 8% improvements on JavaScript, and PHP, respectively (See Table b). We speculate the additional training with python data has improved model perfor- mance in other languages as well.
With few-shot prompting, we observe significant improvements in out-of-domain languages. Specif- ically, accuracy with Ruby (which is typically confused with python by the models) increased from 3.5% to 16.46% with few-shot prompting on CodeGen-mult models with few-shot learning (See Table 8 In translation mode, barring Ruby, we find significant improvements in languages (See Table 10
56

---

Published as a conference paper at ICLR 2023
Table 6: Evaluating pass@] execution accuracy of publicly available models on MBXP
using greedy decoding
Model Family
Model Size
Python
Java
JavaScript
Kotlin
Ruby
PHP
350M
1.54
3.21
3.21
0.83
0.00
2.80
760M
4.52
4.76
4.66
0.83
0.00
5.90
BLOOM
1.3B
5,34
4.66
5,49
2.07
0.00
6.94
2.5B
6.88
6.83
10.14
4.66
0.00
11.59
6.3B
6.98
7.76
7.35
6.21
0.00
6.94
1.38
0.10
0.00
0.83
0.52
0.00
0.41
2.7B
2.05
0.00
1.14
0.93
0.00
0.31
6.7B
2.05
1.97
1.35
1.55
0.00
1.66
OPT
13B
1.35
1.35
1.76
2.17
0.00
0.72
30B
1.64
1.45
2.69
1,45
0.00
1.55
66B
3.70
2.28
3.73
1.76
0.00
2.17
350M
7.90
8.17
7.45
1.14
1.04
0.8
2B
18.78
19.56
17.70
3,93
4.76
2.90
CodeGen-multi
6B
22.48
21.74
22.87
4.55
4.24
5.90
16B
24.22
28.05
26.29
7.04
3.52
10.35
350M
18.37
1.86
6.00
1.04
1.55
1.35
2B
31.72
16.66
22.04
3.21
2.90
9.21
CodeGen-mona
37.16
19.77
27.74
3.83
1.66
10.14
16B
40.55
26.81
32.81
6.63
5.90
18.94
125M
6.37
5.59
6.94
1.04
0.21
0.52
672M
19.71
17.29
21.43
4.55
1.86
7.76
Ours-multi
2.7B
27.00
22.46
27.23
9.73
3.93
11.28
13B
35.32
30.33
36.13
14.18
6.73
18.84
125M
8.11
4.35
7.45
672M
19.82
11.39
14.39
Ours-mono
2.7B
24.13
15.32
19.67
13B
33.57
19.77
23.60
Model Family
Model Size
go
C++
c#
Typescript
Perl
swift
scala
1.3B
0.11
0.35
0.00
0
0.1
0.1
0.52
2.7B
0.43
0.47
0.00
0
0.1
0.83
0.21
OPT
6.7B
1.28
1.53
3.31
3.31
0.52
0.93
0.31
13B
1.7
1.06
3.31
3.31
0.31
1.97
0.31
1.1B
3.3
5.07
0.31
0.31
0.21
0.62
0.21
1.7B
4.15
6.01
0.31
0.31
0.1
2.07
0.41
Bloom
3B
6.28
8.61
10.95
10.95
1.55
4.24
0.62
7.1B
7.77
15.09
13.84
13.84
3.31
5.07
0.21
350M
1.38
5.19
7.13
7.13
0.1
1.14
0.1
2B
5.11
17.69
20.76
20.76
0.83
2.59
0.1
CodeGen-Mon
6B
3.83
17.33
19.21
19.21
1.24
4.14
0
16B
10.54
29.13
29.96
29.96
2.48
4.55
0.31
350M
6.39
9.32
7.13
7.13
0.1
1.66
0
2B
12.03
18.04
17.25
17.25
2.07
2.17
0.62
CodeGen-Muile
6B
11.61
17.69
17.46
17.46
2.07
2.8
0.31
16B
15.23
26.06
21.49
21.49
7.14
3.62
0.41
125M
0.64
1.53
0.62
6.61
0.1
0.41
0
672M
7.67
4.34
19.83
1.35
1.76
0.52
Ours
2B
15.68
8.16
26.14
0.93
3.11
0
13B
5.22
18.75
10.54
32.85
4.14
6.52
0.1
57

---

Published as a conference paper at ICLR 2023
Table 7: Evaluating pass@1 execution accuracy of publicly available models on Hu-
maneval using greedy decoding
Model Family
Model Size
PY
Java
JS
Kotlin
Ruby
PHP
Perl
Swift
Scala
1.IB
3.66
3.73
2.48
0.62
0.00
2.48
0.62
0.62
8.07
1.7B
3.66
1.86
4.97
0.62
0.00
4.35
0.00
0.62
24.22
Bloom
3B
7.93
4.97
5.59
2.48
0.00
4.97
0.62
1.24
29.19
7.1B
7.93
8.07
6.21
0.62
0.00
3.11
0.62
2,48
34.16
L.3B
0.00
0.00
0.62
0.62
0.00
0.00
0.00
0.62
0.00
2.7B
0.00
0.00
0.00
0.00
0.00
1.86
0.00
0.00
0.00
OPT
6.7B
0.61
0.62
0.62
0.62
0.00
1.24
0.00
0,62
9.32
13B
0.61
0.62
2.48
0.62
0.00
1.24
0.00
0.62
12.42
350M
10.37
1.24
0.00
0.62
0.00
0.62
5.59
2B
20.73
4.97
10.56
1.24
0.00
3.73
1.24
0.62
8.07
CodeGen-Mono
6B
19.51
8.70
11.18
1.24
0.00
4,35
1.24
0.62
6.21
16B
22.56
17.39
12.42
0.62
0.00
11.80
2.48
0.62
16.15
350M
7.32
4.97
0.00
0.62
0.00
0.62
1.86
2B
10.98
11.18
6.83
3.11
0.00
1.86
1.24
0.62
21.12
CodeGen-Multi
6B
15.24
10.56
11.80
3.11
0.62
3.73
1.24
0.62
10.56
16B
17.07
16.15
16.15
1.86
0.00
5.59
3.11
0.62
16.15
125M
7.32
3.73
1.24
0.00
0.62
0.00
0.62
672M
17.07
9.32
13.04
1.86
0.00
1.86
1.24
0.62
1.24
Ours
2B
19.51
14.29
14.91
1.86
0.00
4.97
0.00
0,62
1.86
13B
22.56
22.36
20.50
8.07
0.00
11.80
3.11
0.62
4.35
Table 8: Evaluating pass execution accuracy of publicly available models on MBXP with few-shot prompting using greedy decoding
Model Family
Model Size
Python
Java
JavaScript
Kotlin
Ruby
PHP
350M
7.80
10.56
8.28
2.28
4.24
3.2
20.02
22.15
21.43
7.76
12.73
8.8
Codegen-multi
6B
23.10
24.53
24.84
10.66
9.01
13.87
16B
26.69
29.92
28.26
11.59
16.46
17.60
350M
17.04
3.73
5.18
2.69
4.35
2.69
2B
28.95
15.42
15.53
5.69
8.07
11.70
CodeGen-mono
6B
39.53
21.43
19.15
7.14
10.35
16.4
16B
46.41
27.64
26.50
11.08
15.11
20.2
125M
6.06
7.14
6.94
2.90
2.59
0.93
672M
19.40
16.56
19.46
5.90
5.90
8.90
Ours-multi
2.7B
24.23
24.12
27.95
11.08
9.42
14.29
13B
31.93
30.75
37.37
15.11
12.53
20.19
125M
8.93
5.38
7.35
672M
18.89
10.56
16.87
Ours-mono
2.7B
21.77
15.11
21.43
13B
31.31
21.22
25.16
Model Family
Model Size
Go
C++
C#
Typescript
Perl
Swift
Scala
350M
1.17
4.25
4.03
3.93
1.45
2.90
28.78
2B
5.86
19.34
8.99
16.94
4.97
4.45
26.29
CodeGen-Mono
6B
6.50
19.69
12.71
18.08
3.42
4.24
26.50
16B
13.84
32.31
16.63
28.20
8.18
5.49
28.67
350M
6.39
9.91
5.68
9.19
1.66
3.62
30.43
2B
14.59
19.46
11.05
18.80
4.24
3.83
37.68
CodeGen-Multt
6B
12.78
21.58
13.43
19.83
6.00
4.55
28.26
16B
20.77
29.36
17.46
24.38
8.49
4.55
28.57
58

---

Published as a conference paper at ICLR 2023
Evaluating pass@1 execution accuracy of publicly available models on Hu- manEval with feo-shot prompting using greedy decoding
Table
Model Family
Model Size
Java
JS
Kotlin
Ruby
PHP
Perl
Swift
Scala
350M
12.80
3.11
2,48
1.86
2,48
1.24
0.62
0,62
19.25
2B
21.95
8.07
10.56
3.11
3.11
6.83
1.86
0.62
15.53
CodeGen--Mooo
6B
23.17
8.07
8.70
3.11
2.48
7.45
1.24
0.62
13.66
16B
28.66
16.77
11.18
4.97
5.59
9.94
3.73
1.24
18.63
350M
7.93
4.97
4.35
1.86
1.24
1.24
1.24
1.24
22.98
2B
13.41
10.56
12.42
4.35
7.45
4.35
4.35
0.62
24.22
CodeGen-Muult
6B
14.02
12.42
11.80
3.11
4.97
3.11
3.73
1.24
13.0M
16B
20.12
17.39
13.66
4.35
8.07
9.94
5.59
0.62
16.15
125M
7.32
4.35
5.59
1.24
0.62
1.86
0.00
0.62
0.62
672M
17.07
8.70
11.18
3.73
2.48
4.97
1.24
1.24
1.86
Ours
2B
19.51
14.29
14.91
4.97
3.11
8.07
2.48
1.24
3.11
13B
22.56
18.01
26.09
4.97
6.21
10.56
3.72
0.62
5.59
Table 10: Evaluating pass execution accuracy publicly available models on MBXP in translation mode (Python as source language).
Model Family
Model Size
Java
JavaScript
Kotlin
Ruby
PHP
Go
350M
3.83
10.24
3.11
4.66
2.07
3.94
2B
22.57
22.87
5.18
4.35
21.74
8.41
CodeGen- Mono
6B
30.95
35.92
8.07
6.63
30.23
11.93
16B
43.48
54.14
10.56
4.55
36.85
22.36
350M
7.35
9.93
1.55
3.83
1.76
7.14
2B
23.19
36.12
3.42
4.76
22.46
16.83
CodeGen-Mullt
6B
38.41
37.99
6.83
4.66
27.74
20.77
16B
44.82
50.10
8.28
5.18
40.17
26.41
125M
7.04
9.52
3.21
3.73
0.41
2.77
672M
22.98
24.53
7.97
5.07
16.15
6.28
Ours
2B
30.75
28.47
10.56
6.52
33.02
6.92
13B
41.93
37.06
14.80
6.73
37.06
8.52
Model Family
Model Size
C++
C#
Typescript
Perl
Swift
Scala
350M
8.37
3.93
11.78
0.21
1.24
0.00
2B
30.19
16.01
31.51
4.14
6.83
0.00
CodeGen-Moon
6B
35.14
23.45
36.67
7.56
7.04
0.10
16B
49.65
36.67
0.41
7.04
10.97
0.10
350M
10.85
1.86
5.89
0.00
0.83
0.00
2B
23.00
6.71
10.54
5.18
3.62
0.00
CodeGen-Multt
6B
40.45
22.52
26.96
8.39
6.21
0.10
16B
46.58
28.51
5.89
14.18
8.18
0.10
125M
1.06
1.76
10.54
0.52
1.76
0.00
672M
16.51
10.43
19.11
4.66
3.93
0.10
Ours
2B
28.18
13.33
24.90
5.07
6.52
0.10
13B
33.14
25.52
44.52
10.56
8.49
0.10
59

---

Published as a conference paper at ICLR 2023
N TRAINING
N.1 MODEL ARCHITECTURE AND TRAINING DETAILS
We train using 210B tokens for mono-lingual models, and 630B tokens for multi-lingual models with 210B tokens from each language. Across all models, we use max sequence length of 2048, and use larger batch size for larger models, while reducing max steps accordingly to train all models with same amount of per-language tokens. For example, for 13B, we use batch size of 1024 and max steps of 100,000 with 2048 sequence length, resulting in total 210B training tokens for each language. For multi-lingual models, there are three languages and increase max steps by three times to 630B tokens. We use AdamW optimizer (Loshchilov & Hutter 2018) with 31 = 0.9, 10-8 We use warm up steps of 2000 steps with cosine annealing after peak learning rate, and the mininum learning rate being 10% of corresponding peak learning rate, weight decay of 0.01, and gradient clipping of 1.0. We rescale the initialization weight standard deviation for larger models following (Shoeybi et al.) 2019) for better training stability. Our training pipeline is based on PyTorc Lightning and we use b (Kalamkar et al. 2019) and DeepSpeed (Rasley et al., 22200) for training optimization. We randomly split 0.1% data as validation set. The validation loss curve for different sizes of multi-lingual and monolingual models are shown in Fig. 24
= 0.95, and
Jnd
Pretraining
Code Protroining
128m
malli sychon
14
124m symtnn
672m puthon
228m UiA
672 moll.ss
128m. a13
(72m Joas
1.2
2.0
3.0
0.8
3.6
0.0
102 tskera (eil ont
200
200
1.0
10ienn (clllon)
(a) 128M
(b) 672M
Code Phetraninng
Code Preisaining
1.-
2.70 madi cythen
Tu pothia
1.4
250 Ui Uyrthnn
235 gylhen
avancrst
23c pounseipe
7h Jaranerpe
536 Jaasectpt
13300 on 100
2.5
328
150
203
100
200
(c) 2.7B
(d) 13B
Figure 24: Validation loss curves for 128M 672M, 2.7B and 13B multi multi-lingual and mono- lingual models.
N.2 OBSERVATIONS ON VALIDATION LOSSES VERSUS PERFORMANCE
We plot the validation loss of multi/mono- -inguul models on each programming languages in Figure We can see that the trend of validation loss roughly follows log- linear relationship with respect
25
to model sizes.
By comparing the validation loss curves between multi-lingus, models and mono-lingus models, we can see that mono-lingual models consistently achieves lower loss than multi-lingual ones. This
tttps://ww..yrorrehh http:///w.p.prrr....hhn.........
60

---

Published as a conference paper at ICLR 2023
Code Pretraining
multi-py python python
1.0
muttijor java
mutti jovascnpt javascript
0.8
validation loss
0.6
0.5
0.3
207
20
parameters (Billion)
Figure 25: Validation loss vs number of parameters for 128M, 672M, 2.7B and 13B multi- lingual and mono-lingual models.
demonstrate that is more difficult for the same size of model fit multi-lingual datasets since with limited model capacity it needs to learn more diverse information while mono-lingual models can be more concentrated.
However, although the validation loss of mono-lingual models is generally lower, from Section 4.2 we observe that in terms of execution performance pass@k, multi-lingual models actually out- perform mono-lingual ones especially when model sizes go beyond 672M. In fact, as model size increases, the improvement of multi-lingual models over mono-lingual models get more and more significant. The reason could be that although models get distracted to fit multiple languages, the knowledge sharing across different languages helps model learn better in solving problems. For example, similar tasks might exist in different programming languages, hence models are easier to learn to transfer from one language to another. And larger models have better capability in knowl- edge sharing/transfer learning, with the evidence that the zero-shot learning performance of multi- lingual models on unseen programming languages get significantly better than mono- long-lln ones model sizes increases.
61

---

Published as a conference paper at ICLR 2023
O DATASET CONVERSION FRAMEWORK
In this section, we describe a dataset conversion framework that transforms an execution-base evaluation one programming language another. In particular, we focus on a function completion format of snecution-based evaluation as shown in Figure 2 Each problem in a function completion dataset consists of prompt, test statement, and a canonical solution. The prompt contains function signature along with doestring describing the desired functionality of the code. The canonical solution is an example of function body that fulfills such functionalityy, usually written by human annotators. Given candidate function body generated by the model, we can test whether the corresponding function is correct by executing the test statement against the candidate function.
To construct evaluation dataset for function completion in new language, recognize that is sufficient to convert only the prompts and the test statements (Section O.I]. That is, we do not need to transform the canonical solutions, since they simply examples and are not used to measure correctness in the test-based framework. This key feature of a test-based evaluation makes it possible perform mapping of an evaluation set from one language static analyses, as outline below. For other code-related evaluations that require access to canonical solutions, we synthesize solutions by generating many code versions based on the converted prompt and use our converted test statement to filter for correctness (Section 0.1.1I..
many others by
0.1 LANGUAGE CONVERSION OF PROMPTS AND TEST CASES
O.1.1 FORMAT CHOICE
The purpose of this work is to build datasets that allow us to measure multi-lingual code generation abilities. The function completion format helps steer the model to generate code in a specific lan- guage since the prompt consists of partial function that has already been started, i.c. function signature. This is contrast to other formats such as that of the original MBPP where the prompt does not consist of a funetion signature, but contains more implicit information such assert state- ments, example function calls, and a description such as "Write code in Python" (see Appendix R.I.I for examples). Compared to other formats, the execution-based function completion aligns well with how an ML-driven model would perform code suggestion in a typical coding environ- ment. Therefore, process our converted datasets and the original datasets (MBPP, MathQA) to be of this format, except for the original HumanEval dataset whose format is already consistent.
O.1.2 INFERRING ARGUMENT AND RETURN TYPES
This step is applicable for statically typed target languages such as Java, C#, etc. The process starts from inferring the types of function arguments, which can be done by inspecting the argument values. We perform mapping of types from Python to types in a target language; for instance, to convert to Java, we map list + ArrayList or dict + HashMap. Values for different test cases can have different types, therefore we infer the common superclass of all observed types for each argument. Since there can also be many levels of types, due to containers such as list or sets, recursively infer the types among each level to be consistent. For example, "list of list" and "list of object" has a common type of "list of object". The return type is inferred via expected return values in the test cases.
0.1.3 SUPPORTED TYPES OF OBJECT CONVERSTON
Our conversion framework depends on the structure of basic programming problems which involve object types of the following:
Integer or long version of integer
Float or double
Boolean
String. We assume any string of single character is also of type string for the purpose of conversion.
None. This depends whether the target language also supports None/null/nil types.
62

---

Published as a conference paper at ICLR 2023
List. Tuples in Python also converted to list in all languages.
Dictionary
Set
For any container type, we recursively perform object conversion for all nested structure within the container.
O.1.4 CONSTRUCTING REPRESENTATION OF CODE OBJECTS
We convert the argument and return values from Python target language by generating strings that represent the target language's objects. For example, the object [1, 2, 3] in Python is con-
verted Arrays. asList (1, 2, 3) Java, as shown in Figure We recursively construct container elements for any nested structures.
O.1.5 CONVERTING TEST STATEMENT
We construct objects for function argument inputs, using above information regarding constructed objects and types, well as expected output. We build the test statement target language using appropriate assertion to match the returned value with the expected output. We perform deep object comparison with an appropriate comparator each language.
0.1.6 PROMPT CONSTRUCTION
ensure that the converted function, argument, and class names are stylistically appropriate for each language, e.g. camel case versus snake case, etc. We construct function call examples in the docstring to look representative without being too verbose, e.g., we use 2, 3 in Java's 's docstring to represent list, instead of actual ArrayList. We avoid using language-reserv words for variable names such as end for Ruby or char for Java or C++ and escape certain substrings that keywords such /
We also deal with all formats in the prompt with great care. For instance, docstrings for Java and JavaSeript be before the function signature, following the convention. For Java, this is crucial, otherwise it would be too out of distribution and the model would not generate anything, if docstring below function signature.
O.1.7 DOCSTRING AND NATURAL LANGUAGE CONVERSION
The natural language statements for datasets such MBPP can contain python-specific statements that might not be applicable to Java or Javascript such as "Write a function in Python to or "... the object does not exist, return None". We substitute "Python" with the target language name, "None" as appropriate null values.
O.1.8 VALIDATION
We validate that converted objects, test statements and function signatures parse and/or compile with respect each language.
0.1.9 QUALITY CHECK VIA REVIEWERS
To gauge the quality of our conversion, we also request annotators to manually review the converted programming problems in sample languages, namely, Java and JavaScript. We ask language experts to identify issues with converted examples consisting of natural language statement, test cases, and function signature and use this process to help iteratively improve our conversion algorithm. For the final review, annotators have not found issues specifically related to language conversion, but observed ambiguity in some cases attributed to the original dataset. In the future, any updates to the original datasets can propagated to all converted languages programmatically. We provide the detailed analysis of evaluation by annotators in Appendix Q
63

---

Published as a conference paper at ICLR 2023
0.1.10 COMPARISON TO OFF-THE-SHELL ML TRANSLATOR
We find ML translators to be insufficient perform the dataset conversion, due limited support for language pairs, restriction on format, as well as transformation errors related to types and object construction. contrast, our framework can convert data to many target languages and does not have non-deterministic errors related to type inference or object mapping. We provide further discussion and examples in Appendix O.2
0.1.11 SYNTHETIC CANONICAL SOLUTIONS
The availability of canonical solutions in each converted language open up the possibilities to perform other types of evaluation. To generate such synthetic solutions for each language, we sample up 10, 000 versions of code per problem and filter them for correctness with our converted tests. In order to generate at least one correct solution for many problems as possible, we use both the function completion and zero-shot translation settings (see Section (43) where we prepend the Python solution, provided in the original datasets by human annotators, to the beginning of function signature prompt. With high-temperature sampling, we are able to generate correct solutions for large portion of all problems, with 96% coverage for JavaScript, 93% for Java, and even on an out-of-domain language such as PHP with 93% coverage (more details in Appendix
O.2 POTENTIAL USE OF TRANSCODER FOR DATASET CONSTRUCTION
We conduct preliminary experiments using a publicly available code translation model Transcoder (Lachaux et al. 2020b) to perform dataset conversion. Overall, there are two main limitations of this approach. First, these models typically support limited number of language pairs, which means that we would not be able to perform conversion to 10+ languages like with our proposed framework. Second, we find that there are some common errors associated with type inference, for instance, when the return type should be boolean, the translation model can predict int as return type. These types error cause false negatives and can impact overall quality of the converted datasets. In contrast, we do not have these types of errors in our conversion framework due to the static analysis implementation.
In particular, Transcoder (Lachaux al. 2020b) supports Python, Java, and C++. In this setup, we use a complete function in Python as an input prompt. The transcoder model then generates a complete function in Java and C++. Here, function signatures that capture necessary information.
interested in whether the model is able translate
Example While the model seems able translate the function signature, the function name for Java and C++ appear to be in snake case, which is not the standard for these languages.
TASK ID:
Input in Pythen
def differ At Cwt___ne________e_eeeeeeeee____e (o,
return in Aa_Power_of_Twe
Translation in Java
public statie differrt_ono___ Bnt___t___________
translation in C+ bael Diffee__t_one__it_.ee ( int a int
10
12
Example The model seems to adapt the function name to be entirely different, i.e., 1
is_undulating to isAbundulating or
isskknddt isSkundtlating,
ID:
imput in Pythen
is_undullating(m))
if (Len (n) <= 25 retura Palse (2, lentn)):
for
if (n[i 21 != m[i]): return return True
dava
64

---

Published as a conference paper at ICLR 2023
static isAbundulating
12
13
14
Translation in CH
15 boel inskundulating string a
16
17
Example Incorrect type inference where a nested list nestedlist is Python is mapped to string in C++, or simply a flat list of strings in Java.
TASK TD: 111
Input in Python
def common_in itmmmo__n__ested__ii_e 1ists (sestedlist):
result = List (sot.int.orsaction (+map (set, nestedlist)))
return result
Translation in Java
public static String commonineestedliste (String restedlist
translation in C+-
11
string common commantnWostedhiletts { string neatedlist.
13
Example 4 test__iss is incorrectly inferred to have a type string in C++.
TASK_ TD: 117 (fn: 1_of_1)
Input in Python
def jist_to_flost (tesssttttttt______ litst
for tup in test_list: temp
for ele in tup:
if pla aemp.
10
else:
temp. eppand (flost
12
res.append (temp te],temp(1]])
13
recurn (str (res))
14
Translationn in dava
15
public static String listToDouble AzrayList String testList
16
17
19
Translation in Ct
string liDeubee string testlist
21
22
P SYNTHETIC CANONICAL SOLUTIONS
P.I MULTI-STAGE DATA BOOTSTRAPPING
We combine solutions for (1) normal function completion with few-shot prompting if the language is out-of-domain and (2) translation settings. This is because these different generation modes can synthetic correct solutions for different problems, according to our evaluation analyses
Section We perform data generation in multiple stages. We first sample n = 100 samples for all cases, after which we sample for n = 1000 cases for the problems where we have not found at least one correct solution. The last step contains uses 10000 samples. We use temperature 0.8, 1.0 and 1.2 respectively.
4.2
P.2 DISCUSSION: GROUND TRUTH ASSUMPTIONS OF TEST CASES
Our synthetic generation of canonical solutions make heavy use test cases to filter whether each code is correct not. The process implicitly assumes that the test cases act ground truth verifier that provides necessary and sufficient conditions for the correctness of each task's 's functionality. Such assumptions might not hold if the test cases not thoroughly written in the original dataset.
65

---

Published as a conference paper at ICLR 2023
In fact, false positives of ixecuution-based datasets typically occur indicated in several previous work (Li et al. 2022a). Our framework do not aim to inject additional knowledge per each test case but merely acts translate task in one language to another, while carrying the information captured in test cases of the original dataset to the corresponding datasets in other languages. Any corrections to the benchmark shall be done upstream in the original dataset, and can easily propagate to the rest of the converted datasets, since the conversion process for prompts and test cases is purely programmatic. The usefulness of the conversion framework lies in the automated conversion into many languages which can be repeatedly done if the test cases in original datasets are updated, or new tasks are added. This helps reduce human effort perform such manual translation of a dataset in one language to many others.
One step that can he done improve thoroughness of the test cases is to use the provided canonical solutions in the original dataset to synthetically generate additional test cases, with the hope that additional test cases can provide test coverage for the functionality. However, this proposal also relies heavily the canonical solution the ground truth that captures the true functionality of the task, which might not necessarily be true since during the annotation process, annotators might write canonical solutions that only partially correct but pass the specified test cases. Therefore, we leave this investigation as future work.
66

---

Published as a conference paper at ICLR 2023
Q QUALITY CHECK OF CONVERTED DATASETS
During the manual review process, we provide the converted programming problem Java and Javascript to annotators and ask them to check if the problem is correct and clear. Below catego- rize the issues identified by annotators, with examples. Almost all of these cases can be attributed to the source dataset we use for conversion. On the other hand, the conversion process itself does not introduce additional errors. For future work, we plan to thoroughly check for errors in the orig- inal MBPP. Any changes from there can be easily propagated the converted datasets due to the automatic conversion.
Natural language statement is ambiguous.
Import java.io.*?
3
Amport Java. lang.*;
Import java.util.4r
5
7
class CounLCormon
{
8
Write a funct ion to count the most common words in a dietionary. CountCommenn countCommoz [["red", "green", "black", "pink", "black", "uhlte", black", "eyes", "white", "black", "orange", "pink", "pink", "red", "red", "white *osange", "white", "black", "pink", "green", "green", "pink", "green", "pink", "white", "crange", "orenge", "red"];
[1"pink", 6), ["black", 51, ["white", 51, ["red", 4]] CountCommen commmcccoonttoommrr (["ona" "two", "three", *Four*, "five", "ane", "two", "one", "three", "one"])
[1"nne", ["Lwo", 21, ("thcee", 2], ["Cous", 111 >
15
([ (["aacebook", "Apple", "Amazen", "Netflix", "Google", "Apple", "Notflix", "Amazon"])
[1"Apple*, 21, (*Amazen", 21, ("Netflix", 2], ("Facebook", 1]]
ieblic statle 14 <ta 4Dbjeets countcommon (Id ring words)
19
Comment: problem should explicit
commen
Spelling mistake.
Zea
3
Wzite a jevascript functien to count numbers whose oth and nth bits are set. countNum (2)
5
7
DountNum
9
+/
function countNum,n)
12
should "Ces*
One more test cases wrong.
Import java.io.v
3
import lava.lang.*p
import java.util.e;
5
7
class eirstRepestedchar
8
9
/+a
write a java function to find the first repeated character in a given string. FirstRapeatadchar HirstRopeatedChar (*Goog "Google")
12
"g"
"a"
FiestRepeatedChar. http:hhhww...tttteeeeeeeeaaaaaaa (yythohn
"Ax00"
ioblic atatle String FiratRepeatedChar (String str)
19
test case incorrect.
67

---

Published as a conference paper at ICLR 2023
R DATASETS
R.I MBXP
Below, we show examples of the samples from the original dataset as well as the converted dataset.
Table 1I: MBXP Datasets in 10+ Languages. The dataset names are loosely inspired by each language's file extension. For instance, the Scala Perl dataset is MBSCP MBPKP due to the file extension being sc or .pl.
Language
Dataset Name
Python
MBPP
Java
MBJP
JavaScript
MBISP
TypeScript
MBTSP
Go
MBGP
C#
MBCSP
РHP
MBPHP
Ruby
MBRBP
Kotlin
MBKP
C++
MBCPP
Perl
MBPLP
Scala
MBSCP
Swift
MBSWP
R.1.1
MBPP: PYTHON
Note that we convert the original MBPP dataset (Austin et al. (021) which has slightly different format into HumanEval format (Chen et al. 2021) with function signature and docstring, as shown below. The use of function signature and docstring in the formatted MBPP makes it consistent with the converted datasets in all other languages.
MBPP/1:
PROMPE
der min_cost (cost, n)
Write function find minimum path reach m) from (0, g cost mateix cost [) and position [m, n1 in cost [1 []+
min_cost ([[1, 31,
21, 5, 2, 2)
>>> min_cost [[2, 3, 4), [5, 9, 31, 12, 6, 4,1, 2, 2)
12
>>> min_cost ([(3, 4, 5), [6, 10, 41, [3, 3, 511, 2, 2)
10
11
13
= 3
15
16
to [[0 for x zange [C)] for x in zange (R)
to(0] coat[00 [0]
18
For i in range 11, m+1):
tc[i] [0] [o] cost [0]
20
for 1 in range!1, m+1):
tc[o][i] to[0]
cost [0] [1]
22
For i range:l, m+l):
23
for in zange(1, ntl):
tell] [1] min(tell 11 11, tell 11:11, tetil 111 cost[l)
25
return te [m] [=]
27
TEST
28
def
30
assert candidate (1 [1, 2, 31. 14, 8, 2],
11,
5,
311
1
2, 2)
-- 8
assert candidate 12, 3, (5,
12,
41
32
assert candidate [3, 4, 51, 16, 10,
41
[3,
7
511
2.
21
== 16
R.1.2 MBJP: JAVA
68

---

Published as a conference paper at ICLR 2023
MBJ2/1:
Import java-1o.*; Import java. +
import jeva.util.
import jeva.mech.
XinCost
10
/Av
11
Rrite function Sind the minimum cost peth reach
from (0, for the
given cast matrix cost [1 [] and pasition (m, n) in cost (1
12
MinCost.minnoms noost 2, 3), 8, (1, 5, 3)1, 2,
13
MinCost.mincon ([[2, 3, 4), [5, 9, 31, 12, 6, 4)1,
15
12
16
A VinCost.minCost 4, 5], [6, 10, 4), [3, 7, 511, 2, 2)
16
18
+/
fublic statfe Int minCost (tsstttttttttttttegggee.. cost,
20
//
COMPLETION (model genezaled)
int new
[a]
22
Toz (int i 0z
11t)
23
for (int - 3 < as 5+4)
18 0 66 == 01
25
dp(1) (3) cost.get (f)
if [1
27
[p[i] [}] do(ill] 11 cost.get .gt (jiz
(j 0)
29
[p[i][]] dp(i 1][j] cost. (1) .get (j1z
30
else
cp(i) (3) (dp(: [1) cp(i) 1])
(1) get (1)
12
34
return dp [n
36
37
38
//
TEST
39
class Main {
public ststic
compare ject ebjl, jejtct obj21 return abjl.equll (ob)217
41
43
44
public static void moinintinngll arga) throws Exception
Int xo Mi MinCost.m noost (Arrays..a at 2, 31, Arrays.astistt4. Arcays.astist (1, 5, 31), 2, 2);
(! tecapare (x0, 8)11
47
throw new java. wwwa.lang.seee. ixc.ptxctttion ("Exception test case 0 did not pass. x0 x0)
48
49
Int MinCost. minCost aArrays.aatiat (Arrays 3, 41, Arrays.astist 15, Arcays.asList (2, 6, 41), 2, 2); (1 tecapare 12)1)
52
throw new jova. Excoption ("Exception tast case did not paas. xi x1)
54
int. MinCost, minCoet (Ariays (Arcays 4, Arraya 16, serays (3, 7, 5)1, 2, 2);
(! teompare 1))1)
57
throw new java. 1 Exceptiion ("Fxcept inn teat case 2 did not paas. x2 = x2)
58
59
61
1.1.3 MBJSP: JAVASCRIPT
MRJSP/1:
Write function to find the minimum cost peth to reach (m, n) Erom (0, 0) for the given cost matrix cost (1 and position
n) in cost cost[] [1. (I 2, 3), (4, 8, 21, [1, 5, 3)1, 2, 2)
mincost ([12, 3, 4), [5, 9, 31, 12, 6, 411, 2, 2)
69

---

Published as a conference paper at ICLR 2023
mincost([13) 4, 51, [6, 41,
7,
10
11
innetion minCost (cost, COMPLETICN (model generated)
12 //
Array(m 1).fill(() .ap Arroy f1.f
14
for Hlet 1 1; <= 1++)
15
flet
j [++]
16
dp[1] [3] cost 1](3 Kath. 111)), 6p 1111
17
18
19
recurn dp dm[m] [n]
21
//
TEST
23
const require [odash")
24
function compare (objectl, objec:2)
26
return wat.aiiuuulbbeeeeoee.o objact2)
28
31, if (compaze (s, 8)111 elae { throw 'Error at 2th assert stalement, Value J3ON. J30N. (x)
mincost
21, [1, 31:,
30
31 minuest ( 3, 41, (5, 9, 31, [2, 6, 41;,
32 if (compare (x, 121 alce throw "Error at 2th assert atatement, Value JSCK. fy Ix
33 var minCost (1 [3, 4, 51, [6, 10, 41, [3, 7, 5,1, 2, 2 (compare (x, (6))
"Erzor 2th assert statement. Value sttiinigg
R.1.4 MBTSP: TYPESCRIPT
MBTSP/1:
Write Lunction Lo find the minimun cost path to reach (m, n) Ezom (0, 0) for the given matrix cost () and poaltion n) in () [].
Exemples:
>>> min_cost [[L,, 2, 31, [4, 8, 21, 11, 5, 311, 2, 2)
min_cost (1 3, 41, (5, 9, 31, 12, 6, 41
12
min_cost [3, 4, 51, (6, 4), 7, 5.1, 21
11
3>> 16
13
const mir_cost function (cost: AeraycArray<mumbor>> m: numbar, n: number) number ( COMPLRTTON (model generated)
//
15
const []
16
for (let 0j i my 1++]
dp[i] [];
18
for (let 1 - 0; <- 3-4) dplil [5]
20
22
10[0] [o]]
25
for (let i 1, my 1++1 : dp[1][0] cos dpll 1110
25
(let dp(0](5] cost [0] [3 1.:
j [++]
27
28
29
for (let
2++1
30
for (let j = 1, ny 3-+)
dp[1]
32 //
TEST
33 import as assert from "aseert
34
35 let actual_: cost 2, 31, 14, 8, 21, t. 5, 311,2,2);
36 let expected
37 Lasert.doopEquu (astual_1, expactad_]);
39 1st actual_2 min_coat 3, 41, [5, 9, 31, [2, 6, 411,2,21;
expected_2 12;
41 Sssert.deepEqua (actual_2, expected_2);
12
43 actusl_3 min_cost|[[3, 4, 51, 16, 10, 41, 13, 7, 5)1,2,2);
44 lat expected_3 16;
deeert.desSquul factual_3, expected_3);
70

---

Published as a conference paper at ICLR 2023
R.1.5 MBGP: Go
//
MDGP/1: PRONPT
package main
//
Code should be written in Go/Golam lan
// Mrite Lunction Lo rind the minimun cost path to reach (m, n) Ezom (0, 0) For the given
cost
position (m, nJ cost
// Examples:
// min_cest (C[,, 2, 31, (4, B, 21, [1, 5, 31: 2,
//
// >>3 min_coot ([[2, 3, 41, (5, 9, 3), 12, 6, 411 2, 2)
10 //
11 // 2>2 min_cost ([[3, [3, 4, 51, [6, 10, 41, [3, 7, 5:1, 2, 21
12 //
13 fune min_coat (cost [] int, int) int
14 //
COMPLETION (model genezated)
15
16
C := 3
make(
18
tet0: [0] :- cost [0]
20
[o[1] [c] cost(1)
for
23
to(0] :- tc(o] [i-1] coat [0] I1]
25
For 1 2,
for 1 ==
27
teli] (1l :- min(to[l-11 se(1-1):1), [1-1] cost [1)3)
28
30
return to[m] In!
32 //
TEST interface())
fune deepCompareta interface(),
34
aC3ON,
Ma
35
ezt 1- nil
panioterr)
37
bJSON, ezr json.Narshal
39
if err i= mil {
(ante(err)
41
42 ver #2 interface()
Inter
44
e:r json. Uamarshal (aJSON, sa2)
46 panic (err)
47
48
jsonn Unmarshal (bJSON, sb2)
(bJSON,
49
ezr 1- nil :
(anio(err)
51
result Deflactt.Depssqulll (s2, b2)
53
if result
54
return Lrue
55
else
56
panio (frr.ttrint ("év In tv", a, b))
58
60
Tune m)in.
actual_l :- in) 2, 3), Ciitt4, 8, 21. Hiatil 5, 311,2,2
62
expected_i
63
doeptcmpare lastusl_,, expected_1)
65
actuol_2 := (() (2), 3, 4), (Sn 9, 31, 12 6, 411,2,2)
expected_2
67
deepCompare lactual_2, expected_2)
68
actual_3 min_cest (() inint [3 5), [int 10, 4), [3in: 7, 51),2,21
70
expected_3 := 16 deegCompare lactual_3, expected_3)
72
73
R.1.6 MBCSP: C#
71

---

Published as a conference paper at ICLR 2023
NBCSP/1:
gaing System;
using
8ysten.Co Cereen.oollee Collect lons. Gor
using System. Lingz
ssing
System. Text www.eeeet....ee... www.keerrepeeeeseeeee.....
nameapace Solution
public claas Program
10
11
/// <summary>
12
1// Write function to Sind the minimum cost path to reach (m, nl from (C, 01 for the
matrix
and position in,
13
///
Exomples:
15
1/1 >>> XinCoat [I1,, 2, 31, 14, 8, 21, C. 311,
16
1/1
17
1/1 XinCost [[[2, 3, d1, 15, 12, 411, 2,
18
1/1 532
1// XinCost 1113, 4, 16, 10, 41, 13, 5]1,
20
1/7 533 26
</summery>
22
public static int MinCost (List<tist<int>s cost, int int
23
//
COMPLTTION (model generated) 1/ rono: Krite your cada hare 0;
25
27
29
public static void Main (string) azge)
30
Comporologie comparelogic ComparoLagie():
32
var actuall MinCost tnow tistttists nt>> [new Mattintt> 11,2,3, Matkint> 14,8,21,
(1,5,31),2,2);
33
ver expectedl 8;
compareicgie.compan (actuall, expectedl);
35
if (1resultl.Areequall (threw ixception:"ssserttte failed");]
36
actual2 MinCost Metcklatt< Inew list<nnt> 12,3,4), listinnt> 15, 3,31, new new List<int> 4,),221);;
expested2 12; ver result2 wompareecgie.comg.... (actual2, expected2); (tremult2.AArEEqual) (throw new Exception ("sasertion failed");)
39
41
42
ver actuall MinCost fnew Lisr<Liet<int>>> Inew Listkint> (3,4,5), Listkint> 16, new Matkint> 23,,,
43
var expected3 16;
result3 comparelcgic. Compare (actual3, expected3) if (.result3. Arreqqual) (threw new Exception ("aapiitttasssttioonn
45
("assertion failed");)
47
48
R.1.7 MBPHP: PHP
77
MBPXP/1: PROXPT
You are expert PSP programmer, and here is your taak.
-Write function to find the minimun cost path reach (m, n) from (0, 0) for the given cost metrix cost and position
a) in cost [1.
pho minCost ([1), 2, 31, 14, 8, 21, 11, 5, 3.1, 2, 2)
php
minCost ([[2, 3, 41, 15, 9, 31, 12, 6, 4.1, 2, 2)
minCost [[[3, 4, 5), [6, 10, 41, 13, 3, 811, 2, 2)
10
11
-/
12
Funct Lon minCost ($cost, Sm, 5n)
COMPLETION (model generated)
14
Smin $cost [6m] [9n];
15
for 151
17
šn; si++) [
16
for ($) $1 1; 51 8m šn; $it-)
17
if (Scost [$1][8]] Sminl
18
Smin Scost [si)
19
21
return Smin:
72

---

Published as a conference paper at ICLR 2023
24 //
TEST
25
Tunct Ion compare (5x, Sy) [
26
return Sx 8y7
27
28
29
5x minCost (11,, 2, 31, 14, 8, 21. D., 5, 311, 2, 2);
30 (ccompare (9x, 8)
31
throw new Except on E"rrorr at 1th assert statement..
32
33
34 9x mincost (21 3, 41, [5, 9, 31, 12, 6, 411, 2, 2)
(ccompra (8x, 121)
36
throw new Except ion Errcor at 2th assect statement, ")
37
38
39 9x (L 4, 51, [6, 10, 4], 13, 7, 511, 2, 217
40
if (Icompare ($x, 16)) 1
41
throw new Exception ("Error at 3th assert statement. ")
R.1.8 MBRBP: RUBY
MDRD2/1:
PROMPT
You are an expart Ruby programmer, and here is your task.
Write a tunction sind the minimum cost path to reach (m, from 10, for the given
cost matrix cost (1 and position
n) in cost [].
izb> min_cost (C[1, 2, 31, [4, B, 21, [1, 5, 31:, 2,
7 # izb> min_cest (1 [2, 3, 41, [5, 9, 31, 12, 6, 411 2, 2)
# 1rb> mi min_cost (33, 4, 5], (6, 10, 4), [3, 3, 511, 2, 21
10
11
12 def min_cost (cost, my m) COMPLETION (model generated)
14
srite your code here.
15
cost [m] [n] 0:
16
return
17
if m -- and
0: return cost [m] [n]
18
19
if m == 0:
return
[n] min_cost (cost,
21
if n am 0:
22
return cast [m] [n] min_cost (cost, 1, n)
23
return cost [a] + minnmiicoost (coat, 1, n), min_cost (cost, 1)3
24 end
TEST
26 x min_cost ([11, 2, 3), (4, 8, 21, [1, 5, 3)1, 2, 2)
!-
28
raise StandardError, "Berer at test case 1*
29
30 min_cost((12, 3, 4J, [5, 9, 31, 12, 6, 411,
31 if x !!-12
32
raise Standardkrror, "Brrer
33 end
min_cost([13,, 51, [6, 4), 7, 511,
35 if x != 16
36
ralae StandardBeror, "Brror at test case
37
1.9 MBKP: KOTLIN
MBK3/1: PROMPT
/-0
You aze an expert Rotlin programmes, and here is your task.
Write function find minimun cost path reach (m, cost metrix cost [1 and position (m, nl in cost [1 [].
0) For given
minCost ([(1, 2, 3), [4, 8, 21, (1, 5, 3]1, 2, 2)
8
minCost ([[2, 3, 41, [5, 3. 31, 12, 411, 2, 2)
12
>>> minCost ([(3, 4, 5), [6, 13, 4), [3, 7, 51% 2, 2)
11
+
12 minCost (cost Taat <Inb>>, Int, n Int)
73

---

Published as a conference paper at ICLR 2023
COMPLETICN
generated)
14
var dp Azrayin im 11..f11 map Arrayin ())
15
db[6] [3] cost [0] [0]
16
For (1 in 0 until
17
foz C in until n)
18
if == 3 0)
19
dpl1 1J(3 + +1 cost
[ [1]
else
21
dp(1 1][3 11 cost [1]] (3) sp(1] 1), opt1 11
22
23
24
return dp(m)
26
//
28
Fun main())
var arg00 : List<List
mutablelistoE (L 2, 3), autabletlatorre, 8, 2) mutablelistof 5, 3)}
30
vor arg01 : Int 2
argu2
32
vai x0 : Int minCost (azg00, azgol, azg02)
34
throw Exception ion EEcception - Lest case 0 did not pass. x0 x31
35
36
37
var arg'0 : AistelisttIIntt>
mitahletiatof Inttalltttr (2, 3, 4), mutablet:stof 9, 3)
mutabletistor var srgil Int 2 var argi2 :
38
40
Int minCost (azg10, argli, azg12)
41
is (xl 1- 12)
42
throw Exception ("Excoption
: did pass. xl x1)
43
45
ver arg20 : ListcList<Int>>> murablelistof InttablelierOff 4, 5), autablelistOE 10, 4), ( 51)
46
Int
47
var arg22 : Int 2
Int minCost (arg20, arg2i, ang22)
49
if (x2 1- 161
Exception "Exception
case did pass. x2)
51
53
R.1.10 MBCPP: C++
7/
MBCPF/11
PROMPT
innolude colte/stdott.n> using namespace std;
Write Lunction Lo Lind the minimum cost path zeach (al, n) Erom 0) For the given
cost matrix cost () () and poaition [m, n1 in nost () []-
(vedtoreveetorrinntaall
14, 21, (1, 3)1,
6
mincest (ettrreeeetrrreeeeerisssssss (12, 4), 15, 31, (2, 411, 2, 21
11
12
12
minCost (vectorcoeettorritts>: 4, 5), 16, 10, 43, 13, T. 811, 2,
13 16
14
//
minCost (vectorsvectorriiat> cost, int m,
16
17 vectorcint nt> dptm
COMPLETION generated)
18
117
19
[b[0] 0;
20
for (int 1 1++1 dp]1][0] doli
21
1](0] cost[l [][0];
23
for (int : 1, j S++1 dp[0][]] dp 1] cost(C] [1 1];
25
26
For (int 1; mr 1111 :
for (int 17
j-+)
28
dplil (2) cest 1113 1] Moth.min 1)(5), dp(=) 1));
30
dp[m] [m];
74

---

Published as a conference paper at ICLR 2023
32
33
34 //
TEST
35
36
template <typename 1> bool compare &, 511 return h;
18
40
int main arge, chare argv[]] {
41
int XO minCost (eedtorseec.rrciabbttl. 2, 31, 14, 8, 2br 11, 5, 314, 2, 2). if (compare (x0, 8)1)
42
45
throw runtime_erree Exxcepttoon test case did not pas5."17
44
45
minCost (veetorscccorrcinss 41, (5, 9, 31, 12, 6, 4)1, 2, 233 if (compare (X1, 12111 { thzow suntime_eree ("Exception
47
48
Leat case did not pass."
49
50
1tt mincost weetorsveccorceinte0(
10, 4), 13, 513, 212 throw zuntime_ezzor ("Exception test case pass."1y
52
1f (! (compace (x2, 16))1 {
54
55
return
37
R.1.11 MBPLP: PERI.
MBPLP/1:
You are an expert Perl programmeE, and here is your task.
function find the minimum cost path to reach (m, from 10, the given cost mstrix cost () (I and position (m, n1 in cost []
>>>
min_cost ([(1, 2, 3), [4, 8, 21, [1, 5, 311, 2, 2)
8 >>>
min_coat ([ 3, 4J, 15, 9, 31, 12, 6, 4]1, 2, 21
#
>>>
min_cost ([C3, 4, 51, [6, 10, 41, 13, 7, 512, 2, 2)
10
sub min_cost
11
12
my ($cost, Snl
13
COMPLETION (model generated)
15
Smin 0;
16
my Spath 0;
17
$1 3;
18
my $3 = 9;
20
my $min_i 0;
my $min_j whila ($i 3m and 3j 6m)
22
(Scost [$1 [8 Smim)
24
Smin (c18L [$]]7
25
smin $1
Smin
27
($1
1)
29
51 += 1;
30
else {
31
Si
32
$j += 1;
34
35
Spath $min_
36
Spath
37
CANONICAL SOLJTION
38
Data::Compace:
39
my Sazg0u [[1, 2, 3), 14, 8, 21, [1, 5, 3:]
my $azg01
41
$argd2 2;
my
min_cest (Sarg00, sarg01, $arg02);
43
SvD 8;
44 anless (Compare ($x0, (vo))
45
die "Exception test did not pass.";
46
my $argl0 [[2, 41, 15, 31, (2, 4:1;
48
$argll 2:
my $arg12 2
75

---

Published as a conference paper at ICLR 2023
50
my can_cost (Sarg10, sargil, $argi2);
51 $v1 12;
52 unleas (Compare $v1))
53
die "Exception test did not pass.";
54
my Sarg20 [[3, 5), [6, 10, [3, 5]17
56 my $arg21 2;
my $azg22
58 6x2 min_cost (Sarg20, sarg21, $arg22);
39 my $v2 16;
60 unless (Compare (9x2, )+2)
61 dia "Exception test case 2 did not pass.";
R.1.12 MBSCP: SCALA
//
MBSCP/1: PROMPT
object extends App
/.-
You are an expert Scala programmer, here your task.
Rrite Iunotion to Cind the minimum coat path to reach nl Irom 10, Ioz the
given cost matrix cost[] and position (m, n) cost(]
(Liat IList 11,
List 8, 2), Liat
A
>>> mintost (List
3, 4), (is 9, 3), List 41), 2, 2)
12
10
>>> minCost (Mat ILlt 13, 4, 51, (e, 10, 4), 1ist 7, 5)1, 2,
11
16
12
def mintlost (cost List (Lits
Int) Int generated)
14
15
16
vall dp (r + dp(0,
17
18
foz (i <- until
19
for (j 0 until ml
01 continue
21
i3 (1 -- 01 dp dp 1) cp 1) cost (1, d1
cost (i, 31
alse (1
23
else dpti, 1 min 5, 1), dp 13) coat 11,
24
25
26
println(dpim, m)
28 //
TEST
30
azgu0
List [List [Int]] (List 2, 3), List 8, 2), L1s: 3)1
31 var azg01
Int
32
arg02
int
33 vor x0 Int minCost (org03, arg01, arg02)
35
assert 1x0 == v0,
"Exception test case did not pass. x0 x01
37
azgl0 List (Lis(List[nn)) [InL]] List (List (2, 3, 4), List 3, 3), L1s: 4)1
38
var argll
Int
arg12
40
var x1 Tnt mi m'nst (arg13, arg!1, 1, arg12)
12
42
assert ix1 == v1, "Exception -- test case did not pass. x1 x11
43
44
azg20
List (List [Int]] (List 4, 5), List 10, 41, List 51)
45
var arg21
int
46
arg22
47
var x2 Int minOost (argZu, arg21, arg22)
16
49
assert (x2 == v2, "Exception -- cest case did not pass. x2 x21
50
R.1.13 MBSWP: SWIFT
MBSNP/11 PROMPT
You are on expert Swift prograrmer, and here is your tack.
Write function find the minimun cost path
(m, cost metrix cost [1 and position (m, nl in cost [] minCost (L(), 2, 3), [4, 2], (1, 5, 3]], 2, 2)
(0, 0) for given
76

---

Published as a conference paper at ICLR 2023
>>>
minCost (((2, 3, 4), 19, 9, 31, (2, 6, 4]1, 2, 2)
12
>39 minCost ([C3, 5), [6, 10, 4), [3, 7, 512
10
16
w/
12
Func nCost (cost [[Tnt]] Int, Tot) -> Tnt
13
(madel generated)
14
1f == D 11 =e 0)
15
zeturn D5
16
17
if
11 mm 1) [0]
18
19
(cost [m] [n] -1) return coat [m] [n];
21
22
cost [m] [n] (mintoit Icost,
cost [n], minCost (cost, 2) cost [m]
});
return (n];
25
CANONICAL SOLUTION
27
a:g00
[[Int]] I11, 2, 31, 14, 8, 21, [1, 5, 3]]
28
var a=g01
Int - arg02 int =
30
vor x0 Int minCost (cest : arg00, m angtt, n arg02)
32
assert 1x0 -- v0, "Exception - test case did not pass. xo
34
acgl0 [[Int]] [[2, 3, 4), 15, 9, 31, 12, 6, 4]1
35 var argll int
36 arg12
37 var x1 Tnt m! noost (oost : arg 0, m argl1, n arg12)
39 aszert ixl =- vl, "Exseption - cest case did not pass. x1
41
azg20 [[Int]] [[3, 4, 5), 16, 10, 4h 13, 7, 511
42 var arg21 int
arg?2
44
var x2 Int minCost (cost : arg20, m azg21, axg22)
46
assert 1x2 == v2, "Exception cest case did not pass. x2
R.2 MULTI-LINGUAL HUMANEVAL
HumanEval contains 164 cases, most of which are compatible with our conversion framework. For some cases where the tests are not explicit, such using Python for loop iterate over many test cases, we expand them out explicitly to make it compatible with the conversion framework. For instance, the test statement below
range (2, B): assert cand1 date x+1) -- stx
expanded
dandidatell (2, *2*
assert candidete (3, 41 == *3*
assert candidste (4,
assert candidate (5, 61
*5*
assert candidate 71
assert candidate
*7*
There are some cases that we filtered out such as cases that involve a user defined function. total, keep 161 out of 164 cases. We format of multi-lingual HumanEval are similar to that of MBXP each language; therefore, we skip the display of examples in this section for brevity.
R.3 MULTI-LINGUAL MATHQA
By extending MathQA-pythhn datasets Austin et al. (2021) for other programming languages, we obtained MathQA-Java and MathQA-lavaSeript for the purpose of evaluating the ability of the models to reason and synthesize code from more complex text, under multiple languages. The original MathQA-pythho problem contains short text (which describes mathematical question),
77

---

Convert MathQA-Python problem into our canomical MBXP format (Section 3 Specif- ically, we construct a unified function signature, ie. def problem :, followed by doestring. which equivalent to the short text of the original MathQA-Python. Published as a conference paper at ICLR 2023
answer (usually real number) and canonical solution in Python. Based this, to build version in a different language, we perform following two transformation steps:
Additionally, a single test case can be generated based on the given answer, ie. assert problem ()
answer. Obtain prompts and test cases in another language for execution-based evaluation using our proposed rule-based conversion framework.
For the conversion framework outlined in Section we emphasize that handle floating point comparing numbers to be within handling is suitable for floating points and helps avoid potential false negatives. It is also compatible with all conversions in other datasets since it is handled within the abstract compare function in each target language.
1e instead of exact comparison. This
Below, we show converted examples after the first step (including Python prompts, the canonical solution and single test case) and its counterparts for Java and JavaScript generated from the second step.
Bathha-Pyshon
daf problem())
shopkeeper sold article offezing discount earned profit of what would have been the percentage of profit earned if no discount had been offered n0
5.0
n0 -5.0
n1 =3111
t. nl 100.0
100.0
11
t2 t0 100.0
12
13
answer t3 100.0
14
return answer
15
16
Import math
compare (x, y):
18
return math. fabs fabh. (x-) <le-8
19
candidate probl
20
asaert congare (candidate 38.0)
21
23
Matho8 Java
import java.io.";
25
import java.
26
import java.util.43
27
import java.math.
28
Problem
30
/+.
32
shopkesper sold article offering a d' scount of 5 and carned profit of 31.1 4 what would have been the percentage of profit earned if no discount had been offered 7
nc 5.0 31.1
33
*/
public static double problem() // the model output are inserted here.
35
37
38
39
class Main
40
public static bcolesn compare10bjeect obil, Object obj21 tebjl null obj2 muil)
42
retur: true;
(obg1
II ob12 mull):
44
return false;
45
elze
46
if [obj] instanceof Deuble 11 abšl instancect Floas) &6
47
(ab12 instancecf Sauble II obj2 instanceof Float)
49
1: (objl instanceaf Float] (
ob11 (1Float) ob11) (0
78

---

Published as a conference paper at ICLR 2023
52
(obj2 instanceaf Float)
ob12 (1Float) oo32) .dubbeetauue ();
54
55
return Kath.abs ((dcuble)obj- - (double)obj2) 1e-7;
57
elae
return obji caba2
99
62
//axecution-based test case
public static (Strinn)) args) throws Exceptiion double xo Problem. problem(1
64
teompare (xO, 38.0)1) throw new java.lang www.llnng.xcettoon ("Except lon teat case did not paas. x0)
66
67
68
70
72
73
Bath@A-JJvaSeee
14
75 y--
article offering discount what would have been the percentage of profit earned if no discount had been offered n0
shopkeeper
profit 31.
77
</
78
Sunct ion problem() { the model
inserted here.
80
82 // execution test case
const require (*lodash"
85
Sunction compere (objectl, object2)
itypeet objectl "numbar" typeof ohject2 == "number") (
87
return aba tobject1 object2) 1e-7
89
else(
return L.isEqual object2)
92
96
var problem()
(compare (x, (3 else { 'Drror essert stetement. Velue JeON. striniin
(x)
79


Reviews:
[{"review_id": "fqiLA5g_mJ", "reviewer": "Reviewer_rPgV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a framework for deriving multi-lingual data from an execution-based dataset (e.g., MBPP, HumanEval, MathQA), which entails applying rule-based transformations based on static analysis to the prompt and test cases. The key idea is that the canonical solution does not need to be translated since evaluation is done by running the test cases. Through this, they translate MBPP into 10+ languages. They train large decoder-only monolingual and multi-lingual models, of various sizes, using three languages, and they evaluate them in out-of-domain and in-domain settings. They highlight the benefits of the large multi-lingual models especially in the out-of-domain settings. They also consider few-shot learning, which results in fewer errors, and is most advantageous in out-of-domain evaluation. They further consider other evaluation tasks like zero-shot translation, code insertion, and summarization. Additionally, their framework also allows deriving synthetic canonical solutions for tasks which require it.\n\nContributions:\n- A framework for acquiring multi-lingual data from a monolingual dataset, without directly requiring translation of code solutions.\n- A large multilingual dataset which can be used for studying a variety of code-related end tasks, spanning multiple languages.\n- Empirical results and observations, spanning many languages, model sizes, settings, and tasks.\n", "summary_of_the_review": "Overall, the intuition and motivation behind this work are neat. The paper presents interesting empirical results and observations related to multilingual aspects of code generation, especially considering much of the focus in recent years has been on only a few languages like Python and Java. They also present a dataset spanning multiple languages, which could potentially be useful for the broader research community, although I have some concerns about whether their framework which effectively “translates” prompts and test cases in existing datasets might be propagating biases. Additionally, the technical novelty behind the framework seems to be somewhat limited. Nonetheless, I would say that the empirical novelty is still relatively significant.\n\n[After Author Response]\nI have read the response, and the authors have addressed most of my concerns. I have increased my score.", "strength_and_weaknesses": "Strengths:\n- Deriving multilingual code data without having to translate code solutions (especially considering how error-prone code translation models are) is a neat idea and useful.\n- Since it is challenging to get parallel data to study tasks like code translation, this framework which can give synthetic canonical solutions could be particularly useful for studying that tasks.\n- There are extensive empirical results, including supplementary analyses (e.g., best combos of source/target pairs for zero-shot translation) and even minor observations (e.g., sampling efficiency, validation losses) that can be beneficial for guiding future work.\n\nWeaknesses:\n- One concern I have is regarding whether potential biases/assumptions in the original MBPP dataset can transfer over to the MBXP datasets. These could be possible ambiguities (as the authors have mentioned  in Section 2.1) but also more fundamental biases/assumptions, such as language-specific ones. Could the formulation of the prompt and the types of tests that are needed drastically change based on the specific language, depending on for example, whether it is a language that requires explicit memory management?\n- While the synthetic canonical solutions may be useful for training during different tasks like translation, I find that using them for test to be slightly problematic because these are model-generated, and possible errors/biases in this output could propagate, and we don’t want to be evaluating using such references. However, these execution-based sets like MBPP are quite small, so it is not clear whether they offer enough training data to have a meaningful impact.\n- It is not clear whether training the monolingual and multilingual models in Section 3.1 was necessary. As described in Appendix M, CodeGen entails monolingual and multilingual models trained on a subset of the languages studied in this work, including Python Java and Javascript, which were used in this work. Based on the results shown in  Appendix M, it seems that many of those models are better performing than the ones trained in this work. Most (if not all) of the analyses could have been done with these.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "azaxuwsG_L", "reviewer": "Reviewer_DMjf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new benchmark MBXP for 10+ programming languages to facilitate the evaluation of execution-based code completion task. By the design of the conversion framework to translate prompts and test cases from the MBPP dataset, MBXP can evaluate code generation models in a multi-lingual manner. Furthermore, by the code generation model and some defined strategies, they can also provide the canonical samples in different programming languages and these data can also used for different code-related tasks such as source code summarization, code translation etc.", "summary_of_the_review": "This paper propose a framework to construct a new benchmark across different programming languages for different code-related tasks. Based on this bechmark, an entensive experiment is conducted. Although the technique novelty is limited, I appreciate the efforts in this paper for some interesting findings. ", "strength_and_weaknesses": "Strength:\n\n- The provided benchmark is valuable for AI4Code community. Currently, MBPP is a dataset designed for python, which is limited for evaluation, This paper proposes a new benchmark based on MBPP to enrich the evaluated programming languages in Python, Java, Go, etc. (13 in total) and this dataset is very valuable for the community.\n\n- The extensive evaluation on this benchmark is very appreciated. Besides the code completion task including  multi-lingual version and  mono-lingual version, this paper also evaluates on zero-shot translation, prompt robustness, code insertion and the summarization task to show the effectiveness of the constructed benchmark.\n\n- There are also some findings by the extensive evaluation and these findings can provide the readers with some insights about these large-scale pre-trained models.\n\nAlthough I appreciated the authors for the extensive evaluation, I also find some problems and hope to fix these issues in the next version.\n\n- The synthetic canonical solutions are generated by the models, so I do not know how diversity they have compared with the real semantic-equivalent data?\n\n- In the abstract section, the authors mention that they will release part of the benchmark, but the main contribution for this paper is this benchmark, why not release the entire benchmark for the following-up researchers? \n\n- I suggest authors provide more details about the quality check via reviewers part. How many reviewers do you have? How to evaluate the converted version? How to ensure the converted problem is correct? How long do you take to evaluate these converted problems?\n\n- In Figure 3, I cannot know which line belongs to which model.  It should add some clarifications.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q5V93UcQGM", "reviewer": "Reviewer_n1wH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a way to make an existing dataset (MBPP), an execution-based code completion benchmark, multilingual: while the original MBPP only contained python code, the new dataset (MBXP) contains a total of 10 programming languages. Problems in function completion datasets consists of *prompt*, *test statement*, and *canonical solution*. The proposed dataset creation is based on the insight that, for an execution-based benchmark, only translation of the prompt and test statement are necessary.\n\nThe paper further evaluates multiple monolingual and multilingual models on MBXP. This evaluation results in multiple interesting findings, for instance, models have a non-zero performance on languages they have not been trained on, what the authors attribute to code in multiple languages appearing in the same files. The authors further explore few-shot approaches and translation. Multilingual models mostly outperform monolingual ones.", "summary_of_the_review": "As the paper provides interesting insights and contributes a useful dataset, I believe it should be accepted.", "strength_and_weaknesses": "Strengths:\n- The paper is very clear and contains a coherent set of experiments.\n- The paper presents a new dataset, which will be useful for future work.\n\nWeaknesses:\n- The proposed dataset creation and the experimental setup aren't very creative -- but the results are interesting, so this isn't a big weakness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G8lHVsDIFI_", "reviewer": "Reviewer_3QeX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a new framework to convert prompt and test cases, for code generation models, to a new target programming language. The framework is used to generate an execution-based code completion benchmark for more than 10 programming languages.\nThis enables an easier evaluation of a \"multilingual\" model versus the standard monolingual model.\n\nThis work shows that multilingual models tend to be better than the monolingual model and show that code generation models have zero-shot capabilities.\n  ", "summary_of_the_review": "The paper is very interesting and the method will be useful. My only concern is that the size of the PDF file submitted is huge and that I couldn't check all of it thouroughly. Some sections of the appendix must be move to the main paper before acceptance.\nI would definitely recommend the authors to submit the entire paper to a journal.", "strength_and_weaknesses": "Strengths:\n- the paper is very well written and illustrated. For someone like me who is not an expert of this area, I find this paper relatively easy to follow.\n- the framework proposed is well-motivated and very simple\n- this work can easily be applied to extend many benchmarks to many languages.\n\n\nWeaknesses:\n- The appendix is huge. It contain tons of information that would be more useful in the main paper. The limits for instance should definitely be included before the conclusion.\n- most of  the details needed for reproducibility and fully understand the experiments are in the appendix. I couldn't check it thouroughly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%