Venue: NAACL 2025

# mHumanEval - A Multilingual Benchmark to Evaluate Large Language

Models for Code Generation

Nishat Raihan, Antonios Anastasopoulos, Marcos Zampieri

George Mason University

Fairfax, VA, USA

{mraihan2, antonis, mzampier}@gmu.edu

###### Abstract

Recent advancements in large language models (LLMs) have significantly enhanced code generation from natural language prompts. The HumanEvalBenchmark, developed by OpenAI, remains the most widely used code generation benchmark. However, this and other Code LLM benchmarks face critical limitations, particularly in task diversity, test coverage, and linguistic scope. Current evaluations primarily focus on English-to-Python conversion tasks with limited test cases, potentially overestimating model performance. While recent works have addressed test coverage and programming language (PL) diversity, code generation from low-resource language prompts remains largely unexplored. To address this gap, we introduce mHumanEval1, an extended benchmark supporting prompts in over 200 natural languages. We employ established machine translation methods to compile the benchmark, coupled with a quality assurance process. Furthermore, we provide expert human translations for 15 diverse natural languages (NLs). We conclude by analyzing the multilingual code generation capabilities of state-of-the-art (SOTA) Code LLMs, offering insights into the current landscape of cross-lingual code generation.

Footnote 1: github.com/mraihan-gnu/mHumanEval-Benchmark

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1: Long Papers_), pages 11432–11461

## 1 Introduction

LLMs have transformed software development with their ability to generate programming code from simple natural language instructions. LLMs are trained on extensive datasets that include diverse code samples, aiding programmers in code development and debugging. They also make programming more accessible to beginners. However, assessing the performance of these models across different coding tasks is still a major challenge. Comprehensive testing is essential to verify that these models are both effective and adaptable, rather than only performing well under specific conditions.

The most widely used benchmark for evaluating these models is OpenAI's HumanEvalChen et al. (2021), which includes a collection of 164 tasks generated by human experts. Each task includes an English prompt, a canonical solution provided by the authors, and three test cases. Although this benchmark is commonly used, it has significant limitations, such as limited test coverage and minimal support for non-English and non-Python prompts. While recent variations Peng et al. (2024); Cassano et al. (2023) of HumanEvaladdress some of these issues, most do not include prompts in NLs other than English and, in particular, in low-resource NLs. Consequently, current benchmarks fail to provide key insights into the multilingual capabilities of LLMs in the context of code generation.

Figure 1 demonstrates one such example. While the widely used GPT3.5Brown et al. (2020) model performs perfectly for the original prompt "Write a Python codesnippethat detects whether a year is a leapyear or not.", it fails when the same prompt is given in a low-resource language (_Nyanja_, in this case).

Most LLMs, primarily pre-trained on large English corpora like Common Crawl, perform poorly on

Figure 1: Code snippet generated by GPT3.5 when prompted to write a Python code to detect leap years in _Nyanja_ language. Some Python keywords are transformed into _Nyanja_ words, resulting in compilation issues.



multilingual tasks, further propagating inequalities in language technology access Blasi et al. (2022). However, proprietary models like GPT-4 Achiam et al. (2023) and Claude 3 Anthropic (2024), with undisclosed training data, show decent performance in multilingual scenarios. Peng et al. (2024) for instance show that GPT-4 excels in code generation even with mid-resource language prompts. The open-source community is also advancing with multilingual models like Aya Ustun et al. (2024) and LLaMA 3. However, insights into their code generation performance in a massively multilingual setting are lacking due to the absence of comprehensive benchmarks.

In this work, we introduce mHumanEval, a novel multilingual code generation benchmark including coding prompts in 204 NLs and expert human translations for 15 NLs. mHumanEval further includes canonical solutions in 25 PLs, including 4 new PLs that are not covered by any prior benchmarks. The primary contributions of this paper are as follows:

1. The creation of mHumanEval, the first massively multilingual benchmark for code generation.
2. A translation quality evaluation for each prompt.
3. A thorough evaluation of existing SOTA Code LLMs using mHumanEval.

The paper addresses two research questions (RQs):

* **RQ1:** How do the code generation capabilities of LLMs vary when prompts are provided in English, or other high-, mid-, and low-resource NLs?
* **RQ2:** How does the performance of multilingual LLMs compare to specialized, fine-tuned Code LLMs in code generation tasks on the mHumanEval dataset?

Finally, we also report _secondary findings_ related to the translation quality of machine translation (MT) methods on coding prompts.

## 2 Related Work

The most widely used benchmark dataset for evaluating Code LLMs is the adormentioned HumanEval Chen et al. (2021). Another key benchmark is DeepMind's MBPP Austin et al. (2021), which includes 974 tasks with 3 test cases each. Despite their popularity, these benchmarks have significant limitations, such as inadequate test case coverage, limited number of PLs, and small task sets that do not represent real-world scenarios. Other benchmarks, like CONCODE Iyer et al. (2018) Java, AxiBench Hao et al. (2022) Java, CSEPrompts Raihan et al. (2024) Python and CodeApex Fu et al. (2023) C++ focus on a single PL.

To broaden PL coverage, Cassano et al. (2023) combine both HumanEval and MBPP and add 17 more popular PLs besides Python, such as C++, Java, Ruby, and PHP. However, all prompts remain in English, with only 3 test cases per task. Similarly, the authors of BabelCode Orlanski et al. (2023) include 14 PLs and a more extensive test suite. To address test case coverage, Liu et al. (2024) introduce two datasets, HumanEval+ and MBPP+, with significantly more test cases per task, ensuring both node and edge coverage. Notably, Code LLM performance decreases with the additional test cases, highlighting the initial benchmarks' limitations. Nevertheless, these benchmarks also use English prompts exclusively.

Few studies explore non-English coding prompts and evaluate Code LLMs on them. The recent benchmark, HumanEval-XL Peng et al. (2024), extends coverage for both NLs and PLs. This benchmark includes coding prompts in 23 NLs and solutions in 12 PLs. The original prompts from HumanEval Chen et al. (2021) are translated into 23 different NLs using GPT-4 Achiam et al. (2023), with the quality of these translations assessed using a thresholded BERTScore Zhang et al. (2019). While HumanEval-XL explores multilingual prompts for code generation (Table 1), its 23 predominantly high-resource NLs limit insights into mid and low-resource NLs. The BERTScoreZhang et al. (2019) evaluation may be inadequate, with CometKiwi Rei et al. (2023) and X-Comet Guerreiro et al. (2024) offering more robust alternatives. Experimenting with SOTA Code LLMs like WizardCoder Luo et al. (2023) or multilingual models like Aya Ustun et al. (2024) could yield valuable insights. Also, they do not include any human translations.

We argue that NL coverage is more critical than PL coverage when compiling a code generation benchmark. While prompts and tests can be reused across PLs, different NLs require curating contextually and linguistically appropriate prompts. Thus, NL diversity introduces more complexity in benchmark creation than PL diversity. To bridge the 

gap, we present mHumanEval, offering comprehensive experiments with multilingual coding prompts across 204 NLs and 25 PLs--the most extensive coverage to date (see Appendix B for the full list) and the first one to include expert-human annotations (see Table 1). We describe mHumanEval in detail in this paper and we evaluate SOTA models on this dataset.

## 3 The mHumanEval Benchmark

The mHumanEval benchmark is curated based on prompts from the original HumanEvalChen et al. (2021) dataset. It includes a total of 33,456 prompts, significantly expanding from the original 164. The curation process can be divided into several key steps, as illustrated in Figure 2 and elaborated upon in the following subsections.

### Prompt Extraction

A typical prompt from the original dataset includes optional library imports, a function declaration, a docstring, and optional examples (as illustrated in Figure 3).

For translation, we only consider the docstrings (enclosed in triple quotes). These are manually extracted from all 164 prompts to ensure accuracy.

### Prompt Translation

Upon extracting the prompts, we move on to translating them into different languages. We use three different machine translation strategies - leveraging OpenAI's GPT4-omni through API, MetaAI's NLLB Costa-jussa et al. (2022), which is the SOTA model for multiple NLs, and Google translate via API.

Our target languages are all 204 languages from the Flores 200 dataset-Costa-jussa et al. (2022). While we employ GPT4-omni and NLLB for all the target languages, it is important to note that we use only Google Translator for the 108 languages it supports (available through the API). For each extracted prompt, we employ the three translation systems for each target language, generating 5 candidate translation prompts (3 for GPT4o, due to budget considerations). We then evaluate the quality of the translation and keep the best one (see Figure 2). The pseudocode is in Appendix C.

### Evaluating Prompt Quality

We evaluate translation quality using BERTScoreZhang et al. (2019), which focuses on similarity based on contextual embeddings, and CometKiwiRei et al. (2023), which is trained on human judgments of MT quality and incorporates linguistic features. While BERTScore uses BERT embeddings to measure candidate-reference translation similarity (Appendix D), CometKwi evaluates translations reference-free, using human judgments and combining linguistic features with contextual embeddings (Appendix E). Using both ensures holistic evaluation, covering lexical similarity and human-assessed quality aspects.

As illustrated in Figure 2, we generate 13 candidate translations for each prompt. We also perform round-trip translations back to the original language (eng_Latin) to calculate the BERTScore. While CometKiwi is calculated as a reference-free metric. Both metrics generate scores in the [0,1] range. By computing the mean of the two metrics for each prompt, we select the candidate with the highest score. The mean scores for each language and system are provided in Appendices I, J and K. It is worth noting that the CometKiwi metrics are not available for all languages, as it relies on XLM-R models Conneau et al. (2019); Goyal et al. (2021) supporting 100 languages Rei et al. (2023). For the remaining 104 Flores 200 languages, we use round-trip translations to calculate BERTScore, similar to HumanEval -XLPeng et al. (2024).

### Categorization based on Language Classes

To better understand the performance of models on languages considered to be low- or high-resourced, we group the languages in mHumanEval following the methodology of Joshi et al. (2020), who identify six classes of languages based on digital re

\begin{table}
\begin{tabular}{l|c c c c|c|c} \hline \hline  & \multicolumn{7}{c}{Benchmarks} \\  & HumanEval & MBPP & Label Code & MultiPL-E & HumanEval-XL & mHumanEval \\ \hline NL-Covg (MT) & 1 (eng) & 1 (eng) & 1 (eng) & 1 (eng) & 23 & **204** \\ NL-Covg (Human) & ✗ & ✗ & ✗ & ✗ & ✗ & **15** \\ \hline PL-Covg & 1 (py) & 1 (py) & 14 & 18 & 12 & **25** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparing popular benchmarks in terms of NL and PL coverage.

 

source availability. These classes range from 0 to 5, with higher numbers indicating greater resource availability. Joshi et al. classify a total of 2,485 languages, of which mHumanEval includes 204, including 15 with expert translations, as detailed in Table 2.

We present the class-wise evaluation scores for the selected prompts in mHumanEval in Figure 4. The language-specific scores are provided in Appendices I, J, and K. Generally, the quality of the translation decreases as we address languages with fewer resources. However, by implementing Algorithm 1 and selecting from 13 candidate translations, the chosen candidates demonstrate improved quality compared to the model-specific results (see Appendices L and F). The final prompts in mHumanEval exhibit significantly better quality.

### PL coverage

As noted in Section 2, most benchmarks in this subdomain are limited to Python, including HumanEval and MBPP. While recent benchmarks such as MultiPL-E and HumanEval-XL offer broader coverage, they still omit several widely used programming languages. With mHumanEval, we compile a comprehensive set of programming languages covered by existing multi-PL coding benchmarks and extend this set by incorporating four additional languages that have not been previously included: MATLAB, Visual Basic, Fortran, and COBOL (as shown in Table 8).

We provide canonical solutions for the newly included four languages in the same format as HumanEval . These solutions are handwritten by human experts and successfully pass all test cases.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Class** & **Resource** & **Total** & **mHumanEval** & **Expert** \\ \hline
5 & High & 7 & 7 & 6 \\
4 & Mid to High & 18 & 18 & 4 \\
3 & Mid & 28 & 27 & 1 \\
2 & Low to Mid & 19 & 16 & 2 \\
1 & Low & 222 & 98 & 1 \\
0 & Rare & 2191 & 38 & 1 \\ \hline ALL & – & 2485 & 204 & 15 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Class distribution of natural languages based on resource availability. **Expert** denotes human translations done by expert programmers.

Figure 3: A sample prompt instance from the original HumanEval benchmark.

Figure 2: The workflow to generate prompts in a target language from the original HumanEval. Original prompts are first extracted manually. Then 3 Machine Translation models (GPT4o, NLLB, Google Translate) generate 13 candidates as well as roundtrip translations. Next, we evaluate each candidate’s quality using _BERTScore_ using RoundTrip translations and _CometKiwi_ as a reference-free metric (if the language is supported). We then select the best candidate for each original prompt and compile the new benchmark for the target language.

 



### mHumanEval Subsets

We have a total of 33,456 prompts in mHumanEval spanning 204 NLs. Each prompt additionally supports 24 PLs, bringing the total number of prompts to 836,400. The entire dataset is publicly available on GitHub.

We also provide multiple subsets of the dataset for quick usability and interesting ablation studies (Table 3). Separate subsets are available for each NL and PL, in all possible combinations. Additionally, we create several variants for testing purposes- mHumanEval-T50@: a subset consisting of the 500 highest-quality prompts based on BERTScore and CometKiwi, mHumanEval-R50@: a randomly selected subset of 500 prompts, and mHumanEval-B50@: a subset of the 500 lowest-quality prompts. Note that these prompts are drawn from the curated mHumanEval, which compiles the best prompts from 13 candidates each. Finally, we produce mHumanEval-mini which is a subset containing 204 prompts, with each prompt in a different language, where we select one prompt per language.

### mHumanEval - Expert

The mHumanEval-Expert benchmark encompasses human translations across 15 languages, representing all six language classes (Table 4). Native speakers with computer science and engineering backgrounds perform these translations, ensuring precise interpretation of programming concepts and terminology. The curation process unfolds in three stages: (1) selection of 15 natural languages based on native speaker availability, ensuring representation from each language class; (2) translation by native speakers; and (3) quality assessment by expert programmers to verify the integrity of the coding prompts. Figure 5 illustrates the whole curation process.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Language** & **Class** \\ \hline English & 5 \\ Spanish & 5 \\ French & 5 \\ Japanese & 5 \\ Arabic & 5 \\ Chinese & 5 \\ Portuguese & 4 \\ Italian & 4 \\ Korean & 4 \\ Hindi & 4 \\ Bangla & 3 \\ Swahili & 2 \\ Zulu & 2 \\ Telugu & 1 \\ Sinhala & 0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: NLs along with their classes in mHumanEval-Expert.

Figure 4: Evaluating the translated prompt qualities, chosen in mHumanEval. Our method results in better quality prompts compared to the model-specific translations (as depicted in Appendix F).



A comparative analysis between human translations and mHumanEval's machine-translated prompts yields comparable evaluation metrics, with BERTScore variations of \(\pm\)0.02 and CometKiwi variations of \(\pm\)0.03 across the selected languages. Interestingly, annotators report no significant terminology concerns when reviewing machine translations. Further examination of the original HumanEval prompts reveals that the docstrings--the primary translated content--predominantly comprise general task descriptions, minimizing the use of specialized coding terminology. This observation emphasizes the negligible discrepancies between human and machine translations in this context.

We conclude that human and machine translations of programming prompts across 15 languages show similar quality, with minimal differences in evaluation metrics. This similarity is attributed to the general nature of the content, which contains limited specialized coding terminology.

## 4 Experiments

Experimental SetupFor machine translation, GPT-4o and Google Translate are accessed via API keys, eliminating the need for GPU hours. We use recommended hyperparameters without tuning, setting max_tokens to 1000 and temperature to 0.7. The NLLB model runs on a single NVIDIA A100 GPU with 40 GB of memory.

In code generation, proprietary models are accessed through API keys, negating the necessity for GPU hours. We adhere to recommended hyperparameters, setting max_tokens to 1000 and temperature to 0.7. WizardCoder and Aya utilize full precision (FP32) models on four NVIDIA A100 GPUs, each with 40 GB memory, following authors' recommended hyperparameters without additional tuning. MagiCoder, LLaMA 3, and Phin-3-mini employ full precision (FP32) models on a single NVIDIA A100 GPU with 40 GB of memory, using the authors' recommended hyperparameters.

Model SelectionWe experiment with mHumanEval using six models (Table 5), including both proprietary and open-source SOTA models for code generation. We use a mix of general-purpose and finetuned models to gather broader insights.

PromptingWe use the proprietary models through their APIs. Our experiments include all 33,456 prompts from mHumanEval, with 164 prompts for each language. We follow the standard prompt templates for each LLM. These templates are shown in Appendix G.

Code ExecutionFollowing code generation, we move to execution. The six models produce well-structured code blocks, requiring minimal cleaning. We use simple RegEx commands to extract these blocks, and evaluate them locally in batches using Python's subprocess2 library, focusing exclusively on the Pass@1 metric.

Footnote 2: docs.python.org/3/library/subprocess.html

ResultsFor each language, we present the Pass@1 scores as percentages, categorizing them by

Figure 5: Curating mHumanEval-Expert via native human translation followed by expert programmer evaluation.

 

the six language classes as discussed in Section 3.4. As illustrated in Figure 6, Claude3.5 and GPT4o exhibit the most consistent performance, maintaining strong results even with coding prompts in low-resource languages. In contrast, GPT3.5 and DeepSeek experience a significant decline in performance for low-resource classes. Although Aya shows the weakest results for higher resource classes, it maintains relative consistency, even in extremely low-resource languages. On the other hand, WizardCoder achieves excellent results in English and reasonable performance for Class 5, but its performance deteriorates significantly in other languages. The model and language-specific detailed results are presented in Appendix M.

Other PLsWe extend our evaluation to four additional subsets of mHumanEval: mHumanEvalC++, mHumanEvalJAVA, mHumanEval-JavaScript, and mHumanEvalRuby. The average Pass@1 scores across all 204 NLs for the 5 PLs are shown in Table 6.

We observe that GPT-4o and DeepSeek-Coder achieve strong results in Classes 4 and 5, with scores consistently exceeding 0.85 in Python, Java, and C++. Python shows top performance, with scores reaching above 0.88 in Class 5. For lower classes (0-2), models like GPT-3.5, WizardCoder, and Aya underperform, often scoring below 0.70, particularly in JavaScript and Ruby, where scores frequently drop under 0.65. Even in higher classes, JavaScript and Ruby show challenges, with Class 4 scores for most models not exceeding 0.75. This highlights the models' limitations in handling non-Python languages, particularly for lower classes and specific scripting languages. While every model's best scores are generated with English-Python pair, DeepSeek-Coder is the only exception with Chinese-Python.

A detailed analysis and discussion is provided in Appendix H.

## 5 Insights and Analysis

Upon curating the mHumanEval benchmark and completing the model evaluations, we now present some key analyses and gained insights based on the obtained results.

\begin{table}
\begin{tabular}{l|c c c c c} \hline  & **Python** & **Java** & **C++** & **JavaScript** & **Ruby** \\ \hline GPT4o & 0.738 & 0.650 & 0.652 & 0.477 & 0.480 \\ GPT3.5 & 0.360 & 0.270 & 0.270 & 0.099 & 0.103 \\ Claudio3.5 & 0.739 & **0.651** & **0.659** & **0.633** & **0.477** \\ DeepSeek-Coder & 0.229 & 0.139 & 0.136 & 0.000 & 0.000 \\ WizardCoder & 0.098 & 0.009 & 0.007 & 0.000 & 0.000 \\ Aya & 0.445 & 0.355 & 0.356 & 0.186 & 0.183 \\ \hline \end{tabular}
\end{table}
Table 6: Mean performance of models across programming languages.

Figure 6: Comparing model performances (% in Pass@1) for the six models on mHumanEval–Python.

 



### LLMs' Performance Analysis

We observe significant performance discrepancies among the models, as illustrated by Figure 6. While closed-source models perform better, their reliance on proprietary pretraining data complicates definitive conclusions. As suggested by the Chinchilla scaling hypothesis Hoffmann et al. (2022), their superior performance may result from a larger parameter count and extensive training tokens, possibly including diverse and rare languages.

Aya, fine-tuned for multiple natural languages but not specifically for code generation, has the lowest Pass@1 score in English. However, low variability across language classes indicates that multilingual pretraining and fine-tuning enhances code generation across different NLs.

WizardCoder's poor performance in non-English languages is due to its fine-tuning on StarCoder Li et al. (2023), which is primarily pretrained on code and documentation with minimal non-English content. In contrast, DeepSeek performs well for midresource languages but struggles with low-resource ones. These results suggest that effective multilingual code generation requires multilingual pretraining and/or finetuning datasets.

### Performance based on Language Classes

While there are significant discrepancies among the models' performances, a key trend observed is a somewhat consistent performance decline as we move from high-resource to low-resource languages. This decline is not as pronounced for Claude and GPT-4o. However, it is quite substantial for others and exceptionally steep for WizardCoder and DeepSeek-Coder.

### Ablation Study

We present results from a limited ablation study conducted on various subsets of mHumanEval as detailed in Table 3. This study incorporates two additional models including MetaAI's LLaMA 3 (70B), and MistralAI's code-finetuned CodeStral (22B) model.

As indicated by the results in Table 7, mHumanEval-mini serves as an effective preliminary test for evaluating a model's proficiency in code generation following multilingual prompts. Models fine-tuned on code but lacking multilingual exposure perform poorly, whereas base models with some multilingual exposure perform better. The three subsets of mHumanEval are curated by prompt quality: mHumanEval-T500 includes prompts from language class 5, mHumanEval-B500 from classes 0 or 1, and mHumanEval-R500 is randomly selected. These results align with our findings in Sections 5.1 and 5.2.

### Error Analysis

In our analysis of errors, we observe several unique issues. Notably, the models rarely fail to generate any code. Specifically, GPT4o and GPT3.5 generate code with almost no compilation issues. However, a significant number of errors arise from misunderstandings of the problem, resulting in code that addresses incorrect tasks. We discuss such unique cases in this section.

Task MisunderstandingGPT4o generates a code for finding significant digits, when it is prompted to generate a code for detecting prime numbers in Zulu. This is due to the reason of the translation of 'prime number' in Zulu becomes 'Inombolo Eyninloko' which means 'primary numbers'. The model interprets it as 'significant number' instead.

Issues like this persist, even sometimes when prompted multiple times.

Multilingual Keyword IssuesSome code snippets include keywords and identifiers in the target language as well, resulting in compilation errors. As shown in Figure 8, when prompted in Rundi, to find the largest number in a list, Aya generated a piece of code, where keywords like for and return turned into kuri and subiza.

These kinds of errors are also persistent but the outputs differ with similar prompts. Still generate the same issue again and again.

\begin{table}
\begin{tabular}{c|r r r r r r r r} \hline \hline  & **GFt4o** & **GPT3.5** & **Aya** & **WizardCoder** & **Claude3.5** & **DeepSeek-Coder** & **LLaMA 3** & **CodeStral** \\ \hline mhumanEval-mini & 72 & .44 & .47 & .12 & .61 & .57 & .35 & .15 \\ mHumanEval-T500 & 37 & .76 & .6 & .63 & .86 & .73 & .56 & .36 \\ nhumanEval-R500 & .78 & .53 & .47 & .16 & .59 & .63 & .28 & .17 \\ nhumanEval-15500 & 48 & .21 & .42 & .00 & .31 & .22 & .11 & .10 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Comparison of different LLMs’ based on % in Pass@1 metric on multiple subsets of mHumanEval.





## 6 Conclusion

This study introduces mHumanEval, a comprehensive multilingual code generation benchmark for assessing LLMs across 204 languages. We curated high-quality prompts for each language and evaluated various models. Our analyses, including ablation studies, provided insights into LLMs' multilingual code-generation capabilities, addressing the RQs posed in Section 1:

**RQ1: How do the code generation capabilities of LLMs vary when prompts are provided in English, or other high-, mid-, and low-resource NLs?**LLMs generally demonstrate optimal performance when prompted in English. For prompts in other languages, performance varies based on the language's resource level. High-resource languages tend to yield superior results compared to mid- and low-resource languages. The extent of performance variation is contingent upon the specific language of the prompt and the model's prior exposure and training in that language. This variation is likely influenced by the model's training data and the relative abundance of resources available for each language.

**RQ2: How does the performance of multilingual LLMs compare to specialized, fine-tuned Code LLMs in code generation tasks on the mHumanEval dataset?**While code-finetuned language models excel at generating code from English prompts, multilingual models demonstrate strong proficiency across various NLs. Notably, even without specific code fine-tuning for different NLs, they achieve decent results in code generation. This phenomenon suggests that multilingual models can generalize coding capabilities across NLs, leveraging their understanding of multiple NLs to support diverse linguistic contexts in programming. While we draw some insightful conclusions from curating and evaluating mHumanEval, to facilitate further research, we are making it publicly available. We plan to expand coverage to more NLs and PLs in future updates. Despite the high cost of human translation, we included human annotations for 15 NLs, including some low-resource and rare ones. Currently, our dataset includes 164 prompts per language, following the HumanEval benchmark, with plans to increase this number. We will also explore strategies to enhance low-resource language performance, such as transfer learning and diverse training datasets. Comparative studies between general-purpose multilingual LLMs and specialized code LLMs will help optimize multilingual code generation.

Figure 8: When prompted in Rundi, Aya generated Python code to find the largest number in a list, containing multiple keywords in Rundi, resulting in Compilation Error.

Figure 7: GP4o generated Python Code to find significant digits, when prompted to write code to detect prime numbers in Zulu.





### Limitations

We conducted primary evaluations on six LLMs, focusing on key performance metrics. Given the benchmark's extensive 33,456 prompts, the evaluation process is exceedingly costly. This cost is the primary reason why we adopted Pass@1 as our evaluation metric, rather than more resource-intensive metrics like Pass@10 or Pass@100. However, to ensure a thorough analysis, we incorporated additional models in our ablation study. In our next iteration, we plan to comprehensively evaluate all models across the entire benchmark. This future work aims to enhance the benchmark's robustness and provide deeper insights into the performance of various LLMs in multilingual code generation.

## Ethical Considerations

The benchmark introduced in this paper, which focuses on analyzing code generation using large language models (LLMs), strictly adheres to the ACL Ethics Policy. Each prompt in mHumanEval was tested multiple times by different models, and none produced any malicious code. Although there can occasionally be garbage code snippets or similar issues, none have posed any threats to the system.

To ensure safety and reliability, we recommend executing code generated using prompts from mMumanEval in a contained virtual environment. This precaution helps prevent potential issues related to infinite execution loops and memory management. Running code in a safe environment can also stop problems like crashing the system or using too much memory. We believe and hope that researchers and practitioners can maintain a secure and controlled testing environment while utilizing mMumanEval. This approach ensures that users can confidently explore and innovate without risking system integrity.

## Acknowledgments

We would like to thank the human annotators and experts for their valuable time and effort; also George Mason's Office of Research Computing (ORC) for providing the computing resources.

Antonios Anastasopoulos is additionally supported by the National Science Foundation under award IIS-2327143 and benefited from resources provided through the Microsoft Accelerate Foundation Models Research (AFMR) grant program.

## References

* Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, et al. 2023. Gpt-4 technical report.
* Anthropic [2024] Anthropic. 2024. Claude 3: A next-generation ai assistant. [https://www.anthropic.com/news/claude-3-family](https://www.anthropic.com/news/claude-3-family). Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. _arXiv preprint arXiv:2108.07732_. Blasi et al. [2022] Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world's languages. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics. Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_. Cassano et al. [2023] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckeney, et al. 2023. Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. _IEEE Transactions on Software Engineering_. Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. 2021. Evaluating large language models trained on code. _arXiv preprint arXiv:2107.03374_. Conneau et al. [2019] Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. _arXiv preprint arXiv:1911.02116_. Costa-jussa et al. [2022] Marta R Costa-jussa, James Cross, Onur Celebi, et al. 2022. No language left behind: Scaling human-centered machine translation. _arXiv preprint arXiv:2207.04672_. Dai et al. [2024] Damai Dai, Chengqi Deng, Chenggang Zhao, et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. _arXiv preprint arXiv:2401.06066_. Fu et al. [2023] Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, et al. 2023. Codeapex: A bilingual programming evaluation benchmark for large language models. _arXiv preprint arXiv:2309.01940_. Goyal et al. [2021] Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021. Larger-scale transformers for multilingual masked language modeling

In _Proceedings of the 6th Workshop on Representation Learning for NLP RepL4NLP-2021)_.
* Guerreiro et al. (2024) Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andre FT Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. _Transactions of the Association for Computational Linguistics_.
* Hao et al. (2022) Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, et al. 2022. Aixbench: A code generation benchmark dataset. _arXiv preprint arXiv:2206.13179_.
* Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, et al. 2022. Training compute-optimal large language models. _arXiv preprint arXiv:2203.15556_.
* Iyer et al. (2018) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
* Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
* Li et al. (2023) R Li, LB Allal, Y Zi, N Muennighoff, D Kocctekov, C Mou, M Marone, C Akiki, J Li, J Chim, et al. 2023. Starcoder: May the source be with you! _Transactions on machine learning research_.
* Liu et al. (2024) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. _Advances in Neural Information Processing Systems_.
* Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, et al. 2023. Wizard decoder: Empowering code large language models with evol-instruct. In _The Twelfth International Conference on Learning Representations_.
* Orlanski et al. (2023) Gabriel Orlanski, Kefan Xiao, Xavier Garcia, et al. 2023. Measuring the impact of programming language distribution. In _International Conference on Machine Learning_. PMLR.
* Peng et al. (2024) Qiwei Peng, Yekun Chai, and Xuhong Li. 2024. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRECOLING 2024)_.
* Raihan et al. (2024) Nishat Rainhan, Dhiman Goswami, Sadiya Sayara Chowdhury Puspo, Christian Newman, Tharindu Ranasinghe, and Marcos Zampieri. 2024. Cseprompts: A benchmark of introductory computer science prompts. _arXiv preprint arXiv:2404.02540_.
* Rei et al. (2023) Riccardo Rei, Nuno M Guerreiro, Daan van Stigt, Marcos Trevso, et al. 2023. Scaling up cometkiwi: Unbbelist 2023 submission for the quality estimation shared task. In _Proceedings of the Eighth Conference on Machine Translation_.
* Ustun et al. (2024) Ahmet Ustun, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Obemileke Onilude, et al. 2024. Aya model: An instruction finetuned openaccess multilingual language model. _arXiv preprint arXiv:2402.07827_.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.

 

Annotator Details

As mentioned in Section 3.7, mHumanEval-Expert utilizes native-speaking volunteer translators for 15 NLs. Each translator was assigned 164 prompts, with no monetary compensation involved. The experts, also native speakers, possess backgrounds in Computer Science and/or Information Technology, complemented by substantial coding experience. Both translators and experts were carefully selected through a rigorous process, ensuring a diverse demographic representation. This methodological approach enhances the dataset's linguistic diversity and technical robustness across various cultural contexts.

## Appendix B List of NLs and PLs in mHumanEval

mHumanEval supports 204 NLs and 25 PLs. The Expert subset contains human annotation for 15 NLs.

### List of PLs

Comparing PL support provided by most widely used existing benchmarks -

### List of NLs: mHumanEval

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline  & \multicolumn{8}{c}{Benchmarks} \\  & HumanEval & HBPP & Babel Code & MultiPl_E & HumanEval-XL & mHumanEval \\ \hline Python & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ \\ Bash & ✗ & ✗ & ✗ & ✓ & ✗ & ✓ \\ C++ & ✗ & ✗ & ✓ & ✓ & ✗ & ✓ \\ C# & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ D & ✗ & ✗ & ✗ & ✓ & ✗ & ✓ \\ Go & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ Haskell & ✗ & ✗ & ✓ & ✗ & ✗ & ✓ \\ Java & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ JavaScript & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ JavaScript & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ Julia & ✗ & ✗ & ✗ & ✓ & ✗ & ✓ \\ Kottin & ✗ & ✗ & ✓ & ✗ & ✓ & ✓ \\ Lua & ✗ & ✗ & ✗ & ✓ & ✗ & ✓ \\ Perl & ✗ & ✗ & ✗ & ✓ & ✓ & ✓ \\ PHP & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ R & ✗ & ✗ & ✗ & ✓ & ✗ & ✓ \\ Racket & ✗ & ✗ & ✗ & ✓ & ✗ & ✓ \\ Ruby & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ Rust & ✗ & ✗ & ✓ & ✓ & ✗ & ✓ \\ Scala & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ Swift & ✗ & ✗ & ✓ & ✓ & ✓ & ✓ \\ TypeScript & ✗ & ✓ & ✓ & ✓ & ✓ & ✓ \\ MATLAB & ✗ & ✗ & ✗ & ✗ & ✓ \\ Visual Basic & ✗ & ✗ & ✗ & ✗ & ✗ & ✓ \\ Fortran & ✗ & ✗ & ✗ & ✗ & ✓ \\ COBOL & ✗ & ✗ & ✗ & ✗ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 8: Comparing popular benchmarks in terms of NL and PL coverage.

 



\begin{table}
\begin{tabular}{l l|c c|l l|l l} \hline
**Language** & **Class** & **Language** & **Class** & **Language** & **Class** & **Language** & **Class** \\ \hline arb\_Arab & 5 & zsm\_Latin & 3 & gla\_Latn & 1 & tät\_Cyrl & 1 \\ deu\_Latn & 5 & amh\_Ethi & 2 & guj\_Guir & 1 & tel\_Telu & 1 \\ eng\_Latin & 5 & gle\_Latin & 2 & hye\_Armn & 1 & tgk\_Cyrl & 1 \\ fra\_Latin & 5 & hau\_Latin & 2 & ibo\_Latin & 1 & tpi\_Latin & 1 \\ jpn\_Jpan & 5 & isi\_Latin & 2 & iio\_Latin & 1 & tso\_Latin & 1 \\ spa\_Latin & 5 & lao\_Laoo & 2 & java\_Latin & 1 & tuk\_Latin & 1 \\ zho\_Hans & 5 & mar\_Deva & 2 & kab\_Latin & 1 & tum\_Latin & 1 \\ cat\_Latin & 4 & mlt\_Latin & 2 & kan\_Knda & 1 & twi\_Latin & 1 \\ces\_Latin & 4 & pan\_Guru & 2 & kaš\_Arab & 1 & uig\_Arab & 1 \\ eus\_Latin & 4 & san\_Deva & 2 & kaš\_Deva & 1 & vec\_Latin & 1 \\ fin\_Latin & 4 & swh\_Latin & 2 & khk\_Cyrl & 1 & war\_Latin & 1 \\ him\_Deva & 4 & ir\_Ethi & 2 & khm\_Khmr & 1 & ydd\_Hebr & 1 \\ hrv\_Latin & 4 & tsn\_Latin & 2 & ikić\_Latin & 1 & zho\_Hant & 1 \\ hun\_Latin & 4 & wol\_Latin & 2 & kin\_Latin & 1 & awa\_Deva & 0 \\ ita\_Latin & 4 & xho\_Latin & 2 & kir\_Cyrl & 1 & bam\_Latin & 0 \\ kor\_Hang & 4 & vor\_Latin & 2 & kmr\_Latin & 1 & ban\_Latin & 0 \\ nld\_Latin & 4 & zul\_Latin & 2 & lij\_Latin & 1 & bem\_Latin & 0 \\ pes\_Arab & 4 & ace\_Arab & 1 & lim\_Latin & 1 & cjk\_Latin & 0 \\ pol\_Latin & 4 & ace\_Latin & 1 & lin\_Latin & 1 & dyu\_Latin & 0 \\ por\_Latin & 4 & acm\_Arab & 1 & lmo\_Latin & 1 & fon\_Latin & 0 \\ rus\_Cyrl & 4 & acq\_Arab & 1 & ltg\_Latin & 1 & fuv\_Latin & 0 \\ srp\_Cyrl & 4 & aeb\_Arab & 1 & ltz\_Latin & 1 & gm\_Latin & 0 \\ swe\_Latin & 4 & aip\_Arab & 1 & lug\_Latin & 1 & hat\_Latin & 0 \\ tur\_Latin & 4 & aka\_Latin & 1 & mai\_Deva & 1 & hn\_Deva & 0 \\ vie\_Latin & 4 & als\_Latin & 1 & mal\_Mlym & 1 & kac\_Latin & 0 \\ afr\_Latin & 3 & apc\_Arab & 1 & min\_Arab & 1 & kam\_Latin & 0 \\ arb\_Latin & 3 & ars\_Arab & 1 & min\_Latin & 1 & kbp\_Latin & 0 \\ arz\_Arab & 3 & ary\_Arab & 1 & mkd\_Cyrl & 1 & kea\_Latin & 0 \\ ben\_Beng & 3 & asm\_Beng & 1 & mri\_Latin & 1 & kmb\_Latin & 0 \\ bos\_Latin & 3 & ast\_Latin & 1 & mya\_Mymr & 1 & kne\_Arab & 0 \\ bul\_Cyrl & 3 & agr\_Latin & 1 & nno\_Latin & 1 & knc\_Latin & 0 \\ ceb\_Latin & 3 & azb\_Arab & 1 & nob\_Latin & 1 & kon\_Latin & 0 \\ dan\_Latin & 3 & azj\_Latin & 1 & npi\_Deva & 1 & lua\_Latin & 0 \\ ell\_Grek & 3 & bak\_Cyrl & 1 & oci\_Latin & 1 & luo\_Latin & 0 \\ est\_Latin & 3 & bel\_Cyrl & 1 & ory\_Orya & 1 & lus\_Latin & 0 \\ glg\_Latin & 3 & bho\_Deva & 1 & pag\_Latin & 1 & mag\_Deva & 0 \\ heb\_Hebr & 3 & bijn\_Arab & 1 & pap\_Latin & 1 & mini\_Beng & 0 \\ ind\_Latin & 3 & bijn\_Latin & 1 & pbt\_Arab & 1 & mos\_Latin & 0 \\kat\_Geor & 3 & bod\_Tibt & 1 & plt\_Latin & 1 & nso\_Latin & 0 \\ kaz\_Cyrl & 3 & bug\_Latin & 1 & quy\_Latin & 1 & nus\_Latin & 0 \\ lit\_Latin & 3 & ckb\_Arab & 1 & sag\_Latin & 1 & nya\_Latin & 0 \\ lvs\_Latin & 3 & crh\_Latin & 1 & sat\_Olck & 1 & prs\_Arab & 0 \\ non\_Latin & 3 & cyp\_Latin & 1 & scn\_Latin & 1 & run\_Latin & 0 \\ slk\_Latin & 3 & dik\_Latin & 1 & smo\_Latin & 1 & shn\_Mymr & 0 \\ slv\_Latin & 3 & doz\_Tibt & 1 & sna\_Latin & 1 & sin\_Sinh & 0 \\ tam\_Tamil & 3 & epo\_Latin & 1 & snd\_Arab & 1 & sot\_Latin & 0 \\ tgl\_Latin & 3 & ewe\_Latin & 1 & som\_Latin & 1 & taq\_Latin & 0 \\ tha\_Thai & 3 & fao\_Latin & 1 & srd\_Latin & 1 & taq\_Trng & 0 \\ ukr\_Cyrl & 3 & fij\_Latin & 1 & ssw\_Latin & 1 & trim\_Tfng & 0 \\ urd\_Arab & 3 & fur\_Latin & 1 & sun\_Latin & 1 & umb\_Latin & 0 \\ uzm\_Latin & 3 & gaž\_Latin & 1 & szl\_Latin & 1 & yee\_Hart & 0 \\ \hline \end{tabular}
\end{table}
Table 9: All NLs and their classes included in mHumanEval.



Prompt Translation and Evaluation Algorithm

The pseudocode version of the workflow, presented in Figure 2.

```
1:for each extracted prompt from HumanEval do
2:for each translation system do
3:for each target language do
4:if the language is supported then
5: generate \(\underline{5}\) translated candidate prompts
6: do back translation
7: calculate BERT_Score and Comet_Kiwi for each
8: take the average of the two
9: pick the best prompt
10:else
11: do back translation
12: calculate only BERT_Score
13: pick the best prompt
14:endif
15:endfor
16:endfor
```

**Algorithm 1** Prompt Translation and Evaluation

It describes how the originally extracted prompts go through 13 candidate translations and evaluation via BERTScore and CometKiwi to build the new sets of benchmarks in the target natural languages.

## Appendix D Evaluation Metric 1: BERTScore

BERTScore uses pre-trained BERT embeddings to assess similarity between candidate and reference translations. For a candidate sentence \(C\) and a reference sentence \(R\), let \(E_{C}\) and \(E_{R}\) be the sets of BERT embeddings for tokens in \(C\) and \(R\), respectively. The similarity \(S(i,j)\) between tokens \(i\) and \(j\) is the cosine similarity of their embeddings:

\[S(i,j)=\frac{e_{C_{i}}\cdot e_{R_{j}}}{\|e_{C_{i}}\|\|e_{R_{j}}\|}\]

Precision \(P\), recall \(R\), and F1-score \(F1\) are then:

\[P =\frac{1}{|E_{C}|}\sum_{e_{C_{i}}\in E_{C}}\max_{e_{R_{j}}\in E_{ R}}S(i,j)\] \[R =\frac{1}{|E_{R}|}\sum_{e_{R_{j}}\in E_{R}}\max_{e_{C_{i}}\in E_{ C}}S(j,i)\] 

Comparison of the Prompt Qualities by the 3 models vs mHumanEval

Figure 10: Comparing the Machine Translation Quality for GPT4o, NLLB and Google Translator. The metrics used are BERTScore and CometKiwi. As shown in the figure, the prompts chosen for mHumanEval are better in quality upon choosing from 13 different candidates.

 



## Appendix G Prompt Templates

**GPT4o and GPT3.5**: prompt = "Write a Python function for the following: " + mHumanEval[i] + " Ensure your response includes a Python code block." messages=[ {^role": "system", "content":  "You are a helpful assistant  trained to generate Python code.  "},  {^role": "user", "content":  prompt}  ]

### WizardCoder

Below is an instruction that describes a task. Write a response that appropriately completes the request.

#### Instruction: "mHumanEval[i]"
##### Response:

Figure 16: Prompt template - DeepSeek-Coder.

Figure 17: Prompt template - MagiCoder.

Figure 18: Prompt template - Cloude3-Opus.



Evaluation Results: mHumanEval-PL

We evaluate the six LLMs from Table 5 for all 204 NLs in four different PLs. More specifically, we evaluate them on four subsets of mHumanEval - mHumanEval-C++, mHumanEval-JAMA, mHumanEval-JavaScript, and mHumanEval-Ruby. The results with mHumanEval-Python are presented in Figure 6 and discussed in Section 4. The performance trend is similar to Python, as discussed in Section 4. However, the results are slightly worse than those of Python.

### mHumanEval-C++

### mHumanEval-JAVA

Figure 19: Comparing model performances (% in Pass@1) for the six models on mHumanEval-**JAVA**.

Figure 18: Comparing model performances (% in Pass@1) for the six models on mHumanEval-**C++**.

 



### mHumanEval-Ruby

Figure 21: Comparing model performances (% in Pass@1) for the six models on mHumanEval-**Ruby**.

Figure 20: Comparing model performances (% in Pass@1) for the six models on mHumanEval-**JavaScript**.





### Analyzing PL-specific results

Performance Decline in Lower Classes (0-2) Models generally exhibit a noticeable performance decline in lower language classes, particularly Classes 0-2. Across all programming languages, scores in these classes fall well below the performance seen in Classes 4 and 5. This decline is especially pronounced in JavaScript and Ruby, where scores frequently drop to or near 0.000, suggesting these classes pose additional challenges.

General Trends Across Language ClassesIn Classes 4 and 5, GPT-4 and Claude3.5 achieve high scores, often exceeding 0.85 in Python and Java. Python consistently demonstrates the highest scores, especially in Class 5, where models like GPT-4 and DeepSeek-Coder surpass 0.88. However, in Classes 0-3, performance drops across all models, particularly in JavaScript and Ruby, where scores frequently fall below 0.65.

Underperformance of WizardCoder and Aya in JavaScript and Ruby Across ClassesWizardCoder and Aya consistently struggle across all language classes in JavaScript and Ruby. In Classes 0-3, their scores frequently reach 0.000, underscoring limitations in handling these scripting languages regardless of language class.

Mixed Adaptability of DeepSeek-Coder AcrossLanguage ClassesDeepSeek-Coder shows moderate scores in Python for higher classes (Classes 4 and 5) but drops to 0.000 in lower classes, particularly in JavaScript and Ruby, highlighting issues with adaptability across classes.

#### 1.1.1 Implications for Future Model Development

The significant underperformance in JavaScript and Ruby across language classes indicates a need for enhanced training in scripting languages. Models like GPT-4 and Claude3.5 excel in higher classes, particularly in Python and Java, but gaps in lower classes and scripting languages suggest a focus on diversifying training data to boost adaptability.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Model** & **Python (C2)** & **Java (C2)** & **Ce++ (C2)** & **JavaScript (Cl1)** & **Ruby (C1)** \\ \hline GPT+ & 0.600 & 0.939 & 0.951 & 0.000 & 0.000 \\ GTP+ 5 & 0.200 & 0.180 & 0.181 & 0.000 & 0.000 \\ Claude3.5 & 0.620 & 0.600 & 0.601 & 0.748 & 0.473 \\ DuyeSock-Coder & 0.300 & 0.330 & 0.331 & 0.000 & 0.000 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Performance of models in lower classes (0-2) across programming languages, with pronounced drops, particularly in JavaScript and Ruby.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Language Class** & **Python** & **Java** & **C++** & **JavaScript** & **Ruby** \\ \hline Class 5 & 0.880 & 0.850 & 0.852 & 0.000 & 0.000 \\ Class 4 & 0.860 & 0.830 & 0.832 & 0.000 & 0.000 \\ Class 3 & 0.500 & 0.480 & 0.482 & 0.000 & 0.000 \\ \hline \hline \end{tabular}
\end{table}
Table 13: DeepSeek-Coder’s performance across language classes, illustrating high scores in Python and Java in Classes 4 and 5, but collapsing to 0.000 in JavaScript and Ruby.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Dataset** & **Python** & **Java** & **C++** & **JavaScript** & **Ruby** \\ \hline Class 5 & 0.880 & 0.850 & 0.852 & 0.650 & 0.653 \\ Class 4 & 0.860 & 0.830 & 0.832 & 0.640 & 0.643 \\ Class 3 & 0.750 & 0.720 & 0.721 & 0.530 & 0.533 \\ Class 2 & 0.620 & 0.600 & 0.601 & 0.420 & 0.423 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Claude3.5’s consistent performance across language classes and programming languages, with scores remaining stable above 0.477.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Class** & **Python** & **Java** & **C++** & **JavaScript** & **Ruby** \\ \hline Class 5 & 0.880 & 0.850 & 0.852 & 0.650 & 0.653 \\ Class 4 & 0.860 & 0.830 & 0.832 & 0.640 & 0.643 \\ Class 3 & 0.750 & 0.720 & 0.721 & 0.530 & 0.533 \\ Class 2 & 0.620 & 0.600 & 0.601 & 0.420 & 0.423 \\ \hline \hline \end{tabular}
\end{table}
Table 11: General model performance across language classes, highlighting high scores in Classes 4 and 5 and lower scores in Classes 0-3, particularly in JavaScript and Ruby.





## Appendix I Evaluating Prompt Translation by GPT4

\begin{table}
\begin{tabular}{l c r l|l c c c} \hline \hline
**Language** & **Class** & **BERTScore** & **CometKiwi** & **Language** & **Class** & **BERTScore** & **CometKiwi** \\ \hline \hline arb\_Arab & 5 & 0.927 & 0.807 & tha\_Thai & 3 & 0.874 & 0.749 \\ deu\_Latin & 5 & 0.948 & 0.826 & ukr\_Cyrl & 3 & 0.872 & 0.722 \\ eng\_Latin & 5 & 1.000 & 0.930 & urd\_Arab & 3 & 0.841 & 0.682 \\ fra\_Latin & 5 & 0.927 & 0.807 & uzn\_Latin & 3 & 0.885 & 0.740 \\ jpn\_Japan & 5 & 0.948 & 0.807 & zsm\_Latin & 3 & 0.890 & 0.711 \\ spa\_Latin & 5 & 0.927 & 0.839 & amh\_Ethi & 2 & 0.825 & 0.690 \\ zho\_Hans & 5 & 0.921 & 0.784 & gle\_Latin & 2 & 0.824 & 0.666 \\ cat\_Latin & 4 & 0.911 & 0.784 & hau\_Latin & 2 & 0.810 & 0.666 \\ ces\_Latin & 4 & 0.914 & 0.799 & ibo\_Latin & 2 & 0.842 & 0.685 \\ eus\_Latin & 4 & 0.920 & 0.754 & kin\_Latin & 2 & 0.838 & 0.665 \\ fin\_Latin & 4 & 0.870 & 0.798 & lao\_Laoo & 2 & 0.844 & 0.690 \\ him\_Deva & 4 & 0.879 & 0.795 & lung\_Latin & 2 & 0.824 & 0.685 \\ hrv\_Latin & 4 & 0.921 & 0.768 & lua\_Latin & 2 & 0.842 & 0.690 \\ hun\_Latin & 4 & 0.879 & 0.784 & luo\_Latin & 2 & 0.824 & 0.685 \\ ita\_Latin & 4 & 0.916 & 0.768 & mar\_Deva & 2 & 0.811 & 0.676 \\ kor\_Hang & 4 & 0.930 & 0.768 & npi\_Deva & 2 & 0.812 & 0.666 \\ nld\_Latin & 4 & 0.887 & 0.799 & orm\_Latin & 2 & 0.842 & 0.665 \\ pes\_Arab & 4 & 0.929 & 0.768 & prs\_Arab & 2 & 0.827 & 0.685 \\ pol\_Latin & 4 & 0.894 & 0.754 & quc\_Latin & 2 & 0.842 & 0.665 \\ por\_Latin & 4 & 0.879 & 0.798 & sag\_Latin & 2 & 0.811 & 0.676 \\ ru\_Cyrl & 4 & 0.929 & 0.754 & sna\_Latin & 2 & 0.812 & 0.666 \\ spr\_Cyrl & 4 & 0.879 & 0.795 & srd\_Latin & 2 & 0.842 & 0.665 \\ swe\_Latin & 4 & 0.914 & 0.798 & tso\_Latin & 2 & 0.842 & 0.665 \\ tur\_Latin & 4 & 0.920 & 0.754 & uzb\_Latin & 2 & 0.827 & 0.685 \\ vie\_Latin & 4 & 0.894 & 0.795 & zdi\_Arab & 2 & 0.811 & 0.676 \\ arb\_Latin & 3 & 0.894 & 0.731 & fuv\_Latin & 1 & 0.844 & 0.666 \\ afr\_Latin & 3 & 0.890 & 0.711 & gaz\_Latin & 1 & 0.839 & 0.665 \\ arz\_Arab & 3 & 0.891 & 0.748 & hin\_Latin & 1 & 0.841 & 0.682 \\ ben\_Beng & 3 & 0.872 & 0.749 & jav\_Latin & 1 & 0.776 & 0.508 \\ bos\_Latin & 3 & 0.900 & 0.731 & kan\_Knda & 1 & 0.755 & 0.489 \\ bul\_Cyrl & 3 & 0.886 & 0.677 & knhm\_Khmr & 1 & 0.787 & 0.529 \\ ceb\_Latin & 3 & 0.841 & 0.682 & kir\_Cyrl & 1 & 0.765 & 0.578 \\ dan\_Latin & 3 & 0.827 & 0.733 & kmr\_Latin & 1 & 0.784 & 0.567 \\ ell\_Grek & 3 & 0.887 & 0.727 & mal\_Mlym & 1 & 0.785 & 0.529 \\ est\_Latin & 3 & 0.885 & 0.715 & mkd\_Cyrl & 1 & 0.737 & 0.578 \\ glg\_Latin & 3 & 0.867 & 0.701 & mya\_Mymr & 1 & 0.760 & 0.579 \\ heb\_Hebr & 3 & 0.895 & 0.673 & nob\_Latin & 1 & 0.750 & 0.515 \\ ind\_Latin & 3 & 0.874 & 0.677 & org\_Orya & 1 & 0.776 & 0.579 \\ ket\_Geor & 3 & 0.892 & 0.741 & snd\_Arab & 1 & 0.788 & 0.432 \\ kaz\_Cyrl & 3 & 0.850 & 0.745 & som\_Latin & 1 & 0.750 & 0.512 \\ lit\_Latin & 3 & 0.886 & 0.740 & sun\_Latin & 1 & 0.778 & 0.567 \\ lvs\_Latin & 3 & 0.827 & 0.688 & tel\_Telu & 1 & 0.745 & 0.560 \\ roon\_Latin & 3 & 0.895 & 0.722 & ugi\_Arab & 1 & 0.741 & 0.529 \\ slk\_Latin & 3 & 0.886 & 0.731 & ydd\_Hebr & 1 & 0.768 & 0.508 \\ slv\_Latin & 3 & 0.890 & 0.745 & zho\_Hant & 1 & 0.788 & 0.529 \\ tam\_Taml & 3 & 0.887 & 0.677 & sin\_Sinhn & 0 & 0.690 & 0.410 \\ tgl\_Latin & 3 & 0.827 & 0.731 & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 15: Evaluating the quality of machine translation by GPT4 using BERTscore and CometKiwi. The languages are given as Flores-200 codes.





\begin{table}
\begin{tabular}{l c r|c c c} \hline \hline
**Language** & **Class** & **BERTScore** & **Language** & **Class** & **BERTScore** \\ \hline ace\_Arab & 1 & 0.666 & quy\_Latin & 1 & 0.722 \\ ace\_Latn & 1 & 0.719 & sag\_Latn & 1 & 0.735 \\ acm\_Arab & 1 & 0.680 & sat\_Olck & 1 & 0.717 \\ acq\_Arab & 1 & 0.714 & scn\_Latin & 1 & 0.730 \\ aeb\_Arab & 1 & 0.664 & smo\_Latin & 1 & 0.738 \\ ajp\_Arab & 1 & 0.675 & sna\_Latin & 1 & 0.679 \\ aka\_Latin & 1 & 0.683 & snd\_Latin & 1 & 0.705 \\ als Latn & 1 & 0.692 & ssw\_Latin & 1 & 0.739 \\ apc\_Arab & 1 & 0.697 & szI\_Latin & 1 & 0.716 \\ arx\_Arab & 1 & 0.727 & tat\_Cyrl & 1 & 0.724 \\ any\_Arab & 1 & 0.700 & tgk\_Cyrl & 1 & 0.673 \\ ast\_Latin & 1 & 0.716 & trip\_Latin & 1 & 0.730 \\ avr\_Latin & 1 & 0.693 & tso\_Latin & 1 & 0.696 \\ azp\_Arab & 1 & 0.673 & tuk\_Latin & 1 & 0.674 \\ bak\_Cyrl & 1 & 0.702 & tum\_Latin & 1 & 0.724 \\ bho\_Deva & 1 & 0.708 & twi\_Latin & 1 & 0.665 \\ bin\_Arab & 1 & 0.686 & vec\_Latin & 1 & 0.698 \\ bip\_Latin & 1 & 0.701 & war\_Latin & 1 & 0.735 \\ bod\_Tibt & 1 & 0.735 & awa\_Deva & 0 & 0.600 \\ bug\_Latin & 1 & 0.678 & bam\_Latin & 0 & 0.691 \\ ckb\_Arab & 1 & 0.662 & ban\_Latin & 0 & 0.677 \\ crh\_Latin & 1 & 0.682 & ben\_Latin & 0 & 0.656 \\ dik\_Latin & 1 & 0.727 & cjk\_Latin & 0 & 0.691 \\ dzo\_Tibt & 1 & 0.727 & dyu\_Latin & 0 & 0.648 \\ ewe\_Latin & 1 & 0.706 & fon\_Latin & 0 & 0.685 \\ fao\_Latin & 1 & 0.662 & fuv\_Latin & 0 & 0.613 \\ fij\_Latin & 1 & 0.730 & grm\_Latin & 0 & 0.680 \\ fur\_Latin & 1 & 0.733 & hat\_Latin & 0 & 0.649 \\ gaz\_Latin & 1 & 0.669 & hne\_Deva & 0 & 0.633 \\ ibo\_Latin & 1 & 0.680 & kac\_Latin & 0 & 0.665 \\ ilo\_Latin & 1 & 0.723 & kam\_Latin & 0 & 0.623 \\ kab\_Latin & 1 & 0.690 & kbp\_Latin & 0 & 0.646 \\ ka\_Arab & 1 & 0.677 & kea\_Latin & 0 & 0.698 \\ las\_Deva & 1 & 0.698 & knb\_Latin & 0 & 0.680 \\ khk\_Cyrl & 1 & 0.686 & kne\_Arab & 0 & 0.612 \\ kil\_Latin & 1 & 0.694 & kne\_Latin & 0 & 0.684 \\ kin\_Latin & 1 & 0.704 & kon\_Latin & 0 & 0.688 \\ lij\_Latin & 1 & 0.683 & lux\_Latin & 0 & 0.671 \\ lim\_Latin & 1 & 0.691 & luo\_Latin & 0 & 0.667 \\ lin\_Latin & 1 & 0.699 & lus\_Latin & 0 & 0.603 \\ Imo\_Latin & 1 & 0.662 & mag\_Deva & 0 & 0.665 \\ tp\_Latin & 1 & 0.720 & mini\_Beng & 0 & 0.629 \\ ltz\_Latin & 1 & 0.688 & mos\_Latin & 0 & 0.610 \\ lug\_Latin & 1 & 0.685 & nso\_Latin & 0 & 0.601 \\ mai\_Deva & 1 & 0.664 & nus\_Latin & 0 & 0.614 \\ min\_Arab & 1 & 0.669 & nya\_Latin & 0 & 0.649 \\ min\_Latin & 1 & 0.695 & prs\_Arab & 0 & 0.698 \\ miri\_Latin & 1 & 0.706 & run\_Latin & 0 & 0.608 \\ mono\_Latin & 1 & 0.703 & shn\_Mymr & 0 & 0.670 \\ npi\_Deva & 1 & 0.671 & so\_Latin & 0 & 0.624 \\oci\_Latin & 1 & 0.732 & taq\_Latin & 0 & 0.658 \\ pag\_Latin & 1 & 0.662 & taq\_Tfng & 0 & 0.677 \\ pap\_Latin & 1 & 0.728 & tzm\_Tfng & 0 & 0.622 \\ pbt\_Arab & 1 & 0.693 & umb\_Latin & 0 & 0.643 \\ plt\_Latin & 1 & 0.696 & yue\_Hant & 0 & 0.696 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Evaluating the quality of machine translation by GPT4 using BERTscore. These languages are not supported by CometKiwl. The languages are given as Flores-200 codes.





## Appendix J Evaluating Prompt Translation by NLLB

\begin{table}
\begin{tabular}{l c r l|l c c} \hline \hline
**Language** & **Class** & **BERTScore** & **CometKiwi** & **Language** & **Class** & **BERTScore** & **CometKiwi** \\ \hline \hline arb\_Arab & 5 & 0.941 & 0.798 & tha\_Thai & 3 & 0.881 & 0.704 \\ deu\_Latin & 5 & 0.902 & 0.809 & ukr\_Cyrl & 3 & 0.883 & 0.685 \\ eng\_Latin & 5 & 1.000 & 0.910 & urd\_Arab & 3 & 0.872 & 0.697 \\ fra\_Latin & 5 & 0.917 & 0.787 & uzn\_Latin & 3 & 0.875 & 0.697 \\ jpn\_Japan & 5 & 0.935 & 0.807 & zsm\_Latin & 3 & 0.864 & 0.697 \\ spa\_Latin & 5 & 0.935 & 0.809 & amh\_Ethi & 2 & 0.817 & 0.597 \\ zho\_Hans & 5 & 0.911 & 0.831 & gle\_Latin & 2 & 0.816 & 0.555 \\ cat\_Latin & 4 & 0.909 & 0.777 & hau\_Latin & 2 & 0.827 & 0.574 \\ ces\_Latin & 4 & 0.899 & 0.743 & ibo\_Latin & 2 & 0.816 & 0.579 \\ eus\_Latin & 4 & 0.877 & 0.719 & kin\_Latin & 2 & 0.817 & 0.573 \\ fin\_Latin & 4 & 0.875 & 0.704 & lao\_Laoo & 2 & 0.810 & 0.585 \\ him\_Deva & 4 & 0.875 & 0.697 & lung\_Latin & 2 & 0.817 & 0.573 \\ hrv\_Latin & 4 & 0.875 & 0.697 & lua\_Latin & 2 & 0.816 & 0.579 \\ hun\_Latin & 4 & 0.872 & 0.685 & luco\_Latin & 2 & 0.818 & 0.585 \\ ita\_Latin & 4 & 0.872 & 0.719 & mar\_Deva & 2 & 0.817 & 0.573 \\ kor\_Hang & 4 & 0.872 & 0.685 & npi\_Deva & 2 & 0.816 & 0.579 \\ nld\_Latin & 4 & 0.875 & 0.704 & orm\_Latin & 2 & 0.817 & 0.573 \\ pes\_Arab & 4 & 0.875 & 0.697 & prs\_Arab & 2 & 0.816 & 0.579 \\ pol\_Latin & 4 & 0.872 & 0.685 & quc\_Latin & 2 & 0.818 & 0.585 \\ por\_Latin & 4 & 0.875 & 0.704 & sag\_Latin & 2 & 0.817 & 0.573 \\ ru\_Cyrl & 4 & 0.875 & 0.697 & sna\_Latin & 2 & 0.816 & 0.579 \\ spr\_Cyrl & 4 & 0.872 & 0.685 & srd\_Latin & 2 & 0.817 & 0.573 \\ swe\_Latin & 4 & 0.875 & 0.704 & tso\_Latin & 2 & 0.818 & 0.585 \\ tur\_Latin & 4 & 0.872 & 0.685 & uzb\_Latin & 2 & 0.817 & 0.573 \\ vie\_Latin & 4 & 0.875 & 0.704 & zdi\_Arab & 2 & 0.816 & 0.579 \\ arb\_Latin & 3 & 0.875 & 0.697 & fuv\_Latin & 1 & 0.818 & 0.585 \\ afr\_Latin & 3 & 0.875 & 0.704 & gaz\_Latin & 1 & 0.817 & 0.573 \\ arz\_Arab & 3 & 0.872 & 0.685 & hin\_Latin & 1 & 0.816 & 0.579 \\ ben\_Beng & 3 & 0.872 & 0.697 & jav\_Latin & 1 & 0.758 & 0.535 \\ bul\_Cyrl & 3 & 0.875 & 0.697 & kan\_Knda & 1 & 0.740 & 0.577 \\ ceb\_Latin & 3 & 0.872 & 0.716 & khm\_Khmr & 1 & 0.754 & 0.561 \\ dan\_Latin & 3 & 0.851 & 0.666 & kir\_Cyrl & 1 & 0.757 & 0.583 \\ ell\_Grek & 3 & 0.883 & 0.709 & kmr\_Latin & 1 & 0.770 & 0.579 \\ est\_Latin & 3 & 0.877 & 0.661 & mal\_Mlym & 1 & 0.736 & 0.550 \\ glg\_Latin & 3 & 0.864 & 0.697 & mkd\_Cyrl & 1 & 0.736 & 0.559 \\ heb\_Hebr & 3 & 0.828 & 0.701 & mya\_Mymr & 1 & 0.770 & 0.582 \\ ind\_Latin & 3 & 0.864 & 0.697 & nob\_Latin & 1 & 0.766 & 0.535 \\ ket\_Geor & 3 & 0.880 & 0.709 &ory\_Orya & 1 & 0.743 & 0.582 \\ kaz\_Cyrl & 3 & 0.877 & 0.719 & snd\_Arab & 1 & 0.743 & 0.582 \\ lit\_Latin & 3 & 0.872 & 0.697 & som\_Latin & 1 & 0.770 & 0.520 \\ lvs\_Latin & 3 & 0.880 & 0.661 & sun\_Latin & 1 & 0.743 & 0.540 \\ ro\_Latin & 3 & 0.864 & 0.713 & tel\_Telu & 1 & 0.754 & 0.540 \\ slk\_Latin & 3 & 0.828 & 0.716 & ug\_Arab & 1 & 0.751 & 0.579 \\ slv\_Latin & 3 & 0.879 & 0.685 & ydd\_Hebr & 1 & 0.757 & 0.556 \\ tam\_Tamil & 3 & 0.877 & 0.716 & zho\_Hant & 1 & 0.736 & 0.583 \\ tgl\_Latin & 3 & 0.851 & 0.713 & sin\_Sinhn & 0 & 0.645 & 0.490 \\ \hline \hline \end{tabular}
\end{table}
Table 17: Evaluating the quality of machine translation by NLLB using BERTScore and CometKiwi. The languages are given as Flores-200 codes.





\begin{table}
\begin{tabular}{l c r|c c c} \hline
**Language** & **Class** & **BERTScore** & **Language** & **Class** & **BERTScore** \\ \hline ace\_Arab & 1 & 0.672 & quy\_Latin & 1 & 0.700 \\ ace\_Latn & 1 & 0.716 & sag\_Latn & 1 & 0.702 \\ acm\_Arab & 1 & 0.664 & sat\_Olck & 1 & 0.681 \\ acq\_Arab & 1 & 0.669 & scn\_Latin & 1 & 0.668 \\ aeb\_Arab & 1 & 0.664 & smo\_Latin & 1 & 0.739 \\ ajp\_Arab & 1 & 0.690 & sna\_Latin & 1 & 0.679 \\ aka\_Latin & 1 & 0.720 & srd\_Latin & 1 & 0.694 \\ als Latn & 1 & 0.716 & ssw\_Latin & 1 & 0.735 \\ apc\_Arab & 1 & 0.683 & szl\_Latin & 1 & 0.739 \\ arx\_Arab & 1 & 0.669 & tat\_Cyrl & 1 & 0.716 \\ any\_Arab & 1 & 0.666 & tgk\_Cyrl & 1 & 0.672 \\ ast\_Latin & 1 & 0.672 & trip\_Latin & 1 & 0.670 \\ avr\_Latin & 1 & 0.684 & tso\_Latin & 1 & 0.706 \\ azp\_Arab & 1 & 0.688 & tuk\_Latin & 1 & 0.690 \\ bak\_Cyrl & 1 & 0.699 & turn\_Latin & 1 & 0.692 \\ bho\_Deva & 1 & 0.719 & twi\_Latin & 1 & 0.705 \\ bin\_Arab & 1 & 0.734 & vec\_Latin & 1 & 0.709 \\ bip\_Latin & 1 & 0.668 & war\_Latin & 1 & 0.684 \\ bod\_Tibt & 1 & 0.692 & awa\_Deva & 0 & 0.644 \\ bug\_Latin & 1 & 0.670 & bam\_Latin & 0 & 0.607 \\ ckb\_Arab & 1 & 0.733 & ban\_Latin & 0 & 0.645 \\ crh\_Latin & 1 & 0.670 & bem\_Latin & 0 & 0.613 \\ dik\_Latin & 1 & 0.710 & cjk\_Latin & 0 & 0.658 \\ dzo\_Tibt & 1 & 0.726 & dyu\_Latin & 0 & 0.664 \\ ewe\_Latin & 1 & 0.737 & fon\_Latin & 0 & 0.694 \\ fao\_Latin & 1 & 0.710 & fur\_Latin & 0 & 0.615 \\ fij\_Latin & 1 & 0.689 & grm\_Latin & 0 & 0.677 \\ fur\_Latin & 1 & 0.739 & hat\_Latin & 0 & 0.666 \\ gaz\_Latin & 1 & 0.708 & hne\_Deva & 0 & 0.686 \\ ibo\_Latin & 1 & 0.687 & kac\_Latin & 0 & 0.651 \\ ilo\_Latin & 1 & 0.722 & kam\_Latin & 0 & 0.672 \\ kab\_Latin & 1 & 0.680 & kbp\_Latin & 0 & 0.600 \\ kaas\_Arab & 1 & 0.684 & kea\_Latin & 0 & 0.672 \\ las\_Deva & 1 & 0.716 & kmb\_Latin & 0 & 0.636 \\ khk\_Cyrl & 1 & 0.725 & kne\_Arab & 0 & 0.615 \\ kil\_Latin & 1 & 0.668 & knc\_Latin & 0 & 0.611 \\ kin\_Latin & 1 & 0.705 & kon\_Latin & 0 & 0.637 \\ lij\_Latin & 1 & 0.719 & lux\_Latin & 0 & 0.606 \\ lim\_Latin & 1 & 0.706 & luo\_Latin & 0 & 0.679 \\ lin\_Latin & 1 & 0.723 & lus\_Latin & 0 & 0.632 \\ Imo\_Latin & 1 & 0.690 & mag\_Deva & 0 & 0.600 \\ tp\_Latin & 1 & 0.681 & mini\_Beng & 0 & 0.655 \\ ltz\_Latin & 1 & 0.727 & mos\_Latin & 0 & 0.688 \\ lug\_Latin & 1 & 0.712 & nso\_Latin & 0 & 0.635 \\ mai\_Deva & 1 & 0.710 & nus\_Latin & 0 & 0.674 \\ min\_Arab & 1 & 0.666 & nya\_Latin & 0 & 0.699 \\ min\_Latin & 1 & 0.724 & prs\_Arab & 0 & 0.609 \\ miri\_Latin & 1 & 0.726 & run\_Latin & 0 & 0.615 \\ mono\_Latin & 1 & 0.703 & shn\_Mymr & 0 & 0.657 \\ npi\_Deva & 1 & 0.675 & so\_Latin & 0 & 0.601 \\oci\_Latin & 1 & 0.735 & taq\_Latin & 0 & 0.658 \\ pag\_Latin & 1 & 0.709 & taq\_Tfng & 0 & 0.677 \\ pap\_Latin & 1 & 0.664 & tzm\_Tfng & 0 & 0.607 \\ pot\_Arab & 1 & 0.696 & umb\_Latin & 0 & 0.643 \\ plt\_Latin & 1 & 0.683 & yue\_Hant & 0 & 0.677 \\ \hline \end{tabular}
\end{table}
Table 18: Evaluating the quality of machine translation by NLLB using BERTscore. These languages are not supported by CometKiwI. The languages are given as Flores-200 codes.



Evaluating Prompt Translation by Google Translate

\begin{table}
\begin{tabular}{l c r r|c c c c} \hline \hline




## Appendix L Evaluating Prompt Quality in mHumanEval

\begin{table}
\begin{tabular}{l l c c|l r r r} \hline \hline
**Language** & **Class** & **BERTScore** & **CometKiwi** & **Language** & **Class** & **BERTScore** & **CometKiwi** \\ \hline eng\_Latin & 5 & 1.000 & 0.961 & urd\_Arab & 3 & 0.911 & 0.782 \\ spa\_Latin & 5 & 0.98 & 0.919 & bul\_Cyrl & 3 & 0.956 & 0.777 \\ deu\_Latin & 5 & 0.99 & 0.927 & ind\_Latin & 3 & 0.944 & 0.777 \\ fra\_Latin & 5 & 0.98 & 0.896 & tam\_Taml & 3 & 0.957 & 0.777 \\ zho\_Hans & 5 & 0.96 & 0.89 & heb\_Hebr & 3 & 0.965 & 0.773 \\ jpn\_Japan & 5 & 0.97 & 0.88 & amh\_Ethi & 2 & 0.895 & 0.77 \\ arb\_Arab & 5 & 0.96 & 0.867 & mlt\_Latin & 2 & 0.904 & 0.77 \\ ces\_Latin & 4 & 0.964 & 0.839 & isi\_Latin & 2 & 0.898 & 0.765 \\ nld\_Latin & 4 & 0.937 & 0.839 & ir\_Ethi & 2 & 0.913 & 0.765 \\ fin\_Latin & 4 & 0.92 & 0.838 & yor\_Latin & 2 & 0.913 & 0.765 \\ por\_Latin & 4 & 0.929 & 0.838 & zul\_Latin & 2 & 0.913 & 0.765 \\ swe\_Latin & 4 & 0.964 & 0.838 & lao\_Laoo & 2 & 0.887 & 0.756 \\ hin\_Deva & 4 & 0.929 & 0.835 & mar\_Deva & 2 & 0.881 & 0.756 \\ spr\_Cyrl & 4 & 0.929 & 0.835 & xho\_Latin & 2 & 0.881 & 0.756 \\ vie\_Latin & 4 & 0.944 & 0.835 & gle\_Latin & 2 & 0.894 & 0.746 \\ cat\_Latin & 4 & 0.961 & 0.824 & pan\_Guru & 2 & 0.916 & 0.746 \\ hun\_Latin & 4 & 0.929 & 0.824 & san\_Deva & 2 & 0.925 & 0.746 \\ hrv\_Latin & 4 & 0.971 & 0.808 & wol\_Latin & 2 & 0.884 & 0.746 \\ ita\_Latin & 4 & 0.966 & 0.808 & hau\_Latin & 2 & 0.884 & 0.745 \\ kor\_Hang & 4 & 0.98 & 0.808 & swh\_Latin & 2 & 0.913 & 0.745 \\ pes\_Arab & 4 & 0.979 & 0.808 & tsn\_Latin & 2 & 0.925 & 0.745 \\ eus\_Latin & 4 & 0.97 & 0.794 & guj\_Guir & 1 & 0.828 & 0.717 \\ pol\_Latin & 4 & 0.944 & 0.794 & epo\_Latin & 1 & 0.813 & 0.709 \\ ru\_Cyrl & 4 & 0.979 & 0.794 & mya\_Mymr & 1 & 0.828 & 0.709 \\ tur\_Latin & 4 & 0.97 & 0.794 & ory\_Orya & 1 & 0.84 & 0.709 \\ ben\_Beng & 3 & 0.942 & 0.849 & kir\_Cyrl & 1 & 0.819 & 0.708 \\ tha\_Thai & 3 & 0.944 & 0.849 & mkd\_Cyrl & 1 & 0.873 & 0.708 \\ arz\_Arab & 3 & 0.961 & 0.848 & cym\_Latn & 1 & 0.865 & 0.707 \\ kaz\_Cyrl & 3 & 0.92 & 0.845 & kmr\_Latin & 1 & 0.86 & 0.697 \\ slv\_Latin & 3 & 0.96 & 0.845 & sun\_Latin & 1 & 0.854 & 0.697 \\ ket\_Geor & 3 & 0.962 & 0.841 & gla\_Latin & 1 & 0.846 & 0.69 \\ lit\_Latin & 3 & 0.956 & 0.84 & tel\_Telu & 1 & 0.823 & 0.69 \\ uzn\_Latin & 3 & 0.955 & 0.84 & khm\_Khmr & 1 & 0.867 & 0.659 \\ bos\_Latin & 3 & 0.97 & 0.839 & mal\_Mlym & 1 & 0.871 & 0.659 \\ dan\_Latin & 3 & 0.897 & 0.833 & ug\_Arab & 1 & 0.809 & 0.659 \\ arb\_Latin & 3 & 0.964 & 0.831 & zho\_Hart & 1 & 0.876 & 0.659 \\ slk\_Latin & 3 & 0.956 & 0.831 & asm\_Beng & 1 & 0.871 & 0.645 \\ tgl\_Latin & 3 & 0.897 & 0.831 & nob\_Latin & 1 & 0.823 & 0.645 \\ ell\_Grek & 3 & 0.957 & 0.827 & hye\_Armn & 1 & 0.852 & 0.642 \\ ro\_Latin & 3 & 0.965 & 0.822 & som\_Latin & 1 & 0.812 & 0.642 \\ ukr\_Cyrl & 3 & 0.942 & 0.822 & java\_Latin & 1 & 0.828 & 0.638 \\ est\_Latin & 3 & 0.955 & 0.815 & ydd\_Hebr & 1 & 0.875 & 0.638 \\ afr\_Latin & 3 & 0.96 & 0.811 & kan\_Knda & 1 & 0.825 & 0.619 \\ zsm\_Latin & 3 & 0.96 & 0.811 & bel\_Cyrl & 1 & 0.809 & 0.613 \\ gig\_Latin & 3 & 0.937 & 0.801 & azj\_Latin & 1 & 0.829 & 0.562 \\ lvs\_Latin & 3 & 0.897 & 0.788 & snd\_Arab & 1 & 0.851 & 0.562 \\ ceb\_Latin & 3 & 0.911 & 0.782 & sin\_Sinh & 0 & 0.859 & 0.56 \\ \hline \hline \end{tabular}
\end{table}
Table 20: Observing improved prompt quality in mhumanEval upon choosing the best ones from 13 candidates each, evaluated using BERTScore and CometKiwi. The languages are given as Flores-200 codes.





\begin{table}
\begin{tabular}{l c r|c c c} \hline
**Language** & **Class** & **BERTScore** & **Language** & **Class** & **BERTScore** \\ \hline \hline ace\_Arab & 1 & 0.806 & quy\_Latin & 1 & 0.862 \\ ace\_Latn & 1 & 0.859 & sag\_Latin & 1 & 0.875 \\ acm\_Arab & 1 & 0.82 & sat\_Ock & 1 & 0.857 \\ acq\_Arab & 1 & 0.854 & scn\_Latin & 1 & 0.87 \\ aeb\_Arab & 1 & 0.804 & smo\_Latin & 1 & 0.878 \\ ajp\_Arab & 1 & 0.815 & sna\_Latin & 1 & 0.819 \\ aka\_Latin & 1 & 0.823 & srd\_Latin & 1 & 0.845 \\ als\_Latin & 1 & 0.832 & ssw\_Latin & 1 & 0.879 \\ ape\_Arab & 1 & 0.837 & szl\_Latin & 1 & 0.856 \\ arx\_Arab & 1 & 0.867 & tat\_Cyl & 1 & 0.864 \\ any\_Arab & 1 & 0.84 & tkg\_Cyrl & 1 & 0.813 \\ ast\_Latin & 1 & 0.856 & tip\_Latin & 1 & 0.87 \\ agr\_Latin & 1 & 0.833 & tsoC\_Latin & 1 & 0.836 \\ azb\_Arab & 1 & 0.813 & tuk\_Latin & 1 & 0.814 \\ bak\_Cyrl & 1 & 0.842 & tum\_Latin & 1 & 0.864 \\ bho\_Deva & 1 & 0.848 & twi\_Latin & 1 & 0.805 \\ bijn\_Arab & 1 & 0.826 & vec\_Latin & 1 & 0.838 \\ bin\_Latin & 1 & 0.841 & war\_Latin & 1 & 0.875 \\ bod\_Tbit & 1 & 0.875 & awa\_Deva & 0 & 0.75 \\ bug\_Latin & 1 & 0.818 & barn\_Latin & 0 & 0.841 \\ ckb Arab & 1 & 0.802 & ban\_Latin & 0 & 0.827 \\ crh\_Latin & 1 & 0.822 & bem\_Latin & 0 & 0.806 \\ dik\_Latin & 1 & 0.867 & cjk\_Latin & 0 & 0.841 \\ dzo\_Tibt & 1 & 0.867 & dyu\_Latin & 0 & 0.798 \\ ewe\_Latin & 1 & 0.846 & fon\_Latin & 0 & 0.835 \\ fao\_Latin & 1 & 0.802 & fuv\_Latin & 0 & 0.763 \\ fij\_Latin & 1 & 0.87 & grn\_Latin & 0 & 0.83 \\ fur\_Latin & 1 & 0.873 & hat\_Latin & 0 & 0.799 \\ gaz\_Latin & 1 & 0.809 & hne\_Deva & 0 & 0.783 \\ ibo\_Latin & 1 & 0.82 & kac\_Latin & 0 & 0.815 \\ ilo\_Latin & 1 & 0.863 & kam\_Latin & 0 & 0.773 \\ kb\_Latin & 1 & 0.83 & kbp\_Latin & 0 & 0.796 \\ ka\_Arab & 1 & 0.817 & kea\_Latin & 0 & 0.848 \\ ks\_Deva & 1 & 0.838 & kmb\_Latin & 0 & 0.83 \\ khk\_Cyrl & 1 & 0.826 & knc\_Arab & 0 & 0.762 \\ ikik\_Latin & 1 & 0.834 & kne\_Latin & 0 & 0.834 \\ kin\_Latin & 1 & 0.844 & kon\_Latin & 0 & 0.838 \\ lij\_Latin & 1 & 0.823 & u\_Latin & 0 & 0.821 \\ lim\_Latin & 1 & 0.831 & too\_Latin & 0 & 0.817 \\ lin\_Latin & 1 & 0.839 & lus\_Latin & 0 & 0.753 \\ Imo\_Latin & 1 & 0.802 & mag\_Deva & 0 & 0.815 \\ Hg\_Latin & 1 & 0.86 & mini\_Beng & 0 & 0.779 \\ Itz\_Latin & 1 & 0.828 & mos\_Latin & 0 & 0.76 \\ lug\_Latin & 1 & 0.825 & nso\_Latin & 0 & 0.751 \\ mail\_Deva & 1 & 0.804 & nus\_Latin & 0 & 0.764 \\ min\_Arab & 1 & 0.809 & nya\_Latin & 0 & 0.799 \\ min\_Latin & 1 & 0.835 & ps\_Arab & 0 & 0.848 \\ mri\_Latin & 1 & 0.846 & run\_Latin & 0 & 0.758 \\ nano\_Latin & 1 & 0.843 & shn\_Mymr & 0 & 0.82 \\ npi\_Deva & 1 & 0.811 & sot\_Latin & 0 & 0.774 \\ oci\_Latin & 1 & 0.872 & taq\_Latin & 0 & 0.836 \\ pag\_Latin & 1 & 0.802 & tag\_Tfng & 0 & 0.774 \\ pap\_Latin & 1 & 0.868 & tzm\_Tfng & 0 & 0.772 \\ pbd\_Arab & 1 & 0.833 & umb\_Latin & 0 & 0.795 \\ plt\_Latin & 1 & 0.836 & yue\_Hant & 0 & 0.846 \\ \hline \end{tabular}
\end{table}
Table 21: Observing improved prompt quality in mtDNAEval upon choosing the best ones from 13 candidates each, evaluated using BERTScore. These languages are not supported by CometKwi. The languages are given as Flores-200 codes.





## Appendix M Evaluation Results on mHumanEval

\begin{table}
\begin{tabular}{l c r|r r r|r r} \hline \hline
* Class 4 languages. The languages are given as Flores-200 codes.

\begin{table}
\begin{tabular}{l l r r r r r r} \hline \hline
* Class 5 languages. The languages are given as Flores-200 codes.




\begin{table}
\begin{tabular}{l c r r r|r r||c} \hline
* Class 3 languages. The languages are given as Flores-200 codes.

\begin{table}
\begin{tabular}{l c c c c c c} \hline
* Class 2 languages. The languages are given as Flores-200 codes.




\begin{table}
\begin{tabular}{l l c c c c c c} \hline
**Language** & **Class** & **Class3.5** & **Class4.6** & **Class5.7** & **Class5.8** & **DeepSearch-Color** & **Vinner/Cut** & **Avg** \\ \hline Sec.Num & 1 & 0.922 & 0.925 & 0.928 & 0.005 & 0.000 & 0.000 \\ sec.Num & 1 & 0.922 & 0.925 & 0.928 & 0.005 & 0.000 & 0.000 \\ sec.Num & 1 & 0.923 & 0.921 & 0.926 & 0.019 & 0.000 & 0.000 \\ sec.Num & 1 & 0.924 & 0.925 & 0.924 & 0.019 & 0.000 & 0.000 \\ sec.Num & 1 & 0.936 & 0.739 & 0.934 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.936 & 0.739 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.937 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.939 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.939 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.000 & 0.000 & 0.000 \\ sec.Num & 1 & 0.938 & 0.936 & 0.936 & 0.936 & 0.000 & 0.000 \\ sec.Num & 1 & 0.





\begin{table}
\begin{tabular}{l l c r r r r|r} \hline
* Class 0 languages. The languages are given as Flores-200 codes.