CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
Authors: Weixiang Yan
Yuchen Tian
Yunzhe Li
Qian Chen
Wen Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Code translation
Multilingual datasets
Multilingual modeling
Large language models

Abstract: Recent code translation techniques exploit neural machine translation models to translate source code from one programming language to another to satisfy production compatibility or to improve efficiency of codebase maintenance. Most existing code translation datasets only focus on a single pair of popular programming languages. To advance research on code translation and meet diverse requirements of real-world applications, we construct **CodeTransOcean**, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation. CodeTransOcean consists of three novel multilingual datasets, namely, **MultilingualTrans** supporting translations between multiple popular programming languages, **NicheTrans** for translating between niche programming languages and popular ones, and **LLMTrans** for evaluating executability of translated code by large language models (LLMs).  CodeTransOcean also includes a novel cross-framework dataset, **DLTrans**, for translating deep learning code across different frameworks. We develop multilingual modeling approaches for code translation and demonstrate their great potential in improving the translation quality of both low-resource and high-resource language pairs and boosting the training efficiency. We also propose a novel evaluation metric **Debugging Success Rate@K** for program-level code translation. Last but not least, we evaluate LLM ChatGPT on our datasets and investigate its potential for fuzzy execution predictions. We build baselines for CodeTransOcean and analyze challenges of code translation for guiding future research. The CodeTransOcean datasets and code are publicly available at https://github.com/WeixiangYAN/CodeTransOcean.

CodeTransOccan: A Comprehensive Multilingual Benchmark for Code Translation
Weixiang Yan'*
Yuchen Tian
Yunzhe Li
Qian Chen
Wen Wang'
University of California, Santa Barbara
"The University of Hong Kong
University of Illinois at CibanaaChhappan 'Speech Lab, Alibaba Group
weixiargyan@ucsb.ee edu yuchent&cconnect hku .hk yunzhe12@illioiis edu (tanqing. cq, w. wang ) @alilaaaaliaeee cemaba-inn..om com
Abstract
Introduction
Early software systems developed using pro- gramming languages such as Fortran and COBOL, which have significantly smaller user base com- pared to modern mainstream programming lan- guages (e.g., Python and Java). Hence maintaining and modernizing early software systems are ex- pensive (Opidi, 2020). Moreover, the readability and compatibility of the mixed multitude of pro- gramming languages are challenging when migrat- ing existing software systems to new technology ecosystems or integrating software systems using different programming languages. The code trans- lation task aims to convert source code from one programming language to another and is of great value in industry.
Recent code translation techniques exploit neural machine translation models to trans- late source code from one programming lan- guage to another to satisfy production com- patibility to improve efficiency of code- base maintenance. Most existing code trans- lation datasets only focus
single pair of popular programming languages. To ad- vance research on code translation and meet diverse requirements of real-world applications, we construct CodeTTransOees comprehensive benchmark that supports the largest variety of programming languages for code translation. CodeTransOccan consists of three novel multilingual datasets, namely, Mults supporting translations be- tween multiple popular programming lan- guages, NicheTrans for translating between niche programming languages and popular ones, and L.L.Trans evaluating executabil- ity of translated code by large language mod- els (LL.ss). Code TransOcean also includes a novel cross-framework dataset, DLTrans, for translating deep learning code across dif- ferent frameworks. We develop multilingual modeling approaches for code translation and demonstrate their great potential in improving the translation quality of both low-resource and high-resource language pairs and boost- ing the training efficiency. We also propose a novel evaluation metric Debugging Success Rate@1 K for program-level code translation. Last but not least, we evaluate LIMM ChatGPT on our datasets and investigate its potential for fuzzy execution predictions. We build base- lines for Tode'raanOeean and analyze chal- lenges of code translation for guiding future re- search. The Code TransOcean datasets and code are publicly available https: : com/Weds www.aaaaa//eeeeeeanneeeeeee
large-scale
Code translation methods evolve from the inef- ficient, costly, and error-prone manual rewriting method to automatic methods. Automatic code translation methods can be categorized into compil- ers and transpilers, rule-based methods, and neu- ral network based methods. Neural models (Feng et al., 2020; Wang et al., 2021, 2023b) have be- come dominant in code translation. Details of code translation methods are presented in Appendix A. The performance of neural models relies heavily on large-scal high-quality parallel data. However, isting code translation datasets are limited by insuf- ficient coverage of programming languages and mostly focusing on single pair of popular pro- gramming languages, limited scale, and uneven data distribution. The widely used CodeTrans (Lu et al., 2021) is small dataset containing only Java- C# parallel data for quite short code samples. Other datascts (Ahmad et al., 2023; Rozière et al., 2020; Zhu et al., 2022b: Nguyen et al., 2013; Chen al., 2018) suffer from the same limitations. Conse- quently, existing code translation models (Feng et al., 2020; Wang et al., 2021; Ahmad et al., 2021) are confined to a narrow range of one-to-one code
"Fqual contribution. Work is supported by Speech Lab, Alibaba Group.

---

Category
LanguagoFramework
Dataset Name
Train/Dew/Test #Samples
Avg. #Tokens/Sample
Avg. Length
Python, C+,,,
Visual Basic, Go.
MultilnngullTaans
19.115/3.759/7.55
398 /21 491
1099 / 1135 1358
PHP. Java, C#
Swift, R, Rust,
Mulrilingual
Fortran, Ada, Perl.
NiansTrans
(65,457/23,500 447,002
292 /75/ 505
785 785/995/ 995 1372
COBOL, Lun,
Python, C. C++,
Visual Basic, PHP, Java, C#
LLMTTans
-/-/350
-7-1270
-/-1745
py PyTocc, TensorFlow, MXNet, Paddle
CrossFFrmework
DLTraas
282 36
/85
1318 /441 /884
Table I: Summary of our Codear We report #Samples, Avg. #Tokens/Sample and Avg. Length for Train/Dew/Test sets of each dataset. Note that LLMTrans only for testing. #Samples are on the program-level. #Tokens are based RoBERTa tokenizer (Liu et al., 2019). Length is the number of characters.
translation scenarios. Moreover, deep learning has been broadly used and achieved unprecedented suc- cess. However, there are barriers between different deep learning frameworks during the actual pro- duction process. Existing code translation datasets also neglect important demands from real-world applications, including modernizing early soft- ware systems developed in niche programming languages and migrating code across different deep learning frameworks.
DLTrans includes 408 samples covering four main- stream deep learning frameworks.
Multilingual modeling shows great potential in neural machine translation (Aharoni et al., 2019; Wang et al., 2020; Zhu et al., 2023), but it has not been systematically explored for code trans- lation. We investigate multilingual modeling for code translation using our MultilingualTrans NicheTrans, Trans, and DLTrans datasets. Experimen- tal results demonstrate that multilingual modeling significantly improves translation quality for both high-resource and low-resource language pairs and improves the model training efficiency.
To address these limitations and advance neu- ral code translation models, construct large- scale comprehensive multilingual code translation benchmark Code ceen summarized Ta- is an innovative benchmark
Recent research indicates that the proficiency of the LLM ChatGPT in natural language translation is on par with commercial-grann translation sys- tems (Jiao et al., 2023). To the best of our knowl- edge, our work the first to systematically in- vestigate the potential of ChatGPT in code trans- lation. We develop a fully automated translation- execution-evaluatio pipeline TutooTransExecuee to support this study. Note that match-based met- rics and execution-based metrics have been used for evaluating code translation methods, with de- tails in Appendix A. In order to accurately evalu- ate the usability of translated code from ChatGPT. we propose a novel execution based evaluation metric Debugging Success Rate @ @K (DSR @ K), which is the percentage of samples with translation results that successfully execute and produce the expected functionality after K debugging rounds. On our LLM'Trans dataset, the baseline ChatGPT setting achieves 48.57% DSR@0. We find that self-debugging and one-shot improve the perfor- mance while chain-of -ffthhtthttttttt strategies degrade the translation accuracy. Since our Transxxx
ble 1. that aims to provide a unified platform for eval- uating various models on a comprehensive set of code translation tasks that reflect real-world de- mands. Based on this goal, each dataset in Code- TransOcean is specifically designed to tackle key challenge in the field of code translation. Code- TransOcean includes three multilingual datasets, namely, the MultilingualTrans dataset (includ- ing eight popular programming languages), the RicheTraans dataset (translating between thirty- seven niche programming languages and the eight popular ones'), and specialized dataset LLM- Trans (including 350 data samples and their ex- ecuted results) evaluate executability of code translated by large language models (LLMss, and a cross-framework dataset DLTrans facilitating our proposed task for translating code between deep learning frameworks to enhance code reusability.
'We define popular and niche programming languages hased on the TIOBE Programming Community Index, which is a metric of the popularity of programming languages.

---

ecuter still cannot cover arbitrary programming languages, we also propose a novel metric fuzzy ex- ecution, attempting to address the limitations of ex- isting evaluation metrics for code translation. Our preliminary study using ChatGPT shows that Chat- GPT is still inadequate to predict fuzzy execution for any arbitrary programming language, which demands future research.
translation on code translation tasks Chen et al. (2018) explore this dataset from Nguyen et al. and also construct CoffesScript-Javaaeeeo dataset for investigating tree-to-tre to-tree neural models for code translation. Rozière et al. (2020) create dataset containing 852 programs to evaluate un- supervised methods. Recently, Zhu et al. (2022b) construct new translation dataset CoST from the GeeksForGeeks website?. Subsequently, they lease the translation dataset XLCoST (Zhu et al., 2022a), which expands the CoST dataset by 7.3 times. However, the limited language coverage of these datasets and their imbalanced data distri- bution hinder their practical applications. Rozière et al. (2022) construct the TransCoder-ST dataset to perform unsupervised code translation using auto- mated unit tests. Details of these datasets are sum- marized in Table 2. Rithy al. (2022) proposes code translation dataset XTest containing nine programming languages with unit tests, but is not open-sourced". Although CodeNet (Puri et al., 2021) comprises many problem statements and pro- vides corresponding solutions, experts have proven that about half of the CodeNet dataset has incorrect solutions (Zhu et al., 2022b), making it unsuitable for code translation tasks. With the limitations of existing code translation datasets, neural models trained on them may encounter overfitting, underfit- ting, and poor generalizability. Clearly, these issues impede the development of neural models for code translation. Therefore, constructing datasets that effectively address these problems is critical to en- hance performance of code translation algorithms.
parallel
Our contributions can be summarized as follows:
A large-scale multilingual code translation benchmark:
covers the largest number of popular and niche programming lan- guages far with the largest scale. It also in- cludes unprecedented dataset for translating code across different deep learning frameworks and a dataset and an automated pipeline for eval- uating LLMs on code translation. We establish baselines for all datasets in ConeTransOcean, Multilingual modeling for code translation: We are the first to systematically evaluate mul- tilingual modeling code translation for both high-resource and low-resource language pairs. Experimental results demonstrate that multilin- gual modeling significantly improves translation quality for both high-resource and low-resource language pairs and improves training efficiency. ChatGPT on code translation: We conduct the first comprehensive study of the potential of Chat- GPT on code translation, investigating efficacy of prompting strategies, hyperparameters, self- debugging. One-shot, and Chain-of-Thouglee New evaluation metrics: We propose DSR@ to evaluate translation and debugging capabilities of LLMs. We also propose a fuzzy execution metric based on LLMMs and conduct a preliminary study using ChatGPT on this metric.
Code Translation Methods and Evaluation Met- rics Details of code translation methods and evaluation metrics are presented in Appendix A.
2 Related Work
3 The Code TransOcnnnn Benchmark
Code Translation Datasets The success of neu- ral models for code translation relies heavily on large-scale high-quality parallel data. However, existing code translation datasets are plagued by issues such as insufficient coverage of program- ming languages, limited scale, and imbalanced data distribution. The widely used code trans- lation dataset CodeTrans (Lu et al., 2021) in the CodeXGLUE benchmark consists of Java-C# func- tion pairs. The small parallel corpus AVATAR (Ah- mad et al., 2023) is constructed for Java-Python code translation. Nguyen et al. (2013) construct a Java-C# dataset to explore statistical machine
In this section, we provide detailed descriptions and analyses of our CodeTransOcean benchmark, including the code translation tasks, their associ- ated datasets, and dataset statistics. Details of data collection methods and licensing information as well quality control and quality assessment are presented in Appendix A.2. Note that the vast
PI was not possible count specitic information about this dataset because it not released the public and were unable obtain response from the authors.
'In Table we report program-Ieve counts for the CoST dataset to facilitate a fair comparison with our own program-leve datascts.
"We tried to contact the authors but there was no response.

---

Dataset Source
Programming Languages
#Samples
Avg. #Tokens/Sample
Avg. Length
CodeTrrans (Lu al., 2021)
Java, C#
11,800
59/ 58
205 218 /22
Avatar (Ahmad et al., 2023)
Java, Python
9,517
239 / 235 / 234
691 / 687 688
Nguyen et al. al.(Nguyen al., 2013)
Java, C#
16,966
Lachaux al. (Rozièr et al., 2020)
C++, Java, Python
852
-/ 119 120
//313/ 311
C++, Java, Python, C#,
CoST (Zhu et al., 20226)
16,738
/ 199
770/455 450/458/ 551
Javascripe, PHP, C
TransCoder-ST (Rozière et al., 2022)
Java, C++, Python
437,030
C++, Java, Python, C#,, Javascript, PHP,
XLCoST (Zhu et al., 2022a)
122,151
234 232//22/// 222
644 / 634 606
Table Summary of existing code translation datasets. For #Samples, we report program-leve counts for Avatar, CoST, and XLCoST. Given that the original samples from other datasets are not organized at the program-level we report counts at the snippet-level for these datasets. Avg. # #Tokens/Sample and Avg. Length are counted in the same way as Table
majority of the samples in TranOOonn pro- vides explicit input and output, which is equivalent consists
gramming languages. We treat the other languages in the rankings as niche languages and construct the Niche' Trans dataset for translating between niche languages and popular languages. Additionally, in order to quantitatively evaluate the execution capabilities of the code generated by LLMs (c.g., ChatGPT, PaLM2 (Anil al., 2023)), we construct LLMTrans, which includes the execution results for a subset of MultilingullTrans and facilitates evaluating LLMs for multilingual code translation.
to unit tests. Overall, of 270,507 samples (over 200K unit tests), cov- ering 45 programming languages for multilingual code translation and 4 deep learning frameworks for foossfrra code translation Note that all samples in all CodeTransOccan datasets are constructed at the program-level. We ensure a balanced distribution of each languagee/ramework when constructing the datasets (Appendix A.2). There is no overlap between ConeTransOcean datasets and existing code translation datasets.
Multilingual/rans Dataset This dataset con- tains 30,419 program samples covering eight popu- lar programming languages, namely, C, C++, C#, Java, Python, Go, PHP, and Visual Basic. Table shows the statistics of each language pair. Note that XLCoST (Zhu et al., 2022a) is the only existing multilingual code translation dataset. Compared to XI. XL.CSTT MultilingualTrrns is advantageous in more balanced data distribution across various pro- gramming languages, practicality of language pairs, and data quality. For example, the real-world re- quirement for translating Java into JavaScript as in XLCoST is quite limited. As to data quality, our Multilinuull originates from program- ming chrestomathy website, with all data already reviewed and verified by the website.
3.1 Multilingual Code Translation
With the increasing need to unify the language variety when implementing system integration extensions with multilingual programming en- vironments, we construct the MultilingualTrrns dataset for multiple popular programming lan- guages Among programming languages in the rankings, we select the Top-10 10 languages as popu- lar ones except JavaScript and SQL" and construct the MultilingualTrans dataset based on the 8 pro-
*Code Translation also extends to conversions between different versions of the same language. e.g., Python 2 to Python 3. However, according to our survey, these translation tasks are quite straightforvard. Naive Copy methods, specitic translation tools. and tutorials (e.g.. Python 2 to 3 Conversion Guide) already achieve high translation accuracy. As result, no longer include these types of tasks in our benchmark. p
Niche Dataset The NicheTrans dataset contains 236,468 program samples, covering code translation pairs from thirty-seven niche program- ming languages, including Ada, COBOL, Pascal, Perl, Erlang, Fortran, Scala, Julia and others, to the eight popular ones. Table 12 shows statistics of each niche language. Although many studies have highlighted the practical necessity of code transla- tion for modernizing niche programming languages (Chen et al., 2018; Zhu et al., 2022b; Rozière et al.,
"We categorize languages as popular or niche based the TIORE Index Programming Language Rankings released April 2023 hteps:/ //ww. cibbe... com/ tiobe-indexs
'It is important to note that JavaScript and SQL, both within the top 10, are mainly used for front-end programming and database management respectively, signifying consider- able differences in their usage scenarios compared to the other 8 languages.

---

2020), our NicneTTrans dataset is the first dataset for code translation between these niche languages and popular ones. We believe this dataset will not only facilitate modernization of outdated programming languages more effectively, but also augment and evaluate generalizability of neural models.
and export based on agreed data formats convert only the final model through the compu- tation graphs. These technologies have obvious limitations. contrast, we propose a Cross-DL- framework Translation task for code migration between different DL frameworks through code translation (Appendix A.4). Compared to existing cross cross-framework adaptive technologies, Cross-DL- framework Translation achieves re-implementation under multiple DL frameworks through an auto- mated process, which not only generates highly readable code and enables secondary development, but also provides developers with flexibility on combining advantages of multiple frameworks.
LLMTrans Dataset The LLMTrans dataset aims provide benchmark for evaluating the perfor- mance of LLMs on code translation. The dataset translates seven popular programming languages to Python, totaling 350 program samples. We com- pile and test these samples and record the execu- tion results. Based on this dataset, we design and implement automated pipeline, AutoTransS:s TransExe- cuter automatically using LLMs to conduct code translation, execution, debugging, and calculating the success rate. This dataset and the automated pipeline ease investigation of the actual debugging success rate of LLMs on code translation and ef- fectively measure the practical usability of LLMs. Details of the LLM'Trans dataset are in Table 1.
DLTrans Dataset We construct the DLTrans dataset for Cross-DL-framework Translation, in- cluding four deep learning frameworks and span- ning twelve directions. To the best of our knowl- edge, our work is the first to define the cross-Di DL-- framework translation task and construct a corre- sponding dataset. We create two subsets of dif- ferent granularities based on the collected code, namely, coarse-grained at the program level and fine-grained the function class level. Each code pair comprises code that shares the same functionality but is written in different popular DL frameworks, including PyTorch, TensorFlow, MXNet, and Paddle. The coarse-grained and fine- grained datasets have 408 and 3,270 samples, re- spectively. In this work, we only experiment on the coarse-grained subset.
3.2 Cross-framework Code Translation
Cross-Deep-Learning Saaaniinng-raawwrrkk Translation Task The widespread applications of deep learn- ing (DL) has spawned emergence of various DL frameworks, such as PyTorch, TensorFlow, MXNet, and Paddle. However, there significant differ- ences in syntax and dependency libraries between different frameworks, severely impeding reusabil- ity of projects". Moreover, studies illustrate signif- icant disparities in energy consumption and eco- nomic costs during training and inference between various frameworks (Georgiou et al., 2022). Se- lecting appropriate DL framework for green AI has become paramount in els (Ananthaswamy, 2023). Code reusability and onergy-economin efficiency in DL have emerged as critical considerations for both research and practical engineering implementation. Convert- ing code between different DL frameworks is chal- lenging, mainly due to differencee between frame- works, code complexity, structural inconsistencies. and cross-platform compatibility (more details are in Appendix A.3). Existing Mooss-DL-framewor adaptive technologies such as the ONNX 10 model conversion protocol require both parties import
Experiments
We present experiments of multilingual training for code translation (Section 4.1). We then intro- duce a novel evaluation metric Debugging Success Rate @ K for program-level code translation (Sec- tion 4.2) and the first comprehensive exploration of ChatGPT for code translation (Section 4.3).
of large mod-
4.1 Multilingual Modeling
Multilingual modeling has been pivotal in broad- ening the applicability of neural machine transla- tion (Aharoni et al., 2019; Wang et al., 2020; Zhu et al., 2023; Johnson et al., 2017). This is pri- marily evidenced in enhancing the performance of low-resource languages and cross-language trans- fer learning (Mohammadshahi et al., 2022; Zoph et al., 2016; Nguyen and Chiang, 2017: Johnson et al., 2017). CodeTransOcenn covers nearly fifty
TrueettrnsEeeeutter only supports translation from any source language Python. We discuss it Limitations. "https:/ http:////wwwwwwwb/bbb./// assemblyai com/blog/ pytorch-vey vs- tensor flow- innwwiin-- 2
"https: /oonx.

---

Average
One-to-Ome (baseline)
Many-to-One
Many-to-Many
One-to-Many
High-resnunce
4.68
5.56 (1 0.88)
5.94 (f 1.26)
6.18 (f 1.50)
Low-resource
4.83
4.85 (1 0.02)
4,95 (1 0.12)
5.84 1.01)
All
5.19
5.31 (1 0.12)
5.81 (1 0.62)
6.42 (f 1.23)
Table 3: Average BLEU scores of the four multilingual modeling strategies, One-to-One, Many-to-One, Many-to- Many, and One-to-Many, for All language pairs, High-resource language pairs, and Low-resource language pairs.
programming languages and deep learning frame- works. We use its datasets to explore multilingual modeling on code translation tasks.
language features, thereby improving generalizabil- ity of the model. (2) Many-to-Many can be consid- ered expanding the One-to-Me Mo-Many strategy by em- ploying greater volume of non-source language data for training. Since the encoder must be attuned to the features of various languages simultaneously under Many-to-Many parameter sharing may po- tentially undermine the performance. (3) Many- to-One helps the model to learn from a broader range of data than the baseline. Specific patterns or expressions in diverse source languages assist the model in more precisely comprehending how to translate into the target language. The shared semantic representations across different source languages allow the model to implement effective transfer learning strategies. Furthermore, increase in training samples enables the model to optimize the loss function more stably. These results are consistent with previous findings on multilingual modeling for natural language translation (Aharoni et al., 2019): Many-to-Many models, trained across multiple target languages instead of just one target language, can function effectively as a regulariza- tion strategy for Many-to- thereby reducing the possibility of over-matching.
Experimental Setups In this work. we use pre- trained CodeTT++ (Wang et al., 2023b)'l as the backbone based on its superior performance on code understanding and generation evaluations re- ported in (Wang et al., 2023b). We use the Multilin- gual dataset to investigate four multilingual modeling strategies based on data sharing in the source or target language or both, namely, One-to- One, One-to-Many, Many-to-One, and Many-to- Many, with One-to-One as the baseline. Details of the four strategies are in Appendix A.5. To un- derstand the strengths and weaknesses of the four strategies, compare their average performance on all language pairs and focus on low-resource and high-resource pairs. Since the CodeBLEU metric (Ren al., 2020) does not cover all eight languages in MultilingualTran, we use BLEU to measure translation accuracy for the four strategies. Then, we establish baselines for the DLTrans and NicheTrrans datasets.
We rank the resource richness of the eight pro- gramming languages in MultilingualTrans in de- scending order based on their amounts in the CodeT5+ pre-training data, as Java, PHP, C, C#, Python, C++, and Go (Visual Basic is not covered by the Co pre-training data). Based on this ranking, we consider Visual Basic, C++, and Go as low-resource languages and Java, PHP and C as high-resource languages.
For High-resource and Low-resource languages, as shown in Table 3, the ranking of the four strate- gies the same as for All, but there notable difference in their adaptability across languages of varying resource scales. High-resource lan- guages can take advantage more effectively from the shared information across multiple source lan- guages; whereas. low-resource languages are rela- tively less equipped to handle the additional uncer- tainty and noise introduced by shared parameters, and thus often have to rely on a larger volume of source language data to optimize their benefits.
Results and Analysis Detailed experimental re- sults are shown in Table 14 in Appendix. For All language pairs, the performance of the four strate- gies is ranked as One-to-Many > Many-to-Many Many-to-One > One-to-One. (1) Under One-to- Many strategy, the model encoder can provide more comprehensive information for source language translation due to its ability to absorb more source
Results from the Many-to-Man strategy on DL.- Trans and NicheTrans datasets are shown in Ta- bles 4 and 5. The experimental results suggest that significant improvements in translation accuracy can be achieved by swapping the source and tar- get languages in the training set to facilitate data augmentation and training a bidirectional model.
Il We will conduct evaluations of a broader selection of mod- els on our datasets in future work, including a. (Touvron ct al., 2023), WizardCoder (Luo ct al., 2023), etc.

---

Metbod
PyTarch
TeunrEles
MXNc
Public
CodeBLEU
BLEU
CodeBLEU
EM
BLEU
CodeBLEU
EM
BLEU
CodeBLEU
PyTrech
72.77
TewedFlow
27.27
20.09
63.79
57.9
27.27
63.04
65.81
MXNe
18
72.80
74.15
29.08
63.58
06.06
28.15
69.89
71,08
Publi:
30.91
80.25
94.83
27:27
62.94
67.78
28.15
00.43
75.09
PyTrehh
354551091
71.163077
70.5200755
42.73:2.41
81.76.0045
N2.5260.06
43.441158
k3.7660.00
K507210.74
ChdrTS+
Tenontfloe low
MDNNct
32,12:2.29
1.36
77.796013
71.9710.66
76.4360.14
71 08:1777
31
62.2291339
67.621227
36,07: 188
71.77-061
73,01:0118
29,990091
29.701265
74.2660.46
0036:1078
73.27:0.422
06 76:10.332
Puiklc:
43,08e4.10
86.2540.66
86.0910.36
29.39=2.93
29
60 69.35:0.77
69.7=0051
35,75=0.3
7666==022
794660.3
Table Results on DLTrans of Naive and CodeTT++ 220M with Many-to-Many strategy, We run each experiment with 3 random seeds and report the mean and standard deviation of EM, BLEU, and CodeBLEUU scores.
BLEU
Naive
Two-way
One-way
Meanwhile, recent research shows that LLMs such as ChatGPT demonstrate preliminary code debug- ging capabilities (Chen et al., 2023b,a). Hence, we propose a novel and robust evaluation metric for LLM on code translation, Debugging Success Rate@ K (DSR @ K), by measuring whether the translated code can be compiled and executed with the same behavior the input source code, with K rounds of debugging. To the best of our knowl- edge, DSR@ @K the first metric designed to aceu- rately reflect real-world software development scenarios.
Many-to-C
2.36
4.60
4.86
Many-to-C#
2.53
4.48
3.82
Many-to-C++
1.99
4.78
3.32
Many-to-Go
3.11
5.24
3.19
Many-to-Java
3.18
5.23
5.34
Many-to-PHP
4.37
2.46
1.98
Many-to-Python
2.87
2.38
1.67
Mony-to-VB
1.69
2.17
1.97
Average
2.76
3.92
3.27
Table 5: BLEU scores NicheTrans of Naive and CodeT5+_ 2200 with Many-to-Many strategy. One- way denotes training models only from niche to popular, while Two-way denotes training in both directions.
DSR@K K is the percentage of the samples that successfully execute and produce the expected re- sults among all samples. Each sample is given K generation and debugging attempts by LLM. If the generated code successfully executes and pro- duces the expected results with these K rounds, the sample marked as successful. DSR@ is computed as 1 N Σ S(i, K), where N denotes the total number of samples. If the 7th code sam- plc succeeds within K attempts, then S(i, = I; otherwise, S(i, K) = O. Note that DSR @0 can be used for program-level code translation evaluation for any models. In this work, we employ DSR @K to evaluate the ability of LLMs such as ChatGPT for debugging code and translating code with de- bugging results.
Notably, prior studies multilingual neural ma- chine translation often overlook the comparison between One-to-Many and other strategies. Nev- ertheless, One-to-Many demonstrates superiority over the One-to-One baseline across all our exper- iments. Overall, our results strongly recommend a targeted multilingual modeling strategy for code translation, as it not only can translate multiple lan- guage pairs with single model, but also achieves better and more stable accuracy than baselines.
4.2 Debugging Success @ate@K
For evaluations, we adopt existing code translation evaluation metrics in our experiments, including Exact Match (EM), BLEU, and CodeBLEU (de- tails are in Appendix A.1.2). However, all these metries are based on surface-form matching (or with some adaptations as for CodeBLEEU) and are not suitable for our program-level translation tasks since they cannot reliably evaluate functional cor- rectness of translated code. Moreover, in real-world software development scenarios, developers typi- cally ensure the functionality of code by testing and debugging upon completion, rather than writ- ing and testing multiple versions of the code to achieve the expected functionality as measured by the existing pass@k @k (Kulal et al., 2019) metric.
4.3 ChatGPT for Code Translation
The recent LLM ChatGPT demonstrates competi- tive performance on language generation tasks such as summarization and machine translation (Yang et al., 2023; Peng et al., 2023; Gao et al., 2023). However, ChatGPT for code translation has not been systematically explored. We study the effec- tiveness and potential of ChatGPT on code transla- tion and investigate strategies to improve its perfor- mance. We use @ K as the principal evalua- tion metric since we focus on the practical usability of ChatGPT. We use the ChatGPT API and gpt- 3.5-turbo as the default model and evaluate on the

---

LLMTrans dataset for all experiments. We inves- tigate the efficacy of prompts and hyperparameters and context in zero-shot setting, then compare one- shot versus zero-shot and study Chain-of- Thouuht.
Strategy
Expt Anum
EM
BLEU
CodeBLEU
DSRE
Baseline
10.83
48.57%
6.00
11.06
24.36
4.5.4.%
24.48
Role assignments
0.00
10.70
24,08
41.71%
0.00
10.73
24.08
40.36%
Polite inquiry Chiiiyy usage
10.85
24.37
47.71%
Effect of Prompts and Hyperparameters Prior works show that prompts can influence the perfor- mance of ChatGPT (Zhong al., 2023; Peng et al., 2023; Jiao et al., 2023). We set initial prompt "Translate [SL] to [TL] [SC] the baseline, where [SL] and [TL] denote the source language and the target language respectively and [SC] denotes the source code. We also add "Do not return anything other than the translated code." for each prompting strategy to require ChatGPT to return only code in order to ease code execution. We design three prompt variants. Details of the experimental settings and prompt variants are in Appendix A.6. We also investigate the effect of hyperparameters on code translation performance. As shown in Table 6, implementing role as- signments, clarifying usage, and polite inquiry in prompts all degrade the performance compared to the baseline prompt. These results show that the baseline with the most straightforward prompt pro- duces the best performance, possibly because provides clear, short, and unambiguous instructions for the task to the model. More intricate prompting strategies may introduce noise and confuse Chat- GPT. The performance of polite inquiry prompt is comparable to but still worse than the baseline performance. We speculate that the improvement from polite inquiries in prior studies (Akin, 2023) may stem from their explicit and comprehensive formulations which make it easier for the model to understand the task requirements. Wc also observe in Table 6 that same as prior findings, BLEU and Cou have no obvious positive correlations with the debugging success rate (DSR @0). Since the reference target code exhibits the same function- ality the source language code but their execu- tion results could differ slightly, EM also does not correlate with DSR @. Therefore, in subsequent experiments, we only report DSR @0, We also eval- uate the CodeT5+ 220M model on LLMTrans with the Many-to-Many strategy and find that DSR @+ is 0, suggesting that CodeT5+ M20M Zero-shot is unable to generate executable translation results.
0.86
10.87
24,26
47.71%
0.29
10.63
24.11
34.00%
Divide and Conquer
0.00
7.44
25,30
22.86%
Table 6: Zero-shot performance of ChatGPT with differ- prompt variants and contextual strategies. Baseline denotes ChatGPT with the baseline prompt. Details of the prompt variants (Expt #num) are Appendix A.6.
Rthh Debug DSR
K Debug
DSR
48.57%
2
52.29%
51.43%
3
52.57% %
Table 7: ChatGPT performance at the 1 debugging.
lary considered during generation. Higher temper- ature top_p could increase diversity in the gen- erated results from ChatGPT. However, as shown in Table 16 in Appendix, independently varying temperature or top_p does not notably change the performance of ChatGPT; hence for the other Chat- GPT experiments, we set both temperature and top_p as to ensure stability an reproducibility.
Effect of Context We explore a Divide-and- Conquer strategy, which segments the source lan- guage code into snippets (c.g., functions and sub- functions), translate each snippet independently, then merge their outputs as the final result. As shown in Table 6, Divide-and-Conquer signifi- cantly degrades the performance. We hypothe- size that lack of the global context in Divide-and- Conquer could prevent ChatGPT from considering the overall structure and variable configurations of the code for translation.
Effect of Self-debugging Since ChatGPT has shown preliminary capability in error detection and correction during code generation (Shinn et al., 2023; Chen et al., 2023b; Kim et al., 2023; Nair et al., 2023; Madaan et al., 2023), we use Chat- GPT to perform multiple rounds of self-debugging and investigate the impact on DSR. Specifically, ChatGPT first translates the souree language code into the target language (which is Python as in our Exxeuue)s)x and then attempts execute the translated code. If the execution passes and executing the translated code exhibits the same functionality as the source code, is regarded as
ChatGPT selects the token with the highest prob- ability during generation. The hyperparameter tem- perature influences the randomness of the gener- ated text, while top_p controls the range vocabu-

---

successful execution. Otherwise, feedback from the compiler will be also fed ChatGPT for the next round of translation, and this process is re- peated until reaching a pre-defined number K of debugging rounds. The whole process shown in Table 17 in Appendix. As shown in Table 7, DSR improves significantly with multiple rounds of self- debugging. The first self-debugging improves DSR by 3% absolutely. Each subsequent round of self- debugging brings further gain but DSR begins to plateau after the second debugging round. This suggests that ChatGPT has limitations in its capac- ity to rectify errors after multiple debugging cycles, which is consistent with human behaviors.
Strategy
Expts Anum
DSR
Strategy
Expts fnum
DSR@0
Baseline
48.57%
46.00%
49.71%
CUT
42.57%
One-shor
50.29%
48.29%
48.86%
45.43%
Table 8: Performance ChatGPT with One-shot and CoT strategies compared to the Zero-shot Baseline. De- tails of Expt #num are in Appendix A.7 and A.8.
descriptions and translation prompts for each strat- egy are in Appendix A.8. As shown in Table 8, CoT degrades executability of the translated code. In Experiment #2, DSR @1 even declines by 6% absolutely. We study the translation results of Chat- GPT and find that when CoT strategies are applied, the model tends to translate the source code line by line, neglecting compatibility issues between libraries and functions in different languages. CoT also compromises the global planning ability of the model. These observations are consistent with the findings in (Peng et al., 2023) that CoT may lead to word-by-word translations of natural language, thereby degrading the translation quality.
Effect of One-shot In-context learning (Brown et al., 2020) allows the model to learn from input examples, enabling it to understand and manage each new task. This method has been validated as an effective strategy for enhancing the performance of model inference (Peng et al., 2023; Liu et al., 2023a). Therefore, we explore one-shot learning for ChatGPT on code translation. We investigate three one-shot learning sample selection strategies. Descriptions the strategies and the correspond- ing prompts are in Appendix A.7.
F Fuzzy Execution To address the limitations of existing evaluation metrics and our AutoTransEx: ecuter, we propose another novel code translation evaluation metric fuzzy execution using LLMs in Section Limitations, inspired by recent progress in using LLMs as evaluation metrics for NLP tasks. Our preliminary studies evaluates the performance of ChatGPT for predicting whether given code can be executed or not, and if executable, also for predicting the executed output. Experimental sults show that using ChatGPT for fuzzy execution is not yet practical and demands future research.
Table 8 shows that all three One-shot learning strategies effectively improve DSR @0 of ChatGPT over the Zero-shot baseline. The Experiment#2 strategy (provided contextual example has both same source and target languages as the origi- nal task) achieves the best performance, yielding 1.72% absolute gain in DSR @0, with Experiment #1 (example has the same target language but dif- ferent source language) and #3 (example has differ- ent source and target languages) following closely with 1.14% and 0.29% absolute gains, respectively. These results show that One-shot learning entirely tailored to the translation requirements is most ef- fective in boosting code translation performance for ChatGPT. The results corroborate previous findings in natural language translation (Peng et al., 2023) that the performance of ChatGPT is sensitive to the provided contextual example in One-shot learning.
5 Conclusion
We construct CodeTransOcean, a comprehensive code translation benchmark that includes multilin- gual and cross-framework datasets. We demon- strate that multilingual modeling has remarkable potential in enhancing code translation quality. We also reveal the superior code translation capability of ChatGPT and advanced strategies lead to signif- icant performance gains. Moreover, introduce fuzzy execution that may overcome limitations of existing metrics but requires future research. summary, we provide a comprehensive suite re- sources, tools, and baselines for code translation.
Effect of Chain-of-Thought Chain-of-Thought (CoT) allows the model to simulate an orderly and the think-
structured way of thinking by sorting ing process. It helps guide the model to output the final answer step by step (Wei al., 2022; Peng et al., 2023; Kojima et al., 2022). For code transla- tion, we investigate four CoT strategies. Detailed

---

6 Limitations
Zero-Shot
Few-Shot
PP
EN
TP
FP
FN
TP
Existing match-based evaluation metrics for code translation (Papineni al., 2002; Ren al., 2020; Eghbali and Pradel, 2022: Zhou et al., 2023; Tran al., 2019) focus solely on semantics, overlooking executability of the code and the functional equiva- lence under different implementations. Execution- based metrics (Kulal et al., 2019; Hao et al., 2022; Hendrycks et al., 2021; Rozière et al., 2020; Dong et al., 2023) that require providing test cases are expensive to conduct in practice, and the significant overhead of executing numerous test cases and the heightened security risks during the execution pro- cess remain unresolved. is crucial to establish an evaluation metric that overcomes these limitations.
292
8
238
62
294
4
242
58
12
x280
14
x
Table 9: Confusion matrix of fuzzy execution prediction by ChatGPT with Zero-shot and Few-shot settings.
Metrics
Calculation formula
Zero-Shot
Few-Shot
TP+ TN + TN + + TP
Accuracy
59.00%
59.00%
Precision
88.57%
93.55%
TP + FP TP PN Precision Recall
Recall
20.67%
19.33%
FI scores
2
33.52%
32.04%
Our proposed DSR @ K and the automated Auto- TransExecuter aim to measure the executability of the code and reflect the real-world software devel- opment scenarios. However, Auto Transtxxcuterr currently only supports Python as the target lan- guage. This mainly due to the fact that different programming languages necessitate distinet run- time environments and libraries, making it particu- larly challenging to automatically detect and install the required dependencies for each code. While certain existing tools, such as Dynatrace can carry out dependency detection, the range of sup- ported programming languages remains limited. Moreover, the configuration methods for compilers vary substantially among different programming languages, which further complicates automated configuration. In addition, fully automated execu- tion systems could be exploited by malicious code, thus necessitating further security measures. There- fore, achieving this goal requires overcoming many technical and practical difficulties.
Precision + Recall
Table 10: Performance of ChatGPT on predicting fuzzy execution.
Each entry in this dataset includes the execution status and, if executable, the result of the execution. We use ExecuteStatus and AutoTraasExxecute to evaluate the performance of ChatGPT for predict- ing whether given code can be executed not, and if executable, also predict the executed output. The Zero-shot prompts are shown in Table 18 in Appendix. For the Few-shot strategy. in addition to the Zero-shot baseline, we include an example executable code and an example of non-executable code, as detailed in Table 18.
We define fuzzy execution as first testing the consistency between the actual pass rate and the predicted pass rate ChatGPT, followed by fur- ther testing the accuracy in predicting execution results using ChatGPT without relying a com- piler. Since we are interested in the ability of Chat- GPT identify samples that cannot actually be executed accurately, we present the confusion ma- trix in Table 9 based on the results. To evaluate the performance of ChatGPT on the fuzzy execution prediction task, we use the standard accuracy, pre- cision, recall, and F1 scores. Experimental results based on these evaluation metrics are in Table 10. The low accuracy, recall and Fl scores show that ChatGPT still has difficulty in identifying emrors in the code, exhibiting about an 88% tendency to pre- dict that the code is executable. Overall, ChatGPT has low accuracy in the binary classification task of "whether it can be executed". and its ability to pre- dict execution results, being at a scant 4%, clearly
To address limitations of existing evaluation met- rics and limitations of Exxeerrrsex we pro- pose another novel code translation evaluation met- ric fuzzy execution.
Recent studies have begun to utilize LLMs as evaluation metrics in the field NLP (Chen et al., 2023c; Wang et al., 2023a; Fu et al., 2023; Koemi and Federmann, 2023; Ji et al., 2023). Inspired by these works, we create a new dataset ExecuteSta- tus by randomly selecting 300 executable samples from MultilingualTram and 300 non-executable samples from the translation results of ChatGPT.
hhtt::://ww.yynatrac.ea p plat fott/o// l ligence/ Sependency-detectioon

---

requires further enhancement. Thus, using Chat- GPT for fuzzy execution is not yet practical (Liu et al., 2023b). Despite this, fuzzy execution with LLMs holds the potential to overcome the deficien- cies of current code translation evaluation metrics. We will continue this exploration in future work.
et al. 2023. Palm 2 technical report. CORR. abs/2305.10403
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners In Ad- vances in Neural Information Processing Syxtems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
References
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 3874- 3884, Association for Computational Linguisties.
Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T. Devanbu. and Baishakhi Ray. 2022. Natgen: generative pre-training by "naturalizing" source code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Sym- posium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 18, 2022, pages 18 10-30.44 ACM.
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, NAACL- HLT 2021, Online, June 6-11, 2021, pages 2655 2668, Association for Computational Linguistics.
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bow- man, Kyunghyun Cho, and Ethan Perez. 2023a. Im- proving code generation by training with natural lan- guage feedback. CoRR, abs/2303.16749.
Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai- Wei Chang. 2023. AVATAR: A parallel corpus for java- java-python program translation. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 2281.. Association for Computa- tional Linguistics.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023b. Teaching large language mod- cls to self-debug. CoRR, abs/2304.05122
Fatih Kadir Akm. 2023. swesome-chatept-prompp
Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree- to-tree neural networks for program translation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 2552-2562.
Badr AlKhamissi, Siddharth Verma, Ping Yu, Zhijing Jin, Asli Celikyilmaz, and Mona T. Diab. 2023. OPT- R: exploring the role of explanations in finetuning and prompting for reasoning skills of large language models. CoRR, abs/2305. 12001.
Anil Ananthaswamy, 2023, In ai, is bigger always bet- ter? Nature.
Yi Chen, Rui Wang. Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023c. Exploring the use of large lan- guage models for reference-fre text quality eval- uation: A preliminary empirical study. CoRR. abs/2304.00723.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin. Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meisr-Helllseen, Gau- rav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay. Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego. Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Chov Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxi- aoyu Feng, Vlad Fienber. Markus Freitag. Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and
Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin. 2023. Codescore: Evaluating code generation by learning code execution. CoRR. abs/2301.099 .00043.
Aryaz Eghbali and Michael Pradel. 2022. Crystalbleu: Precisely and efficiently measuring the similarity of code. In 37th IEEE/ACM International Confer- ence on Automated Software Engineering, ASE. 2022, Rochester, MI, USA, October 10-14, 2022, pages 28:1-28:12 ACM.
Zhangyin Feng. Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin,

---

Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- bert: A ure-trained model for programming and nat- ural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL., pages 1536- 1547. Association for Computational L Linguistics.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. CoRR, abs/2303. 17491.
Tom Koemi and Christian Federmann. 2023. Large lan- guage models are state-of-the-art evaluators of trans- lation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023, Tampere, Finland, 12-15 June 2023, pages 193-203. European Association for Machine Translation.
Jinlan Fu, See- See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. CoRR, abs/2302.04166.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human- like summarization evaluation with chatgpt. CoRR, 00552234..0254..
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In NeurlPS.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. Spoc: Search-based pseudocode to code. In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pages 11883-11894,
Stefanos Georgiou, Maria Kechagia, Tushar Sharma, Federica Sarro, and Ying Zou. 2022. Green AI: do
deep learning frameworks have different costs? In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pirtsburgh, PA, USA, May 25-27, 2022. pages 1082-1094, ACM.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svy- atkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng. Colin B. Clement, Dawn Drain, Neel Sundare- san, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graphcodebert: Pre-training code representations with data flow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvveRevie..net.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023a. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., (5(9):195:1 19::1119::::13
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907) 11692.
Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, He Zong, Siyuan Jiang, Yang Liu, and He Wei, 2022. Aixbench: A code generation benchmark dataset. CoRR, abs/2206.1311
Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, and Hinrich Schütze. 2023b. Evaluate what you can't 't evaluate: Unassessable generated responses quality. CoRR, abs/2305. 1bs/2800..4668.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. 1. Measuring coding challenge com- petence with APPS. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrlPS Dalasets and Bench- marks 2021, December 2021, virtual.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Li- dong Zhou, Linjun Shou, Long Zhou, Michele Tu- fano, Ming Gong, Ming Zhou, Nan Duan, Neel Sun- daresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. Codexglue: A machine learning bench- mark dataset for code understanding and generation. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks 1, NeuriPS Datasets and Benchmarks 2021, December 1021, virtual.
Yunjie Ji. Yan Gong, Yiping Peng, Chao Ni, Peiyan Sun, Dongyu Pan, Baochang Ma, and Xiangang Li. 2023. Exploring chatgpt's 's ability to rank content: A prelim- inary study consistency with human preferences. CoRR, abs/2303.076111
Wenxiang Jiao, Wenxuan Wang. Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt good translator? yes with gpt-4 as the engine.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xi- ubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol- instruct. CoRR, abs/2306.08566 08568.
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Tho- rat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation sys- tem: Enabling zero-shot translation. Trans. Assoc. Comput. Linguistics, 5:339-351.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter

---

Clark. 2023. Self-refine: Iterative refinement with self-feedback. CoRR, abs/2200.116511.
for automatic evaluation of code synthesis. CORR, abs/2009.10297
Alireza Mohammadshahi, Vassilina Nikoulina, Alexan- dre Berard, Caroline Brun, James Henderson, and Laurent Besacier. 2022. Small-100 Introducing shal- low multilingual machine translation model for low- resource languages. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages Association for Computational Linguistics.
Israt Jahan Rithy, Hasib Hossain Shakil, Niloy Mondal, Fatema Sultana, and Faisal Muhammad Shah. 2022. Xtest: A parallel multilingual corpus with test cases for code translation and its evaluation*. R In 2022 25th International Conference on Computer and Informa- tion Technology (ICCIT), pages 623-628.
Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanus- sot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. In Advances in Neural Information Processing Systems 33: An- nual Conference Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual.
Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan. 2023. DERA; enhancing large lan- guage model completions with dialog--nabledd resolv- ing agents. CoRR, 17077
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N.
Baptiste Rozière, Jie Zhang, François Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2022. Leveraging automated unit tests for unsuper- vised code translation. In The Tenth International Conference on Learning Representations. ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Nguyen. 2013. Lexical statistical machine transla- tion for language migration. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE' 13, Saint Peters- burg, Russian Federation, August 18-26. 26, 2013, pages 651-654. ACM.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27- December 1, 2017, Volume 2: Short Pupers, pages 296-301, Asian Federation Natural Language Processing.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Ngoc M. Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien N. Nguyen. 2019. Does BLEU score work for code migration? In Proceedings of the 27th International Conference on Program Comprehen- sion, ICPC 2019, Montreal, QC, Canada, May 25-31, 2019, pages 165-176. IEEE/ ACM.
Alfrick Opidi. 2020. Why your legacy software is hard to maintain and what to do about it.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6- 1-1 2002, Philadelphia. PA, USA, pages 31 311-318. ACL.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxi- ang Shi, Zhixu Li. Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is chatgpt good NLG evaluator? A preliminary study. CoRR, abs/2303.044 0400448
Kegin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. CoRR, 133/0.
Yiren Wang, ChengXiang Zhai, and Hany Hassan. 2020. Multi-task learning for multilingual neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2020, Online, November 16-20, 2020, pages 1022-1034. Association for Computational Linguistics.
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, and Ulrich Finkler. 2021. Project codenet: A large- scale AT for code dataset for learning a diversity of coding tasks. CoRR, abs/2105.12655.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li. and Steven C. H. Hoi. 2023b. Codet5+: Open code large language mod- els for code understanding and generation. CoRR, 0et.2330507922..
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method
Yue Wang. Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi 2021. Codet5: Identifier-aware unified

---

pre-trained mncoer-ddccodr models for code under- standing and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, Virtall Event / Punta Cana, Dominican Republic, 7-11 November, 2021. pages 8696-8708, Association for Computa- tional Linguisties.
Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 MMarch I, 2022, pages 11783-117999 11790. AAAI Press.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong. Jiajun Chen, Lei Li, and Shujian Huang. 2023. Multilingual machine translation with large language models: Empirical results and analy- sis. CoRR, 0bo//334400465...
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian lchter. Fei Xia, Ed H. Chi. Quoe V. Le. and Denny Zhou. 2022. Chain-of-thouge prompt- ing clicits reasoning in large language models. In NeurlPS.
Barret Zoph. Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for now-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1568-1575. The Association for Computational Linguistics.
Justin D. Weisz, Michael J. Muller, Stephamic Houde. John T. Richards, Steven Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection not required? human-ai partnerships in code translation. IUI '21: 26th International Con- ference on Intelligent User Interfaces, College Sta- tion, TX, USA, April 13-17, 2021, pages 402-412. ACM.
A Appendix
A.1 Related Work
Justin D. Weisz, Michael J. Muller, Steven I. Ross, Fer- nando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, and John T. Richards, 2022. Better together? an evaluation of ai-supported code translation. In IUI 2022: 27th International Confer- ence on Intelligent User Interfaces, Helsinki, Finland, March 22- 25, 2022, pages 369-391. ACM.
A.1.1 Code Translation Methods
Naive Copy directly duplicates the source code the target code without making any modifications. Given that the results produced by this method are often unusable, it is treated as the lower bound of performance for code translation. Early code trans- lation relies heavily on manual rewriting, which requires developers to have a deep understanding of both source and target languages along with the ability to navigate various complex programming structures and semantic challenges. This method is inefficient, costly, and prone to errors.
Ning Wu, Ming Gong. Linjun Shou, Shining Liang, and Daxin Jiang. 2023. Large language models are di- verse role-players for summarization evaluation. In Natural Language Processing and Chinese Comput- ing- 12th National CCF Conference, NLPCC 2023, Foshan, China, October 12-15, 2023, Proceedings, Part I, volume 14302 of Lecture Notes in Computer Science, pages 695-707. 6priner..
Automatic code translation methods fall into sev- eral categories. Compilers and transpilers 13 can automatically translate the source code into a tar- get language, significantly saving time and effort. However, these methods cannot fully preserve all the linguistic features and behaviors of the source code, nor can they comprehend the intent and mantics inherent to the source code as humans do. Rule-based methods (Weisz et al., 2021, 2022; Rozière et al., 2020) treat the code translation task as a program synthesis problem. They define a set of transformation rules and employ the rules or pattern matching for code translation. Research on rule-based methods is quite scarce, mainly because they overly rely the completeness of the rules and also require considerable amount of manual preprocessing.
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. Harnessing the power of Ilms in practice: survey on chatgpt and beyond. CoRR, abs/2304.13712.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du and Dacheng Tao. 2023. Can chatgpt understand too? comparative study on chatgpt and fine-tuned BERT. CoRR, abs/2302. 10198.
Shuyan Zhou, Uri Alon, Sumit Agarwal, and Gra- ham Neubig. 2023. Codebertscore: Evaluating code generation with pretrained models of code. CoRR, abs/2302.05527.
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravin- dran, Sindhu Tipimeni. and Chandan K. Reddy. 2022a. Xlcost: A benchmark dataset for cross- lingual code intelligence. CoRR, abs/2206.08474.
Neural network based methods have become dominant in the field of code translation in recent
Ming Zhu, Karthik Suresh, and Chandan K. Reddy. 2022b. Multilingual code snippets training for pro- gram translation. In Thirty-Sixth AAAI Conference on Artificial Intelligence AAAI 2022, Thirty
Is Ptttps://en baon.wike wiki ped! orgg/wkk / Source- to- www.rc_euuuuullllleeeee

---

years. These methods mainly treat code transla- tion as sequence-to-sequenee generation prob- lem. Among them, Chen et al. (Chen et al., 2018) are the first to successfully apply neural networks to code translation, designing a tree-to-tree neu- ral model. CodeBERT (Feng et al., 2020) sig- nificantly improves code translation accuracy by pretraining models with masked language mod- eling and replaced token detection. GraphCode- BERT (Guo et al., 2021) further improves code translation aceuracy by introducing two additional pre-training tasks as edge prediction and node alignment. CodeT5 (Wang et al., 2021), based on the Transformer encoder-decoder architecture, achieves excellent performance on code translation through four pre-training tasks, namely, masked span prediction, identifier tagging. masked identi- fier prediction, and bimodal dual generation. With a similar architecture as CodeT5, PLBART (Ah- mad et al., 2021) adopts three tasks of token mask- ing, token deletion and token infilling for denois- ing seq2seq pre-training, which enables PLBART to infer language syntax and semantics and to learn how to generate language coherently. Nat- Gen (Chakraborty et al., 2022) forces the model to learn capture intent of the source code by setting tasks during pre-training. and forces the model to make the generated code closer to the human-written style.
ation Understudy (BLEU) (Papineni et al., 2002) metric evaluates the similarity between the transla- tion output and the reference translation by multi- plying the geometric average of n-gram precision scores with a brevity penalty. The CodeBLEU (Ren et al., 2020) metric extends BLEU by consider- ing syntactic and semantic characteristics of pro- gramming languages; it not only considers shal- low matching but also pays attention to syntactic and semantic matching. CrystalBLEU (Eghbali and Pradel, 2022) focuses more on the inherent differences between source code and natural lan- guage, such as trivial shared n n-gram syntax. Code- BERTScore (Zhou et al., 2023) uses pre-trained models to encode the translation output and ref- erence translation, then calculates the dot product similarity between them, enabling comparisons of code pairs with distinct lexical forms. However, CodeBLEU, CrystalBLEU, and CodeBERTScore have limitations as they only support a limited range of programming languages and cannot be used in general multilingual scenarios. Ruby (Tran et al., 2019), new method for evaluating code translation, considers the lexical, syntactic, and se- mantic representations of source code. However, its codebase has not yet been open-sourced. These match-based evaluation metries can only evaluate the surface form and semantic differences of the code, while neglecting the exccutability of the code and the functional equivalence of implementation variations.
up
In the line of neural network based methods, re- cently released large language models (LLMs) (e.g., ChatGPT (OpenAT, 2023)) have shown re- markable performance in a wide range of NLP tasks with instructions and few in-context ex- amples. ChaiGPT is built upon GPT and opti- mized with Reinforcement Learning from Human Feedback. ChatGPT can efficiently understand and generate code sequences, and can lelf-learn from human feedback to improve the quality and accu- racy of its outputs. This significant advancement has markedly propelled progress in the field of code translation.
Execution-Based Evaluation Metries Execution-based evaluation metrics mainly compare the executed result of the generated code with the expected result. The PASS@K score (Kulal et al., 2019) is evaluated by unit tests: if any of the l samples meets the expected result, the generated result is deemed successful. AvgPassRatio (Hao et al., 2022; Hendrycks et al., 2021) evaluates the overall executable result of code by calculating the average pass rate of test cases. Computational accuracy (Rozière et al., 2020) measures the quality of the generated code snippet by comparing the output of this snippet with the reference code snippet when given the same input. Additionally, CodeScore (Dong et al., 2023) claims that it can estimate the PassRatio of test cases for the generated code without executing the code, but its codebase has not yet been open-sourced. These execution-based evaluation metrics require construction of executable test
A.1.2 Code Translation Metries
Match-Based Evaluation Metrics These evalu- ation metrics are based on the similarity between the translation output and the reference translation. Among them, the Exact Match (EM) metric cal- culates the percentage of translation outputs that exactly match the reference translation, which over- looks the fact that the same function can be im- plemented in various ways. The Bilingual Evalu-

---

Method
Train
Dev
Test
Total
Method
Train
Dev
Test
Total
C + C#
796
84
169
1049
C++ C++
799
149
298
1246
C+ Go
877
227
454
1558
C Java
813
171
343
1327
C + Python
901
213
426
1540
C PHP
296
51
102
449
C < VB
617
97
194
908
C++ <> C#
748
79
160
987
C++ + Go
792
208
418
1418
C++ + Java
753
172
345
1270
C++ + Python
842
202
405
1449
C++ < PHP
291
53
106
450
C++ + VB
586
97
195
888
C# +
777
100
202
1079
C# <> Java
750
86
174
1010
C# Python
813
99
199
1111
C# + PHP
293
40
80
413
+ VB
597
70
142
809
Java + Go
793
221
443
1457
Java < Python
838
217
436
1491
Java PHP
574
119
239
932
Java VB
610
104
210
924
Go + Python
887
314
628
1828
Go ++ PHP
606
128
258
992
Go VB 618
116
232
966
PHP Python
927
185
370
1482
РHP <> VB
267
44
88
399
VB Python
644
114
229
987
Table 11: Composition and Distribution of the Multilingual Dataset. The numbers refer the number of code pair
samples. VB short for Visual Basic. [Return to Section 3.1]
sets, which could be costly. Furthermore, due potential security threats from the execution environment and the code, they need to be run in an isolated sandbox.
adherence to these licenses.
A.2.2 Data Processing
Multilingual Datasets Given the variations compilation requirements among programming lan- guages, we keep the original format as much as pos- sible to ensure the compilability of the data while ensuring its accuracy. Additionally, we employ a cualicate-file-dtteed
A.2 Data Management
A.2.1 Data Sources & Licenses
We collect CodeTransOcean from two different platforms. The MultilingualTrans and NicheTrrans datasets are collected from Rosetta Code a pro- gramming site presenting solution strategies for identical tasks across many programming lan- guages as possible, thereby demonstrating both similarities and differences among these languages. We strictly adhere to the data distribution license the platform as Seaibbutioo-ShhreAAiee national (CC BY-SA 4.0) 15cense
tool to identify and remove duplicate data from the dataset to avoid any poten- tial data leakage problems during model training.
Pross-Framework Dataset To ensure the compi- lability of Python, we keep the original formatting information. We manually verify all automatically collected samples, identify and exclude samples that do not meet the requirements.
4.0 Inter-
A.2.3 Data Quality
The DLTrans dataset is derived from open- source teaching platform Dive into Deep Learn- ing"6, which is dedicated to teaching deep learning knowledge ranging from thcoretical background, conceptual understanding, to coding practices. We strictly adhere to the data distribution license of this platform as Apache-2.0" To ensure legal and regulated use of these datasets, we require strict
We randomly select IK samples from each dataset within CodeTransOcean for manual quality as- sessment. We find that for the MultilingualTrans dataset with compilation requirements, the compil- ability rate exceeds 90 %. We verify that the code pairs in each dataset are functionally identical and confirm that
is of high quality. Additionally, during data collection, we pay spe- cial attention the diversity of domain knowledge and code styles. CodeTransOcean includes vari- ous code examples ranging from basic syntactic structures to complex algorithm implementations, as well as building neural networks from scratch and conducting training and inference. This rich
"httas//roosttccod.oookkkkk/
Code "attep///eaaaieeeemsssoo////o/as by-sa/4.0/
"nttps://g1 http:///thhhbcccoo/////aaa//// clob/master/L.ccenet

---

diversity ensures that CodeTransOcean reflects wide variety of real-world scenarios.
Py
class Kastediottrasttn ose fnn Grosefit reppe" ose]: det forvardiaelf, Label, velie]lent:
torst.ores Clabell
A.3
Specific Challenges in Implementing Cross-framework Translation
Noights
haiighta, valid]len)
relf.reduction ioducctooonnnone "none"
woneighied supas BaskedBolunnceEioss
Ioforwazdl
(f 2, 23, labsi:
Firstly, there are significant design differences be- tween frameworks, including data processing meth- ods, model-building strategies, and network con- nection techniques. Secondly, the inherent com- plexity of DL code increases the difficulty of con- version, as these codes usually contain various components such as neural network layers, loss functions, optimizers, and learning rate schedulers. Thirdly, there are significant inconsistencies in the code structure of different frameworks, such code organization and variable naming rules. Lastly, cross-plan posss-pltffomm compatibility must be con- sidered because DL code may encounter compati- bility issues when executing on different hardware platforms (e.g., GPUs, CPUs, TPUs) and operating systems.
wnightedLioss - (onweighted Lozz seights)
disel) return veightee_less
TensorFlow
claxs Waskedtiftton wra.ttttaatttioo www.eea..ooeee......... wow.es.sssss............... init (self, valid_len) supert) colf.velid valid_ion
def celliself, labei, pred):
seights sequenze_ mack self Ien)
Label_ons_ Lf.oow flabsi, dep.tepced.
shape [-11
sowiighteddoo L..bere.
detegorica tores
Lron_logitaaeee seduct mnon"))
label one hot, predt
weighted aecdicc_nanntaaeeeeeeee
seights), axte=1) zoturn veigh:ad_
MXNet
clazs MaskedBoftraxcSLees Igluon, Scf.zaxcattssa def lorwerd(owlf, pesd, label, valid lent
axpaadddd__da
1ike (labal),
weights apa.sequence aaseuunee__aaaeee (weights, val.d true, axixel) return Munper//sceeiooteeeeeoo celt). pred, Jabel, smighta:
A.4 Code Examples on Different Deep Learning Frameworks
Paddle
Figures 1 and show the implementation of two different deep learning components in various deep learning frameworks.
Class Haskenlin waa.ttifiattttttteeeeoee fon. Consaaiinteooooooooo eapyloas: det prrerdisnit, prec, (sbel, va walid lear:
seightz paddie.
flabel)
soigate sequence mask txeights, valid len)
self. ceceuuctionn"ie unweighted
supor MasteddootesscEEooss
A.5 Multilingual Modeling
I.foznazdr
gred, label]
wiigited (snwiigite weighes) awwan axia=l]
One-to-One For each language pair in the dataset, we train an independent model, e.g., trans- lating C++ Java.
roturn weightod
Figure I: Sofimax cross-entropy loss fanction with mask- ing in PyTorch, TensorFlow, MXNet and Paddle.
One-to-Many We train individual models from one language to many other languages, e.g., trans- lating Python to all other languages.
model to simulate more Nomain-adaptable and spe- cialized expert roles.
Many-to-One We train individual models from multiple languages to one language, e.g., translat- ing all other languages to Python.
Polite inquiry (Akin, 2023) These strategies add polite expression and set up imperative and inter- rogative requests. Given that ChatGPT is designed to simulate human conversation styles as closely possible, including understanding and simulating polite language expressions. Therefore, we expect these strategies to boost the comprehension of the model and augment the quality of its generated results.
Many-to-Many We train unified model for the multiple to multiple languages in the dataset, which can handle translations between all languages.
We ensure all experiments are performed un- der the same hyperparameters and environment for comparison. Table 13 shows these in detail.
A.6 Prompt Variations
Clarify usage This strategy aims make the model clearly aware of its requirements during the code translation process the generated code needs to guaranteed to execute without issues. The translation prompts the above four strategies
Role Assignment (Peng et al., 2023; AlKhamissi et al., 2023; Wu et al., 2023; Akin, 2023) We con- figured two distinct roles for the model, each with unique skills. This arrangement empowers the

---

Method
Train
Dev
Test
Total
Method
Train
Dev
Test
Total
Ada
6022
464
937
7423
Elixir
3618
297
599
4514
Arturo
3802
470
947
5219
Erlang
3269
217
449
3935
Aut
4305
555
1120
5980
Factor
4724
860
1756
7340
AWK
3880
578
1162
5620
Forth
3619
339
690
4648
BBC Basic
3663
239
485
4387
Fortran
4050
305
617
4972
Clojure
3617
310
633
4560
Groovy
3773
227
467
4467
Common Lisp
5085
467
933
6485
Haskell
5647
1045
2097
8789
D
5541
825
1662
8028
Icon
2646
158
326
3130
Delphi
3547
436
889
4872
J
5476
1204
2422
9102
Julia
5829
1511
3055
10395
Ruby
5793
1290
2600
9683
Lua
5316
677
1366
7359
COBOL
2438
167
355
2960
Mathematica
5485
1046
2105
8636
REXX
5595
1118
2241
8954
MATLAB
2872
157
322
3351
R
2803
197
402
3402
Nim
5814
1321
2675
9810
Racket
5646
901
1817
8364
OCaml
4286
405
817
5508
Rust
5146
717
1439
7302
Pascal
3393
465
942
4800
Tel
5354
740
1502
7596
Perl
5818
1445
2914
10177
PowerShell
3563
240
490
4293
Scala
5852
1074
2164
9090
F#
4517
638
1287
6442
Swift
3653
404
818
4875
Table 12: The number of code samples for each language the NicheTrans datasets. [Return to Section 3.
MultilingualTrans
Niche'Tram
DLTrans
Experiment #3 randomly selects code pair an example, in which neither the source the tar- get languages match the translation requirements. The specific translation prompts are shown Table
Learning rate
3c-5
20-5
36-5
Beam size
1
Max souree length
1536
1536
512
Max target length
1536
1536
512
Batch size
19.
Max spoch
Fp16
GPU
NVIDIA Tesla 32GB
A.8
Chain of Thought
Experiment #1 First, describe the function of the source code in the natural language, then translate it according the source code and the corresponding natural language description.
Table 13: Parameters and hardware configuration for training CodeTS+_220 2200 (220M is the model size) on CodeTransOcean.
Experiment #2 First, let ChatGPT understand the function of the source code, followed by the translation, while ensuring that the function of the code remains unchanged during the translation pro- cess.
are shown in Table 15,
A.7 One-Shot
Experiment #1 selects training sample from high-resource language code pair as an example. this case, the target language type aligns with the target language type of the translation request, but the source language does not.
Experiment #3 First, let ChatGPT understand the function of the source code, then predict the output result of the source code, and finally perform the translation, demanding that the translated code successfully executes.
Experiment #2 selects code pair whose source and target language directions congruent with the translation requirements as an example. That is, the source and target languages of the exam- ple dynamically adjust following the translation requirements.
Experiment #4 Building upon Experiment #3 and the one-shot approach of Experiment #2, we introduce a CoT one-shot variation. That is, first, provide case in the same direction for ChaiGPT reference, then require to understand the func-

---

TensorFlow
Claaa Sec22egleecndes :esodee):
PyTorch
def init _niik (sulf, yocak_pize, arbei_atza,
mun mun_hidoens, munulayers, dropost=l, exteagge): super init. (asisangs)
3
class SegeEcatoredar : (smiC,
selt.ensddding -t..keass Froeddngg
c1ze, embed size)
siddes, layers, sropout-nl, sspgangs)
self.ena S...oraa.layeraa www.aaiec...eaaaappo......
super (SeeeSee awlf) iinns [eokoergs)
StackcdRMMcellll [LL waw.ssaayasasa... wtt.s...Gooc11 foarsel despout
cnle.enbedsing nh, -n.Stbedeinggnoonn 6176,
crbci_siz3)
"dropouti
colt. Jember, size nun hiddens, sunnhid payyiay deopoctndropouzz nn.einear toun hiddone, voceb_ cize)
2os in sance Loan_
return sernnnegggeeeeserrr retorn_states True]
relt.dense tt.rersz. layers.Pense (voceb s1ze)
(sit enc_outputs, smrge): suluen uns_oalpat][])
inst stateiselt, ourputs, eangel:
dct forwardicelf, x, statal:
seturn enc_pcipate
13
12
x = se ",erheading ng (1) (1, 6, 21
concext - statel-i] ropeat (X shage [01, 1, 17
14
13
daf taeli, X, state, vioeaggs::
X sell..nbedsiin
14
x and context tores.eat cat (ix, contextl, 2)
culput delc.deme jou.pu.) pesmuteth, roturn output, state
cutput,
self. CCan context,, state!
15
16
context if.rupe.. (Lf, dina (atata(-1),
axia=ll, Xaand context concat (N, contextl, exte-21
(I, axix-()
(L stare,
keargst
cotpot selt.denee (ran, oungur [03)
19
zeturn output, [c
Paddle
-3325 Boggieggeender Bezoder);
daf init (self, vocak_pize, arbod_aiza,
MXNet
nun nun_hiddon hiddone, nun layers, dropout- Dy +exxagge):
3
supur (Be22Buueeee selli. _init, (vrloazes)
class Sog2feglosodrrdi2 Decoder):
selt welttennwddingg en. sizee
_init_ (vell,
anbediise,
nun Moden,, nun loyeeayer,, dropout =
anight pado
auper (Bee28ueeeeess awl() -in - (ookusp)
weight sttr - pade ie P.Parrrtte Pl
self. enbeddiag zn, www.bbbeciinggooooostssst wibel[sise)
oelt. = rnn,GR siidens, layeleyers, thopputiiropous)
weight an ater = pade le
Indriaiizes wec.aaaiee.aviirrn...o.. Kavicruni swlf.enn ni, 500 virbeii manglidiea
izerrnn.
6
colt.cenae Denee
Flatten-Palze)
panchiddens, ioyer, drapout-éropous
joreSruw, iuaatta eeegght
tnit state sutpote, +args]:
sotuEn [nc_oatputs[1]
11
12
Jel foivardfself, X, stacs):
X se.i. webbiiinn (X) -vvspaees 1)
10
selt.denze mn..inmar (nom hiddens, voosbaaise, woight ateerveight acts!
neight hh attr)
14
13
context app.broddsst__t tolonttet OK.ppape
context.shage rhl context,
state]0] [-1]
(113)
11
def stateisclf, outpoup eargel: unc_psipuls
rp.concstenss
contextl,
cutput, state self.rra anddannnnnttdodooaeeeeeeeeo context, stato!
Iornaeditael,, X, a.alels
cntpot n.dense poutpur) cvagasee 13, 1)
X eentedtt ny (X) ranngeoe (11, 0, 213
18
roturn output, state
17
context = staies 1) ct [D6 slaa 1, 111
18
Xnndd X_and_ - pansle..concst 10, contest), 30
19
cutput, state self, ai__aad__ooteeeedddd stacs) cutpot delee cont rangsaon 211, 2) zeturn output, state
20
Figure 2: Implementing RNN Decoder for Seq2Seq Learning in PyTorch, TensorFlow, MXNet and Paddle. [Return to Section 3.2]
tion of the source code, then predict the output of the source code, and finally translate it, with the condition that the translated code must successfully execute.
The specific translation prompts are shown in Table 19.

---

Method
C++
C#
Go
VB
Python
Java
PHP
Naive
14.61
9.08
4.52
1.64
2.17
11.23
2.09
OIO
11.13=0.28
4.77:0085
8.18:1040
2.02:10233
1.8510.14
6.50:12.1
1.71=0.32
C
OIM
12.33=0.58
7.56=0.98
9.5821077
2.22=0.89
2.8240.58
8,92-0067
3,42=0.09
MIO
7.99:11.94
5.35+0.41
6.94:10.83
3.77:10.44
2.09::0.25
8.65
0.93
MtM
9.61+0032
1.15+141
7.54++033
2.30+0.71
2.11+0.43
8.29:10.55
3.30=0.92
Naive
14.88
10.08
3.87
3.76
1.69
11.39
1..2
OIO
10.58:40.37
6,52=1196
6.77+1108
2.3420.90
1.2:40.37
9.41
1,50=0.03
C++
OIM
13.3242246
9.87=1.15
9.83
2.62+1.15
2.9340.18
12.10=1.51
2.87=0.82
MIO
10.54:2.33
1.48
7.29:2073
3.57=0074
2.2840.18
7.8211.26
1.96=0.20
MIM
9.92111.14
1.79
7.70:10.48
2.06+0.74
2.3840300
10,20::1.00
4.12-0811
Naive
9.05
10.03
3.00
7.20
- 1.83
13.60
2..58
OIO
5,41+1,25
6.45++168
7.10++0.3
9.42+5.500
1.73++0.46
10.55+
1,97-0.76
C#
OIM
7.42=1882
8.99=1.17
9.86
6,13+2292
2.80#0.43
11.92:2.52
4.55=0.81
MIO
6.37.1069
6.13:1.60
7.53
9,9211000
2.47:0062
10.16:1440
1.70=0.20
MIM
6.19+-088
7.22++0.3
9.36+
4.60+2.87
2.23+0.36
+
4.70-0.27
Naive
4.52
3.75
5.04
2.46
3.00
6.56
2.14
OIO
5.65:10.41
6.8811047
1.70
1.8710.54
2.8910.16
5.41
2.48=0.23
Go
OIM
6.06+0.99
6.56++0.89
7,28
2.12+0.65
3.32+0.36
8.79+1.06
3.48-0.20
MIO
5.41:10.33
4.771:0.66
6.8930.33
2.94-0.63
3.12:0..8
7.81:0.32
1.79=0.51
MIM
5.1811.23
6.06:10.28
7.12:0.72
18240.93
2.83+0.50
8.9911,28
2.55=0.40
Naive
1.64
3.76
7.29
2.54
1.42
2.89
0.46
OIO 3.9611.20
5.25:10.68
17.34:11.79
5.85:10.64
1.34:10.06
5.51:10.43
0.61=0.18
VB
OIM
5.15110.15
6.10:10.30
19.63411.03
7.83:1044
2.02:10.32
7,91
2.02=0.20
MIO
4.25++0.6
4.15+1009
1..9
6.38+00.5
1.66
7.26
1.18-0.33
MIM
5.18:0065
5.09:10115
14.1341883
6.97+1.30
2.3440.11
8,39-1077
2.92-0.22
Naive
1.73
1.14
1.44
2.50
0.89
1.82
1.28
O1O
4.35=0.44
3.66:20.5
5.3820.76
5.67:10.34
2.53+1.11
4.70=10.07
3,24=0.60
Python
OIM
4.51:11.33
5.18:10.52
6.48
6.04:10.9
1.50
6.29:10.39
6.95
4.7011028
3.4410.76
5.4810.55
4.8010.33
2.42-10.88
5,32110.6
2.74-0.39
MIM
4.54+00.7
4.74+0.11
5.69
5.84
2.09+0..4
6.45+
4.49-0.63
Naive
T1.23
T1.13"
13.44
6.37
2.84
2.24
"2.6
OtO
1.15
9.09+11.0
10.92++1.11
11.05=0.52
2.66:10113
2.54+:0.35
2.05=0.95
Java
OLM
9.27+1152
9.3311086
13.52:40.95
12.5720.96
2.9220.78
3.98
6.20=1.08
6.69:10.83
6.32:11.66
10.5741.16
8.7910.44
3.91
2.8611.36
0.48
MIM
6.56++0.33
7.60+00.66
9.40+1.08
8.30+0.87
2.20+0.85
2.53++0.15
3.57-0.94
Naive
1.79
1.31
2.36
2.00
0.37
1.30
2,30
OIO
4.18:11.44
2.91:10.43
6.45
5.85:10.54
0.42:10.99
1.98:10.26
6.47:10.59
PHP
OtM
3.78++180
1.93+1.35
5.51
5.13:10.00
0.56+0.36
2.70+4..00
5.51+11.1
4.97-1.09
3.69110.6
6.0823.03
6.44110.8
2.2840.12
3.5540.18
8.66.10.4
MIM
6.03-10.61
5.15:10.67
10.64
6.69110.11
1.69
2.46:00.66
7.87+1190
Table 14: BLEU scores from different multilingual modeling strategies by fine-tuning the pre-trained CodeT5+ 220M model (220M the model size) (Wang al., 2023b), Naive denotes Naive Copy. which directly duplicates the source code the target code without making any modifications. Method OtO, OtM, MIO, and MIM denote One-to-One, One-to-Many, Many-to-One, and Many-to-Man Many,, respectively. The rows correspond to the source language while the columns correspond the target language. We run each experiment with three different random seeds and report the mean and standard deviation of BLEU scores. [Return to Section 4.1.]

---

Method
Translation Prompt
Role Assignment
"role": "system", "content":
"Your are a code translation system.", ", "role": "user", "content": "Please provide the [TL] translation for the following [SL] code:
Role Assignment "role": "system", "content":
"You are a code translation system that specializes in [SL] and [TL] programming languages.", "role":
"user", "content"; "Please provide the [TL] translation for the following [SL] code:
Role Assignment #3 "role": "system", "content";
"You are programmer proficient in multiple programming languages.", "role": "user", "content": "Please provide the [TL] translation for the following [SL] code:[SC]
Role Assignment #4 "role": "system", "content";
"You are programmer proficient
in [SL] and [TL] programming languages.", ", "role": "user", "content": "Please provide the [TL] translation for the following [SL] code:[SC)
Polite Inquiry
Please translate the following
[SL] code into [TL] code: [SC]
Polite Inquiry #2
Can you rewrite this [SL] code in [IL]][SC]
Clarify Usage
Translating [SL] to [TL] ensures that Python code can be executed. [SC]
Divide & Conquer
Translas ate [SL] to [TL] : [SC]
Table 15: Translation prompts for prompt variants and contextual strategies on ChatGPT. (SL] refers the source language, [SC] refers to the source code, [TL] refers to the target language. [Return to Section 4.3]
Strategy
Temp.
EM
BLEU
CodeBLEU
DSR@0
Strategy
Top-K
EM
BLEU
CodeBLEU
DSR@0
10.83
24.45
48.57%
0.57
10.83
24.45
48.57%
0.2
0.57
10.82
24.43
47.71%
0.2
0.57
10.81
24.45
48.29%
0.4
0.57
10.80
24.37
48.00%
0.4
0.57
10.81
24.44
48.29%
Top-K =
Temp.
0.6
10.78
24.45
48.00%
10.82
24.42
48.29%
0.57
10.82
24.49
47.71%
0.57
10.80
24.38
47.71%
1.0
0.57
10.81
24.41
48.00%
1.0
0.57
10.81
24.38
48.00%
Table 16: Code translation performance of ChatGPT under different parameter settings. Temp. refers to temperature.
[Return to Section 4.3)

---

Self-debug@0
Translate @ource_languaae to Laagettlanguagel) @oorce__coder
CтO
Here is the Laarget_languaar code equivalent of the given [source languagl code: granslated_cooee
Self-debug@n
The above python code executes with the following errors, please correct them. [Compiler reports errors]
сто
SpC
Here is the modified harget_language code: [translated_codee
Table 17: A simple demo: Translation prompting of ChatGPT in the multi-round debugging strategy. The content in red is returned by the compiler. [Return to section 4.3.]
Zero-shot prompting
Does the following Python code execute? [python_codes
Sp
Yes, the Python code executes without errors.
Please predict the executed output of the Python code above. The predicted execution result of the Python code above is [output].
CTO
Few-shot prompting
This is executable Python code [python_code), and this is a Python code [python_ccom _odee] that cannot be executed. Does the following Python code execute? ppython_code/,
Yes, the Python code executes without errors.
Please predict the exccuted output of the Python code above.
CтO
The predicted execution result of the Python code above is [output].
Table 18: Two simple demos: prompting in fuzzy execution experiments. [Retum Section 6.]

---

Method
Translation Prompt
One-Shot
Here is example of translation from [ESL] to [ES]]: [ESC], [ETL]: [ETC]
Please imitate this example to
translate following code from [SL]
to [TC]:
One-Shot #1 where [ESL] [ [SL] and [ETL]= [TL]
One-Shot #2 where [ESL]= and [ETL]=
One-Shot #3 where [ and [ETL)
CoT #1
1.Please explain the function of
the following [SL] code, which
is limited to 200 words. (SC)
2.Please translate into [TL]
code according to the following
[SL] code and its functional description. [SL] Function description: [DSC]
CoT #2
First, understand the function of the following [SL] code. Then, translate the [SL] code into [TL]
code while keeping the function [SCh First, understand the functionality of the following [SL] code and predict the execution output. Then, translate the [SL] code into [TL] while maintaining the same functionality. ensuring that the translated code can be successfully executed.[s1) First, learn how to translate [ESL] code to [ETL] based on the example, [SL]: [ESC], [ES]] [TL] Then, understand the functionality the following [SL] code and predict the execution output, [SL]: Finally, translate the [SL] code into [TL] while maintaining the same functionality ensuring that the translated code
CoT #3
CoT #4
[ETC], [ETC]
be successfully executed.
Table 19; ChatGPT translation prompts one-shot & CoT strategies. [SL] refers to the source language, (SC) refers to the source code, [TL] refers to the target language, refers to source language the example, [ESC] refers the source code in the example, [ETL] refers the target language in the example, [ETC) refers to the target code in the example, [DSC] refers to the natural language description of the source code. [Return to Section 4.3]


Reviews:
[{"review_id": "psDSbYk3V0", "reviewer": "Reviewer_RZux", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our contributions of providing valuable new paired data for code translation tasks, facilitating execution of important specialized fields in real-world code translation applications (such as NicheTrans) and inspiring subsequent research, and our thorough performance analysis of LLMs on code translation tasks and in-depth ablation analysis. We sincerely thank the reviewer for all valuable feedback. Below we address all your questions and concerns.\n\n**Q1: Earlier match-based and execution-based metrics have shortcomings. The new metric proposed does not fully address the core issue at hand.**\n\n**Response:** Firstly, we would like to emphasize that the primary contribution of this paper is to establish a comprehensive code translation benchmark including new datasets for high-resource and low-resource programming languages, cross-deep-learning framework datasets, and for effectively evaluating LLMs on code translation tasks. Our goal is not to fully address all the issues in existing code translation evaluation metrics, such as neglecting executability and functional equivalence with match-based metrics and constrained by execution environment with execution-based metrics. We propose the Debugging Success Rate@K (DSR@K) metric to meet the needs of **code debugging** with LLMs and to complement the previously proposed Pass@k metric (Line 1143-1146). As explained in Line 484-492, DSR@K is the first metric designed to accurately reflect real-world software development scenarios. With the promising capabilities of LLMs for code debugging, we consider this new metric both practical and reasonable.\nNote that we did point out the limitations of our current automatic pipeline AutoTransCompiler and the DSR@K metric in Sections \nLimitations and Related Work (Line 696-734 and 1101-1163), as AutoTransCompiler currently only supports Python as the target\nlanguage hence there is no constraint from compilation/execution environment. \n\nWe consider two potential solutions to address core issues of code translation evaluation metrics. **One solution is Extending AutoTransCompiler to support a wide range of programming languages** by implementing automatic detection of code dependencies and automatic compilation and execution in a sandbox for safe execution. However, even recent commercial products such as [coderpad.io](https://rosettacode.org/wiki/Rosetta_Code) only provide a multilingual execution engine and cannot yet fully automate the entire process. There are still many technical challenges to overcome. **Another solution is further exploring our proposed fuzzy compilation**. As discussed in Section Limitations Line 731-779, we believe that fuzzy compilation using LLMs has the potential to address the limitations of current match-based and execution-based evaluation metrics. Although ChatGPT for fuzzy compilation is not yet practical, we are exploring fine-tuning methods to improve it.\n\n**Q2: Need to provide a clear explanation of the code pair quality.**\n\n**Response:** \nHere we clarify how we ensured high data quality for our CodeTransOcean datasets.\n\n**(1) Data quality:** \nSection 3 Line 220-222 points out that the details of data quality control are in Appendix A.2, where A2.2 and A2.3 Line 1165-1215 elaborate how we ensure the quality of the data. We verify that for the MultilingualTrans dataset with compilation requirements, the compilability rate exceeds 90\\%. We verify that the code pairs in each dataset of CodeTransOcean are functionally identical and confirm that CodeTransOcean is of high quality.\n\n**(2) Unit tests:** Our CodeTransOcean provides unit tests. The vast majority of the samples in our datasets provides explicit input and output, which is equivalent to unit tests. These input and output information is collected from our data source [rosettacode.org](https://rosettacode.org/wiki/Rosetta_Code) as the vast majority of tasks in the source provides a clear input and output. Our CodeTransOcean, with **over 200,000 unit tests** and **45 covered languages**, far exceeds the scales of Avatar and TransCoder  (Avatar includes Java and Python, with a total of 250 unit tests; TransCoder encompasses Java, C++, and Python with 2,112 unit tests).\n\n**(3) Execution environment:** As in our response to Q1, the focus of this paper is not to fully address all the issues in existing code translation evaluation metrics. In this paper, we provide an execution environment for Python, given that many code intelligence tasks, such as HumanEval, MBPP, and DS 1000, are based on Python. And our AutoTransCompiler currently supports automatically using LLMs to conduct code translation, compilation, debugging and calculating success rate, with the target language as Python.\n\n**(4) Evaluation metrics:** We respectfully disagree on the reviewer's comment that “most of the author’s experimental section\nonly involves metrics like EM”. Except for our proposed **DSR@K** metric used in the experiments on the LLMTrans dataset, we used both **EM** and **BLEU** evaluation metrics in all other experiments (Table 3, Table 10-13, Table 15) following prior works. Table 3, 12, 15 also report CodeBLEU in addition to EM and BLEU.  We explained these metrics in detail in Appendix A.1.2. EM and BLEU have been extensively used in prior studies such as CodeTrans (NeurIPS 2021), CoST (AAAI 2022), Avatar (ACL 2023) and TransCoder (NeurIPS 2020). As explained in Sections Limitations Line 696-709 and Related Work Line 1101-1163, there are no other match-based metrics other than EM and BLEU that can support the 45 programming languages covered by our CodeTransOcean. Moreover, as described in our response in Q1, it would require great effort to implement execution-based metrics for this large number of programming languages.\n\n**(5) Version differences in the Avatar paper:** It is worth noting that when we wrote this paper, the Avatar version (Ahmad et al., 2021b) ([version 1](https://arxiv.org/abs/2108.11590v1)) that we cited was published on August 26, 2021, which did not include unit tests and an execution environment. The reviewer referred to the [version 2](https://arxiv.org/abs/2108.11590v2) published on May 4, 2023, which incorporated unit tests. According to the EMNLP 2023 guideline that “papers (whether refereed or not) appearing less than 3 months before the submission deadline are considered contemporaneous to your submission, and you are therefore not obliged to make detailed comparisons that require additional experimentation and/or in-depth analysis.”,  our work is contemporaneous to version 2 and does not need to compare to version 2.\n\n**Reproducibility: 3**\n\n**Response:** \nTo ensure the transparency and reproducibility of our experiments, we have already provided all the relevant code, datasets and experimental results in the supplementary materials. We will make CodeTransOcean's datasets, code, experimental results, and model checkpoints publicly available, so that researchers can reproduce all our results. We also created a website and leaderboard for this benchmark. To demonstrate these to reviewers, we uploaded the dataset, code, experimental results, website, and model checkpoints in [anonymous Github](https://anonymous.4open.science/r/CTO-5263/Readme.md) as a way to demonstrate the reproducibility of our work. All in all, based on our rigorous internal testings, we are confident that our experimental results are easily reproducible.\nIn addition, Line 382-383 points out that the details of multilingual modeling strategies are in Appendix A.5, where A.5 provides the details of the specific hyperparameter settings and environment used in all experiments in Table 9. The reason that the training and evaluation data has not been widely available is because the training/evaluation data is from our CodeTransOcean and we have not yet disclosed the datasets before the anonymous review period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Since the emergence of ultra-large language models, the focus of the NLP community has increasingly shifted from model-driven to data-driven research. This paper contributes valuable new paired data for code translation tasks, making it a noteworthy and commendable effort.\n2. The NicheTrans Dataset and LLMTrans Dataset, which involve more specialized fields, hold significant value. In the context of industrial-level code, the framework migration for these less popular fields urgently requires high-quality annotated data. Such data has the potential to facilitate the execution of related projects and inspire subsequent research within the community.\n3. The author engages in a thorough performance analysis of the current code translation task on large language models, as well as in-depth ablation experiments. These efforts are commendable and deserving of recognition.\n", "reasons_to_reject": "1. Regarding the evaluation metrics for code translation tasks, earlier metrics such as CodeBLEU had shortcomings in reflecting the true effectiveness of the generated code. Subsequently, the community introduced metrics like executable accuracy and computation accuracy. However, these methods were constrained by the cost of constructing the execution environment, resulting in limited versatility. The new metric proposed by the author still relies on the compilation environment, and thus, it does not fully address the core issue at hand.\n\n2. Besides the scale of the collected data, the quality of the gathered code pairs is also crucial for current code translation task datasets. Some recent new datasets, such as Avatar and TransCoder, often include unit tests and executable environments. However, most of the author's experimental section only involves metrics like EM, which do not provide a clear explanation of the code pair quality.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about advancing research on code translation by constructing a large-scale comprehensive benchmark called CodeTransOcean. It addresses the problem of limited code translation datasets that usually focus on a single pair of popular programming languages. To tackle this issue and meet diverse requirements of real-world applications, the authors propose three novel multilingual datasets: MultilingualTrans, NicheTrans, and LLMTrans. Additionally, they include a cross-framework dataset called DLTrans for translating deep learning code across different frameworks.\n\nThe main contributions of this paper are:\n1.\tThe creation of CodeTransOcean, a benchmark that supports the largest variety of languages for code translation.\n2.\tThe development of multilingual modeling approaches for code translation, which improve the translation quality of both low-resource and high-resource language pairs and boost training efficiency.\n3.\tThe proposal of a novel evaluation metric called Debugging Success Rate@K for program-level code translation.\n4.\tThe evaluation of LLM ChatGPT on their datasets and investigation of its potential for fuzzy compilation predictions.\n5.\tThe establishment of baselines for CodeTransOcean and analysis of challenges of code translation to guide future research.\n\nThese contributions align with the EMNLP's interests in new data resources, approaches for data and compute efficiency, and publicly available software and pre-trained models.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fh9Y8XYkLF", "reviewer": "Reviewer_XoXb", "summary": "", "questions": "- How did you decide which models to evaluate using the CodeTransOcean benchmark? Why methods that are assessed in previous work are not included in this paper?\n- Why existing work like [1][2] is not referenced and compared in this paper?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our contributions of creating unique datasets for code translation tasks, evaluating reliability of translated code, and providing empirical validations of potentials of multilingual modeling for improving translation quality and increasing training efficiency. We thank the reviewer very much for all thoughtful reviews and valuable feedback. Below are our responses to all of your questions and concerns.\n\n**Q1: Limited model evaluation.**\n\n**Response:** We respectfully disagree the reviewer's comment that “The paper primarily focuses on the evaluation of ChatGPT, neglecting other significant models such as Transformer, CodeBERT, DOBF, and TransCoder, and open-source LLMs like LLaMa.”\nIn this work, we aim at investigating various multilingual training strategies and studying the potential of LLMs on code translation tasks, hence we did not report experimental results on a wide range of models; instead, we comprehensively evaluate the strategies and potentials using the best performing models. **We conducted careful model selections based on both prior works and our experimental validations**. \nFirstly, Table 2, 3, 4, 5 and Figure 6 in the CodeT5+ paper (Wang et al., 2023b) which is cited in our paper shows extensive performance evaluations of open-source and closed-source LLMs on code tasks.  The results show that CodeT5+ significantly outperforms prior models including encoder-only models such as RoBERTa, CodeBERT, and GraphCodeBERT, decoder-only models such as GPT-2 and CodeGPT, and encoder-decoder models such as CodeT5 and PLBART. Details of these baselines that CodeT5+ compared to are in Section 5 of the CodeT5+ paper.\nThe results in the CodeT5+ paper show that CodeT5+ is the best performing **open-source pre-trained language model for code tasks**, and CodeT5+ outperforms LLaMA on code tasks. Since the sizes of the three datasets of MultilingualTrans, NicheTrans and DLTrans require fine-tuning, we chose the pre-trained CodeT5+ as the backbone for all multilingual training experiments, shown in Table 10-13. Secondly, we also evaluated CodeBERT, CodeT5 and CodeT5+ on MultilingualTrans, NicheTrans and DLTrans and observed that CodeT5+ significantly outperforms CodeBERT and CodeT5. \n\nFor evaluations on our LLMTrans dataset, we chose GPT-3.5 to evaluate because GPT-4 and GPT-3.5 are the Top-2 LLMs on code tasks, as shown in Table 2 of the CodeT5+ paper and at the time of this work, we did not have access to GPT-4 API. Hence GPT-3.5 is the best performing LLM available to us. Since GPT-3.5 has significantly better code task performance than LLaMA, as shown in Table 2 of the CodeT5+ paper, we focus on conducting comprehensive evaluations of various algorithms and studying potential of ChatGPT on code translation tasks, with results reported in Table 3, 4, and 19. In our revised version, we will add experimental results for CodeBERT, CodeT5, and LLaMA for the sake of completeness.\n\nRegarding DOBF and TransCoder that you mentioned, DOBF is a pre-training method related to programming languages, not a specific model for code tasks, hence not related to our work. TransCoder is a fully unsupervised neural transcompiler based on unsupervised machine translation approaches. Its code translation performance is mediocre and is not widely used or evaluated. Hence, we did not discuss this model among code translation models in Related Work nor compare to this model. We only cited the dataset used in the Transcoder work in Line 189-193 of our paper. We will add TransCoder into related code translation models in Related Work.\n\n\n**Q2: Missing reference [1][2].**\n\n**Response:** Thank you for suggesting these two papers. Regarding paper [1], the code intelligence benchmark it introduced comprises five sub-tasks, and its “Code Translation” section is actually based on augmenting the data from the CoST paper we already cited in Line 191-193. It is worth noting that both studies are authored by the same team, and their data was directly collected from the GeeksForGeeks website. From the perspective of contributions to code translation, there is no significant difference between these two datasets. Hence we did not cite paper [1]. However, to ensure completeness, we will add a reference to paper [1] in the revised version.\n\nAs for paper [2], its main contribution is to propose a code translation dataset XTest containing nine programming languages with unit tests, but **it is not open-sourced**. Its contribution is similar to that of our MultilingualTrans dataset, which aims to expand code translation pairs between popular programming languages. As can be seen from Table 2 and Line 178-212 of our paper, we performed statistical analysis and verification of each dataset. However, the actual usability of the dataset in paper [2] cannot be analyzed or verified. We tried to contact the authors of paper [2] but there was no response. Hence we chose not to cite paper [2] in our paper.  Once we can access and validate this dataset, we will add comparisons and in-depth analysis of this paper in the revised version. \n\nAs shown in the above analyses, it is clear that while paper [1] and [2] contribute to the field, **their works do not impact nor overshadow the contributions of our work**. \n\n**Q3: How did you decide which models to evaluate using CodeTransOcean? Why some models assessed in previous works are not included in this paper?**\n\n**Response:** Please refer to our response to Q1.\n\n**Q4: Why existing work like [1][2] is not referenced and compared in this paper?**\n\n**Response:** Please refer to our response to Q2. \n\n**Presentation Improvements: Would be beneficial to include the main results table in the main body of the text rather than in the appendix.**\n\n**Response:**\nWe structure the paper based on two considerations. Firstly, the primary contributions of this paper are introducing the new CodeTransOcean benchmark for code translation. Therefore, we allocated the majority of the main body of the paper to elaborate the motivation and the process for creating this benchmark and dataset analysis, to demonstrate the efficacy of multilingual modeling on code translation, to propose a new evaluation metric, and to report comprehensive evaluations and studies of the representative LLM ChatGPT on these datasets. \nSecondly, due to the page limit, we only put some experimental details and some experimental result tables such as those of CodeT5+ (Table 10-13) in Appendix. However, **our paper is self-contained**, since for **all** these tables of experimental results that we put in Appendix, **we have included the text describing the purpose of the corresponding experiment and the detailed discussions corresponding to the results in the main body**. In the main body, for CodeT5+ results, Line 401-434 provides detailed analyses and discussions for Table 10, Line 435-447 for Table 11, Line 448-455 for Table 12 and 13, and Line 456-466 provides an overall analysis of the observations in Table 10-13. In the main body, for ChatGPT results, Line 569-580 provides detailed discussions for Table 15, and Line 592-618 details the experimental setup and results for Table 16 and 19. In the revised version, we will refine the organization of the content and move the key result tables from Appendix to the main body.\n\n\n**Reproducibility: 4**\n\n**Response:**\nTo ensure the transparency and reproducibility of our experiments, we have already provided all the relevant code, datasets and experimental results in the supplementary materials. We will make CodeTransOcean's datasets, code, experimental results, and model checkpoints publicly available, so that researchers can reproduce all our results. We also created a website and leaderboard for this benchmark. To demonstrate these to reviewers, we uploaded the dataset, code, experimental results, website, and model checkpoints in [anonymous Github](https://anonymous.4open.science/r/CTO-5263/Readme.md) as a way to demonstrate the reproducibility of our work. All in all, based on our rigorous internal testings, we are confident that our experimental results are easily reproducible. In addition, Line 382-383 points out that the details of multilingual modeling strategies are in Appendix A.5, where A.5 provides the details of the specific hyperparameter settings and environment used in all experiments in Table 9. The reason that the training and evaluation data has not been widely available is because the training/evaluation data is from our CodeTransOcean and we have not yet disclosed the datasets before the anonymous review period. \n\nCodeT5+ (Wang et al., 2023b) Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023b. CodeT5+: Open code large language models for code understanding and generation. arxiv:2305.07922.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well structured and easy to follow\n- The proposed benchmark includes creating three unique datasets to support translations between various popular and niche programming languages, and to evaluate the reliability of translated code. In addition, they develop a cross-framework dataset specifically for translating deep learning code across different frameworks. \n- Experiments demonstrate the potential of multilingual modeling approaches to improve code translation quality for various language pairs and increase training efficiency.", "reasons_to_reject": "- Limited model evaluation: The paper primarily focuses on the evaluation of ChatGPT, neglecting other significant models such as Transformer, CodeBERT, DOBF, and TransCoder, and open-source LLMs like LLaMa. This narrows the scope of the benchmark's applicability and restricts comprehensive comparative analysis.\n- Missing reference: By not referencing or discussing key works such as [1][2], the authors may miss out on important comparative analyses.", "questions_to_authors": "- How did you decide which models to evaluate using the CodeTransOcean benchmark? Why methods that are assessed in previous work are not included in this paper?\n- Why existing work like [1][2] is not referenced and compared in this paper?", "paper_topic_and_main_contributions": "The paper introduces a novel multilingual benchmark for code translation, named \"CodeTransOcean\". Recognizing that existing code translation datasets typically focus on just one pair of popular programming languages, the authors created this comprehensive benchmark to cater to a wide variety of languages, reflecting the diversity in real-world applications. CodeTransOcean contains three distinct datasets designed to support translations between multiple popular programming languages, niche languages, and also to assess the reliability of translated code by large language models. Moreover, it incorporates a dataset specifically for translating deep learning code across different frameworks. The authors also pioneered multilingual modeling approaches for code translation and proposed a new metric for evaluating code translation at the program level. The paper includes an evaluation of a large language model, ChatGPT, using this new benchmark, and provides baseline results and an analysis of challenges in code translation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Zhu, Ming, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni and Chandan K. Reddy. “XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence.” ArXiv abs/2206.08474 (2022): n. pag.\n[2] Rithy, Israt Jahan, Hasib Hossain Shakil, Niloy Mondal, Fatema Sultana and Faisal Muhammad Shah. “XTest: A Parallel Multilingual Corpus with Test Cases for Code Translation and Its Evaluation*.” 2022 25th International Conference on Computer and Information Technology (ICCIT) (2022): 623-628.", "Typos_Grammar_Style_and_Presentation_Improvements": "In the paper, it would be beneficial to include the main results table in the main body of the text rather than in the appendix. This change would enable readers to easily relate the experimental results to the methods and discussions as they progress through the paper."}}, {"review_id": "4SkumXCASG", "reviewer": "Reviewer_ebpA", "summary": "", "questions": "* Do you analyze whether there are overlap between CodeTransOcean and the prior code translation datasets?\n* Can you elaborate how can CodeTransOcean complement to prior code translation datasets?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our contribution of (1) providing a novel and comprehensive code translation benchmark, which is beneficial for measuring code translation capabilities of LLMs and contributes to the development of code LLMs, (2) conducting comprehensive evaluation of both open-source LLMs such as CodeT5+ and closed-source models such as ChatGPT on these datasets, and (3) proposing a new evaluation metric Debugging Success Rate@K for program-level code translation. Thank you very much for your constructive feedback and valuable suggestions. Below we address all of your questions and concerns.\n\n**Q1: Some important experimental results are in Appendix.**\n\n**Response:** \nWe structure the paper based on two considerations. Firstly, the primary contributions of this paper are introducing the new CodeTransOcean benchmark for code translation. Therefore, we allocated the majority of the main body of the paper to elaborate the motivation and the process for creating this benchmark and dataset analysis, to demonstrate the efficacy of multilingual modeling on code translation, to propose a new evaluation metric, and to report comprehensive evaluations and studies of the representative LLM ChatGPT on these datasets. \nSecondly, due to the page limit, we put some experimental details and some experimental result tables such as those of CodeT5+ (Table 10-13) in Appendix. However, **our paper is self-contained**, since for **all** these tables of experimental results that we put in Appendix, **we have included the text describing the purpose of the corresponding experiment and the detailed discussions corresponding to the results in the main body**. In the main body, for CodeT5+ results, Line 401-434 provides detailed analyses and discussions for Table 10, Line 435-447 for Table 11, Line 448-455 for Table 12 and 13, and Line 456-466 provides an overall analysis of the observations on Table 10-13. In the main body, for ChatGPT results, Line 569-580 provides detailed discussions for Table 15 of ChatGPT performance under different Top-K and Temperature parameters, and Line 592-618 details the experimental setup and results for Table 16 and Table 19, evaluating the effect of multi-round self-debugging on DSR.\nIn the revised version, we will refine the organization of the content and move the key result tables from Appendix to the main body.\n\n**Q2: Whether there are overlap between CodeTransOcean and the prior code translation datasets**\n\n**Response:** \nWe confirm that **there is no overlap between all the four datasets included in our CodeTransOcean and the existing code translation datasets**. Our work is not a simple data amplification over prior works. We provide three novel and unique datasets NicheTrans, DLTrans and LLMTrans for code translation tasks and our MultilingualTrans dataset is advantageous in larger size, more balanced data distribution across various programming languages, practicality of language pairs, and data quality compared to the existing code translation datasets (explained in lines 191-196, 259-264, 1200-1215 of the paper). For the construction of CodeTransOcean, we carefully selected data from [Rosetta Code](https://rosettacode.org/wiki/Rosetta_Code) and [Dive into Deep Learning](https://d2l.ai/) from multiple code sites. When building our datasets, we carefully compared with the data sources of other code translation datasets to ensure that **the same data sources are not used**. \nWe will add the specific data sources for each dataset in Table 2 of our paper to further clarify this point.\n\n**Q3: How can CodeTransOcean complement to prior code translation datasets?**\n\n**Response:**  \nWe elaborate on the complementary relationship between CodeTransOcean and the existing code translation datasets as follows. Prior to our work, the most comprehensive code translation dataset, CoST, covers seven programming languages. The XTest paper claims their dataset covers nine programming languages, but has not been open-sourced. In contrast, we released the MultilingualTrans dataset, which encompasses 8 popular programming languages and is advantageous in larger size, more balanced data distribution across various programming languages, practicality of language pairs, and data quality compared to the existing code translation datasets (explained in lines 191-196, 259-264, 1200-1215 of the paper). Moreover, we have also introduced the NicheTrans dataset, which covers 37 programming languages with medium or low usage rates. In addition, we have specifically designed the LLMTrans dataset for evaluating LLMs on code translation and provided the DLTrans dataset that spans four major deep learning frameworks. It is worth noting that while the MultilingualTrans dataset shares some similarities with some previous datasets, to date MultilingualTrans covers the broadest range of programming languages and the largest number of unit tests and comprises **program-level** parallel data, while the data samples in previous datasets are usually simple parallel data at the function-level. The three datasets LLMTrans, DLTrans and NicheTrans are all created for the first time, are all novel and unique, and are also **program-level** parallel data. **Therefore, CodeTransOcean covers the largest number of programming languages to date and is a valuable resource in the field of code translation, as acknowledged by Reviewer RZux and Reviewer XoXb. More importantly, CodeTransOcean fits perfectly with EMNLP's interest in \"[New data resources, particularly for low-resource languages](https://2023.emnlp.org/calls/main\\_conference\\_papers/\\#contributions)\".**\nFurthermore, we have provided evaluation tools and an automated pipeline for code translation using LLMs. To further help researchers exploit this benchmark, we have also established corresponding baselines on multiple strategies for each dataset of CodeTransOcean. \n\n**Reproducibility: 4**\n\n**Response:**\nTo ensure the transparency and reproducibility of our experiments, we have already provided all the relevant code, datasets and experimental results in the supplementary materials. We will make all datasets of CodeTransOcean, code, experimental results, and model checkpoints publicly available, so that researchers can reproduce all our results. We also created a website and leaderboard for this benchmark. To demonstrate these to reviewers, we uploaded the dataset, code, experimental results, website, and model checkpoints in [anonymous Github](https://anonymous.4open.science/r/CTO-5263/Readme.md) as a way to demonstrate the reproducibility of our work. All in all, based on our rigorous internal testings, we are confident that our experimental results are easily reproducible.\nIn addition, Line 382-383 points out that the details of multilingual modeling strategies are in Appendix A.5, where A.5 provides the details of the specific hyperparameter settings and environment used in all experiments in Table 9. The reason that the training and evaluation data has not been widely available is because the training/evaluation data is from our CodeTransOcean and we have not yet disclosed the datasets before the anonymous review period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This is a novel and comprehensive code translation benchmark, which is beneficial for measuring LLMs’ code translation capabilities and contributes to the development of code LLMs.\n* This paper provides comprehensive evaluation of both open LLMs such as CodeT5+ and closed-source models such as ChatGPT on these datasets. Besides, it proposes a new evaluation metric Debugging Success Rate@K for program-level code translation.", "reasons_to_reject": "* The presentation of the paper can be improved, some of the important experimental results are placed in the  Appendix, which violates the self-contained requirement of the main content.", "questions_to_authors": "* Do you analyze whether there are overlap between CodeTransOcean and the prior code translation datasets?\n* Can you elaborate how can CodeTransOcean complement to prior code translation datasets?", "paper_topic_and_main_contributions": "This paper introduces a comprehensive multilingual benchmark for code translation tasks. The benchmark consists of 4 new datasets including MultilingualTrans for  translations between multiple popular PLs, NicheTrans for translating between niche PLs and popular ones, LLMTrans for evaluating compilability of translated code by LLMs, and DLTrans for translating deep learning code across different frameworks. This paper also proposes a new evaluation metric Debugging Success Rate@K for program-level code translation and evaluates both open LLMs such as CodeT5+ and closed-source models such as ChatGPT on these datasets. Given its large scale and comprehensiveness, this benchmark would be beneficial for measuring models’ code translation capabilities. A minor downside of this paper is that lots of results are placed into the Appendix, which violates the self-contained requirement of the main content.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The presentation of the paper can be improved, some of the important experimental results are placed in the  Appendix, which violates the self-contained requirement of the main content."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces comprehensive new data sets for code translation between many different programming languages.

Pros:
- Valuable new data sets, high number of programming languages covered. Much larger than other data sets in the space.
- Detailed experiments involving LLMs and code translation using these new data sets.
- Inclusion of even "niche" programming languages like Fortran, for which code translation can be useful as part of modernization of legacy systems.
- Reproducibility and extensibility: Datasets, code, experimental results, and model checkpoints are openly available, enabling further research down the line.
- Well structured and easy to follow.

Cons:
- Some concerns about missing comparisons with other data sets, and unclarity around the selection of the models that were chosen for initially evaluating the benchmark on. This can be addressed in a minor revision.

The main focus of this paper is the creation of the new data sets, not the experimental results on various models, so it's not a problem that the tables with the detailed experimental results only appear in the appendix, given that the top-level analysis of these experiments appears in the main body of the text.
Decision: Accept-Findings