AVATAR: A Parallel Corpus for Java-Python Program Translation
Year: 2023

Abstract:
Program translation refers to migrating source code from one programming language to another. It has tremendous practical value in software development, as porting software across languages is time-consuming and costly. Automating program translation is of paramount importance in software migration, and recently researchers explored unsupervised approaches due to the unavailability of parallel corpora. However, the availability of pre-trained language models for programming languages enables supervised fine-tuning with a small number of labeled examples. Therefore, we present AVATAR, a collection of 9,515 programming problems and their solutions written in two popular languages, Java and Python. AVATAR is collected from competitive programming sites, online platforms, and open-source repositories. Furthermore, AVATAR includes unit tests for 250 examples to facilitate functional correctness evaluation. We benchmark several pretrained language models fine-tuned on AVATAR. Experiment results show that the models lack in generating functionally accurate code.

heading: 1 Introduction
text: Software developers and researchers often require to convert software codebases or research prototypes from one platform to another or rewrite them in the target programming languages. Manually rewriting software is time-consuming, expensive, and requires expertise in both the source and target languages. For example, the Commonwealth Bank of Australia spent around $750 million and 5 years translating its platform from COBOL to Java (Lachaux et al., 2020). A program translation system that converts the source code of a program written in a programming language to an equivalent program in a different programming language is known as a transcompiler, transpiler, or source-tosource compiler. Transcompilers have a prodigious practical value; they could help to reduce the translation efforts of developers and researchers by not
requiring them to write code from scratch, instead, they can edit the translated code with less effort.
The conventional transcompilers are based on rule-based approaches; they first convert source code into an Abstract Syntax Tree (AST) and then apply handwritten rules to translate to the target language. Development and adaptation of transcompilers need advanced knowledge and therefore are available in a handful of programming languages. Undoubtedly, the automation of program translation would facilitate software development and research tremendously.
With the recent advancements in data-driven neural machine translation (NMT) approaches between natural languages, researchers have started investigating them for programming language translation. Lachaux et al. (2020) trained an NMT system in an unsupervised fashion using large-scale monolingual source code from GitHub that showed noteworthy success in source code translation between Java, Python, and C++ languages. Pre-trained language models (PLMs) of code have been shown to work well on Java-C# translation after fine-tuning on a small amount of parallel examples (Feng et al., 2020; Guo et al., 2021; Ahmad et al., 2021; Wang et al., 2021). Motivated by these favorable results, in this work, we propose a new parallel corpus of Java and Python programs.
We propose a corpus, AVATAR (jAVA-pyThon progrAm tRanslation) that consists of solutions written in Java and Python for 9,515 programming problems collected from competitive programming sites, online platforms, and open source repositories. AVATAR includes 250 examples with unit tests to facilitate functional correctness evaluation of program translation. We train several baselines, including models trained from scratch or pre-trained on large-scale source code collection and fine-tuned on AVATAR. The experiment results indicate that while the models perform considerably in terms of the lexical match, they lack Fur-
2268
thermore, AVATAR offers 3,391 parallel functions that we use to train models or fine-tune pre-trained language models and perform function translation evaluation on the dataset released by Lachaux et al. (2020). Our code and data are released at https: //github.com/wasiahmad/AVATAR.

heading: 2 AVATAR Construction
text: Data Collection We construct AVATAR based on solutions of computational problems written in Java and Python collected from open source programming contest sites: AtCoder, AIZU Online Judge, Google Code Jam, Codeforces, and online platforms: GeeksforGeeks, LeetCode, Project Euler. We crawl Codeforces and GeeksforGeeks sites to collect the problem statements and their solutions. We collect the AtCoder and AIZU data from Puri et al. (2021), Google Code Jam data from Nafi et al. (2019)1, and LeetCode and Project Euler problem solutions from open source Github repositories.2,3 We collect [1 – 20] accepted solutions for a single problem written in Java and Python. Preprocessing & Filtering At first, we tokenize the solution code and remove docstrings and comments from them. We use the javalang4 tokenizer for Java and the tokenizer5 of the standard library for Python. After tokenization, we filter out solutions that are longer than a specified
1https://github.com/Kawser-nerd/ CLCDSA
2https://github.com/qiyuangong/ leetcode
3https://github.com/nayuki/ Project-Euler-solutions
4https://github.com/c2nes/javalang 5https://docs.python.org/3/library/
tokenize.html
length threshold (= 464). In the initial data collection, there are [1 – 20] accepted solutions for each problem. We filter out solutions and only keep at most 5 solutions per problem. Our goal is to keep the solutions that are maximally different from others in order to increase diversity among solutions of the same problem. We use the open source library difflib6 to compare all the solutions pairwise (individually in Java and Python) and select five solutions that differ most from others.
Data Statistics We split 9,515 problem statements into a 75:5:20 ratio to form 7,133 training, 476 validation, and 1,906 test examples. Table 1 summarizes the data statistics. Since we collect [1 – 5] accepted solutions for each problem statement in both languages, we form [1 – 25] parallel examples per problem for training. In evaluation, we use multiple ground truths and select the best performance according to the evaluation metrics.
Unit Tests AVATAR presents unit tests for 250 evaluation examples (out of 1,906) to perform functional accuracy evaluation of the translation models. The unit tests are collected from the publicly available test cases released by AtCoder.7
Parallel Functions AVATAR includes 3,391 parallel Java and Python functions.8 The functions are extracted by parsing programs that include only one function. We use them for training models and evaluating using the dataset released by Lachaux et al. (2020).
6https://docs.python.org/3/library/ difflib.html
7https://atcoder.jp/posts/21 8Deduplicated against the evaluation dataset released by Lachaux et al. (2020) using https://github.com/ microsoft/dpu-utils.

heading: 3 Experiment & Results

heading: 3.1 Evaluation Metrics
text: BLEU computes the overlap between candidate and reference translations (Papineni et al., 2002).
Syntax Match (SM) represents the percentage of the sub-trees extracted from the candidate program’s abstract syntax tree (AST) that match the sub-trees in reference programs’ AST.
Dataflow Match (DM) is the ratio of the number of matched candidate data-flows and the total number of the reference data-flows (Ren et al., 2020).
CodeBLEU (CB) is the weighted average of the token level match, syntax level match (SM), and Dataflow match (DM) (Ren et al., 2020).
Execution Accuracy (EA) indicates the percentage of translated programs that are executable (results in no compilation or runtime errors).
Computational Accuracy (CA) Lachaux et al. (2020) proposed the metric to evaluate whether the candidate translation generates the same outputs as the reference when given the same inputs.

heading: 3.2 Models
text: We evaluate a variety of models on program and function translation using AVATAR and the evaluation dataset released by Lachaux et al. (2020).
Zero-shot This set of models is evaluated on AVATAR without any training or fine-tuning.
• TransCoder is pre-trained in an unsupervised fashion that can translate programs between Java, Python, and C++ languages (Lachaux et al., 2020).
• DOBF uses deobfuscation pretraining followed by unsupervised translation (anne Lachaux et al., 2021).
• TransCoder-ST is developed by fine-tuning TransCoder on a parallel corpus created via an automated unit-testing system (Roziere et al., 2022).
Models trained from scratch These models are trained from scratch using AVATAR. We use the sentencepiece tokenizer and vocabulary from Ahmad et al. (2021) in these models.
• Seq2Seq+Attn. is an LSTM based sequence-tosequence (Seq2Seq) model with attention mechanism (Bahdanau et al., 2015).
• Transformer is a self-attention based Seq2Seq model (Vaswani et al., 2017). We use the Transformer architecture studied in Ahmad et al. (2020).
Pre-trained Models We evaluated three types of pre-trained models (PLMs). First, we evaluate decoder-only PLMs (e.g., CodeGPT) that generate auto-regressively. The second category of PLMs is encoder-only (e.g., CodeBERT). We use a randomly initialized decoder to finetune such models in a Seq2Seq fashion. The third category of PLMs is Seq2Seq models (e.g., PLBART), which we directly finetune on translation tasks. • CodeGPT and CodeGPT-adapted are GPT2 (Radford et al., 2019) style models pre-trained on CodeSearchNet (Lu et al., 2021). Note that CodeGPT-adapted starts from the GPT-2 checkpoint, while CodeGPT is pre-trained from scratch. • CodeBERT is an encoder-only model that is pre-trained on unlabeled source code via masked language modeling (MLM) and replaced token detection objectives (Feng et al., 2020). • GraphCodeBERT is pre-trained using MLM, data flow edge prediction, and variable-alignment between code and its’ data flow (Guo et al., 2021). • PLBART is a Transformer LM pre-trained via denoising autoencoding (Ahmad et al., 2021). • CodeT5 is a Transformer LM pre-trained via identifier-aware denoising (Wang et al., 2021).
In addition, we fine-tune TransCoder-ST, which is the best translation model in the literature.

heading: 3.3 Hyperparameters Details
text: We individually fine-tune the models for Java to Python and Python to Java program and function translation, respectively. We fine-tune the models for a maximum of 20 epochs using the Adam (Kingma and Ba, 2015) optimizer with a batch size of 32. We tune the learning rate in the range [1e − 4, 5e − 5, 3e − 5, 1e − 5]. The final models are selected based on the validation BLEU score. We use beam decoding with a beam size set to 10 for inference across all the models.

heading: 3.4 Results
text: Program Translation The performance comparison of all the experiment models is presented in Table 2. In general, all the models perform well in terms of match-based metrics, e.g., BLEU and CodeBLEU. However, the computational accuracy (CA) clearly indicates that these models are far from perfect in generating functionally accurate translations. Overall, the best-performing model is PLBART, resulting in the highest execution accuracy (EA) and CA in Java to Python translation.
Note that the zero EA score of TransCoder, DOBF, and TransCoder-ST in Python to Java translation is due to not generating a class correctly that fails execution of all translated programs.
Function Translation The performance comparison of all the experiment models is presented in Table 3. Apart from models trained from scratch, CodeBERT, and GraphCodeBERT, all the models perform well in terms of match-based metrics, execution, and computational accuracy. Overall, the best-performing model is fine-tuned TransCoderST, and PLBART is the closest competitor model.

heading: 3.5 Analysis
text: Execution-based Evaluation Breakdown We present the breakdown for the test-case-based eval-
uation in Table 4 (in the Appendix). We present the number of success, failure, and error counts. For program translation evaluation, AVATAR consists of 250 evaluation examples with unit tests. For function translation evaluation, we use the test examples released by (Lachaux et al., 2020). Among the examples, 464 Java to Python and 482 Python to Java examples have test cases. We further present the compilation and runtime error breakdown in Table 5 (in the Appendix).
To analyze program translation errors, we manually examine the errors made by PLBART. We observe that PLBART does not generate the import statements in Java properly, resulting in many failures to find symbols (e.g., StringTokenizer, BufferedReader). Moreover, a quick look at the error made by all models reveals that type mismatch is one of
the primary causes of compilation errors in all the models. We also notice that models fail to translate longer programs.
Qualitative Examples We demonstrate a couple of qualitative examples of Java to Python program translation by PLBART in Figure 1. We observe that PLBART correctly translates Java API Math.pow() to pow() in Python. We also observe that PLBART learns to translate a class with a function in Java to a function only in Python.
In Figure 2, we present an example of Python to Java program translation. We see PLBART fail to translate correctly. We notice PLBART unnecessarily generates InputReader class that uses BufferedReader to read from standard input. Furthermore, we observed another behavior: when translating from Python to Java, PLBART generates classes with the name either Main or GFG. This is presumably due to the generic class name used in many programming solutions and GeeksforGeeks examples.
We present qualitative examples of Java to Python and Python to Java function translation by PLBART in Figure 3 and 4. Overall, we observe a pretty good quality of translations, although there are translations that do not pass all the unit tests, as demonstrated by the performance in terms of computational accuracy in the main result.

heading: 4 Related Works
text: Several works in the past have contributed to building a parallel corpus for source code translation. Nguyen et al. (2013) curated the first parallel corpus of Java and C# functions by developing a semiautomatic tool to search for similar class names and method signatures from two open source projects, Lucene and Db4o. Similarly, Karaivanov et al. (2014) built a mining tool that uses the Java and C# ANTLR grammar to search for similar methods from five open source projects - Db4o, Lucene, Hibernate, Quartz, and Spring. Subsequent works used libraries and transcompilers to construct parallel corpus. For example, Aggarwal et al. (2015) used 2to3, a Python library9 and Chen et al. (2018) used a transcompiler to create a parallel corpus between Python 2 – Python 3 and CoffeeScript – Javascript, respectively. Recently, Lachaux et al. (2020) collected programming problem solutions in Java, Python, and C++ (∼850 func-
9https://docs.python.org/2/library/ 2to3
tions in each language) from GeeksforGeeks to evaluate their proposed translation model. Concurrent works (CodeGeeX, 2022; Athiwaratkun et al., 2023) present unit tests-based benchmarks to evaluate zero-shot translation capabilities of large language models. Different from these works, we propose a sizeable parallel corpus of Java and Python programs by collecting programming problem solutions from competitive programming sites, online platforms, and open-source repositories.

heading: 5 Conclusion
text: This work proposes a parallel corpus of Java and Python programs to contribute to the development of translation systems for programming languages that have a sizeable impact on software development. We evaluate several neural machine translation systems on the proposed dataset and perform analysis to reveal crucial factors that affect program translation accuracy. In our future work, we want to increase the size of the parallel corpus and support more programming languages.

heading: Limitations
text: The proposed benchmark has a few limitations. First, AVATAR has a smaller training data size which limits training deep neural models from scratch. Second, the dataset covers only two programming languages. Third, AVATAR includes parallel examples of programs and functions that mostly focus on the use of data structures and algorithms. On the other hand, most software developers write programs as part of software projects that include API dependencies. Therefore, it is unknown whether AVATAR could facilitate program or function translation for such settings. Due to a lack of computational resources, we could not evaluate large language models (LLMs) (Nijkamp et al., 2023; Fried et al., 2023; CodeGeeX, 2022). Therefore, it is unknown how much AVATAR could bring value for LLMs. However, our code release would help to evaluate LLMs.

heading: Ethics Statement
text: License The LeetCode examples we crawled from the GitHub repository are under an MIT license. On the other hand, Project Euler and Code Jam examples collected from GitHub do not have any license information. The AtCoder and AIZU examples are collected from CodeNet
which is under Apache-2.0 license. We crawl examples from GeeksforGeeks and Codeforces and release them under CC BY-NC-SA 4.0 license. To use the AVATAR benchmark, we are required to adhere to these licenses strictly. Carbon Footprint We avoided fine-tuning large models due to computational limitations, resulting in a reduced impact on the environment. We finetuned nine models on program and function translation tasks and due to the smaller size of the training data, all jobs took a total of 1–2 days on RTX 2080 Ti GPUs. A total of 100 hours of training in a single RTX 2080 Ti GPU results in approximately 7.5kg of carbon emission into the environment.10
Sensitive Information AVATAR composed of parallel programs and functions that do not have any natural language (NL) comments or docstring. We remove them to get rid of any personally identifiable information or offensive content. However, there could still be such content in the form of string as we do not manually check each example.

heading: Acknowledgements
text: We thank the anonymous reviewers for their insightful comments.

heading: JAVA TO PYTHON TRANSLATION

heading: INPUT PROGRAM IN JAVA

heading: JAVA TO PYTHON TRANSLATION

heading: INPUT PROGRAM IN JAVA

heading: PYTHON TO JAVA TRANSLATION

heading: INPUT PROGRAM IN PYTHON

heading: INPUT FUNCTION IN JAVA

heading: INPUT FUNCTION IN JAVA

heading: INPUT FUNCTION IN JAVA

heading: INPUT FUNCTION IN PYTHON

heading: INPUT FUNCTION IN PYTHON

heading: INPUT FUNCTION IN PYTHON
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section after conclusion
3 A2. Did you discuss any potential risks of your work? In the limitations section.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2
3 B1. Did you cite the creators of artifacts you used? Section 2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Ethics Statement
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics Statement
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2
C 3 Did you run computational experiments? Section 3 and Appendix
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Ethics Statement
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We will release the source code.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We will release the source code.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In our future work, we want to increase the size of the parallel corpus and support more programming languages