Data Augmentation for Code Translation with Comparable Corpora and Multiple References
Authors: Yiqing Xie
Atharva Naik
Daniel Fried
Carolyn Rose
Venue: EMNLP 2023 Findings (2023)
Keywords: Code translation
Machine learning for code
TL;DR: Improve code translation by training on comparable corpora and extending the finetuning set with additional translation references.

Abstract: One major challenge of translating code between programming languages is that parallel training data is often limited. To overcome this challenge, we present two data augmentation techniques, one that builds comparable corpora (i.e., code pairs with similar functionality), and another that augments existing parallel data with multiple reference translations. Specifically, we build and analyze multiple types of comparable corpora, including programs generated from natural language documentation using a code generation model. Furthermore, to reduce overfitting to a single reference translation, we automatically generate additional translation references for available parallel data and filter the translations by unit tests, which increases variation in target translations. Experiments show that our data augmentation techniques significantly improve CodeT5 for translation between Java, Python, and C++ by an average of 7.5\% Computational Accuracy (CA@1), which verifies the correctness of translations by execution. The code is available at https://github.com/Veronicium/CMTrans.

Data Augmentation for Code Translation with Comparable Corpora and Multiple References
Yiqing Xie
Atharva Naik Daniel Fried Carolyn Rosé Language Technologies Institute Carnegie Mellon University
(yiqingxi, arnaik, dfried, cprose cmu. edu
Abstract
CMTrans
Finetune on Multiple references
Train on Comparable corpora
One major challenge of translating code be- tween programming languages is that parallel training data is often limited. To overcome this challenge, we present two data augmen- tation techniques. one that builds comparable corpora (i.e., code pairs with similar function- ality), and another that augments existing par- allel data with multiple reference translations. Specifically, we build and analyze multiple types of comparable corpora, including pro- grams generated from natural language docu- mentation using a code generation model. Fur- thermore, to reduce overfitting to single ref- erence translation, additional translation references for available parallel data and filter the translations by unit tests, which increases variation in target transla- tions. Experiments show that our data augmen- tation techniques significantly improve CodeT5 for tramslation between Java, Python, and C++ by an average of 7.5% Computational Accu- racy (CA@1 1), which verifies the correctness of translations by execution.
Denoising Pretrain
Finetune on Parallel
Test
Standard pipeline (E.g., CodeT5)
Figure 1: The standard pipeline for code translation and the pipeline of CMTrans. The comparable corpora are both naturally occurring and model generated. We generate multiple references by our method.
automatically generate
either in the same language between languages. We show an example of parallel data in Figure 2. One major challenge of code translation is that parallel data is typically limited. For instance, the TransCoder dataset (Roziere et al., 2020) only con- tains 466 Python C++ pairs. Constructing parallel data requires substantial human effort and cannot be easily scaled. With limited fine-tuning examples, it is difficult for model learn functional equiva- lence across programming styles and domains.
1 Introduction
To mitigate the data scarcity issue, we hypoth- esize that the functional equivalence between lan- guages can also be learned from comparable cor- pora, a term we borrow from natural language translation, where it refers to texts similar top- ics in different languages (Gete and Etchegoyhen, 2022: Irvine, 2014). Here, we use it to refer programs with similar functionality different lan- guages (Ahmad et al., 2021b; Khan al., 2023). As shown in Figure 2, although the programs paired in a comparable example may have different algo- rithms and structures (c.g.. global code vs. class functions), they are likely to have similar constructs and may even have some lines that can be matched.
Code translation a special type machine trans- lation that translates between programming lan- guages. is widely applied in software engineering to migrate a codebase into another programming language. Recent code translation models typically follow the pretrain finetten pipeline, shown in Figure I. In pretraining, with denoising objectives such masked span identifier prediction (Ah- mad et al., 2021a; Wang et al., 2021; Roziere et al., 2020), the model learns to produce sequences in both languages. When finetuned on parallel data (Ahmad et al., 2021b), which are program pairs in the source and target language that are aligned line- by-line, the model learns functional equivalence: identifying programs with the same functionality,
In this paper, we study what the model learns from comparable corpora by building three types of comparable examples: (1) Naturally existing, where we leverage independently-writtee solutions
Code available at http://ggitbb.coom/eeonnmmm////

---

board "
Source (Python)
Stringl] board new Stzingl8]s Soolean check Ialoe;
Parallel (Java)
import Jeva.utt2,";
Comparable (Java)
check Faloo
import Lava.long."5
Scannerr Bysten.in
import java.io.*)
range (8)s
for( int 07
pablic
Ioeone
board.aggend) wtr (er(it)t)))
board[i] in-next([]:
public static void naint Beringt] arge I( Scannaz in now Ecansoer Systen. )
Cox x in board:
foz (int 1 0, 1 8, it-ld
String hoard
int f =
11 "BB" In or "W6" in xt
if (8.consaiini (nsonnaiinn"BB0 ("B8")
k.costaina| "M6")
[as( int 1- 1 a It+ H
print check
Systen.out.peiatae "90"
String in.nextLine())
check trees
break
break;
for int 3 K 5a+ H 12 ( 6.charAe(2) (.0harAE(2+1) H
if x 1- "RM" +4:
14 (Ix.equalnt ("mm" "mm" (regent(()))
if i- "Mar*4:
if (Ix.equala "ip" (1
print( "BO"
Bysten.out. printinn "30" traes
break
break;
iejt 01 Syatem..ut..praaaee "EaS"
clse cesten.out.prinnae. "NO" 11
if check Fallee:
if (chack falan)
"VES"
Syatez..u. printing "LES"
Figure 2: An example of parallel and comparable data. Parallel examples are line-by-line aligned. Programs in a comparable example may have different algorithms and structures (e.g., global code vs. class in this case), but may still contain lines that can be matched, as highlighted in yellow, green, and blue.
of the same coding problem; (2) Generated, where we collect programs with docstrings in one lan- guage and apply a code generation model to gen- erate programs in another language: and (3) Re- trieved, where we either retrieve program's 's k nearest neighbors (KNN) or simply choose a ran- dom program in another language. Among them, (1) contains cleaner examples. which are guaran- teed to be bug-free.
1, which is initialized from the same pretrained checkpoint, for an average of 7.5% Computational Accuracy (CA @ 1) over translation between 6 lan- guage pairs. CMTrans also significantly outper- forms the state-of-the of--tt--rtt-tttt method in out of 6 lan- guage pairs, while reaching parity on the other one. Analyses of our two techniques suggest that: (1) All three types of comparable corpora (including random program pairs) improve the syntax accu- racy and perplexity of the translation outputs and lead to better final performance.
(3) covers programs from a larger variety of sources, providing more diverse training signals.
(2) Both natu- rally existing and generated comparable corpora help the model generate constructs that match the input. The combination of them gives the largest performance gain. (3) By training with multiple references, the model generates more unique cor- rect translations within a certain budget, which indicates better functional equivalence of the target language is learned.
In addition to the functional equivalence between languages, the model should also learn the equiva- lence between different programs in the target lan- guage. This is challenging due limited finetuning data. Furthermore, the majority of finetuning data only have one reference translation (c.g., 82.5% in AVATAR, Ahmad et al. 2021b), which is likely to cause overfitting to a single translation without fully capturing its semantics.
Contributions. (1) We build and study three types of comparable corpora for code translation. (2) We improve the modeling of target language by generating, verifying, and selecting additional ref- erences for existing parallel translation data. (3) We demonstrate that our model significantly outper- forms tfate-of-hhe-an methods on 5 language pairs. Our analyses provide insights into what the model learns in code translation for future researchers.
As result, in this paper, we generate multi- ple translation references for the finetuning data. Specifically, after training on comparable corpora, we finetune model on the original parallel data, generate multiple translations for each example, and use automatically generated test cases to filter out incorrect translations. By training on different functionally equivalent programs, we reduce over- fitting to a single output and improve the modeling of the target language.
2 Related Work
Combining the two techniques, we name our full approach CMTrans, a code Translation model trained with Comparable corpora and Multiple ref- erences. Extensive experiments show that CM- Trans significantly improves CodeT5 (Wang et al.,
Code Translation. Previous work has tackled the problem of translating code written in one program- ming language to another. Karaivanov et al. (2014), Nguyen et al. (2013, 2015), Phan et al. (2017); Oda

---

al. (2015) applied statistical machine translation techniques to code translation, while Chen al. (2018) introduced a tree-to-tree neural translation approach. Further improvements were achieved by pre-trained language models code such as Code- BERT (Feng et al., 2020), PLBART (Ahmad et al., 2021a), and CodeT5 (Wang et al., 2021). However, the above approaches require finetuning parallel data, which is often scarce.
why comparable corpora are beneficial.
Another data augmentation strategy is using mul- tiple translation references. Qin and Specia (2015) and Khayrallah et al. (2020) found that using mul- tiple references can be beneficial in low-resource settings. An advantage of working with code is that test cases can be used to filter model-generated translated references to ensure functional equiva- lence the source, which we exploit in CMTrans.
Data Scarcity in Code Translation. To tackle the data scareity issue, TransCoder (Roziere al., 2020) uses back translation for unsuper- vised code translation. DOBF (Lachaux et al., 2021), TransCoder-ST (Roziere et al., 2022), and S&G (Ahmad et al., 2022) respectively improve TransCoder with de-obfuscation pre-training, self- training, and pairing up the model with generation and a summarization model. However, the best- performing approach, TransCoder-ST,, is only able generate parallel data for standalone functions where the model can already generate correct so- lution within a limited budget. contrast, the com- parable corpora we use to train CMTrans include code with arbitrary structure and content. CMTrans also has much better efficiency, as self-training re- quires running a much larger number of test cases than generating multiple references. MultiPL-E (Cassano et al., 2022) also automatically generates test cases, but retries until the output passes all test cases. We do not directly compare to since requires multiple rounds of translation.
3 Methodology
In this section, we introduce our data augmentation method that trains the model first on comparable corpora, which are either naturally existing, gener- ated, or retrieved (Section 3.2) and then on parallel data with additional references generated by our model (Section 3.3).
3.1 Problem Formulation
We formulate code translation as a sequence-to sequence (Seg2Seq) generation problem following existing work (Lu al., 2021; Ahmad al., 202 lb; Roziere et al., 2020). The input is the code in the source language
{s1, 82; .2..... 8n.). which is se- quence of tokens. We apply a model to translate the MM(())
input code into the target language {h, ..., ..., 4m J. where the model Mo is typically finetuned for code translation. Alternatively, the model may generate k candidate outputs for each input: Ma(S, k) T ( (())..
3.2 Training on Comparable Corpora
Data Augmentation for Natural Language Translation. The lack of parallel training data is a fundamental problem in the field of transla- tion, which has led the NLP community to develop several data augmentation techniques in response. Comparable Corpora, as defined by Munteanu and Marcu (2005), *are texts that, while not parallel in the strict sense, are somewhat related and convey overlapping information". Paramita et al. (2013) and Wolk et al. (2015) present methods for collect- ing comparable corpora to study when and how to use them. Etchegoyhen and Gete (2020) and Gete and Etchegoyhen (2022) identify information balance, alignment, and length differences between source and target as key factors affecting translation quality. In this work, we extend the study of com- parable corpora to code translation. We show that code translation can benefit from multiple types of comparable corpora even if there is already high- quality parallel data. We also provide analyses on
As shown in Figure 3, we study three types of com- parable corpora: naturally existing, generated, and retrieved ones. Here we introduce how we build each type of comparable examples and train model on them. We provide details of the compara- ble corpora datasets we use or build in Section 4.1 and provide examples in Appendix A.3.
Naturally Existing Comparable Examples. We make of comparable corpora collected from programming contests by existing datasets (Ahmad et al., 2021b; Khan al., 2023; Puri al., 2021). which mainly consist of solutions to the same con- test problems written in different languages. These comparable examples are typically confined to spe- cific domains such as dynamic programming or graph theory.
Generated Comparable Examples. To cover programs in more diverse domains, we present a method that automatically generates comparable

---

Problem: 9-6 board. uach square black
for Java
Codo
XWA Comp.
1 Java
Ongy
Input
Generation
Model
Stop candidate
Stop tost generation
sococa
genaration
PyL
static
int mandua Ant
arr Lot
public clase avi
the giver säsk
P22
Bin-
pabits acasse
Rocatas Eyatum, in1p (ist 100p ..
pibl
acsg copytol
cut
python
Comp.
tmmalation
Our codo
Pyto
Error
Cloase closus Closer.cnentel It
read it reader.crmater we ertertttttttr while conlitkr
python
Py2
loxrd for canguct)y
Reader cunder cicser. coçiatere
10:2
Sm [10, 4. 13
X
Py5
Pv1
sareeddappnddddde (epett))
uperBtreond))
rend
** java
[4.7]
acd
Pg
18 "to" in 1. us
keitar veicee
i d putton
Pyto
PyB
Stop candidate fitarinng
4 reforance selection
Natually Existing
Train on Comparable Corpora
Finatune with Multiple References
Extend by CMTrans
Denoising Pretraining
Finetune on Parallel Data (Majority: single reference)
Test
Translation
Self-train Unit Tests (single reference)
Figure 3: An example of CMTrans for Java-to-Python translation. We compare the pipeline of CMTrans to the standard pipeline of code translation (e.g.. CodeTS, Wang et al. 2021) and the self super-supeiiio----ee and fine-tuning method of TransCoder-ST (Roziere et al. 2022).
examples using natural-language-tooco genera- tion models.
do not contain any information on functional equiv- alence and allow us to better understand whether comparable corpora can improve the modeling of the target language and hence improve the transla- tion quality.
Specifically, we leverage a monolingual corpus of functions with natural language documentation (i.c., docstrings) to describe their functionality. In our experiments, we use GitHub functions with docstrings extracted by CodeSearchNet (Husain et al., 2020). For each function S. in the cor- pus, we feed the natural language documentation to code generation model finetuned in the other language, which is trained to generate program based on the given natural language description. Similarly to code translation, the code generation model can generate multiple candidate outputs (T("), ( ?!))) ). We then select the one with the highest probability, which is paired up with the extracted program as comparable example.
Model Training. After constructing corpus of comparable examples D. (SS..T.), we input the program in the source language S. into our model MM and maximize the log-likelihood of its corresponding program in the target language To = [tc,1, te,2, t.
LMT(Dc) Σ
So(T)(Se)
(Se.T.)DD
Po(T))S)) Σ log (Po (Se, ... tcg-1))
(1)
Retrieved Comparable Examples. To study the effect of the quality comparable corpora, we further build KNN Comparable Corpora. We compute the embedding of all the programs in the source and target language in the dataset us- ing finetuned code retrieval model (Husain et al., 2020). For each program S in the source language with embedding emb(S), we retrieve its k nearest neighbors in the target language by the cosine sim- ilarity between their embeddings: SSm( = (emb(S), emb(T)).
Here 0 is the parameters of Ma, which is initial- ized from an encoder-decoder model pretrained on code (Wang al., 2021). After training the model on the corpus comparable examples D. till conver- gence, we finetune on the dataset of parallel ex- amples Dp. where the loss (Mr(D,) is computed by Equation as well.
By maximizing the probability of the target Te the model will learn generate fluent code in the target language. Furthermore, in general, programs in a comparable example often exhibit the same types of constructs. In the examples Figure 2, to check whether a board is valid, no matter what algorithm is used, the program will always need
Finally, we build Random Comparable Cor- pora by pairing random programs in the source and target language. In principle, such program pairs

---

Train (Comparable)
Train (Parallef)
Test
Datasel -
Gen-comp
XCodeEval
A-ATAR-comp
AVaTAR-para
Transtoder-test
C++ es Java
4,053
3,414
3.226
482 C-to-J, 467 J-toCC
# C++ es Python
4,053+
4,376
3,226
464 C-w-PP 467 P-to-C
#JJava 69 Python
22,181*
5,937
3,391
464 J-to-P, 482 P-to-J
Github (Java + Python) AIZU, AtCoder (Others)
AtCoder, Codeforces, ProjectEuler, CodeJam, GeeksforGeeks,, LeetCode,
Source
Codeforees
GookstorGecks
GoeksforGocks
Table I: Number of problems per dataset. "Gen-comp" is the comparable corpora dataset we generate. C-to-J, C-to-P, denote the test examples of C++-toJJaa,, Ct++to-Python translation, etc. * denotes data we build.
a loop to read the board and use statements for the validity check. As a result, in principle, the model can also learn to generate the same types of constructs as the input program S., which is beneficial for generating accurate translations.
Experiments
In this section, we conduct experiments answer four research questions: (RQ1) How do CMTrann and its ablations perform compared against state- of-the-art approaches for code translation? (RQ2) What can the model learn from comparable cor- pora? (RQ3) What can the model learn from multi- ple references? (RQ4) How is CMTrans affected by the size of comparable corpora and the number of references?
3.3 Finetuning with Multiple References
In addition to comparable corpora, we also provide our model with more diverse training signals by finetuming with multiple references. By providing the model with programs with the same functional- ity, we encourage the model to learn a better repre- sentation space for the target language and hence benefit the translation.
4.1 Experimental Setup
We initialize CMTrans from the pretrained check- point of CodeT5 (Wang et al., 2021), an encoder- decoder model pretrained on code files from varied languages with denoising objectives.
Since the majority of source programs only have one reference in existing datasets of parallel ex- amples (Ahmad et al., 2021b), we apply a series of steps to generate additional references, which are illustrated in Figure 3. The first step to fine- tune a model with the original parallel data, and then use the finetuned model to generate multi- (8))
Datasets. We list the dataset statistics in Table 1. All the methods are evaluated on the TransCoder- test dataset (Roziere et al., 2020). We train our method on xCodeEval (Khan et al., 2023), a compa- rable corpora dataset, and AVATAR (Ahmad et al., 2021b), which contain both comparable corpora and parallel functions (denoted as AVATAR-comp and AVATAR-pare respectively). All the super- vised baseline methods are finetuned on AVATAR- para before evaluation.
ple translation candidates (T!"), TT(2) !!) for each source program in the parallel data.
In the second and third steps, similar to TransCoder-ST (Roziere et al., 2022), we lever- age automatically generated unit tests to filter out candidates with different behaviors as the source program. Specifically, we extract the input argu- ments of the source programs, randomly produce set of test inputs based the argument types, and feed the test inputs the source program. We filter out test inputs that cause compilation or runtime errors. Finally, we feed the remaining test inputs to each candidate and only keep the ones that have exactly the same output as the source program.
Since AVATAR -praa does not contain C++ func- tions, we add a parallel C++ function for each train- ing example in AVATAR- Specifically, we generate 50 C++ translations for each Java func- tion by TransCoder-ST ST, TransCoder, and finetuned CodeT5 and filter the translations with unit tests. Construction of Comparable Corpora. We con- duct the study on different types of comparable cor- pora (Comp-Corp) on Java < Python translation. We use AVATAR- -ompp as the naturally existing Comp-Corp. KNN and random Comp-Corp are also retrieved from AVATAR-comp. To build the
Notice that some of the translations may only have small differences (c.g., i++ vs. i+=1). To obtain a diverse subset of references, select the most distinct k translations for each source pro- gram by their string edit distance. These trans- lations are added as additional references to the finetuning set.
Theree are two versions of AVATAR-comp with slight dif- ferences. We use the first version because most of our experi- ments were finished hefore the second version was released.

---

Java-to-Python
Python-to-Java
Avg Pairs
Model 4
BLEU
CB
@
BLEU
CB
CA@1
BLEU
CB
@1
TransCoder (Roziere al., 2020)
72.4
67.9
49.1
65.4
70.7
35.7
72.0
75.0
51.7
DOBF (Lachaux 2021)
72.2
67.5
52.2
67.7
71.2
44.4
TransCoder-ST (Roziere al., 2022)
73.1
68.7
68.5
70.0
71.9
71.3
74.9
66.3
CodeBERT (Feng et al., 2020)
52.0
48.9
10.4
45.4
45.0
4.2
CodeTS (Wang et al., 2021)
79.4
72.5
61.0
79.0
75.9
52.7
83.6
80.0
62.6
PLBART (Ahmad et al., 2021a)
79.9
73.2
68.9
80.5
76.8
57.5
TransCoder-STT (Roziere al., 2022)
79.3
72.9
69.4
81.4
78.4
62.0
81.8
80.2
67.6
CMTraas
80.1
74.2
73.5
84.3
82.1
66.0
84,9
82.0
70.1
Table 2: Java-Python translation results on TransCoder-test. We copy the results of all the baselines reported by Ahmad et al. (2021b). CB and CA@ stand for CodeBLEU and Computational Accuracy. We highlight the best results under each metric with Bold and underline the second-best results.
C++-to-Java
C++itu Python
Juva-toC++
Python-te-C++
Model
BLEU
CB
CA@1
BLEU
CB
CA@1
BLEU
CB
CA@1 1
BLEU
CB
CA @1
TransCoder (Roziere et al., 2020)
84.0
86.7
65.1
75.2
73.4
47.1
83.6
85.4
79.8
51.6
65.7
32.6
IransCoder-ST (Roziere et al., 2022)
78.8
85.2
68.0
73.1
73.0
61.3
76.7
83.7
84.6
55.8
67.1
56.7
CodeTS (Wang et al., 2021)
90.9
90.0
65.1
82.9
75.4
56.5
89.1
88.5
81.6
79.8
77.9
58.5
TransCoder-ST-it (Roziere et al., 2022)
88.7
90.1
75.4
74.3
62.5
89.3
76.7
77.6
59.1
CMTrans
91.6
90.5
71.4
83.7
77.2
64.2
89.7
88.4
84.4
82.0
79.3
61.2
Table 3: Java-C++ and Python-C++ translation results on TransCoder-test. The CA@ I results TransCoder and TransCoder-ST are copied from the TransCoder-ST paper. We evaluate BLEU and CodeBLEU using their released checkpoints. We finetune and evaluate CodeT5 and TransCoder-ST on our own.
dataset of generated Comp-Corp (denoted as Gen Comp), we generate from functions with docstrings in the CodeScarchNet (Husain et al., 2020) dataset. We train CMTrans first on the best combination comparable corpora, which is natural and gener- ated Comp-Corp, and then finetuned on AVATAR- para. For language pairs other than Java + Python, we use XCodeEval as naturally existing Comp-Corp and generate Comp-Corp from CodeNet (Puri et al., 2021). More details can be found in Appendix A. Evaluation metrics. Our primary metric is the Computational Accuracy (CA (k),,) which evalu- ates whether at least out of k translation candi- dates generates the same outputs the reference for all the given test inputs. Following previous work (Ahmad et al., 2021b), we report CA @ re- sults and also report BLEU, which computes the overlap between candidate and reference transla- tions (Papineni et al., 2002), and CodeBLEU: the weighted average of token level match, syntax level match, and Dataflow match (Ren et al., 2020). Appendix A. contains more details about the implementation and baselines.
CMTrans substantially outperforms CodeT5, which is initialized from the same pretrained check- point and finetuned with the original parallel data, by an average improvement of 7.5% CA @ 1. CM- Trans also significantly outperforms the state-of- the-art methods, TransCoder-ST-ft on 5 out of 6 language pairs, and reaches parity on Java-to-C++ translation. Note that TransCoder-STT generates test cases for 103,45 (08,488 functions and executes these test cases over 4 iterations of training. In compari- son, our method only generates test cases for 3,391 functions and executes the test cases once, resulting in better efficiency.
Our results show that the advantage of CMTrans is larger on translations between Java and Python. The reason may be that the parallel data gener- ate for translations involving C++ are only verified by automatically generated test cases, which might not cover all the boundary cases and could intro- duce noise to the finetuning set.
Following previous work (Roziere et al., 2022; Ahmad et al., 2021b), report the results of one checkpoint. We also conduct the t-test in Ap- pendix A.2, which indicates that CMTrans signif- icantly outperforms the best baseline over CA @ with p-value < 0.01 for 4 out language pairs and with p-value < 0.05 for one language pair.
4.2 Main results
Table 2 and Table 3 show the code translation re- sults on trastt.

---

Model
J-to-P CA@I
P-to-J CA@1
(Comp Corp Only
Java-to-Python
Python
CodeT5
61.0
52.0
-205%
Random Comp-Corp
64.7
54.6
+714%
145.6%
+ KNN Comp-Corp
63.8
55.4
188.5%
+ Generated Comp-Corp
64.0
63.1
Natural Comp-Corp
68.8
62.9
20
+ All Comp-Corp
62.5
55.4
+ Natural & Generated
70.0
64.1
100 300 500 1000 all(4770)
100 500 500
1000 all(4770)
Paraliel Pairs Finetuning
Panallel Pairs in Finetuning
Multi-Rel
68.1
61.6
CMTrrans
73.5
66.0
Figure 4: Translation results with different amount of parallel data. We mark the relative gain of CMTrams over CodeT5.
Table 4: Ablation studies CMTrans. We report CA @1 I results. J-to-P and P-to-J stand for Java-to-Python and Python-to-Jaye results. We put the experiments on each type comparable corpora an individual block.
1.35
+1.660
130
1.296
> 1:30
1.25
g 1.25
Only)
CMirans
(Comp-Corp Only)
5 1.20
4.3 Performance Analysis
1115
115
1.10 1.072
1.051
1.060
110
1,075
1
1.61
Ablation studies. Table 4 presents the ablations of our approach. We first study the effectiveness of each type of comparable corpora and some combi- nations of them. All the "+ Comp-Corp" ablations denote training CodeT5 on comparable corpora and then finetuneing on AVATAR-parr We also ablate "+ Multi "eet which is finetuned on multiple refer- ences directly after pretraining. The results show that both Multi-Ret and the best Comp-Cur Corp pro- vide large performance gain and stacking them (the full CMTrans) has the best performance.
1.05
1.05
500
1000
1300
2000
2500
3000
3500
500
1000
1500
2000
2500
Train Step
Train Step
(a) Java-to-Python
(b) Python-to-Java
Figure 5: Perplexity of validation set during finetuning.
ral & Generated" ablation and use CMTrans (Multi- Ref Only) to denote "CodeT5 + Multi-Ref" As shown in Figure 4, the relative gains of CM- Trans as well as our ablations are more pronounced when the parallel data is more limited. For example, when there only 100 parallel examples for Java- to to-Python translation, CodeT5 obtains 6.5 CA @ while CMTrans, (Comp-Corp Only), and (Multi- Ref Only) obtain 43.1, 39.7, and 26.9 CA@ 1, re- spectively. This demonstrates the effectiveness of our two data augmentation techniques to tackle data scarcity.
As for the effectiveness of different Comp-Corp, we can see that code translation can benefit from all these types of Comp- Corp. The combination of Natural and generated Comp-Corp has the best performance. The reason might be both Comp- Corp have relatively high data quality and further adding the other two Comp-Cory in training intro- duces noise to the model and hinders the learning of functional equivalence.
4.4 Influence of Comparable Corpora
Notice that with the same finetuning set, Comp- Corp Only still outperforms TransCoder-ST which indicates that compared to self-training, training on comparable corpora has not only better efficiency but better effectiveness. We hypothesize that the reason may be that the comparable corpora contain more diverse structures and contents. This provides more diverse training signals the model and hence improves the generalization.
To answer RQ2, we hypothesize that training on comparable corpora allows the model to produce fluent code in the target language while using simi- lar constructs as the input.
Fluency of outputs. To validate our hypothesis, we compare the perplexity of the reference trans- lations (in the target language) during finetuning, which reflects the fluency of outputs. As shown in Figure 5, after training on comparable corpora, the perplexity is substantially reduced before fine- tuning. It also converges to lower value after finetuning parallel data.
Translation with Limited Parallel Data. To ama- lyze how well method tackles the data scarcity challenge, we compare the performance of CodeT5 and CMTrans when finetuned on parallel data with different sizes. For simplicity, we use CMTrans (Comp-Corp Only) to denote the "CodeT5 + Natu-
Similarly, Table 6 shows that in most scenarios, training on comparable corpora leads to fewer syn-

---

Juva-to-Python
Python-to-Java
Model
LOOP
IF
ELSE IF
LOOP
ELSE IF
CodeTS (No finetune)
100.0
28.5
44.4
99.1
35.4
522
0.0
0.0
0.0
100.0
14.6
25.5
100.0
36.6
53.6
0.0
0.0
0.0
KNN Comp-Coop
89.4
94.6
91.9
82.1
64.3
72.1
41.6
57.6
48.3
87.9
98.9
93.1
90.2
50.9
65.1
30.5
16.4
21.3
Generated Comp- Corp
93.2
98.3
Natural Comp.
95.8
85.4
90.3
91.2
73.9
81.6
50.0
49.6
49.8
87.4
98.9
92.8
88.2
80.4
84.1
57.8
33.6
42.5
Table 5: The overlap between the types of constructs in the translation outputs and the ground truth translations.
Model
J-to-P SA
P-to-J SA
CMTrans (Mutt) Rer
CMiiress (Nuui-Re Only
200
200
No finetuning
150
150
CodeTS (Wang al., 2021)
1.1
1.7
1.0
1.00
+ Random Comp-Corp
20.0
41.3
+ KNN Comp-Corp
22.0
56.4
+ Generated Comp-Corp
41.4
43.2
1-5
Conrect Translations Candidetes
6-15
16-20
3:5
Correct Translations Candicater
6-15
16-20
+ Natural Comp-Corp
34.1
54.6
(a) Java-to- lava-to-Python
(b) Python-to-
With finctuning
CodcT5 (Wang al., 2021)
95.3
69.7
Figure 6: Number of unique correct translations in 20 candidates for each test example. We use beam search for each method, the generated candidates are guar- anteed to be distinct.
+ Random Comp-Corp
96.3
69.7
KNN Comp-Corp
96.3
71.0
+ Generated Comp-Corp
97.6
71.2
Natural Comp-Corp
97.4
76.4
Table Syntax Accuracy (SA) on TransCoder-test be- fore finetuning, which evaluates whether the program can be compiled without syntax errors.
the generation model, so the generated program follows the same algorithm as the input program, which results in larger percent of comparable examples that share the same types of constructs.
tax errors both before and after finetuning, which means the programs our method generates not only have high token-level overlap with the reference translations but also syntactically correct. The reason might be comparable corpora (including the random Comp-Corp) train the model to maximize the probability complete program in the target language, improving its language modeling ability, Generation of matching constructs. We observe that programs in a comparable pair typically con- tain similar constructs (e.g., in AVATAR-come for 83.41% % of Java programs with if statements, the corresponding Python programs also contain if statements). To assess whether our model also learns generate the correct types of constructs, for each type of construct, we consider whether the reference translation has this type of construct as the ground truth and whether the translation output has it as the predictions. Then we compute the accuracy, recall, and Fl scores.
4.5 Influence of Multiple References
As for RQ3, we hypothesize that training on addi- tional references can reduce overfitting to single translation. To validate this hypothesis, we show in Figure 6 that when trained with multiple references, model generate more unique correct transla- tions using beam search within the same number of candidate outputs. For instance, in lava-to-Python translation, there are 214 test examples where our model generates 2 16 unique translations, while there are only 145 and 146 examples for Code and TransCoder-STT: respectively. Furthermore, the scores under the "0" group indicate that after training with multiple references, the model gen- erates at least one correct translation for more test examples. In other words, in addition to CA @ 1, training CodeT5 on multiple references also im- proves on CA
4.6 Hyper-parameter analysis
As shown in Table 5, after training KNN, generated, and natural Comp-Corp. the Fl score is highly improved. The generated Comp-Corp has the highest Fl scores. The reason might be we input both the documentation and the program to
To answer RQ4, we analyze two copee-parraetent Size comparable corpora. We sample the same percentage of comparable examples from AVATAR- comp and Gen-comp and combine the sampled

---

Returne (elink Contic) object from, the supplied 1ist with the supplied nane, it exists. 24 it exists, the supplied list of (Boods configs) will be rodified euch that it no langer contains the returned value. <input NL Documentations
public static Config getAndiannevaConn Lights <Cunfig> configs, String nane] final Config config gettonf twith naze))
getAnddennveeonn for 1 in configs 1 14 1.nane == name confige.remen return
name):
it contig 1- null ) concige, renovee contiy }
configy
<Input return None
<Generated Python
Figure 7: Case study: a comparable example generated by method with NL-to-code generation model. We highlight the lines in the Java program and the generated Python program that can be matched.
Jan Diva-to vaa-to-Python
Python-to-jave
Java -yythn
Python to
CMTTrans
CMitrans (Camp-Corp Only!
CMTraan
CMitrans (Comp-Corp Only)
75
CMiiram
(Multi/fff Oriyy
67
CMiians
(Mirans
Only)
72
TranstaderS11
TannCode
6/69
61
66
63
To
55
Code15
63
Code15
55
CodeT5
20%
50%
100%
0 10%20%%30
50%
100%
2
2
5
10
(a) Effect of the size of comparahle corpora.
(b) Effect of the max number of references for cach example.
Figure 8: The CA@1 score for CMTrans and its ablations with different hyper- paper-parameters..
data. CMTrans is finetuned with at most refer- ences in all trials. As shown in Figure 8a, as we increase the size of comparable corpora, CMTrans (Comp-Corp Only) always has better performance. While the trends are not monotonic for CMTrans, it still obtains the best performance with full com- parable corpora.
to a large software project, which has a different nature from coding problems (e.g., the example in Figure 2). As a result, combining collected and generated comparable examples provides diverse training signals to our model.
Conclusion and Future Work
Number of references. We also examine the ef- fect of the maximum number of references for each parallel example in finetuming. We observe that the performance of CMTrans does not always increase when finetuned with more references. The reason might be our model may generate more unique cor- rect translations for some training examples than others. As a result, the training signals the model received from different examples could be unbal- anced, especially when the maximum number of references for each example is large.
We present two data augmentation techniques for code translation: comparable corpora and multiple references. We study multiple ways of building comparable corpora, such as feeding natural lan- guage code documentation to code generation mod- els. Additionally, we generate multiple reference translations using our model and filter references using automatically generated unit tests. Experi- ments on translation between 6 language pairs show that our two techniques improve CodeT5 substan- tially, by average of 7.5% CA@ 1. Our analyses further show that after training comparable cor- pora, the model learns generate more fluent code with the same types of constructs. With multiple references, the model learns to generate a larger number of unique correct translations.
4.7
Case Studies
We show a constructed comparable example Fig- ure 7. More case studies for comparable corpora and generated references can be found in Appen- dices A.3 and A.4. As shown in Figure 7, both the Java function extracted from GitHub and the gen- crated Python function remove config by name and return this config. The only difference is the Java function obtains the name of each config by helper function, while the generated Python func- tion directly accesses the "name" attribute. We also observe that this function is likely to belong
Following our analyses of what the model learns from comparable corpora, future work may con- duct more comprehensive study what the model learns in code translation. One may also ex- plore combining our data augmentation techniques with data distillation from large language models (LLMs), where LLM may generate data with higher quality, but our techniques are less expensive.

---

Limitations
References
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. 2021a. Unified pre- training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2668, Online. Association for Computational Linguistics.
Despite the empirical advantages of using compara- ble corpora (Comp-Corp) shown in our work, there are some inherent biases and limitations in how we collect and construct them. The collected Comp- Corp are from competitive programming websites, leading to biased data distribution. The Comp- Corp constructed using code generation models are also biased by the training data seen by these models and can potentially contain errors. Further- more, we only evaluate methods on the TransCoder dataset, which is currently the largest code transla- tion dataset with test cases. The TransCoder dataset only contains standalone functions that don't con- tain any imports outside the standard libraries for each language. Translation of longer code with arbitrary external modules is an extension we plan to explore in future work. Another possible risk in our system is that the data may also contain infor- mation that uniquely identifies users or offensive. For example, we generate comparable examples based on users' comments, which could contain inappropriate content.
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Chang. 2022. Summarize and gener- ate to back-translate: Unsupervised translation of programming languages. In Conference of the Furo- pean Chapter of the Association for Computational Linguistics.
Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 202 Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108. 11590.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Eilln Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models.
Federico Cassano, John Gouwar, Daniel Nguyen, Syd- ney Nguyen, Luna Phipps-Costin, Donald Pinckney. Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha. Michael Greenberg, and Abhinav Jangda. 2022. Multipl-e: A scalable and extensible approach to henchmarking neural code generation.
Ethics Statement
License. We use public datasets AVATAR (Ah-
Xinyun Chen, Chang Liu, and Dawn Xiaodong Song. 2018. Tree-to-tree neural networks for program trans- lation. In Neural Information Processing Systems.
mad et al., 2021b), xCodeEval (Khan et al., 2023), and TransCoder (Roziere et al., 2020) in our ex- periments. The data of AVATAR, xCo,eEval, and TransCoder are all distributed under 4.0 license.
T. Etchegoyhen and Harritxu Gete. 2020. Handle with care: A case study in comparable corpora exploita- tion for neural machine translation. In International Conference on Language Resources and Evaluation.
Carbon Footprint We avoided the usage of large language models when constructing our models. CMTrans has the same architecture as C5deT5- base, which has 220 million parameters. The two models we use to construct comparable corpora have the same architecture as CodeTS-base and CodeTS- which have 220 million and 770 mil- lion parameters, respectively. We train CMTrans first on the comparable corpora and then the paral- lel data. Training on comparable corpora took 4-5 hours on average and training on parallel data took less than one hour on one RTX A6000 GPU. There- fore, training CMTrans results in approximately 0.78kg of carbon emission into the environment."
Zhangyin Feng. Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng. Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- BERT: pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536-1544 Online. Association for Computational Linguistics.
Harritxu Gete and Thierry Etchegoyhen. 2022. Making the most of comparable corpora in neural machine translation: case study. Language Resources and Evaluation, 56(3):943-97)
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020. Code- searchnet challenge: Evaluating the state of semantic code search.
"Estimations were conducted using the MachineL_cariig Impact calculator presented in Lacoste et al. (2019). We use Amazon Web Services as the provider.
Ann Irvine. 2014. Using comparable corpora to aug- ment statistical machine translation models in low resource settings.

---

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguistics.
Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical ma- chine translation (t). 2015 30th IEEE/ACM Interna- tional Conference on Automated Software Engineer- ing (ASE).
Svetoslav Karaivanov, Veselin Raychev, and Martin T. Vechev. 2014. Phrase-based statistical translation of programming languages. Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Soft- ware.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02. Association for Compu- tational Linguistics.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang. Md Rizwan Parvez, and Shafiq Joty. 2023, xcodeeval: A large scale multi- lingual multitask benchmark for code understanding, generation, translation and retrieval.
Monica Lestari Paramita, David Guthrie. E. Kanoulas, Robert J. Gaizauskas, Paul D. Clough, and Mark Sanderson, 2013. Methods for collection and eval- uation of comparable documents. In Building and Using Comparable Corpora.
Huda Khayrallah, Brian Thompson, Matt Post, and Philipp Koehn. 2020. Simulated multiple reference training improves low-resource machine translation. In Conference on Empirical Methods in Natural Lan- guage Processing.
Hung Dang Phan, Anh Tuan Nguyen, Trong Duc Nguyen, and Tien Nhut Nguyen. 2017. Statistical migration of api usages. 2017 IEEE/ACM 39th Inter- national Conference on Software Engineering Com- panion (ICSE-C), pages 47-50.
Marieanne Lachaux, Baptiste Roziere, Mare Szafraniec, and Guillaume Lample. 2021. DOBF: A deobfus- cation pre- pre-training objective for programming lan- guages. In Advances in Neural Information Process- ing Systems.
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi. Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Freder- ick Reiss. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning.
Ying Qin and Lucia Specia. 2015, Truly exploring mul- tiple references for machine translation evaluation. In Proceedings of the 18th Annual Conference of the European Association for Machine Translation.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Hoi. 2022. CodeRL: Master- ing code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Li- dong Zhou, Linjun Shou, Long Zhou, Michele Tu- fano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shu- jie LIU. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and gener- ation. In Thirty-fffth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 1).
Baptiste Roziere, Marie-Anne Lachaux. Lowik Chanus- sot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33.
Baptiste Roziere, Jie M. Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2022. Leveraging automated unit tests for unsuper- vised code translation.
Dragos Stefan Munteanu and Daniel Marcu. 2005. Im- proving machine translation performance by exploit- ing non- non-parallel corpora. Computational Linguistics.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien Nhut Nguyen. 2013. Lexical statistical machine translation for language migration. In ESEC/FSE 2013.
Krzysztof Wolk, Emilia Rejmund, and Krzysztof Marasek. 2015. Harvesting comparable corpora and mining them for equivalent bilingual sentences using statistical classification and analogy-hased heuristics. ArXiv, abs/1 51 11.5511...0 1002855
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien Nhut Nguyen. 2015. Divide-and-comquer approach for multi-phase statistical migration for source code (t). 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

---

A Appendix
A.3 Case Studies for Naturally Existing and Generated Comparable Corpora
A.1 Experimental Details
We show case studies of naturally existing and generated comparable examples in Figure 9 and Figure 10. The example in Figure 9 is from the XCodeEval dataset, which contains different users' submissions of the "Food for Animals" problem on Codeforces. In Figure 10, the Java function and its doestring are from the Node dataset. The Python program is generated by our method. Quality of the Generated Comparable Exam- ple. As shown in Figure 10, the Python program generate has similar functionality as the input Java program. Specifically, both programs define a list of variables, compare the sum of term off- set and aligned length with the term length, and return an offset of the term. The major differ- ences are that the Java program calls a global func- tion "getAndAddRawTail" to compute the value of "rawTail", while the generated Python program calls class function "termBuffer "ternBuffer..attive>" Both functions are not defined in the context. Also, the Java program calls another function, "handleEndOfLLogCondd"" tion", without defining it, while our method also generates the content of "handleEndOfl oggondi tion".
Implementation details. To generate the com- parable corpora dataset (denoted as "Gen-comp" Table 1), for Java ++ Python translation, we obtain Java functions with Docstrings from Code- SearchNet (Husain et al., 2020) and use CodeRL (Le et al., 2022) finetuned on MBPP (Austin et al., 2021) to generate Python programs. For Cpp Java translation, obtain Cpp solutions with prob- lem descriptions from CodeNet (Puri et al., 2021) and use CodeT5 (Wang et al., 2021) finetuned on CONCODE (lyer et al., 2018) to generate Java pro- grams. For Cpp ++ Python translation, we again obtain Cpp programs from CodeNet and use the finetuned CodeRL model to generate Python pro- grams.
To train CMTrans, we tune the number of source/target programs per problem in the range of [1, 5] and tune the maximum number of gen- erated references for finetuning in the range of [5, 10]. For the training on both comparable corpora and AVATAR-part we train CMTrans with a learn- ing rate of le-4 and batch size of 32 for at most 20 epochs. As for the baselines, with learning rate of le-4 and batch size of 32 for at most 20 epochs. We finetune TransCoder-ST-it with a lcarning rate of le-4 and batch size 64 for at most 20 epochs.
finetune Code'T5
We notice that large percentage of differences between the generated and input program are due to calling functions without presenting their defi- nitions in the input. Such input programs contain limited information on the purpose of these func- tions. As result, it is challenging for the code generation model to generate code with exactly the same functionality.
Baselines. We compare CMTrans with unsu- pervised and self-supervised models, including TransCoder (Roziere et al., 2020), DOBF (Lachaux et al., 2021), and TransCoder-ST (Roziere et al., 2022). We also compare with supervised models, which are initialized from CodeBERT (Feng et al., 2020), PLBART (Ahmad et al., 202 la), CodeT5 (Wang et al., 2021), or TransCoder-ST (Roziere al., 2022) (denoted as TransCoder-ST-fi) and finetuned on AVATAR-para. We only compare with DOBF, CodeBERT, and PLBART on Java +> Python translation because these models are not pretrained on Cpp.
Comparison Between Naturally Existing and Generated Comparable Examples. We can ob- serve that the naturally existing and generated com- parable examples are different in several ways. For instance, the naturally existing comparable exam- ples are mainly collected from coding problems, while the generated examples can belong to other sources, such as a large software project. In addi- tion, the programs in the naturally existing example are self-contained, while both the Java and Python programs in the generated example contain user- defined classes and external functions that are de- fined elsewhere, including "HeaderWriter"" "Head- erWriter. wr and "align()". Besides, programs in one naturally existing comparable example typi- cally have the same output format (c.g.. "YES" or "NO" in this case), while programs we generate
A.2 Statistical Significance Test
We present the t-test results in Table 7, where we run each experiment with 3 random seeds when finetuning on AVATAR-para.

---

Java-to-Python
Python-to-Jan
BLEU
CB
CA@1
BLEU
CB
CA
Best baseline
79.9
73.2
69.4
81.4
78.4
62.0
CMTrans
82.1 + 0.4**
76.1 + 1,48*
72.1 + 1.6**
84.3 + 0.4*s
82.0 + 0.5
64.6 + 1.0*s
C++-to-Python
Python-to-C++
BLEU
CB
CA@1
BLEU
CB
CA
Best baseline
82.9
75.4
62.5
79.8
77.9
59.1
CMTrans
83.8 + 0.6**
76.8 = 0.3**
64.6 + 0.8**
82.1 = 0.9
79.8 + 0,5**
63.2 + 1.4
Java-to-C++
C++-to-Java
BLEU
CB
CA@1
BLEU
CB
CA
Best baseline
89.3
88.5
84.6
90.9
90.1
68.3
CMTrans
89.9 - 0.4*
88.1 + 0.2
83.7 + 1.1
91.1 0.4
90.3 + 0.1**
69.9 1.1*
Table 7: T-test results. We copy the results from the papers of the best baselines and run CMTrans for 3 different random seeds. **means significant results with p-value < 0.01. *means significant results with p-value < 0.05.
from the documentation may have different output formats, especially when the doestring unclear about the return value. With all the differences, the combination of naturally existing and gener- ated comparable corpora cover programs with a large variety of styles and domains, which provides diverse training signals to our model.
including the ground truth translation.
We can observe that the differences between candidates can be as small adding deleting parentheses and semicolons. For instance, the only difference between Candidate #3 and Candidate #9 is the parentheses around "i%10 == k". In this example, by computing the string edit distance, we avoid having both Candidate #3 and #9 in the reference set.
A.4 Case Studies for Model-generated References
Figure 11 shows an example the multiple refer- ences generated by CMTrans. The example is from the AVATAR- para dataset. We use our model to generate 50 candidate translations, 23 of which are correct (i.e., have exactly the same output the source program on all the automatically generated test cases). We show the correct translations with the highest probabilitics, of which are selected to the reference set.
Quality of the Generated References. In this example, the source program contains a for loop and an if statement. Note that the for loop has "< high" as the end condition. When we translate it to the "for $ITER_VAR in range()" grammar in Python, we have to instead use "high + 1" as the end value. This correctly handled in several different candidate translations.
Diversity of the Generated References. There are two main options for translating loop: using either for loop or while loop. the example in Figure 1 I, the selected reference set contains translations with a while loop and with a for loop,

---

obviously, the best way to buy food for every pet is buy maximam poseible food for dogs and cate, then nax(O, x-a) dogs and maxt y-b) cata will not get food, for these doge max(o, x-2) (0 and No else.
for in (inge b, X, neptint, (oput().eplie()) a >E and b y print( 'YaS" y
buy
YES, if
Instruction Coding
#include ciostrean> #include<estdion us using nanaspace atd; int [
import Jeva.util."; public class Heint
if b F.c. yi print( 'xO"
publ static main(String() arga)
Scanner se new (y)neer()))tenninn)
print 'YES' if a + XI print( 'NO'
while int cin 55 =i
for (int sc.nextint()1 t--201)
e=sc-nextInt((,
If (
besc.nextInt((), evsc.nextInb(! xsso.nextInt(), ( int Hath.max(O, X-2)i int B Nath.max(0, y-b)i Syston.cot.printlln >- c "YES" "NO")r
print( 'YES'
if
elser
((x-a) (y-b)) if v >- 0
if (b >- cout dc "YsSin",
"NOAn";
print( 'YES'
elee
else cout << "HO'n";
print( 'XO'
<C+ Solution (User A)>
<Java Solution (User B)>
<Python Solution (User C)>
Figure 9: A comparable example collected from coding problem. We highlight the lines in the user-written C++, Java, and Python programs that can be matched.
length
vriting uessage with copy semantics.
Sparanetass:
header for vriting the defeult header.
length of the ressage to be writren.
butfarcisin to be ugdated with the claixed region. activenerntd used for flow costrol.
ERetarns: the resulting offeet the atter
othervise PATEED.
clnput NL Documentations
public int clain[final mnddrdiittar header, final ine langth, final Bufferclair baffertlain, final int s
def clain(bander, length, bufferclain, activeterstd):
Iranetangth lenqth SEADER, DENGZH
final int frametangth alignediength align(frameLengs)
alignedtength align[frunetanghh FRANE ALTGWENT
FRME ALIOMEND))
terrBuffer bufferClain tersBaffer
sersiength cerpputter..
temmeenth ternButeer (3) long raviail (ateddddiootuilllllleeeeeeo
ternid att.urreeeaetiveeeeteeeeeee
serxDfisut LermBuffez activej activ//endd seteeeeeeeeeeeeeeeeeeeeeeeee
OxFEzy FEEEL
final int temidd tormid]zwwtail);
terxofiset aaddeeaadoriopgdddd...oo tertuuufo. termöffset,
final long terabffset ravtail 6xt57r_57753 checkTern activelemtt, serntd }: long wwultingafffees tersoffaet: alignadteogth;
Leader, temntangth, tersid);
return tersDtfaet def aundla@ndDiiagiiniiiiio ternraatfe tamoffset,
it tingötfett ternlength rescitingfffset SandieEsdbtoogoondtuee
beader, ternlangth, ternid):
termöffset termlenghh
torsttfeet, header, tembeegth, temid)
else
temboffar..nd centuffar..cttver activwtemte
final int frameOCisot (eat)terttftee;:
Lersbuffer.acte activelerid t- torniength
weteerrrritttttttene fraraoffset, Frametength, tacnid)? bafferclain, wrapttersButter, fransoIiset, Eranetengsh);;
framntangth tamteeggh FRANS, ALTONEET
header www.eetteetteteeee....t....ttttt termoffact, franstength, temile
bufferclainn tersfloffer, farsoifant, Franetength) termatter. activel activeferid
retarn teslttteettt
<input
return
Python>
Figure 10: A comparable example generated by our method. The natural language documentation is the docstring of the Java program. We highlight the lines in the Java program and the generated Python program that can matched.

---

Input Source program (Java)
Ground Truth Reference Translation (Python)
public countastDigitx(int int high, int count 0;
def countaetDigitx(1ow, high, count
int count ++1
high;
range( 1ov,
( i810 k
if { is10
return count
count
retuen count
Correct Candidate
Correct Candidate
Correct Candidate
def countiassUUgiitk high, ;)
def comntsstDigits(1ov, high, k)
def couniastDigitk (low, high, k)
for in range(low, hight] )
range(low, hight]):
range( (Iow, hight1, 1)=
it is10 == k
it is10 == kt
it i010 = kt
coont retura count
count
return count
retura count
Correct Candidate 4
Correct Candidate
Correct Candidate
def counLastDigi=:(low high, ;); count. = lov while i <= high
def counLastDigitx(1ow high, k1 count.
def couniastDigitki (loetiiiiiiK(,, high, x)
count 0; low; while i <m hight
while i <= high ): count =-
if 1810
count --
count +- 1; 1;
retura count
return count
return count:
Correct Candidate 7
Correct Candidate # 8
Correct Candidate
deř counLastDigi:E(Lov high, *) i lov while high 1r 1910 -- k
def counLastDigitx(Lov high, k)1
det counLastDigitk (Low, high, k)
i low; while high ):
for in range(low, high+l, 1); ir 1610
12 { 1410 -- k )=
count +-
retura
1;
Figure II: An example of the multiple references we generate. We show the top correct translations generated by our model with the highest probabilities. Candidates with checkmark are those we select to add the reference set. We add these translations one by one based on their string edit distance with all the existing translations.


Reviews:
[{"review_id": "RFOhlqZ0Co", "reviewer": "Reviewer_9cbG", "summary": "", "questions": "Have you experimented with current large language models like ChatGPT and GPT-4 on this dataset? I'm intrigued to know how your work fares in comparison to these models.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your thoughtful comments and valuable suggestions! \n\n* Here are the key points of our response:\n    * [Clarification] We emphasize the difference between our work and Transcoder-ST\n    * [Response to “Limited novelty”] We highlight the novelty of our first contribution compared to the “Summarize and Generate” EACL’23 paper\n    * [Response to “Limited scalability”] We clarify the details of our method\n    * [Response to “GPT potential”] We discuss why we do not use GPT models in this work\n    * [Response to “Experimental design”] We discuss the selection of base model and dataset\n\n&nbsp;\n### Clarification: Difference to TransCoder-ST\n* We’d like to further highlight the difference between our work and Transcoder-ST. The main contributions of our work are two-fold:\n    * (1) Training on comparable corpora before finetuning (there are multiple means of creating comparable corpora, where leveraging docstrings is only one of them);\n    * (2) Augmenting the finetuning set with multiple references.\n\nTransCoder-ST uses neither the comparable corpora nor multiple references.\n\n&nbsp;\n### Response to “Limited novelty”\nThanks for pointing out the connection between our first contribution and the “Summarize and Generate” (S&G) EACL’23 paper. We agree that S&G can also be viewed as one way to create comparable corpora. Nevertheless, **the contribution of our work lies in conducting a more in-depth study of when and how to use comparable corpora for code translation**.\n\n“When and how to use comparable corpora” is a broad topic in natural language translation. After [1] introduced the definition and potential utility of comparable corpora, there have been research works that identify scenarios where comparable corpora do not improve the performance and study the conditions when they are beneficial [2][3]. There are also a series of works that present different ways to collect comparable corpora [4][5].\n\n\nFor the study of **when** to use comparable corpora, **we present a surprising result that training on comparable corpora is beneficial even if there is already high-quality parallel data** (i.e., the finetuning set). This goes beyond the conclusion of the S&G paper that comparable corpora obtained by summarization and generation are beneficial when no other parallel data is available.\n\n**We also provide an analysis on why comparable corpora are beneficial**. Specifically, Sec 4.4 shows that by training on comparable corpora, the pretrained model learns to (1) produce fluent code in the target language, and (2) produce code with matching constructs as the source input (e.g., if statements, for loop, etc). The S&G paper does not provide analysis on what can be learned from the comparable corpora.\n\n\nFor the study of **how** to use comparable corpora, we show that **code translation can benefit from multiple types of comparable corpora, and construction from docstring is only one of them**. We additionally study a type of naturally existing comparable corpora: different submissions to the same online coding problem. Experimental results show that both types of comparable corpora can improve performance, and combining them gives the best results.\n\nDuring the response period, we further explore two other types of comparable corpora: (1) KNN: retrieve k code pieces in the target language that have the most similar sequence embeddings to the source code, and (2) Random: a random piece of code in the target language. Experiments below show that both of them can improve the performance of Java-Python translation:\n\n**Java-to-Python**:\n\n| CC      | Implementation Details                                                            | Finetuning data | CA (Pass@1, k=10) |\n|---------|-----------------------------------------------------------------------------------|-----------------|-------------------|\n| —-      |                                                                                   | AVATAR-para     | 0.6099            |\n| Random  | Each Java function in AVATAR is paired up with a random Python function in AVATAR | AVATAR-para     | 0.6466            |\n| KNN     | Each Java function in AVATAR is paired up with K nearest python functions         | AVATAR-para     | 0.6530            |\n\n\n**Python-to-Java**:\n\n| CC      | Implementation Details                                                            | Finetuning data | CA (Pass@1, k=10) |\n|---------|-----------------------------------------------------------------------------------|-----------------|-------------------|\n| —-      |                                                                                   | AVATAR-para     | 0.5203            |\n| Random  | Each Python function in AVATAR is paired up with a random Java function in AVATAR | AVATAR-para     | 0.5456            |\n| KNN     | Each Python function in AVATAR is paired up with K nearest Java functions         | AVATAR-para     | 0.5809            |\n\n\nOur analysis in Sec. 4.4 helps explain why “Random comparable corpora” is also beneficial: the model learns to generate fluent code in the target language when the input is in the source language. In other words, the model learns to capture the statistical patterns and relationships between tokens in the target language.\n\nWe will include this analysis in the next version of the paper.\n\n&nbsp;\n### Response to “Limited scalability”\nRegarding the concern about the scalability of using docstrings to construct comparable corpora, we want to note that one can use any programs publicly available from GitHub that have docstrings for this purpose, which is already of a large scale. In our experiments, we only used 22K functions from the CodeSearchNet dataset. Large-scale GitHub datasets are publicly available, e.g. The Stack [6]\n\nRegarding multi-reference code generation, the test cases are **automatically** generated using fuzzing, so our method can be applied to any dataset of functions to obtain multiple references. The process of automatically generating test cases is described in Line 284-295. We also provide an illustration in “step 2: test cases generation” in the right half of Figure 3.\n\n&nbsp;\n### Response to “GPT potential”\nThere are two reasons why we do not use GPT models in our training. First, it’s expensive to apply/query GPT models at our scale. More importantly, **we don’t know whether GPT has already been trained on the test data we use**. Our test set, the TransCoder dataset, is collected from the GeeksforGeeks website. A large number of GeeksforGeeks code pieces have been crawled and uploaded to GitHub, which GPT-3.5 (and presumably GPT-4) are trained on. Our work uses only fully open-access models with known training data.\n\n&nbsp;\n### Response to “Experimental design”\nFor the choice of base model, we have tried TransCoder in early experiments and found that the performance of fine-tuned CodeT5 is slightly better than fine-tuned TransCoder. \n\nFor the evaluation dataset, we do not use CodeXGLUE because it does not provide test cases and only supports CodeBLEU/BLEU as evaluation metrics. These metrics are less accurate than execution-based metrics such as Computational Accuracy. For example, semantically equivalent programs with different implementations will have low CodeBLEU and BLEU scores. In our experiments (Table 2), CodeT5 has higher BLEU and CodeBLEU scores than TransCoder-ST in Java-to-Python translation (79.4/72.5 vs. 73.1/68.7 BLEU/CodeBLEU), but much lower Computational Accuracy (61.0 vs. 68.5 CA@1).\n\nFor the above reason, most code translation papers (TransCoder, DOBF, TransCoder-ST, S&G, AVATAR) use only TransCoder as the test dataset and we are following their setting.\n\n&nbsp;\n### References\n[1] Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics.\n\n[2] T. Etchegoyhen and Harritxu Gete. 2020. Handle with care: A case study in comparable corpora exploitation for neural machine translation. In International Conference on Language Resources and Evaluation.\n\n[3] Harritxu Gete and Thierry Etchegoyhen. 2022. Making the most of comparable corpora in neural machine translation: a case study. Language Resources and Evaluation, 56(3):943–971.\n\n[4] Monica Lestari Paramita, David Guthrie, E. Kanoulas, Robert J. Gaizauskas, Paul D. Clough, and Mark Sanderson. 2013. Methods for collection and evaluation of comparable documents. In Building and Using Comparable Corpora\n\n[5] Krzysztof Wołk, Emilia Rejmund, and Krzysztof Marasek. 2015. Harvesting comparable corpora and\nmining them for equivalent bilingual sentences using statistical classification and analogy-based heuristics. ArXiv, abs/1511.06285.\n\n[6] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, Harm de Vries, The Stack: 3 TB of permissively licensed source code. ArXiv, abs/2211.15533.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Code translation tasks pose greater challenges compared to the more mature field of neural machine translation. As code-assisted software like Copilot gains prominence, the potential for future research in this area is immense.\n2. The earlier work, Transcoder-ST, introduced an effective approach for code translation tasks by automatically generating code pairs, such as back translation, to create an extensive range of potential data pairs for the task. Subsequently, filtering was performed using compilation tools like unit tests to overcome the issue of scarce labeled data. The author further enhances this method by incorporating docstrings as a supervisory signal, resulting in improved performance.", "reasons_to_reject": "1. Limited novelty: One of the challenges for code translation is the scarce parallel training data. This paper addresses the issue by using models to generate parallel code. Specifically, it employs docstrings (summarized text) as a bridge for source-target code alignment to generate related parallel code for model training. However, this idea has already been explored in the work \"Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languages.\"\n2. Limited scalability: Utilizing docstrings for code generation restricts scalability, as it requires that source code functions contain docstrings. Furthermore, the multi-reference code generation relies on test cases to select the correct candidates, making it difficult to apply this method to labeled datasets without test cases.\n3. GPT potential: Considering the effectiveness of GPT3.5/GPT4 in code generation, Why not use GPT as parallel code generator to create parallel pairs?  \n4. Experimental design: This work conducts all experiments based on the CodeT5 model, using only a small evaluation dataset (TransCoder dataset). In the original CodeT5 paper, the model is evaluated on the CodeXGLUE benchmark for C#-Java translation without testing on the TransCoder dataset. Conversely, the TransCoder model is pretrained on C++/Java/Python and evaluated on their dataset. It’s strange to select CodeT5 as base model to validate the proposed methods on TransCoder dataset.", "questions_to_authors": "Have you experimented with current large language models like ChatGPT and GPT-4 on this dataset? I'm intrigued to know how your work fares in comparison to these models.", "paper_topic_and_main_contributions": "This paper focuses on program translation tasks and tries to generate more parallel training data automatically for supervised translation training.  The authors use two methods to generate code pairs: 1) use docstring of one of program code method (program A) to generate another program (program B) to get A-B as supervised data.  2) based on existing parallel code, they use a model to convert source code to multiple candidates for a target program language, then select some of these candidates as multi-references codes.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y5rhpIfc2x", "reviewer": "Reviewer_jFgc", "summary": "", "questions": "jFgc_A. In fact, unlike natural language, formal grammars for programming languages are fully defined. So why not program a \"compiler\" from language A to language B, instead of using deep learning or machine learning?\n\njFgc_B. How could your proposition and previous state-of-the-art propositions work for very different language (for example, Python versus Prolog)?\n\njFgc_C. The parallel data is aligned line-by-line. But several lines are not aligned. Are this lines cut off before training?\n\njFgc_D. Could you discuss the quality of docstrings? How did you train the generative model (lines 228-231)? What is its performance?\n\njFgc_E.Table 2: give precision about significance of results\n\njFgc_F. Table 2: you copy/paste previous results. Are you sure that the condition tests are strictly the same?\n\njFgc_G. lines 389-392 : Comparable corpora allow to build fluent translations, not precise translations according to the input. Therefore, the effectiveness of models trained on comparable corpora should be lower. Contrary to intuition, the results are better than baseline models. The proposed explanation is unconvincing.\n\njFgc_H. Figure 6 : loop, if, else-if are present in the vast majority of program, I guess. Even by chance, a model should obtain more than 21%, shouldn't it? Please, could you comment this?\n\njFgc_I. lines 485-488: the explanation is not clear. Why could the training be unbalanced?\n\njFgc_J. Limitations: what is the bias given by competitive programming websites.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and insightful feedback on our paper. We address your concerns and answer your questions as follows:\n\n&nbsp;\n### Using compilers for code translation (Reason to reject, point 1, and also jFgc_A)\nIt is reasonable to build a compiler for code translation. However, such transcompilers are time-consuming to develop, requiring expertise in both source and target languages. Transcompilers rely on manual rewrite rules applied to the abstract syntax tree (AST) of the source code. Furthermore, building transcompilers is especially hard for language pairs with different amounts of information or levels of abstraction. An example is Python-to-Java translation, where variable type information is required for Java when declaring variables, but is not required and usually omitted in Python.\n\n&nbsp;\n### Explanation of Results (Reason to reject, point 2, and also jFgc_E, jFgc_G, and jFgc_I)\n*(jFgc_G)* For **lines 389-392**, just a reminder, our final model is trained first on comparable corpora and then on parallel data. While we agree that training on just the comparable corpora would lead to unsatisfactory performance, we show that models trained on comparable corpora followed by parallel data **perform better and converge faster** than models trained on just parallel data. Our analysis of the fluency of output (Line 423-435) indicates that the reason may be that training on comparable corpora already increases the fluency of the translation, so the model will not need so much of the limited parallel data to improve both the fluency and the accuracy of the translation.\n\n*(jFgc_I)* For **lines 485-488**, while generating multiple references we initially generate the same number of candidates for each program, but then we **filter out incorrect translations, which vary in number across programs**, leading to an unequal number of references per program in the filtered data.\n\n*(jFgc_E)* We present the **t-test** results as follows, where we run each experiment with 3 random seeds when finetuning on parallel data:\n\n&nbsp;\n### \n**Java-to-Python:**\n|               |     BLEU     |      CB      |     CA@1     |\n|---------------|--------------|--------------|--------------|\n| Best baseline |     79.9     |     73.2     |     69.4     |\n| CMTrans       | 82.1 ± 1.4** | 76.1 ± 1.4** | 72.1 ± 1.6** |\n\n&nbsp;\n### \n**Python-to-Java:**\n|               |     BLEU     |      CB      |     CA@1     |\n|---------------|--------------|--------------|--------------|\n| Best baseline |     81.4     |     78.4     |     62.0     |\n| CMTrans       | 84.3 ± 0.4** | 82.0 ± 0.5** | 64.6 ± 1.0** |\n\n&nbsp;\n### \n**C++-to-Java:**\n|               |    BLEU    |      CB      |     CA@1    |\n|---------------|------------|--------------|-------------|\n| Best baseline |    90.9    |     90.1     |     68.3    |\n| CMTrans       | 91.1 ± 0.4 | 90.3 ± 0.1** | 69.9 ± 1.1* |\n\n&nbsp;\n### \n**C++-to-Python:**\n|               |     BLEU     |      CB      |     CA@1     |\n|---------------|--------------|--------------|--------------|\n| Best baseline |     82.9     |     75.4     |     62.5     |\n| CMTrans       | 83.8 ± 0.6** | 76.8 ± 0.3** | 64.6 ± 0.8** |\n\n&nbsp;\n### \n**Java-to-C++**\n|               |     BLEU    |     CB     |    CA@1    |\n|---------------|-------------|------------|------------|\n| Best baseline |     89.3    |    88.5    |    84.6    |\n| CMTrans       | 89.9 ± 0.4* | 88.1 ± 0.2 | 83.7 ± 1.1 |\n\n&nbsp;\n### \n**Python-to-C++**\n|               |     BLEU     |      CB      |     CA@1     |\n|---------------|--------------|--------------|--------------|\n| Best baseline |     79.8     |     77.9     |     59.1     |\n| CMTrans       | 82.1 ± 0.9** | 79.8 ± 0.5** | 63.2 ± 1.4** |\n\n\n\n**means significant results with p-value < 0.01\n\n*means significant results with p-value < 0.05\n\n&nbsp;\n### \nNote that we use computational accuracy (CA) as the primary evaluation metric because it directly reflects the functional correctness of the program, while BLEU and CodeBLEU (CB) only compare text overlap with references. Our model significantly outperforms the baselines on BLEU, CodeBLEU, and CA@1 for most language pairs, and our improvements are typically large on CA@1.\n\n\n&nbsp;\n### Questions for the authors \n\nTranslations between very different languages (jFgc_B): Our work and the majority of previous code translation works train a Seq2Seq model to translate between two languages. The details of training are provided in Lines 241-254. In principle, we can train a Seq2Seq model to translate other language pairs as well. An example of “very different languages” is the translation between programming languages and natural languages, which are called code summarization and code generation. Previous work [1] shows that Seq2Seq models trained for code summarization and code generation can also have strong results.\n\nUnaligned lines in parallel data (jFgc_C): We do not cut off any lines. The reason is removing such lines leads to influent and “unnatural” code pieces in the truncated data. Training with such code will cause inconsistency with testing, where the input will typically be fluent and complete code.\n\nQuality of docstrings (jFgc_D): We use the CodeSearchNet dataset for the functions with docstrings [2]. They apply preprocessing such as removing short docstrings and truncating long ones to improve the quality of docstrings. \n\nWe do not train the code generation model by ourselves but apply CodeRL finetuned on MBPP (for translations involving Python) and CodeT5 finetuned on CONCODE (for other translations) as the code generation models. More details can be found in the Appendix (Lines 736-749).\n\nTest cases used by baseline models (jFgc_F): Assuming that condition tests refer to the test cases, yes – we use the same test cases as all baseline methods, which are provided by the TransCoder dataset. \n\nComparison to random chance in construct overlap analysis (jFgc_H): We compute the percentage of each type of construct in the test set:\n\n**Java-to-Python:**\n|                                                              |  loop  |   if   | Else if |\n|--------------------------------------------------------------|:------:|:------:|:-------:|\n| Percent of test examples with this construct                 | 76.29% | 68.75% |  7.97%  |\n| Construct overlap rate of CMTrans (CC. Only) (no finetuning) | 84.18% | 74.61% |  21.62% |\n| Construct overlap rate of CodeT5 (no finetuning)             | 28.53% | 35.42% |    0    |\n\n**Python-to-Java:**\n|                                                              |  loop  |   if   | Else if |\n|--------------------------------------------------------------|:------:|:------:|:-------:|\n| Percent of test examples with this construct                 | 75.31% | 69.08% |  7.68%  |\n| Construct overlap rate of CMTrans (CC. Only) (no finetuning) | 99.72% | 78.68% |  10.81% |\n| Construct overlap rate of CodeT5 (no finetuning)             | 14.60% | 36.64% |    0    |\n\n\nAll the numbers are lower than the construct overlap rate of CMTrans (CC. Only) (no finetuning) and higher than that of CodeT5 (no finetuning). \nAs shown in the experiment in Figure 5, when the input and output programming languages are different, CodeT5 (no finetuning) is not able to generate fluent code in the target language. As a result, even if the percentage of loops or if structures are high in the training data, without finetuning, CodeT5 may not learn to generate the code with the same distribution.\n\n\nBias of programming websites (jFgc_J): The code on competitive programming websites mostly uses only standard inbuilt libraries. They are also typically short in length and do not import functions from external files.\nAll the above properties typically do not hold for other code pieces on GitHub. However, as these websites provide functional tests, they have become a standard benchmark for evaluating code translation, where functional tests are crucial to evaluate accuracy of the translated functions.\n\nReproducibility: We will release the code and data upon acceptance.\n\n\n### References\n[1] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. Arxiv. abs/2102.04664.\n\n[2] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020. Codesearchnet challenge: Evaluating the state of semantic code search.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and easy to follow. It is not too technical.\nThe experiments are rich and try to explore several pertinent questions. Past results from 7 systems are given for comparison.\nSeveral propositions are evaluated, and each one is separately evaluated, this allows to have more refined results.", "reasons_to_reject": "The usefulness of deep learning and machine learning is not discussed. In fact, unlike natural language, formal grammars for programming languages are fully defined. So why not program a \"compiler\" from language A to language B?\n\nThe explanations of the results are not convincing:\n1) Lines 389-392 : Comparable corpora allow to build fluent translations, not precise translations according to the input. Therefore, the effectiveness of models trained on comparable corpora should be lower. Contrary to intuition, the results are better than baseline models. The proposed explanation is unconvincing.\n\nEdit: the given explanation is convincing. This should be added to the camera ready paper\n\n2) The explanation of lines 485-488 is not clear to me.\n\nEdit: the given explanation is now clear, and should be added to the camera ready paper\n\nThe results are sometimes very close to the baseline models, and no information is given about the significance.\n\nEdit: the authors gave this information.", "questions_to_authors": "jFgc_A. In fact, unlike natural language, formal grammars for programming languages are fully defined. So why not program a \"compiler\" from language A to language B, instead of using deep learning or machine learning?\n\njFgc_B. How could your proposition and previous state-of-the-art propositions work for very different language (for example, Python versus Prolog)?\n\njFgc_C. The parallel data is aligned line-by-line. But several lines are not aligned. Are this lines cut off before training?\n\njFgc_D. Could you discuss the quality of docstrings? How did you train the generative model (lines 228-231)? What is its performance?\n\njFgc_E.Table 2: give precision about significance of results\n\njFgc_F. Table 2: you copy/paste previous results. Are you sure that the condition tests are strictly the same?\n\njFgc_G. lines 389-392 : Comparable corpora allow to build fluent translations, not precise translations according to the input. Therefore, the effectiveness of models trained on comparable corpora should be lower. Contrary to intuition, the results are better than baseline models. The proposed explanation is unconvincing.\n\njFgc_H. Figure 6 : loop, if, else-if are present in the vast majority of program, I guess. Even by chance, a model should obtain more than 21%, shouldn't it? Please, could you comment this?\n\njFgc_I. lines 485-488: the explanation is not clear. Why could the training be unbalanced?\n\njFgc_J. Limitations: what is the bias given by competitive programming websites.", "paper_topic_and_main_contributions": "The paper focuses on machine translation of code.\nThe goal is to translate (for example) Python code into Java code.\nTo do this, the authors use existing benchmarks. But the parallel data for training is not huge.\nTo solve this problem, they propose two ideas:\n\n1) they generate comparable corpora, train a model on them, and tune it on an existing parallel corpus. For the generation of comparable corpora, they use the human user documentation of the source file, and they automatically generate code in the target language based on this documentation.\n\n2) They enrich the corpus with several target references. To do this, they ask the system to automatically generate several translations. They select the translations that pass the automatically generated unit test. And they select translations in order to promote diversity.\n\nThe authors propose numerous experiments and study the results according to each of their proposals (separately), and they describe the behaviour of their system according to hyperparameters. The system is compared with several state-of-the-art systems.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GTluBt0Mf4", "reviewer": "Reviewer_PYWi", "summary": "", "questions": "Could you provide some statistical significance testing (e.g. bootstrap resampling) for the BLEU scores - at least for the ones that have less than one point difference?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper! We address your concerns and answer your questions as follows:\n\n&nbsp;\n### New pipeline, not new model\nThe contribution of our paper is not just presenting a pipeline that reaches state-of-the-art results, but **the study and analysis of data augmentation methods of code generation**. Our two proposed data augmentation methods — training on comparable corpora and with multiple references — can be applied to a wide range of code generation models. Our analyses including what the model learns from comparable corpora and multiple references also provide insights into what the model learns in code translation training.\n\n&nbsp;\n### Fit for the conference\nWe agree that other papers on code translation that were published in NLP venues, such as [1][2], help demonstrate the fit of our paper. There was also a “NLP for programming” workshop co-located with ACL 2021 (NLP4Prog 2021), which focuses on deep learning based techniques for modeling both natural language and computer programs. We categorize code translation as a subdomain of “NLP for programming”.  Additionally, our paper draws motivation from classic methods in machine translation (the use of comparable corpora).\n\n[1] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2022. Summarize and generate to back-translate: Unsupervised translation of programming languages. In Conference of the European Chapter of the Association for Computational Linguistics.\n\n[2] Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, Kai-Wei Chang. 2022. AVATAR: A Parallel Corpus for Java-Python Program Translation. In Findings of the Association for Computational Linguistics: ACL 2023.\n\n&nbsp;\n### Writing and Reproducibility \nIn the next version of the paper, we will revise the parts that cause confusion for you and other reviewers, such as experimental details and descriptions of our method. \n\nWe will release the code and data upon acceptance.\n\n&nbsp;\n### Clarification of Figure 2 (Questions for the authors, point 1)\nCould you be more specific about what Figure 2 is missing? We highlight the lines with similar functionalities in the same color. Take lines with the green background as an example, the input Python code contains a line: “```if \"BB\" in x or \"WW\" in x:```”. Its parallel example (in Java) has a similar line “```if (x.contains(“BB\") || x.contains(\"WW\")){```”, which has exactly the same logic and algorithm. Similarly, the comparable example (in Java) has a line “```if ( s.charAt(j) == s.charAt(j+1) ){```”, which also checks whether the board has contiguous “B” or “W”, but with a different algorithm.\nWe will add line numbers in the next version to make it clearer.\n\n&nbsp;\n### Significance Test (Questions for the authors)\nWe present the **t-test** results as follows, where we run each experiment with 3 random seeds when finetuning on parallel data:\n\n&nbsp;\n### \n**Java-to-Python:**\n|               |     BLEU     |      CB      |     CA@1     |\n|---------------|--------------|--------------|--------------|\n| Best baseline |     79.9     |     73.2     |     69.4     |\n| CMTrans       | 82.1 ± 1.4** | 76.1 ± 1.4** | 72.1 ± 1.6** |\n\n&nbsp;\n### \n**Python-to-Java:**\n|               |     BLEU     |      CB      |     CA@1     |\n|---------------|--------------|--------------|--------------|\n| Best baseline |     81.4     |     78.4     |     62.0     |\n| CMTrans       | 84.3 ± 0.4** | 82.0 ± 0.5** | 64.6 ± 1.0** |\n\n&nbsp;\n### \n**C++-to-Java:**\n|               |    BLEU    |      CB      |     CA@1    |\n|---------------|------------|--------------|-------------|\n| Best baseline |    90.9    |     90.1     |     68.3    |\n| CMTrans       | 91.1 ± 0.4 | 90.3 ± 0.1** | 69.9 ± 1.1* |\n\n&nbsp;\n### \n**C++-to-Python:**\n|               |     BLEU     |      CB      |     CA@1     |\n|---------------|--------------|--------------|--------------|\n| Best baseline |     82.9     |     75.4     |     62.5     |\n| CMTrans       | 83.8 ± 0.6** | 76.8 ± 0.3** | 64.6 ± 0.8** |\n\n&nbsp;\n### \n**Java-to-C++**\n|               |     BLEU    |     CB     |    CA@1    |\n|---------------|-------------|------------|------------|\n| Best baseline |     89.3    |    88.5    |    84.6    |\n| CMTrans       | 89.9 ± 0.4* | 88.1 ± 0.2 | 83.7 ± 1.1 |\n\n&nbsp;\n### \n**Python-to-C++**\n|               |     BLEU     |      CB      |     CA@1     |\n|---------------|--------------|--------------|--------------|\n| Best baseline |     79.8     |     77.9     |     59.1     |\n| CMTrans       | 82.1 ± 0.9** | 79.8 ± 0.5** | 63.2 ± 1.4** |\n\n**means significant results with p-value < 0.01\n\n*means significant results with p-value < 0.05\n\n&nbsp;\n### \n\nNote that we use computational accuracy (CA) as the primary evaluation metric because it directly reflects the functional correctness of the program, while BLEU and CodeBLEU (CB) only compare overlap with references. Our model significantly outperforms the baselines on CA@1 under 5 out of 6 language pairs.\n\n&nbsp;\n### The construct overlap statistics (Questions for the authors, point 2)\nWe compute the Construct Overlap Rate for each type of construct as follows: for each test example, we consider the ground truth label as 1 if the source input contains this type of construct, and 0 otherwise. Similarly, we consider the prediction result as 1 if the translation output by some model contains this type of construct and 0 otherwise. Then we compute the accuracy, recall, and F1 scores for each type of construct. The “construct overlap rate” in Figure 6 is the recall score.\n\nWe will also report the construct overlap accuracy and F1 in the next version of the paper. Below are the accuracy, recall, and F1 scores for CMTrans (CC. Only) (no finetuning). \n\n&nbsp;\n### \n**Java-to-Python:**\n|                                                   |  loop  |   if   | else if |\n|---------------------------------------------------|:------:|:------:|:-------:|\n| Accuracy                                          | 0.9582 | 0.9119 |  0.2857 |\n| Recall  (the Construct Overlap Rate in our paper) | 0.8418 | 0.7461 |  0.2162 |\n| F1                                                | 0.8962 | 0.8207 |  0.2462 |\n\n&nbsp;\n### \n**Python-to-Java:**\n|                                                   |  loop  |   if   | else if |\n|---------------------------------------------------|:------:|:------:|:-------:|\n| Accuracy                                          | 0.8744 | 0.8822 |  0.4444 |\n| Recall  (the Construct Overlap Rate in our paper) | 0.9972 | 0.7868 |  0.1081 |\n| F1                                                | 0.9318 | 0.8317 |  0.1739 |\n\n\nWe will revise the Figure captions and the explanations of the experiments in the next version.\n\n&nbsp;\n### Typo in line 450 (Questions for the authors, point 3)\nLine 450 does not contain “15”. Do you mean the “our model generates more than 15 unique translations for 214 test examples” in Line 459? \nWe are sorry for the confusion. We intended to describe the statistics of the “16-20” group in Figure 7(a). In Java-to-Python translation, there are in total 464 test examples. For 214 of them, CMTrans (MultiRef. Only) generate >=16 unique translations. In contrast, CodeT5 and TransCoder-ST-ft only generate >=16 unique translations for 145 and 146 examples.\nWe will rephrase the result descriptions here in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* well written paper, good structure and easy to understand\n* good experimental method and positive results\n* thorough specification of hypotheses and analysis based on them", "reasons_to_reject": "* substance: the authors do not suggest a new model but pipeline several models as a preprocessing step\n* fit for the conference: I have a doubt because code translation is not really NLP but I have seen many similar papers in these venues, so I don’t think this is irrelevant to the conference\n* this paper extends on prior experiments of a particular mode. For readers who are not very fit with those experiments, there are some gaps in the understanding. \n* the authors train a model similar to a previous paper. No details of this model are given here, that might be an issue for reproducibility", "questions_to_authors": "Could you provide some statistical significance testing (e.g. bootstrap resampling) for the BLEU scores - at least for the ones that have less than one point difference?", "paper_topic_and_main_contributions": "The paper deals with code translation and suggests two methods of data augmentation in order to solve the problem of data scarcity. For this, they create comparable corpora by using a code generator for the target language, prompted based on a docstring. Secondly, they generate multiple references (as training material too) by using a fine-tuned model. The automatically generated references are filtered by using test cases. \n\nThis seems to be the first paper that uses comparable corpora for code translation. Both methods seem to have positive impact as compared to the baseline model and are additionally useful if the conditions are even more low resource.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* figure 2 must be missing some lines, particularly the coloured ones.\n* the construct overlap statistics are not very well described\n* line 450, the number 15 must be a typo, the figure shows 150"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents data augmentation techniques to generate comparable corpora for using machine translation for code translation (e.g. from Python into Java). The paper is well written, and the experimental results show that their approach outperforms CodeT5.

As the paper is heavily based on previous work, the contributions of the present work and its novelty, as compared to the previous studies, should be made more  clear.  The confusion regarding the experimental design should be clarified. Statistical significance testing should be added to the paper (thank you for providing the numbers in the author response).
Decision: Accept-Findings