Leveraging Automated Unit Tests for Unsupervised Code Translation
Authors: Baptiste Roziere
Jie Zhang
Francois Charton
Mark Harman
Gabriel Synnaeve
Guillaume Lample
Venue: ICLR 2022 Spotlight (2022)
Keywords: unsupervised
translation
code
self-training
pseudo-labelling
unit tests
programming languages
deep learning
transformer

Abstract: With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation. However, the majority of unsupervised machine translation approaches rely on back-translation, a method developed in the context of natural language translation and one that inherently involves training on noisy inputs. Unfortunately, source code is highly sensitive to small changes; a single token can result in compilation failures or erroneous programs, unlike natural languages where small inaccuracies may not change the meaning of a sentence. To address this issue, we propose to leverage an automated unit-testing system to filter out invalid translations, thereby creating a fully tested parallel corpus. We found that fine-tuning an unsupervised model with this filtered data set significantly reduces the noise in the translations so-generated, comfortably outperforming the state-of-the-art for all language pairs studied. In particular, for Java→Python and Python→C++ we outperform the best previous methods by more than 16% and 24% respectively, reducing the error rate by more than 35%.

Published as a conference paper at ICLR 2022
LEVERAGING AUTOMATED UNIT TESTS FOR UNSUPERVISED CODE TRANSLATION
Baptiste Rozière
Jie M. Zhang University College London zhangj e fb. com
François Charton
Facebook Al Research Paris-Dauphine University broz fb. com
Facebook Al Research fcharton fmmc com
Mark Harman
Gabriel Synnaeve Facebook AI Research gab fb. com
Guillaume Lample Facebook AI Research glample8 fb. com
Facebook
ma markharman@ fb. com
ABSTRACT
With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation. However, the majority of unsu- pervised machine translation approaches rely on back-translation oped in the context of natural language translation and one that inherently involves training on noisy inputs. Unfortunately, source code is highly sensitive small changes; a single token can result in compilation failures or erroneous programs, unlike natural languages where small inaccuracies may not change the meaning of a sentence. To address this issue, we propose to leverage an automated unit-testing system to filter out invalid translations, thereby creating a fully tested parallel cor- pus. We found that fine-tuning an unsupervised model with this filtered data set significantly reduces the noise in the translations so-generated, comfortably out- performing the state-of-the- the-art for all language pairs studied. In particular, for Java + Python and Python - C++ we outperform the best previous methods by more than 16% and 24% respectively, reducing the error rate by more than 35%.
method devel-
INTRODUCTION
Ancient languages such as COBOL still underpin much of the financial industry and government ser- vices. Their outdated structures and thinning developer bases induce costs and severely slow down development, prompting businesses to modernize their codebases. For instance, the Commonwealth Bank of Australia spent around $750 million over 5 years to migrate its COBOL codehase to more recent language. More generally, most large companies own code written in several programming languages, which can hinder interoperability and make programmers less efficient. Automatic trans- lation systems could make codebase migrations faster and cheaper, and help programmers learn new languages or understand existing code. Systems to automatically translate between program- ming languages with approximately the same level of abstraction are called transpilers or source- to-source compilers. They need to be distinguished from compilers which translate source code to a lower-level language. The particularities of some languages allow the creation of very successful rule-based transpilers for a few language pairs (e.g. Java-Scalla, SoffeeScript-JJvvaSSee ods leveraging verified lifting (Kamil et al. 2016), which offer formal guarantees, can significantly speedup some pre-defined code fragments (Ahmad & Cheung 2016 Ahmad et al.) 2019).
Meth-
However, source-to tonicceto-oouue source translation for arbitrary programming languages is still an open problem. Rule-based systems are commonly used, but they are never exhaustive due to the considerable num- ber of translation rules that should be written to translate every function and object from every stan- dard library. Unlike in natural languages, there is little to no parallel data available for source code, making it impossible to train standard machine translation models. Recently, TransCoder (Roziere) ct al. 2020) showed that unsupervised methods can be used to translate source code. However, is
1 Work done while Facehook

---

Published as a conference paper at ICLR 2022
trained without any supervised signal and only learns the semantics of tokens from their contexts. As shown in Figure [ it can confuse tokens that have different semantics in different languages, for instance the float division in Python and integer division in C++ and Java which use the token or more subtle operator priority differences (e.g. Java prioritizes == over & unlike Python). While small inaccuracies often merely hinder comprehension in natural languages, they often make the entire translation erroneous in the context of programming languages.
Input function
TransCoder
TransCoderSTT
del 20_ return
static boolean 1a0dd (int x) return
static boolean isodd/int return X
static void printbrint XI
def 0r] while 0z print (x t 2:21 /=
def prin
while 1x 01:
while 01 print (x 21 x // 2
systom.ott. ot..prriiiiiiiii 237 27
static String reverse (chael str)
def severselstrll
def reverse (datal: at I| for data st. stpeeedddtet
Stack scheracter> at new Stack (int 0j idstz.lengshy 244)
Stack
in range (stri):
st st..puhietrr lills
st st-gush [ [1]]
for (int 1 3j i<sts. 241
for i in range (1en
for in range (lon
str st.ppek
str st.pop
dita(i) st(1) st.pep() return ".jodaldass)
st.
st.pushietr
return str
return String. wwl.eoe
Improvements over TransCoder. The first function returns whether an input integer is odd and is translated from Python to Java. The translation of TransCoder does not compile because the == operator has precedence over in Java, and parentheses are required unlike in Python. The second example is function that prints an integer in base two, which is translated from Java to Python. TransCoder translates does not modify the expression x/=2, even though it corresponds to the integer division in Java and to the float division in Python. In the third example, function reversing a char array. TransCoder does not manage to translate the Java Stack object into the right Python object and uses the unsafe str parameter name. In all three cases, TransCoder-ST manages to leverage the semantics contained in unit to translate the function correctly.
Figure I:
TransCoder leverages back-translation (Sennrich et al. 2015), an effective data-augmennation scheme where the model translates source sequences generate training data for the target-to- source direction, and vice versa. Although being highly effective in low-resource translation, back- translation also has issues, as the model is trained on potentially invalid input-output pairs. Neural machine translation models being highly sensitive to input noise (Belinkov & Bisk, 2018 Khayral- lah & Koehn, 2018), this can severely deteriorate the performance. Fortunately, many programming languages come with relatively mature tools and technologies for automated test data generation. In this paper, we propose to leverage these tools to guide the translation process, weeding out unsuc- cessful translations, thereby increasing the overall confidence in the machine translation process.
The topic of automated test data generation has been active for over three decades in the software engineering research community (Myers| 1979 Miller & Spooner 1976). There now many existing mature tools for test data generation, both open source research tools (Fraser & Arcuri, 2011 Lakhotia al. 2013 Cadar al. 2008), and production testing systems (Alshahwan al., Tillmann et al. 2014). Because of its pivotal impact on practical software engineering, auto- mated testing remains highly active research area (Anand al., 2013). with the result that future automated testing advances will lead to ongoing improvement in automated translation.
2018
We use one such open source automated test generation tool, EvoSuite (Fraser & Arcuri 201T). in this paper. EvoSuite is well-established test generation tool for Java which uses coverage metrics (Chekam al. 2017) and mutation scores (Jia & Harman 201T) to generate high-quality tests. has been widely used in the Software Testing research literature for test data generation although has not, hitherto, been used as part of an automated code translation approach, the topic of the present paper.
More generally, software testing tools have been largely ignored by the machine learning community (Zhng et al.) 2020). In this paper, we propose to use automatically created unit tests to guide unsupervised translation models for programming languages. More precisely, we create unit tests automatically for a large number of functions from the source dataset. Since the unit tests composed of simple inputs and asserts, they can easily be translated to semantically equivalent tests in the target languages using simple scripts. Using our unit-tests and pre-trained unsupervised
2

---

Published as a conference paper at ICLR 2022
translation model, create parallel datasets by translating functions and selecting the translations that have the same semantics as the original function for the tested inputs. Overall, we make the following contributions:
We introduce novel approach, TransCoder-ST (for Self- Self-Trained), that leverages au- tomated unit test generation pipeline to filter out invalid translations and reduce the noise coming from the back-translation process in unsupervised machine translation.
We present two implementations of this approach (online and offline), and show that it sig- nificantly outperforms the previous state of the art in code translation on all the language pairs we considered. In particular, Java, Python and C++ by an average of 12.6% Computational Accuracy (CA@ I), corre- sponding to average relative improvement of 25.5%. For Python - C++, we improve the CA@ 1 by 24%, reducing the error rate by 35.7% compared to previous models.
improve the state of the art for translating between
We generate multilingual unit tests for hundreds of thousands of Java functions and create large parallel dataset of 135,000 parallel functions between Java, Python, and C++.
Our method is completely unsupervised and could easily be generalized to other program- ming languages and unit test creation tools.
2 RELATED WORK
Unit Test Generation. Software testing is challenging due to the large number of possibilities to be tested, and the inherent cost of covering reasonable representative sample (Myers) 1979). When test design is performed by humans, the cost can be prohibitive. To reduce such cost, much research over the last three decades has focused on automating the process of test generation (Anand al,, 2013). Although automated test generation has been studied since the mid-1970s (Miller& & Spooner 1976 it was only in the last decade that industrial-strength tools have become widely available. There are now several test data generation tools for languages, including C (Cadar et al., 2008
Lakhotia et al., 2013) and Java (Fraser & Arcuri 201I). Popular test data generation techniques include symbolic execution of the code (Cadar& & Sen 2013), dynamic execution guided by fitness function (Harman et al. (013)), and hybrids of these two techniques (Baars et al., (0IT). Recently,
neural networks have also been used successfully to generate unit tests (Tufano al. 2020
One of the most well-established and widely-uses open source tools for test data generation is the EvoSuite system (Fraser & Arcuri 2011). EvoSuite uses search based software engineering (SBSE) (Harman et al., 2012) to generate test cases. Like all SBSE techniques, EvoSuite is guided by fitness functions, in this case aimed capturing the test suite's coverage and mutation score of the code being tested. We use EvoSuite in our work for three reasons: it is publicly available in open source (thereby facilitating replication), it is under current active development (thereby supporting future work), and it is widely used by other researchers (thereby enabling interoperabilityy. The test framework can be considered as a parameter in our overall approach and could be substituted with another.
In order to assess the effectiveness of the test suites generated, we use mutation testing, topic also widely-studied since the 1970s (DeMillo et al. 1978). A mutant is version of the program into which a fault deliberately inserted, thereby assessing the test suite's fault detection ability (Jia & Harman 2011 Papadakis et al. 2019). For given set of mutants and test suite, the mutation score is defined to be the proportion of mutants for which the test suite distinguishes the behavior of the mutant from that of the original program. The mutation score is thus proxy for the fault- revealing power of the test suite on a set of simulated faults (the mutants). Mutation scores have been empirically demonstrated be correlated real fault revelation (Chekam al. 2017). motivating our adoption of this approach.
Machine Learning for Programming Languages. In recent years, deep learning methods have been used to tackle various tasks in software engineering, with particular interest in bug detection
and repair (Wang et al. 20188] Chen et al. 2019 Allamanis et al. 2018 Tarlow et al. 2020 Murali eet al. 2021 Dinella al. 22200; Yasunaga & Liang 2020 Tufano et al. [019 Drain et al.
2021
and code completion (Li et al. 2018) 8 Liu et al. 2020 Kim et al. 20.1 Svyatkovskiy et al.
2021
Unsupervised pre-training methods for code based on BERT (Kanade et al. 2020) Feng al.
2020
3

---

Published as a conference paper at ICLR 2022
Training
Python Translation
def naxía, bl:
Java Python
return a if b etse
New parallel data Python
Input code
Python Test
static int max int b){
def Leste():
Running unit tests
return b
assert nax(0,
Multilingual Unit Test Ganerator
def testi():
assert nax -1) =
def test2(1:
Our iteratíve self-training method. Using EvoSuite, generate unit tests in Java, Python and C++ corresponding several input Java functions. With machine translation model (e.g. TransCoder), generate several candidate translations of the Java function Python C++. Generated translations that pass the unit tests are used to create paralicl dataset on which we fine-tune the model. Discarding translations that fail the unit tests reduces the noise of data coming from the back-translation process, and significantly improves the overall performance of the model.
Figure 2:
BART (Ahmad et al. 2 or other objectives tailored to source code (Guo et al. 2020 Roziere
al.
2021) have shown strong results on benchmarks such as CodeXGLUE (Lu et (021).
Recently, Hendrycks et al. (2021) evaluated the competence of several language models for solving coding challenges. Chen al. (202T) trained large model to generate programs from docstrings and are able to solve 28.8% of the problems in their HumanEval dataset. Austin et al. (2021) also evaluated the capabilities of large language models for generating code solving problem statements written in natural language. The goal of code translation is also to generate code solving a specific problem, but the input (code written a different language) is more precise and often more concise.
Translation of Programming Languages Several studies used statistical methods to translate between programming languages. Early methods extracted parallel datasets and trained phrase- based models translate between C# and Java (Nguyen al., 2013 Karaivanov et al. 2014) or from Python 2 to Python 3 (Aggarwal et al. 2015). Later, Chen et al. (2018) proposed tree-to-tree neural network to translate between CoffeeScript and JavaScript and between C# and Java using the dataset created by [Nguyen et al. (2013). However, these approaches are limited to few language pairs which small parallel datasets were created manually (e.g. C#-Java) or can be created with rule-based tools (e.g. Python 2-Python 3 and CoffeeScrrit--JaaSSee
Instead, Roziere al. (2020) proposed TransCoder, an unsupervised model that leverages the prin- ciples of unsupervised machine translation (Lample et al. (2018), translate between Python, Java and C++. They showed that their method outperforms well-established rule-based baselines, does not require any parallel data expert knowledge, and easily be generalized to other languages. They pre-trained their model with the Masked Language Modeling (MLM) objective of Devlin al. (2018), and trained with the denoising auto-encoding (DAE) (Vincent al. 2008) and the back-tranllation (BT) (Sennrich et al. 2015) objectives. Later, Roziere et al. (2021) 1) showed that augmenting MLM with deobfuscation objective (dubbed DOBF) can substantially improve the performance of TransCoder. In the rest of the paper, we will refer to their model as DOBF.
Even though unsupervised methods can be trained on large amounts of data, they sometimes lack the signal needed to differentiate between semantically different tokens that often occur in similar contexts (see Figure [T. There is need for method providing supervised signal directly related to the semantics of the code without manually crafted parallel datasets.
3 METHOD
3.1 PARALLEL DATA CREATION
Parallel unit test generation: We use EvoSuite to automatically generate unit tests for Java func- tions. EvoSuite is well-established open source tool for automated test generation in Java, which is
4

---

Published as a conference paper at ICLR 2022
still under active development and frequently used. It is designed for Java programs but its search- based technique is general and could be used for any programming language. Unit tests can be thought of as lists of imputs and asserts testing the semantics of program (e.g.. the output of the function, the side effects on its arguments such as sorting the input list). EvoSuite uses evolutionary methods derive tests that maximize criteria such code coverage mutation score. During its search, each candidate solution in EvoSuite is a test input. The candidate inputs are evolved using crossover and mutation, and filtered by a fitness function (e.g., mutation score). With each genera- tion the fitness improves until it reaches a plateau or the budget is exhausted. The final test inputs are wrapped up as test cases. Each program is associated to a test suite containing a series of test cases. Figure 3 shows an example of a test case generated by EvoSuite.
Java function
A generated unit test
public class CLANP CLASS
(t irouut 4000)
public static double clamp
public void test. ()
throws Throwable
double double min, double return asminPmin: (armax/max:a)
double doubles
Exemple.clang
742.9, 0.0, 0.0);
assertEquala (Q.0, doubled, 0.0117
Figure 3: A unit test generated by EvoSuite. The Java function clamps the given value between the given min and maz. This test case is not sufficient to test the semantics of the function thoroughly but could be part of a suitable test suite. See Figure Flin the Appendix for a generated test suite with high mutation score.
Parallel test suites selection: Some test suites created by EvoSuite only cover a few parts of the semantics of functions. We only trust the translations verified by test suites which examine the function semantics thoroughly. We use the mutation score, which is the most effective test assessment metric in the literature (Jia & Harman 2011), to pick out these test suites. The mutation score is computed through mutation testing, in which mutants (i.e., program variants with syntactic changes) are generated from the original program based details in Appendix A.2 A mutant is said to be killed if at least one test from the test suite has different results on the mutant and the original program. Otherwise, the mutant is said to survive. The mutation score is the ratio of killed mutants. A test suite with a higher mutation score checks the code semantics more thoroughly. We adopt a strict strategy test suite selection: we keep only the Unit test suites with mutation score larger than 90% for building the parallel dataset.
set of transformation rules (more
Parallel dataset building: The generated test suites can be used to test the semantics of programs written in any programming language as long as there is clear mapping between the types of the output and parameters in the original language and the language of the translated unit tests. We transform the generated Java tests into C++ and Python tests with identical inputs and expected outputs and side effects (i.e., assertions). In practice, selected the Java functions which can be compiled and run in isolation and with simple output and parameter types. These types are the Java primitive types (e.g. int, long, bool, float...), standard data types (e.g. Integer, Double, string...), array and List or ArrayList types of elements of supported types (e.g. double [1, List<Integrr>....
We use the best unsupervised translation models available for Java to Python and Java to C++ trans- lation, namely TransCoder (Roziere et al., 2020) for Java to C++ and DOBF (Roziere et al. 2021) for Java to Python. For each Java function, we generate 20 Python and C++ translations with heam search and select the first element in the beam that passes the unit tests. The created tests are exe- cuted against the translated functions. If all the tests pass, the Python and C++ functions have the same semantics assessed by the generated tests. Our method is illustrated in Figure 2
3.2 TRAINING METHOD
Our parallel data generation method relies on a pre-existing model to translate from Java to Python and C++. There is little parallel data for these tasks and the best performing published models are unsupervised. TransCoder (Roziere et al. 2020) is trained using the MLM, denoising and back- translation objectives and is able to translate between Java, C++ and Python. DOBF (Roziere al.) 2021) provides clear improvements over TransCoder for translating between Java and Python but was not trained C++. Therefore, we use DOBF translate from Java to Python and TransCoder
5

---

Published as a conference paper at ICLR 2022
Table 1:
Size of the parallel datascts generated offline at each iteration.
Languages
First iteration
Second iteration
Third iteration
Fourth iteration
Java + C++
27,875
37,769
47,729
60,495
Java ++ Python
33,496
43,194
43,956
45,311
C++ + Python
14,935
21,026
27,080
32,869
to translate from Java to C++. When fine-tuning, we also reload these models. For DOBF, we initialize the C++ language embeddings with those of Java.
The parallel examples we generate can be used to improve the performance of pre-existing trans- lation models. Since the number of examples we generate also depends on the performance of the translation model, it creates positive feedback loop where improving the model allows to improve the parallel dataset which in turn can be used improve the model again. We propose offline and online approaches to use our method to maximize the unsupervised translation performance.
Offline training. With the offline training method, we use the method described in Section 3.1 to create parallel Java ++ Python, Java ++ C++ and Python ++ C++ datasets using every input Java function we selected. For the first iteration, we fine-tune the model on these parallel examples until convergence. We can iterate this process by selecting the best checkpoints for Java - Python and Java + C++ using the validation dataset and using them to generate new parallel datasets, which in turn be used train a better model. We iterate this process until convergence, i.e. when we see no significant improvements on the validation set.
Online training. With the online method, we create parallel examples on the fly while training the model. Compared to the offline method, it allows to always use the last model to generate new examples and much more convenient to automate. However, this process can be unstable done naively. For instance, the model can start over-fitting only few examples and stop generating anything that passes the unit tests for any other example. In order to stabilize the training, we follow Likhomanenko et al. (2020) and implement cache mechanism storing the previous examples that passed the unit tests. At each step, the model can either train on parallel functions sampled from the cache or create new parallel functions to add to the cache. When example is sampled, we remove from the cache with a given probability. The online training allows the model to always benefit from the performance of the latest model and the cache mechanism ensures that the model does not forget the correct examples that it was able to generate at previous time steps.
3.3 EVALUATION
In the context of natural languages, machine translation models are generally benchmarked against a reference solution using the BLEU score (Koehn 2009; Bahdanau et al., 2015 Vaswani et al., 2017]. Early studies on source code translation used the same metric evaluate the quality of the generated functions (Nguyen et al.) 20133) Karaivanov et al.] 2014 Aggarwal et al.] 2015 Miceli- Barone & Sennrich, 2017), or the exact match score which requires the translation to exactly equal to the ground truth (Chen et al. 2018). However, these metrics fail to capture the semantics of the code and typically correlate poorly with the correctness of the generated function, prompting the use of new metrics checking if the generated solution passes series of test cases (Kulal et al. 2019
Roziere et al. 2020; Hendrycks et al. 2021 Chen et al. 2021 Drain et al. 2021).
We evaluate models the full validation and test sets of TransCoder. It contains few hun- dreds of parallel functions extracted from GeeksforGeeks along with associated unit tests. As our TransCoder and DOBF baselines, we evaluate our models with the CA@N metric, which checks if any of the top-N solutions proposed by the model passes all the corresponding unit tests. This metric can be computed independently of the beam size (as long as the beam size is greater or equal to N).
EXPERIMENTS
4.1 TRAINING DETAILS
Model architecture. We use a sequence- to-eeee-------enceeeeee model with attention composed of an en- coder and decoder model with transformer architecture (Vaswani al. 2017). In order to pro-

---

Published as a conference paper at ICLR 2022
Computational accuracy scores for methods and baselines. We show the CA@1 metric computed with beam size 10. For the baselines, we ran the evaluations again reported the best result between those reported in the original paper and those we obtained. Both the offline and online self self--raining methods lead to significant improvements over our baselines for every language pair and direction. Online self-training outperforms offline self-training, even after several iterations.
Table 2:
C++ + Ja
C++ + Py
Ja la+C++ C++
Ja + Py
Py + +++
Py + Ja
AVG
TransCoder
65.1%
47.1%
79.8%
49.0%
32.6%
36.6% 45.7%
51.7%
DOBF
52.7%
Offline ST
65.5%
56.2%
81.6%
61.8%
46.8%
55.1%
61.1% %
Offline ST 2
65.5%
58.3%
83.7%
63.3%
46.4%
52.2%
61.6%
Offline ST
66.5%
56.2%
85.2%
66.3%
48.1%
56.6%
63.1%
Offline ST 4
65.3%
48.2%
81.1%
58.1%
48.9%
54.7%
59.4%
Online ST
68.0%
61.3%
84.6%
68.9%
56.7%
58.2%
66.3%
vide fair comparisons, we use the exact same architecture as TransCoder: an encoder and a decoder of 6 layers each, a hidden dimension of 1024 and 8 attention heads. We limit the size of the input to 512 tokens. Rozzere et al. (202T) train models with two different architectures. For Java <> Python, we compare ourselves to the version of DOBF using the same architecture as TransCoder. We ini- tialize our models with either the best TransCoder checkpoint for Java - C++ or the best DOBF checkpoint for Java - Python with C++ language embeddings initialized with those of Java.
Datasets. As TransCoder and DOBF, we use the GitHub public dataset available on Google Big- Query filtered to keep only projects with open-source licenses]] As our unit test creation tool can only be used on Java code, we only use the Java files and we select only the functions that can be compiled in isolation. We obtain dataset containing 333,542 Java functions. We run EvoSuite with a budget of 20 seconds and criterion including the line, branch, cbranch and output coverages, as well as the weak and strong mutation scores. We set the maximum absolute value of integers that can be generated as an input to / -I to limit the number of overflows. We manage to obtain high-quality (mutation score >> 0.9 and at least two asserts) test cases for 103,488 functions. See Figures 3 5 6 in the appendix for examples of selected and filtered out test suites.
Training details. During the training, we alternate between batches for every source and target language so that language pairs for which we managed to create more parallel examples are not overrepresented in our training batches. For the online version, we set a cache warm-up parameter ensure that we always generate new parallel examples if there are less than 500 examples in the cache for any language pair. Otherwise, we sample from the cache with probability 0.5, generate new examples, train on them once and put them in the cache also with probability 0.5. The sampled elements are removed from the cache with probability 0.3, so that each element we create is trained on about 4 times in average before being removed from the cache. We initialize the cache with parallel examples created offline.
During beam decoding, we compute the score of generated sequences by dividing the sum of token bog-probabilities by n where / is the sequence length. We found that taking 0.5 (and penalizing long generations) leads to the best performance on the validation set.
4.2 RESULTS AND DISCUSSION
Results. In Tables 2 and we compare the results of our offline and online training methods with those of TransCoder and DOBF. DOBF outperforms TransCoder for the Java ++ Python pair. We compare our models against the best baseline for each language pair and direction.
Training the generated parallel examples brings substantial improvements for every language pair, direction, and metric. Offline training already provides clear improvements over the baseline after iteration. The computational accuracy (CA @ 1) computed with beam size 10 is higher for every direction and it is substantially higher for the language pairs involving Python. It allows reduce the emoor rate of the best baseline by 25.5% for Java Python. In average, it increases the CA@ 1 by 7.4% over the best previous models, and reduces the error rate by 16.6%. In the
We select the open-source licenses: 'apache-2.0', 'mit', 'gpl-2.0', 'gpl-3.0', 'bsd-2-clause', 'bsd-3-clause'

---

Published as a conference paper at ICLR 2022
Table 3: CA @n metric for several beam sizes averaged all language pairs. The value k corresponds the beam size. For instance, CA@1 1 k=10 means that we use beam decoding generate 10 translations, and select the one with the highest score. The best baseline corresponds to taking the best model between TransCoder and DOBF for every language pair and direction. The error rate reduction of the offline and online self-training methods over the best baseline are high (> 20%) across all CA@ N metrics and beam sizes.
@A@1 k=1
CA@1 1 k=10
CA@1 k=20
CA@ 10 k=10
CA@ 20 k=20
Best baseline
32.2%
53.7%
53.4%
67.3%
70.5%
Offline ST
60.8%
61.1%
61.1%
72.9%
75.3%
Offline ST
61.4%
61.6%
61.4%
73.3%
75.8%
Offline ST 3
61.7%
63.1%
63.0%
73.3%
75.8%
Offline ST
58.5%
59.4%
59.2%
70.8%
73.6%
Online ST
6
66.3 %
66.3% %
75.4%
77.2% 2%
Table 4: Ablation study. We show the CA@1 metric computed with greedy decoding evaluation time except the last line where the beam size is set 10. evaluate models trained with no cache system, without initializing the cache (with or without selecting the tests with a minimum mutation score of 0.9), and a beam size of I when generating examples. We also compare the CA@ score of our full model when evaluating with greedy decoding and with beam size 10. Using a pre-filled cache and selecting only the tests with high mutation score lead substantially better performance, although these steps are not necessary to outperform our baseline. The online method already performs well with greedy decoding at generation time, but generating with beam size 20 further improves the results.
C++ Ja
C++ + Py
Ja C++
Ja + Py
Py + C++
Py
Ja
AVG
No cache
66.5%
52.7%
83.7%
60.3%
41.2%
51.8%
59.4%
Cache not initialized
64.9%
51.6%
82.4%
62.4%
46.6%
52.6%
60.1%
No min mut, score
64.0%
50.1%
82.6%
60.9%
47.4%
47.0%
58.7%
ST greedy decoding
65.9%
54.2%
82.2%
60.9%
56.2%
56.6%
62.7%
Full model (ST beam 20)
66.7%
61.1%
84.1%
67.8%
52.2%
56.7%
64.7%
Eval beam
68.0%
61.3%
84.6%
68.9%
56.7%
58.2%
66.3%
two next iterations, the model is trained on significantly more examples (see Table T It results in average improvements of 2% points between the first and third iteration. Although the model for the fourth iteration is trained on more parallel samples, its performance on the test set of TransCoder is actually worse than after the third iteration. After three iterations, the model learned generate more samples that pass the unit tests but some of them are actually incompatible with the types of translations expected by TransCoder (e.g. example with overflows in Figure 4). causing the computational accuracy score to go down.
The online self-training method provides further improvements over training the pseudo-labeled examples offline. It outperforms every other method in every case except the third iteration of of- fline training for Java + C++. In average. this model outperforms the baseline by 12.6% points, corresponding to an eтror rate reduction of 25.5%. For Python - C++, it improves previous perfor- mance by more than 24% points, which corresponds to reducing the error rate by 35.7%. Examples of avoided errors can be found in Figure T and Appendix B Overall, all our models significantly improve previous results. As shown in Table 3 these improvements are stable across several beam sizes and CA @n metrics. The CA 20 metric shows that the number of examples for which none of the 20 elements in the beam are correct is reduced by more than 22% with online self-training. indicates that, even though we train only on the output of the model, our method does much more than reordering the elements in the beam and allows the model to find correct solutions that were not assigned a high probability by the baseline model. See Table in the appendix for more results.
Ablation study. The results of our ablation study are shown in Table Training online with no cache makes the training much less stable. The model improves at the beginning of training and we can select a few checkpoints where it performs well, but it ends up over-fitting a few examples it generated and the performance drops after a few epochs. Starting with an empty cache slows down the training and hinders generalization, leading the minimum mutation score requirement for the model with no initial cache, which leads to even lower scores the model is trained partly on lower-qualle parallel data.
clear drop in performance. We also try removing
All these models were trained using a self-training beam size of 20 when generating new examples. Training with greedy decoding is much faster since computing the results for all the 20 elements
8

---

Published as a conference paper at ICLR 2022
of the beam is costly. However, generating new examples with greedy decoding leads to loss of about two percentage points in average compared to our full model using beams of size 20. shows that initializing the cache of the model with beam size 20 is not sufficient and creating new examples with beam search is necessary to reach our best performance. Our full model provides some improvements over the ablated versions for every language pair and direction, except over the model trained with greedy decoding for Python - C++ translation. Evaluating with beam size 10 (still returning only the first element) leads some improvements for every language pair.
Limitations. We found that the unit tests we create with this method are sometimes incompatible with those of the test set of TransCoder, and that the capacity of model to generate functions that pass these unit tests is not perfectly correlated to its score on the test set. It raises the deeper issue of defining what constitutes a correct translation. For instance, most programmers would translate a factorial function implemented with 1 long integers into a factorial function implemented with Python's 's integer type. However, these functions are not semantically equivalent since the Java implementation would return negative number for the input 21 due to integer overflow while the Python implementation would return 21! correctly. The human developers who wrote the parallel functions in the test set of TransCoder often assumed that these functions would only be used on a limited domain where no overflow occurs (see Figure 4. However, the test cases of EvoSuite and TransCoder are not limited this domain and they sometimes assert different semantics. By using the test suites from EvoSuite as source of truth, we sometimes train the model to generate translations that are more rigorous but also less natural.
Input Java function
Gold translation
Translation passing multilingual tests
static int factorial (ánt n) E
def Eactorial (n) if
def factorial (n): np. int322 if x 2: return np. 311 return n. factorial 1)
return 1;
return n factorial In 1)
return
return factorial((-1)
Figure 4: Example of disagreement between our multilingual tests and the test set of TransCoder. The gold translation is only equivalent the input Java function on a small domain where there is no integer overflow and does not pass our unit tests. The version that passes the unit tests casts uses the np. 32 type, reproducing the behaviour of the original Java code but causing it to fail some of the unit tests of TrannCooder.
CONCLUSION
In this paper, we introduced novel method to grow a parallel corpus for automated code translation, from completely monolingual data. We leverage multilingual unit tests to filter good pseudo-labels, improving the model, and in turn the candidate translations. We show that both offline and online methods substantially improve the state of the art in unsupervised code translation, with an average improvement of 12.6% points in computational accuracy, and up to 24% points for Python - C++, corresponding to translation error rate reductions of 25.5% and 35.7% respectively, without using any unit test generation tool for Python and C++ (exclusively for Java).
Our method would automatically gain from improvements of automatic unit test generation tools. We could also increase the size of the dataset we generate by using test creation tools written for other languages in addition to Java, or by generating tests with EvoSuite on translated examples. Similarly, we could also extract the semantics of -irtten unit tests found in open-source projects to obtain larger, and possibly higher-quality datasets. In this paper, we focused on transla- tion correctness and our parallel example validation criterion was only based on semantics. could be supplemented with other requirements, such as specific code formatting or the output of linters to generate code verifying arbitrary criteria. Finally, the approach presented in this paper could eas- ily be transferred to natural languages. Although there is no concept of unit tests in natural language, traditional grammar and syntax checkers could be used to filter out some incorrect generations, and reduce the noise coming from the trecslaaanlttoon process. Neural machine translation systems being highly sensitive to noise coming from parallel data (Belinkov & Bisk 2018 Khayrallah & Koehn 2018), this may improve the performance in low-resource machine translation significantly.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY
We made sure to use the same architecture and framework as previous works in source code transla- tion so that our results are comparable (see Section 4.1). We submit our code with this submission, along with a ReadMe file detailing clear steps to reproduce our results, including script to set-up a suitable environment. We will open-source our code and release our trained models. Our models were trained using standard hardware (Tesla V100 GPUs) and libraries (e.g. Pytorch, Cuda) for machine-learning research.
ETHICAL CONSIDERATIONS
In this paper, we improve source code translation methods. Our methods could facilitate codebase migrations and interoperability, encouraging companies to move away from ancient programming languages and making software developers more efficient. Although increased efficiency could re- duce the number of developers needed to perform task, its impact on the labor market unclear as lower costs would also lower the bar for starting new projects and increase the demand for software engineers. Today, the demand for software engineering skills is high despite (or thanks to) the devel- opment of software (c.g. git, IDEs), programming languages (e.g. python), libraries (c.g. pytorch) and methodologies (c.g. continuous deployment) improving the efficiency software developers, and we believe that the development of automatic translation tools would not drastically affect the prospects of software developers either. the long term, the migration codebases written anti- quated programming languages (e.g. COBOL) could negatively impact experts in those languages, who are in particularly high demand at the moment. However, it would also benefit society by facil- itating debugging and updating software still used in most of our financial transactions and for many government services, and by increasing the demand for other software engineering skills.
REFERENCES
Karan Aggarwal, Mohammad Salameh, and Abram Hindle. Using machine translation for convert- ing python 2 to python 3 code. Technical report, PeerJ PrePrints, 2015.
Maaz Bin Safeer Ahmad and Alvin Cheung. Leveraging parallel data processing frameworks with verified lifting. SYNT@CAA 2016.
Maaz Bin Safeer Ahmad, Jonathan Ragan-Kelley, Alvin Cheung, and Shoaib Kamil. Automatically translating image processing libraries to halide. ACM Transactions on Graphics (TOG), 38(6): 1-13, 2019.
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655 2668, 2021.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations, 2018
Nadia Alshahwan, Xinbo Gao, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, Taijin Tei, and Ilya Zorin. Deploying search based software engineering with Sapienz at Facebook (keynote paper). In 10th International Symposium on Search Based Software Engineering (SSBSE 2018),
pp. 3-45, Montpellier, France, September 10th 2018 Springer LNCS 11036.
Saswat Anand, Antonia Bertolino, Edmund Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Jenny Li, Phil McMinn, and Hong Zhu. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software, 86(8):1978-25 2001, August 2013.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint (iXXx:2108.00733 2021.
10

---

Published as a conference paper at ICLR 2022
Arthur Baars, Mark Harman, Youssef Hassoun, Kiran Lakhotia, Phil McMinn, Paolo Tonella, and Tanja Vos. Symbolic search-base bascch-baedd testing. In 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011), pp- 53 62, Lawrence, Kansas, USA, 6th 10th November 2011.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In 3rd International Conference Learning Representations, 2015.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. In International Conference on Learning Representations, 2018.
Cristian Cadar and Koushik Sen. Symbolic execution for software testing: Three decades later. Communications of the ACM, 56(2):82-90, February 2013. ISSN 0001-0782.
Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. OSDI, volume pp. 209-224, 2008
Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, and Mark Harman. An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In Proceedings the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017, pp. 597-608, 2017
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Ed- wards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In Advances in neural information processing systems, pp. 2547-2557, 2018.
Zimin Chen, Steve James Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: srqueecce-o-seqqenee learning for end-to-end program repair. IEEE Transactions on Software Engineering, 2019.
Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. Hints on test data selection: Help for the practical programmer. IEEE Computer, 11:31-41, 1978.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs// 810.04805, 2018.
Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learning Representations (ICLR), 2020.
Dawn Drain, Colin Clement, Guillermo Serrato, and Neel Sundaresan. Deepdebug: Fixing python bugs using stack traces, backtranslation, and code skeletons. arXiv preprint arXiv:2105.09352, 2021.
Zhangyin Feng, Daya Guo, Duyu Tang. Nan Duan, Xiaocheng Feng. Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Findings, pp. 1536-15500 1566,117,455 2020.
Processing:
Gordon Fraser and Andrea Arcuri. EvoSuite: automatic test suite generation for object-oriented software. 8th European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE '11), pp. 416-419, ACM, September 5th 9th 2011. ISBN 978-1-4503-0443-6
Gordon Fraser and Andrea Arcuri. Achieving scalable cnutationbbsse generation of whole test suites. Empirical Software Engineering, 20(3):783-811 2015.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. In International Conference on Learinng Representations, 2020.

---

Published as a conference paper at ICLR 2022
Mark Harman, Afshin Mansouri, and Yuanyuan Zhang. Search based software engineering: Trends, techniques and applications. ACM Computing Surveys, 45(():11:1-11:61 November 2012.
Mark Harman, Yue Jia, and Yuanyuan Zhang. Achievements, open problems and challenges for search based software testing (keynote paper). In 8 JEEE International Conference Software Testing, Verification and Validation (ICST 2015), Graz, Austria, April 2015.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. Xiv preprint arXiv:2105.09955 2021.
Yue Jia and Mark Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engincering, 37(5):649 - 678, September-October 2011.
Shoaib Kamil, Alvin Cheung, Shachar Itzhaky, and Armando Solar-Lezama. Verified lifting of stencil computations. ACM SIGPLAN Notices, (1)6):111-72) 2016.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning, pp- 5110-5121. PMLR, 2020.
Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. thrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium New Ideas, New Paradigms, and Reflections on Programming Software, pp. 173-184, 2014.
Huda Khayrallah and Philipp Koehn. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop Neural Machine Translation and Generation, pp. 74-83, 2018.
Seohyun Kim, Jimman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 160-162. IEEE, 2021.
Philipp Koehn. Statistical machine translation. Cambridge University Press, 2009.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy $ Liang. Spoc: Search-based pscudocode to code. Advances in Neural Information Processing Systems, 12:11906-1199 19919-996699,,6-1111111111111191 2019.
Kiran Lakhotia, Mark Harman, and Hamilton Gross. AUSTIN: An open source tool for search based software testing of C programs. Journal of Information and Software Technology, 55(1):112-125, January 2013.
Guillaume Lample, Myle Ott, Alexis Comneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In EMNLP, 2018.
Jian Li, Yue Wang, Michael R Lyu, and Irwin King. Code completion with neural attention and pointer networks. IJCAI, 2018
Tatiana Likhomanenko. Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, and Ronan Collobert. slimipl: Hanguage-model-ffe iterative pseudo-labeling. arXiv preprint arXiv:2010.11 15544, 2020.
Fang Liu, Ge Li, Bolin Wei, Xin Xia, Zhiyi Fu, and Zhi Jin. A self-attentional neural architecture for code completion with multi-task learning. In Proceedings of the 28th International Conference on Program Comprehension, pp. 37-47, 2020.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy. Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint ar.ii:2202
2021.
Antonio Valerio Siccei-BBrron and Rico Sennrich. A parallel corpus of python functions and docu- mentation strings for automated code documentation and code generation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 314-319, 2017.
12

---

Published as a conference paper at ICLR 2022
W. Miller and D. Spooner.
Automatic generation of floating-pone point test data. IEEE Transactions Software Engineering, 2(3):223-22) 1976.
Vijayaraghavan Murali, Lee Gross, Rebecca Qian, and Satish Chandra. Industry-scale ir- ir-based bug localization: A perspective from facebook. In 2021 IEEE/ACM 43rd International Conference Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 188-197. IEEE, 2021.
Glenford J. Myers. The Art of Software Testing. 0-471-04328-1.
Wiley Interscience, New York, 1979. ISBN
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 651-654, 2013.
Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. Mutation testing advances: analysis and survey. In Advances in Computers, volume 1 12, pp. 275-378. Elsevier, 2019.
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33, 2020.
Baptiste Roziere, Marie-Anme Lachaux, Mare Szafraniec, and Guillaume Lample. DOBF: A deob- fuscation pre-training objective for programming languages. arXiv preprint arXiv:2102.07492, 2021.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 86-96, 2015.
Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Vicente Franco, and Miltiadis Allamanis. Fast and memory-efficien neural code completion. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pp. 329-344 IEEE, 2021.
Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, and Edward Aftandilian. Learning to fix build errors with graph2diff neural net- works. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, pp. 19-20, 2020.
Nikolai Tillmann, Jonathan de Halleux. and Tao Xie. Transferring an automated test generation tool to practice: From Pex to Fakes and Code Digger. In 29th ACM/IEEE International Conference on Automated Software Engineering (ASE), pp. 385-396, 2014.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. An empirical study learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1- 29, 2019.
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers. arXiv preprint arXiv:2009.05617 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pitrre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Ke Wang, Rishabh Singh, and Zhendong Su. Dynamic neural program embeddings for program repair. In International Conference on Learning Representations, 2018.
Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference on Machine Learning, pp. 10799-10808 PMLR, 2020.
13

---

Published as a conference paper at ICLR 2022
Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering, 2020.
14

---

Published as a conference paper at ICLR 2022
A MULTILINGUAL UNIT TESTS CREATION
A.1 GENERATED UNIT TESTS
Java function
Generated test suite
public static int pow (int b, el int
public void ()ato throwa Throwsble int into Example. 1111, (-1)17
17
while (e
sssertEquale (2, int0);
AE ((e 6 11 --
1)
b - bj
public void teats () throwa Throwable int into Sxample.pow101 assertEquals int0))
return 2)
public void teat2 throws Throwable int into Exarple. pow 2743);
intl)s
public void teat3() throwa Throwable int Example.pow il, 1)
assertFquats intD);
Figure 5: A generated unit test suite with high mutation score. The mutation score of this test suite 95% and we selected in our dataset for pseudo-labelling. The third test case (i.e. test2) may be too strict as it would make translations using the python int type fail the unit tests.
Java function
Generated test suite
public static int sizsBits_cmd() ( reburn B,
public void test0 throws Throwsble sssertEquale (8, wizebeeee.eeiietee.o ())
Figure 6: A test suite with a good mutation score but only one assert. Even though it contains only one test and assert, this test suite tests the semanties of the function the left properly since it only returns a constant and its mutation score is 100%. We found that test suites with good mutation scores and only one assert generally correspond to uninteresting input functions. Removing these functions and tests from our dataset for self labelling improves the performance of our model.
As discussed in Section 3.1 we only generate unit tests for static functions with selected return and parameter types. It makes it easy to map the types of inputs and outputs in Java to Python C++ types in the translated unit tests. While most of the unit tests are translated correctly, the translation sometimes fails due to EvoSuite generating test cases expecting exceptions. Our analysis shows that it happens for about 5.6% of all tests and less than 2% of the tests with high mutation scores. In that case, the candidate translations cannot pass the translated tests and no parallel examples created.
A.2 MUTATION SCORE
In mutation testing, mutants are programs transformed from the original programs based on series of syntactic transformation rules called mutation operators. Mutation testing consists in introducing minor syntactic faults on the code and running the tests against the mutated code. strong test suite is expected to detect the code changes by having at least one test failing. Table the examples of mutation operators adopted in EvoSuite when generating mutants (Fraser & Arcuri 2015).
A mutant is said to be killed by test case if the output of this test on the mutant is different from its output on the original program (i.e., the test fails the mutant). Otherwise, the mutant is said to have survived. Figure 7 shows example of mutant generated by changing the < in the return statement into >. The test with input (-800, -800, -1), as shown by Figure 33 does not kill this generated mutant, because its outputs on the original program and the mutant the same.
Mutation score is considered as the most effective criteria in accessing the fault-revealing ability of test suites. Other criteria, such code coverage, are weak: they check only whether the test
15

---

Published as a conference paper at ICLR 2022
Table 5: Examples mutation operators in EvoSuite.
Mutation operator
Explanation
Delete call operator
Remove method invocation
Delete field operator
Remove field access and replaces with default value (0 null)
Insert Unary Operator
Add to, subtract from, negate numerical value after it was loaded on the stack Replace an arithmetic operator in an expression with other operators. E.g., + - Replace constants with the special values -1, O, +1
Replace arithmetic operator
Replace constant operator
Replace variable operutor
Replace variables with other variables the same type
Original Java function
Mutant
public class CLAME CCAASS
public class CLAMP
public static double clamp
public static double clamp
double double min, double max) return ascmi Prrt nz (armax 7
double double min, double maxt return aomin im Jaamax amax:a)
Figure 7: A mutant generated by the "Replace arithmetic operator" mutation in EvoSuite. The < operator the return statement is replaced with >.
executes the code, but do not check whether the execution result is correct. A test suite without any assertions can achieve 100% code coverage, but could not detect any faults.
0.40
0.35
0.25
Frequency
0.20
0.15
0.10
0.00
0.0
0.2
0.4
0.6
0.8
10
Figure 8: Histogram of mutation scores for our generated unit tests. We select about 40% of the unit tests with threshold 0.9. Many of the remaining unit tests have mutation of 0.
B TRANSLATION EXAMPLES
16

---

Published as a conference paper at ICLR 2022
Input Python function
TransCoder C++ translation
TransCoder-ST C++ translation
daf rangeGcD In, returo if (c--m) else
int
range3CD ny retura n (n--m) 7 1: -2g
rangeeCh int
zeturn (a==a] ? melg
def count Set
int SauntSa (int n)
Int count n) in return 0;
12
zeturo
return 02
else:
else
else countsetBitt >5 1);
countsetBits 35
countSetBits (n 55 11
Figure 9: Python to C++ translation examples. TransCoder sometimes fails to capture the se- mantics of the incoming code and translates them to other expressions that could occur in similar contexts. Self-training helps the model to avoid such mistakes.
Input Python function
TransCoder Java translation
TransCoden-ST Java translation
csesini
public return mn 2) 7:1: 2)
public
return -1 iftn else
return in 1 0.7 -:1 1: 13
def Cn,
static turnOFfK (ánt i 01 (k ai)
turnDffxjint int
ik <- 0y:
iflik <- 0))
return <CIk 1111
return ni
return nz -(<c< 23337
(1<<< 11132
deř (sh
static boolean (St e) for (int iengiongth 1++1
static boolean swwubberittcien 812
in (1en slil isdigit True:
for lint i<s. lenoth 1+-) (character.
ifis.cherät return false:
(1)3
return False
true
return True
return true;
Figure 10: Python to Java translation examples. Similarly to Python to C++, TransCoder often fails to get the right semantics, especially for conditions where can hallucinate extra clauses or write incorrect comparisons. TransCoder-ST often solves these issues.
Input Python function
TransCoder Java translation
TransCoder-ST Java translation
int cassln' (int inturnin
daf cassiriing [n]
def cassint (n] return
zeturn
1)
liner n) if(n -- 1) retuen 27
power (B):
power [b] if n 1:
if n
power
return power
return
power in
Figure 11: C++ Python translation examples. For C++ to Python translation, many of the errors of TransCoder come from incorrectly translated conditions, wrong operators and badly translated functions. TransCoder-ST better comprehends the semantics of the code and generally able to solve these issues.
17

---

Published as a conference paper at ICLR 2022
Input C++ function
TransCoder Java translation
TransCoder-ST Java translation
minxto (int ac=(],
wczll, Ant
(int:
TNT
nin Integer.MIN VALOEJ
Integer.NoX VA84TR)
(int 0:
119)
Zor (int 0: C: 1-+)
for (int 0:
1+*)
(int 1 1 1-+) nin_xor min (KiTLAKE,
foriint nin_aor Math.minn (min_Kor
1-+)
fox (int 1 minkor Math.mini (nandos,
1 1++)
are (S
return nin xory
return nin xOTy
return mintor
public static int Stnddoonggtttttneneaaeee int arelly 1nt n
public static int sindoongeett int arell, int n
int Lind-ongeettConageeee int aret), int n
1:
) #cchset=Integee> 8
)( Bashset-Intoger>> 8 new (astSater
cottdero 37
new Blantdet Cinteger)
(intint i 0z 1 oz 1113
for (int 0; 1 111) 1 tarditari
(int i 0: 1 K 111) 1 5.adstarrisip
5. (arel1l33
9z < 1013
0z L0o
(inddiir 11305..0dd
1013
int 3 acctils
contociss (arcts)
I falee int axell];
while(S.findff) 1- B.ead()) 1-+)
Lnt aze(l))
(allo (3)1
whilo (31) )
and nax Jans, acrlills
and Mith (ano, (x)
ana
Kath (ano, 1 ax2()))
return cod1
return and]
return ansi
Figure 12: C++ to Java translation examples. In the first example computed the mini- mum XOR between two elements of an array, TransCoder erroneously translates INT _MMX into Integer .IN_ VVLUEE This value is used similar contexts (i.e. compute a maximum in- stead of a minimum) but is inappropriate here. TransCoder-ST manages to correct this and outputs function with the right semantics. In the second example, where the function computes the size of the largest subset of elements of the list that could form sequence of consecutive integers, TransCoder- ST manages to translate the semantics of S. find (arr [i] -1) --S . end () appropriately while TransCoder translates it into its negation.
Input Java function
TransCoder C++ translation
TransCoder-ST C++ translation
static boolean LsCoxnerB rasent
bool
isCornerPresonz
bool
LscarsesPrascatt
String MEE, String CoENEE
string st string comnez
atring att, atring cornes
int wtw..eenghhhh 1
int atr h:
size str. leng:he xize el comeer, cormer.langghh Ol i6 ol) return false)
int el cengee length 12:
int el corner. length
ifin a eli return talso/
in ell zetuzn falss)
return lstx. substaiing (G, c11 Legue ln (oorne))
(ocurn we coxnez[0]
return wtw..eeeeteeeeeee.e.eeeee....e.. all mcl-nrr
65 stein ctl -n conner 'n11;
66 etr.enbetr (n-al, n3-mporre:)
stx dab.tubatiing (a. ci, E)
loquels (coznce));
PriorityQueue <Iateqor> PriorityQueue <Iateger>
pelority_qucus int
prioeity
vector-int>,
greater-int=
Figure 13: Java to C++ translation examples. In the first example, which returns whether a given string corner is present at the beginning and the end of string str, TransCoder completely fails to translate the last logical expression correctly while TransCoder-ST manages translate the logic to get the right substrings and return the right output. The second exam- ple is a line defining a priority queue extracted from the kthLargestSum function in the test set of TransCoder. The PriorityQueue object Java returns the smallest elements first by default, while
in C++ returns the largest. TransCoder, which was not trained on any semantic signal, manages to instantiate priority queue object but instantiates a max queue instead of a min queue. TransCoder-ST, which was trained with some supervised signal directly linked to the semantics of the code, manages to instantiate the right type of priority queue.
18

---

Published as a conference paper at ICLR 2022
Input Java function
TransCoder Python Translation
TransCoder-ST translation
static int divisorSum lint al
def divisor_suminn
def divisor_scm
int
0
for (int 1 17 1 ny ++1)
for in renge n 11: 1- return sum
for in range (1 A1: 1- return sum
i-tn / 1) return sumj
static
def check (degres, al: deg_aum
def check (degzee, R):
boolean chack (int degreell, int n)
int deg_sum 07
for in renge in)
for i in range deg_sum 1- degree return /2+ (t-11 medg__smmg
for lint i
44)
i- degree( return(2+>> (a-1) sum
dum + degree
return (2+ (n-1) -=deg_amm)
static int decimalToBinary (int Ni
def dbcimal_to_binarylll D_Number -
def decimal to bo_narryy B_number ent while 1- C:
int N_Number int ont 0; while !-
0 while N 0:
rem V
ren N
rem $
double Moth pow cnt);
pow130, ent)
poc entl
Number rem 02
B Number rem
number t- rem ///- 2 ent return D_mumerr
N /- 2;
M /-
return B_Number
return Number;
Figure 14: Translation examples for Java to Python. When translating to Python, TransCoder was often failing to distinguish between float division (/) and integer division (//). It also often confuses other operations, for instance x and **. Training on self-created labels often solves this issue
Input Python function
TransCoder C++ translation
TransCoder-ST C++ translation
tranapose IA, B) for L. in range (3) for in yange R(1T(3)AA()11
transpose
template ttyaantaettyppaeee T>
const double & double- B[M]
void transpose (2+ A, Is B)
for (int 14M 1+t)
(int j=0; 5-N; fer)
for (int 1-0; 188; 144)
B[i] A[])(A);
for lint j-0; ((A)())-())()
504)
def exponentisl a1 to, 83
void exponent fal (int n, double xif double sum-1,
template <typenane T>
sum-1.0
void expenential fint T
for in range (n, D. -11
fint I-n; 130; f--)
sum-1, 1,0;
sun- -un-L+x-uum/
sun- 1xxxumm111 7
for (int i-m; 180; i--)
sum)
*e"x
endi
1-xxsun/2]
cout
end1;
Figure 15: Our parallel unit tests lead to the generation of more general solutions using tem- plates. Solutions using templates pass the unit tests for several parameter types, while guessing the wrong parameter type can lead to some errors. Solutions using templates succeed more often, are more likely appear in the parallel data we generate and, result, in our model's generations. It leads to our model generating more templates (three times more often for our online model trained the longest).
19

---

Published as a conference paper at ICLR 2022
C EXTRA RESULTS
C.I BEAM REORDERING
We also evaluate simpler method where create unit tests for the Java functions in the dataset and use them to reorder the elements of the beam at test time. We compute the results of the tests for every proposed C++ or Python translation and prioritize the elements that pass the unit tests.
As shown on Table 6 reordering the elements of the beam at test time when translating from Java leads only to small improvements compared to the best baseline (up to 1.7% CA @ for Java + Python) and the scores of this method are far from those obtained when requiring any of the 10 clement of the beam to correct (i.c. CA@ 10). It can be explained by the fact that the tests generated by EvoSuite on these functions can have low mutation scores and be insufficient to thor- oughly test the semantics the functions. Moreover, the tests we create are sometimes incompatible with those of our test set (see Figure 4 for example).
Table 6: Extra results table. We show the CA@1 metric computed with beam size 10 for baselines, our offline and online methods, the beam reordering, and model trained from scratch with our dataset. Beam reordering leads only to small improvements compared to our oftline and online self oeff-trainin methods. Training on our generated parallel dataset from scratch leads to decent performances, but that are still below those of TransCoder and TransCoder-STT.
C++ + Ja
C++ - Py
Ja + C++
Ja - Py
Py - C++
Py + Ja
AVG
TransCoder
65.1%
47.1%
79.8%
49.0%
32.6%
36.6%
51.7%
DOBF
52.7%
45.7%
Beam reordering
80.3%
54.4%
Offline ST scratch
43.0%
41.3%
54.3%
43.2%
31.1%
39.7%
42.1 1%
Ofline
65.5%
56.2%
81.6%
61.8%
46.8%
55.1% 1%
61.1%
Offline ST 2
65.5%
58.3%
83.7%
63.3%
46.4%
52.2%
61.6%
Ofline ST
66.5%
56.2%
85.2%
66.3%
48.1%
56.6%
63.1% 1%.1%
Offline ST
65.3%
48.2%
81.1%
58.1%
48.9%
54.7%
59.4%
Online ST
68.0 %
61.3%
84.6%
68.9% %
56.7% %
58.2%
66.3 %
20


Reviews:
[{"review_id": "rirBcR59K9-", "reviewer": "Reviewer_a7Vt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like this paper. The idea of using unit tests for transpilation is, although not formally guaranteed, a reasonable approach for transpilation in my opinion.\n\nSome of the language in the paper is a little off-putting as it seems to be disconnected (in my opinion) from deeply understanding the space of natural language translation (unstructured ambiguous languages) vs. programming language translation (structured unambiguous languages). It's my hope that if this paper is accepted, that the authors will polish the writing slightly.\n\nFor example: \"translation systems are not as effective for source code as for natural languages.\" This is a weird claim. The work that Alvin Cheung (Berkeley) and JRK (MIT) for Maaz Ahmad (UW Washington / Adobe Research) is wildly effective and has automatically transpiled ~300 functions with a geometric mean performance improvement of 3.36x. Moreover, this approach is *formally verified* unlike the authors approach. In addition, it's already being used in production quality code (Adobe Photoshop v.21).\n\nThere are many other such examples that seem to demonstrate the authors perhaps need to do a bit deeper literature review before making such bizarre claims. For example, the word \"transpilation\" never even appears in the paper, yet the whole paper is about transpilation. Do the authors not know about the field of \"transpilation\" and that they are working in this space?\n\nhttps://en.wikipedia.org/wiki/Source-to-source_compiler\n\nThese minor issues aside, I actually like the approach and am favorable toward the paper.\n", "summary_of_the_paper": "This paper is aimed at using the automatic generation of unit tests as a means to perform transpilation (i.e., the translation of code from one language to another). The authors demonstrate their technique from Java to Python and from Python to C++. They claim that their approach outperforms prior methods by +16% and +24% over prior state-of-the-art.\n", "summary_of_the_review": "TL;LD: it's an important emerging area. It's a reasonable (although not formal) way to perform transpilation. Using self-supervision is likely one of the most important aspects of the system because 99% of the code that we have today is currently unlabeled. To my knowledge, only a handful of labeled datasets exist (e.g., POJ-104, Google Code Jam, IBM/MIT's Project CodeNet, Microsoft's CodeXGLUE, etc.)\n\nWeak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iHbP72io3GB", "reviewer": "Reviewer_LnDd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to understand. It provides a very nice overview of related work and the history of software testing. The core methods is very practical and easy to reproduce.\n\nUsing unit tests to improve quality of parallel datasets generated via backtranslation is not entirely new in the ML for software engineering domain. There were attempts to use it for bug patching applications, see for instance, DeepDebug (https://arxiv.org/abs/2105.09352) paper. It is also frequently used during inference, as a postprocessing step, and evaluation. \n\nNevertheless, a technique that would work in a multilingual code-to-code translation setting is non-trivial and novel. Generating a high-quality parallel corpus starting with monolingual data is non-trivial, it relies on a pre-trained translation model for the target task, and translating unit tests generated by EvoSuite from Java to target languages using heuristics and mapping of types. Using mutation scores to select test suites is also novel.\n\nAuthors claim that the unit tests suites can be used to test program semantics in any programming languages, provided there is a way to establish the mapping between types and parameters. While this is true in most cases, have you observed and analyzed some corner cases, for instance, when an object needs to be instantiated in the body of the unit test method prior to assertions. Instantiating such variables may rely on imported classes or require translating extensive surrounding code context other than the unit test itself. How often does it happen, are these cases ignored?\n\nStarting with 333k Java functions, authors obtain 103k high-quality test case suites with EvoSuites which are then iteratively used to generate parallel training corpus. Have you analyzed the failed cases: \n•\tDoes the model successfully generates data for small methods only (e.g. a single statement), and majority of failed cases are long methods? How does it handle test cases with multiple asserts?\n•\tWhen does EvoSuite fail to generate high-quality test suites? I wonder if EvoSuite scaffolding is an issue\n\nMaybe I am missing something, but why would generated test set mismatch to TransCoder tests set be an issue? Are you generating tests sets for evaluation as well? Does this change the baseline (published) TransCoder computational accuracy numbers?\n\nMinor:\n\nFor multiple published papers the authors cite corresponding arXiv pre-prints. \nFor instance:\n1. Miltiadis Allamanis, Marc Brockschmidt, and M. Khademi. Learning to represent programs with graphs. ArXiv, abs/1711.00740, 2018 -> Published at ICLR 2018\n2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. -> published at ICLR 2015\n… etc\n\n\n\"maximize a criteria such as code coverage or mutation score\" -> \"maximize criteria such as code coverage or mutation score\"\n\n“Otherwise, the mutant is said to be survived” -> “Otherwise, the mutant is said to have survived”\n\nA1 caption (right), should perhaps be changed to “A generated unit test suite”", "summary_of_the_paper": "The paper introduces a pipeline to generate parallel data corpus for code-to-code translation instrumented with unit tests, starting with monolingual data in Java programming language. This allows to improve data quality over noisy backtranslation approach and subsequently improve the model accuracy finetuned on these data. The approach is evaluated on the code-to-code translation against TransCoder baseline.\n\nThe authors also introduce two methods (online and offline) to train on these data, substantially improving upon the state of the art in unsupervised code translation, and reducing the error rate.", "summary_of_the_review": "A well written paper that introduces a practical method to generate a supervised parallel corpus for code-to-code translation. The findings are  generally well supported by evaluation, but some additional tests could strengthen the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fPwuMozkb01", "reviewer": "Reviewer_19G3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, easy to follow and presents an interesting and novel idea. Technically, the work touches many parts - “static” text-to-text transformer, executing code, checking tests passing, observing and optimising test coverage. The paper has quite some effort into it in getting precise results using different decoding strategies and has ablation studies on these factors. What would have been more interesting though is how well would it work with differences in the test generation/acceptance strategies.\n\nThe experiment about dropping mutation scores is the most interesting one from this perspective. If I understand correctly, it seems to have no huge effect on the results (other things like the cache are also ablated?). Probably a related question is if computing mutation scores is needed or one can replace it with much cheaper code coverage metrics.\n\nOne of the biggest limitations of the approach seems to be around how unit tests are translated between the languages. The paper downplays this step in its expositions by simply saying that simple scripts can translate unit tests. While I agree that unit tests tend to be inputs and outputs for which translation should be easier, this is probably limiting the scope of what can possibly be translated. The authors could include an appendix or discussion on how many of the tests could be translated successfully and what incorrect translations look like - e.g. tests that perform a sequence of APIs or expect exceptions. A related question is if tests could be translated with some of the baseline models such as Transcoder/DOBF and if such an idea was considered and was/wasn’t successful.\n\nIt is not completely clear if the data filtering techniques (to only include functions for which tests can be generated) also play a role in the selection of the evaluation data. I would be happy to get a confirmation of this from the authors.\n", "summary_of_the_paper": "The paper proposes a technique to generate parallel training data for machine translation between programming languages. This data is generated based on specifics of the programming languages and the availability of programming tools in these languages.\n\nUsing this additional data, the paper shows that machine translation using state-of-the-art transformers between programming languages significantly improves over prior data collection methods with back-translation.\n", "summary_of_the_review": "The idea of the paper is specific to translation between programming languages and may be of interest to a subset of the ICLR community. There are interesting ideas and overall good evaluation of the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P13DQSgs-Es", "reviewer": "Reviewer_zd7L", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- A simple method is proposed to fine-tune existing code translation models without the need to manually curate parallel data\n- The method does not require any changes to the training procedure of the model, and can be independently applied in the fine-tuning stage\n- The method can potentially be applied to fine-tune a model for characteristics other than correctness. for ex, runtime efficiency, code style, etc.\n- Results show significant improvement over the baseline TransCoder and DOBF models\n\n\n**Weaknesses:**\n- If ground truth parallel data were available for this task, then fine-tuning a model on this data would naturally result in gains in performance. Since this parallel data is extremely scarce, the authors propose to utilize an automated test generation suite to build this parallel data from the model translations itself. I have 2 concerns with this proposal:\n   1. It requires an automated unit test generation tool to be available for at least one of the languages translated. Unit tests generation is an active area of research and tools are available only for limited sets of languages and primarily for modern languages such as Java, Python, etc. An important premise of automated code translation -- which the authors mention in their introduction and ethical concerns section -- is about modernizing legacy languages such as COBOL, where automated test generation systems might not be available at all. The proposed method therefore, in my opinion, is bottlenecked on the availability of automated unit test generation tools, and is not applicable when they are not available.\n   2. Automated unit test generation tools can themselves generate incorrect test cases -- test cases testing the incorrect method, incorrectly testing the method resulting in false positives or false negatives. AthenaTest [1], for instance, works better than EvoSuite and generates correct unit tests for the intended method only 16.21% of the time as per [1]. 26.71% of the times the test case generated by AthenaTest test for the wrong behavior. Other factors such as dynamic typing (as in Python) [2] or the use of more complicated data structures (such as Generics in Java [3]) further affects the effectiveness of these test generation systems. It would have been nice to get some insights into the quality of test cases generated and how this affects the performance of the translation model. This effect, I believe, would be especially pronounced in cases where the code translated is a piece of more complicated codebase and not standalone functions that can be compiled in isolation.\n\n- The authors utilize EvoSuite more as a test data generation tool than a test case generation tool. EvoSuite generates test cases for Java, while this work trains models for Java, Python and C++. Therefore, if I understand correctly, the way test cases are created for Python and C++ is by using the expected input/output pairs and assert statements generated for corresponding Java code, and putting these in a template for the other languages. I'm not completely convinced that this is an ideal strategy that can generalize to a broader set of languages or for more complicated pieces of code. For example: if the source code implements a Java class and the generated test operates on this class and their methods, how would this be translated to Python test without changing the procedure for Python test case generation? Or, how would this procedure work when there isn't a direct mapping of data types and constructs between the languages. This further reduces the applicability of this approach.\n\n- If I understand correctly, the evaluation used in this work includes a reference implementation and a translated implementation, which are tested to generate the same output given the same input? This creates problems as authors note in Figure 4, where the gold translation is equivalent to the source code only on a small domain, while the model generated translation, though correct and better is incorrectly marked wrong. This, to me, indicates an issue with the utilized test set where gold implementations might not be correct translations, thereby affecting the scores. Contemporary works in code generation and translation [4,5] have utilized expected input / output pairs to test the performance of the code generation system, and I wonder why the authors here chose to compare the performance against a reference implementation and not against input-output pairs directly. Additionally, we don't know how prevalent the problem of incorrect reference translations is in the utilized test set.\n\n\n**Other concerns/Suggestions:**\n- The authors mention that they collected data from GitHub with permissive licenses. Can the authors please share the list of licenses used for this data collection step? Having this information in the appendix will provide a reference point for future work in this space on which licenses might be acceptable to use when working with GitHub data.\n\n\n[1] Tufano, Michele, et al. \"Unit Test Case Generation with Transformers.\" arXiv preprint arXiv:2009.05617 (2020).\n\n[2] Lukasczyk, Stephan, Florian Kroiß, and Gordon Fraser. \"Automated Unit Test Generation for Python.\" International Symposium on Search Based Software Engineering. Springer, Cham, 2020.\n\n[3] Vogl, Sebastian, et al. \"EVOSUITE at the SBST 2021 Tool Competition.\" 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST). IEEE, 2021.\n\n[4] Hendrycks, Dan, et al. \"Measuring Coding Challenge Competence With APPS.\" arXiv preprint arXiv:2105.09938 (2021).\n\n[5] Chen, Mark, et al. \"Evaluating large language models trained on code.\" arXiv preprint arXiv:2107.03374 (2021).", "summary_of_the_paper": "- This work presents a way to fine-tune code translation models by providing weak indirect supervision from automated test case generation tools\n- The proposed approach has 3 main components\n   - Pre-trained models for code translation: the authors use Transcoder and DOBF for this.\n   - Automated test case generation tool: the authors use EvoSuite, an evolutionary computing based automated test case generation tool for Java, and\n   - Dataset: the authors collect approximately 330k Java code samples from Github with permissive licenses\n- The proposed approach then proceeds as follows:\n   - It starts by creating unit tests for Java code samples using EvoSuite\n   - It uses the generated expected input / output pairs to create unit tests for other languages -- Python and C++\n   - The method then translates code samples from one language to another and then validates the generated translation using the unit tests created in the step above\n   - In this manner, the approach collects parallel translation data samples that are later used for fine-tuning the model\n- The authors propose 2 variants of the method: offline and online\n   - In the offline variant, the approach utilizes all the parallel code samples collected to fine-tune the model, while\n   - In the online variant, the approach maintains a cache which is updated with new examples generated from the model\n- Models fine-tuned on parallel data created in this manner show substantial increase in performance over TransCoder and DOBF models.", "summary_of_the_review": "The proposed method achieve significant improvements over the baseline TransCoder and the DOBF models. However, I found the following weaknesses, resulting in my recommendation.\n\n1. Bottleneck on the availability of test generation tools thus limiting the number of languages it can be applied to.\n2. Quality of generated tests dips on more complicated code segments, and we do not know what effect that has on the performance. The method is trained on Java functions that can be executed in isolation.\n3. Method to generate test cases for translated Python and C++ codes seems to be dependent on expected input/output pairs, and it's not clear how the proposed method would work for more complicated code segments, or for languages with no direct mapping of data types and constructs.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is about unsupervised translation between programming languages. The main positive is that it introduces the idea of using a form of unit test generation and execution behavior within a programming language back-translation setup, and it puts together together a number of pieces in an interesting way: text-to-text transformers, unit test generation, execution and code coverage. Results show a substantial improvement. The main weaknesses are that there are some caveats that need to be made, such as the (heuristic, not learned) way that test cases are translated across languages is not fully general, and that limits the applicability. There are also some cases where I find that the authors are stretching claims a bit beyond what experiments support, e.g., in the response to zd7L about applicability to COBOL.

All-in-all, though, it's a good implementation of an idea that should have a lasting place in this line of work, so it's worth accepting.
Decision: Accept (Spotlight)