Leveraging Automated Unit Tests for Unsupervised Code Translation

1 INTRODUCTION . Ancient languages such as COBOL still underpin much of the financial industry and government services . Their outdated structures and thinning developer bases induce costs and severely slow down development , prompting businesses to modernize their codebases . For instance , the Commonwealth Bank of Australia spent around $ 750 million over 5 years to migrate its COBOL codebase to a more recent language . More generally , most large companies own code written in several programming languages , which can hinder interoperability and make programmers less efficient . Automatic translation systems could make codebase migrations faster and cheaper , and help programmers learn new languages or understand existing code . Systems to automatically translate between programming languages with approximately the same level of abstraction are called transpilers or sourceto-source compilers . They need to be distinguished from compilers which translate source code to a lower-level language . The particularities of some languages allow the creation of very successful rule-based transpilers for a few language pairs ( e.g . Java→Scala , CoffeeScript→JavaScript ) . Methods leveraging verified lifting ( Kamil et al. , 2016 ) , which offer formal guarantees , can significantly speedup some pre-defined code fragments ( Ahmad & Cheung , 2016 ; Ahmad et al. , 2019 ) . However , source-to-source translation for arbitrary programming languages is still an open problem . Rule-based systems are commonly used , but they are never exhaustive due to the considerable number of translation rules that should be written to translate every function and object from every standard library . Unlike in natural languages , there is little to no parallel data available for source code , making it impossible to train standard machine translation models . Recently , TransCoder ( Roziere et al. , 2020 ) showed that unsupervised methods can be used to translate source code . However , it is †Work done while at Facebook trained without any supervised signal and only learns the semantics of tokens from their contexts . As shown in Figure 1 , it can confuse tokens that have different semantics in different languages , for instance the float division in Python and integer division in C++ and Java which use the token / or more subtle operator priority differences ( e.g . Java prioritizes == over & unlike Python ) . While small inaccuracies often merely hinder comprehension in natural languages , they often make the entire translation erroneous in the context of programming languages . TransCoder leverages back-translation ( Sennrich et al. , 2015 ) , an effective data-augmentation scheme where the model translates source sequences to generate training data for the target-tosource direction , and vice versa . Although being highly effective in low-resource translation , backtranslation also has issues , as the model is trained on potentially invalid input-output pairs . Neural machine translation models being highly sensitive to input noise ( Belinkov & Bisk , 2018 ; Khayrallah & Koehn , 2018 ) , this can severely deteriorate the performance . Fortunately , many programming languages come with relatively mature tools and technologies for automated test data generation . In this paper , we propose to leverage these tools to guide the translation process , weeding out unsuccessful translations , thereby increasing the overall confidence in the machine translation process . The topic of automated test data generation has been active for over three decades in the software engineering research community ( Myers , 1979 ; Miller & Spooner , 1976 ) . There are now many existing mature tools for test data generation , both open source research tools ( Fraser & Arcuri , 2011 ; Lakhotia et al. , 2013 ; Cadar et al. , 2008 ) , and production testing systems ( Alshahwan et al. , 2018 ; Tillmann et al. , 2014 ) . Because of its pivotal impact on practical software engineering , automated testing remains a highly active research area ( Anand et al. , 2013 ) , with the result that future automated testing advances will lead to ongoing improvement in automated translation . We use one such open source automated test generation tool , EvoSuite ( Fraser & Arcuri , 2011 ) , in this paper . EvoSuite is a well-established test generation tool for Java which uses coverage metrics ( Chekam et al. , 2017 ) and mutation scores ( Jia & Harman , 2011 ) to generate high-quality tests . It has been widely used in the Software Testing research literature for test data generation although it has not , hitherto , been used as part of an automated code translation approach , the topic of the present paper . More generally , software testing tools have been largely ignored by the machine learning community ( Zhang et al. , 2020 ) . In this paper , we propose to use automatically created unit tests to guide unsupervised translation models for programming languages . More precisely , we create unit tests automatically for a large number of functions from the source dataset . Since the unit tests are composed of simple inputs and asserts , they can easily be translated to semantically equivalent tests in the target languages using simple scripts . Using our unit-tests and a pre-trained unsupervised translation model , we create parallel datasets by translating functions and selecting the translations that have the same semantics as the original function for the tested inputs . Overall , we make the following contributions : • We introduce a novel approach , TransCoder-ST ( for Self-Trained ) , that leverages an automated unit test generation pipeline to filter out invalid translations and reduce the noise coming from the back-translation process in unsupervised machine translation . • We present two implementations of this approach ( online and offline ) , and show that it significantly outperforms the previous state of the art in code translation on all the language pairs we considered . In particular , we improve the state of the art for translating between Java , Python and C++ by an average of 12.6 % Computational Accuracy ( CA @ 1 ) , corresponding to an average relative improvement of 25.5 % . For Python→ C++ , we improve the CA @ 1 by 24 % , reducing the error rate by 35.7 % compared to previous models . • We generate multilingual unit tests for hundreds of thousands of Java functions and create a large parallel dataset of 135,000 parallel functions between Java , Python , and C++ . • Our method is completely unsupervised and could easily be generalized to other programming languages and unit test creation tools . 2 RELATED WORK . Unit Test Generation . Software testing is challenging due to the large number of possibilities to be tested , and the inherent cost of covering reasonable representative sample ( Myers , 1979 ) . When test design is performed by humans , the cost can be prohibitive . To reduce such cost , much research over the last three decades has focused on automating the process of test generation ( Anand et al. , 2013 ) . Although automated test generation has been studied since the mid-1970s ( Miller & Spooner , 1976 ) , it was only in the last decade that industrial-strength tools have become widely available . There are now several test data generation tools for languages , including C ( Cadar et al. , 2008 ; Lakhotia et al. , 2013 ) and Java ( Fraser & Arcuri , 2011 ) . Popular test data generation techniques include symbolic execution of the code ( Cadar & Sen , 2013 ) , dynamic execution guided by a fitness function ( Harman et al. , 2015 ) , and hybrids of these two techniques ( Baars et al. , 2011 ) . Recently , neural networks have also been used successfully to generate unit tests ( Tufano et al. , 2020 ) . One of the most well-established and widely-used open source tools for test data generation is the EvoSuite system ( Fraser & Arcuri , 2011 ) . EvoSuite uses search based software engineering ( SBSE ) ( Harman et al. , 2012 ) to generate test cases . Like all SBSE techniques , EvoSuite is guided by fitness functions , in this case aimed at capturing the test suite ’ s coverage and mutation score of the code being tested . We use EvoSuite in our work for three reasons : it is publicly available in open source ( thereby facilitating replication ) , it is under current active development ( thereby supporting future work ) , and it is widely used by other researchers ( thereby enabling interoperability ) . The test framework can be considered as a parameter in our overall approach and could be substituted with another . In order to assess the effectiveness of the test suites generated , we use mutation testing , a topic also widely-studied since the 1970s ( DeMillo et al. , 1978 ) . A mutant is a version of the program into which a fault is deliberately inserted , thereby assessing the test suite ’ s fault detection ability ( Jia & Harman , 2011 ; Papadakis et al. , 2019 ) . For a given set of mutants and a test suite , the mutation score is defined to be the proportion of mutants for which the test suite distinguishes the behavior of the mutant from that of the original program . The mutation score is thus a proxy for the faultrevealing power of the test suite on a set of simulated faults ( the mutants ) . Mutation scores have been empirically demonstrated to be correlated to real fault revelation ( Chekam et al. , 2017 ) , motivating our adoption of this approach . Machine Learning for Programming Languages . In recent years , deep learning methods have been used to tackle various tasks in software engineering , with a particular interest in bug detection and repair ( Wang et al. , 2018 ; Chen et al. , 2019 ; Allamanis et al. , 2018 ; Tarlow et al. , 2020 ; Murali et al. , 2021 ; Dinella et al. , 2020 ; Yasunaga & Liang , 2020 ; Tufano et al. , 2019 ; Drain et al. , 2021 ) and code completion ( Li et al. , 2018 ; Liu et al. , 2020 ; Kim et al. , 2021 ; Svyatkovskiy et al. , 2021 ) . Unsupervised pre-training methods for code based on BERT ( Kanade et al. , 2020 ; Feng et al. , 2020 ) , BART ( Ahmad et al. , 2021 ) or other objectives tailored to source code ( Guo et al. , 2020 ; Roziere et al. , 2021 ) have shown strong results on benchmarks such as CodeXGLUE ( Lu et al. , 2021 ) . Recently , Hendrycks et al . ( 2021 ) evaluated the competence of several language models for solving coding challenges . Chen et al . ( 2021 ) trained a a large model to generate programs from docstrings and are able to solve 28.8 % of the problems in their HumanEval dataset . Austin et al . ( 2021 ) also evaluated the capabilities of large language models for generating code solving problem statements written in natural language . The goal of code translation is also to generate code solving a specific problem , but the input ( code written in a different language ) is more precise and often more concise . Translation of Programming Languages Several studies used statistical methods to translate between programming languages . Early methods extracted parallel datasets and trained phrasebased models to translate between C # and Java ( Nguyen et al. , 2013 ; Karaivanov et al. , 2014 ) or from Python 2 to Python 3 ( Aggarwal et al. , 2015 ) . Later , Chen et al . ( 2018 ) proposed a tree-to-tree neural network to translate between CoffeeScript and JavaScript and between C # and Java using the dataset created by Nguyen et al . ( 2013 ) . However , these approaches are limited to a few language pairs for which small parallel datasets were created manually ( e.g . C # -Java ) or can be created with rule-based tools ( e.g . Python 2-Python 3 and CoffeeScript-JavaScript ) . Instead , Roziere et al . ( 2020 ) proposed TransCoder , an unsupervised model that leverages the principles of unsupervised machine translation ( Lample et al. , 2018 ) , to translate between Python , Java and C++ . They showed that their method outperforms well-established rule-based baselines , does not require any parallel data or expert knowledge , and can easily be generalized to other languages . They pre-trained their model with the Masked Language Modeling ( MLM ) objective of Devlin et al . ( 2018 ) , and trained it with the denoising auto-encoding ( DAE ) ( Vincent et al. , 2008 ) and the back-translation ( BT ) ( Sennrich et al. , 2015 ) objectives . Later , Roziere et al . ( 2021 ) showed that augmenting MLM with a deobfuscation objective ( dubbed DOBF ) can substantially improve the performance of TransCoder . In the rest of the paper , we will refer to their model as DOBF . Even though unsupervised methods can be trained on large amounts of data , they sometimes lack the signal needed to differentiate between semantically different tokens that often occur in similar contexts ( see Figure 1 ) . There is a need for a method providing supervised signal directly related to the semantics of the code without manually crafted parallel datasets .

Summary: This paper is aimed at using the automatic generation of unit tests as a means to perform transpilation (i.e., the translation of code from one language to another). The authors demonstrate their technique from Java to Python and from Python to C++. They claim that their approach outperforms prior methods by +16% and +24% over prior state-of-the-art.