Leveraging Automated Unit Tests for Unsupervised Code Translation

1 INTRODUCTION . Ancient languages such as COBOL still underpin much of the financial industry and government services . Their outdated structures and thinning developer bases induce costs and severely slow down development , prompting businesses to modernize their codebases . For instance , the Commonwealth Bank of Australia spent around $ 750 million over 5 years to migrate its COBOL codebase to a more recent language . More generally , most large companies own code written in several programming languages , which can hinder interoperability and make programmers less efficient . Automatic translation systems could make codebase migrations faster and cheaper , and help programmers learn new languages or understand existing code . Systems to automatically translate between programming languages with approximately the same level of abstraction are called transpilers or sourceto-source compilers . They need to be distinguished from compilers which translate source code to a lower-level language . The particularities of some languages allow the creation of very successful rule-based transpilers for a few language pairs ( e.g . Java→Scala , CoffeeScript→JavaScript ) . Methods leveraging verified lifting ( Kamil et al. , 2016 ) , which offer formal guarantees , can significantly speedup some pre-defined code fragments ( Ahmad & Cheung , 2016 ; Ahmad et al. , 2019 ) . However , source-to-source translation for arbitrary programming languages is still an open problem . Rule-based systems are commonly used , but they are never exhaustive due to the considerable number of translation rules that should be written to translate every function and object from every standard library . Unlike in natural languages , there is little to no parallel data available for source code , making it impossible to train standard machine translation models . Recently , TransCoder ( Roziere et al. , 2020 ) showed that unsupervised methods can be used to translate source code . However , it is †Work done while at Facebook trained without any supervised signal and only learns the semantics of tokens from their contexts . As shown in Figure 1 , it can confuse tokens that have different semantics in different languages , for instance the float division in Python and integer division in C++ and Java which use the token / or more subtle operator priority differences ( e.g . Java prioritizes == over & unlike Python ) . While small inaccuracies often merely hinder comprehension in natural languages , they often make the entire translation erroneous in the context of programming languages . TransCoder leverages back-translation ( Sennrich et al. , 2015 ) , an effective data-augmentation scheme where the model translates source sequences to generate training data for the target-tosource direction , and vice versa . Although being highly effective in low-resource translation , backtranslation also has issues , as the model is trained on potentially invalid input-output pairs . Neural machine translation models being highly sensitive to input noise ( Belinkov & Bisk , 2018 ; Khayrallah & Koehn , 2018 ) , this can severely deteriorate the performance . Fortunately , many programming languages come with relatively mature tools and technologies for automated test data generation . In this paper , we propose to leverage these tools to guide the translation process , weeding out unsuccessful translations , thereby increasing the overall confidence in the machine translation process . The topic of automated test data generation has been active for over three decades in the software engineering research community ( Myers , 1979 ; Miller & Spooner , 1976 ) . There are now many existing mature tools for test data generation , both open source research tools ( Fraser & Arcuri , 2011 ; Lakhotia et al. , 2013 ; Cadar et al. , 2008 ) , and production testing systems ( Alshahwan et al. , 2018 ; Tillmann et al. , 2014 ) . Because of its pivotal impact on practical software engineering , automated testing remains a highly active research area ( Anand et al. , 2013 ) , with the result that future automated testing advances will lead to ongoing improvement in automated translation . We use one such open source automated test generation tool , EvoSuite ( Fraser & Arcuri , 2011 ) , in this paper . EvoSuite is a well-established test generation tool for Java which uses coverage metrics ( Chekam et al. , 2017 ) and mutation scores ( Jia & Harman , 2011 ) to generate high-quality tests . It has been widely used in the Software Testing research literature for test data generation although it has not , hitherto , been used as part of an automated code translation approach , the topic of the present paper . More generally , software testing tools have been largely ignored by the machine learning community ( Zhang et al. , 2020 ) . In this paper , we propose to use automatically created unit tests to guide unsupervised translation models for programming languages . More precisely , we create unit tests automatically for a large number of functions from the source dataset . Since the unit tests are composed of simple inputs and asserts , they can easily be translated to semantically equivalent tests in the target languages using simple scripts . Using our unit-tests and a pre-trained unsupervised translation model , we create parallel datasets by translating functions and selecting the translations that have the same semantics as the original function for the tested inputs . Overall , we make the following contributions : • We introduce a novel approach , TransCoder-ST ( for Self-Trained ) , that leverages an automated unit test generation pipeline to filter out invalid translations and reduce the noise coming from the back-translation process in unsupervised machine translation . • We present two implementations of this approach ( online and offline ) , and show that it significantly outperforms the previous state of the art in code translation on all the language pairs we considered . In particular , we improve the state of the art for translating between Java , Python and C++ by an average of 12.6 % Computational Accuracy ( CA @ 1 ) , corresponding to an average relative improvement of 25.5 % . For Python→ C++ , we improve the CA @ 1 by 24 % , reducing the error rate by 35.7 % compared to previous models . • We generate multilingual unit tests for hundreds of thousands of Java functions and create a large parallel dataset of 135,000 parallel functions between Java , Python , and C++ . • Our method is completely unsupervised and could easily be generalized to other programming languages and unit test creation tools . 2 RELATED WORK . Unit Test Generation . Software testing is challenging due to the large number of possibilities to be tested , and the inherent cost of covering reasonable representative sample ( Myers , 1979 ) . When test design is performed by humans , the cost can be prohibitive . To reduce such cost , much research over the last three decades has focused on automating the process of test generation ( Anand et al. , 2013 ) . Although automated test generation has been studied since the mid-1970s ( Miller & Spooner , 1976 ) , it was only in the last decade that industrial-strength tools have become widely available . There are now several test data generation tools for languages , including C ( Cadar et al. , 2008 ; Lakhotia et al. , 2013 ) and Java ( Fraser & Arcuri , 2011 ) . Popular test data generation techniques include symbolic execution of the code ( Cadar & Sen , 2013 ) , dynamic execution guided by a fitness function ( Harman et al. , 2015 ) , and hybrids of these two techniques ( Baars et al. , 2011 ) . Recently , neural networks have also been used successfully to generate unit tests ( Tufano et al. , 2020 ) . One of the most well-established and widely-used open source tools for test data generation is the EvoSuite system ( Fraser & Arcuri , 2011 ) . EvoSuite uses search based software engineering ( SBSE ) ( Harman et al. , 2012 ) to generate test cases . Like all SBSE techniques , EvoSuite is guided by fitness functions , in this case aimed at capturing the test suite ’ s coverage and mutation score of the code being tested . We use EvoSuite in our work for three reasons : it is publicly available in open source ( thereby facilitating replication ) , it is under current active development ( thereby supporting future work ) , and it is widely used by other researchers ( thereby enabling interoperability ) . The test framework can be considered as a parameter in our overall approach and could be substituted with another . In order to assess the effectiveness of the test suites generated , we use mutation testing , a topic also widely-studied since the 1970s ( DeMillo et al. , 1978 ) . A mutant is a version of the program into which a fault is deliberately inserted , thereby assessing the test suite ’ s fault detection ability ( Jia & Harman , 2011 ; Papadakis et al. , 2019 ) . For a given set of mutants and a test suite , the mutation score is defined to be the proportion of mutants for which the test suite distinguishes the behavior of the mutant from that of the original program . The mutation score is thus a proxy for the faultrevealing power of the test suite on a set of simulated faults ( the mutants ) . Mutation scores have been empirically demonstrated to be correlated to real fault revelation ( Chekam et al. , 2017 ) , motivating our adoption of this approach . Machine Learning for Programming Languages . In recent years , deep learning methods have been used to tackle various tasks in software engineering , with a particular interest in bug detection and repair ( Wang et al. , 2018 ; Chen et al. , 2019 ; Allamanis et al. , 2018 ; Tarlow et al. , 2020 ; Murali et al. , 2021 ; Dinella et al. , 2020 ; Yasunaga & Liang , 2020 ; Tufano et al. , 2019 ; Drain et al. , 2021 ) and code completion ( Li et al. , 2018 ; Liu et al. , 2020 ; Kim et al. , 2021 ; Svyatkovskiy et al. , 2021 ) . Unsupervised pre-training methods for code based on BERT ( Kanade et al. , 2020 ; Feng et al. , 2020 ) , BART ( Ahmad et al. , 2021 ) or other objectives tailored to source code ( Guo et al. , 2020 ; Roziere et al. , 2021 ) have shown strong results on benchmarks such as CodeXGLUE ( Lu et al. , 2021 ) . Recently , Hendrycks et al . ( 2021 ) evaluated the competence of several language models for solving coding challenges . Chen et al . ( 2021 ) trained a a large model to generate programs from docstrings and are able to solve 28.8 % of the problems in their HumanEval dataset . Austin et al . ( 2021 ) also evaluated the capabilities of large language models for generating code solving problem statements written in natural language . The goal of code translation is also to generate code solving a specific problem , but the input ( code written in a different language ) is more precise and often more concise . Translation of Programming Languages Several studies used statistical methods to translate between programming languages . Early methods extracted parallel datasets and trained phrasebased models to translate between C # and Java ( Nguyen et al. , 2013 ; Karaivanov et al. , 2014 ) or from Python 2 to Python 3 ( Aggarwal et al. , 2015 ) . Later , Chen et al . ( 2018 ) proposed a tree-to-tree neural network to translate between CoffeeScript and JavaScript and between C # and Java using the dataset created by Nguyen et al . ( 2013 ) . However , these approaches are limited to a few language pairs for which small parallel datasets were created manually ( e.g . C # -Java ) or can be created with rule-based tools ( e.g . Python 2-Python 3 and CoffeeScript-JavaScript ) . Instead , Roziere et al . ( 2020 ) proposed TransCoder , an unsupervised model that leverages the principles of unsupervised machine translation ( Lample et al. , 2018 ) , to translate between Python , Java and C++ . They showed that their method outperforms well-established rule-based baselines , does not require any parallel data or expert knowledge , and can easily be generalized to other languages . They pre-trained their model with the Masked Language Modeling ( MLM ) objective of Devlin et al . ( 2018 ) , and trained it with the denoising auto-encoding ( DAE ) ( Vincent et al. , 2008 ) and the back-translation ( BT ) ( Sennrich et al. , 2015 ) objectives . Later , Roziere et al . ( 2021 ) showed that augmenting MLM with a deobfuscation objective ( dubbed DOBF ) can substantially improve the performance of TransCoder . In the rest of the paper , we will refer to their model as DOBF . Even though unsupervised methods can be trained on large amounts of data , they sometimes lack the signal needed to differentiate between semantically different tokens that often occur in similar contexts ( see Figure 1 ) . There is a need for a method providing supervised signal directly related to the semantics of the code without manually crafted parallel datasets .

Summary: The paper proposes a technique to generate parallel training data for machine translation between programming languages. This data is generated based on specifics of the programming languages and the availability of programming tools in these languages.  Using this additional data, the paper shows that machine translation using state-of-the-art transformers between programming languages significantly improves over prior data collection methods with back-translation.