Using Document Similarity Methods to create Parallel Datasets for Code Translation

Translating source code from one programming language to another is a critical , time-consuming task in modernizing legacy applications and codebases . Recent work in this space has drawn inspiration from the software naturalness hypothesis by applying natural language processing techniques towards automating the code translation task . However , due to the paucity of parallel data in this domain , supervised techniques have only been applied to a limited set of popular programming languages . To bypass this limitation , unsupervised neural machine translation techniques have been proposed to learn code translation using only monolingual corpora . In this work , we propose to use document similarity methods to create noisy parallel datasets of code , thus enabling supervised techniques to be applied for automated code translation without having to rely on the availability or expensive curation of parallel code datasets . We explore the noise tolerance of models trained on such automatically-created datasets and show that these models perform comparably to models trained on ground truth for reasonable levels of noise . Finally , we exhibit the practical utility of the proposed method by creating parallel datasets for languages beyond the ones explored in prior work , thus expanding the set of programming languages for automated code translation . 1 INTRODUCTION . As the pace of software development increases and the famous adage “ software is eating the world ” ( Andreessen , 2011 ) is borne out , there is a corresponding increase in the amount of source code and number of software artefacts in active use for which support has lapsed . At the same time , the number of software professionals and programmers who can support and understand such code is unable to keep pace with the rate at which it is produced . This problem , while important when it comes to relatively modern programming languages ( such as Java and Python ) , becomes even more pressing when it come to legacy languages ( like COBOL ) that mission-critical applications and systems are written in ( Charette , 2020 ) . In recent years , there have been multiple instances of organizations struggling to maintain their legacy systems and making considerable investments to upgrade them . In 2021 the Commonwealth Bank of Australia upgraded its core banking platform originally written in COBOL : this ultimately took 5 years and more than 1 Billion AUD to complete ( Irrera , 2017 ) . During the COVID-19 pandemic , software systems implemented in COBOL slowed down the release of US unemployment stimulus checks ( Kelly , 2020 ) , leaving governments scrambling to find COBOL experts who were already hard to come by . A recent study by the United States Government Accountability Office ( Walsh , 2021 ) has identified 65 critical federal legacy systems in need of urgent modernization . Some of these systems are over 50 years old , and cost millions of dollars annually to operate and maintain . Parallel to these developments are recent efforts at the intersection of software engineering , machine learning ( ML ) , and natural language processing ( NLP ) , which have posited the naturalness hypothesis of software ( Hindle et al. , 2016 ) . The hypothesis states that “ ... Software is a form of human communication ; software corpora have similar statistical properties to natural language corpora ; and these properties can be exploited to build better software engineering tools ” ( Allamanis et al. , 2018 ) . This hypothesis has been used to extend breakthroughs and advances from various NLP sub-fields to software engineering tasks such as code translation . Prior works in the code translation domain have proposed the application of statistical , supervised , and unsupervised machine translation techniques to learn code translation models to varying degrees of success . A key limitation of a majority of the proposed code translation approaches , however , is the lack of availability of parallel data for training . Unlike natural language , where a piece of text is verbatim translated in multiple languages – legal documents , parliamentary proceedings in multilingual societies – code is rarely implemented as is in multiple languages ; thus making it hard to create parallel datasets . A few limited datasets – such as Java↔ C # ( Nguyen et al. , 2013 ) and AVATAR for Java ↔ Python ( Ahmad et al. , 2021b ) – are currently available . However , these are extremely limited in the number of programming language they cover , and manually curating a dataset for a specific use-case is impractical . To bypass this limitation , unsupervised techniques have been applied to the code translation task . Unsupervised techniques come with their own limitations however ; and often , supervised techniques can outperform them when the source and target corpora are from different domains , the source and target languages use different scripts , and on low-resource language pairs , among other concerns ( Kim et al. , 2020 ; Marchisio et al. , 2020 ) . It is for this reason that in this work , we focus on one of the main blockers impeding the application of supervised techniques to code translation : the availability of parallel corpora and datasets . Specifically , we propose to utilize document similarity methods to create parallel source code datasets that are noisy by design . In this work , we empirically demonstrate the effectiveness of document similarity methods in creating such parallel datasets with high levels of accuracy . Given that datasets created in this manner are bound to be noisy , we study the performance characteristics of models for code translation that have been trained on data with varying degrees of noise ; and show that these models have considerable resistance to noise and perform well even with moderate amounts of noise . Finally , we demonstrate the practical utility of the proposed approach by training models to translate between 10 pairs of languages – a majority of which have not been looked at in prior work . 2 RELATED WORK . Code translation datasets : Typical methods for creating parallel datasets for code translation have either relied on the availability of open-sourced projects with implementations in multiple languages , or on the existence of transpilers . The earliest widely-used large-scale dataset for code translation was for Java↔ C # ( Nguyen et al. , 2013 ) translation , created by indexing open-sourced projects implemented in both languages . Aggarwal et al . ( 2015 ) used the Python 2to3 1 transpiler to create a dataset ; while Chen et al . ( 2018 ) used CoffeeScript ’ s compiler ( which compiles down to JavaScript ) to create a parallel dataset . More recently , Ahmad et al . ( 2021b ) released AVATAR – a parallel corpus of Java to Python manually curated through submissions on competitive programming websites . Publicly available datasets for code translation are however extremely limited , and manually curating these datasets for a specific use-case is expensive and often impractical . Source-to-Source translation : The earliest code translation models were rule-based systems , operating on handcrafted rules . These systems require a lot of effort to build , are not easily extendable to other languages , and are also outperformed by neural techniques . Some of these systems are : Java2CSharp 2 , Java2Python 3 , SmallTalk to C ( Yasumatsu & Doi , 1995 ) , Cobol to Java ( Mossienko , 2003 ) , and Tangible Software Solutions 4 ( VB.NET , C # , Java , C++ , and Python ) . Moving away from rule-based systems , Nguyen et al . ( 2013 ) , Karaivanov et al . ( 2014 ) , and Nguyen et al . ( 2014 ) applied different versions of Phrase-Based Statistical Machine Translation to translate between Java and C # . Chen et al . ( 2018 ) proposed a tree-to-tree neural network to translate the parsed tree of the source code into the target code parse tree . The aforementioned supervised techniques have all been benchmarked on the Java↔ C # dataset , and are limited by the availability of parallel datasets . To bypass this limitation , Roziere et al . ( 2020 ) used unsupervised neural machine translation techniques to translate between languages using only monolingual corpora , and showed impressive results for translation between Java , C++ , and Python . While Roziere et al . ( 2020 ) trained the model specifically for code translation , large language models – such as GPT-2 ( Radford et al. , 2019 ) , GPT-3 ( Brown et al. , 2020 ) , and Codex ( Chen et al. , 2021 ) – have also been shown to have some competence in generating code ( Hendrycks et al. , 2021 ) . Parallel corpus mining : Prior work in natural language research has looked at various ways of creating parallel corpora from a non-parallel corpus . Munteanu & Marcu ( 2005 ) train a maximum 1https : //docs.python.org/3/library/2to3.html 2https : //sourceforge.net/projects/j2cstranslator/ 3https : //github.com/natural/java2python 4https : //www.tangiblesoftwaresolutions.com/ entropy classifier to identify if two given sentences are translations of each other . They extract parallel data from large-scale Chinese , Arabic , and English non-parallel newspaper corpora , and show improvement in model performance when trained with a combination of a small parallel corpus and the extracted dataset . Uszkoreit et al . ( 2010 ) describe a system that uses n-gram features to mine parallel documents from a billion-scale corpus . Smith et al . ( 2010 ) focus on aligning Wikipedia documents by creating features suitable for such documents . Artetxe & Schwenk ( 2019 ) utilize specific scoring functions based on multilingual sentence embeddings to create parallel corpora , and Hangya & Fraser ( 2019 ) rely on continuous parallel segments rather than word similarities to find parallel sentences . Banón et al . ( 2020 ) released the largest publicly available parallel corpora of sentences ( 223 million parallel sentences ) by aligning sentences from data crawled over the web . There is a substantial precedence of parallel corpus mining in the natural language domain ; however , such studies in the code translation domain are non-existent . Machine Translation using noisy data : Prior studies have aimed to study the impact of noise on the performance of machine translation systems . Formiga & Fonollosa ( 2012 ) study the impact of misspelled words on the performance of Statistical Machine Translation and suggest strategies to deal with them , while Goutte et al . ( 2012 ) study the impact of sentence alignment errors on the performance of SMT . Further , Khayrallah & Koehn ( 2018 ) define 5 categories of artificial noise in Neural Machine Translation , and study the impact each of these types has on performance . We motivate our work from these prior efforts in order to study the impact that noise has on the performance of code translation models . 3 PROPOSED METHOD . In this work , we propose to utilize document similarity methods to create noisy parallel datasets for code translation . We refer to the datasets created in this manner as “ noisy ” because unlike manually curated datasets , there is no guarantee of a parallel implementation of the specific source file/code being available in the corpus : this may result in near-similar code samples being paired as ground truth examples instead . Algorithm 1 presents the proposed approach as pseudocode . Algorithm 1 Creating parallel code corpus 1 : CreateParallelCorpora ( D , D ′ , M , δ ) 2 : initialize P = { } 3 : for i = 1 to |D| do 4 : Dsim = GetSimilarDocuments ( di , D ′ , M ) 5 : for ( di , d ′ j ) in Dsim do 6 : if ( · , d′j ) /∈ P and M ( di , d ′ j ) ≤ δ then 7 : P = P ∪ ( di , d ′ j ) 8 : break 9 : Dres = sort ( ( d1 , d2 ) ∈ P , key =M ( d1 , d2 ) ) 10 : Return : Dres The algorithm expects two non-parallel sets of documents D = { d1 , · · · , dn } and D ′ = { d′1 , · · · , d ′ m } as input . Within the context of our work , the documents in these two sets represent code samples from two distinct programming languages . Along with the documents , the algorithm also expects a similarity measure M ( d , d ′ ) as input , to compare two given documents for similarity . A lower score from the similarity measure indicates higher similarity between documents . Finally , the algorithm expects a similarity threshold δ to help keep only sufficiently similar documents in the resulting parallel corpus . Thereafter , the algorithm follows a simple procedure of iterating over all documents ; finding the most similar documents in the target set ; and adding the newly found similar document pairs to the result only if the target document has not been paired before , and if the similarity is below the threshold value . Once all the documents are iterated upon , the algorithm produces a list of unique pairs of code segments ( documents ) ordered by their similarity , ready to be used for downstream tasks .

Summary: This paper mines noisy parallel datasets of code by calculating the similarity between two non-parallel sets of documents. The authors first show that the document similarity methods can indeed align parallel documents and find that the word movers distance (WMD) is the most effective one. Then, the authors show the high tolerance of models trained with noisy datasets. Based on the two findings, the authors finally apply the proposed method to a large, non-parallel code dataset, and observe a performance boost of using a noisy parallel dataset compared to randomly paired datasets.