Using Document Similarity Methods to create Parallel Datasets for Code Translation
Authors: Mayank Agarwal
Kartik Talamadupula
Fernando Martinez
Stephanie Houde
Michael Muller
John Richards
Steven I Ross
Justin D. Weisz
Venue: ICLR 2022 Submitted (2022)
Keywords: code translation
machine translation
document similarity

Abstract: Translating source code from one programming language to another is a critical, time-consuming task in modernizing legacy applications and codebases. Recent work in this space has drawn inspiration from the software naturalness hypothesis by applying natural language processing techniques towards automating the code translation task. However, due to the paucity of parallel data in this domain, supervised techniques have only been applied to a limited set of popular programming languages. To bypass this limitation, unsupervised neural machine translation techniques have been proposed to learn code translation using only monolingual corpora. In this work, we propose to use document similarity methods to create noisy parallel datasets of code, thus enabling supervised techniques to be applied for automated code translation without having to rely on the availability or expensive curation of parallel code datasets. We explore the noise tolerance of models trained on such automatically-created datasets and show that these models perform comparably to models trained on ground truth for reasonable levels of noise. Finally, we exhibit the practical utility of the proposed method by creating parallel datasets for languages beyond the ones explored in prior work, thus expanding the set of programming languages for automated code translation.

Under review as a conference paper at ICLR 2022
USING DOCUMENT SIMILARITY METHODS TO CREATE PARALLEL DATASETS FOR CODE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Translating source code from one programming language to another is critical, limeccosumiigg task in modernizing legacy applications and codebases. Recent work in this space has drawn inspiration from the software naturalness hypothesis by applying natural language processing techniques towards automating the code translation task. However, due to the paucity of parallel data in this domain, super- vised techniques have only been applied to a limited set of popular programming languages. To bypass this limitation, unsupervised neural machine translation techniques have been proposed to learn code translation using only monolingual corpora. In this work, we propose to use document similarity methods to create noisy parallel datasets of code, thus enabling supervised techniques to be applied for automated code translation without having to rely on the availability or expen- sive curation of parallel code datasets. We explore the noise tolerance of models trained on such automatiiaally-createe datasets and show that these models per- form comparably to models trained on ground truth for reasonable levels of noise. Finally, we exhibit the practical utility of the proposed method by creating parallel datasets for languages beyond the ones explored in prior work, thus expanding the set of programming languages for automated code translation.
INTRODUCTION
As the pace of software development increases and the famous adage "software eating the world" (Andreessen) (011T) is borne out, there is a corresponding increase in the amount of source code and number of software artefacts active use for which support has lapsed. At the same time, the number of software professionals and programmers who can support and understand such code is unable to keep pace with the rate at which it is produced. This problem, while important when it comes to relatively modern programming languages (such as Java and Python), becomes even more pressing when it come to legacy languages (like COBOL) that mission-critical applications and systems are written in (Charette) 2020). In recent years, there have been multiple instances of organizations struggling to maintain their legacy systems and making considerable investments to upgrade them. In 2021 the Commonwealth Bank of Australia upgraded its core banking platform originally written in COBOL: this ultimately took 5 years and more than Billion AUD to com- plete (Irrera 2017). During the COVID-19 pandemic, software systems implemented in COBOL slowed down the release of US unemployment stimulus checks (Kelly) 2020). lcaving governments scrambling to find COBOL experts who were already hard to come by. A recent study by the United States Government Accountability Office (Walsh] 202T) has identified 65 critical federal legacy sys- tems in need of urgent modernization. Some of these systems are over 50 years old, and cost millions of dollars annually to operate and maintain.
Parallel to these developments are recent efforts at the intersection of software engineering, machine learning (ML), and natural language processing (NLP), which have posited the naturalness hypoth- esis of software (Hindle et al. 2016). The hypothesis states that "...Software is a form of human communication; software corpora have similar statistical properties natural language corpora; and these properties can be exploited to build better software engineering tools" (Allamanis al.) 2018). This hypothesis has been used to extend breakthroughs and advances from various NLP sub-fields to software engineering tasks such as code translation. Prior works in the code trans- lation domain have proposed the application of statistical, supervised, and unsupervised machine translation techniques to learn code translation models varying degrees of success.

---

Under review as a conference paper at ICLR 2022
A key limitation of a majority of the proposed code translation approaches, however, is the lack of availability of parallel data for training. Unlike natural language, where a piece of text is verbatim translated in multiple languages legal documents, parliamentary proceedings in multilingual soci- eties - code rarely implemented as is in multiple languages; thus making it hard to create parallel datasets. A few limited datasets such as Java ++ C# (Nguyen et al.] (013)) and AVATAR for Java <+ Python (Ahmad et al., 2021b) - are currently available. However, these extremely limited in the number of programming language they cover, and manually curating a dataset for a specific use-case is impractical. To bypass this limitation, unsupervised techniques have been applied to the code translation task. Unsupervised techniques come with their own limitations however; and often, supervised techniques can outperform them when the source and target corpora are from different domains, the source and target languages use different scripts, and on low-resource language pairs, among other concerns (Kim et al. 2020 Marchisio et al. 2020).
It is for this reason that in this work, we focus on one of the main blockers impeding the application of supervised techniques to code translation: the availability of parallel corpora and datasets. Specif- ically, we propose to utilize document similarity methods to create parallel source code datasets that are noisy by design. this work, we empirically demonstrate the effectiveness of document sim- ilarity methods in creating such parallel datasets with high levels of accuracy. Given that datasets created in this manner are bound to be noisy, we study the performance characteristics of models for code translation that have been trained on data with varying degrees of noise; and show that these models have considerable resistance to noise and perform well even with moderate amounts of noise. Finally, we demonstrate the practical utility of the proposed approach by training models to translate between 10 pairs of languages majority of which have not been looked at in prior work.
RELATED WORK
Code translation datasets: Typical methods for creating parallel datasets for code translation have either relied on the availability of open-sourced projects with implementations in multiple lan- guages, or on the existence of transpilers. The earliest widely-used large-scale dataset for code translation was for Java ++ C# (Nguyen et al. 2013) translation, created by indexing open-sourced projects implemented in both languages. Aggarwal al. (2015) used the Python 2to3 transpiler to create a dataset; while Chen et al. (2018) used CoffeeScript's compiler (which compiles down to JavaSeript) to create parallel dataset. More recently, Ahmad et al. (2021b) released AVATAR parallel corpus of Java to Python manually curated through submissions on competitive program- ming websites. Publicly available datasets for code translation are however extremely limited, and manually curating these datasets for a specific use-case is expensive and often impractical.
Source-to-Source translation: The earliest code translation models were rule-based systems, op- erating on handcrafted rules. These systems require a lot of effort to build, are not easily extend- able to other languages, and are also outperformed by neural techniques. Some of these systems are: Java22Sh arhpaapp rp 1ava2Python $ SmallTalk to C (Yasumatsu & Doi 1995), Cobol to Java (Mossienko 2003), and Tangible Software Solutions Fl (VB.NET, C#, Java, C++. and Python). Saattes,,Nuyenn
Moving away from ule-based systems, Nguyen et al.] (2013). Karaivanov et al. (2114). and Nguyen
et al. (014)) applied different versions of Phrase-Based Statistical Machine Translation to translate
between Java and C#. Chen al. (2018) proposed tree-to-tree neural network to translate the
parsed tree of the source code into the target code parse tree. The aforementioned supervised tech- niques have all been benchmarked on the Java ++ C# dataset, and are limited by the availability of parallel datasets. To bypass this limitation, Roziere et al. (2020) used unsupervised neural machine translation techniques to translate between languages using only monolingual corpora, and showed impressive results for translation between Java, C++, and Python. RozieRRoziere et al. (2020) trained the model specifically for code translation, large language models- such as GPT-2 (Radford et al., 2019) GPT-3 (Brown et al.) 2020). and Codex (Chen et al., 2021) have also been shown to have some competence in generating code (Hendrycks et al. 2021).
Parallel corpus mining: Prior work in natural language research has looked various ways of creating parallel corpora from a non non-parallel corpus. Munteanu & Marcu (0055 train a maximum
2

---

Under review as a conference paper at ICLR 2022
entropy classifier to identify if two given sentences are translations of each other. They extract parallel data from large-scale Chinese, Arabic, and English non-parallel newspaper corpora, and show improvement in model performance when trained with combination of small parallel corpus and the extracted dataset. [szkoreit et al.] (2010) describe a system that uses n- n-gram features to mine parallel documents from billion-scale scale corpus. (mmith et al. (2010) focus on aligning Wikipedia documents by creating features suitable for such documents. Artetxe & Schwenk) (2019) utilize specific scoring functions based on multilingual sentence embeddings to create parallel corpora, and Hangya & Fraser (2019) rely continuous parallel segments rather than word similarities to find parallel sentences. Banón et al. (2020) released the largest publicly available parallel corpora of sentences (223 million parallel sentences) by aligning sentences from data crawled over the web. There a substantial precedence of parallel corpus mining in the natural language domain; however, such studies in the code translation domain are non-existent.
Machine Translation using noisy data: Prior studies have aimed study the impact of noise on the performance of machine translation systems. Formiga & Fonollosa (202)) study the impact of misspelled words the performance of Statistical Machine Translation and suggest strategies to deal with them, while Goutte et al. (2012) study the impact of sentence alignment errors on the performance of SMT. Further, Khayrallah & Koehn (2018) define 5 categories of artificial noise in Neural Machine Translation, and study the impact each of these types has on performance. We moti- vate our work from these prior efforts in order to study the impact that noise has on the performance of code translation models.
PROPOSED METHOD
In this work, we propose to utilize document similarity methods to create noisy parallel datasets for code translation. We refer to the datasets created in this manner as "noisy" because unlike manually curated datasets, there is no guarantee of parallel implementation of the specific source file/code being available in the corpus: this may result in near-similar code samples being paired as ground truth examples instead. Algorithm pressents the proposed approach pseudocode.
The algorithm expects two non-parallel sets {d1.-..,dd ,dn} and D' (dl....,, d as input. Within the context of our work, the documents in these two sets represent code samples from two distinet pro- gramming languages. Along with the docu- ments, the algorithm also expects a similarity measure M (dd d') as input, to compare two given documents for similarity. A lower score from the similarity measure indicates higher similarity between documents. Finally, the algorithm expects similarity threshold ò help keep only sufficiently similar documents in the resulting parallel corpus. Thereafter, the algorithm follows a simple procedure of iterating over all documents; finding the most similar documents in the target set; and adding the newly found similar document pairs to the result only if the target document has not been paired be- fore, and if the similarity is below the threshold value. Once all the documents are iterated upon, the algorithm produces a list of unique pairs of code segments (documents) ordered by their similarity, ready to be used for downstream tasks.
of documents
Algorithm 1 Creating parallel code corpus
1: CreateParallelCorppra(]] D', M,
2: initialize P =
3: for i = 1 to [ do
4: Daim -GeiSinianDDocuennstd, D', M)
5: for (ds, d,)) in Daim do
if (-, d,)) $ and M d,) S 8 then
7:
P= PU (d;.ddd))
break Dres sorl((d1, d2) P, key M(h, d2)) 10: Return: Dres
8:
4 EXPERIMENTAL SETUP
To better understand the effectiveness and practical utility of the method proposed in Section 3 we devise 3 research questions and design experiments to empirically answer them (see Section S). In this section, we briefly summarize the different document similarity methods, datasets, pre-trained models, and evaluation metrics we use in our experiments.
3

---

Under review as a conference paper at ICLR 2022
4.1 DOCUMENT SIMILARITY METHODS
TF-IDF:
TF-IDF (Salton & Buckley 1988) computes the product of the term frequency (TF) (fraction of times a term appears document) with the inverse document frequency (IDF) (log- arithm the inverse fraction of documents particular token occurs in). The cosine similarity between the document vectors thus created computes the similarity of documents.
Okapi-BM22: The Okapi- model (Robertson al. 1995) uses the following scoring func- tion (Equation T to score the importance of a word w in a document D. Here, IDF(w) (w) represents the inverse document frequency of the word w. TF(w,, D) represents the term frequency of the word w the document D. [D] and Dang are the lengths the current document and the average document lengths respectively, and bą and are free parameters of the model.
(F, D)(k 1)
(M25 D) IDF(w)= x
TF D) k1 (1=11 b Dang
Latent Dirichlet Allocation (LDA): LDA (Blei al. 2003 is hierarchical generative Bayesian model that models each document as finite mixture over an underlying set of topics. The cosine similarity of the topic distribution of two documents computes their similarity.
Latent Semantic Indexing (LSI): LSI (Deerwester al. 1990) computes the Singular Value Decomposition of the Bag of Words representation of documents. The cosine similarity of the decomposed vectors of documents computes their similarity.
Word Movers Distance (WMD): WMD (Kusner et al. problem variant of the Earth Movers Distance (Monge 1781 the optimization problem defined in Equation
2015
models the document distance Rubner et al.) 1998) and solves
min T20
Teyc(i,
+==1
subject to:
Ti = d, Vi fI.
(2)
n Σ 1=1
Ti = Vj (1, ,n)
Here, T' E IR is flow matrix where Ti 2 0 and denotes how much of word in document d travels to word in document d'. c(i.j) = I x; ll2 is the cost associated with travelling from one word to another, d and d' are the nBOW representations of the documents, and X E Rdxx is the word embedding matrix where x; gd represents the d-dimensional embedding of the 7th word.
4.2 DATASETS
we utilize the following datasets. We provide representative code samples and statistics from the datasets in Appendix A
For the experiments whose results are detailed in Section 5
Java ++ C#: The Java <+ C# dataset is one of the earliest large-scale datasets introduced for the code translation task (Nguyen et al.) 20133 Zhong al.] 2010). It is created by indexing several open-source projects which have both Java and C# implementations, and pairing methods with the same file name and method name. The earlier version of this data was created by indexing the 4 and Lucene projects. More recently however, [Cen et al. (2018) indexed 6 open-sourced projects to create the dataset. We use the version provided by Chen et al. (0118) in work.
Java < Python, Java Java ++ C++, and Python <> C++: Roziere et al. (2020) extracted parallel func- tions in C++, Python, and Java from the online competitive programming platform GeeksForGeek
4

---

Under review as a conference paper at ICLR 2022
and used these code samples validation and test sets. We, however, concatenate the two datasets and use the unified dataset for our experiments. The code samples in this dataset are function-scope code samples that solve an algorithmic problem.
CodeNet: Project CodeNet (Puri et al. 2021) is a recently released large-scale Al for Code dataset, created by indexing two online competitive programming websites. The dataset is organized into about 4000 different problem sets, and contains little under 14 million total solutions 55 pro- gramming languages. Besides providing the code samples, CodeNet also provides input-output pairs to evaluate solutions to the problem sets.
4.3 MODELS
CodeBERT (Feng et al. 2020) a Transformer (Vaswani et al. 2017) based model, pre-trained a unimodal data of function-level code samples, and bimodal data of code and the associated documentation in natural language. The pre-training data contains code samples in Go, Java, JavaScript, PHP, Python, and Ruby, and is trained using the Masked Language Modeling (Devlin et al. 2019) (MLM) and the Replaced Token Detection (Clark et al. 2019) objectives.
CodeBERT:
GraphCodeBERT (Guo et al., 2020) is Transformer based model for code that also considers the inherent structure code by integrating the data flow in the pre-training stage. The model is trained on the CodeSearchNet dataset (Husain et al. 2019) using the MLM, Edge Prediction, and Node Alignment objectives.
GraphCodeBERT:
PLBART: PLBART (Ahmad al., 202 BART (Lewis et al. 2020 based model pre- trained on over 700 million Java, Python, and natural language documents collected from open-sourced code on Github and posts on StackOverflow. The model is pre-trained via denoising autoencoding, where the model learns to reconstruct input corrupted by noise function. The authors use three noising strategies: token masking, token deletion, and token infilling to create the corrupted inputs.
4.4 EVALUATION METRICS
BLEU score (Papineni et al. 2002 is a common automatic evaluation metric for machine-gen -eeeeneeeeee text, and exhibits high correlation with human judgment of quality. BLEU score is computed as the overlapping fraction of n-grams between the machine-generatee text and the reference text. The metric has however been shown not be reliable measure for source code (Ren et al. 2020 Allamanis et al. 2018 Austin et al. 2021).
BLEU score:
CodeBLEU score: Ren et al. (2020) propose the CodeBLEU score leverage the tree structure and semantic information in code. It is computed as combination of the standard BLEU score,
weighted n-gram match, syntactic abstract syntax tree match, and the semantic data flow match.
Exact Match (EM): EM (Nguyen et al. 2013) evaluates if the generated code matches exactly to the reference code.
Computational Accuracy @ k (CA@k); Recent work in code synthesis has adopted the CA @k metric (Austin et al.) 2021 Roziere et al., 2020) to evaluate code generation models. To compute CA @ k; k samples are generated from the model, and the problem is considered solved if any of the generated samples pass the unit tests associated with the problem.
5 RESEARCH QUESTIONS AND RESULTS
To validate the central hypothesis of this paper using document similarity methods to create datasets for supervised training of code translation models we define and seek answers to the following research questions (RQ):
RQI: How accurate document similarity methods in creating parallel datasets for code?
RQ2: Given the created dataset will be noisy, what is the effect of varying degrees of noise on code translation models?
RQ3: Can the proposed method be used in practice to train models for programming languages not explored in prior work?
5

---

Under review as a conference paper at ICLR 2022
5.1 RQ1: EFFICACY OF DOCUMENT SIMILARITY METHODS
We start our analysis by examining how effective document similarity methods are in creating code translation datasets. For this experiment, we utilize 4 datasets with known ground-truth mapping between pairs of programming languages - Java ++ C#, Java + Python, Java ++ C++, and Python ++ C++. For each of these datasets, we create parallel dataset using different document similarity methods, and compute the match accuracy as the number of correctly matched code samples.
We summarize the results for this experiment in Table 1 and observe that similarity methods that operate in a latent space (such as LDA and LSI) perform much worse than methods that operate in the original space (such as TF-IDF, Okapi-BM25, and WMD). We posit that because code is written in more formal language than natural language, and each data sample in the datasets used this experiment implements an independent unique function, there is likely no underlying topic or latent semantic associations that can be captured by LSI and LDA. Therefore these methods perform worse than methods that directly utilize the tokens in the original space.
Table I: Match accuracy of parallel datasets created using various document similarity methods. Match accuracy computes True if the matched code sample is the same as the code sample in the ground truth dataset.
Java < C#
Java ++ Python
Java + C++
Python + C++
(
10,300
1,418
1,418
1,418
LDA
47.21%
21.44%
35.83%
16.64%
LSI
57.21%
66.08%
87.66%
78.84%
TF-IDF
87.36%
86.10%
94.08%
89.35%
Okapi-BM25
87.86%
89.91%
95.77%
89.99%
WMD
89.53%
91.04%
95.06%
94.08% %
We note that the datasets used for the experiments in Table I are not true representatives of code expect to find while using this method in practice. This is due to the fact that the 4 datasets the other language exists.
used contain code samples for which a true parallel implementation When trying to create a parallel dataset from code collected in the wild, we cannot be sure of the availability of a true parallel implementation, which might affect the performance of the proposed method. Thus, to account for this phenomenon, we conduct a similar experiment with the CodeNet dataset. Since code samples in the CodeNet dataset not parallel implementations, this gives us a better idea of the effectiveness of document similarity methods in creating parallel datasets from code in the wild. We select 6 languages and randomly sub-sample 50 problems from the dataset. For each the code sample in each of the 50 sampled problem sets, we create a parallel dataset using different document similarity methods. Since we do not have ground-truth parallel implementation available for the CodeNet dataset, we cannot compute the match accuracy like we did in the previous experiment. We therefore compute the pseudo- match accuracy instead. The pseudo-match accuracy computes True if the matched code sample is solution of the same problem set, and False otherwise.
We report the pseudo- match accuracy results for this experiment in Table We note that while the TF-IDF and Okapi-BM25 methods performed well in the previous experiment, their performance varies greatly for this experiment. Datasets created using TF-IDF and Okapi-BM25 methods are matched to the correct problem set with as little as 30% accuracy and as high as 70% accuracy in some cases. Datasets created using the WMD method however achieve a high match accuracy for both experiments (Table [ and T
From the experiments designed to answer RQ1, we conclude that document similarity methods capable of creating parallel datasets code with a significantly high degree of match accuracy. Specifically, the WMD metric seems to be quite adept at delineating datasets for code translation.
5.2 RQ2: NOISE TOLERANCE OF MODELS TRAINED ON CODE
In the previous section, we showed that document similarity methods - and specifically the Word Movers Distance (WMD)- - are quite adept at creating parallel datasets for code. However, datasets created in this manner contain errors; therefore in this section, we seek to understand the effect that varying the degree of such noise has the performance of code translation models.

---

Under review as a conference paper at ICLR 2022
Table 2: Pseudo-match accuracy of datasets created by different document similarity methods 50 subsampled problems from CodeNet. Pseudo-match accuracy computes True the matched code sample is from the problem set, and False otherwise.
Source languuge: Go
Source language:
Java
JavaScript
PHI
Python
Raby 27.07%
Go
JavaScripe
PHP
Python
Ruby
TF-IDF
29.56%
41.91%
30.70%
40.35%
TF-IFF
65 05.005
53.40%
29.41%
31.32%
23.47%
WMD
71.47%
55.70%
51.35%
60.89%
45.12%
WMD
79.30%
73.93%
57.51%
66.49%
56.06%
language:
JavaScript
Source language: PHP
Go
Java
PHP
Python
Kuby
Go
Java
JavaScript
Python
Ruby
TFADF
50.93%
46.64%
44.89%
41.18%
35.34
TE-DF
64.31%
49%
45.66%
26.04%
37,62% 55.679
BM25
51.74%
58.12%
56.26%
59,74%
61 6.0%0
BM25
62.54%
50.48%
48.71%
27.97%
WMD
67.52%
WMD
71.38%
61.25%
73.79%
85.05%
84.08%
Source language: Python
Source language: Ruby
Java
JavaScript
PHP
Ruby
Go
Java
JavaScript
Python
TF-DD
67.19%
52.79%
66.67%
69.54%
72.46%
TE-IF
61.94%
51.53%
69.85%
61.42%
67.51%
73.33%
58.66 54.35%
62.22%
82,459
WMD
82.08% %
73.75%
77.39
79.59%
87.31
WMD
68.80%
78.59%
77.05%
90.39%
We use the CodeBERT and the GraphCodeBERT pre-trained models for this experiments, and fine- tune these models on different pairings of the Java ++ C# dataset (Nguyen et al., 2013) created using the different document similarity methods. We compare the performance of models trained these paired datasets with models trained on the ground-truth dataset, and a random baseline with random pairings of code samples from the two programming languages, Following Ahmad et al. (2021a). we compute the BLEU score, CodeBLEU score, and the Exact Match score.
The results for this experiment are summarized in Table We additionally refer the reader to Table see the corresponding match accuracy of the different document similarity methods. Interestingly, we find that models trained on noisy code datasets have a certain degree of resistance to noise: and while the performance drops with increasing levels of noise, the degradation is sudden. Even with high levels of noise, the models perform considerably well. With a high-performing method such the Word Movers Distance (WMD) with about 90% match accuracy the degradation in performance roughly percentage point across the three measures and for both directions of translation. For methods with a higher level of noise such as LDA with 47.21 % match accuracy while the performance goes down significantly. it is still significantly higher than the performance of the random baseline. We posit that although noisy datasets create pairs of code with incorrect parallel implementations, much of the semantics is still retained by the dataset due to the formal nature of programming languages. For example, even if code samples are incorrectly paired, the syntax for function and variable definition, code blocks, and indentation stays the same and is preserved. This allows the model to learn the translation task to a certain degree.
Table 3: Model performance the Java C# dataset matched using various document similarity methods. Fach method introduces a different amount of noise in the resulting dataset (see Table thereby affecting the performance.
Java
C#
C#
Java
BLEU
CodeBLEU
EM
BLEU
CodeBLEU
EM
Random baseline
12.2
31.71
0.0%
4.4
16.56
0.0%
LDA
57.55
65.97
33.2%
43.75
55.39
23.7%
LSI
71.8
77.64
47.9%
52.44
66.26
30,9%
CodeBERT
Okapi-BMi BM25
79.39
83.54
59.5%
73.83
80.64
57.6%
TF-IDF
78.78
82.70
58.1% 1%
72.52
77.34
57.1%
WMD
79.59
83.85
58.2%
75.16
80.93
59.2%
Ground-truth
80.83
84.86
60.6%
75.84
81.64
59.9%
Random baseline
6.88
20.09
0.0%
2.94
14.31
0.0%
LDA
60.34
67.91
37.3%
47.14
S8.79
26.3%
LSI
73.74
79.30
49.1%
52.86
66.49
30.5%
CodeECO
TF-IDF
79.14
83.04
57.8%
73.14
77.55
57.9%
Okapi-BM25
79.65
83.42
59.4%
74.24
80.56
58.0%
WMD
79.47
83.72
59.0
75.63
81,06
60.2
Ground-truth
85.05
61.1
76.76
82.03
62.3
While the preceding experiment allowed us to understand the performance of models trained on
noisy datasets created using different document similarity methods, wish understand the per-

---

Under review as a conference paper at ICLR 2022
formance characteristics of models trained with varying levels of noise more granular level. Thus we create datasets for translation between Java and C# by artificially injecting noise of varying levels. For explanation, in dataset with x% noise level, we randomly misalign 2% of code sam- ples in that dataset, while keeping the remaining code samples correctly paired. We then train the GraphCodeBERT model on these datasets, and compute the BLEU score, CodeBLEU score, and the Exact Match (EM) score. Figure [ shows the performance curve with levels of noise varying from 0% (ground-truth dataset) to 100% (complete random pairings). We observe that the degra- dation in the performance is gradual for initial levels of noise compared to performance at 0% noise, the performance at about 30% noise goes down slowly by about 20 percentage points across all measures and both ways of translation. Post this 30% noise level, we see sharper degradation in performance; and post 70% noise, we observe that the performance is just slightly better than the performance at 100% noise.
CodeBLEU
BodeBBEUU
60-
Evalusatoon metric
Evaluaatio metric
20
40
60
80
100
20
60
80
100
Noise (N)
Noise (%)
Figure 1:
Noise performance curve for GraphCodeBERT model fine-tuned for Java - C# translation (left) and C# - Java translation (right). While the performance decreases with increasing levels of noise, the degradation gradual for initial levels of noise.
Through the experiments in Table AableBand and in Figure I we get a better idea of the performance charac- teristics of models for code under varying levels of noise. We conclude that the performance of these models is not severely affected with moderate amounts of noise; therefore when creating datasets from code in the wild where we expect a certain amount of noise, we can expect the models to perform reasonably well.
5.3
RQ3: TRANSLATING BETWEEN A WIDER SET OF PROGRAMMING LANGUAGES
In Section 5 we concluded that document similarity methods are adept at creating parallel datasets for code with acceptable levels of noise; and in Section 5.2] we concluded that models trained on noisy datasets of code perform reasonably well under moderate levels of noise. In this section, we take advantage of these two findings and demonstrate the practical utility of the proposed method by creating noisy code translation datasets for languages not explored previously in the literature; and by training models for translating between these languages.
For this experiment, we utilize the CodeNet dataset by creating noisy parallel datasets between the following 10 languages C. C#, C++, Go, Java, JavaScript, PHP, Python, Ruby, and Scala. We choose these languages in order of their frequency in the CodeNet dataset, thereby maximizing the number of data samples we can potentially create. We also sub-sample about 2500 problem sets from the original 4000 problem sets from the CodeNet dataset for computational reasons. Thereafter, we match the solutions in one programming language to another for each of the 2500 sub-sampled problem sets using the WMD metric. Since the PLBART model can only translate sequences with a maximum of 512 tokens, we only match code samples with less than 512 tokens. We additionally use a similarity threshold of 3.0 and filter samples accordingly. In Appendix B.3 we provide statistics of the final dataset along with some representative code samples and their corresponding similarity scores. To create the test set for the language pairs, we sub-sample 100 problems that are not seen in the training and the validation set, and randomly sample 5 different implementations in the source language from each of the 100 problem sets for a final test set size of 500 code samples.
8

---

Under review as a conference paper at ICLR 2022
CP
C++
Java
PHP
Python Ruby
java
Python
Ruby
17577
11.05
6.75
2.71
1.52
3.25
32.55
22.47
C#
21.74
17.31
2.52
5.64
2.37
2.05
C+-
C+++
5.19
2.05
2.41
2.92
-0.52
Go
20.07
16.63
30. 18
37.73
17.83
22.20
22.43
20.07
13.49
8:29
17.66
-2.21
1.71
Soutce lmnuung
Spurce brquage
jaon
33.86
19.37
41.00
20.97
18.35
24.87
30.49
Javs
5.27
24.71
0.50
1.84
3.91
InaaSriit
25.59
19:06
28.35
14:53
26.50
16.75
13.92
24.91
JowSSript
9.60
19.86
5.11
0.00
-7.98
22.03
PHP
25.60
23.28
31.53
17.46
28.33
30.56
32.99
PHР
2.08
4.18
1.37
-0.86
6.91
Python
20.08
10.46
30.05
14.31
16.69
9.64
10.47
Python
4.38
28.60
0.50
.1.70
7.94
Ruby
11.74
11.04
24.21
15.53
16.24
28.43
1.76
23.25
2.59
0.48
3.83
Scala
19.21
13.47
29.35
17:83
21.34
20.44
27.61
28.66
1.52
8.85
0.00
.1.37
3.38
0.60
language
Figure 2: CA@5 for PLBART model fine-tuned on code translation dataset created from the CodeNet dataset using the WMD metric.
Figure Difference between the CA@5 of PLBART trained using data created using the proposed method and the CA @ of PLBART trained using randomly matched dataset.
We fine-tune the PLBART model on the matched training data for each language pair, and evaluate the computational accuracy @ 5 on the test set. We compare the performance of this fine-tuned model against a model fine-tuned on a dataset created by randomly matching solutions from each problem set rather than using the WMD metric. To compare these two models fairly, we keep the dataset sizes, problem sets, and all the hyperparameters constant across the two training procedures. The CA @5 results for the model trained on the WMD-matched dataset are shown in Figure while Figure shows the difference in the performance of the model fine-tuned using the WMD-matched data and the performance of the model fine-tuned using the randomly matched data. We present some of the model generated code samples in Appendix B.4
Overall, we observe that models trained using the WMDDmmatched datasets achieve noteworthy per- formance across language pairs. More importantly, when compared with models trained ran- domly paired data, we see substantial improvements for a majority of the language pairs. While the common language pairs, such as C + C++, Python - C++, Ruby - C++ see the biggest improve- ments, obscure language pairs such PHP + C++, PHP Ruby, JavaScript C++, and Python > Ruby also demonstrate substantial improvements over their random counterparts. This leads to the conclusion that the proposed method is a viable way of creating high-quality datasets for code translation, thereby alleviating the paucity of training data the domain.
DISCUSSION & CONCLUSION
Modernizing legacy applications into programming language is a process that requires lot of time, intellect, and monetary investment. Automatic code translation techniques have the potential to speed up this process, and to reduce the human effort required by either working in tandem with humans or automatically translating legacy code to a modern language of choice. While multiple techniques have been proposed to improve the quality of code translation, their practical utility is hampered due the limited availability of parallel data required train these models between languages of choice. In this work, we proposed a simple technique to utilize document similarity methods create noisy datasets for code translation; and demonstrated that models for code have certain amount of tolerance for noise and perform well even under significant amounts noise. We specifically demonstrated the effectiveness of the Word Movers Distance (WMD) metric in creating parallel datasets between numerous language pairs that have not been explored in prior literature: and showed significantly improved model performances as compared to models trained on randomly matched datasets. Future work will explore better metrics terms of both match accuracy and computational efficiency, thereby further reducing the noise in the dataset; and incorporating the similarity score in the model weight samples according to their computed similarity.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
One of the major ethical points to consider when dealing with the automatic creation and translation of source code centers around the effects humans: both humans who create and maintain code for a living: and humans that are affected by the decisions and outcomes produced by the execution of such code. For the former concern, our work merely seeks to align pre-existing bits of open- sourced code so that downstream data-hungry techniques may have more reasonable approximations of correct and on-purpose code to learn from. Our work does not replace jobs that humans are trained do and more adept at; and indeed defers and takes inspiration from prior studies (Weisz et al. 2021) that show that human generators of code are very likely to engage in partnerships with automatically learned models to produce or maintain code better. For the latter concern, we acknowledge that it is possible to use the output of artefacts from our work in downstream systems that can produce automatic code with little to oversight. Similar to work on examining the effects of large language models for human natural languages (Bender et al. (021]. much attention is needed where it comes to automatic code generation and translation techniques and models. We look forward to studying some of these issues partnership with colleagues in the future.
8 REPRODUCIBILITY STATEMENT
To allow for the reproducibility of experiments conducted in this work, we provide the source code of the experiments in the supplementary material attached with the submission. The supplementary material is grouped by the three research questions we define in our work. Each set of files within a folder corresponding to research question contains the source code to the respective experiments. Another critical aspect of our work is the creation of parallel code translation datasets across many languages from the CodeNet dataset. Along with the source code, we also provide the train, valida- tion, and test data sets for small subset of language pairs in the attached supplementary material. We provide a small subset due to the limitations on the amount of data that can be provided as supplementary material. However, plan to release the complete noisy dataset created for our experiments with the final version of the paper.
REFERENCES
Karan Aggarwal, Mohammad Salameh, and Abram Hindle. Using machine translation for convert- ing python 2 to python code. Technical report, PeerJ PrePrints, 2015.
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang, Unified pre- pre-training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 2655-2668, 2021a.
Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai- Kai-Wei Chang. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108. 1et.:21088.19900 11590, 2021b.
Miltiadis Allamanis, Earl Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4):1-37, 2018.
Mare Andreessen. Why software is cating the world. Wall Street Journal, 20(2011):C2 2011.
Mikel Artetxe and Holger Schwenk. Margin-bases parallel corpus mining with multilingual sen- tence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computa- lional Linguistics, pp. 3197-3203, 2019.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint AXX::2008.077::: 2021.
Marta Banón, Pinzhen Chen, Barry Haddow. Kenneth Heafield, Hieu Hoang, Miquel Espla- Gomis, Mikel Foreada, Amir Kamran, Faheem Kirefu, Philipp Koehn, et al. Paracrawl: Web-scale acquisition of parallel corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4555-4567, 2020.
10

---

Under review as a conference paper at ICLR 2022
Emily M Bender, Timnit Gebru, Angelina AcMillan-Maipp and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings ofe the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623, 2021.
David M Blei, Andrew Ng, and Michael Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022 1022, 2003.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14]66 2020.
Robert N Charette. No one notices the creaky software systems that run the world until they fail. IEEE Spectrum, 57(9):24-30, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Ed- wards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107, 107334407,,,377,, 2021.
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. Advances in Neural Information Processing Systems, 31, 2018.
Kevin Clark,
Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41 (6):391 4-17, 1990.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1171-4186, 2019.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin. Ting Liu, Daxin Jiang, et al. Codebert: A pre-trainn trained model for programming and natural languages. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: Findings, pp. 1536-1547, 2020.
Lluis Formiga and José AR Fonollosa. Dealing with input noise in statistical machine translation. In Proceedings of COLING 2012: Posters, pp. 319-328, 2012.
Cyril Goutte, Marine Carpuat, and George Foster. The impact sentence alignment errors on phrase-based machine translation performance. In Proceedings of the 10th Conference of the Association for Machine Translation the Americas: Research Papers, San Diego, California, USA, October 28 28-November 1 2012. Association for Machine Translation in the Americas. URL https://accanttoooogg
2tgg/0122 amt awtaaattaapppaa. paperss.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. In International Conference on Learning Representations, 2020.
Viktor Hangya and Alexander Fraser. Unsupervised parallel sentence extraction with parallel seg- ment detection helps machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1-24-1234, 2019.
Dan Hendrycks, Steven Basart, Sauray Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song. et al. Measuring coding challenge competence with apps. arXiv preprint S.Xv:22055009333 2021.
Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. On the natural- ness of software. Communications of the ACM, 122-):1 2016.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis. and Mare Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv: 190.009999 2019.
11

---

Under review as a conference paper at ICLR 2022
Anna Irrera. Banks scramble to fix old systems it 'cowboys' ride into sunset. Apr 2017. URL http::///wwwwwwwteeee cww wtt.ttttrr..ccoo.atttteeeeeeeeee coboll cobol/ banks- Gandss--crrambl----f-----00----0
as- - www.:eittttsssttttteeeee idm
Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software, pp. 173-184, 2014.
Makena Kelly. Unemployment checks are being held up by a coding language almost no-
wtt.e//eewww.eeeeeeeeeeee com 2020/ http://///2//2///222222222 /ttp////////1115//11// body knows, Apr 2020. www.puunmmmmeeemmeee systems coaettees-ccoooloo legacy serrsaccysoffttee soft infrastructure
coronavi us
Huda Khayrallah and Philipp Koehn. On the impact of various types of noise on neural machine
translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 74-83, 2018.
Yunsu Kim, Miguel Graça, and Hermann Ney. When and why is unsupervised neural machine translation useless? In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 35 45,44,25 2020.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning, pp. 957-966. PMLR, 2015.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 787 78810, 2020.
Kelly Marchisio, Kevin Duh, and Philipp Koehn. When does unsupervised machine translation work? In Proceedings of the Fifth Conference on Machine Translation, pp. 571-583, 2020.
Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Histoire de l'Académie Royale des Sciences de Paris, 1781
Maxim Mossienko. Automated cobol to java recycling. In Seventh European Conference onSoftware Maintenance and Reengineering, 2003. Proceedings., pp. 40-50. IEEE, 2003.
Dragos Stefan Munteanu and Daniel Marcu. Improving machine translation performance by ploiting non non-parallel corpora. Computational Linguistics, 31(4):477-504, 2005.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 651-65 1-654, 2013.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Migrating code with statistical ma- chine translation. In Companion Proceedings of the 36th International Conference on Software Engineering, pp. 544 1144-57,222 2014.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311-318, 2002.
Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. Project codenet: A large-scale ai for code dataset for learning diversity of coding tasks. arXiv preprini arXiv:2 /2655, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint 2009 20299 2020.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulien Mike Gatford, et al. Okapi at tree-3. Nist Special Publication Sp, 109:109, 1995.
12

---

Under review as a conference paper at ICLR 2022
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. In Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 20601-20611. Curran Associates, Inc., 2020. URL http:ss///rrccceeinnss.ooo neurips.. pdf
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. A metric for distributions with applica- tions image databases. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pp. 59-66. IEEE, 1998.
Gerard Salton and Christopher Buckley. Term-weightin approaches in automatic text retrieval. Information processing & management, 24(5):513-523, 1988.
Jason Smith, Chris Quikk, and Kristina Toutanova. Extracting parallel sentences from comparable corpora using document level alignment. In Human Language Technologies: The 2010 Anmual Conference of the North American Chapter of the Association for Computational Linguistics, pp- 403-411, 2010.
Jakob Uszkoreit, Jay Ponte, Ashok Popat, and Moshe Dubiner. Large scale parallel document mining for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010). pp. 1101-1109, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Kevin Walsh. Agencies need develop and implement modernization plans for critical legacy systems. 2021. httpa//////a.a///////o/oo///////
pdf
Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Mar- tinez, Mayank Agarwal, and Kartik Talamadupula. Perfection not required? human-ai partner- ships code translation. In 26th International Conference on Intelligent User Interfuces, pp. 402-412, 2021.
Kazuki Yasumatsu and Norihisa Doi. Spice: system for translating smalltalk programs into environment. IEEE Transactions on Software Engineering. 21(11):902 -122, 1995.
Hao Zhong, Suresh Thummalapenta, Tao Xie, Zhang, and Qing Wang. Mining api mapping for language migration. In Proceedings the the 32nd ACM/IEEE International Conference on Software Engineering- Volume 1, pp. 195-204, 2010.
13

---

Under review as a conference paper at ICLR 2022
A REPRESENTATIVE CODE SAMPLES FROM UTILIZED DATASETS
In this section, we provide representative code samples from the datasets we use in this work. List-
ings] and 2 are data samples from the Java + C# dataset. Listings and 4 are data samples from
the Java <- Python dataset. Listings aare data samples from the Java C++ dataset. Listings
7 and 8 are data samples from the C++ < Python dataset. Finally, listings 9 [0 11 12
15
16 provide code samples from one particular problem set from the CodeNet dataset.
Listing Java C#: Java code sample
Listing 2: Java + C#:: C# code sample
public static Cell getCell (Row row, int Ir.ddn)r.ee))
public static 10ell GeR rew, int oc cclum)
Call cell cuw-gel Dwll (
ICe1l cell row. Gellll (oalum));
(cell selli
(cell aulli
cell cow.creat Cell (ndlemnIndexx]
wow.CreateCell annl
seturn cell)
securn coll)
Listing 3: Java < Python: Java code sample
Listing 4: Java ++ Python: Python code sample
stacie iitt bi neey int
def bineryTofecinel
17)
des_valua 0 ins base :
dec_valus
inc Leitg - nam 7
bass = 1
shile canp > 0
shilw Lemp
temp nun
Lengi Leep 10
last_digit
last_digit
dec_value last_digit bass
temp int { temp
base bass 2
base base 2
dec_value + last_digit bade
des_value
recurn des_value
Listing 5: Java ++ C++: Java code sample
Listing 6: Java + C++: C++ code sample
stacid int finds int 1
int Finds Lnt
inz for sun 5
1
12 R. -+ 1
eun < m7
sum B R. ++ 1:
sum + +1
return
securn
1;
Listing 7: C++ + Python: C++ code sample
Listing 8: C++ ++ Python: Python code sample
void prineoietinct int aze n.)
peintDiscinct aEc acr,sort C for in range (n 1 it ( 1 <
eort art, art 15 for int i 0; 1 <
and arz [.A arr 1.1. +1+11
arr
1
cout ec asc (11 sen.
else print aer
end
Listing 9: CodeNet: C code sample
Listing 10: CodeNet: C# code sample
#inel und.ciideetdii...
int. meint1
iin Systems claas
For (
atat void Maint)
peint (Meddddddeeed 3ta
(lus Link i = 1f 10; 1+t) For Cink = 1, be < 10y (++) Mondoia.NiiteLinee (1ett feet
+1+d)71
14

---

Under review as a conference paper at ICLR 2022
Listing 11: CodeNet: Go code sample
Listing 12: CodeNet: Java code sample
main
public
Import
"Ent"
public stocio vold main (Stringt) argst
cene main
fortint amlpacelpass))
12 10; 14-
tia bel:boeoghann)
344 Ent E EMaasaaeed tup Je 1+
System, out. pat.printin (a 464"""44a"""
Listing 13: CodeNet: JavaScript code sample
Listing 14: CodeNet: PHP code sample
Ox 1843 fox Ivar jel; je10p 1011 consoleilogit is +. 'x" +.d +d- i-d1
<Ppop
1oo110111p1 01---433 for 2 038ge13
wcha $2, hwf.Bjj. j. fer. $1..8. Saty
Listing 15: CodeNet: Ruby code sample
Listing 16: CodeNet: Scala code sample
Your code here!
object Maini
def Acray (String]] tor 92
lwiel
1.tinesi131
For <- 1. 994
println(i tg 1.3 3 Exgt
3-jat
puls Lo (joooattf (1 staa
B PARALLEL DATA CREATED FROM CODENET
In this section we look at the various properties of the parallel dataset created from the CodeNet dataset. In section B.I we present most similar data samples identified by the WMD metric across various language pairs. In section B.2 we present the histograms of similarity scores in the final dataset. Finally, section B.3 we present the number of data samples the final dataset created from the CodeNet dataset.
B. IDENTIFIED MOST SIMILAR DATA SAMPLES ACROSS LANGUAGE PAIRS
In table 4 we provide the mist-similar code samples identified by the WMD metric across various language pairs along with the computed similarity between the two samples.
15

---

Under review as a conference paper at ICLR 2022
Table 4: Most similar code samples across various language pairs identified by WMD metric
Python dataset: Similarity: 0.75
Minclude <stdio.b> omath.n>
int fraw input (1)
int maint)
For sange (a)
int
yl, ya, y3 mlptl raw.inpat
x1, x2, x3, yh, y2, 38, pay PY,
apliL
("Sd""" anty
(1y2 (x1xx2 + y1+y1ı
ty3
y1)
x2 y2xy2) lyl 621
y3
Cox
1993
(2x
+yy y2)21
Scant (1L1E 1iz VIE 912 E12 s12", 5x1, 631, 6X2
PV
(1x2 -31-1x1-11 + y1+y11 003
x1)
19 4y2, 483, sy31;
x2 y2xy2) 181 x21 (x3
y3xy31
1/ (2+ (yl+ X31 +.42 (x3 x1)
x2+x2 y2-y21 131 (2343)
(2
y31 K2-
math. wwt.haggtttpppeee........ (pow:0d px), 21 pow(lyl pyl.
y21 0y1
y21131
2)1
(082 x2) (x1--x
y1-y1) + 183 2
y24y21
ixl (x3433
print 14.35 4.36 k.31"
13 (y1
(2.
xi) 33x
x2113r
agrt (powd (ed pe), 23 powttyl FV). 21
priacr("N.32 4.32 N.32507, pa. PV: 212
return Oz
C#
Java dataset: Similarity: 0.62
using System;
cleas Main
claes Program
public static vold (Sinintri
atabie voud Rain(steengg]l aegs)
Gin 02 1000; 1:1l
for Grt i - O5 5 1000; 144)
System.ecc.griieennteellll World");
Corsole.Riiteian ("Hello Nosld")>
Scala Ruby dataset: Similarity: 0.91
ocdect Kein extence App
gete.to_i
well Arrey
1.
2,
1,
5.
2.
by 1. 1. 1, 14, 1, 5, S, 2, 2,
2,
< 5,
2, 1,
14,
1,
5,
1,
P,
4 15, 2, 2, 5, 6.
6
2,
517
4 1, 4, 1, 51 (ep.it (1. .nap (s:to_A)
val scnla.1o.
elk-1]
printlatsik)
РHР Python dataset: Similarity: 1.34
16

---

Under review as a conference paper at ICLR 2022
<Pppp
shile Tzuez
lint For im input ressrip splitt
(Bline trin figets (S7DDN0)
Scun explode (1lnw)
cun[0] and nun[]]
i0 (Snan[0] -- 44 Snanf1] -- oy
break
(1)
aun.scrt(
"-jointste (xt Cor num)
(Smund)
91ine imploda 1M S0un))
ocho Gline EHP_BOL;
3 <2pbp
191ine trimitgets (STD1N001
9nom expl ode -,81ine)2
12 (Smun[0] 44 Snun[1] =n 0)
exic
(8nun)
$3ln implude (* Soun)
echo 9iiie EHP_5002
B.2 SIMILARITY HISTOGRAMS IN THE CREATED DATASET
We show the histograms of the similarity scores in the datasets matched through the WMD metric on the CodeNet dataset in figure
B.3 DATA STATISTICS
In table we provide the number of data samples in the final dataset created from the CodeNet dataset.
Table 5: Number of data samples for each code translation dataset. Fach row represents the source language, while each column represents the target language. Due to the skewed number of submis- sions in different languages in the CodeNet dataset, see the same in the created parallel dataset.
C
C#
C++
Go
Java
JS
PHP
Python
Ruby
Scala
C
x
2,295
15,534
709
8,900
1,354
837
10,499
5,564
1,247
C#
2,504
3,145
595
2,629
846
635
2,639
2,007
821
C++
30,977
3,388
x
1,358
17,809
1,793
1,550
44,707
14,375
2,183
Go
1,732
974
2,496
2,077
642
669
1,755
1,315
Java
15,909
5,016
23,821
2,858
2,262
2,202
12,846
7,746
2,034
JS
2,876
1,907
2,954
998
2,653
x
1,238
2,893
2,307
1,349
PHP
2,802
1,741
2,898
1.180
2,454
1,389
x
2,996
2,310
1.133
Python
27,947
7,778
76,825
7,235
29,007
6,666
7,641
X
48,301
7,126
Ruby
28,423
12,595
48,985
8,838
23,515
8,322
9,252
62,298
6,457
Scala
5,506
4,720
6,660
3,343
5,953
3,536
3,249
6,937
5,586
17

---

Under review as a conference paper at ICLR 2022
W0L-700 Maat
Jendin ware
Figure Histogram similarity scores various language pairs matched by WMD
B.4 CODE GENERATED FROM FINE-TUNED PLBART MODEL
In table 6 we show examples of code generated from a PLBART model fine-tuned on dataset created using the WMD metric. We show examples from C + Python, C# + Java, C++ - Ruby, Java Ruby, JavaScript language pairs. For given input source code, we show an example of the correctly generated and incorrectly generated code.
PHP Python, Python C, Ruby - C. and Scala Python
18

---

Under review as a conference paper at ICLR 2022
Table 6: Examples of correct and incorrect translations given the source code to PLBART model fine-tuned on a dataset matched using the WMD metric
c Python translation
Source C code
Correct Python translation
Incorrect Python translation
Hinolurk catdio.
iin Fingat (1) List (rig I Riity Anpus 12 -pptit
I'n Cingat (1)
int main miintvold) I Long nj
List (mage (inkl Eoput 13 -apppt + 34
+ 24 tn):
scenf acanfitklis pakld", 5n)s
For:
For
In)
#[1]92
w(1]92
fox (Lend scan21"91d", 4a11111
peint ("Clestr)
("Ciea")))
exit D
exit.)
1
print ("eecond")
print ("swcond")
For (Cogg 1 1 2011
(a:1]82 1-
return 3:
""sot(""ecoodaa")
C# Java translation
Source C# code
Correct Java translation
Incorrect Java translation
using Systems
impost Java.uti1.
import jova.
namespace TenkslA
public class Main
public class Main
public static woid main (String
public static void main (Stringg
class Program
Scannar now Scanner
Scannor sc now Scanner osyston..
statio void Maintstring argsi
+ in);
+ in)
set sc.next Tat (3:
iet à sc.next Tnr (1:
string[] isput Sonsale.
int se.next Int ():
int B se.nex: ():
Readisne .ppltt
int so.nextInt
sc.nextInE ()3
Perse (irpot
falsez
int Part_ (inpat (1)
(A B1
int Pat.Pa Paoppt 121);
LE (C B 66 AoC) [
if (C > 8.8 6s AoC) [
bool tig talee;
tig truez
tig truez
1= (A
}
wlsw if
(C BELAPC)
CHAI
64 APCI
[
f13 2500)
21q C290)
fg Lruep
}
}
jelse (3>A)
(f1g)
(BHCS6CHN)
: alee
3 elce Byazen. out...ouu...eoo
f1g-trus)
suc. (*
co.close1))
(ellg)
Sancolo.Mestclaa ("Yos")
#lse
Consale Li (?
C++ > Ruby translation
Source C++ code
Correct Ruby translation
Incorrect Ruby translation
19

---

Under review as a conference paper at ICLR 2022
Finclude seatdio>
N 5w-a.5o_4
N guts.co_1
Hincloee <iostream>
get (5
gets. mep.eetttt.m 11
sinclude selgorichr>
Artey.nex -Posts:: IIFFIIITV)
Arrey.nexiN Plost:t
tinclude Seatring?
IIFINIIY)
tinclude <enath> using nanespaco std; Lypedar Long long Lig int a,k,ans f1
nx[0] ez.each do 141 -f uxtal
nx13] ac.each do |a1
if nxfa]
int a/520035]7
eissf mx]a] mxta mxta] mxls - 1] end
elsif mxja] mxia
int mx(300305] 1997
mxia] mxta - 1] end
int meint)
end
end
cin neneet (tw.-t0t
pets (nx.nax
mx100
forlint - 171 <n ar2 +-) cin
+ >> al-17
tortint 1
+-11
iFte[5] naxtnafad (2)
mx]a[]]]
- 1f 112
ans maxi(ans, mx
lirlzig) -1 endl; andi)
cost
return 3;
Java + Ruby translation
Source Java code
Correct Ruby translation
Incorrect Ruby translation
jasa, Hashhhpp
imgort Jeva.utii 11.Mpp
gecs.oplit. it (ash 111
gets.aplit kach.n2wi01
import S.va.util.Scamme..
public class Main
a.each so k(i) 4- end h.valvos.el12 INI
a.each d0 111 a[1] 4= end hevalues.el17
publ fiz statiee volid mann (g
C- argst throws Except lon
File
(ffeett
+ "xx");
17 Scantwz ac new Scannes iflle c
Scanner
Scanner (8ysten,
int se.naxtint ())
Hap<Inteoss, Boolcan> nap BOM
- EantNopcxx1;;
For - Dy 1 [+-]
Int sc.naxtInt
isinull nap. (
Systen.out..pe wtt.:outt..inntnnnoo"")...
sc.close 1) rezernJ
Jolces
nap.pot (ay 2rue)27
sc.close 1)
Javascript C translation
Source JavaScript code
Correct C translation
Incorrect C translation
20

---

Under review as a conference paper at ICLR 2022
"use striet!
Linelude <oldio.3> int main (void)
Hncllde <aidio.h> int maintrole)
main irgut
arga ingut i (nf)..
nag (arg azg.aplil(
-t My by acant (Ptd Id", 63, sb)s
my scanf sd", sa, sb)
b ')
const A (arg: [0],
if ta 3 om D Dilbe 3-3mm 1
if (2 6.3 mm 0 11 b.b. (MMosssaiPaa else
(- 10:
5 0) printct "Fossibla
www.eInt (arga
-+ 10:
prince ("InpcsaiDeeha") return 07
printe p ("Impossibieaa"))
12 (8 t
0 8
return 07
- 11 1A + $
- console.iog 1on.ooie..g""ooollaaa Poosohlle")
ccaselo. (tm.oesiieeii
Impocsibler)
main(requi re Fat) reacr: ledync 4 (f /dd///sccinn, (uttsr))
PHP Python translation
Source PHP code
Correct Python translation
Incorrect Python translation
<2pbp
ingut 13 cla Palse Elg True
inpul ()
Escant (37DIN, "LS", Ro)
falaer
for range (len(st):
Caratepo (8n, (AT) --- falaw) Jalsa:
elawz 1E "X" in
*A" in at-l: 18 "g* in s141:
(fstr (B") --- Islcoll
True
relee:
print (*Yoa" tig "No"1
bices
tru0s
if be princ ("ioa"))
elses
prine
osho "Yoo"?
telees
wcho *N077
Python - C translation
Source Python code
Correct C translation
Incorrect C translation
iinclude caldiv.ho
#include caldiu,h> (void) int Nj (cant"" 6312
(ingut O.
int. main
1ist tmap, impat (1 .pp1i: - DX2)
:
int EJ
acant an]2
int a[N]; for Hint s"ad" 63(1]17
1 144)
int a1100005[7
(-int
tor 11 02 i < 1-+1
sys.exit
scant Aalills
irt ans 0; int count
for La zange (N)
int ans 0y int
st
lint [ali]
149)
ror 11
else
araitz
ana 1=
[al-] -- count] count+-?
coaaL
print lane)
else
seturn
frintei""ddis ans); return 02
Ruby - C translation
Source Ruby code
Correct C translation
Incorrect C translation
21

---

Under review as a conference paper at ICLR 2022
A, gets..ppitt.oo (a:to_A) gets.chorp
uinelude satdio.1i int main (xo1d)
Emeiudesatdio..s
maintvote)
("NGJA (A (d(4)))
puta else
-ot R,8; acanf (P 4d), char 9:1001;
A, acanf 4d")
puts "Bo"
ctar 9[100]:
end
auant (*9u", (F(800]==
printe("Mo"])
return 02
printsi"Yerk""12
21(8(0) (4660))
eiset
Priote
print PNNo")
roturn 07
return
6810 --/ ( (
pxintE((Mo")) return 07
-E(R (a
pzintE("Mo")
relurn
+ '][ printe("Mo"]
rotuzn
(5(0)) (0G600)--126660 "F
- '11 print F("N")); rctazn
C- ( printe(""oo")
print Fi""t"));
printz (1
return 02
Scala -> Python translation Correct Python translation
Source Scala code
Incorrect Python translation
import
import eys
import eye import ou
object Main extands App
age. ays.atdin. read)l
vall lines: Iterator (Stiing]
for 1ine in lines:
Lamoda liise (map (int, input + split (113
ce Source, sccin. getlinees
S, b cint, line lineveplit ca
wall line next it
if "lons ne.envi ron
.upp Lutnn (aake
upen /ängut.
line.head
peint
fup txtr, faft
vall (1ne())
elees peint Cb 1)
def eclive ()
inlif (b al eille
c- 1).LoBleing!
B [C)
print
2olve()
22


Reviews:
[{"review_id": "kJrFH4foW1l", "reviewer": "Reviewer_nRDP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The proposed technique can be useful for legacy code translation where annotated data is scarce. \n+ The empirical investigations are solid.\n+ The paper is well-written and easy to understand.\n\n- The authors did not provide any explanation/intuition/evidence why such a simple data augmentation technique works.\n\n- The main problem with this approach is that the performance suffers significantly with CodeNet dataset. The reason behind similarity-based data augmentation will work is because for implementing similar functionalities often similar variable names are used. However, such an assumption may not be true for CodNet where developers did independent implementations. Thus, the dropping of performance for CodeNet is not surprising.  However, this raises questions about the usability of this technique.\n\n- The contrast between performance with noise is useful, but again that shows how sensitive this technique is with noise and for some legacy code (e.g., COBOL --- a language authors use to motivate the study) the amount of noise might be high.\n\n\n", "summary_of_the_paper": "The paper proposes a simple similarity-based data augmentation technique to translate code written in one programming language to another. With empirical investigation, they show such apparently simple data augmentation closely matches the performance of the models trained on the manually annotated datasets. ", "summary_of_the_review": "The authors propose an approach of data augmentation for code translation for document similarities. However, the results show the approach suffered for independently developed code (CodeNet), which is the most realistic case. The approach is also susceptible to noise. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DJ2IQx6jgs", "reviewer": "Reviewer_rSze", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although I recognized that the main contribution of this paper is not proposing a sophisticated method to align samples, I thought the paper need more attention to design efficient strategy. The entire alignment algorithm takes O(D*f(D')) where f(D') is the complexity of GetSimilarDocuments(). Unless any assumption this takes O(DD') meaning that it tends to be infeasible by increasing the data. Also, the algorithm seems a simple greedy method and may be heavily affected by the sampling order.\n\nAccording to Table 2 and 3, I observed that the pseudo-match accuracy introduced here does not reasonably reflect the real characteristics needed to be considered in code translation models since the BLEU of several systems compete with each other.\n\nI recognized that Figure 1 shows every range of noise ratio, except around 100%, has certain gradient against the accuracy, and got the opposite conclusion \"the less noise the better\" against that the paper concluded \"we expect a certain amount of noise, we can expect the models to perform reasonably well.\" The root cause of this is that the paper did not provide a reasonable criterion about acceptance of this experiment first.\n\nAs far as I saw the Figure 2 and 3, the accuracy is heavily reflected by the amount data available during training and some normalization should be needed to compare these metrics each other without biases. Figure 3 also show some remarkable characteristics in translating only into C++, but the paper did not mention about this point.\n\nThe motivation of the paper includes \"modernizing legacy applications,\" but the experiments does not reflect this perspective. For example, the first section repeatedly referred to the COBOL applications, but we maybe not be able to reflect the experiments into this application due to large discrepancy between languages.", "summary_of_the_paper": "This paper proposes to use similarity metric to generate pseudo alignments between source/target program pairs. Generated pairs are utilized to train program translation model. The paper described a simple greedy method to align both codes, and experimented 5 types of similarity metric as its inner measure.\nAccording to the experiments, the word mover's distance works notably well for this purpose, but other metric can also improve the translation accuracy significantly against random selection.\nThe other experiment investigating a performance curve by changing noise ratio in the ground-truth parallel corpus hypothesized that certain amount of alignment errors can be acceptable since the actual performance can be maintained.\nThe paper also challenged to construct translation systems between arbitrary pairs in 10 programming languages using the proposed framework and observed that the trained system works with certain accuracies.", "summary_of_the_review": "- The alignment algorithm need to be improved. This is not critical: this is still acceptable for the first observation.\n- Conclusion sounds sometimes not reasonable according to experiments. Some results are hard to discuss due to unnormalized data.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9T7r7mqqEB", "reviewer": "Reviewer_3acE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In its current form, the problem statement and approach seem very trivial. First, the data sets seem rather unsophisticated and it is not clear what the pathway is from this method to say mining parallel corpora from github or some large-scale code repository websites. In its current form, it seems like this is preliminary work and with some more sophistication (in method + data sets) it is going to be a good future task. I suggest that this be submitted to an appropriate workshop and a future attempt be made to submit this to ICLR. In particular, good directions can include in-the-wild performance when tested on something outside of coding-problem data sets (especially since those are so well-organized by language and problem).\n\n", "summary_of_the_paper": "The paper constructs (weak) parallel corpora for code translation tasks using nearest neighbor sampling in the code-document-embedding space. A variety of similarity metrics / base LMs are evaluated.", "summary_of_the_review": "The method and data set in this paper in its current form are rather trivial and more sophistication is required. The techniques seem highly tuned for mining well-organized data sets from coding-problem websites. A better and more improved model that can scale in the wild is recommended and would be a better fit for this venue.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9DU0yZ8E2uy", "reviewer": "Reviewer_xtd8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. The paper is easy to follow. Although I am not super familiar with code translation, I can catch most of the points.\n2. The proposed method is simple and easy to use.\n3. The experiments are relatively thorough, covering different types of document similarity methods, programming languages, model architectures, and evaluation metrics. The results seem to be convincing.\n\n**Weaknesses**\n1. My main doubt is the results presented in \"Section 5.3 RQ3: TRANSLATING BETWEEN A WIDER SET OF PROGRAMMING LANGUAGES\". The authors report CA@5 scores here but in the other place, BLEU/CodeBLEU/EM is used. So, what is the intuition that uses different metrics? BLEU can also handle multi-reference evaluation. It would be nice if the authors could add these results in the author response.\n2. The research is somewhat superficial. The authors only show the models can tolerate certain noises but do not propose any simple heuristics to alleviate the impact of noises. For example, penalizing the noises (i.e., the targets not belonging to the source) during model training (fine-tuning). [1] might inspire the authors.\n\n[1] Wu, Lijun, Jinhua Zhu, Di He, Fei Gao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. \"Machine translation with weakly paired documents.\" In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4375-4384. 2019.\n\n**Typos**  \nSection 3 Proposed Method: curated -> created", "summary_of_the_paper": "This paper mines noisy parallel datasets of code by calculating the similarity between two non-parallel sets of documents. The authors first show that the document similarity methods can indeed align parallel documents and find that the word movers distance (WMD) is the most effective one. Then, the authors show the high tolerance of models trained with noisy datasets. Based on the two findings, the authors finally apply the proposed method to a large, non-parallel code dataset, and observe a performance boost of using a noisy parallel dataset compared to randomly paired datasets.", "summary_of_the_review": "This paper is interesting, but the research is somewhat superficial. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Motivated by addressing the problem of lacking parallel training data for supervised code translation, this paper proposed to construct noisy parallel source code datasets using a document similarity based approach, and empirically evaluated its effectiveness for code translation tasks. 

The paper is in general well-written, easy to follow, and the method is simple and empirical results look positive. Some major concern by reviewers is that while the proposed method is simple and may be easy to use, the overall technical novelty/contribution is limited, e.g., there generally lacks of more thorough discussions on how to deal with the critical noise issue in a more robust or sophisticated way. In addition, there were also other concerns about the experimental issues, such as datasets, metrics, ablation analysis, usability, etc. 

Overall, the paper presents some preliminary positive results for an interesting research problem, but the overall technical novelty and contributions are incremental and the paper is not strong enough for the acceptance by this conference. Nonetheless, this work could be potentially valuable for the niche area of code translation research, and authors are encouraged to continue to improve this research with more thorough investigation for a future venue.
Decision: Reject