iPTR: Learning a representation for interactive program translation retrieval
Authors: Binger Chen
Ziawasch Abedjan
Venue:  (2021)

Abstract: Program translation contributes to many real world scenarios, such as porting codebases written in an obsolete or deprecated language to a modern one or re-implementing existing projects in one's preferred programming language. Existing data-driven approaches either require large amounts of training data or neglect significant characteristics of programs. In this paper, we present iPTR for interactive code translation retrieval from Big Code. iPTR uses a novel code representation technique that encodes structural characteristics of a program and a predictive transformation technique to transform the representation into the target programming language. The transformed representation is used for code retrieval from Big Code. With our succinct representation, the user can easily update and correct the returned results to improve the retrieval process. Our experiments show that iPTR outperforms supervised baselines in terms of program accuracy.

Under review as a conference paper at ICLR 2021
IPTR: LEARNING A REPRESENTATION FOR INTERAC- TIVE PROGRAM TRANSLATION RETRIEVAL
Anonymous authors Paper under double-blind review
ABSTRACT
Program translation contributes to many real world scenarios, such as porting codebases written an obsolete or deprecated language to modern one or re- implementing existing projects in one's preferred programming language. Exist- ing data-driven approaches either require large amounts of training data or neglect significant characteristies of programs. In this paper, we present IPTR for interac- tive code translation retrieval from Big Code. IPTR uses a novel code representa- tion technique that encodes structural characteristics of program and predictive transformation technique to transform the representation into the target program- ming language. The transformed representation is used for code retrieval from Big Code. With our suceinct representation, the user can easily update and correct the returned results improve the retrieval process. Our experiments show that IPTR outperforms supervised baselines in terms of program accuracy.
INTRODUCTION
Numerous programs are being developed and released online. To port codebases written in obsolete or deprecated languages to modern one (Lachaux et al. (0200), or to further study, reproduce and apply them on various platforms, these programs require corresponding versions in different lan- guages. In cases when developers do not make the translation efforts themselves, third-party users have to manually translate the software to their needed language, which is time consuming and error prone because they have to be the expert in both languages. Also, hard-wired cross- rross-language com- pilers still require heavy human intervention for adaptation and are limited between some specified types of programming. In this paper, discuss the potentials of data-driven methods that exploit existing big code resources to support code translation. The abundance of open source programs on the internet provides opportunities for new applications, such as workflow generation (Derakhshan ett al. 2020), data preparation YYan & He (020), and transformation retrieval (Yan & He 2018)
Code translation another application that is gaining attention (Lachaux et al. 2020).
Data-driven program translation. Inspired by natural language translation, one line of approaches trains translation model from large amounts of code data either in a supervised (Nguyen et al.
2013 2015 Chen et al., 2018) or weaakly-uuperviedd fashion (Lachaux et al., 2020). Supervised approaches require parallel dataser to train the translation model. In parallel datasets, programs in different languages are considered to be "semantically aligned". Obtaining the parallel datasets in programming languages is hard because the translations have be handwritten most of the time. Besides massive human efforts, it is also a tricky problem to extract general textual features that apply to every programming language. A recent supervi- method (Lachaux et al. 2020) pretrains the translation model on the task of denoising randomly corrupted programs and optimizes the model through nack-translation. However, this method still relies on high-quality training data. Further, all these approaches directly reuse NLP approaches that neglect the special features of programming languages. Another potential approach is to use a retrieval system to obtain translation candidates directly from Big Code, which refers to well-maintained program repositories. However, existing code retrieval systems, such as Sourcerer (Linstead et al. 2009), lack the proper capabilities for code-to-con co-e search and lanss-language code retrieval. These methods ask users to give feedback on several preset metrics and questions (Wang et al.) 2014 Dietrich et al.) 2013 |arric et al. 2015 Sivaraman et al. 2019). None of these methods is tailored cross-language retrieval.

---

Under review as a conference paper at ICLR 2021
In this paper, we propose an interactive program translation retrieval system IPTR based novel and generalizable code representation that retains important code properties. The representation not only encodes textual features but structural features that can be generalized across all imperative programming languages. We further propose query transformation model based on autoencoders transform the input program representation guage. Due to the succinct form of our code representation, IPTR can adapt the original query based on user annotations. This methodology can compete with existing statistical translation models that require a large amount of training data. In short, we make the following main contributions:
representation that has properties of the target lan-
We propose IPTR, an interactive tross-language code retrieval system with program feature representation that additionally encodes code structure properties.
We further propose a novel query transformation model that learns a refined code representation in the target language before using it for retrieval. This model can be trained in an unsupervised way but also improved through active learning.
Based on our succinct code representation, propose user feedback mechanism that enables IPTR to successively improve its results.
2 SYSTEM OVERVIEW
We propose IPTR, an interactive cross language code retrieval system that supports program trans- lation on multiple programming languages.
Problem Definition, Given a piece of source program P. written in language L,, selected target language L,, and large program repository D, = (P1, P2. Pa]. the goal is find the best possible translation P. of P. in L, from Dp. The problem is to design an effective program feature representation that generalizes to many languages and can updated through user feedhack.
Our solution: IPTR The workflow of IPTR shown in Figure IPTR first constructs succinet but informative feature representation for input programs (3.1tion(311) Since the target is to identify a similar program in the target language, 1PTR then applies query transforma- tion model (QTM) to transform this representation into an estimated feature representation of the translation (Section 3}. The transforma- tion model is trained in an unsupervised manner but can also be up- dated dynamically through active learning (Section B.2.2). Finally, this new representation will be used as query to retrieve the program that addition, as an interactive
Input
program
Representation constructinn
Representation
Transfomation
model
Raiined representation Queryy
Active labeling
has similar features from the database. system, IPTR allows the user to give feedback the retrieved transla- tion (Section 4 The user can either accept the result or make correc- tions. Based on our structured and informative feature representation, IPTR can easily and quickly adapt the query based on raw user cor- rections. Then with the new query, it may identify more appropriate translation candidate in the second retrieval attempt.
Translation reltivval
Transation candidate
azcept
Figure 1: IPTR overview
PROGRAM REPRESENTATION
To retrieve promising program translation from large code database, IPTR needs effective and efficient query. Directly retrieving based on raw code is impractical. In contrast to existing methods that generate queries based on either keywords (Linstead et al. (2099) or preset metrics and questions (Martie et al. 2015). IPTR generates feature representation that effectively combines structural properties of the program and textual features. further uses a query transformation model (QTM) to generate features in the target language.
3.1 BASIC ENCODING OF PROGRAM STRUCTURE AND TEXT
Due to the special and non-trival structure of programming languages compared to natural lan- guages, we take both structural and textual features into consideration. The structural features of program be represented by its syntax tree where each tree node denotes code construct.
2

---

Under review as a conference paper at ICLR 2021
inStatement
[IStatement, (ddnritter, Idartiter))
JavaScript Program [|| dalaCxample -- 0.1) Type exemple 'eeal number";
Syntox Pansing
ExpeeiityEpression
(esiigmmenEExreesionn
Faths Extraching
[iStatement. (Ittrra, Identiforr;:
(rStatement, [ideniiferr (ttral))
(IStatement [Iherrl, Iharaly)
identifier
llteral
[dentifier
literal
(equaltyExpressinn [Idontifeer, litelly)
(asdggmemnExo ressiloon (Identifer, Iteral)
data@xemple
0.7
ape_exomple
'eai nomber'
Figure 2: Program representation
One can also use control flow graph (CFG) that captures the dependence between code basic blocks and procedures represent the code behavior. However, our goal is support program transla- tion for any granularity of program, code behavior is hard to measure when the code fragment is not a complete code block. Further CFG is much more difficult to construct than syntax trees. As syntactical similarity also plays significant role, we pick syntax trees as the basis of our represen- tation to also capture the low-level syntactic structure within code blocks. One can also combine CFG and syntax trees to reserve more information. However this would trade-off the simplicity of the retrieval query. We show that with analysis of the static AST features we already achieve high program accuracy. The syntax tree can be either concrete syntax tree (CST) or abstract syntax tree (AST) (Alon et al., 2018 2019] Chen et al. 2018). CST depicts nodes with complete struc- tural information, such as all the tokens in the code while the AST is more abstract and only displays structural or content- related details. For more details on CSTs and ASTs, we provide examples in the Appendix A.2
As the CST is quite verbase and the AST does not generalize to multiple languages, we fall back on the low-level CST as basis, and take the philosophy of AST as an inspiration to construct unified abstract representation. Specifically, IPTR first simplifies the CST by removing semantically repetitive nodes such as equalityExpres
and == and intermediate nodes and generates more simplified but still informative syntax tree. As shown in Figure 2 piece of Javascript code is first converted into a simplified syntax tree (details of simplifying CST are shown in Appendix Inspired by prior work (Alon et al., 2018). 1PTR further simplifies it by extracting a set of one- dimensional paths that connect the program elements of the two-dimensional tree. Our method abstracts these paths by only keeping the three nodes that enclose the most critical information on a path: for each pair of leaf-nodes in the CST, IPTR keeps the nodes with their values and the root node of the statement. It drops all other intermediate nodes this path. The extracted abstract paths of the JavaScript Program example are also shown in Figure 2 By simply matching nodes with similar names from different languages, 1PTR can classify these paths into different types and generalize them to multiple programming languages. Thus, the structural feature of a program can be succinetly represented by the different types of paths pi, P2. P/ it contains.
To additionally incorporate textual features, IPTR treats extracted paths as plain text and keeps all the text tokens (1, 12.-. appearing in each extracted path generate text features. Fi- nally, we can use these tokens (textual features) together with different types of paths (struc- tural features) illustrated above as feature elements of programs. For each input program, we generate feature vector consisting of the feature element frequencies in that program. Let f be the occurrence frequency of feature elements, then the final feature representation will be J/a...... Jt.. In our experiment, the number of different
[Jey Lea. J=,,] [pp Spmp. path types is about 5,000 on average for each language pair. The number of tokens can be restricted by
max_vocal to trade off effectiveness and efficiency. Our default setting is 10,000. These elements can also be used index keys to filter the database, and by calculating the similarity of feature vectors IPTR can retrieve the most similar program in target language, which is the translation candidate. Noted that we simplify the feature representation presented here due to space limitation. In actual IPTR, we also consider the dependencies between structural and textual features. In contrast to existing work (Cheng et al. 2016 2017) that suggests to extract all the text from a program and treat it in isolation, IPTR considers textual features in strong dependency with the structural features to leverage more context from the structure. For the sake of simplification and to avoid computation overhead, 1PTR only processes text that appears in the extracted paths. In this way, 1PTR can run an efficient hierarchical retrieval mechanism: first calculating the structural similarity, then comparing the textual similarity in each common path type. We show the details of our comprehensive feature representation in Appendix A.3
3

---

Under review as a conference paper at ICLR 2021
3.2 QUERY TRANSFORMATION MODEL
We can directly use the feature vector described in Section 3.1 as a query to retrieve translation can- didate. However, the feature vector will still fail to accommodate some cross languages hurdles. For example, C# supports goto statement while its Java translation has to use break or cont cuee with label instead of goto. In this case, IPTR cannot directly use the features of C# program to retrieve its Java translation. The result can be improved if the retrieval can be conducted based on the features of the translation. While translation of complete program is our original problem and hard to solve, we propose a query transformation model (QTM) that solves a smaller problem. QTM transforms the original query (feature vector of the imput program) into an optimized query (esti- mated feature vector of the translation). While the estimated feature vector is not a full program is better estimate for finding the correct program in the database.
3.2.1 MODEL DESCRIPTION
As shown on the left of Figure 3 in the on- line phase. to translate from language Lg to lan-
Online
Offline
guage Lt iPTR extracts the features as F, of
Task: Translate from L, to
Program Database
a program from source language T., and feeds into the QTM. the QTM, a one layer en- coder (red in Figure 3 maps the original feature vector to a low dimensional latent space and pro- duces shorter hidden vector H. Then // is re- constructed to the estimated translation feature vector F2 by one layer decoder (yellow in Fig- ure 3) Fg will be used as query to retrieve po-
Query Transformation Model meme -a
porgram
For each 4 In Database train
Input:
Encoder,
from Af
Enceddrt
Decoder
(Encoder aow Decoder
Decoder,
AEra (Encoderay. Decader,
tential translation in target language T.,
outpot: F, Manm
from AE,
Decodar,
Myyon (Encoderr, sytuos pecoder
Since there is no available training data for QTM, we leverage an unsupervised method au- toencoder (AE) to train the encoder and the de- coder. An AB an encoder-decoder that aims to reproduce its input. That is, it encodes the input to hidden vector, then reconstructs the input from this hidden vector. In IPTR, we exploit this property to learn the weights of the encoders and decoders separately. As shown on the right of Figure in the offline phase, for each programming language T. in the database IPTR trains a separate A AE; on all programs that are written in this language. Thus it obtains a pair of Encoder; and Decoder; for each programming language. For the actual translation task, we combine the appropriate encoder and decoders depending on the source and target language of a translation task. We can see in the QTM in Figure 3 it selects the encoder Encoder, of the source language L, from AE,, which is trained in offline phase, to transform Fg into the hidden layer representation. And it picks the trained decoder Decoder, of the target language L, from AE, to estimate the F,. This way, can build a pretrained model with an encoder that learns significant information from the feature vector of the input program and decoder that can generate features of its translation.
Figure 3: Query transformation model (QTM)
3.2.2 SAMPLING STRATEGIES FOR ACTIVE LEARNING
To make the QTM more suitable for specific real-world code data, we also design an active learning mechanism to enable the user to fine-tune the mixed encoder model. As shown in Figure [ during each translation retrieval, the most useful programs are selected with sampling strategy for user la- beling. IPTR will ask the user to give the correct translation as its label. We propose an aggregation of four sampling strategies to approximately measure the informativeness of the given program as following (We take program A that contains set of feature elements {E1, eg....} as example):
1. Coverage sampling. It picks the programs that cover more different feature elements, which may reveal more information. We consider over half of the average feature elements amount as high coverage. In a database, the average amount of feature elements contained in a program is À and program A contains more than X/2 different feature elements, it is qualified sample.
2. Rarity sampling, Rarity sampling considers programs with rare feature elements, i.c. programs that contain features that appear in at most e% of the database programs. For example, if feature
4

---

Under review as a conference paper at ICLR 2021
element e1 from program A appears in x% (z e) of the database programs, A is qualified sample.
3. Uncertainty sampling. Uncertainty sampling picks retrieved programs with low certainty, i.e., lower similarity score than 75%. For example, if program B the top retrieved translation of program A, but their similarity score is 50%, which is lower than 75%, program A is qualified sample.
Random sampling. It randomly selects program for labeling (Zhu al. 2007
After running the above four sampling strategies for an input program, IPTR employs query-by- committee method aggregate the results. With this approach, we make sure to have incorporated a diverse set of characteristics that might be relevant for sampling. The final decision is made by selecting program data where the largest disagreement occurs among those sampling strategies. The level of disagreement of a program can be measured by vote entropy VE (Dagan & Engelson 1995
VE(x) = V(x) N.
V(x)
N V())() N
N₂ (x)) N
log
log
N
((x) is the number of sampling strategies that select/vote as valuable sample. N₂ is the number of sampling strategies, which equals 5 in our case. Programs with higher vote entropy are returned as samples. After task was successfully completed new label can be stored for the QTM at hand.
4 QUERY ADAPTION BASED ON USER FEEDBACK
IPTR can also adapt query based on user's feedback on returned results to improve the query. Ex- isting interactive code retrieval methods ask the user to give feedback on the preset metrics and ques- tions (Nie et al. 2016) Sivaraman et al. (211)). On contrary, IPTR directly leverages user's s correc- tions to the retrieved results feedback circumventing the efforts needed to pose the right questions. The simplest form of using the user corrections is to just use the feature vector of the corrected pro- gram. However, we also can make use of the fact that user corrections lead to manually curated fea- tures. To reflect this in feature representation, we extend it with a weighting scheme. As shown in Figure 4
we obtain a new feature representation R, where each element consists a feature ele- ment fe, and its weight wy. The initial weights uniform. After the makes corrections the result, IPTR featurizes the correction the same way and compares it with the original code to gener- ate the weights. We classify corrections into three categories and the weights are tuned accordingly:
Fenture Extraction
Weights
Inttialiation
Emphasize. If the correction increases feature element fe; that already exists in R, the weight of fe, will be increased. Add. the correction adds feature element fe, that does not exist in R, fe, and its initial weight w; will be added into R. Delete. If the correction decreases feature element fe, in R. its weight w; will also be decreased.
Weighted Representation
[feew.)
update
(few)))
R==
[feow.)
corr1: emphasize feyE responsel: W. corr2: add fe, &. --> response2: (L) corr3: fe -> response wi
Then the feature representation is updated and used for a new round of retrieval. When calculating the similarity between query and instance in database, we consider an instance is bet- ter if it has higher similarity in features with high weights. We Figure 4: User feedback mechanism show an example of user feedback in Appendix A.4
5 EXPERIMENTS
In this section, we first compare different variations of IPTR with existing work from data-driven program translation and code search. We further evaluate IPTR's performance for different lan- guages and discuss the user feedback. In addition, we also conduct an experiment on large real- world dataset without ground truth to show the practicality of iPTR. This experiment is shown in Appendix[A.3 All our experiments are conducted on a PC with an Intel Xeon E5-2650 v2 2.60GHz CPU and NVIDIA Tesla K40m GPU.
5

---

Under review as a conference paper at ICLR 2021
Dataset. Datasets with ground truth are generally scarce. We our experiments on the parallel dataset used in previous work with ground truth on Java to C# translations (Nguyen et al. 2013 2013 Chen et al. 2018) and the dataset GeekforGeeks provided by Lachaux et al. (2020) The former was built based on several authoritative open source projects, such as Lucene, ANTIR, and JGit, which have both official Java and C# versions that have been correctly aligned to 39.797
matched methods. The latter gathered and aligned 698 coding problems and their solutions in several programming languages. We train the AEs of QTM on database generated from the Public Git Archive (PGA) database with more than 260,000 top bookmarked Git repositories (web) (). We cleaned the dataset beforehand by gradually removing duplicates at file level, and files that cannot be successfully parsed due to format, errors, version compatibility or other issues. Finally, we obtain a dataset with the size of 260GB. We split all the files into methods or functions.
Metric. To measure effectiveness, we use program accuracy as proposed by prior work (Nguyen al. 2015; Chen al., 2018). Each time, we pick one program from the dataset and try retrieve its translation from the database. Program accuracy is the percentage of the retrieved translations that are exactly the same as the ground truth in the dataset. Note that, it is an underestimation of the true accuracy based on semantic equivalence. That is. program and its translation should have the same functionalityy but minor differences, such as variable names and writing habits, can be tolerated. Since we use a parallel dataset for evaluation, we can directly use the ground truth.
5.1 COMPARISON WITH BASELINES
We compare 1PTR with four program translation baselines (1pSMT (Nguyen et al. 2013),
mppSMT (Nguyen et al., 2015), Tree2tree (Chen et al. 2018), TransCoder (Lachaux et al. 20200, )
which use a translation model to generate the results. The supervised baselines use 90% matched method pairs as training data predict the translations for the rest of programs. We used the openly available implementation of Tree2tree. For TransCoder, we follow their method to pre-train the lross-language model on the Public Git Archive dataset (30GB of Java and C# data). For the pro- gram translation baselines IpSMT, mmpSMT and Tree2tree, we report the results from their work on the same dataset as their code and configurations are not available. Finally, we report the results of two code search baselines Sourcerer (Linstead et al. 2009 and CodeHow (Lv et al. 2015)).
We generate different versions of iPTR with variations in interaction and feature representation:
Interative versions: We discuss three different interactive versions of IPTR. All of which use the same feature representation and the QTM module. AL#FB the full fledges interactive system with active learning for QTM and user feedback as described in Table [ IPTR AL and 1PTRFg only use the active learning component or the user feedback component, respectively. Representions: PTR refers to the basic program translation retrieval module with the feature vector of input program as query (Section B.Ip. PTR+QTM uses the QTM module without active learning on top of PTR. We further analyze PTRWoRD2VEC and PTRcoDEZVEC as two feature representations variations of PTR based on Code2vee (Alon et al., 2019) and Word2vee (Mikolov et al. 2013a), respectively.
Table T shows the results and the degree of supervision. We observe that holistic IPTR with at most one user correction per task and optimized QTM outperforms all the baselines. The improvement in program accuracy is ranges from 19.5% to 65.5%. As expected, the code search baselines perform poorly because they are designed for retrieval with more accurate and detailed input than raw pro- gram code. The results of partial components of IPTR are also encouraging. PTR and PTR+QTM, which do not leverage any supervision, outperform the TreeZtree, which shows the effectiveness of our program feature representation and translation retrieval methodology. Leveraging the QTM successfully improves the result by 7.5% showing that generating features the target language more promising than using features of the source language. If the QTM is trained by active learning, the accuracy can increase by 8.5% (IPTRAL)). The table also shows that the Word2vec and Code2vec variants of PTR cannot outperform PTR with our proposed feature representation based on structural features. Word2vec is designed for natural languages so that it can not capture the special features of programming languages. Although Code2vec is designed specifically to represent code, their model can only be trained within a same programming language, which makes it less suitable for tross-language similarity comparisons.

---

Under review as a conference paper at ICLR 2021
Table Comparison of different methods on program accuracy (PA) and supervision extent
Genre
Method
Description
PA
Supervision Extent
Data-driven program translation
1pSMT
Phrase-based SMT
24.1%
fully supervised
multi-phase
mmpSMT
phrase-based SMT
41.7%
Tree2tree
tree-to-tree neural networks
70.1%
weakly-supervise neural translation
TransCoder
49.9%
weakly supervised
Lucene-based code search, free-text queries
Code search
Sourcerer
13.5%
system
CodeHow
free-text queries
13.5%
PTR
71.1%
no Jabels (directly retrieye translation with input)
Variations of IPTR
PTR woRDevvv
word2vec as queries
67.7%
PTReoDEIVEC
code2vec queries
63.4%
PTR+QTM
78.6%
IPTRAl
87.1%
80 labels for QTM
1PTR
79.3%
most correction per task
IPTRALFFR
the full system
89.6%
combination 1PTRAL and iPTRra
Table 2: Comparing program accuracy with Transcoder GeeksforGeeks dataset
C++-Java
C++-Python Java-C++ Java-Python
Python-C++
Python-Java
Transcoder
3.1%
6.7%
24.7%
3.7%
4.9%
0.8%
PTR
69.2%
65.3%
70.9%
59.3%
55.4%
54.2%
PTR+OTM
76.6%
74.2%
78.1%
68.1%
59 2%
59.6%
APTRAL AL 87.2%
79.5%
84.8%
72.5%
66. 2%
68.8%
IPTRED 75.9%
71.3%
76.5%
64.2%
59.5%
63.3%
IPTRAL-FB 84.8%
83.0%
90.5%
77.5%
67.8%
68.1%
We further explored the supervision impact on IPTR. this experiment, we let the user give at most one correction to each retrieved task. Without optimizing the QTM, 1PTR slightly improves on PTR+QTM. The same improvement rate can be observed when comparing 1PTRAL+FB to IPTRAL suggesting that AL and FB have independent influence on the results. Compared to the fully super- vised methods IpSMT, mmpSMT and Tree2tree, 1PTR leverages very limited human supervision to achieve better results. In the first retrieval round, the user does not make correction to any wrong results, IPTR achieves 87. % accuracy with only 80 labels for QTM. Also the reproduced weakly supervised approach TransCoder does not achieve better results than IPTR. Although some of the performance loss can be contributed to the fact that our training data is not as large as they report in their paper, we also observed that their model often generates invalid translations with regard to grammar. For example. it often mistakes the input type of a function for different languages. This phenomenon is also acknowledged in their own paper and can be attributed to the fact that only textual features have been used. IPTR avoids this problem by reusing existing code.
5.2 COMPARISON ON MULTIPLE PROGRAMMING LANGUAGES
We further compare IPTR and the state-of-the-art method Transcoder on the GeeksforGeeks bench- mark provided by Lachaux et al. (020)). It contains groundtruth for Java, Python, and C++. Ta- ble g shows that the program accuracy of 1PTR is significantly higher than Transcoder on their own datasets also reported in their own paper. This is because of two reasons: (1) Our targets are different- Transcoder outputs machine-generated translations while 1PTR directly retrieves ex- isting programs as translation candidates, which makes IPTR always output syntactically correct programs. (2) Transcoder generally treats programming languages plain text and aims to generate semantically similar programs. does not encode the non-trival syntactical features of program- ming languages in their model. Thus, it has difficulties in generating results exactly the same as the ground truth. This experiment shows that IPTR is generalizable for multiple languages including dynamic languages, such Python. Further it shows that it is easier to find translation candidates for grammatically similar programming languages, such as C++ and Java compared to Python as a high-level dynamic language.
7

---

Under review as a conference paper at ICLR 2021
5.3 INFLUENCE OF USER CORRECTIONS
In Table I we showed the influence of a single user correction on the result. We further investigate the number of user corrections required in order to retrieve the correct translations. We simulate users with the ground truth in our parallel dataset and for
Number completed tasks
1000
100
Program accuracy/ 9%
number of translation tasks
program accuracy
600-
each returned result we fix the first different line between true result and returned result.
200
shows the number of required user correc- tions to obtain the correct result for all 5.134 trans- lation tasks that failed the first retrieval round and the improvement in the overall accuracy. We observe that in most cases, 1PTR only requires a single user correction to successfully complete the translation task. About 98% of the failed retrieval tasks can succeed after 10 user corrections. Con- sidering the average length of an input programs is 168 lines, we can conclude that with small amounts of user feedback the accuracy of IPTR can be significantly improved. Note that, we might even achieve better results if we do not restrict the users to fix the first difference each time. real user might fix more significant errors that improve the success rate of the retrieval task. The average response time of 1PTR to each user feedback 9.4ms using an efficient index structure.
10 15
20
25
30
35
Number of maximum allowed user corrections
Figure 5: Required amount of user feedback
6 RELATED WORK
Data-Driven program translation. Existing research mainly focus on building a translation model. Nguyen et al. (2013) directly applied the phrase-based statistical machine translation (SMT) model on the lexemes of source code to translate Java code to C#. In their follow-up work, they develop a multi-phase, phrase- based SMT method that infers and applies both structure and API mapping rules (Nguyen et al. 2015). But they are limited to languages that are similar on either structural or textual level, such as Ct Chen et al.] (2018) binarize the code tree using left-Child right-sibling and translate code by an LSTM-based encoder-decoder model. However, all these approaches re- quire large parallel dataset for training. In contrast to them, a recent work also proposes weakly- supervised system TransCoder (Lachaux et al., 2020). They first train a cross language model for programming languages through predicting randomly masked words task, then acquire a pre- trained translation model from denoising randomly corrupted program task. Finally they improve this model through back translation. Although they do not need parallel translation data, this transfer learning method highly relies on the similarity of the data for pre-trained. And this approach is originally designed for natural language processing, which makes produce results neglecting the features of programming languages. Our code retrieval system can support program translation without paral- lel dataset and violating the strict grammar rules of programming languages, which non-trivial problem in all the above methods.
Interactive program retrieval. In addition to the Sourcerer (Linstead et al. 2009 and StackOver- flow search engines that were discussed in the introduction, there are some other work incorporating each result and re- order the rest ranking list. Nie et al. (2016) extract relevant feedback from StackOverflow for the initial query and reformulate it using Rocchio expansion. |Dietrich et al. (2013) utilize novel form of association rule mining to learn a set of query transformation rules from user feedback and use them to improve the efficacy of future queries. |Martie et al.) (2015) propose CodeExchange that uses the previous results to formulate the query so that user can find new results based on the previ- ous characteristics. [iivaraman et al. (2019) propose an active learning system ALICE to iteratively refin query based on positive or negative labels. All these approaches are designed for the mono- language setting and their queries are in natural language. IPTR performs cross language retrieval and directly uses raw code as input. Moreover, their method asks the user to feedback on the preset metrics and questions so that the result improvement largely depends on the subjective design from the developer. Due to the different understandings of different users, the feedback cannot be evalu- ated fairly and uniformly. 1PTR directly uses user's corrections to the retrieved results as feedback and integrates active learning based query transformation model to refine the query.
user interaction. [Wang et al. (2014) refine query based user's feedback
8

---

Under review as a conference paper at ICLR 2021
Program representation. By constructing program representations, of data processing to a wide range of programming langunggg tasks including program translation and code search. Kamiya et al. (2002) and Allamanis et al. (2016) treat a program as natural language and use the sequence of tokens as representation to detect code clones and summarize code. |Allamanis) et al.] (2018) present a Gated Graph Neural Network in which program elements represented by graph nodes and their semantic relations are edges in the graph to predict variable name and select correct variable. These methods rely on semantic knowledge, which requires more expert analysis and is not generalizable as the semantic analyses need to be implemented differently for every language,
enable the application
recent approach uses paths program's 's abstract syntax trees (AST) as code representation to predict program properties such names or expression types (Alon et al., 2018). And they further leverage tree-based neural network to encode these paths and generate more abstract representations (Alon et al. 2019). Yin et al. employ neural networks to express source code edits (Yin et al. 2019). However, these methods are not designed for rettival of them too simple to effectively represent program's 's features (Kamiya et al., 2002 Allamanis et al. 2016). Others are too abstract for users to understand intuitively, which makes it impossible easily interact with users (Allamanis al. 2018 Alon et al., 2018 2019 Yin et al.,, 2019). The program representation we use as query is more structured. It is not only informative for translation retrieval but also succinct for user interaction.
retrieval. Some
CONCLUSION
We presented IPTR, novel interactive translation retrieval system that can support program trans- lation. Different from traditional data-driven methods that train translation model, 1PTR directly searches for the most suitable translation candidate in existing code database. We propose pro- gram representation based on query transformation model that can learn a succinct but informative feature vector to retrieve the translation of input raw program. 1PTR casily adapt the rep- resentation to user corrections for interactive retricval improvements. Our experiments show that IPTR outperforms existing solutions in terms of effectiveness and requires no training data.
Future work. One important direction to design a convenient user interface that allows user to make relevant corrections to translation suggestions and enable the system to converge faster to the desired result. Another possible direction to explore is other forms of user interaction that do not require the user to provide corrections in the target language.
REFERENCES
code2vec. website, a. htttpp//ggttub.ccc////eeeaa///// [Online; accessed 15- May-2020].
Antlr. website, b.
(ww.:orggg//Ooiiiiceeeecesee//e/ 15-
Public git archive. website, c. PublicsitArchive [Online; accessed 15 M0-M0.
Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. A convolutional attention network for ex- treme summarization of source code. In Proceedingy the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2091-2100. JMLR. 2016
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path-based representation for predicting program properties. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018, pp. 404-419. ACM, 2018
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: learning distributed represen- tations of code. Proc. ACM Program. Lang., 3(POPL)::0:144000) 2019.

---

Under review as a conference paper at ICLR 2021
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to to-ere neural networks for program translation. In 6th International Conference on Learning Representations. ICLR 2018, Vancouver. BC, Canada, April 30 May 2018, Workshop Track Proceedings. OpenReview.net, 2018.
Xiao Cheng, Zhiming Peng, Lingxiao Jiang, Hao Zhong, Haibo Yu, and Jianjun Zhao. Mining vision histories to detect cross-language clones without intermediates. In Proceedings of the 31 IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, Singapore, September 3-7, 2016, Pp. 696-701. ACM, 2016.
Xiao Cheng, Zhiming Peng, Lingxiao Jiang, Hao Zhong, Haibo Yu, and Jianjun Zhao. CLCMiner: Detecting cross-language clones without intermediates. IEICE Trans. Inf. Syst., 100-D(2):273- 284, 2017.
Ido Dagan and Sean P. Engelson. Committee-based sampling for training probabilistic classifiers. In Machine Learning, Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, USA, July 9-12, 1995, pp. 150-157, Morgan Kaufmann, 1995.
Behrouz Derakhshan, Alireza Rezaei Mahdiraji. Ziawasch Abedjan, Tilmann Rabl, and Volker Markl. Optimizing machine learning workloads in collaborative environments. In Proceedings of the 2020 International Conference on Management of Data, SIGMOD Conference 2020, online conference [Portland, OR, USA], June 14-19, 2020, pp. 1701-1716. ACM, 2020.
Jacob Devlin, Ming-Woo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Association for Computational Linguistics, 2019.
Short Papers), pp. 4171-4188
Timothy Dietrich, Jane Cleland-Huang, and Yonghee Shin. Learning effective query transformations for enhanced requirements trace retrieval. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA, USA, November 11-15, 2013, pp. 586-591. IEEE, 2013.
Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. Cefinder: A multilinguistic token-based code clone detection system for large scale source code. IEEE Trans. Software Eng., 28(7):654- 670, 2002.
Maric-Anne Lachaux. Baptiste Rozière, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. CoRR, Abs/20006005111 2020.
Erik Linstead, Sushil Krishna Bajracharya, Trung Chi Ngo, Paul Rigor, Cristina Videira Lopes, and Pierre Baldi. Sourcerer: mining and searching internet-scale software repositories. Data Min. Knowl. Discov,, 18(2):00-333, 2009.
Fei Lv, Hongyu Zhang, Gan-Guang Lou, Shaowei Wang. Dongmei Zhang, and Jianjun Zhao. Code- how: Effective code search based on API understanding and extended boolean model (E). In 30th JEEE/ACM International Conference on Automated Software Engineering. ASE 2015, Lincoln, NE, USA, November 9313, 2015, pp. 260-270. IEEE Computer Society, 2015.
Lee Martie, Thomas D. LaToza, and André der Hoek. Codeexchange: Supporting reformulation of internet-scale code queries in context (T). In 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, November 9-13, 2015, pp. 24- 35. IEEE Computer Society, 2015.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In 1sl International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Informa- fion Processing Systems 26: 27th Annual Conference Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 3111-3119, 2013b.
10

---

Under review as a conference paper at ICLR 2021
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. Lexical statistical machine translation for language migration. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium the Foundations of Software Engineering, ESEC/FSE Saint Saint Petersburg, Russian Federation, August 18-26, 2013, pp. 651-654. ACM, 2013.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. Divide-and Slvide-and-come -nn-cuu approach for multi- mulse- statistical migration for source code (T). 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, November 9-13, 2015, pp- 585-596. TEEE Computer Society, 2015.
Liming Nie, He Jiang, Zhilei Ren, Zeyi Sun, and Xiaochen Li. Query expansion based on crowd knowledge for code search. IEEE Trans. Serv. Comput., 95):771-783, 2016.
Aishwarya Sivaraman, Tianyi Zhang, Guy Van den Broeck, and Miryung Kim. Active inductive logic programming for code search. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 292-303. IEEE / ACM, 2019.
Shaowei Wang, David Lo., and Lingxiao Jiang. Active code search: incorporating user feedback to improve code search relevance. In ACM/IEEE International Conference on Automated Software Engineering, ASE '4, Vasteras, Sweden September 15 19, 2014, pp. 677-682. ACM, 2014.
Cong Yan and Yeye He. Synthesizing type-detection logic for rich semantic data types using open- source code. In Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018, Houston, TX, USA, June 10-15, 2018, pp. 35-50. ACM, 2018.
Cong Yan and Yeye He. Auto-suggest: sooriingtt-recommmend data preparation steps using data science notebooks. In Proceedings of the 2020 International Conference Management of Data, SIGMOD Conference 2020, online conference [Portland, OR, USA], June 14-19, 2020. pp. 1539- 1554. ACM, 2020.
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Learning represent edits. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Xingquan Zhu, Peng Zhang, Xiaodong Lin, and Yong Shi. Active learning from data streams. In Proceedings of the 7th IEEE International Conference Data Mining (ICDM 2007), October 28-31, 2007, Omaha, Nebraska, USA, pp. 757-762, 2007.
A APPENDIX
A.1 CST AND AST
JavaScript Program: dalaExample
CST
AST equal
equalityExpression
sinleExpressionn
singleExpressiin
doteExample
Identifter
llteral
deteEremple
0.1
Figure 6: CST and AST of simple JavaScript program
The structure of a program can be described by syntax tree where each tree node denotes a con- struct occurring in the code, e.g., if_stmt Python denotes an if construct. The tree can be either comprehensive CST or a more abstract AST. CST is tree representation of the gram- mar (rules how the program should be written). It represents the source program exactly in parsed form. In other words, it defines the way programs look like to the programmer. There is example in Figure 6 The CST of the simple JavaScript program is verbose with all the detail information about parsing the code. It keeps all the tokens in the program and their types, such 1 literal

---

Under review as a conference paper at ICLR 2021
JavaScript Program
if daaaaExampee 0.1) (type example real number"
Parsing
CST
statement
iSttatement
expressionSequence
statement
angloExpression
block
equalityExoression
statementfitt
singleExpreessin
singleExpression
Identiet
expresskontttemmen
cemppi
0.1
SxpuessienSequence
assigmmentExpeeesion
singloExoression
literal
identifier
'reel number
type_exeeee example
Pruning
Pruned Tree
ifStatement
equalityExpression
assignmenEExpression
Identifier
literal
identifier
literal
detaExample
0.1
type_ app__exapple
'real number'
Figure 7: Generating simplified syntax tree
and identifier. And it reveals all the grammar rules, such as equalityExpressstttettttt singleExpression
singleExpressione ion. On contrary, AST is of the abstract syntactic structure of source code. Each node of the tree denotes a construct occur- ring in the source code. The syntax "abstract" the sense that it does not represent every detail appearing in the real syntax, but rather just the structural or content-related details. It defines the way the programs look like to the evaluator/compiler. As shown in Figure the AST directly shows the equal structure without any detail information. It discards the intermediate structural information that would not contribute to semantics.
representation
A.2 GENERATING SIMPLIFIED SYNTAX TREE.
IPTR first employs left-to-right parser, such ANTLR web (b), to parse the source code and generate the original CST. 7 the CST of a raw JavaSeript program. However, as shown in Figure 7 the original JavaScript code is simple while its CST is comparably intricate with all the details preserved. The large amount of nodes and branches will raise the computation complexity while having little contribution to the representation. For example, the node type i ifStateees already specifically indicates the existence of an IF expression. Therefore its parent node type stat ement
necessary as it does not reveal more information. On syntax tree, all the nodes are already enclosed by their parent node. Thus, nodes of parentheses are not needed to reveal en- closure relationships. Further, node already explains the content. Thus, IPTR follows two rules to prune the CST: (1) Node types that only have one child node are discarded, unless this child node represents specific content in the source code (Italics in Figure 77. For example, statemed ement and s carded, while i dentifies and literal are retained. (2) If the parent node of a terminal node has more than one child node, this terminal node is removed. This is the case for symbols, such as and ==, The pruned tree in Figure 7 is much simpler than the original CST and retains all major structural features.
not necessary when node type equa tyyyxpresssss on
are dis-
12

---

Under review as a conference paper at ICLR 2021
Table
The representation of the program from Figure
Path type
Freq.
Text statisties
Pi
""data": "1""mmappxl
11:"pee": "exxampp"""""
P2
2
"0.1":1, "rr"al""lll"" 11 "1"mnmuberr"
P3
("O..":1,"reall" 1:, 11numberr""""
Pi
Pb
{"type" :1," "exxmmlee"; :1,, "real":1, "number":1]
A.3 COMPREHENSIVE PROGRAM REPRESENTATION
In our comprchensive program representation, we not only consider the structural and textual fea- tures but also their dependencies.
Initially, IPTR regards the text appearing in each extracted path in structural features as natural language text. The words in the text are tokenized and stemmed accordingly. Our tokenization process also considers camel case, spaces, and underlines to accommodate code-specific language. Further, IPTR does not remove and tokenize numeric values, such as hard-coded floating points and integers, as they might be integral to the program representation. 1PTR vectorizes these generated tokens based on the Bag-of-words model (BoW). One could also resort to more sophisticated em- beddings, such as word2vec (W2V) (Mikolov et al., 2013b). However, in programming languages the structural features are more important than the textual ones and word order can be ignored to accommodate different programming styles. Besides, we only compare text for each single path type, which means the amount of words is very few for each similarity calculation. Same reasons hold for more complex language model like BERT (Devlin et al.] 2019). BoW is sufficient for this process. In our experiment, W2V does not show worthwhile improvements, but rather introduces extra training time for building the language model.
To avoid repeated computation for every new input and improve the efficiency, IPTR splits the process of building BoW models. The basic part of this process is generating word statistics. 1PTR counts the words of each candidate program and stores the results during the offline phase. In the online phase, IPTR only needs run word statistics on the input program and build the BoW model.
Table the final comprehensive representation of the program example from Figure 2 con- sists of three components. First is the list of path types that appear in this program. There is a look-up table in IPTR to maintain all types of paths so that each path from a program is described by its path {identifies
type ID P.. as shown Table It should be noted that path { affftatemmen literal]] and path {i lifstat ement, {literal, identifier) in Figure are regarded as the same path type P3 because the order of end nodes not relevant. The second component of the feature representation is the frequency of different path types in program. The third compo- nent contains the structure-dependent textual features. IPTR integrates the information of relative position of text and structure into the feature representation. As shown in Table 3 the text statistics with respect to each path type are stored in the final representation.
A.4 EXAMPLE OF USER FREDBACK
Figure 8 is an example of user feedback module. The input code is an Ackermann function in C++. However, IPTR retrieved a greatest common divisor function as its Python translation. The ground truth is shown in Figure 8 in blue. The possible reasons are the variable names are different in the ground truth and the weight of each feature element is not assigned properly. As soon as user corrects the first wrong line (change return n to return n+1), IPTR construts feature representation for the correted code. Then IPTR compares it with the feature representation of the input code and summarizes user's corrections. Based on this, IPTR update the query. In the example in Figure 8 user's correction emphasizes the path type (identifier, Iiteral]] and token so that the weight of these two feature elements will be increased accordingly. Finally, with the updated query, IPTR will run With higher weights on structural feature and token 1 and lower weights on variable names, the ground truth will be more likely to be retrieved, and in this example it does
round of retrieval.
13

---

Under review as a conference paper at ICLR 2021
Ground fruth
def funolaa bj: i 0:
retum
Inpor
if 0:
ursigned int funcqunsigned unsigned E (m 0){
return functe-1 1, 1)
return funcja-1, functa, b-1))
return n 1:
Retrivved
Conrecte code
O return tunc 10m 1, 1):
funojmun):
functrun):
If return n
uner m=
return n+1
return func__(m 1, func 1(m, n- 13):
alae:
clse:
funcin.m8inn
luncin,m
Upda'ss repore
Emphastze element
path type: (assignmenExpeessinn, (ddentiferr llteraly: (2) Emphasize fnature element
tokon: "1"
Run new round of retrieval with updated query
Figure 8: An example of user feedback
Table 4: Statistics of the real-world dataset
Size
Files
Lines
Methods/Functions
3.8GB
280.128
75,567,192
2,023,546
A.5 EXPERIMENT ON A LARGE REAL-WORLD DATASET
Dataset. We choose the four programming languages with the most pushes on github JavaScript, Python, Java, and C++ They are also representative of programming languages with different char- acteristics. Based on the number of stargazers, we pick 1% files in these four languages from PGA to be our raw dataset. Theoretically, IPTR works for programs of any length, but considering the practical value and the intuitiveness the validation process, we aim to translate programs at method level in our experiment. Longer programs whose complete translations are not existing in the database can be translated by merging translations cach part. We split all the files into meth- ods or functions the input/output of IPTR. One limitation of PGA is that there duplicate files across different repositories. To ensure the quality of the dataset and increase the efficiency of our program translation task, we gradually remove duplicates at file level by taking hashes of these pro- grams and comparing their hashes. In addition, we also remove the data that cannot be successfully parsed due to format, errors, version compatibility or other issues. Since our approach does not require additional explanation except the program itself, we remove annotations and descriptions. Finally, we obtain a dataset with the size of 3.8GB shown in Table 4
Experiment. In this experiment, we only evaluate the basic program translation retrieval module PTR without user interaction to show the feasibility of the translation retrieval idea. Since there is no labeled data, we can not evaluate the translation accuracy automatically. As an alternative, we carry sampling inspection and manually judge the correctness of the results based the program accuracy stated above. Specifically, we take a random number x E [20, 30] and randomly sample programs from the dataset as a small input set. PTR translates each program in this set for one round of experiment. Then we conduct 10 rounds of experiments to obtain the average result. In the results, PTR returns the top 10 candidates with highest overall similarity for each input program.
To show the advantage of our representation method, we compare its results to other methods:
1. P Only the structural part of PTR.
2. PTR It extracts all the text from the program and uses BoW model to only construct representation for textual features. This is the common method used in cross language code clone detection (Cheng et al.] 2017). We follow the approach of Chhng et al. (2017). They implement a sliding window with a length of 20 tokens and compare the similarity of these token sequences.
3. P This representation is the combination of (1) and (2) without the feature dependency described in Appendix A.3
4. PTR W Same as PTR except changing the textual feature from ofg-of-words (BoW) to Word2vec (W2V). We train cross language word vector PGA database.
14

---

Under review as a conference paper at ICLR 2021
Table 5: Comparison of different representations
(a) JS as source language
Program accuracy
Mean reciprocal rank
Representation
C++ Python
Java
C++
Python
Java
PTR STRUCTURAL
53.1%
39.8%
46.7%
0.74
0.68
0.73
TTRTTNU
48.0%
23.4%
31.7%
0.36
0.42
0.57
PTR
56.2%
43.7%
49.9%
0.80
0.71
0.83
PTRoonazvee
51.0%
32.0%
40.2%
0.53
0.62
0.61
PTR WORDZVEC
61.4%
53.9%
60.2%
0.81
0.83
0.89
PTR
61.4%
52.3%
59.8%
0.88
0.84
0.89
(b) Python as source language
Program accuracy
Mean reciprocal rank
Representation
JS
C++
Java
JS
C++
Java
PTR PTR STRUCTURAL
41.8%
45.5%
43.3%
0.73
0.73
0.75
PT RTEXTUAL
39.5%
31.2%
40.9%
0.63
0.58
0.64
PTR
44.3%
46.7%
43.8%
0.71
0.73
0.76
PTR PTR ReudE2vuc
41.9%
43.4%
43.1%
0.66
0.53
0.71
PTR WORDZVEC
47.9%
54.7%
48.3% %
0.80
0.87
0.79
PTR
48.4%
54.2%
47.9%
0.79
0.85
0.77
(C) Java source language
Program accuracy
Mean reciprocal rank
Representation
JS
Python
C++
JS
Python
C++
PTR RIRUCTUAL
53.3%
47.9%
62.2%
0.84
0.76
0.85
PTR TEXTUAL
45.1%
33.9%
50.7%
0.71
0.61
0.55
PTR S+T
54.7%
51.6%
67.1%
0.88
0.78
0.86
PTReOBEIVRE
50.0%
38.6%
63.4%
0.82
0.77
0.83
PTR WORDZVBE
59.1%
57.1%
67.7%
0.87
0.84
0.88
PTR
60.6%
57.5%
69.1%
0.91
0.82
0.92
(d) C++ as source language
Program accuracy
Mean reciprocal rank
Representation
J$
Python
Java
IS
Python
Java
PTR STRUCTURAL
PTR
59.6%
55.8%
63.9%
0.78
0.76
0.82
PTR PTR TEXTUAL
54.4%
43.1%
49.4%
0.65
0.41
0.59
PTR
59.7%
57.4%
64.8%
0.83
0.79
0.86
PT R CODE2VEC
58.9%
47.5%
64.4%
0.71
0.53
0.88
PTR WORDZVEC
64.0%
63.6%
66.9%
0.89
0.87
0.91
PTR
64.0%
63.5%
67.0%
0.90
0.87
0.93
PTRCODEZVECt Code2vec the state-of-the-art code representation which transforms code syn- tax tree vector trained by neural network. For Java, directly the provided trained model to generate code vectors (webl a). For other languages, we train code vectors on PGA database. Then determine the candidates by calculating the cosine similarity.
In this experiment, we evaluate the top pick and the top 10 most likely candidates, for which we calculate the mean reciprocal rank (MRR).
Results. In Tables 5a 5d we observe that our approach can successfully translate 58.8% programs between four popular languages. Note that this is the amount of successfully translated programs. Considering PTR is positioned as a translation support system, the accuracy is considerable. Trans- lation tasks that cannot be fully automated can still be assisted by the results returned by PTR. Moreover, When the translations do not appear at top in the retrieved results, they are very likely appear higher rank in the candidates list. The mean reciprocal rank is around 0.8 and 0.9.
In general, compared to the other program representations, our novel representation performs sig- nificantly better. Unique structural features of program languages play a more important role than textual features in programs. And the dependency of structural features and textual features can effectively enrich the representation. W2V does not contribute much to the results and introduces extra model training time compared to simple BoW model. But we also observed W2V can help in some cases where the text of the original code quite different from its translation. In IPTR we re- tain BoW for of. Code2vec also considers the structure of programming languages. However,
15

---

Under review as a conference paper at ICLR 2021
their model can only be trained within same language, which makes it perform less satisfactory cross-language similarity comparison.
We can also infer from the results that it is generally casier to find translation candidates for gram- matically similar programming languages, such as C++ and Java. Finding translations for C++ as low-level basic programming language can lead to higher accuracy than finding translations for Python as high-level dynamic programming language.
16


Reviews:
[{"review_id": "yvosYpMGFed", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper seeks to solve program translation problem through code retrieval. It proposes an interactive code retrieval system, called iPTR to perform cross-language retrieval with minimum code pairs. The method extracts textual features and structural features from code and transform the features into target-language features through an autoencoder, which is trained through a uni-language manner. It further leverages users' correction to update feature representation and for new round of retrieval.\n\nThe problem is challenging, especially since the parallel data is scarse. I think the idea of utilizing both textual features and structural features is interesting. The idea of training autoencoder for each language and fuse encoder and decoder of different languages is also very interesting, though questionable as mentioned below.\n\nConcerns:\n- The major concern is that this paper does not made significant novel and technical algorithm/theory contribution, but rather like proposing a system. Therefore in my opinion the paper does not fit ICLR very well.\n\n- Due to lack of parallel training corpus for query transformation model (QTM), the paper utilizes simple uni-language autoencoder training for each language, and take the encoder and decoder from the corresponding language respectively as the transformer model. This could cause significant issue if not taking seriously. The hidden space of different language can vary drastically, and hence the encoder output is very likely to not make any sense to decoder to perform the transformation. There should be a mechanism that encourages the hidden space of different languages to the similar.\n\n-The paper tries to solve the program translation problem with retrieval method. In the experiment, what is the database to retrieve from? Is it that all training code form the database, or is it all training and testing code form the database? In practical situation, the desired translation is very likely to not exactly match the some code in the database. Therefore, I don't understand why the program accuracy of retrieval method can achieve so high if the database doesn't overlap with ground truth codes a lot.", "extra_scores": {}, "extra_text": {}}, {"review_id": "X7yshqaya6j", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a retrieval-based approach for program translation. Existing ML/DL models for program translation typically design a decoder to directly generate the code in the target language. On the contrary, this work designs iPTR, which first computes a feature representation of the target code, then retrieves the most similar code in a large code corpus. Specifically, iPTR includes a query transformation model (QTM), which generates the feature representation of the target code, given the feature representation of the source code as the input. The feature representation includes the information of tokens in the code, and the paths in the syntax trees. The idea of using the paths is similar to the code2vec paper. QTM could be trained without parallel data between source and target codes. Specifically, encoders and decoders of different programming languages could be trained in a similar way to the training of autoencoders. This idea is similar to TransCoder. Meanwhile, they show that iPTR could be trained with active learning and user feedback, where they use active learning to acquire limited parallel training data, and user feedbacks are corrections of the wrong output code. They compare their approach to existing ML/DL program translation models, as well as code retrieval systems. They show that iPTR performs better than other baselines, even without active learning and feedbacks. Not surprisingly, active learning and incorporating user feedbacks further improve the performance.\n\nProgram translation is an important application, and the authors did a good job of evaluating on existing benchmarks and comparing with different types of baseline models. Intuitively, it makes sense that retrieval could at least provide more syntactically and semantically correct code, while synthesis models may struggle with generating coherent code. However, I have a couple of questions about the assumption of the task setup and the implementation of the algorithm, and I list them below.\n\n1. It seems that the feature representation is summarized per complete code snippet for translation. Therefore, do the authors assume that the target code always exists in the retrieval corpus as a single piece of code? If this is the case, one clear limitation is that the retrieval approach could only search for existing code in the corpus, while the synthesis model could combine lines from different code snippets to construct the output code.\n\n2. How do you simulate users for active learning and user feedback? For active learning, do you ask people to annotate the ground truth output programs for input programs, and then add them into the corpus for retrieval? For user feedback, in some parts of the paper you mention that the user modifies the extracted features of the code, but sometimes you also say that the user corrects the first wrong line. Could you provide some concrete examples of how you incorporate human annotations in training and inference loops?\n\n3. More concrete examples of how to sample code for active learning could help. Now the description in Section 3.2.2 is not clear. For example, why N_s is 5 instead of 4? What is the benefit of \"selecting program data where the largest disagreement occurs among those sampling strategies\"? Do you compare among the entire training corpus?\n\n4. What are the numbers of different paths and tokens for feature representation of code? Are text tokens representing terminal nodes or non-terminal codes? For example, are variable names included in tokens, or are they simply denoted as \"identifier\"? My understanding is that the feature vectors follow the bag-of-words representation, thus I don't quite understand why this simplified representation could work better than code2vec or other more advanced program encoders. The authors say that \"theWord2vec and Code2vec variants of PTR cannot outperform PTR with our proposed feature representation based on structural features\", but I think code2vec encodes very similar structural features in a potentially more concise way.\n\nWriting comments: the paper requires proofreading, and there are a couple of typos. For example, (1) \"to approximate measure the informativeness of the given program\" on page 4: \"approximate\" should be \"approximately\". (2) \"To reflect this in out feature representaion\" on page 4, \"out\" should be \"our\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "GnfiX7LMoUr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a program translation (retrieval) method by combining  syntax tree, transformer with specific encoder/decoder, and interactive signals from user. The novelty of each used techniques is limited, but they are reasonable for the code translation task with high accuracy, even for unsupervised version without interactive signals.\n\nThe experimental results are solid and demonstrate the power of representation of the proposed variant of syntax tree. The idea of AE/AD with specific language is quite simple, but it seems that the QTM successfully transfer the representation cross different languages. The performance gain achieved by interactive signals is reasonable.\n\nQuestions:\n- there are other structural features of programming language, such as graph, which can provide rich information of the code snippets rather than syntax tree. \n- Is the proposed method a better code representation? Could it be applied to other downstream tasks after pre-training?\n- Will the authors release their code to public?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I found the setup for this paper a bit contrived. The tool is presented as a code translation tool, but it really functions more as a multi-language code search tool. The Idea is that one has a program in language A, and a database that contains the same program in language B, so one can translate from A to B simply by searching for the right program in the database. 

When evaluated as a language translation tool, it appears to outperform existing language translation schemes, but this is an unfair comparison, because iPTR is being given a database that contains the exact translation of the program in question. The performance is also compared with code search tools, but these are also apples-to-oranges comparisons, because the tools in question are operating from very high-level queries. A much more comparable baseline would be the Yogo tool  recently published in PLDI (https://dl.acm.org/doi/abs/10.1145/3385412.3386001), or for compiled languages you could compare against statistical similarity tools for binaries (https://dl.acm.org/doi/10.1145/2980983.2908126). 

The experiment in the appendix A5 is more fair to standard language translation, and it yields results that are much less impressive. I would be much more comfortable with this paper if it were written around this experiment, or alternatively if it were evaluated against a more comparable approach for semantic code search. 
Decision: Reject