日本ソフトウェア科学会第 38 回大会 (2021 年度) 講演論文集

Attempts on using syntax trees to improve
programming language translation quality bymachine learning
Dai Feng, Shigeru Chiba
While the number of programming languages is increasing, scenarios of developing programs with samefunctionality in diﬀerent programming languages are becoming frequent. Although translation between
natural languages has been extensively studied, translation between programming languages has not been
well explored yet. In this paper, we present a method to improve translation quality between programming
languages by adopting syntax tree information to represent source code. Compared with existing methods,
our method pays more attention to structural information in syntax trees, and utilizes the structural information in code representation. We show that our method could improve the translation quality between
Java code and Python code in a certain degree. We compare our results to the state-of-the-art results in theﬁeld of cross programming language translation, and show that we might be able to achieve better accuracy
in some metrics.

tion. Some researchers have already tried to use

1 Introduction

machine learning techniques for programming lan-

According to [3], the number of programming

guage translation. Work by [8] gives it a try, but

languages has been growing and the popularity ofeach programming language also keeps changing,

we believe there is space for improvement.
In this paper, we present an approach for code

indicating that programmers nowadays have more

representation.

This approach utilizes abstract

choices and switch languages more often. In fact,syntax trees for structural information. We use ourthere are many cases in which programmers haveapproach and existing approach to represent source

to reconstruct a project with another programming

code, and do experiments on programming lan-language, or implement same function in multiple

guage translation task. We present the comparison

languages. This is not only time-consuming, but

between experiment results by two approaches, andalso costly in terms of money.

make a discussion about underlying indications.

Therefore, an automatic translator for sourcecode is desirable.

Researchers have been look-ing into this topic a lot, but no general translator

2 Translation between Programming
Languages

is developed until machine learning demonstrates

Program translation, or code translation, refers

its power in the ﬁeld of natural language transla-

to converting a code snippet written in a high-level

This is a non-referred paper. Copyrights belong to
the Author(s).
Dai Feng, Shigeru Chiba, Graduate School of Information Science and Technology, The University ofTokyo.

programming language, such as Java and Python,to another high-level programming language. It isuseful and necessary in many scenarios.
Aging software has been increasingly problematic

in recent years. Maintenance for old software, espe-

languages without having to write a lot of hand-cially that written in old programming languages,

crafted rules.

is often costly and time-consuming. During the be-

Among the very few attempts to use machine

ginning period of Covid-19 pandemic, the Depart-learning for code translation, the state-of-the-art

ment of Labor of the US government was in ur-

model is developed by Roziere et al [8].

gent demand of programmers for COBOL, which

adopt unsupervised machine learning techniques

is a programming language developed in the 1950s,

from natural language translation, and use thebecause the system was overloaded due to the in-techniques in this task. When representing source

creasing number of claims of unemployment. Thiscode, they follow the similar way to represent nat-

is not an easy job as old programmers have retiredural language text. They tokenize the source code,

while young programmers don’t use COBOL any-learn sub-tokens with BPE, and linearize the source

more. Similarly, the Commonwealth Bank of Aus-

code with BPE vocabulary. This is straight for-tralia spent around $750 million and 5 years of work

ward, but losses the special features of program-to convert its platform from COBOL to Java. In

ming languages.

such scenarios, an automatic tool for code conversion is desirable.

They

In programming languages, syntax trees are tree
representation of source code, and they contain rich

Another reason for the necessity lies in the diver-information regarding syntactic structure. There-sity of programming languages. Usually, program-fore, combining syntax tree information in codeming languages are chosen adapted to the native

representation is a potential method to consider

platforms. Android projects usually require a pro-

special features of programming languages.

gramming language running on the Java Virtual

Besides, current model requires large amounts of

Machine like Java and Kotlin, while Web projects

computing resource and data, and is not aﬀordable

running in the browser require to be written inby everyone. A smaller model that is friendly in

JavaScript. Therefore, when developing a project

resource is desirable.

for multiple platforms, writing same logic in diﬀer-

For the above reasons, we want to do experiments

ent programming languages is necessary, and au-

to see whether combining tree information in code

tomatic translation for such logic can save a lot ofrepresentation can contribute to machine learningtime.

models, and also whether the model can work when

Many researchers have been looking into thisgiven limited computing resource and data. To the

topic. Existing literature regarding this task are

best of our knowledge, no existing literature has

mostly rule-based. Rule-based approaches tend to

examined the idea so far.

result in translations containing bugs or lacking
readability, and can only deal with translation be-

3 Model Details

tween two speciﬁc languages. They cannot trans-

The purpose of this paper is to compare two ap-

late from an arbitrary programming language toproaches of code representation, one is our pro-another. Recently, as machine learning has shown

posal, another is from the paper by Roziere et al.,its power in many ﬁelds, researchers have been con-and test the performance on a programming lan-sidering adopting machine learning in programming

guage translation model.

language translation. They want to develop a gen-We deﬁne our proposal as tree-based code rep-eral translator between two arbitrary programming

resentation, while the approach from Roziere et al.

def

get_valuea
図1Comparison of two code representation

図2

if

<

return10

AST example

approaches

After obtaining these abstract syntax trees, we use
is text-based code representation. The overview of

pre-order depth-ﬁrst-search algorithm to get a lin-

two approaches is shown in Fig.1. The blue arrors

earized version of the ASTs.

refer to approach by Roziere et al., and the red ar-Consider the following piece of code in Lst.1 andrors refer to our proposal. We describe the details

in Fig.2, the tree-based code representation is inof two code representation approaches in this sec-

Lst.2.

tion.
Besides, we also describe the machine translation

1
2
3

def get_value () :if a < 1:
return 0

model and the training details in this section. We
Listing 1

use the same model as in the paper by Roziere etal.

3. 1 Tree-based code representation

1

Code representation example

def , get_value , if , <, a , 1 , return , 0
Listing 2

Tree-based code representation

Tree-based code representation refers to code representation with abstract syntax tree information.

3. 2 Text-based code representation

Using AST for machine learning input is not our

Similar as dealing with text in natural languages,

original idea, but we borrow this idea and apply

we separate source code into individual words for

it in the task of programming language translation

further processing, the separation is also called to-task.

kenization.

Tree-based code representation refers to generat-

One traditional way to tokenize source code ising abstract syntax trees of source code, and repre-

by identiﬁers, built-in key words and punctuation

senting source code with such trees. Abstract syn-marks. Identiﬁers include method names and classtax trees don’t preserve all the detailed information

names, built-in key words include for, if, while,...,

such as parentheses, but only programming con-and punctuation marks include commas, colons,

structs and relationships between such constructs.

brackets and so on. The tokens are linearized by

the order of appearance.
Another way is called Byte-Pair Encoding (BPE)[10], which is to learn sub-tokens from the original
vocabulary to deal with rare tokens, such as classnames and method names. These names are usually concatenated by several words using camel caseor snake case. The BPE algorithm iterates the vocabulary to count the most frequent byte pairs, andmerge two byte pair tokens into one token. Roziere
et al. use BPE algorithm in their paper.
The text-based code representation with traditional tokenization is in Lst.3, and the BPE tokenized code representation is in Lst.4.
1

def , get_value , ( , ) , : , if , a , <, 1 , : ,return , 0
Listing 3

Token-level code representation

図3

Model overview

language model pretraining is to map tokens withsimilar meanings to close positions in vector space.

1

def , get_@@ , value , ( , ) , : , if , a , <,1 , : , return , 0

Denoising auto-encoding is to make the model learn
how to decode a sequence and be robust to noise.

Listing 4

Token-level code representation BPE

Back translation is to make the model able to generate sequence based on cross-language input rep-

3. 3 Machine translation model

resentation.

After getting the representation of source code,
we will use it as input for machine translation

4 Experiments

model. It is a typical sequence-to-sequence [11] pro-In this section, we present the designs and results

cess, in which we use code representation from pre-for following experiments. Both experiments are

vious sections as input sequence, and source codeevaluated by translating both ways between Javaas output sequence.

and Python.

The basic model for training is Transformer.
Transformer [12] is consisted of an encoder and adecoder that both are composed of blocks stacked
on top of each other. In each block, there are sev-

Roziere et al. Text-based representation ofsource code with BPE tokenization
Our proposal AST-based representation ofsource codeeral modules which are meant for diﬀerent purposes, mainly multi-head attention modules and4. 1 Dataset

feed-forward modules.

The training data both in Java and Python is

The training is consisted of three steps [8],collected from GitHub. We sort the repositories

namely cross-lingual masked language model pre-by stars for each language, and download top 100

training [5], denoising auto-encoding [13] and backrepositories. In each repository, aside from the ﬁlestranslation [9]. We show the overview of the train-

in its main language, there might also be some ﬁlesing pipeline in Fig.3. Brieﬂy, cross-lingual masked

in other languages, such as scripts or conﬁgure ﬁles.

We exclude these ﬁles and only keep the ﬁles in ev-

表1

BLEU score for translation results

ery repository s main language. After obtaining theExperimentScore

source ﬁles, we extract the functions and methods

Text-based Java to Python

1.28436e-231

in the source code. We use the entire source ﬁlesText-based Python to Java

1.62241e-156

for pretraining in order to get meaningful numerical

Tree-based Java to Python

1.15311e-231

vectors for tokens. However, we only use functions

Tree-based Python to Java

1.097212e-231

for denoising auto-encoding and back-translation
for simplicity and eﬃciency. After removing ﬁles

4. 3. 1 Translation quality by BLEU scorethat are too long or contain rare symbols, we ﬁ-

The BLEU scores of the translation results isnally get 117028 Java source ﬁles and 999007 Java

shown in Table.1. It can be seen that the absolute

methods, and 23518 Python source ﬁles and 171538

values of the scores are very low, and are almost 0.

Python functions.

It means the model can barely generate valid trans-

The evaluation dataset contains 97 parallel im-

lations compared with the ground truth references.

plementations of functions both in Java andWe take a look into the results, and ﬁnd that the

Python. The data is collected from GitHub andmodel fails to generate consecutive tokens such as

has been manually checked to make sure that par-2-grams.

allel functions in a pair implement same logic.

4. 3. 2 Translation quality by important
structures

4. 2 Conﬁguration

Loops and conditional statements are important

The machine we use is an Nvidia Quadro P6000

structures in programming. From the translations

GPU card with 24GB GPU memory, an Intel Coreby two models, we observe that the model with

i7-6850K CPU with 3.60GHz frequency, 6 cores andtree-based code representation can generate more

12 threads, and 64GB memory.

structures such as loops or conditional statements

For neural network model, we use a hidden em-

than the text-based model. Therefore, we evaluate

bedding size of 256, and train each model for 100

the translation quality by counting the number of

epochs. The training takes an average of 4 days foroccurrences of such structures.

each model.

We deﬁne an occurrence of a for or a if in the

As a comparison, Roziere et al. use 32 Nvidia

ground truth code as a positive case. If the trans-

V100 GPU cards, each of them has 32GB GPU

lation successfully generates the for loop or the if

memory. They set the hidden embedding dimen-

statement, we regard this translation as a true pos-sion to be 1024.

itive, or if the translation does not generate theground truth for or if, we regard this translation

4. 3 Results

as a false negative.

We will evaluate the models by two kinds of cri-

In all the cases, there are 80 ﬁles containing a forteria. One is called BLEU score. it considers theloop or a if statement. In the text-based model,

similarity between a generated sequence and theit successfully generates 5 if statements and one

ground truth sequence. Another one is about im-

for loop when translating from Python to Java,portant structures in source code, such as for loops

and successfully generates only one for loop whenand if statements.

translating from Java to Python. All the generations are true positive. In the tree-based model,

表2

Results from text-based model

Type

TP

FP

TN

FN

Java to Python

1

0

17

79

Python to Java

6

0

17

74
図4

表3

Results from tree-based model

Type

TP

FP

TN

FN

Java to Python

15

6

11

65

Python to Java

13

0

17

67

表4

Result scores from text-based model

Type

accu

prec

recall

F1-scoJava to Python

0.186

1

0.01250.0247

Python to Java

0.237

1

0.075

0.1395

表5Ground truth code

図5

Translation by tree-based model

図6

Translation by text-based model

Result scores from tree-based model

text-based model, and all other scores are higher or
Type

accu

prec

recall

F1-scoJava to Python

0.268

0.714

0.1875

0.297

Python to Java

0.309

1

0.16250.2796equal to text-based model. Therefore, tree-basedmodel does better in generating important structures than text-based model. This could be theevidence to show that embedding AST information

when translating from Python to Java, we success-in the code representation can contribute to gen-fully generate 12 if statements and one for loop,

erating more important structures than text-based

and all the generations are true positive. Whencode representation. To look more deeply into thistranslating from Java to Python, we generates 15

phenomenon, we provide a case study to see what

if statements and 6 for loops, however, 6 genera-exactly do the models generate.

tions of if statements are incorrect.
From Table.2 and Table.3, we can see that treebased model can generate more important structures than text-based model.

4. 4 Case study
We provide a real example of the generated

However, all thetranslation by tree-based model and by text-based

generations by text-based model are correct, whilemodel, and discuss the indications from this exam-

tree-based model may generate some incorrectple.

translations. We calculate the accuracy, the precision and the recall score for each model.
From Table.4 and Table.5, we can see that only

The code we use to translate is shown in Fig.4,
and the code generated by tree-based model andtext-based model is shown in Fig.5 and in Fig.6.

the precision score in tree-based model when trans-

From the generated translations, we can see thatlating from Java to Python is lower than that ofboth generations follow the Java code grammar.

However, the generated translations both contain2 code to Python 3 [7].

a lot of syntactical mistakes, such as wrong identi-

are mostly rely on a set of human-deﬁned rules,

ﬁers and false method calls. Such kind of mistakes

and creating the rules requires a lot of time. Re-

make the translations impossible to pass the com-

cently, with the development of machine learning,

piler. If we calculate the BLEU score of the gener-

learning-based translation has been investigatedated translations, the absolute value of both scores

more than before. Nguyen et al. [6] developed awill be low. However, if we look at the structure

model on a Java-C# corpus using phrase-basedof the code, we can see that the translation gener-

statistical machine translation technique. Unfor-

ated by tree-based model successfully captures thetunately, these attempts are supervised, and rely

for-loop in the ground truth code, which indicateson parallel data that is not easy to obtain.

that our translation is slightly more like the ground

2020, Roziere et al. [8] published a paper dis-truth code from a topological viewpoint.

cussing the approach to translate source code from

Combined with the results shown in the previous

However, these toolsIn

one programming language to another in an un-section, we can conclude that tree-based model issupervised manner.

better at generating important structures such asmachine translation model to translate between

loops and conditional statements than text-based

Java, Python and C++ pairwise with high accu-

model. This is possible because tree-based model

racy, proving that unsupervised machine transla-

embeds abstract syntax tree information in thetion could be a useful tool in solving this task.

They adopted unsupervisedcode representation. To some extend, an abstract
syntax tree preserves more structural information

6 Conclusion

than a linear sequence of tokens. Therefore, whenIn this paper we present a comparison between

feeding the AST-based representation of code into

text-based code representation and tree-based code

the neural machine translation model, the model

representation. Text-based code representation re-might have extra knowledge about the structure ofgards source code as plain natural text, while tree-

the source code, and this extra knowledge mightbased code representation adopts abstract syntax

be able to contribute to ﬁnal translation quality

tree information. The comparison is based on acompared with text-based representation of source

programming language translation task.

code. To prove this, we still need more experiments
to ﬁnd concrete evidence.

We present the experiment results for tree-basedmodel and text-based model, and show that treebased model has potential advantages over text-

5 Related Works

based model in generating important structures of

Applying natural language processing techniquessource code. We also provide a case study to dis-to programming language research has been a pop-

cuss the reasons for the results. The results areular topic for a long time. In the area of code sug-

open for more discussion.

gestion [1], error detection [2] or code comment generation [4], natural language processing techniques

We believe there is space for further improvement, and that part will be our future work.

have been of much help.
Translating between programming languages isalso not a new research topic.

For example, aPython library called 2to3 can help to port Python

参 考 文 献[ 1 ] Allamanis, M., Barr, E. T., Bird, C., and Sutton,
C.: Learning Natural Coding Conventions, Pro-

ceedings of the 22nd ACM SIGSOFT International[ 7 ] Python: https://docs.python.org/2/library/2to3.html.
Symposium on Foundations of Software Engineer[ 8 ] Roziere, B., Lachaux, M.-A., Chanussot, L., anding, FSE 2014, New York, NY, USA, Association
Lample, G.: Unsupervised translation of programfor Computing Machinery, 2014, pp. 281–293.
ming languages, Advances in Neural Information[ 2 ] Chen, Z., Kommrusch, S. J., Tufano, M.,
Processing Systems, Vol. 33(2020).
Pouchet, L.-N., Poshyvanyk, D., and Monperrus,[ 9 ] Sennrich, R., Haddow, B., and Birch, A.: ImM.: SEQUENCER: Sequence-to-Sequence Learnproving Neural Machine Translation Models withing for End-to-End Program Repair, IEEE TransMonolingual Data, 2016.
[10] Sennrich, R., Haddow, B., and Birch, A.: Neuralactions on Software Engineering, (2021), pp. 1–1.
[ 3 ] Frederickson, B.: http://www.benfrederickson.com/rankingMachine Translation of Rare Words with Subword
programming-languages-by-github-users/, 2018.
Units, 2016.
[ 4 ] Hu, X., Li, G., Xia, X., Lo, D., and Jin, Z.: Deep[11] Sutskever, I., Vinyals, O., and Le, Q. V.: SeCode Comment Generation, Proceedings of the 26th
quence to Sequence Learning with Neural Networks,
Conference on Program Comprehension, ICPC ’18,
CoRR, Vol. abs/1409.3215(2014).
New York, NY, USA, Association for Computing[12] Vaswani, A., Shazeer, N., Parmar, N., UszkorMachinery, 2018, pp. 200–210.
eit, J., Jones, L., Gomez, A. N., Kaiser, L., and[ 5 ] Lample, G. and Conneau, A.: Cross-lingual LanPolosukhin, I.: Attention Is All You Need, 2017.
[13] Vincent, P., Larochelle, H., Bengio, Y., andguage Model Pretraining, 2019.
[ 6 ] Nguyen, A. T., Nguyen, T. T., and Nguyen,
Manzagol, P.-A.: Extracting and Composing RoT. N.: Lexical Statistical Machine Translation
bust Features with Denoising Autoencoders, Profor Language Migration, Proceedings of the 2013
ceedings of the 25th International Conference on
9th Joint Meeting on Foundations of Software
Machine Learning, ICML ’08, New York, NY,
Engineering, ESEC/FSE 2013, New York, NY,
USA, Association for Computing Machinery, 2008,
USA, Association for Computing Machinery, 2013,
pp. 1096 – 1103.
pp. 651–654.