Code Translation with Compiler Representations
Authors: Marc Szafraniec
Baptiste Roziere
Hugh James Leather
Patrick Labatut
Francois Charton
Gabriel Synnaeve
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Neural Machine Translation
Unsupervised Learning
Programming Languages
LLVM
Decompilation
Deep Learning
Program Translation

Abstract: In this paper, we leverage low-level compiler intermediate representations (IR) code translation. Traditional transpilers rely on syntactic information and handcrafted rules, which limits their applicability and produces unnatural-looking code. Applying neural machine translation (NMT) approaches to code has successfully broadened the set of programs on which one can get a natural-looking translation. However, they treat the code as sequences of text tokens, and still do not differentiate well enough between similar pieces of code which have different semantics in different languages. The consequence is low quality translation, reducing the practicality of NMT, and stressing the need for an approach significantly increasing its accuracy. Here we propose to augment code translation with IRs, specifically LLVM IR, with results on the C++, Java, Rust, and Go languages. Our method improves upon the state of the art for unsupervised code translation, increasing the number of correct translations by 11% on average, and up to 79% for the Java → Rust pair with greedy decoding. With beam search, it increases the number of correct translations by 5.5% in average. We extend previous test sets for code translation, by adding hundreds of Go and Rust functions. Additionally, we train models with high performance on the problem of IR decompilation, generating programming source code from IR, and study using IRs as intermediary pivot for translation.

CODE TRANSLATION WITH COMPILER REPRESENTATIONS
Mare Szafraniec"
Baptiste Rozière*
Hugh Leather
François Charton
Patrick Labatut
Gabriel Synnaeve
Meta AI
(mszafraniec, @ metta.com
ABSTRACT
In this paper, we leverage low-Ievel compiler intermediate representations (IR) to improve code translation. Traditional transpilers rely syntactic information and handcrafted rules, which limits their applicability and produces unnatural- looking code. Applying neural machine translation (NMT) approaches to code has successfully broadened the set of programs on which one can get natural- looking translation. However, they treat the code as sequences of text tokens, and still do not differentiate well enough between similar pieces of code which have different semantics in different languages. The consequence is low quality translation, reducing the practicality of NMT, and stressing the need for an approach significantly increasing its accuracy. Here we propose augment code translation with IRs, specifically LLVM IR, with results on the C++, Java, Rust, and Go languages. Our method improves upon the state of the for unsupervised code translation, increasing the number of correct translations by 11 % on average, and up to 79% for the Java - Rust pair with greedy decoding. With beam search, it increases the number of correct translations by 5.5% in average. We extend previous test sets for code translation, by adding hundreds of Go and Rust functions. Additionally, we train models with high performance on the problem of IR decompilation, generating programming source code from IR, and study using IRs intermediary pivot for translation.
INTRODUCTION
Automatic code translation allows port old codebases slow) languages to low-level (and fast) ones. Current industry solutions, known as transpilers or transcompilerg] rely handcrafted rules that are applied systematically. They produce unidiomatic translations that prove hard to read for human programmers. This is a serious limitation: the translated code should be easy to read and understand, as will eventually be maintained by human developers.
frameworks, high-level (but
In recent years, Neural Machine Translation (NMT) was proposed as an alternative to rule-based code translation (Roziere al. 2020 Weisz al. 202 2022). These models, trained from existing code, produce idiomatic, easy to understand, translations. Unfortunately. neural transpilers are unreliable, and often fail to translate the semantics of the input program accurately. This is serious limitation, as some of the human work saved by the transpiler has to be reinvested debugging its output.
We propose to improve the reliability of NMT by leveraging information from compiler toolchains. When processing source code, compilers create Intermediary Representations (IR): language-agnostin pseudocode that describes the semantics of the program. Augmenting training data with the corre- sponding IR can benefit a Neural Transpiler in two ways: helps align embeddings for different languages and improves the semantic understanding of the code. As shown in Figure I this can greatly improve the semantic quality of neural translations.
In this work, we leverage LLVM (Lattner and Adve 2004) to augment source code with corresponding Intermediate Representation and train models for code translation and decompilation. We compare
Equul contribution
https: //e..//// wi ki kipeiia wwrg/wwki /Swukcii//ooeeetteee / 1er

---

Input function
TransCoder
TransCoder-IR
Cet
Ruat
int mchlern I int n
Ia ter (oz usize )-> usize
In nehutorn ( 132
132
zeturn non mearn
static int dlviaorSum(AAn nf
Fune divisorSun(n int) (int, int) sur :- for suB
rune divisorSum (an int aun for
Int
int 0
fint imts 1erny
sun -- ( n / i
return
return SUm
1/ Java
// Go
// Go
static boolean IsDiVRy9 (int n]{ 12 return true;
func TantvRys 1 int bool | 12 return true
fune TaDivRys ( n int bool
9
return true
if 9
zetuxo false) return LsDivBy9
retuzn falce
return false
(int) (int) zetuza IaDivBys (int
1;
> 3
return TaDivRys
int
1nt (n >9 1nt (n 701
Figure 1: Improvements over TransCoder. The first example shows a translation from C++ to rust, where TransCoder generates code using unsigned instead of signed integers. In the second example, translation from Java Go, it generates function with the wrong return type. the third example, which is also translation from Java to Go, the model outputs function that looks similar the correct solution but it confuses with > and closes an expression with a parenthesis too early. In these cases and many others, TransCoder makes mistakes that are small in terms of edit distance, but have large impact on the semantics of the code. Using the IR to ground the representations to the semantics often helps solving these issues.
it to TransCoder, which uses only code and no IR. We also design an IR-only baseline, dubbed the pivot method, which generates translation solely by decompiling IR generated from the source language to a different target language. We experiment with four languages: C++ Java, Rust and Go, and show that utilizing both the code and the IR allows for average relative improvement of 5.5%. Moreover, our method only uses the IR at training time and does not require extra computations at inference time.
Our main contributions are:
We implement new IR-augmented translation method, which leverages LLVM IRs to improve code representations. It allows to increase the number of correct translations generated by TransCoder for C++, Java, Go and Rust by 5.5%. Compared to our IR-only pivot method, the improvement reaches 170%
Our method is especially useful in the low data regime: with relative improvements reaching 29.7% when translating Rust and 25.6% when translating from it.
We extend the parallel evaluation dataset of 852 functions in C++, Java and Python from Roziere et al. (0200) with 343 more functions in Go and 280 more in Rust, along with corresponding test cases
In addition, we achieve 78% accuracy when decompiling LLVM IRs to C++
2 INTERMEDIATE REPRESENTATIONS IN COMPILERS
Compilers translate programs written in computer language into executable code for specific machine. Most compilers consist front-end taking source code as input, and a back-end which produces machine binary code. The front-end lexes (tokenizes) and parses the program. Then, it produces an abstract syntax tree (AST), and translates it into some Intermediate Representation (IR). The back-end converts the IR into machine-specine executable code.
In modern compilers such as LLVM (Lattner and Adve, 2004), the IR is generic across different input languages (and thus different front-ends). It allows the application of transformations and target
2

---

language front-end
language agricsto middle--nn
target
4
Lexing and Parsing
Semanticc anayss
LLVM IN
Generic optimizatioss
Targel anch.
.5
Binary
exo
optimization
.lx varaaor tonaa
20% Canntant lant aree, chare argef1l
der and 1st geiintase We, 180e 92) a2
deflune 122 ghosr L8ve
.cfi, def offsst
20:
store Lo2 25, 112e align 4 align ret 132 s8
loca untruns__azer aazer
esta_ortset hrag,
return 18
not
.cfi def cra. register wrbp
pops rots
krop
Figure 2: A bird's eye view of a compiler toolchain, exemplified with LLVM. The unoptimized version (-00) is shown here for illustration. In practice used the size-optimized version (-Oz) of the IR as boxed, which does the compile time optimization of computing the addition of 26 and 16.
agnostic optimizations the IR, in a middle-end module independent from the source language and target machine. This results in an efficient compiler structure: new languages can be implemented by rewriting the front-end, and new target machines by rewriting the back-end.
Several IRs usually co-exist in a compiler: each stage in the toolchain (Figure introduces new representation. Early stage IRs are languag-deepenenn (e.g. ASTs mirror the syntax of the source language). Late stage IRs replace named variables by registers and reflect the specifics of the target architecture. this work, we are interested in middle-end IRs, which are independent from the target machine, and similar for all source languages (like dialects in natural languages).
TRAINING OBJECTIVES
Unsupervised machine translation consists of learning multilingual sequence embeddings, and generating sequence in any output language from these embeddings (Lample et al. 2018a). We now present the objective functions for these tasks. In section
3.1 we review the three basic objectives baseline NMT system. In section 3.2 introduce three new functions that
used by TransCoder, leverage LLVM IRs to improve the multilingual representation of source code, and the performance of our translation models. During training, we alternate between all six objectives, running each for the same number of optimisation steps. At inference, the model is only provided with the source code, i.e. the IR is not needed.
Formally, let x 21 NN. be the source sentence, 2(0) = 2(()) (1) (2) the corresponding IR, and ... y = y/ ... .N.., the target sentence. We write Ler = = (1 y:). with (e y1i the pairwise cross-entropy loss between di and yi- We define the machine translation loss (or seq2seq loss) from x to y,, LMT as the sum of the negative log-likelihood of each token yıs given x and previous tokens yo ... Vi-1 (note that z and y can have different lengths):
LMr(L,
Σ log (P(y Jr y1 (i-1))
3.1 COMMON OBJECTIVE FUNCTIONS
TransCoder (Roziere et al.) 2020) learns to translate between programming languages by leveraging three unsupervised objectives developed for natural language (Lample et al. 2018b):
Masked Language Modeling (MLM) trains an encoder to predict randomly masked inputs.
is commonly used to pre-train embeddings for natural (Devlin et al. 2018 Liu et al. 2019) and programming languages (Kanade et al., 2020 Feng et al. 2020p.. MLM allows the model to learn the syntax and semanties of programs. Alternative objectives, have been proposed for programming languages (Guo et al. 2020; Lachaux et al., [202]] Ahhma et al. 2021 Wang et al. 202I). We do not them here, MLM remains effective and easy on wide range of programming languages.
Denoting mask(z) (z) the masked version the code sentence æ, and enc(t) the encoder output, MLM uses the following loss:
LMLM LCE (onno(mas(())))..)
3

---

IR TLM
TLM Input
Recovered Tokens
int +MASK> Lint a, int =MASK*H
int maxiint 6, int bi
<MASK>
<SEP>
<SEP>
(132 <HASKS 132 1b)
Model
[132 1b1 local, addee
local unnaxed adds
bbl:
bal:
tarp 1cmp egt <MA5K> 16, tb
temp Lomp agt 132 8a,
temp,
kemp, va, ret 132 koond
132 <MASKK>
IR Generation (MT)
LLVM
C++ function
define deo local 132 2'32 90) Lona unnamnd ande #0
fO{ return 22;
Model
zet 132 42
Figure 3: IR for code representation objectives. We show examples of masking (used in TLM and TAE) and IR generation to improve code representations IRs. The masking objective in TLM or TAE makes model understand the relationship between code IR. The generation objective helps the model build semantic representations of the code. For instance, another C++ function computing 39 + 3 would result in the same IR. A Go function that returns 42 would also have similar LLVM IR. Therefore, the IR Generation objective encourages the model to build similar representations for these three semantically equivalent functions.
Denoising Auto Encoding (AE) trains a sequence to sequence (seq2seq) model to retrieve an original sequence from corrupted version. Corruption is done by masking spans of tokens randomly sampled from Poisson distribution, well as removing and shuffling tokens. It uses the following loss (noise(r) denotes the corrupted version of z);
CAE LMT (noise(r),
(2)
Trecksrransttion (BT). Back- Translation (Sennrich et al. 2015) uses the model to generate noisy translation the input sentence, and then trains the model to recover the original input from the translation. It is simple yet powerful objective for unsupervised machine translation (Lample et al. 2018a| AAtttte et al. 2018). In practice, it a required loss to get competitive performance, so is staple of all our experiments. Formally, use the model translate sequence into ý and train the model to reverse the translation process, using the loss:
LBT LMT ($.z)
(3)
3.2 IR FOR CODE REPRESENTATIONS
Intermediate representations (IR) provide additional information about the code to translated. We add them to the training dataset, as described in section 4.2 and leverage them by adding three new objective functions to those described in section 3.1
Translation Language Modeling (TLM), first introduced in Lample and Conneaul 2019), strives generating common representations for parallel sentences in different languages. Like the masked language modeling (MLM) objective, it trains an encoder to predict random masked inputs. However, TLM is trained on pairs of parallel sentences, concatenated together and separated by a special token. Here, we concatenate functions in their source language and their corresponding IR, using the source code and language embeddings, and train the encoder to predict randomly masked tokens. This allows the model to learn correspondences between the source and the IR. The corresponding loss is (I) denotes concatenation):
LTIM = LOE (mask(o @ 2(x)), ) @ 2(1))
(4)
4

---

IR Decompilation
LLVM IR
Local, antaaed
Jnt maxcint int BK
return
Clang
bbl1 Verp Leep kay
soleos
132
MT Model
int mox a, Art by return
Recovered code
Figure 4: IR Decompilation objective. Here, we generate the IR corresponding to each function and train model to decompile it. The IR pivot model uses this objective, well as back-translation objectives, allowing it generalize IRs generated from any language.
Translation Auto-Encoding (TAE) amounts to transposing the TLMM objective into a denoising auto-encoder. The source code and corresponding IR are corrupted and masked, and then concatenated into one sequence (using the language embeddings for code and IR, as previously). TAE is then tasked to recover the original, using the following loss:
LTAE CusT (noise(x) noise(2 ), @ 2(1))
IR Generation (MT) trains the model to translate the source code into the corresponding IR. This allows the encoder learn source code representations from the semantics of the IR. The loss is:
LIRGen = LAST (Ir, 200)
(6)
These three objectives need both the source code and the corresponding IR. However, only a fraction of the functions and files in our dataset could be compiled. To mitigate this, we also train the models on the full monolingual data using the MLM and AE objectives described above. In this setup, the back-translation (BT) objective is the same as in Rozzere et al. (2020), and allows our model to translate directly from source code only inference time.
3.3 ADDITIONAL LOSSES: IR DECOMPILATION AND PIVOT
We study two alternative uses of intermediary representations: IR decompilation, and IR pivot translation. IR decompilation consists of recovering source code corresponding to given IR. In practice, it reverses the computations performed by the compiler. IR Pivot is a translation method built upon IR decompilation. Since LLVM can compile many languages (C++, Java, Rust, Go) into the same IR, an obvious approach to code translation consists of decompiling the IR generated from the source language into code the target language. We call this method "IR pivot". Note that, whereas the IR for code representation techniques only used IR during training, both the decompilation and pivot method also need the IR for inference.
Decompilation. In this supervised task, we use LLVM generate IR from source code, and train language model to reverse the process, i.e. learn to predict the source code from the IR. Models are pre-trained using the MLM and AE objectives, and decompilation learned using the machine translation loss:
Decomm = LATT (2(x)
IR Pivot. This task leverages the IR as a pivot for code translation. For instance, to translate from Rust to C++, we first use LLVM to compile Rust program into IR and then decompile the IR to C++ using a neural decompiler.
practice, slight variations exists between the IR generated for different languages: the Rust-IR and C++-IR behave like dialects of the LLVM-IR. This often leads to poor performance of the IR Pivot method. We mitigate these issues using a variety of techniques, which describe in section of the appendix.
5

---

4 DATA
4.1 TRAINING DATA
Our training data was extracted with Google BigQuery, which indexes over 2.8 million open source repositories from GitHut We selected projects whose license explicitly permits re-distribution of parts, and extracted all individual C++, Java, Rust and Go functions. To learn to decompile IRs, we also used the CodeNet dataset (Puri al. 2021), repository of 14 million competitive programming solutions in 55 languages. Our models work at function level: this reduces compilation failures over missing dependencies, while keeping sequence lengths short.
Table Dataset coverage across languages, number of standalone functions. More details can found in Table in the appendix.
C++
Go
Java
Rust
Monolingual data Code / IR Parallel Data
6.6 M
9.4 M
7.8
576.3
344.4 K
384,4 K
2.2 M
19.2 K
Successful IR Compilation
5.2%
4.1%
28.2%
3.3%
4.2 GENERATING INTERMEDIATE REPRESENTATIONS
While the LLVM ecosystem is large, not every language has an LLVM front-end, and not every front-end can produce LLVM IR out-of-the-box. We use clang+ + Lattner and Adve (2004) from the established LLVM C++ compilation toolchain, JLang" for Java, Gollvm for Go and ruste Matsakis and Klock DI (014) for Rust. For the same program, written in different languages, different front-ends may produce different IR. To minimize these variations, we process the source code as follows. First, we generate the most size-optimized IR (-Oz flag), which makes the IR more uniform across languages. Second, we strip all unnecessary information (c.g. header and footer with attributes, debug information, comments). Finally, block names are canonicalized and symbol names demangled to facilitate their recovery. The functions that fail to compile at this point (e.g. because of missing dependencies) are not included in the parallel dataset, seen in the last row of Table [
4.3 EVALUATION
Traditional NMT evaluation relies metrics such as BLEU, that are based on n-gram overlaps. However, when dealing with programming languages, syntax and in particular compilation and computation outputs can differ widely despite minor changes in the code. Conversely, semantically equivalent code, that differ only in variable names or order of operations can have low BLEU score. To take this into account, we use and enhance the computational accuracy test suite from Roziere et al. (2000), that contains 852 parallel competitive programming solutions in C++, Java and Python. Using C2Rust. CxGo and some manual code cleaning, we translated 280 functions and test suites in Rust and 343 in Go to measure the performance of our models in these languages. We measure our performance using the computational accuracy (CA@) 1) metric (Kulal al. 2019 Roziere et al. 2020), which considers that translation is correct if it passes series of unit tests.
5 RESULTS
5.1 EXPERIMENTAL DETAILS
For TransCoder, consider sequence-to- se-euuence (seq2seq) transformer model (Vaswani et al., 2017) with attention (Bahdanau et al. 2015) Sutskever et al. 2014) and the same architecture as Roziere et al. (020). Our model has 12 layers (6 in the encoder and in the decoder), 8 attention heads, and a dimension of 1024. For the objectives that add noise and masks to the input sentence, such MLM, TLM, AE, and TAE,
choose the masked tokens and noise randomly on the fly at each epoch. We mask 15% of the tokens in MLM and TLM. In AE and TAF, we mask 20% of the tokens. MLM is trained streams of data, while the other objectives are trained function level. We
github-rae repos
https://clanq.11m.cr
https://bollyglot-ccoo io//an//
http:://go.poogeessurccc vm/

---

Table 2: Translation performance (CA@1), for greedy decoding and beam size 5. "To X": average performance when translating to language X. "From X": average performance when translating from language X. See ia the appendix for more detailed results. All these methods except for the IR pivot also use the three objectives defined in TransCoder: MLM, DAE and Back- Tack-Translttion (BT). All combinations of the TLM. MT and TAE objectives improve the performance compared TransCoder. The best results are obtained when all three are used the same time. Beam search, using beam size returning only the top element from beam results improved performance. The IR Pivot method generates a translation the target language from an IR generated from the source, and performs poorly in our setting.
from
C++ from Go to Go from Java to from Rust to Rust AVG
Greedy decoding
IR Pivot
17.4
24.0
19.9
11.5
11.9
22.2
16.3
7.8
16.4
TransCoder (baseline)
46.4
52.1
42.1
45.6
41.2
44.5
29.6
17.0
39.8
TLM
47.5
54.8
45.4
41.2
39.8
52.1
31.1
15.7
40.9
MLM + TAE
47.3
53.3
47.2
44.8
41.8
45.9
25.1
17.4
40.4
TLM TAE
46.9
55.9
45.0
37.9
38.5
54.5
34.9
16.8
41.3
MLM + MT
45.5
51.0
44.0
48.9
46.6
45.2
25.7
16.6
40.5
TLM MT
45.6
51.5
45.1
47.1
46.9
45.5
24.4
17.9
40.5
TAE MT
47.8
54.3
43.8
43.9
39.1
49.2
33.4
16.7
41.0
TLM TAE
47.8
54.3
46.6
51.6
47.1
49.6
35.3
21.4
44.2
Beam size
TransCoder (baselinc)
53.8
53.4
45.2
54.4
46.1
51.5
35.9
20.9
45.3
TIM + TAE + MT
52.9
53.5
48.8
57.1
51.5
53.4
37.9
27.1
47.8
use the Adam optimizer (Kingma and Ba 2015) and an inverse squared-root learning rate scheduler, with an initial learning rate I0 in most of our experiments. Our models are implemented in PyTon PyTorch using translation models presented in Tables and were trained for a week on 32 NVIDIA V100 GPUs.
floats. The pre-trained models were trained until convergence. The
5.2 IR-AUGMENTED CODE REPRESENTATION FOR TRANSLATION
Models using combinations of the three objectives TAE. TLM and MT introduccd leverage IR, were trained to translate between pairs of four languages (C++, Java, Rust, Go). Their average performance when translating and from every language are presented in table 2 Additional information, including comparison to TransCoder-ST for C++ ++ Java, can be found in Table 3) in the appendix. As baseline, we use a TransCoder (Roziere et al. 2020) model, trained with MLM on the same dataset.
Using greedy decoding, the new TLM, TAE and MT objectives, which leverage the IR, improve performance for every language. The best average results are obtained when combining all of them. Compared to TransCoder, they improve performance by an average 4.4% point (11% relative). The largest impacts observed in the low data regime: translations from and into Rust (a language less represented in our training set) are improved by 25.6% and 19.3% (relative). Beam search improves the results of both TransCoder and our models, using IR-au "R-augmented representation still results in better performance. Qualitatively, we observe that IRs help our model translate types when the source and target types are represented by different tokens. For instance, in the first example of Table [ translates the semantics of int correctly using 1.32 instead of an unsigned integer type (us ze). See Appendix for more analysis on how our objectives improve word embeddings.
Compared to IR- RR-augmented translation models, the "obvious" IR Pivot method proves disappointing, even though achieves non-trivial performances. is heavily dependent the size the training set: the IR pivot performs relatively well when translating from low-resource to high-resouree languages (e.g. from Rust), and badly when translating low-resource languages (e.g. Rust).
5.3 DECOMPILATION RESULTS
To compute the IR pivot, trained neural decompiler retrieve source code from IRs. We tried two separate configurations for decompilation: a shared decoder with 6 layers for all language IR pairs, or four separate decoders of with two layers each (one per language). Using shared decoder
7

---

improves the performance for all languages, and particularly when the data is scarce (e.g. Rust). See Table in the appendix for more information.
We compare the performance of model to RetDee (Kroustek et al. 2017 2. rule-based decompiler. obtains a computational accuracy of 68.75 on our C++ dataset and BLEU score of 8.54. In comparison, model obtains computational accuracy of 77.9 and BLEU score of 63.6 in the same setting. In particular, RetDec fails to decompile LLVM files generated from C++ code, especially snippets leveraging the standard library structures such as unordered _aa or std allocator. The limitations of RetDec, which was implemented by a team of 24 developers in 7 years [ shows how difficult it is to build exhaustive rule-based decompilers, especially when the comes from different languages or tools.
6 DISCUSSION
Different IR and interpreted languages The four languages considered in this work have front- ends that can output LLVM Intermediary Representation. LLVM presently covers more than 30 computer languages. Using IR as pivot requires that the source and destination language have front-ends that use the same IR. This rules out some widely-used languages (e.g. Python). Using the IR to improve embeddings is less restrictive: the source and destination language be trained different IR, and aligned with trecstrannlttion In this paper, we focus on compiled languages, but important note that Intermediary Representations are usually available for interpreted languages as well: modern interpreters translate the source code into byte-code, that can serve as an IR.
Pivot vs Embedding TransCoder is an unsupervised model that learns to align code representations and translate code from one language to another. It is based solely on source code and does not use IRs. The pivot method uses automatically generated parallel sentences to learn to decompile IRs, and back-transe trakslatnnltooon adapt different IR dialects. This method learns to translate using only IR-level similarities, and does not use the source code itself except to compute the IR. Although underperforms other methods, it performs relatively well when little data is available for the source language, because the IR can be computed using rule-based compiler. However, it requires to compute IRs at test time, which can be cumbersome. Instead, adding the TLM, TAE, and MT objectives to the objectives generally used for unsupervised code translation allows the model to get the best of both worlds. It can learn multilingual representations source code from similarities in the IR and in the source code itself. As shown in Table [2] it outperforms both TransCoder and the pivot method. At the same time, this model does not require compute IRs at test time, and is as easy to use as TransCoder.
Using our model at inference time. Our self-supervised IR IR-augmented TLM, TAE and MT objectives are designed improve the multilingual code representations used in translation models. However, the translation task does not require to compute these objectives. Therefore, they lead to models that are just as simple to use as TransCoder: computing the IR is not required at test time and the model generates the translation directly from the source function.
7 RELATED WORKS
Source-to-Source Translation, Many rule-based methods are available for transpilation, inventory of which can be found " In particular, C2Rust and CxGe 88 along with manual corrections. central for us in translating evaluation tests to Go and Rust (See Section 4.3) Similarly, 2to
a Python library porting Python code to Python 3, was used in Aggarwal et al. (2055) to create a parallel dataset and train machine learning model.
Neural Machine Translation for code hampered by the lack of parallel data between programming
languages. Indeed, apart from a few language pairs, such Java-C# (Nguyen et al. 2013 Chen
ida lugin.
htm
https///tthbb.or//mma//////1
c/cc
8

---

al. 2018), and specific domains (e.g. competitive programming code), is difficult to collect large datasets of semantically equivalent code in different languages. TransCoder (Roziere et al.) (020) bridges this gap by introducing unsupervised machine translation to programming languages. They take advantage of large monolingual code bases to learn to translate between C++, Python and Java with high performance. Later, DOBF (Lachaux et al. 202T) improved the model pre-training method used in TransCoder, and Roziere et al. (2022) used automatically generated unit tests to improve translation performance between Java, C++ and Python. Recently, large language models trained on code, such as Codex (Chen et al., 2021 and PALM (Chowdhery et al. 2022). have been used for unsupervised code translation.
Using the Transcoder model, Weisz et al. 2021) and Weisz et al. 2022 survey the links between humans and NMT methods for code translation. They view neural translation methods as aids to programmers. In this context, they demonstrate that even imperfect models can improve the quality of an engineer's work for code translation, and plead for the improvement of numan-machine interfaces.
Decompilation. Like transpilation, decompilation is usually performed using rule-based methods that rely on pattern matching to parse the control flow structure of the program. RetDec, an open source decompiler created by Avast (Kroustek et al. 2017). can decompile an executable to C and a Python--ikee like language via LLVM IR. Other tools exist, such as the Hex-Rayy Decompile:" TT and Brumley et al. (2013). A thorough review of rule-based methods can be found in papers such as Liang et al. (2021a) and Katz, et al. (201)). With these methods, decompilation can fail if the code is if it contains language features that were not explicitly translated. Most methods also produce unstructured programs, relying on a large number of goto statements to simulate the control flow of the lower level programming languages. This is semantically correct, but very rarely found in numan-written code.
convoluted,
few works have studied the use of seeuuencett-seqeeeeee neural networks for neural decompilation. Katz et al. (2019) uses LSTM networks to decompile LLVM IRs and assembly code to C. Their approach generates code templates based on the IR, that determine the structure of the output. Then, they fill them with correct variable assignments and numerical values. In the same vein, Fu et al. (2019) tries address limitations of neural decompilation with two sequential phases: code sketch generation and iterative error correction. Finally, Liang et al. (2021b) use method close to ours, and train Transformer models to translate between binary code and C.
Intermediate representations are almost as old as compiler design. The first IR, UNCOL (Strong ct al., 1958) was introduced in the mid-1950s, together with the idea of reusing the same compiler for several languages and machines. In 1960, NELIAC (a variant of ALGOL) (Huskey al. 1960) was the first retargetable compiler, portable to different architectures. Feldman ( (177)) describes how a compiler for Fortran 77 can be added the C compilers Johnson (1979) and Ritchie (1979). GCC (Stallman 2001) introduces Register Transfer Language (RTL) a level IR inspired by Davidson
and Fraser (1980), and then GENERIC and GIMPLE (Merrill 2003 precursors of the IR used in
LLVM (Lattner and Adve, 2004).
CONCLUSION
In this paper, we leverage LLVM IRs improve neural machine translation for source code. The IR provides common semantically-rich language, into which C++, Go, Java and Rust code can all be compiled. We develop three objectives, designed to leverage IRs for better multilingual representations of source code, which lead to a 5.5% relative average improvement for code translation. We also show that -o-eencee---seuencee transformers perform well for neural decompilation, and use this for pivot translation.
We only worked with the LLVM IR, but our approach is broadly applicable to any pair of languages that share common Intermediate Representation. More generally any IR can help improve the code representations by tying them to the semantics. Another limitation is the scale of our current source and target sequences. As future work, LLVM IRs could be generated a larger scale by compiling entire projects, which would greatly improve the percentage of successful IR compilations in Table [ More languages and IRs could be used, and those extensions could be powered by larger models.
https:://hett rays. com/ decompi /
9

---

REFERENCES
Karan Aggarwal, Mohammad Salameh, and Abram Hindle. Using machine translation for converting Python 2 to Python 3 code. Technical report, PeerJ PrePrints, 2015.
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2665-2668, 2021.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Unsupervised statistical machine translation. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, 2018.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, 2015.
David Brumley, JongHyup Lec, Edward Schwartz, and Maverick Woo. Native x86 decompilation using semantics semantics-preserving structural analysis and iterative control-ffor floww structuring. 22nd USENIX Security Symposium (USENIX Security 13), pages 353-368, 2013.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards,
Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 110.::2107.33774,, 2021.
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In Advances in neural information processing systems, pages 2547-2557, 2018.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Jack W. Davidson and Christopher W. Fraser. The design and application of retargetable peephole optimizer. ACM Trans. Program. Lang. Syst., 2(2): 1910)191122222 apr 1980. ISSN 0164-0925.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre- training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805,, 2018.
Stuart I. Feldman. Implementation of portable Fortran 77 compiler using modern tools. SIGPLAN Not., 14(8):98-006, aug 1979. ISSN 0342-1340. doi: 10.1 10.1445972233.806655., URL https 806959
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Findings, pages 1536-1547, 2020.
Processing:
Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushanfar, and Jishen Zhao. Coda: An end-to-end neural program decompiler. In Advances in Neural Information Processing Systems, pages 3703-3714 14, 2019.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. In International Conference on Learning Representations, 2020.
Harry D. Huskey, M. H. Halstead, and R. McAnthur. NELIAC - dialect of ALGOL. Commun. ACM, 3(8):463-468 aug 1960. ISSN 0001-0782 doi: 10. 1.111//367368.6677777 URL https
S. C. Johnson. A tour through the portable C compiler. In Unix Programmer's Manual, 7th Edition, 2B, Section 33, 1979.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning, pages 5110-5121. PMLR, 2020.
10

---

Omer Katz, Yuval Olshaker, Yoav Goldberg, and Eran Yahav. Towards neural decompilation. arXiv preprint 1.Xivv1905.0822.. 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Jakub Křoustek, Peter Matula, and P Zemek. RetDec: An open-source machine-code decompiler, 2017. / gitgtgthh......
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pscudocode code. Advances in Neural Information Processing Systems, 32:11906-11 1917, 2019.
Marie-Anne Lachaux, Baptiste Roziere, Mare Szafraniec, and Guillaume Lample. DOBF: A deob- fuscation pre-training objective for programming languages. arXiv preprint arXiv:2102.07492.
2021.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems, 32:7059- 7069, 2019.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. ICLR, 20
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase- based & neural unsupervised machine translation. In EMNLP, 2018b.
Chris Lattner and Vikram Adve. Llvm: A compilation framework for lifelong program analysis & transformation. International Symposium on Code Generation and Optimization, 2004. CGO 2004., pages 75-86. IEEE, 2004.
Ruigang Liang, Ying Cao, Peiwei Hu, and Kai Chen. Neutron: an attention- basedd neural decompiler. Cybersecurity, 4(1):1-13, 2021a.
Ruigang Liang, Ying Cao, Peiwei Hu, Jinwen He, and Kai Chen. Semantics-recoverrig decompilation through neural machine translation. arXiv preprint arXiv:2 2021b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1907. 1 1692, 2019.
Nicholas D Matsakis and Felix S Klock II. The Rust language. In ACM SIGAda Ada Letters, volume 34, pages 104.104. ACM, 2014.
Jason Merrill. GENERIC and GIMPLE: A new tree representation for entire functions. In in Proc. GCC Developers Summit, 2003, pages 171-180, 2003.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pages 651-654, 2013.
Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, and Ulrich Finkler. Project CodeNet: A large-scale AI for code dataset for learning diversity of coding tasks. CoRR, abs/2105. 12655, 2021. URL httpp:://axxi...oo
2105.1263
D.M. Ritchie, A Tour Through the UNIX C Compiler. 1979. URL httttts://books.gogg//
com/ www.ks/iidsSvVEe.......
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33, 2020.
Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. Leveraging automated unit tests for unsupervised code translation. ICLR, 2022.

---

Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 56-96, 2015.
Richard Stallman. Using and porting the GNU Compiler Collection. M.I.T. Artificial Intelligence Laboratory, 2001.
J. Strong, J. Wegstein, A. Tritter, J. Olsztyn, O. Mock, and T. Steel. The problem of programming communication with changing machines: proposed solution. Commun, ACM, 188):12-18, aug 1958. ISSN 0001-0782. doi: 10. 10.114573889223368955 UtL.https:///o..org// / 368892.368911
1145/
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008, 2017.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696-87000 2021.
Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. Perfection not required? human-ai partnerships in code translation. In 26th International Conference on Intelligent User Interfaces, pages 402-412, 2021.
Justin Weisz, Michael Muller, Steven Ross, Fernando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, and John T Richards. Better together? an evaluation of Al- supported code translation. 27th International Conference on Intelligent User Interfaces, pages 369-391, 2022.
12

---

A FULL SCORES TABLE
Table 3: Results on unsupervised code translation. The metric shown is the computational accuracy for single generation (CA@) 1), measuring the translation correctness using unit tests. It is the full version of Table The models all trained with the same budget. As Table 2 all these methods exeept for the IR pivot also the three objectives defined in TransCoder: MLM, DAE and Back-Translation (BT). Although it not the case for every language pair, TransCoder-IR, which uses TLM,, TAE, methods on average. TransCoder-ST (Roziere et al. 2022) uses parallel dataset generated with automated unit tests and outperforms other methods for C++ ++ Java. Their method is orthogonal to ours, and we could also improve performance with similar methods.
objectives outperforms other
C++ + Go
C++ + Java C++ + Rust
Go - C++
Go + Java Go + Rust
Baseline TransCoder
57.7
63.3
18.2
56.1
46.9
23.3
TransCoder-ST
68.0
Pivot
16.1
22.0
14.0
30.5
26.5
2.7
TLM
61.8
62.5
18.2
57.6
56.4
22.2
TAE
57.7
62.5
21.7
63.0
54.7
23.8
TLM TAE
58.2
63.3
19.2
55.2
57.0
22.8
MT
56.8
60.6
19.2
60.3
53.8
18.0
TLM + MT
58.6
58.5
19.7
57.3
54.1
23.8
TAE MT
61.4
60.2
21.7
55.5
53.8
22.2
TLM + TAE + MT
55.9
62.9
24.8
61.8
55.7
22.2
Java + C++
Java Go
Java
Rust
Rust - C++
Rust + Go
Rust + Java
Baseline TransCoder
77.9
35.9
9.6
22.4
43.2
23.4
TransCoder-ST
84.6
Pivot
19.5
9.4
6.7
22.0
8.9
18.1
TLM
80.9
31.8
6.6
25.9
30.0
37.5
TAE
80.3
38.6
6.6
16.6
38.1
20.6
TLM TAE
82.2
24.6
8.6
30.4
31.0
43.3
MT
76.2
50.9
12.6
16.6
39.1
21.3
TLM MT
77.9
52.7
10.1
19.2
30.0
24.1
TAE + MT
77.5
33.6
6.1
30.0
36.6
33.7
TLM TAE MT
74.5
49.6
17.2
26.5
49.2
30.2
13

---

B BEAM SIZE EVALUATION
Table 4: Results on unsupervised code translation with different beam sizes. The metric shown is still the computational accuracy for a single generation (CA@ 1). BS N refers to beam search decoding with beam size N, and returning only the top clement of the beam. Using beam search improves the average performance of every model. BS N means that the model is evaluated with beam size N. When the beam size not given, we use greedy decoding. Surprisingly, beam size outperforms beam size Our method using intermediate representations still outperforms the baseline with heam size 5 and 10 in average. With the baseline, we obtain average CA@ scores of 45.3 with beam size 5 and 44.0 with beam size 10. Our method yields CA@I scores of 47.8 with beam size and 46.8 with beam size 10.
C++ Go
C++
Java C++
Rust
Go
C++
Go
Java
Go
Rust
Baseline TransCoder
57.7
63.3
18.2
56.1
46.9
23.3
Baseline TransCoder (BS
65.5
67.6
28.3
52.1
59.3
24.3
Baseline TransCoder (BS 10)
65.0
68.7
27.8
52.1
57.0
23.8
TLM +
55.9
62.9
24.8
61.8
55.7
22.2
TLM + TAE MT (BS 5)
61.4
66.6
30.8
57.3
59.0
30.2
TLM + TAE + MT (BS 10)
61.4
67.4
29.3
56.4
59.0
29.1
Java + C++
Java + Go
Java
Rust
Rust
C++ Rust -> Go
Rust - Java
Baseline TransCoder
77.9
35.9
9.6
22.4
43.2
23.4
Baseline TransCoder (BS
82.9
45.5
10.1
25.2
54.8
27.5
Baseline TransCoder (BS 10)
80.9
46.4
7.6
23.6
51.8
23.0
TLM + TAE MT
74.5
49.6
17.2
26.5
49.2
30.2
TLM + TAE MT (BS 5)
76.4
57.7
20.2
26.8
52.3
34.7
TLM + TAE + MT (BS 10)
77.7
57.3
18.2
26.2
51.8
28.2
C PIVOT METHOD DETAILS
As mentioned in Section 3.3 IR generated from different languages contain slight variations and can be seen as dialects of the same language. In practice, these variations prevent us from simply using our best decompilation model to generate source code another language than the one used to generate the IR. Although we prompt the model to generate code in the target language with language embeddings, learns focus on the particularities of each dialect and ignores the language embeddings. Therefore, it generates code in the source language, which results in computational accuracy score for translation.
One way solve this issue is to use one decoder target language. Then, the model is able to generate code in the target language. However, this method still performs poorly due to the small differences between the IR dialects. The method tested that performed the best, and which is reported in Table uses back-translation to make the model to translate from any IR dialect to any language. This model is also grounded by supervised translation steps making generate IR from code and code from IR. In practice, we create new language embeddings for every IR dialect (i.e. IR-C++, IR-Go, IR-Java, IR-Rust) for depending the model generate noisy translations in the IR-Go, IR-Java and IR- "languages" for every C++ sequence, and train re-generate the C++ sequence from the noisy translation. To allow the model to generate good training data for IR-X + C++, we also generate noisy translations in Go, Java, and Rust for every IR generated from C++ in our dataset and train the model to retrieve the IR. Using our parallel code//IR dataset, we also train the model to translate between C++ and IR-C++ sequences. do the same for every language and alternate between them.
the source language. At training time, we make
14

---

D IR DECOMPILATION
Table Performance LLVM IRs Decompilation. This table shows the computational accuracy (CA@) of our neural decompiler and the RetDee C+ rule-based decompiler. Our neural decompiler outperforms RedDec on C++ and is more broadly applicable.
C++
Go
Java
Rust
Baseline RetDec
68.8
Separate Decoders
52.7
42.2
60.1
19.5
Shared Decoder
77.9
70.1
82.2
61.0
E TRANSLATION EXAMPLES
Input function (C++)
LLVM IR
Decompilation code (C++)
define 132 @*max" (132 $a, 132
(int b)
local unnamed addı 40
for (int 1 = i 10y ++1)
lint int return > by
Semp icmp egt 132 8a, Scone select 11 kamp, 132 Na, 132 1b
define deo local zeroext g"is_ever""" local unnamed addr 40 {
bool (int return n 6 3g
bool return false
i1 false
define dao
164 e"muitiplysysevvar (164 10) unnaned 32 mul 164 80, 164
long muitiplyRySeven, (long return (dn <c 31 n)
long molt.plyeySeren (long return
Figure 5: Code simplification examples with Decompilation Pivot. Since the LLVM IR is optimized, functions that are semantically equivalent after optimization map to the same IR. In the first example, it allows to remove useless code by decompiling the generated LLVM IR. In the second example, the simplification allows find bug: the operator has precedence over It is not obvious when looking at the input code, but becomes clear with the IR and simplified C++ code. In the third example, it replaces a bitwise operation by more straightforward multiplication. In all examples, we can the compiler again to check that the IR of the decompiled code is exactly the same that of the input. guarantees that the input and simplified code have the same semanties.
C++, causing this function to always evaluate fa lse.
15

---

Input function
TransCoder
TransCoder-IR
// Java
// Rust add_one
// Rust
static int addone int
and_ooee (x
return
x11
Rust
static boolean isEven (int ni En is_even
u32 >> bool [En
is_even
(
132 -> bool
0
Figure 6: Java Rust translation examples. In the first example, the IR allows the model to understand that the Java bitwise complement operator should be replaced by in Rust. Also, allows the model translate the type correctly in both examples and avoids unnecessary casts. The IR allows the model to generate the right types (e.g. 132 instead of u32 when translating int) and operator (c.g. instead of in Rust).
Input function
TransCoder-IR
// Rust pub binary_search (nums: Vecci32>, target: 132,
// Go
fune binarySearch (nums [. int, target int) int
if lon (numa]
-> 132 |
return
nums, (s_empty() return
target
[
return
target < nums [0] (
return
right: (numa) for left <= right
let mut 1aft = 0;
let right nums.len() 1;
left (right left)
while left <- right {
if numa [mid] -- target
let mid left (right lett) /
return mid
nums [mid] target
nums [mid] target
return mid 132;
left mid
else if nums [mid] target
else
laft
1;
right
else
right mid 14
zeturn left
return left as 132;
Figure Rust Go translation example, This function performs binary search the insertion index for an element in an ordered vector. The model translates types, function definitions, variable definitions, and while loops correctly.
16

---

Input function
TransCoder-IR
// Go
// C++ peirestnSertcdRRtttelllii (int arr (), int int x)
func Pr int,
int, x int)
int
int i;
for 01 1, LHFE
for (1 - 0; 1 < n 1: 1 H)
arr(i) arr
(arr [1] arr 12 1]) break;
break
int = [1 + 6 n;
int r 17
int
int
var int = 1
while (1 I=
var ent int
larr [1]
arr [x] x)
ent
if arr(1] + arr(r] --
if (1 -m 15 1 ml
n) return enty
ent
(1 1)
6 my
n)
return ent
else (arr *arr(r] < x) 1 (1 & mz
(1 8
else if arr(1] + arr <
return ent;
else
return ent
Figure 8: Go C++ translation example. This function computes the number of pairs elements that sum to a given target in sorted and rotated array. Our TransCoder- IR model translates it correctly.
17

---

Input function
TransCoder-IR
// Rost
// Go getMoxxepetirionsse
pub get_max wwwextteettttiiiiieeeeeeeei (s1: String, nl: 132,
string,
n1 int, 82 string, int,
$2: String, 132,
132
int if n1 -- raturn
if n1 --
return 0g
let mut indices vec! [0pm1 as usize + 11;
indices :- make int, n1 + 11
let mut counts [0sn1 usize 117
counta :- make [[] int, n1 11
let mut indox
indox :=
let mut count 0z
count :=
for in 1.. ml as usize
for :- 1;
j c..al.lea {
ni:
if
1++
s1.as_ bytts_yttttti (1 (i)
for j: 0;
sy.es_ [index]
(el);
f++t
index += 1,
if a1 [3] s2 [index]
{
}
index++
if index ()
Index = 0;
if index len
couns 1;
index
count
courts[s] caunt;
indices - index;
counts [1] count indicea - index
for k in C..1
indices(k] irdax let pre_count counts[k];
0;
let pattern_covnt = (nl as 132)
k++1
(1 kt as 132
if indices [k] index
(counta (1] pre_count);
preCount counta
let remain_ counts
pattersCount := int (nl - k)
(1
(counts preCouat)
+ [n] as usize - k)
remainCoume counta
$ (1
+ (nl - x) (i k)
pre_count;
preCount
return
return
pre_count
preCount
pattezn_couat
patteznCoust
remain_count
remainCount
n2;
n2
[m] usize]
counts [n1]
Figure 9: Rust to Go translation example. We call S1 the string n.. repeated s1 times and $2 the string n:2 repeated s2 times. This function finds the largest number of repetitions of S2 appearing in any subset of S1. The model translates us i ze) is not required in Go, and correctly translates other Rust constructs to Go.
types correctly, understands that casting vector indices unsigned int (i.e, with
18

---

F DATASET SIZE DETAILS
Table 6: Dataset details: number tokens in our function-level dataset.
This dataset contains only
functions defined outside of classes and static funnttios.
Number of tokens
Number of sentences
Monolingual data
C++
2.33B
6.6M
Go
1.9B
9.4M
Java
1.5B 130.0M
7.8M
Rust
576.3K
Code / IR Parallel Data
C++-IR
946.7M 7M
343.9K
Go-IR
971.8M
384.4K
Java-IR
1.7B
2.2M
Rust-IR
77.7M
19.4K
G ADDITIONAL ABLATIONS
Training on IRs with different objectives. We perform some additional ablations to determine whether our performance improvements come from training on IRs or from our TLM, TAE and MT objectives. When training model with the three objectives of TransCoder (i.e. MLM, DAE and BT) and considering the IR as an extra language, we obtain an average computational accuracy of 37.4, which is lower than that of our baseline TransCoder. As the structure of the IR is not similar that of any of our source languages, there is not much to gain from adding the IR as an extra language. Moreover, the model is wasting some time compute the AE and BT objectives for the IR which can be better spent on the source languages. It confirms that our objectives are required to map IRs and their corresponding source code similar representations embedding space.
Language ablation: no Java. As Rust and Go are more similar to Java than to C++, we also train baseline model C++, Go and Rust only to evaluate whether including Java hurts the translation performance. We observed similar performance for C++ ++ Go. However, we also observe clear decrease in performance in the very low data regime (i.e. when translating to from Rust). The computational accuracy for Rust + C++ goes down from 22.4% to 20.1% and goes down from 43. 13.1% 32.5% for Rust Go.
H WORD EMBEDDINGS
We notice that our method generates improved word embeddings. is visible when looking the cosine similarity for embeddings of rust types and their equivalents in C++. For instance, Figure I0 shows that the embedding of u32 from our model leveraging LLVM IRs is most similar uint (with cosine similarity of 0.4869). ui uint, which is also correct translation, comes in 11 position with a cosine similarity (0.3716). contrast, u32 a similarity of only 0.2828 with int. This token, which would be an incorrect translation, comes only in 29th position.
The baseline model, which does not use the IR, learns similar representations for rust types since they appear in similar contexts. Hence, its embedding of :32 is most similar to other rust types tokens such as u 132 or u16. 6. uint 32 comes only in fourth position with a cosine similarity of 0.4218. Moreover, uint and int have almost the same cosine similarities with u32 with the baseline model. causes the model to often confuse unsigned and signed integer types, and to incorrectly translate u32 into in instead of uá nt.
19

---

Baseline TransCoder:
Our model with TIM + TAE. MT
0.4789 u64
0.4869
vint32
2 0.4478
132
2 0.4779
u64
3 0.4223
u16
3 0.4681
u16
4 -0.4218
uint32
4 0.4302
uint32_t
0.4119 usize
5 - 0.4270
i32
...
14 0.3169 uint
11 0.3716 uint
16 3.00 int
29 0.2828 int
Figure 10: Token similarities. Rank and token similarity with u32 for our model (right) and the baseline model (left). Our model generates embeddings that better capture token semantics.
I ANALYSIS OF ERROR TYPES
Table Rust error types. validate our intuition on the usefulness of the representations decrease the number of type- related errors (see Fig or Fig we perform an in-depth analysis of the types errors encountered for the Java -> Rust direction. Here we count the total number of erross (there can be several for single translation). We notice that the number of type-related errors (excluding E0433, E0425 and Others) decreases by 24% (609 463) and the number of mismatched types decreases by 49%.
Error Code
Error Description
Baseline (Transcoder)
TLM + TAE + MT
E0308
Mismatched Type
414
210
E0412
Type Does Not Exist
I0277
Type has Missing Trait Undefined Variable
180
250
B0425
27
E0433
Use of Undefined Crate, Module or Type
15
32
Others
28
33
TOTAL
670
555
20


Reviews:
[{"review_id": "olL_xAkUEd", "reviewer": "Reviewer_ikso", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For programming language translation tasks (e.g., C# to Rust), the paper augments the source language with intermediate language representation (IR) generated by LLVM. They show that such IR-augmented translation can significantly improve the SOTA. \nThey also showed two alternate design choices where IR is used for neural decompilation and as a pivot task. ", "summary_of_the_review": "The paper shows Intermediate Representation (IR) augmentation to source code can improve NMT. While this is a good direction to improve programming language translation, novelty-wise, it is rather thin. The empirical evaluation also requires some in-depth study.", "strength_and_weaknesses": "Strength:\n\n+ The paper is generally well-written and tries to address an interesting problem. The augmentation of IR during translation is a direct way to incorporate structures and some semantics into the code representation, and IR is generated by the compiler. This significantly cuts down the search space and improves the outcome.\n\n+ The paper evaluates low-resourced languages like Go and Rust. \n\n+ IR is also not needed for inference time.\n\n+ They propose alternative design and decompilation as another application.\n\nWeakness:\n\n- The paper is not super novel. It essentially shows augmenting data from a different information source helps to improve NMT. However, I also believe compilation-ware IR augmentation is a good direction toward programming language translation.\n\n- The evaluation is not in-depth. For example, in Tables 2 and 3, different settings (TLM, TAE, and MT) proved to be effective for different settings. There are no explanations for such behavior. Why certain objective functions are more effective in one language translation than another?\n\n- Low-resourced languages like Go and Rust are more similar to C/C++ than Java. I am wondering whether including Java can hurt their translations, especially for translating C++ to Rust and C++ to Go and vice versa. It would be good to understand the impact of different language designs on translation.\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ktxI_l28xdu", "reviewer": "Reviewer_d6WQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to improve automated program translation from unsupervised data by leveraging compiler intermediate representation (IR), a machine-code like format that is unified across many programming languages. Rather than train a model to translate each language into and from this domain, the work argues for using intermediate representation to augment inputs at training time in order to learn better representations, and omit it during inference. The result is a tool that performs somewhat better than a baseline without IR (TransCoder) without requiring the use of a compiler for inference.", "summary_of_the_review": "The proposed use of IR to improve unsupervised program translation is reasonably effective and goes beyond the naive (and ineffective) approach of using IR as a proxy for an intermediate domain for universal machine translation. The results are largely convincing, but suffer from a number of smaller issues that should be remedied to make the paper stronger.", "strength_and_weaknesses": "The main strength of this work is the observation that IR can be made useful for unsupervised program translation by extending the conventional unsupervised translation objectives to include program IR, _rather than_ attempting to map programs into and from IR directly (what the paper calls \"IR Pivot\"), which tends to yield poor performance. As such, the conceptual contribution is somewhat straightforward (adding several loss terms based on concatenating IR and code), but nevertheless novel and significant, especially when considering the rather substantial amount of effort invested to collect hundreds of thousands of Code/IR pairs.\n\nThe main weakness, to me, lies in the evaluation. For one, the work is somewhat unclear in its presentation of the results. The introduction to Section 5 states that all six objectives are used during training, which presumably means both the three TransCoder and three new objectives. That would lead me to read Table 2, from the \"TLM\" row onwards, as including all three TransCoder objectives by default, but some rows also include \"MLM\". Does that mean that no row here represents the combination of all six components, and that the \"TLM\" row works rather well using just a single objective? Furthermore, given that TLM and TAE are very similar, I would like to see the results of \"TAE + MT\" given that \"TLM + MT\" is presented. There are a few other issues here:\n- The \"to Java\" result has the wrong number bold-fonted (should be: \"TLM + TAE: 54.5\");\n- I would expect Katz'19 to be included in the comparison since it is cited as a closely related, if older, technique;\n- It is quite surprising that beam-size 5 yields _worse_ performance for \"to C++\" than using greedy decoding (may be worth double-checking this number);\n- The \"79%\" improvement of Java to Rust mentioned in the abstract appears to be relative to an MLM baseline (Tab. 3, App. A), not TransCoder. Is this supposed to be the full TransCoder model? If not, it seems off to cite this as such a large improvement given that the baseline is realtively weak compared to the de facto standard model, even if TransCoder did not evaluate with this setting.\n\nPlease aim to improve the above results. Given that the score improvement over TransCoder is modest on average (and quite small for some languages), it is important for judging the significance of this work that the results given are reliable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "K7DIE4Nzt-", "reviewer": "Reviewer_2BBL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper takes an interesting approach to training neural models for programming language (PL) translation - use of Intermediary Representations (IR): language-agnostic pseudocode that describes the semantics of the program. The paper proposes to augment code translation with IRs, specifically LLVM IR, to facilitate translations between C++, Java, Rust, and Go languages. The main idea of training neural sequence-to-sequence models relies on pretraining via translation language modeling (TLM), translation auto-encoding (TAE), and IR generation via machine translation (MT). The paper extends a prior dataset of parallel functions to Go and Rust languages. The experiment results show improvements in PL translation over the prior approaches.", "summary_of_the_review": "I would not prefer to see this paper accepted at the ICLR conference. The paper reads more like a technical report that describes a system using IR, which is interesting. The paper could go into a system conference or on a different track other than the main research track in an ML conference like ICLR. I am unsure if this paper passes the bar of a research paper of ICLR. However, I am not absolutely certain about my judgment, therefore, I am ready to be convinced otherwise.", "strength_and_weaknesses": "__Strengths__\n- The main proposal of the work is well-motivated. The related work discussions are very comprehensive.\n- The use of IR is interesting and seems new. While the idea seems straightforward, the underlying effort to build such a system proposed in this work is significant.\n- The paper extended an existing translation dataset to new languages with test cases.\n\n__Weaknesses__\n- The writing of the method section could be improved. It became confusing with the mentions of the TransCoder pretraining objectives and the methods used in this work (sections 5.1 and 5.2).\n- Table 1 shows the percentage of successful IR compilation is small. Is it worth pursuing such an approach to building PL translation systems?\n- In Table 2, rows 2 - 7 in the greedy decoding block (starting from TransCoder) show the performance difference across models is very small, but row 8 shows TLM + TAE + MT performs quite well. It is not clear why?\n- While the empirical performances show improvements, there is not enough analysis to emphasize what makes the proposed approach truly effective.\n- In the introduction, the paper mentions, \"Augmenting training data with the corresponding IR can benefit a Neural Transpiler in two ways: it helps align embeddings for different languages and improves the semantic understanding of the code.\" I do not see the authors validating this statement.\n- TransCoder has subsequent improvements in the form of DOBF and TransCoder-ST models. Why they were not mentioned in the empirical results? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JMbq9PuDXv", "reviewer": "Reviewer_DPxt", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses code translation. While learning to translate from language `A` to language `B` (i.e., `A<->B`), the paper proposes to also train on translating `A<->LLVM` and `B<->LLVM`. \nThis way, LLVM serves as a proxy language that both `A` and `B` translate to.\nAt test time, no intermediate language is required, and the model translates directly `A<->B`.\nThis way, the `A<->LLVM` and `B<->LLVM` training serves as an effective training time supervision.\nThe approach is demonstrated on the task of unsupervised translation, with 4 programming languages, and shows significant improvements over TransCoder.", "summary_of_the_review": "The paper is very clear, easy to read, and shows strong results over TransCoder using a simple and elegant approach.\n\nThe higher-level idea of transforming the \"unsupervised translation\" problem into a \"supervised translation to a proxy language\" is very novel and clever.\n\nThe proposed approach is appealing since the LLVM information it requires can be generated from the existing training data. \nEven if not all data can be compiled to LLVM, the authors show that a small amount of such supervised LLVM data is sufficient.\nFurther, at test time, the trained model can be evaluated without any additional information.\n\nOverall this is an excellent paper and I would be happy to see it accepted.", "strength_and_weaknesses": "## Strengths\n* The approach is simple and provides strong empirical results in unsupervised translation over TransCoder.\n* The paper basically transforms the \"unsupervised translation problem\" into a \"supervised translation into a proxy language\": Instead of learning an unsupervised translation between `A<->B`, the authors transform the task into a **supervised** translation between each of `A` and `B` languages to `LLVM` (that is, `A<->LLVM` and `B<->LLVM`). That is, they transform the unsupervised translation problem into a supervised translation to a proxy language. \nThe LLVM language serves as a proxy that every language can be **supervisedly** translated to,\nwhich is very clever and very novel in my opinion. \n* The main Section 5 is very clear.\n* At test time, the model does not require additional analysis / conversion / intermediate representation of the input program, which is a huge advantage compared to prior work that relies on compiler information. Much such prior work which leverages compiler/runtime information - requires that information at test time as well.\n* The evaluation is convincing and thorough, evaluating 4 languages and many objective ablations.\n\n## Weaknesses\n* Nothing serious, see the next section\n\n## Questions:\nWould it be meaningful to run the following ablation: the standard objectives (as TransCoder), but also include the LLVM data of the source and target languages (train on the LLVM as additional, separate examples from their C++/Java/Go original programs)?\n\nThe reason I am asking this is because the proposed approach is different from TransCoder in two aspects:\n1. Trains on LLVM as well\n2. Slightly different objectives TLM/TAE/MT.\n\nWhat other experiments could disentangle these two contributions?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%