Forklift: An Extensible Neural Lifter
Authors: Jordi Armengol-Estapé
Rodrigo C. O. Rocha
Jackson Woodruff
Pasquale Minervini
Michael O'Boyle
Venue: COLM (2024)
Keywords: Transformer, translation, transpilation, binary translation, compiler, software migration
TL;DR: We train a Transformer to translate legacy assembly code into LLVM intermediate representations to later recompile to the architecture of choice. We show that we can freeze the decoder and fine tune the encoder to continually support new ISAs.

Abstract: The escalating demand to migrate legacy software across different Instruction Set Architectures (ISAs) has driven the development of assembly-to-assembly translators to map between their respective assembly languages. However, the development of these tools requires substantial engineering effort. State-of-the-art approaches use lifting, a technique where source assembly code is translated to an architecture-independent intermediate representation (IR) — for example, the LLVM IR — and use a pre-existing compiler to recompile the IR to the target ISA. However, the hand-written rules these lifters employ are sensitive to the particular compiler and optimization level used to generate the code and require significant engineering effort to support each new ISA. We propose Forklift, the first neural lifter that learns how to translate assembly to LLVM IR using a token-level encoder-decoder Transformer. We show how to incrementally add support to new ISAs by fine tuning the assembly encoder and freezing the IR decoder, improving the overall accuracy and efficiency. We collect millions of parallel LLVM IR, x86, ARM, and RISC-V programs across compilers and optimization levels to train Forklift and set up an input/output-based accuracy harness. We evaluate Forklift on two challenging benchmark suites and translate 2.5x more x86 programs than a state-of-the-art hand-written lifter and 4.4x more x86 programs than GPT-4 as well as enabling translation from new ISAs.

Published as a conference paper at COLM 2024
Forklift: An Extensible Neural Lifter
Jordi Armengol-Estape"
Rodrigo O. Rocha Huawei Research
Jackson Woodruff School of Informatics
School of Informatics
University of Edinburgh
University of Edinburgh
jordi armengol. es estape@ed ac. uk
Pasquale Minervini
Michael F.P. O'Boyle
School of Informatics
School of Informatics
University of Edinburgh
University of Edinburgh
Abstract
The escalating demand to migrate legacy software across different Instruc- tion Set Architectures (ISAs) has driven the development of assembly-to- assembly translators to map between their respective assembly languages. However, the development of these tools requires substantial engineering effort. State-of- the-art approaches use lifting, a technique where source assembly code is translated to an architectur-indeeendeet intermediate representation (IR) - for example, the LLVM IR- and use a pre-existing compiler to recompile the IR to the target ISA. However, the hand-written rules these lifters employ are sensitive to the particular compiler and opti- mization level used to generate the code and require significant engineering effort to support each new ISA.
We propose Forklift, the first neural lifter that learns how to translate assembly to LLVM IR using token-level encoder-decoder Transformer. We show how to incrementally add support to new ISAs by fine tuning the assembly encoder and freezing the IR decoder, improving the overall accuracy and efficiency. We collect millions of parallel LLVM IR, x86, ARM, and RISC-V programs across compilers and optimization levels to train Forklift and set up an input/ ootoouubasseeeeeee accuracy harness. We evaluate Forklift on two challenging benchmark suites and translate 2.5x more x86 programs than a stfte-of-the-art hand-written lifter and 4.4x more x86 programs than GPT-4 as well as enabling translation from new ISAs.
1 Introduction
We are witnessing a rapid diversification of computer hardware to overcome the limits of technology scaling. This diversification has been accompanied by an increase in the number and variety of Instruction Set Architectures (ISAs) (Hennessy & Pattersso,, 2019). However, for the successful adoption of any new hardware, pre-existing legacy applications must be ported to their new ISAs (Pegoraro, 2023). Unfortunately, legacy software frequently exists only in binary form with no access to the original source code, preventing easy recompilation to new ISAs (Di Cosmo & Zacchiroli, 2017).
Binary translation solves this problem: taking binaries and translating them to new archi- tectures (Rocha et al., 2022; D' A'Atra et al., 2017). However, developing binary translation tools is difficult task that requires substantial engineering effort for each new ISA. For each new ISA, the same challenges must repeatedly be overcome, reconstructing informa- tion typically destroyed during compilation such as function signatures and pointers, and undoing architecture-specinn et al., 2002). These tasks are non-trivial to overcome compilers are not designed to be bidirectional - and must be redone for each new architecture.
steps such as register allocation and vectorization (Cifuentes
*Corresponding author.

---

Published as a conference paper at COLM 2024
Existing software (potentially legacy)
x88 5
arms
Deogram Norm
Recompiled, retargeted. optimized
addi Pardx, 4), %eee
add x8, x3, x10
add a1, a1. 32
software
Fine tune
Fine tune
ARM encoder
RISC encoder
encoder
Extensible neural lifter (Forkliit)
LLVM IR (frazen)
1% - now 22 %10, 1513
LLVM
Compiler
Compiler intermediate representations
Figure 1: Forklift lifts source x86, ARM, RISC-V code into the intermediate representations (IR) of the LLVM compiler. We leverage the LLVM's ability to compile to a wide range of target ISAs. As the target of Forklift is always LLVM IR, we freeze the decoder and fine-tune existing encoders to incrementally add support for new sources, maintaining accuracy on prior sources.
Lifting is the standard approach to this problem, where manually engineered tool lifts the source binary language into a shared compiler intermediate representation (IR) (Cifuentes et al., 2002). In computational linguistics terms, compiler IR can be thought of as a sort of interlingua (Alansary, 2014). The existing compiler infrastructure for this IR can then be used to re-optimize and lower the code to new binary format. While this removes significant barrier in binary translator design by reusing existing infrastructure, it still leaves the most challenging part of the problem unsolved. Critically, existing lifters require significant engineering efforts. They must overcome the information loss that compiling entails, and they are typically designed not just for single ISA but also for a compiler and optimization level.
In this paper, we tackle the heart of this engineering problem: how can we automatically learn lifting? We present Forklift, the first neural lifter, implemented as token-level encoder-decoder Transformer. We pose lifting as a sequence-to-sequens
task in which the model is directly fed with assembly instructions and is expected to generate the equivalent IR of the LLVM compiler. Unlike previous work in learnable binary translation (Lee et al., 2024), which translates directly to the target assembly rather than lifting, we sidestep the need to learn to generate code for every translation direction. Once Forklift lifts code to LLVM IR, we can leverage the existing power of LLVM to re-optimize and recompile to the desired target, directly benefiting from the LLVM ecosystem.
We collect million-scale parallel dataset of LLVM IR, x86, ARM, and RISC-V for several compilers and optimization levels to train Forklift and set up an 1/O-based accuracy harness.
We evaluate Forklift on two challenging benchmark suites against strong hand-written lifter, Lasagne (Rocha al., 2022), and a state-of-the-art Large Language Model (LLM), OpenAl's GPT-4 (OpenAl et al., 2024). Forklift outperforms existing approaches without requiring manual engineering effort, automatically learning the intricacies of each ISA, compiler, and optimization level. On optimized x86 code, Forklift is 2.5x more accurate than Lasagne and 4.4x more accurate than GPT-4. Forklift is able to perform on new ISAs in a way traditional tools are not. Furthermore, we lift all assembly targets to LLVM IR, we freeze the decoder and incrementally fine-tune the encoder for each new architecture. This
Code will be made available at https://gi wegithuu..ccoom htanub.coomjoordiaeffok....
2

---

Published as a conference paper at COLM 2024
incremental learning enables parameter sharing in the decoder and transfer learning in the encoder, yielding improved accuracy and efficiency.
2 Background
LLVM IR The LLVM project (Lattner & Adve, 2004) is an industry-standard compiler infrastructure. It is designed around an intermediate representations language, LLVM IR, which is a generic static single-assignment (SSA) low-level language akin to assembly. Language developers need only translate their programs into LLVM IR to make use of the pre-existing code-gentn sede-generation backends. LLVM supports all ISAs and has a extensive set of optimizations available. LLVM first applies the desired optimization passes to the IR, and then lowers it to the assembly language of choice.
Neural code translation Neural code translation (Chen et al., 2018; Roziere et al., 2020) the task of learning a neural network model T that takes as input a unit of code (e-g. function) in a given language A and outputs an equivalent program in the desired target language, B:
T: A+ B
(1)
Each instance of A, B is represented as sequence of tokens. [ap...aal anl A, B (b.....bu E B. For the translation between some source E A and some target code E B to be correct, we require a and b to be semantically equivalent:
M M[a = M [b]
(2)
with M, M being functions denoting the meaning of programs in A and B, respectively. While undecidable in general (Winskel, 1993), practice, there are looser bounds of equiva- lence such observational equivalence on finite subsets of input/outpin pairs.
Incremental learning Standard machine learning approaches train models on a given dataset and then remain frozen. However, the model's information can quickly become outdated, or need to be adapted to a new task. A solution is fine tuning the original model with new data. However, this leads to catastrophic forgetting (McCloskey & Cohen, 1989), a phenomenon in which we see a drop in the model's performance on the original task or data it was originally trained model for each new task, or fine tune with a mix of data including the original dataset. Both options are impractical. In incremental learning (van de Ven et al., 2022; Wang et al., 2024) settings, the machine learning system is explicitly designed to incrementally support new tasks, classification tasks, or domains the system requirements are updated. For example, adding support for new languages to multilingual machine translation system (Escolano et al., 2019).
Thus, one is required to either create a copy of the
3 Extensible Neural Lifting
Our goal is to port legacy assembly to new ISAs in scalable manner. We assume that we have access to C compiler for all the source ISAs of interest and to an L.VVM-based compiler (Clang) for the target ISA, which is the case in practice. We take functions the basic unit of translation, following previous work on code translation (Lachaux et al., 2020).
We present our method, Forklift, in four key steps. First, we describe how to generate machine learning-scall data to train our model (Section 3.1). Second, we discuss the critical design choice of which specific IR to lift to (Section 3.2). Third, we describe the architecture of Forklift (Section 3.3). Finally, we propose a method for continually adding new ISAS to Forklift (Section 3.4).
3

---

Published as a conference paper at COLM 2024
3.1 Data generation
Our method requires access to a large C dataset, attainable by the crawling of publicly available repositories. We also require the dataset to be at the function-level, and parallel with the corresponding assembly and LLVM IR functions. However, such functions are rarely compilable. They typically need additional context (inclusion of headers, type definitions, constant definitions). Thus, following da Silva et al. (2021)'s methodology, we take a function-level crawling augmented with synthetic dependencies generated with type inference to make them compilable (Armengol-Estann et al., 2022).
Previous work, however, lacks a parallel dataset with diverse assembly code. We, therefore, augment the dataset in Armengol-Estape et al. (2022) with the corresponding LLVM IR and RISC-V code. We also add assembly generated by two compilers at three optimization levels 00, 03, Oz. The resulting training dataset features between 2.7 and 3.3M parallel functions depending on the specific target after deduplication.
3.2 Target IR
We wish to select single larget that a) is portable and b) aids learning.
IR language LLVM IR fits the portability requirement. It is designed to capture all op- erations provided by traditional processors, while working at a higher level in a machine- independent manner (Lattner & Adve, 2004). has low semantic gap to hardware but is generic. Portability is provided by the LLVM compiler framework, which allows programs compiled to the LLVM IR to be recompiled any target with an existing backend, a feature that existing lifters frequently take advantage of (Rocha et al., 2022).
For the same function, LLVM can emit semantically equivalent IR in different forms. With the 00 setting, Clang will emit unoptimized IR, directly translated from the source code; with 03, the compiler will emit IR optimized for execution time, and with Oz, the compiler will emit IR optimized for code size.
IR optimization level
We observe that Oz presents several advantages in terms of easing the learning task. Firstly, its more compact nature leads to shorter sequences for the model. Secondly, 0z transforma- tions act as a normalizer. However, in terms of portability, we require the selected IR to be recompilable to the desired optimization level. We exploit the fact that LLVM infrastructure provides many transformations and optimizations. This is empirically confirmed in our benchmarks, where we observe that source functions that were automatically vectorized when compiled with 03, are also often vectorized when first lifted targeting Oz and later recompiled with 03. In Appendix G, we provide examples of these reoptimizations.
Thus, we conclude that our lifting target to be LLVM IR Oz. In section 5, we will see the empirical confirmation of this choice.
3.3 Neural lifting
We pose neural lifting sequence-fo-sequenec task. Given dataset D with N pairs {(S, Li]l E {O..N -1 } where Sn is a function assembly code and L, is the corresponding compiler IR code that equivalent to it, we want to train model parameterized by 0 with maximum likelihood estimation:
N N-1
0' arg max
I P(L 1=0
In practice, we minimize the negative log- lieeliikl.hood.
Model We opt for an encoder-decoder Transformer (Vaswani et al., 2017a) using the architectural modifications in BART (Lewis et al., 2019). The first model is trained from
4

---

Published as a conference paper at COLM 2024
scratch, without any pretraining. We do not fine tune from an existing LLM due to the additional computational cost not being justified given the poor performance of code LLMs on low-level code observed in Lee et al. (2024). If an LLM pretrained on LLVM IR was available, 2 one could opt for starting the training from the LLM as the decoder, and randomly initialized encoder attached to the decoder via randomly initialized cross- attention modules added to the decoder. Our method is orthogonal to LLM improvements.
Tokenization For the vocabulary, we train Unigram tokenizer (Kudo, 2018) of 16k tokens. We split individual digits and apply domain-specific normalization (such as space and end- of-line normalization, unnecessary assembly boilerplate removal). 3 We note that, unlike assembly languages, LLVM IR has native struct type for user-defined types, like C. This means that the model will learn to generate structs from code that does not have structs, which can lead to complications for the model. In Armengol-Estape problem was solved with type inference. However, since lifting, unlike decompilation, does not require realistic type names, we instead opt for simplifying the task to the model by normalizing struct names.
al. (2024), a similar
Verifier We need method to verify that a given low-level translation is correct. We opt for observational equivalence, i.e., unit tests. While unit tests don't formally guarantee semantic equivalence, in practice passing unit tests with good coverage strongly correlates with correctness. Due to the imperative, low-level nature of the code we tackle, we implement code evaluator that aside from return values, also checks for side effects on arrays, struct pointers, and global variables. We do not check for side effects involving system calls. We assume that the desired functions have reference unit tests, which is an increasingly realistic assumption given the proliferation of automatic unit test generation methods (Kind et al., 2022; Siddiq et al., 2024).
3.4 Extensibility
We tackle incrementally multi-source neural lifting, that is, we need to incrementally support multiple assembly languages. In this work, we start training an x86 lifter, as x86 is mainstream ISA. We, then, freeze the decoder, copy the x86 encoder and fine-tune it for ARM. Afterwards, for new architecture, e.g., RISC-V, we proceed similarly starting from the previous ARM encoder, motivated by the similarities between ARM and RISC-V.
4 Experimental framework
To evaluate the effectiveness of our approach, we train neural lifters both from scratch and with the extensibility approach outlined in Section 3.4. We evaluate the models against state-of-the-art LLM, ChatGPT4 (OpenAI et al., 2024), and strong handwritten baseline, the lifter used in Lasagne (Rocha et al., 2022) using input/ output accuracy.
Baselines As our first baseline, we use the lifter in Lasagne (Rocha et al., 2022), which is state-of-the-art hand-written binary translator built by extending the LLVM-Mctoll lifter (Yadavalli Smith, 2019). Since this lifter is coupled with a disassembler (similar to Unix's objdump), we provide it with an object file. As an LLM baseline, we use the most recent version'
4 of GPT-4 (OpenAl et al., 2024). We provide GPT-4 with the textual assembly of the function, and experiment with different prompts.
Evaluation We provide each translation system with the source assembly function. We then take the output LLVM IR, compile it back, and call the function from C interface with
2ffer the submission of this article, an LLM pretrained on LLVM IR was released (Cummins et al., 2024). In Section 6, we discuss their results and how they compare to Forklift's.
3 Unigram was shown to outperform BPE Bostrom & Durrett (2020), but it out of scope of this
work to perform an ablation study on tokenization strategies.
4 Most recent version of ChatGPT4 as of March 2024.
5

---

Published as a conference paper at COLM 2024
X86
ARM
RISC-V
Lifter
Exe
Synth
Exe
Synth
Exe
Synth
Lasagne
30.26%
37.86%
1.03%
0,97%
GPT-4
17.07%
0.95%
22.74%
0.97%
16.11% 71.61% %
3.37%
Forklift 75.57%
51.46%
75.31%
67.96%
67.42%
Table 1: Result summary. The columns represent input-output accuracy for each ISA on each benchmark set for Clang optimized assembly (O3) translation into LLVM IR. Lasagne does not support RISC-V, and has limited support for ARM. On ExeBench in x86, Forklift outperforms Lasagne by 2.5x, and GPT-4 by 4.4x.
the original context of the function. We then check for input/ output accuracy using the unit tests from the corresponding evaluation benchmarks, following the verifier implementation described in Section 3. For our models, we sample with beam search and evaluate the top 5 hypotheses.
Benchmarks We evaluate our approach on two benchmarks suites: the program synthesis benchmark in Collie et al. (2020), Synth, with a curated set of features in C (e.g,, numeric arrays and strings) and a subset of 512 functions ExeBench (Armengol-Estape et al., 2022), scrapped from public repositories, to allow wider-scale evaluation and interaction with user-defined types and external function calls. Synth has the interesting characteristic of being rich in automatically vectorizable functions, i.e., loops that the compiler automatically vectorizes when using e.g. the O3 optimization code. While faster, this further complicates recovering the original LLVM IR. For realistic, yet challenging evaluation, for all ISAs, we evaluate on assembly generated with the O3 optimization level, considerably more complex than unoptimized, O0. To avoid train-evaluattin overlaps, the benchmarks in Synth were already filtered out from the train set in Armengol-Este original dataset; the same work performed exact C token-level decontamination to minimize train-test overlaps. In this work, we apply additional deduplications using assembly tokens rather than C tokens (unlike Armengol-Estan --sapp-téée et al. (2022) to search for duplicates, given the fact that different C functions could be lowered to identical assembly.
et al. (2022) in the
Implementation We implement our models in PyTorch (Paszke et al., 2017) using Hugging- face Transformers (Wolf et al., 2020). We build an encoder-decoder Transformer (Vaswani et al., 2017a) with the architectural modifications in Lewis et al. (2019), for a total of 153M trainable parameters. We use a context window of 2048 tokens for both source and target sequences. For further details please see Appendix A.
5 Results, discussion, and analysis
Table shows the overall performance of the lifters across source ISAs and benchmark suite when translating between optimized assembly (O3), generated by the Clang compiler, into LLVM IR. Forklift consistently performs well across the ISAs with a stable accuracy on Exebench of 71 75%. Lasagne has at best half this accuracy, achieving this only on x86. The Lasagne ARM port is considerably weaker, demonstrating the difficulty of manual lifters. GPT-4 is not competitive in this task, highlighting the intrinsic difficulty of lifting and hinting at a scarcity of low-level code in mainstream LLMs pretraining data. However, obtains non-trivial performance on Exebench, ranging between 16% and 22%.
The Synth dataset is especially challenging to lift from optimized assembly, considerable amount of the loops are vectorized. Vectorized assembly is typically longer and difficult to get right (custom vector instructions, padding, etc). GPT-4 especially struggles in this benchmark. Instead, Forklift achieves accuracies between 51% and 67.49 67.42%. The perfor- mance is the lowest on x86, x86's vectorizations typically require more instructions than their ARM counterparts.
'The first 512 functions in the test set, to minimize the overhead incurred in the executing the functions and running the different systems.

---

Published as a conference paper at COLM 2024
In the remainder of this section, analyze the impact of the a) learned LLVM IR target, b) incremental learning, c) the original compiler, and d) prompting strategies for GPT-4. Finally, perform error analysis and discuss the limitations of our approach. Appendix D provides additional results.
5.1 What's the optimal LLVM IR target to learn?
REPRESENTATION
EXEBENCH
SYNTH
x86 O3
467 1079
1089 + 745
ARM O3
540
LLVM IR O0
1314 1729
934 1 465
LLVM IR O3
956 + 1866
2470 1811
LLVM IR Oz
814 1014
458 + 194
Table 2: Mean length and standard deviation (model tokens). Note the considerable variation.
Table shows that while LLVM IR is generally more verbose than x86 and ARM assembly, LLVM IR Oz is the most compact representation among other IRs, and it is even less verbose than ARM and x86 on Synth, due to the presence of vectorizations, which requires much setup code. Table 3 shows the evaluations of neural lifters from x86 using different IR optimization levels, compared almost on par with the IR O3 one on ExeBench, but it is dramatically better on Synth. This, together with the fact that IR can be recompiled to the ISA of choice, and that vectorized assembly can be regenerated from IR Oz makes IR Oz the preferred target for Forklift.
direct translation from x86 to ARM. The IR Oz model is
TRANSLATION x86 O3 + ARM O3
EXEBENCH
SYNTH
71.89%
26.21%
x86 + LLVM IR
63.53%
32.04%
x86 03 + LLVM IR O3
75.93%
8.74%
x86 O3 - LLVM IR Oz
75.57%
51.46%
Table Direct and best lifting target experiments.
5.2 How does incremental learning and decoder sharing affect the results of Forklift?
Table 4 shows the performance of Forklift (with the extensibility methodology described in Section 3.4 and decoder sharing) across ISAs. As baseline we compare to models trained from scratch. The incremental approach has the advantage of smaller parameter count due to the shared decoder. However, we observe additional advantages. Firstly, there is a a faster convergence (see appendix for details), which would help in scenarios with restricted data or compute. Secondly, accuracy slightly improves on x86 ARM and significantly improves RISC-V, which we attribute to the regularization effect of the fixed decoder and the transfer learning of the encoder.
ARM
RISC-V
Lifter
Exe
Synth
Exe
Synth 24.72%
From scratch Forklift
71.15%
67 %6
63.78%
75.31%
67.96% 77.96%
71.61%
67.42%
Table 4: Incremental learning. The columns represent the accuracy for each approach when lifting ISA to LLVM IR Oz across benchmark sets. From scratch trains a new en- coder deder/decoder for each ISA. Forklift has fixed decoder. The encoder for ARM Forklift is pretrained with the x86 encoder and RISC-V Forklift encoder is pretrained with the prior ARM Forklift encoder.
6See examples in the Appendix Appendix B explains the optimization levels.
7

---

Published as a conference paper at COLM 2024
Table 5 shows comparison between full fine-tuning approach the frozen decoder approach (Forklift). There is no clear winner in terms of accuracy, with one model being slightly better on Synth and the other being slightly better ExeBench. We prefer the frozen decoder one due to the decoder parameter sharing, leading to lesser overall parameter count.
MODEL
ExE
SYNTH
Neural full-FT
72.37%
69.90%
Forklift
75.31 %
67.96%
Table 5: Full fine-tuning vs frozen decoder ARM.
5.3 Can we adapt to new compilers?
Hand-written lifters can be sensitive to the original compiler that generated the assembly. Lasagne was originally evaluated with Clang and its performance significantly drops when using GCC-generan code (see Appendix, Table 11). Following similar approach for extending Forklift ARM and RISC-V with Clang, we fine tune the original x86 encoder with GCC- -CC--genertee assembly, and outperform Lasagne (Table 6).
MODEL
EXE SYNTH
Lasagne 22.64%
35.24%
Forklift
64.97%
53.85%
Table 6: Compiler adaptation results: Evaluation on x86 03 compiled with GCC.
5.4 How does prompting affect GPT-4's performance?
Table 7 shows the performance of GPT-4 with different prompting strategies. First, we simply ask GPT-4 to translate the assembly function into the corresponding LLVM IR. We observe that GPT-4 especially struggles when literally translating architectural intrinsics, Thus, we explicitly ask the model not to use intrinsics and come up with the original LLVM IR code. We also experiment with one-shot learning, which slightly degrades performance. Including prior experiments implies feeding costly long sequences to the model. We use the best-performing prompt, Prompted no intrinsics strategy, for the rest of the experiments.
Prompt
Exe
Synth
Simple prompt
12.87%
0.95%
Prompted no intrinsics
17.07%
0.95%
Prompted no intrinsics One-shot
12.28%
0.95%
Table 7: GPT-4 performance on Synth lifting from x86, breakdown.
5.5
Error analysis
Table shows the correlations between different features (e.g., compilability of the predicted translation) and the input/ output accuracy of each system on x86. We refer to Appendix for similar tables on other architectures.
We observe that the main feature driving Lasagne failure is vectorization (vectorized) Synth, and global variables thas_global on ExeBench. The correlation with compilability (compiles) of generated code is strong Synth but weaker on ExeBench. This suggests that on Synth, when Lasagne's LLVM IR compiles, it is likely to be correct. However, on ExeBench, we encounter runtime crashes in many cases, leading to a lack of input/ output accuracy even when it compiles. In Appendix E, we provide additional results and analysis for Lasagne.
Forklift's imput/outuut accuracy generally correlates with compilability, which is an interesting feature because it minimizes the unreliability of having compilable yet subtly
8

---

Published as a conference paper at COLM 2024
incorrect translations. The length of the original C code (c (c_length) is negatively correlated with input-output accuracy, suggesting an increased difficulty with longer sequences.
Synth
ExeBench
Lasagne
Forklift
Lasagne
GPT4
Forklift
compiles
0.94
0.77
0.25
0.70
0.86
edit_sim
0.29
0.50
0.26
0.46
0.43
c_length
-0.14
-0.31
-0.10
-0.23
-0.44
Lun_args
-0.36
-0.09
0.22
-0.21
-0.18
has float
-0.10
-0.04
0.08
-0.02
-0.08
has has_strings
0.28
-0.08
0.05
-0.04
-0.09
vectorized
-0.75
0.12
has has_glos has_globals
-0.87
0.14
-0.05
Table 8: Pearson correlations between different features and the input/output accuracy of each system on Synth and ExeBench with Clang O3 for x86; compiles (does the code generated by the system compile?); edit_sim, the normalized edit similarity (i.e., 1 edit_dist, with edit_dist normalized by length) of the generated code with respect to ground truth LLVM IR; c_length, the character length of the original C function; n_ f_ffu_args, the number of arguments of the function; has_float, whether the function has floating point types; has__strile str_ns, whether the function deals with strings; vectorized whether the function's s assembly is vectorized; and has globals,,,, whether the function uses global variables. GPT-4 correlations for Synth are omitted due to its almost 0.0% accuracy on this benchmark. Similarly, the vectorized column is omitted from ExeBench due to its sparsity on its test set, and lasball is omitted from Synth as its functions have no global variables.
Limitations
We note the following limitations of our system. Firstly, tackle functions with maximum length of 2048 tokens as opposed to full programs with unbounded lengths. Secondly, Forklift is fed with textual assembly, for which assume the presence of correct disassembler, which is a typical assumption in the literature (Hosseini & Dolan-Gavitt, 2022). Thirdly, Forklift does not formally guarantee the semantic equivalence of its translations. Fourth, for the incremental learning strategy we start from the intuition that the encoder essentially in charge of understanding the source assembly, while the decoder mainly concerned with IR generation; however, since the system is trained end-to-end, both roles might be more intertwined. Finally, this work was performed under limited compute budget, and we were not able to pretrain an LLVM IR LLM to work the decoder.
6 Related work
Binary translators and lifters A large number of binary translators exist for a wide range of tasks, from optimizing code (Bruening, 2004; Bansal & Aiken, 2008) to instrumenting code (Nethercote & Seward, 2007; Luk et al., 2005). Common approaches are to do this statically (Engelke Schulz, 2020; Reidel et al., 2021), where binaries are translated ahead of time, and dynamically (Bellard, 2005; Altman & Ebcioglu, 2000), where binaries are translated on the fly. However, these approaches are challenging port to new architec- tures (Hazelwood & Klauser, 2006; Chen et al., 2013; Bansal & Aiken, 2008). LLVM IR has been used as a target-independent language for many lifters (Shen et al., 2012). Lasagne (Rocha et al., 2022) introduces state-of-the-art x86 + LLVM IR lifter by extending the industry-grade LLVM-McToll lifter, and leverages this lifter to build an effective static binary translator x86 ARM.
Neural approaches Transformer-based (Vaswani et al., 2017b) approaches are state-of-the the- art in code translation (Lachaux et al., 2020), and LLMs (Brown et al., 2020) have further improved (Chen et al., 2021a). However, even code-specific LLMs still struggle with low- level languages (Lee et al., 2024), developers don't typically upload assembly to the

---

Published as a conference paper at COLM 2024
Internet. Rather, low-level representations of code are implicitly present codebases. Domain-speccfic Transformers have been trained in decompilation settings (Chen et al., 2021b; Armengol-Estape et al., 2024; Tan et al., 2024). Unlike decompilation, lifting is more concerned with obtaining accurate and fast portable code rather than readability user-level language.
More recently, Cummins et al. (2023) trained an LLM to predict LLVM IR for code size optimization. However, the model cannot handle assembly language and is not publicly available. Guess&Sketch (Lee et al., 2024) tackled RISC-V <> ARM with an encoder-decoder Transformer augmented with symbolic engine fix partially correct translations. However, they evaluate on unoptimized assembly and are limited by the quadratic complexity of all translation directions. Our work is orthogonal to theirs and Forklift could benefit from the symbolic augmentation, which in our case would involve less engineering work to implement due to only having to support one target language (LLVM IR). Starcoder2 (Lozhkov et al., 2024) is another recent LLM work that includes LLVM IR in the training set, albeit the IR is not paired with the corresponding assembly, and the overall proportion of runtime-optimized assembly is tiny. Relatedly, Ivanov et al. (2024) introduced large executable LLVM IR dataset based on Grossman et al. (2024)'s compilable dataset.
After the submission of this article, Cummins et al. (2024) introduce pretrained LLMs LLVM IR and showcase (x86, ARM) lifting as one of their fine-tuning use cases. While the larger scale of their models shows promise (albeit at higher cost), direct comparison with our approach is difficult due to the different granularity (functions vs. files) and metrics used. For example, forklift obtains better compilability (64.08%-80.88%% than the Meta LLM Compiler (49,4%), but in different setups. Additionally, unlike Cummins et al. (2024), Forklift evaluates on assembly optimized for execution time and reports input-output accuracy. As future work, it would be interesting to combine both approaches (initializing Forklift's decoder with the weights in Cummins et al. (2024)).
Incremental learning and interlingua machine translation Escolano et al. (2020) propose jointly training a multilingual translation system with per-language encoder and decoder, resulting in system with shared latent space but no sharing of parameters, inspired by interlingual machine translation (Alansary, 2014). Then, they can incrementally add languages by freezing an existing encoder-decoder pair and training the new ones. Unlike natural languages, assembly languages do have explicit, complete interlingua available, LLVM's intermediate representations. Thus, we can have a single frozen decoder and incrementally add new encoders which in our case we, unlike Escolano et al. (2020), fine tune from existing encoders rather than from scratch. Interestingly, Szafraniec et al. (2023) also exploit the interlingual nature of LLVM IR, but as a pivot language for improving the training of code translation models between source languages.
7 Conclusions
This paper introduces Forklift, the first neural lifter. It effectively translates from optimized assembly into generic compiler intermediate representations. This enables porting legacy applications between ISAs, and reoptimizing them the new architecture, exploiting existing compiler infrastructure. Forklift outperforms Lasagne, a state-of-the-art hand- written lifter, and GPT-4, strong general-purpose LLM, and capable of efficiently supporting new TSAs. As future work, we suggest replacing Forklift's decoder by an LLM pretrained on LLVM IR, and integrating symbolic approaches.
Acknowledgments
This work was partly sponsored by Huawei Research UK. We thank the reviewers for their comments and suggestions.
As shown in Table in the Appendix D.
10

---

Published as a conference paper at COLM 2024
References
Sameh Alansary. Interlingua-based machine translation systems: Unl versus other inter- linguas. The Egyptian Journal of Language Engineering, 1, 01 2014. doi: (221168/ejje.2111
59863.
Erik R Altman and Kemal Ebcioglu. Full system binary translation: RISC to VLIW. IBM Research Report, (RC23262), 2000.
Jordi Armengol-Esstap,, Jackson Woodruff, Alexander Brauckmann, José Wesley de Souza Magalhaes, and Michael F. P. O'Boyle. Exebench: An ml-scale dataset of executable functions. In Proceedings of the 6th ACM SIGPLAN International Symposium on Ma- chine Programming, MAPS 2022, pp. 50-59, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392730, doi: 10.145/3220312.534466 URL https://doi, org/10. 10000.114400 /111/11233111.11111 3534867.
Jordi Armengol-Estape, Jackson Woodruff, Chris Cummins, and Michael F. O'Boll. Slade: A portable small language model decompiler for optimized assembly. In 2024 IÊEE/ACM International Symposium Code Generation and Optimization (CGO), pp. 67-80. Association for Computational Linguistics, 2024.
Sorav Bansal and Alex Aiken. Binary translation using peephole superoptimizers. OSDI, 2008.
Fabrice Bellard. QEMU, fast and portable dynamic translator. Usenix, 2005.
Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pre- training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4617- 4624, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1 v 10000..indings--mml.... 414.
URL
org/ org/2020. findings-emnlp.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/ 2/////005..1116 2020. URL https: //tp:/v../vvv/aa/aooeeooaio http:/vww.org/b///00555//// 14165.
Derek L Bruening. Efficient, Transparent, and Comprehensive Runtime Code Manipulation. PhD thesis, 2004.
Jiunn-Yeu Chen, Wuu Yang, Wei-Chung Hsu, Bor-Yeh Shen, and Quan-Huei Ou. On static binary translation of ARM/Thumb mixed ISA binaries. CASES, 2013.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107 2107 /tts//200.337/ 03374,, 2021a. URL https: haaps://aaa //rxxvaxvvvv////////////////iia www.v.org//oo org/abs/ 2107. 0107.03374.
11

---

Published as a conference paper at COLM 2024
Qibin Chen, Jeremy Lacomis, Edward Schwartz, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. Augmenting decompiler output with learned variable names and types. CoRR, abs/ 2tts//218..00 21666063338,, 2021b. URL https://arxiv wor.ii....///aaaee htrg//b////// /200. 06363.
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program trans- lation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp. 2552-2562, Red Hook, NY, USA, 2018. Curran Associates Inc.
Cristina Cifuentes, Mike Ven Emmerik, Norman Ramsey, and Brian Lewis. Experience in the design, implementation and use of retargetable static binary translation framework. (SMLI TR-2002-1055) 2002.
Bruce Collie, Jackson Woodruff, and Michael F. P. O'Boyle. Modeling black-box components with probabilistic synthesis. In Proceedings of the 19th ACM SIGPLAN International Con-
ference on Generative Programmiing: Concepts and Experiences, GPCE 2020, pp. 1-14, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978145038174 doi: /tt.Lntss//.i...r/////////////// 1110////...1//325899988 44114/332288988.2665222.2 3426952.
01111453342898.8442222
URL httpps:///oi. wrw.1/......./...//
Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Kim Hazelwood, Gabriel Synnaeve, and Hugh Leather. Large language models for compiler optimization, 2023.
Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste
Roziere, Jonas Gehring, Gabriel Synnaeve, and Hugh Leather. Meta large language model compiler: Foundation models of compiler optimization, 2024. URL https:///rrkt
/rggabs//// 2407 .05524.
A. F. da Silva, B. C. Kind, J. W. de Souza Magalhães, J. N. Rocha, B. C. Ferreira Guimarães, and F. M. Quinão Pereira. Anghabench: A suite with one million compilable e benchmarks for code-size reduction. In 2021 IEEE/ACM International Symposium on Code Generation and Opfimization (CGO), pp. 378-390, 2021. doi: 10.1109/CG05) /t////CCO5511111 202211999..222.3303322
Amanieu D' D'Antras, Cosmin Gorgovan, Jim Garside, and Mikel Luján. Low overhead dynamic binary translation arm. SIGPLAN Not., 52(6):333--346 jun 2017. ISSN 0362- 1340. doi: 10.1145 31555/31110777.63371... URL
10. 1111/11411111 40587. 3062371.
Reoberto Di Cosmo and Stefano Zacchiroli. Software heritage: Why and how to preserve software source code. International Conference on Digial Preservation, 2017.
Alexis Engelke and Martin Schulz. Instrew: Leveraging LLVM for high performance dynamic binary instrumentation. Virtual Execution Environments, 2020.
Carlos Escolano, Marta R. Costa-jussà, and José A. R. Fonollosa. From bilingual to multi- lingual neural machine translation by incremental training. CoRR, abs /1///990000033. 2019, 2019. URL http:///axxx org/abs// 00735.
Carlos Escolano, Marta R. Costa-jusså, José A. R. Fonollosa, and Mikel Artetxe. Training multilingual machine translation by alternately freezing language-specific encoders- decoders. CoRR, abs/ abs/2006.01594, 2020. URL htttp:://aaaiii / http:///aaaaiv.oo////////00 01594.
Aiden Grossman, Ludger Paehler, Konstantinos Parasyris, Tal Ben-Nun, Jacob Hegna, William Moses, Jose M Monsalve Diaz, Mircea Trofin, and Johannes Doerfert. Compile: large ir dataset from production sources, 2024. URL https: /ttp///xxvvvvve//////eee////aevl org/ argg/bbs/2099... 15432. 1532.
Kim Hazelwood and Artur Klauser. A dynamic binary instrumentation engine for the ARM architecture. CASES, 2006.
John L Hennessy and David A Patterson. A new golden age for computer architecture. Communications of the ACM, 2019.
Iman Hosseini and Brendan Dolan-Gavitt. Beyond the c: Retargetable decompilation using neural machine translation. In Proceedings 2022 Workshop on Binary Analysis Research, BAR 2022. Internet Society, 2022. doi: 10.14722/bbr. 2302,, URL http: //dp. doi. org/10. 14722/bar 202.2, 23809.
12

---

Published as a conference paper at COLM 2024
Ivan R. Ivanov, Joachim Meyer, Aiden Grossman, William S. Moses, and Johannes Doerfert. Input-gen: Guided generation of stateful inputs for testing, tuning, and training, 2024. URL https:://axxiv.org/bb/////
Cecilia Conde Kind, Michael Canesche, and Fernando Magno Quintao Pereira. Jotai: methodology for the generation of executable c benchmarks. Technical Report 02-2022, Universidade Federal de Minas Gerais, 2022.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. CoRR, abs 1004, 2018. URL http://arxia org/abs/ 1804. 10959.
Marie-Anne Lachaux, Baptiste Rozière, Lowik Chanussot, and Guillaume Lample. Un- supervised translation of programming languages. CoRR, abs/2006.03511, 2020. URL https://arxiv.. horg/abs//000/ 03511.
C. Lattner and V. Adve. Llvm: compilation framework for lifelong program analysis and transformation. In International Symposium on Code Generation and Optimization, 2004. CGO 2004., pp. 75-86, 2004. doi:
Celine Lee, Abdulrahman Mahmoud, Michal Kurek, Simone Campanoni, David Brooks, Stephen Chong, Gu-Yeon Wei, and Alexander M. Rush. Guess & sketch: Language model guided transpilation, 2024.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension. CoRR, 19100.911.,3411,, 2019. URL http://arxit: http::/arxiv.oor/abb///s0
13461.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024. URL
wwwprriii.ooo///ooaeeeeeee wrtp 191777
19173.
Chi-Keung Luk, Robert Cohn, Robert Muth, Harish Patil, Artur Klauser, Geoff Lowney, Steven Wallace, Vijay Janapa Reddi, and Kim Hazelwood. Pin: Building customized program analysis tools with dynamic instrumentation. PLDI, 2005.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motication, pp. 109-165. Academic Press, 1989. doi: https:/ wttp:ooorr.........//////////00/
URL https: //ww. www.sciieiiiceee rect. com/ wttttttccccccccccceeeeeecetetcc
Nicholas Nethercote and Julian Seward. Valgrind: A framework for heavyweight dynamic
binary instrumentation. PLDI, 2007.
OpenAT, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke
13

---

Published as a conference paper at COLM 2024
Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Eukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista P'araccandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Iseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Rob Pegoraro. Windows arm report card: 11 years in and still not bulls-eye. PCMag UK // Windows 10, 2023. Available at https: pcmag 00s109/windwwwss aa0---rm---pppe -oapoos------------------------- htt.rrrrrrrrrr.......aaaaaaaaaaa
Samuel Reidel, Fabian Schuiki, Paul Scheffler, Florian Zaruba, and Luca Benini, Banshee: A fast LLVe-based RISC-V binary translator. ICCAD, 2021.
Rodrigo Rocha, Dennis Sprokholt, Martin Fink, Redha Gouicem, Tom Spink, Soham Chakraborty, and Pramod Bhatotia. Lasagne: A static binary translator for weak memory model architectures. PIDII, 2022.
14

---

Published as a conference paper at COLM 2024
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsu- pervised translation of programming languages. Advances in Neural Information Processing Systems, 33, 2020.
Bor-Yeh Shen, Jiunn-Yeu Yeu Chen, Wei-Chung Hsu, and Wuu Yang. LLBT: An LLVM-based static binary translator. CASES, 2012.
Mohammed Latif Siddiq, Joanna C. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fah- mid Al Rifat, and Vinicius Carvalho Lopes. Using large language models to generate junit tests: An empirical study, 2024.
Marc Szafraniec, Baptiste Roziere, Hugh James Leather, Patrick Labatut, Francois Charton, and Gabriel Synnaeve. Code translation with compiler representations. In The Eleventh International Conference Learning Representations, 2023. URL https://openreview.nee
Hanzhuo Tan, Qi Luo, Jing Li, and Yuqun Zhang. LIm4decompile: Decompiling binary code with large language models, 2024. URL https://arxiv.. httpaaawwwi...r///////////// 05286.
Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three types of incremental
learning. Nature Machine Intelligence, (112)1185-1197, Dec 2022. ISSN 2522-5839, doi: 10.1038/ s42256-022-00056- URL https::tp///t 2-0.:aaoooo.........0/02220--000 00013///42225--22-005--- 3.
1038/
Ashish Vaswani, Noam shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, and Lukasz Kaiser. Attention is all you need. 2017,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1776.00372. 2017b. URL htttp://arxiv.orr/aa/////00 03762.
Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey continual learning: Theory, method and application, 2024.
Glynn Winskel. The formal semantics of programming languages: An introduction. 1993.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Syl- vain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State- of-the-art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653 / v1 v/// w1tp022..emm/////.. 2000.emmmppddmmoo..6. org//020..... emnlp-demos.6 6.
URL https: // httpss:///laaaaahaocoooo
S. Bharadwaj Yadavalli and Aaron Smith. Llvm-mctoll, 2019. URL https://doi.org/11 5281 2281/zenodo. ptw.//o..ooo..........o
15

---

Published as a conference paper at COLM 2024
A Implementation details
We use a learning rate of 1 x 10-4, with an inverse square root scheduler and 10k warmup updates. We train each model with global batch size of 16 function pairs for total of 900k updates on 4 NVIILA A100 for total of 72 hours, using weight decay of 5 x 10-3 as regularization. For the adaptations, use learning rate of 5 x 10 and 5k warmup updates, freezing all decoder parameters.
Regarding the training data, we augmented the compilable C dataset in Armengol-Estapp et al. (2022) with the required assembly and LLVM IR targets. Aside from the token-based C deduplication performed in the original dataset, we additionally perform an assembly- based deduplication, which leads finding further duplicates (different C code can lead to the same assembly), and use similar strategy to prevent data leakage to the evaluation sets. We also filter for length, but only for the training set. For realistic evaluation, we do not filter for length on evaluation sets for any of the systems. After filtering, we are left with about 2.7-3.3M parallel functions depending on the translation pair. We use the validation set to monitor training and select the best checkpoint.
As for the tokenizer, we train a Unigram tokenizer with vocabulary of 16k tokens. using the tokenizers library. 9 For incrementally adding new languages to the vocabulary, we would train new tokenizers and merge them back into the shared vocabulary, training only the embeddings of the new tokens (the ones without overlap with previous vocabulary). To simplify the implementation, do one-off shared vocabulary for LLVM IR, x86, ARM, and RISCV).
There are several variants of ARM. For evaluation, aar carch6--linux-eee However, Lasagne does not have 64 bit support for ARM. Thus, for fair evaluation, for Lasagne we use the target used in their tests, man--liuu-ggnuaabl
use the target triple
Before evaluating the translation systems, we check that the original function compiles in our host machine, and discard those which don't for all systems. Depending on the ISA and benchmark, the number of discarded functions ranges between 3% and 23%.
B Compiler optimization levels
These are the compiler (Clang, GCC) optimization levels used in this work:
O0: Unoptimized code.
O3: Aggressive execution in terms of execution time.
Oz: Aggressive optimization in terms of code size.
The length of IR target varies across formats and benchmark suite, O0 is expected to be more verbose as deliberately unoptimized aid programmer debugging. O3 targets performance and optimizes unnecessary computation away and is hence normally smaller in size. However, for Synth it is considerably larger due to the presence of extensive vectorization which requires much setup code. Oz aims at code size reduction and hence is the smallest of the IRs.
C Convergence
Figures 2 (ExeBench) and 3 (Synth) show the faster convergence of the ARM lifter when fine-tuning from x86 as opposed to starting from scratch.
&ogetherr with the encoder embedding layer, tied with the decoder embeddings. httpp: thub. htt.uu..cooouuggnngfcceeee http://ggginnaaaaeee.eeeeeeeee torenizers
16

---

Published as a conference paper at COLM 2024
From scratch
Best end training
Best end training
Accuracy (%)
epoch)
Number of Updates
Figure 2: Input/output output accuracy on different stages in the training, on ExeBench for ARM.
From scratch
x86
Best at and of training
Best training
Accuracy (%)
opach)
Number of Updates
Figure 3: Input/output accuracy on different stages in the training, on Synth for ARM.
17

---

Published as a conference paper at COLM 2024
D
Additional results
Table 9 shows the % of compilable translations generated by each system from O3 assembly.
X86
ARM
RISC-V
Lifter
Exe
Synth
Exe
Synth
Exe
Synth
Lasagne GPT-4
88.89%
40.78%
1.03%
0.97%
29.94% 19.94%
6.67%
39.61%
8.74%
31.97% 78.26%
5.62%
Forklift
80.88%
64.08%
80.44%
78.64%
76.40%
Table 9: Compilability results.
Table 10 shows the normalized edit distance of the generated translations with respect to the ground truth LLVM IR Oz.
x86
ARM
RISC-V
Lifter
Exe
Synth
Exe
Synth
Exe
Synth
Lasagne
0.74
0.89
1.00
1.00
GPT-4
0.73
0.62
0.67
0.65
0.68
0.68
Forklift
0.21
0.34
0.20
0.26
0.21
0.32
Table 10: Edit distance results.
18

---

Published as a conference paper at COLM 2024
E
Additional Lasagne Results
Given the relatively poor performance of Lasagne, is worth investigating, Table 11 shows the performance of Lasagne across the experimental parameters. Lasagne performs well in restricted settings, but struggles when moved out of its "comfort zone". On numerical benchmarks where the x86 binary has been generated by Clang with no optimizations, is able to perform extremely well, outperforming all techniques.
However, its performance drops by two-thirds when changing benchmark suite with more complex data types or compiler optimization level which introduce vector instructions.
The immaturity of the ARM lifter is emphasised across each category. Although it performs best on Clang O0 Synth, the performance is less than half that achieved on x86. Changing compilers again has an overall negative effect, surprisingly though, using GCC with O0 on Exebench is better than using Clang.
x86
ARM
Compiler
Opt
Exe
Synth
Exe
Synth
Clang
O0
33.01%
91.26%
10.35%
39.18%
03
30.26%
37.86%
1.03%
0.97%
GCC
O0
26.42%
41.35%
16.90%
1%
03
22.64%
35.24%
2.11%
0.97%
Table 11: levels.
Lasagne performance across benchmarks, ISAs, compilers and optimization
19

---

Published as a conference paper at COLM 2024
F
Additional analysis
In this section, we include additional input/outpus output accuracy correlation tables. We have excluded the systems for which the accuracies were close 0 on the corresponding dataset.
GPT4
Forklift
compiles
0.64
0.85
edit_sim
0.51
0.43
length
-0.27
-0.48
n fun _unnaagg
-0.20
-0.18
has_float
0.04
-0.02
has_strings
-0.02
-0.10
has globals
0.18
-0.06
Table 12: ExeBench feature correlations with input/outpul accuracy, Clang RISC-V O3
Lasagne
GPT4
Forklift
compiles edit_sim
0.89
0.67
0.87
0.36
0.46
0.39
c_length fun _nng
-0.05
-0.31
-0.48
0.01
-0.23
-0.19
0.00
-0.05
-0.08
has_stringg has_strings
-0.03
0.03
-0.16
has sosssgllbll
-0.26
0.18
-0.02
Table 13: ExeBench feature correlations with input/output accuracy, Clang ARM O3
Lasagne
Forklift
compiles
0.35
0.84
edit_sim
0.37
0.39
c_length
-0.09
-0.51
n fun args
0.18
-0.23
has has_float
0.05
-0.08
has_strings 0.02
-0.12
has_globab, -0.77
-0.07
Table 14: ExeBench feature correlations with input/ accuracy, GCC x86 O3
20

---

Published as a conference paper at COLM 2024
Forklift
compiles edit sim
0.80
0.31
c length
-0.22
n L'ffu__arg has_strings
0.07
-0.24
Table 15: Synth feature correlations with input/outpin accuracy, Clang RISC-V O3
compiles
0.76
edit_sim c_length n_fun_args
0.34
-0.37
-0.18
has float
-0.01
has stssstiings
-0.20
vectorized
0.21
Table 16: Synth feature correlations with input/output
accuuarc accuracy, Clang ARM O3
x86 03 GCC
Lasagne
Forklift
compiles edit_sim
0.98
0.74
0.55
0.21
c length n_ fun_aags
-0.03
-0.38
-0.25
0.02
has fas_float
-0.19
-0.06
has__stsiil strings vectorized
has
0.17
-0.13
-0.51
0.34
Table 17: Synth feature correlations with input/output accuracy, GCC x86 O3
21

---

Published as a conference paper at COLM 2024
G
Examples
This section provides detailed examples to illustrate where existing lifters fail to accurately lift code and features that Forklift struggles lift. Existing lifters struggle to lift features they were not designed for. For example Lasagne Rocha et al. (2022) fails to lift code that is vectorized, which frequently happens on higher optimization levels. Similarly, we show examples where Forklift is able to remove architecture-specffe features such as vectorization when lifts the code, allowing LLVM to re-introduce vectorization for the new targets. Finally, we show an examples where Forklift produces incorrect code, and highlight the future work that should be done to improve this.
G.1 Benchmark array_inc (Forklift correct, rest incorrect)
This benchmark (Listing 1) increments each element of an integer array by one. When compiled with clang on O3 (Listing 2) and Oz (Listing 3), it is vectorized, as seen by the use of the psubd instructions in the _ block.
Forklift lifts this correctly (Listing 4), recreating code in a similar style to Oz (Listing 3). GPT-4 fails to recreate the correct code (Listing 6), producing syntactically correct LLVM IR that does not perform the correct function. Rule-based Lasagne lifter does not support all the vector instructions the assembly code (Listing 8).
Interestingly, we also provide the O3 recompilations to x86 (Listing 5) and ARM (Listing 6) of the LLVM IR Oz predicted by Forklift. Critically, both are vectorized, showcasing that we can recover generic code from an optimized assembly and use LLVM to recompile with new optimizations.
Listing 1: Original C
void array_inc(int (int(int int n) (int 0; ++1) arr [i] 17
Listing Clang x86 O3
globl airay_ine p2align 4, 0x90 -type array_inc,
Begin function array jine
array_ine,Bfunctioo @function
arcay_inc:
@array_inc
cfi_startproc
* %bb. .0:
"esi, Sesl .LBBO_11
%b.. 1:
movl
"esi %eax $7 Nesi
cmpl
ja
LBB0_3
%bb. 2:
xorl
"ecx, %ecx
Jmp
LBBOO_00
_LBBO_3
mov]
"eax, Secx
andl
S- Necx -//5rex) Krsi
leaq
movq
shrq
$3, Nedx
addg
$1. Sedx
movi
"Seds, Srdd
andi
testg
firsi, Srsi
.1BB0.4
%lbb...
movq
subg
Mardx, Sesii
xorl
"edd, Redd "Smmm), 0x90
pempeqd
pZalign
LBBB_6:
ThThis Inner Loop Header: Depth-1
movdqu mevdqu
(Serdi, Sindx, 44 Samm1
16(%vri Srdx
Samm2
32(%rdi frrdx 4),
movdqu
48(%)rdi Serdx .4), Sammá
psubd
"ammn) Namml
paubd
Samm), Somm2
movdqu
"amml, (Arrd Srdx ,4)
22

---

Published as a conference paper at COLM 2024
meváqu
"amen, 16(%rdi,Nrrdx,
psubd
"amm)) Nocmm3
poubd
"ammn) Socmun
mevéqu
"ammn, Nrdy
mevequ
"amrnd. 18(%rdi, Nrrxx
addg
$16. %rddx
addq
$2. Sasi
LBBB0_6
testq
Nar8 Nra
je
LBB00_9
LBBB0_:
mevéquu
(Sarii 4).
movéqu
16(%rrii/%rdx, 4)) 4), Samml
pempead
"amm2, Namm2
psubd
"smm2. Nammn)
psubd "anmn,, Nwoml
movéqu Naman)) ('rrd, (rrdx, 44)
movdqu
"amen,, Ardxrdi,Nrdx,U Nr)x,4)
.LBBO
amp4
je p2align
.LBBO_11
4, 0x90
.LBBO_1 10:
HoTThis Inner Loop Header: Depth=1
addi
(rox.))))x.44
addq
$1, Srex
cmpq
"arx, Srax .1BBBO100
relq 1 ARR 4:
xorl testg jne .LBBB L_BBo_9
Sedx, Sedx
L.unne end0:
array__ine, Inc, -Luun_ende endo array_ _fi_enndrroc
siie
Listing 3: Ground truth LLVM IR Oz
define dso, Jocal void (132. nocapture 80, i32 %1) unnamed___de addr 40 53 sext 132 % 164
preds %8, %2
55 phi 161 1612 N8 0 iemp br N6, label 58, label 97
%4
preds %4
58 getelementptr inbounds 132, 132 80, 164 % load
59, align 4, Itbaa %11 add nsw 132 %110, 1
store 182 511. 132- 16, align 4, !tbaa 12
%/22 add 164 55,
Listing 4: LLVM IR predicted by Forklift (IO correct)
"Sstruct.strin structo type 132 define dso_local void
Aarray_inc(Ssttuctt..tt strucctt necapture 50, 132 %1) local_ unnamed___ddr
label 944
4: preds %8, %2
%5 phi 164 5.12, %8 0
lemp
N5, N3
il %6, label 88, label
7: preds 54 void
preds getelementptr inbounds Satruct..struct0,
strucco N0, 85, 132
%10 load 132, 132. 89, align 4
S11 add
i32 i32 %110 132. align %12 add raw rew 164 %5, br label %4
Listing 5: Vectorized x86 recompilation of the previous LLVM IR Oz
glohl array
Begin function array
23

---

Published as a conference paper at COLM 2024
-2allgn 0x90 array_ine @function
Sype
array. inc: %bb.
@array_ing iine
%esi,
ile
LB80_11
%bb.. 1:
%.Ir.ph prehhedde
mavslg
"esi, Srax $7,
empl ja
LBB0__ Seex, Seex -LBBO_10
%bb 2:
xorl
jmp
LBB0_3;
Svector -ph
move
"arax, "arex
andq
S-8, "irex
leaq
8(Sree))
movq
Nassi, Sardx
shrq
$3, Sedx
addg
s1 Sedx
movi
andl
$1, Se8d
testq
Marsl, Srrl LLB00_4
Svector cetooo.ph.noo
movq sube xorl pempaad "bamnd) Nwaund -2allgn
Par8 , Nrri
Mardx, Srra
"Sdds, "Seds
0x90
1 L.RRR 6:
Swector bady
ToThhi lnner Loop Header: Depth-1
mevequ
(ardx
Samml
mováqu
16(%rri Sirdx 44
Samm2
mováqu
a2(%rdi Mardx 4). 3amm3
movdqu
48(%vrdi "lammn, Namml
psubd
psubd
"ammn)) Namm2
movdqu
"amm), (ferdt Mards
movdqu
"amm2, 16(%rds,,Nrds Nrrdx
psubd
"bamnd) Nammm
poubd
Nammm) Nommms
movdqu
"tamm3, 32(%rrdi, Nards,
muveguu
"Namm,, 48(%rdd Nrddx $16,
addq
addq
$2, Marsi
jne
118B0.C
%bb..
Smiddle block falockkunr-leer unr-lessa
testi
LB880
1 RRD
Svector
muvelqu
(Srrii Sedx
movdqu
16(5rrdi,
(xrdx 4), N6mml
pempegd
"amm2, Swmm2
psubd
"smm2. Nemmn
poubd
"amm2,
movdqu
"ammn)) (%rdd (ardx
muváqu
"Amml,
16(%rdi, Ard%rdi,Kddxd Ards,4)
1,8B0
Smiddle block
ampy
je p2align
LB80 11
4, 0s90
L.BBB_ 10:
%.1r.ph Inner Loop Header: Depth=]
addi
(Rrdi (Nrcx
addg
S1. Seex
cmp4
Srex, Srax
1,B80
L.BB0_11 11:
S__crit_edge
retg
LLBBO_4:
xorl
%edx
testy %
jne LBBB0_
L8800_9
end0:
size
array_ine,
Luunc_end0-array.iing
Listing 6: Vectorized ARM recompilation of the previous LLVM IR Oz
gloht array
Begin function array
PZalign
type array_ine , @function
inc:
// Barray inc
Skb..0
1/ kill: def killed def 1/ =1
cmp
wl, #1 1
b.
1/ prehhaded
cmp
w1,
11
24

---

Published as a conference paper at COLM 2024
sxtw
LBBB0_3
1/ %bb.2
x9, LBB0__6
LBB0_3;
11 Svector. ph
and
x9, x8
add
x10, x0, #16
// =16
movi
mow
x11, x9
LBB0_4:
11 Svector.
// T>This Inner Header: Depth-1
q1. q2. 1x10, #-16]
subs
xi1, xi1.
//
add
v1.4s. v1.4s, vo.4s
add
v2.45, v2.4s. v0.4s
stp
q1, q2, [x10, 1-16]
add
x10, x10, #32
// -32
.LBBO_4 b.ne Sbb.5
// %middle. block
amp b.eq
.LBBO_8
.LBBO_6
1/ S.lr ph. preheader4
x8, x8,
x0, x9,
.LBBO_7:
1/ 8.1r.ph
1/ Thih Inner Loop Header: Depth-1
w10,
subs
x8, x5,
11
add
w10, w10,
17
sir
w10, [x9], 84
b.ne
.LD00_7
LBBB_8:
11 S._erit_edge
ret
Lfunc_endO:
array inc, .Luune endo-array
Listing 7: LLVM IR predicted by GPT-4 in the x86 case (compiles but not correct)
SLC) internal constant 132> <i32 1, 132 1, i32 1, 132 1>
declare void @llvm. man.yy.mpppppp... p018, (18 nocapture writeonly, 18 necapture readonly, 161. immarg)
define void Barray_ineci322 50, i32 %1)
entry:
52 iemp sle 132 %1, 0 label
3:
preds %entry
%4 add 132 iemp
br %5, label 8L6, label %6
preds %3
132
s8 lemp eq 132 N7. 0 be il %8. label NL9. label %
preds
%11 zext 132 to 161
%111 getelemeniptr 132, 132+ N0, 164 %10
%:12 add 132 87, iemp
%9 br 11 $13, label 'N14, label S11
14:
132+ getelementptr
store 132 1, 132. %15, align %16 add 132 N7, 2
%17 temp sle 132 %16, %17,
18:
preds %51
132+ 132 %19, align %20 add 192 N7, #21 temp ale 132 520, %0
getelementptr
br %21, label N22, label 11.
22:
preds 5588
W23 getelementptr 132, 132+ N0, 132 920 label 1324 152,, align
16:
preds %3
br label
19:
preds %6
br label SL11
2 prods N22,, N188 %114 59, Sentry, %1.
25

---

@str_cspa Published as a conference paper at COLM 2024
Listing 8: Error of Lasagne from x86
llmmmmttoll:
/app/Ilvm project sajject//llm/ttollee aettoo//////////ddddddaaaaaalaaa h08 metoll InstructionKind metoll :: getinstruction (unsigned int):
Assertion
Ieer Im metall wwwAdddinnttttooo.oooo (nd "Unlnnown opcode " failed
G.2 str_cspn (both Forklift and Lasagne correct)
In this benchmark (Listing 9), the string is searched for characters that exist in the string s. In this case, we can see that the compiled code is not vectorized, it relies only on simple x86 instructions (Listing 10). As a result, Lasagne is able to work. However, the generated code is verbose: much longer than the original code (Listing 15).
This unnatural code means that during future lowering passes, LLVM has a much more challenging task to generate performant code. We can that GPT-4 fails to create compilable sequence of LLVM IR (Listing 16), calling a strlen function that not defined.
that Forklift is able to generate correct lifted code (Listing 12) with one key difference from the original (Listing 11): the injection of a struct type around the string. Forklift tends to proactively inject these types due to the GitHub code it was trained on, which is very struct-heavy.
Finally,
Listing 9: Original C
stt__cspnn_cccccc css.epccccchrr (char va, char bb) [
int c
(char
for (char 1s by 405 ++s)
it (4k mn +8) I
return
return C:
Listing 10: Clang x86 03
globl str_cspn
Begin function str_cspn
-p2align
0x90
-ype str_cspn. @function
str_cspn:
-ffi_startproc
%bb..
movb
(%rdi),
testb
9r9b, Sr9b L_BB_1
je
testb %bb. 2:
movb
(Srsi), %r8b
Barib, fabb
je
LBBO_5
%bbb
addg xorl 1 LBRB 4
Neax, Seax
-oThis Loop Header: Depth-1
Child Loop BB1_8 Depth 2
movg movl p2align
Sidx Rrrdd Reex
4, 0x90
Parent Loop Depthel This Loop Headers Depth-2
cmph
%el. , %r96 1.880 10
Sbbb. 7:
Loop:
HBader-BBL Depth-2
(Srdx)),
addg testb
S1, Sedx *el %el
LL880.8
1oop Header-BB0 Depthe1
26

---

Published as a conference paper at COLM 2024
movb
1[Srdi), Se9b
addq
st, Sadi
testb
%r96, Sr9b
LBB0_4 L880_10
jmp
LBB0_1:
xorl
Seax, Seax
def Seax killed Seax killed $rax
relq
LBB0_S:
xorl
Seax
Seax
-2allgn
LLBB0_6
soThii Inner Loop Header: Depth=1
cmpb
$0 1(%rdi, Sarax)
leaq
1(%rax), Seax L.BB0_6
LLBBO_10
kill: def Seax killed Seax killed Srax
relq
Lfunc_endO:
size str_cspa. Lifun__nn0--st__spp efi_eeddroc
Listing 11: Ground truth LLVM IR Oz
define dso alccll 132 @str_cspn(i8. nocapture readonly %0, 18 nocapture readonly %1) add e0 label
3:
preds %15, %2
%4
132 92 516,
phi %00
%17,
56
load 18, 18 955, aliggn 1, 2thaa 12
97 lemp 18 56.
br %2, label %18, label %8
preds 'N3, N12
508 phi 18 1514, %12 1. S1, 53
load 18 align lthaa temp eq
br il %31, label %15, label 5:12
12:
%8 %:14 eetelementptr inbounds 18, 18+ 59, 164
iemp eq %6, %40
br 11 %13, label N18, label 88
preds
%16 add new new 132 %d.
%17 getelementptr inbounds 18, 18+ 55, 164 br label %3
18:
preds 163, M12
132 15
Listing 12: LLVM predicted by Forklift (IO correct, but uses new types)
Estruct.strucee type [
define docal 164 Betr_eapn(Kstruct..tee structo nocapture readunly 50, i8s nacapture readonly %1)
local unnamed addr
preds %17,
%4 phi 164 0, 52 1, 518, %17
55 phi 132 0, %2 1. 519. %17
getelementptr inbounds Sotruct, structo, gotelementptr inbounds Nstruct Netruct..struto,, Notruct structo. N6,, 0, 132
80,
58 load i8 18+ 57, align
59
18 %8, 0
%9, label
10: preds %14,
511 phi i8. 816, %14 1. (s,, 93
%12 load 18 18- %11, align
kemp eq
i1 %13. label %17, label %44
14: preds = %10
515 semp 18 58, %12
getelementptr
18
164
11 %11. label 1520, label %10
17: preds = %:10
%18 add 164 %4.
label %3
20: preds %14, %3
i61 %4
27

---

Published as a conference paper at COLM 2024
Listing 13: LLVM IR predicted by Lasagne (IO correct)
define dso Jocal 132 Bstr carr_cspn(66 Sarg],
164 "arg2)
entry: :
NEAX SKT LOC%
alloca 164. align
SRAV-SKT-ILDC
align
NRDI-SKT-IEX alloca 164, align
NEAX SKT LOC
alloca 164, align
"RON-SKT-LOO" alloca 164, align
CCT-SKT-IOC- alloca 164, align
9898-SKT-IOC- alloca 164, align
so = inttoptr 164 Sarg] to i8-
Nmemloadd load 18, 18, %0, align
seq_est
and Smerlood, Smemlond Shighhit and i8 128, %5 SSF = iemp i8 Shighhit, iemp eq i8 %1,
52 call
httpmm..pp...1..1111 %1)
53 = and 18 %2, 1
WPF temp i8 %3, Smmmmladd store 164 54, 164. 9R9B-SKT-LOC, align store 164 Sargl, 164, SRDI-SKT- SKT--C align 1 SCmpZF_JE iemp il SZF, true br SCmpZF
1bb.1
bb.1:
preds Sentry
55 inttoptr 164 Sarg2 to 18-
Smnemloadt fence seq_est
align
s6 = and 18 Smemlond1, Smemload1 %highbit2 = and -128, 86 SSF3 kem
Shighbit2,
R2F4 kemp eq 56,
57 call i8 Gilvm. 18 %6)
88 and 18 %7, 1
SPF5 semp eq
SCmpZF JESS iamp eq il SZF4, true br il SCmpZF JES8, label 9,b.9, label Nbb
preds
%RSI add 164 Sarg2,
s8 = call [ 164, i1 @livm uadd. with, overflow 164(16 Margg, 164 1)
<CF extractvalue 1 161. '8,
%11 call 164 cttpem.ctppp..iiiie. %10)
%12 and 164 %.1.
%PF6 iemp 164 %12.
%ZF7 iamp eq
%highbit8 and 164 -9233770665475888 5.51
*SP9 kemp ne 164 "Shighbits,
%13 call 161, i1 'ellvm.sadd
with. (61 Sarg2, 164 164, %13,
extractvalue
%14 zext 192 0 to 164
store 161 511. 161- SEAX-SKT-LOC, align
$15 zext 152 "EAX-SKT-LOCK,, align
br label %bb.3
bb.3:
preds 1bb..2 %ob.6
SBCX zext Smemlondl i32 %16 = zext 152 SECX to 164
store 16d %16. 164. SCL-SKT-LOC, align SRSI,
NRDX LOC, align %bb.4
bb. load 164, 164. NRYB-SKT-LOC, align trune
preds 3,b.3, %bb.5
%18 load 164, 164. SCL-SKT-IOS align
XLL trunc 164 %18 to 18 call 18 SCP10 extraetvalue f 18, il 520,
ølivm
overflow 18(18 5898, SCL)
%/FF1 temp eq 511, 0
%highbil12
519
lemp ne Shighbit12.
%21 call | 18, NOP14 extractvalue 18, 11 821. 1
ellvvmssub. with, overflow (88 5898, 18 NCL.
@llvm. @livmmctpop. etpop-ipp. (8 5.19 5,2,,
NPF15 lemp eq 18 523,
SCmpZF JE39 temp i1 92FII, true br SCmp2P_JJeP label Sbb.11, 11, label 516.5
NADX load 164 164. NRDX SKT LOC, align bb.
preds 9bb
inttoptr 164 9ROK 132+
%ewemoad16 load 1324 1824, align
28

---

Published as a conference paper at COLM 2024
seq_est
925 trunc 132 "mmemlaad66 to
SBCX17 zext i8 525 to 132
NRDX24 add *REX, 164.
uadd, wverflo...low.... (i6
SCF18 extractvalue [ 164, 11 826,
%27 and 164 *RDX24, 255
%28 call 161
(16 522))
%:2,
SPF19 lemp eq 164 %22,
%2F20 iemp eq 164 SRDX21, 0
Abighbi121 lemp ne %highhi111
164 -922372086654775988, NRDX24
%30 call 164,
Gilvm sadd. with. overflow (161 SRDK, i64
NDP23 extractvalue 164, 11 %30,
%31 132 SECX17 i8
%33 and %31, N32
%highbit25 and 18 -128, %33 NSP26 lemp
Whighhit25,
lemp eq 533,
call i8 @llvm. etpop..8 18 5:3) %35 and %,4,
NPF28
eq %35,
SCmpZF komp 936 = zext 152 SECX17 to 164 store 16d %36, 164. CLL-SKT-LOC align NRD024, 164- SRDX SKT-IOC, align JCNpZE_JNE,
Nbb.4, 52b.6
bb. 6:
preds 99bb
%.37 load 164, SEAX NEAX SKT-LOC, align
SEAX trunc
NEAX35 add 132 "EAX,
*:38 - call 182,
Gilvm uadd overflow. overflow.132(133 (12(132 SEAX, 132 N38,
REAX35, 255
%40 call 182 allvm sllvm.ctpop..aa 1320p.112(112 %39)
%41 and 132 840,
%PF30 lemp eq 132 %41,
lemp eq
%highbit32 and 132 -2147483648, SEAX35
XSP33 lemp ne 132 "thighbi132,
sadd overfloerflow..... (132 SEAX, 842,
extractvalue 1 132,
XRDI load 164, 164. HRDI SKT LOC, align
Semeret-ddsp add 164 SRDI 1 inttoptr 164 Smmmmrf- to 18 18, 18. %44, align
Nmemload36
fence seg est XRD143 add 164 SRDI 1
undd. with wwwrthooeerwww..ooo (64 SRDI, SCF37 extractvalue 1 164, i1 844,
945 and 164 SRD143, 255
%46 call 164
(14 5:55)
164 846,
lemp eg 164 %47,
%2F39 iemp 164 SRDHS, Shighbit40 and 164 -9223720065475500 NRDHS
iemp ne Shighbiti0.
call I 164. SOF42 extractvalue 164, il 848,
Gillm sadd. with overflow (164 SRDI, i64
%19 and Smmelood366. Smmmload33
Shighbitds
%SP45 lemp ne Whighhit44,
SZF16 icmp eq i8 %41, 0
%30 call 18 @llvm. @llvmmeeppp cttl:mmctppp..iiii. 18(18 5.49) 9,50,
SPF17 iemp eq %51, %52 = zext 132 SEAX35 to 164
161 %52, 164+ SEAX-SKT-LOCS, align SCmp2F = zext 132 SEAX35 store 161 %53, 164. SEAX-SKT-LOC, align 1 161 SRDH3, 164- SRDI-SKT-LOC, align
temp eq
164
SR9B-SKT-LOC, align store 161 %54, 164 be il XCmp2__JE600 label %bb.3, label 8
preds
br Jabel %bb.11
bb.9:
preds %bb.1
zext store 164 %55, 164. SRAX-SKT-LOC, align br Jabel 1bb.10
preds %.b..,
SRAX = load 164, 161+ SRAX-SKT-LOC, align 1 baseree-basereg add 164 Sargl, "RAX
Nimemref-disp15 add 164 SSmemref-baseee inttoptr Smnemref-dis188
29

---

Published as a conference paper at COLM 2024
load 555, align fence seq_est
%58 zext 18 %55 to i64
call 164,
6ilvm usub with. www.ooow..wwwooooo..oo. (164 955, 164 8.59)
SCF49 extractvalue 164. 11 %61,
%ZF50 iemp eq 164 %60, 0
Shighbi151
-9223770066647759888,
%SP52 lemp ne 164 %highbit51.
%62 = call I 161, il @llvm. saub
(66( %55, 164 5.59)
NOF3 extractvalue 164,
N62,
%6t = call 164 gilvm @llvm.ctpop.ik ctpop..6ff 863)
%65 and 164 864,
SPFS4 lemp eq 164 565, Nevemref diap55 store 164 -isem5e--dipp3. 164- SCmp2F INB6 icmp eq il KZF50, false store Smemred disp55. 164. SRAX SKT 1OC, align SCmp2F_
align
%bb.10, label
bb. 11:
preds <bb. 10, "bb.?7, Nab.4
%.66 load 164, 164. NEAX-SKT-LOCS, align SEAX57 trunc
be label SLanfiedRetuonBock
bb.8:
preds SSentyy
StalfiedReturnBlck
Unified
preds 9bb.8, Sbb.11 Sbb.8
NUnifidReeVaal phi 132 (NEEAX7, %bb.11 SUnifidRetVal
Fanction
nosyne
speculatable willreturn
declare @llvm.clpop-ii8 (18)
Function Attrs: nofree nounwind readnone speculatable willreturn
declare 164, Allvm. uadd with overflow 164 164) ao
; Function Attrs: notree nosyne nounwind readnone speculatable willreturn declare 164 giivm. htwm..cppp..ccm (14(()) e0
; Function Attrs: nofree nosyne nounwind readnone speculatable willreturn declare 164, II allvm. sadd with overflow 16444 164) 50
Function Atrs: nofree nosyne nounwind readnone speculatable willreturn declare 18,
usub, with. www.ooow....000.... 18)
Function Attrs: nofree nosyne nounwind readnone speculatable willreturn declare
ssub, overflow i8(188,
; Fanction Attr: nofree nosyne nounwind readnone speculatable willreturn declare 132, il I 8ilvm. uadd with overflow (332, 132) 40
; Function Attrs: nofree nosyne nounwind readnone spoculatable willreturn declare i32 8llvm. sepop.i32(ia) 40
; Function nofree nosyne nounwind
speculatable willreturn declare 132, il 8llvm sadd with overflow 132 (132, 132) 40
Function Attrs: nofree nosyne nounwind readnone speculatable willreturn 164,
164) a0
Function Attrs: nofree nosyne nounwind readnone speculatable willreturn declare 164, il 8ilvm ssub with overflow 164 (i64, 164) a0
attributes nofree nosyne nounwind readnone speculatable willreturn
Listing 14: LLVM predicted by GPT-4 (does not compile).
campiler Running pass DAG-SDAG Instruction Selection
function "#str_cspn'
clang: error: clang frontend command failed due to signal
declare 132 (18)))
define 132 Mast__cspn(18 56, 18+ Scharset) I entry:
50 call 132 Estrien
@strlen(is (18- Scharset)
52 lemp eg 132 'N0,
br il %2, label Necturn zero,, label Souter loop
outer_loop:
: preds Nentry, Souter_loop
%5.0 phi 182 [ Sentry 1, Nnext i, Souter loop %3 geielementptr 18, 184 N6. 132 %4.0 load i8, 58+ %3
kemp eq 18 %4,
30

---

Published as a conference paper at COLM 2024
label Secturn,t, label "aoner__lopp
inner_los loop:
phi 132 Souter__oop 1, Smext_i preds = Nouter_loop, Sinner_loop 56 getelementptr 57 load 18, %
Scharset, Sj Ninner_loop
58 temp 18 87,
br 58, label Ninner_loop_extt, label kinner_loop_bee
inner Jnop.__oo 58 temp i8 S4, 87 br %8, label Necturn,,, label Simeer__oop_continee preds Ninner_ Lonner_loop_body
preds Ninner_loop
inner unner_loop_comee loop_continee: [next_] add 132 %1.0, br label Sooper_loop
soner_loop_ext:
preds
Ninner_loop
"inex__ add 132 Si.0. br label loop
return_i:
preds
Minner_loop_bode Souter_loop
br label Sreturn
return_zero:
preds
Nentry
br label Sreturn
preds Secturn_f, Seeturn. Nret_valld phi [ (%,,,, Sreturn_] 1, Sreturn_zere
ret 132 Nret_val.0
G.3 Benchmark multiply_evens (Forklift Fails, Lasagne Works)
In this benchmark, the function: (arr[i] arr2[i]) arr2[i] is computed for cases where (arr[i] arr2[i]) is even (Listing 15). This code is not vectorized by LLVM on O3 (Listing 16) due to the branch that guards the access to the out array As a result, Lasagne is able to generate correct results for this benchmark all the instructions within its realm of capability (Listing 19).
However, in this case, Forklift generates incorrect code (Listing 18). The type signatures are equivalent, but the behavior of the function is incorrect. Again, we can see that Forklift produces code that relies simple structs.
GPT-4 produces code that does not compile: it is syntactically invalid: it declares the function twice, and then produces error when compiling one of the instructions (Listing 20).
Listing 15: Original C
void multiply_eee -itnsply-evensiattt sarr, int sarr2, int filtered for (int i - 0; i e. s+1)
int x arr[i] arr2[i]:
out filtered] (fltterdd
Listing 16: Clang x86 O3
globl multiply_eree evenss
Begin function multiply_evens
p2align 4, 0x90
-type multiply @function
multiply_ese
@multiply_evenns
-ff_sstartproc
* %bb.
Sedx, Sedx
.LBBO_6
* Nbb.. 1:
movl
"edx,, %r9 Wrld. Seld
cmpl
st. Nedx
jne
1880
Sbbb
xorl
%edx
1 1.R 3:
testq
Mrr %
je
LB800.6
31

---

Published as a conference paper at COLM 2024
mov]
Sardxx 4),,
subl
(frstt (8rdx, Neax
testb
$1, Sal
LBBB0_6
imull
(Srsi), ), Seax Neax, (Srex)
mov]
LBB0_6:
retq LBBB0_:
sube
%rr8 %r9
xorl
"eddx, SSdxx .LBBO_S
jmp
-Paalign
0x90
_LBBO_12:
in Loop: Header-BBO_8 Depth-1
addq
$2, Sedx L.DBO_3 je
empy
LBB00_:
Th-Thi Inner Loop Header: Depth=1
mov] subl
(%eri, Saddx
(Srrs
testb
$1, Sal .LnBo_9
%bbb. 10:
Loop: BBQ_8 Depth-1
mov]
(rdxx,) 4),
subl
40% (rrdx,4), Neax $1. Sal
testb
-1BBB
jmp
LLBBO_11
-Pallign
0x90
.I 9:
in Loop: Header-BB0. Depth-1
imull movl movl
(%rsl). Seax
(Sarx)
4[%rdi,/rrds Nardy ,4), Seax 4[Nrsi (.rdx,4). Seax Sal _LBBO_12
subl
jne
_LBBO_11
in Loop: Header-BBO_8 Dopth-1
imuil movl jan
(Srri),, Seax
"eax, (Srex) LLBBO_12
Luuee end0:
size
multiply_evens
LLune _Lfune_endO- end0- multiply_even
endproc
Listing 17: Ground truth LLVM IR Oz
define dso_lar clccal void #multiply Sereetippyyyyessee__ necapture readonly 55 sext 132 52 to 164 br label %6
eevens (132 nocapture readonly NO, 1324
12, nocapture 53) uncal_nnnaee_aadd
preds 122, %4
%7 phi 164 1222, 521 1. 0. 144
%8 temp sit 164 %5, %5
br %8, label label s9
preds %6
ret void
10:
%:11 getelementptr inbounds 132, 132 No, 164 z3 preds
%:12 load 132. 132, S11. align 4, libaa 12
getelementptr inhounds 132, 132
load 132,
%11, align 4, libaa
%15 sub nsw i32 %12, %14
%16 and 132 8:15,
temp br %17, label %18, Jabel 821
516,
18: load nsw %119 %.5
preds %10
align Itbaa
store i32 520, 132+ %3, align 4, Itbaa 12
br label 821
21:
preds %11, %10
9:2 = add mew new 164 N7, br label 56
Listing 18: LLVM IR predicted by Forklift (IO incorrect but compiles)
example: [4, 0, 12, 12]
sos arr = [5, 2, 8, 7] s00 arr2 [1, 2, 2, 3]
200 [0]
with zeros preallocate space
32

---

Published as a conference paper at COLM 2024
expected: 14, 0, 12,
obtained: 16, 0, 0,
"struct. type 132
define dso_lacal readonly 50, sattuct.structo. necapture readonly %1, 132 %2, 132. nocapture 53) local_unmame__edd
nocapture
%5 getelementptr inbounds Satruct. Satruct.structu structo. Satruct.
structo+ Natruct.struetto
S1. 164 0, 132
getelementptr inbounds Sstruct. structo, Satruct.structO- %0,
132
label %
8: preds 821, %1
phi 522, 1,
semp
11 %100 label S11, label %:3
11: preds %8
%12 geielementptr inbounds Nstruct. Sstruct. %0, 164 N9, 132
132,
5:2, align
%14- load 132. 132 %5, align
%15 sub nsw 132 %13, %14
%36 and 911,
semp eq
S16,
11 %17. label %18. label 821
18: preds %611
%19 load 132. 132+ %5, align
store 132 8,20, 132+ 13,, align br label 8211
preds '618, %11
164 19,
label %88
23: preds %8
Listing 19: LLVM IR predicted by Lasagne (IO correct but verbose)
define dso_local void Bmultiply_evens((fe Sargl, :64 Sarg2, 132 Narg3, 132 %argd) entry:
<RDN-SKT-LOCH
align
SEAX-SKT-LOC alloca 132, align
"RON-SKT-LOC alloca 164, align
50 and 132 Narg3. Sarg3 Shighbit
132 -2147483648,
SSF iemp ne i32 Shighbit, 22F iemp 132 %0.
% and 132
255
132 sllvm.cclppp.. htwmm.cpop..32(155 (132 %1)
%3 and 132 52, SP = jamp 132 %3,
SCmp2F_JE JLÉ kmp SCmpOF_LL kmp
NZF,
SZFOiSF JLE or SCmpZF JLE, XCmpOF JLE
be :1 "L,,,SPP_LEE, label %bb
label %bb.1
1b
preds Sentry
SR&D and 132 Sarg3,
% and 132 NRDD, 255
55 call 132 Stivm. wtwom.ppppp.iii.... 132 %4)
SPFI kemp eq i32 %6. 0
X2F2 sem 132 NRSD,
%highbit3 and 132 -2147483648, RRDD SSPi kmm
ShighbitS, 20 sub 132 Nargă,
%8 = call [ 132, s1 @llvm. usub with, overflow...oooooo.ooooooooorroom overflow..3221122 Karg3, 132 SCF extractvalue semp eq 132
11
Shighbité and 132 -2147483648, %7
NSP7 kmm ne 132 %highbit6,
s8 call I 132.
sllvmm.suu.... with overflow. 132(132 Narg3, 132 %10 and 132 N7, 255
%11 - call 132 allmm
132 sallv..ctpop.o. 11pppp.132(132 %10)
XPFS kmp eq
SCmpZF_JNE JNE temp eq il NZF5, false br i1 SCmp/F_JNE JNE, label Sab.7, label Rbb.2
preds
%13 zext 182 D 164
store 164 %13, 164- 9RDX SKT LOCS4, align
br label
bb.7:
preds %bb.
%114 zext 132 Sarg3 to 164
588D 164
%16 - call I 164
8ilvm usub. with. overflow..........o..oo.oooo.... (164 %11, 164 5.1)
SCP9 extractvalue I 164, i1 816,
iemp eq i64 -922377006854475800,
%highbit11
33

---

Published as a conference paper at COLM 2024
lemp ne "Shighbit11,
%11 call I 164, i1 6illvm ssub wwth. www.rhowww.o...eo.... (64 %14, 164 %115
%0F13 extractvalue 164, il %.7,
%18 164 588, 255
and 164 %,1,
SPF14 lemp eq 164 %20,
%62 zext 152
522, br label %25.10
align
ab.10:
.bb..9, %ob.7
164, SEDN-SKT-LOC, align Simeeneefidxreeg mul 164 1. 9RIX
":22 inttoptr 164 Kargl to 18+
%:2 getelementptr 18, 18 N22, 164 Smemref-idxreg
bitcast Namemlond load 132, 132- 924, align 1 fence seq_est
sinemref-ixxeepe
164 AREX
inttoptr Narg2
getelementptr 18, 18. %25, 164 Smeemef--dxrre115
%,27 bitcast 18. 826 to 132+ load
1227
seq_est
NEAX sub 152 Smemlood, 9228
%,29 = call I 192.
Cilvvm,uset overflo.. overflo...2w.. Smemload, 132 1628
SCF17
lemp eq
%highbil19 and 132 -2147483648, SEAX
XSP20 lemp ne 132 "hiihhit119 call | extractvalue %31 and 132 SFAX, 253
6ilvvm @llvm..ssub ssub owerfoower (32 Smemload, 132 %22)
%:32 call 152 allvm adlvm.cipop..e...
%31)
i32
lemp eg
%33,
%34 trunc 132 NEAX
*33 and 5,34,
call stlvmm httlvm.ctpop.fffee. cipop.18(18 535)
%33,
SPF23 iemp eq i8 537, 0
*2F24 lemp eq 535, 0 -128,
Shighbit25 lemp ne Whighbis25
SCmpZF.JJ JE temp eq 11 42F24, true br il SCmpZF label S8b. 14, label 58b.11
preds 10
Semenret-idsreg27 = mul 164 4, 9EEX poverre-bbsereggh add 164 Sargt, Smennef-idxrege Senemref-disp add Samemeef- inttoptr 164 Sememref-disg to 132
Nememload29 load 132, 132. %,58, align
fence seq_cst
Sememre--idd Smmerref-ixxeegll
164 SREX
Marg?, Smeenref idxreg30 Sememret-disp22 add i64 Sanemtef-baeeee331, 9:39 inttoptr 164 Simemref-disp22 132- load fence seq.
align
SEAXS sub 132 "mmemledd29 %40
Gilvm. usub withhoverrfoow (132 Smemlead29, 132 %:40 SCF3 extractvalue i32,
%11 = call 132
RZF35 temp eq 132 SEAX33,
Shighbit36 and 132 -2147183648, SEAX33
*SF37 iemp 132 Shighbit36, 0
call
oww..o.............oooooo.o..... (132 Smmeload22,
SOF38 extractvalue I 132, il 842,
%43 = and i32 SEAX33, 255
Su call 132 ctppp..ttpp....21.... %43)
i32
%PF39 iemp eq 132 %45,
% trunc 132 NEAX3S to i8
%17 and 546.
ctpop ctpop.i8(i8 547)
% and %48.
SPF40 iemp eq i8 549, 0
SZFI1 iemp eq %47,
Shighbitd2
-128.
%SP43 iemp ne Shighbit42,
store 132 SEAX33, 132- SEAX-SKT-1CC. align 1
"CCpZZF br SCmpZF
eq S2F1l, false
%bb
bb.12:
preds %1
br label 9bb.15
bb.14:
preds %00.1
%500 inttoptr 164 Narg2 to 132+
Simemooadtt = load 132. 132+ N500, align seq_est
34

---

Published as a conference paper at COLM 2024
NEAXSS mul "EAX, "mmemlodd44
fence seq_est
store 132 SEAX55, 132- Sargl, align 1 Ninenref-idxregd6 mul 164 4, NRDX Sanemref--asereed7
Sargl. "mmenee-idxregg6 Senenref-ddssp88 add 161 "ememrei-baereeg1.
%.51 inttoptr 164 Ninemref-disp45 to 132+ "6memload19 load 132, 132+ %51, align
seq_est Sememaeef-ixxeee50 mul 164 4, NRE add 164 Sarg2, "mmennef-idxreg50 Nanemref disp52 add 164 "mmmmref "inemeef-baseregeh basereg51, inttoptr Sinemref-disp52 %53 = load 132, 132. 552, align fence seq_est REAXS sub 132 "mmmmood49, %53
132-
call
wwth.ooer.oo.....oooooooo (132 Nmmeload49, 1853)
SCPS4 extraetvalue 132. 11 %54,
NZFSS iemp eq 132 SEAS3, 0
Shighbit56
132 2147483648, SEAX53
lemp ne "highbit55,
%55 call 132. i1 6ilvm ssub wwth..oerrloo...... (132 Smemload119 132 %S3)
NOFSS extractvalue 132. 11 835,
% 132 SEAXS3, 255
8ilym 538 and 132 837, 1ttpmmmccppp.i221122
NPF59 iemp eq 132 %558 0
18
%,9,
%61 call 18 sllvm.ctppop.... clpop..8(18 18(18 %60)
%62 and %,61, 1
%PF60 eq %62, iemp eq %highbil62 and 128, %60 XSP63 lemp ne 18 Nbiihbit62,,,
SEAX53, 132- SEAX -T--IIC align 3Cmp/F lemp eq 92P61,
br 11 "CmppF_ label %bb.9
label %bb.15
bb.15:
'bb. 14, 9bb.12
NEAX64 132, 132 SAX-SSKT-LX LOC, align
9663 = inttoptr 164 Warg? to 132.
Nenemlodd63 load 132, 132. 1863, align
seq
NEAX66 mul SEAX64, Nmmemloa655
fence seq_est
store 132 SEAX66, 132- Sarg4, align br label 5bb.9
bb ab.
preds 9bbb. 15, 1bbb.114 1966.11
XRDX73 add 164 9REX, 2
undd. with overflow. (66 SRDK,
WCF67 extractvalue i 164, il 8664,
963 and 184 5RDD73, 255
%66 call 164 8itvm.ctpop..6f (16 5:65)
164 866,
lemp eg 164
%2F69 iemp 164 SRDX73,
Shighbit70 and 164 -9223720865475500 NRDA73
iomp ne Shighbit70.
Gilvm sadd. with overflow (164 SRDK, 164 2) SOP72 extractvalue 1 164, il 868,
call | 164,
%89 sub 161 5R9, %RDN73
with www.hh.verllww...o 164(164 5R9, NRDX73) 870,
extractvalue 164,
%2175 iemp 164 %69, 0 %highbit76 and 164 -9223720065575500 %69 %SF77
164 Shighhitt7,
%71 call | 161, %0178 extractvalue 164, il %71,
Gilvm ssub with. overflow. (i61 589, 161 NRDX73)
%72 and i6t 869, 255
silvmmctpop. ctppp..pppooeee and 164 873,
SPF79 iemp eq 161 %74, 161 SRD03, 164- SHDX-SKT-LOCS,, align SCmpZF komp eq NZ175,
store 161 NRDV3, 164- SRDX-SKT-LCC, align br il SCmp2F label Sab.3, label S8b.10
preds %.b...
%67 zext i32 5RDD to 164
%26 zext 182 SRDD to 164
%77 and i61 875, %76
-92233220685475586
%SP61 iemp ne 164 Shiihhbtt8,,
%2182 iemp eq 164 %77,
%78 and 164 N77, 255
Slllmmccpppp itp..ccppp...iii...... %80 and i6t %79,
NPF83 iemp eq 164 %800 SCmp2F_IEs kemp il NZP82, be SCmpZFJJA
%bb.6.
5bb.1
35

---

Published as a conference paper at COLM 2024
bb.d:
preds %9b.3
SRRDXX load 164, 164+ NRDY-9K7-LOCM, align
"ineeneef-idxreg66 mul 164 NRDDSS
inttoptr Sargl
18+
%882 getelementptr 18 i8. *88, 164 Smmemed-idxxee866
%.8 bitcast 18. 882 to 132+
Simemload85 = load 132, 132+ %83, align seq_est
Sememreff-dxxeeg99 mul 164 4, NRDXSS
%51 = inttoptr 164 "arg2 to 18+
585 getelemeniptr 18, 18- 884, 164 Smemref idxregg8
bitcast
%57 load 132, 132. 5886, align
fence seq_est SEAX9 sub 132 "memloadsS, %87
call
Gilvmm
wwwrloow.......o.....o (132 Xmemload88, (N87)
NCP2 extraetvalue [ 132. 11 888,
92F93 iemp eq 132 SEAX9L, 0
%highbi194
132 2147483648, SEAX91
lomp ne "highhi194,
%,99 = call 132. i1 Gilvm ssub. with wwerth.werrroww... (132 Smemload68, 132 %87)
XOP% extractvalue 132. 11 %88,
132 SEAX9, 255 call 8ilvm cttp.ccppp....1122.......
5,9 and 132 N91,
NPF97 iemp eq 132 %92, 0
NEAX91 18 %93, % call 18 slllvmccppop. clpop..8811 %94)
%96 and 5,95,
%PP98 lemp eq 596,
iemp eq
Xhighbit100 and i8 -128, %94
NSP101 lemp ne 18 "highhit100. 0
SCmp/F iemp eq S2F99, SCmpZF_ JNES, label 6, label
bb.5:
preds 9bb.
inttoptr 164 "arga 132
Nmemlood1102 load 132, 132, 887, align fence seg est *EAXI03 mul 132 NEAX91, Smeelood112 seg store SEAXI03, 132+ Nargt, align br label %bb.6
bb.6: )
preds 19bb.5, Nob.4, 58 Sentry
; Function Attes: nosyne nounwind readnone speculatable willreturn declare 132 Bilvm, etpop.i32 (32()32) ) 40
Fanction Attrs: nofree nosyne nounwind readnone speculatable willreturn overllow (132, 182) a0
declare 132, , il
Function Attrs: nufree nosyne nounwind readnone speculatable willreturn declare 132, il I 8llvm. ssub with overllow (132, 132) a0
; Function Attrs: nofree nosyne nounwind readnone speculatable willreturn declare 164, il I 8llvm, usub with overtlow 164 (164, 164) a0
; Fanction nofree nosyne nounwind readnone speculatable willreturn declare 164, il gllvm, ssub with overllow (664, 164) a0
Function Attes: nofree declare @llvm.ctppp... cipop.i644(64)) )
nounwind readnone speculatable willreturn
Function Attrs: notree nosyne nounwind readnone speculatable willreturn declare i8 @llvm.ccpop. 18 (i8)
; Function Attes: nofree nosyne nounwind readnone speculatable willreturn declare 161, il I Gilvm. uadd. with overflow (661, 161) ao
; Function nofree nosyne nounwind readnone speculatable willreturn declare 161, i1 8llvm. sadd. with. overflow (i61, i61) 40
attributes nefree nosyne nounwind readnone speculatable willreturn
Listing 20: LLVM IR predicted by GPT-4 (does not compile).
Aside from ilegally redeclaring the function, the function definition itself doeent compile) oai evens. :1:1: error: label expected to be numbered 'g' preds %12, %
10:
declare void ovens 132,, 132,, 164)
define void @maltiply. evens(132- 50, S1, 182+ N2, 164 N3)
36

---

Published as a conference paper at COLM 2024
start:
Sol kemp sle 164 93,,
br il %4. label S14, label %
preds
56 add 164 563,
%7 xor 164
%3, %3
br label 5:0
10:
preds %.12 %
%8 phi 164 15, 55 1. %.1, %12 132- %0, 88
58 gelelementptr
load %11 getelementptr 132. 132+ %1, 164 %8
%112 load 132. 132+ %.1
%15 132 %10, %12
%15 iemp eq 132 %144 O
br i1 %,1, label %16, label 812
preds
%:17 load 132. 132+ %1
%18 mul 132 111, %13
%19 getelementptr 132. 132. 82, 164 %8
br label %12
preds '616,
%20 imm eq 164 %13,
br i1 %20, label N21, label 1530
preds
ret void
37


Reviews:
[{"review_id": "1vTtlF29uY", "reviewer": "Reviewer_zjVU", "summary": "This paper proposes a neural lifter to translate the source binary language into a shared compiler intermediate representation. The approach is to generate LLVM IR and source binaries from a large set of C functions, and reverse the translation from binaries to the LLVM IR.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. \n\n### Data generation\n\nWe took the dataset in [1] and added the corresponding versions of  LLVM IR (O0, Oz, O3) plus (x86, ARM, RISC) assembler when compiled by (GCC, Clang) with (Oz, O0, O3). The original dataset features over 4M compilable functions crawled from GitHub. The main challenge was engineering: systematically generating the data by compiling millions of functions and processing the corresponding assembly. We will clarify this in the paper.\n\n### Fine-tuning\n\nFine-tuning an existing model rather than training from scratch is interesting, as there is probably an overlap between binary lifting and other code-comprehension tasks. However, based on the findings in [2], in which an encoder-decoder trained from scratch clearly outperformed (e.g., 68.9% compared to 11.1%/ 8.9% or 81.2% compared to 36.4%) strong fine-tuned code LLM baselines (CodeLlama, Starcoder) in a related binary translation task\nwe trained from scratch. We will clarify this in the paper.\n\n### Results and overlap\n\nThe data set in [1] is claimed to be constructed such that there is no overlap, based on exact token-level deduplication with some normalizations (e.g. space normalization, comment removal). We did an additional deduplication not done in the original dataset: assembly-level deduplication (rather than C-level deduplication, as different C functions can be lowered to the same assembly).  We will clarify these deduplication steps taken in the paper, and all the data will be public.\n\n### Negative correlation\n\nGenerally, the longer the original C, the longer the corresponding LLVM IR and assembly, and sequence modeling models typically struggle with longer sequences. Increasing the number of arguments increases the complexity of e.g. stack management code, leading to a greater risk of failure. We will clarify.\n\n### Unit tests\n\nUnit testing is a common way of evaluating code correctness. Both evaluation sets (Exebench test set and Synth) provide unit tests. We compare the behavior of running Forklift’s translation against these. This includes not only return values but also arrays modified in place and mutated global variables. We will clarify this in the paper.\n\n[1] Armengol-Estape et al. ExeBench: an ML-scale dataset of executable C functions. https://dl.acm.org/doi/abs/10.1145/3520312.3534867\n\n[2] Lee et al. Guess & Sketch: Language Model Guided Transpilation. https://openreview.net/forum?id=qPFsIbF3V6", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The resulting model performs well compared to two strong baselines (manual lifter and GPT4), and if open-sourced, will be a significant contribution to not only the research community, but also industry.\n2. The paper is very clearly written and easy to follow. \n3. The analysis experiments are reasonable.", "reasons_to_reject": "1. The details on 3.1 Data Generation, however are quite lacking. Can you quantify the amount of the publicly available repositories crawled and describe the type of sources? Were there any challenges in applying the method of Armengol-Estape et al., 2022? \n\n2. I am not fully convinced by \"We do not fine tune from an existing LLM due to the\nadditional computational cost not being justified given the poor performance of code\nLLMs on low-level code observed\" - I think an experiment is required is to confirm that fine tune from an existing LLM (even if poor) is worse than train from scratch.\n\n3. The results in Table 1 look v good - was there any check for train-test overlap since you crawled from public repos?", "questions_to_authors": "Please see and respond to reasons to reject as well as \n\n3. Why do you think the model is experiencing negative correlation on length of original C code (c_length), and number of arguments of the function?\n\n4. \"We assume that the desired functions have reference unit tests\" - I don't see any particular method here. This seems like standard approach to function execution accuracy? Also, where do you construct the unit tests? Later you write that \"We then take the output LLVM IR, compile it back, and call the function from a C interface with the original context of the function. We then check for input/output accuracy\" (presumably against unit tests).", "extra_scores": {}}, {"review_id": "HkitTzLJEh", "reviewer": "Reviewer_WGiA", "summary": "The paper proposes Forklift, a neural lifter using a Transformer model to translate assembly code of different ISAs to the architecture-independent LLVM IR. Forklift achieves better accuracy and efficiency than both a state-of-the-art hand-written lifter and GPT-4 when translating x86 programs and enables translation from additional ISAs.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback.\n\n### Novelty\n\nThe main novelty of this paper is the demonstration that purely neural approaches (Transformers) can address a core challenge in binary portability: replacing the time-consuming task of manually writing lifting rules with an automatically learned approach targeting an intermediate language rather than another target directly, so that compiler infrastructure can be reused.\n\nTo tackle this challenge, we introduce several novel features.  We leverage incremental learning, where models that are initially trained for one assembler can be retrained to run on a new assembler, reusing the LLVM IR decoder component.  We demonstrate that for assembly, there is significant benefit in taking this incremental learning approach between architectures.  We also augmented the Exebench dataset [1] with LLVM IR and architecture targets and optimization levels not previously present in the dataset. Forklift is the first model that is capable of lifting from assembly.  LLVM IR is also a challenging target for language models, as it lacks the structure and readability that language models for code are typically able to take advantage of. We will update the paper to highlight novelty.\n\n### Data leakage\n\nThe data set in [1] is claimed to be constructed such that there is no overlap, based on exact token-level deduplication with some normalizations (e.g. space normalization, comment removal), and a similar strategy was applied for Synth. We run an additional deduplication not done in the original dataset: assembly-level deduplication (rather than C-level deduplication, as different C functions can be lowered to the same assembly). We will clarify this in the paper.\n\n### Lasagne failures\n\nGenerally, Lasagne failures are not due to specific bugs that could be fixed, but due to unsupported features that could be implemented but would require significant engineering effort. Lasagne has little to no support for vectorized instructions, and there are large bodies of instructions that are not supported in its ARM port. Additionally, Lasagne also has issues in code involving global variables, which would require  significant new analysis to resolve. which has traditionally been a challenging source language for large language models to LLVM IR.\nWe will provide a greater analysis in the final paper.\n\n[1] Armengol-Estape et al. ExeBench: an ML-scale dataset of executable C functions. https://dl.acm.org/doi/abs/10.1145/3520312.3534867", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper demonstrates the viability of lifting assembly code to LLVM IR using neural language models. The discussion on which IR language and IR optimization level works best is valuable.\n- The training data collected by this paper would be useful to the community working on neural code generation.\n- The discussion on fine-tuning the assembly encoder while keeping the IR decoder fixed is interesting and seems to provide some benefits in terms of accuracy and efficiency.", "reasons_to_reject": "- The main novelty in this paper is the application along with its setup (training and evaluation data, etc.) while there is little to no innovation over standard LM training.", "questions_to_authors": "- How do you ensure that there is no leakage between the training data and the evaluation data?\n- It’s great that Forklift performs significantly better than Lasagne out-of-the-box. Do you have a sense of whether the failures of Lasagne that you identified in this could be fixed? Do you know what it would take and how much better the performance would be?", "extra_scores": {}}, {"review_id": "DsFfg7BrG5", "reviewer": "Reviewer_9Een", "summary": "The paper introduces Forklift, which converts assembly into an intermediate representation for generic compiler optimization across different hardware platforms. Forklift outperforms Lasagne and GPT-4, gradually adding support for new ISAs by fine-tuning the assembly encoder and freezing the IR decoder to improve overall accuracy and efficiency. Overall, it leans towards an application-oriented approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. \n\n### Structured Learning Modules\n\nWhile the idea of structured modules such as GNNs for code is interesting, following previous work (e.g. [2]) we relied on a token/sequence-based approach with Transformers, as it’s currently the state of the art for generative code settings.  However, it is possible that there are structured models that are more effective than text-based models for assembly and is an open question.  We will add a discussion.\n\n### Scaling Laws for Size of Intermediate Representation\n\nThere are two dimensions that we can scale the data size in: the size of the individual examples and the number of training instances.\n\nFor the size of individual examples, in Table 8 we provide the correlation between the length of the original C code (correlated with both the length of assembly and LLVM IR) and the IO accuracy, showing that Forklift has a negative accuracy correlation with length. Interestingly, we observed that this was not only due to the context window limitation: longer examples are more difficult to handle for the model, even if they fit in the context window. GPT-4, for which all examples fit in their context window, also has a negative accuracy correlation with length. We will clarify and add data on the relation between performance and length.\n\nFor the number of samples used in training, Figure 2 in the Appendix shows Input/output accuracy on different stages in the training on a subset of ExeBench for ARM (Figure 3 shows the same for Synth). Again we will add clarification.\n\n### Other LLMs\n\n\nWe followed the findings in [1], in which an encoder-decoder trained from scratch clearly outperformed (e.g., 68.9% compared to 11.1%/8.9% or 81.2% compared to 36.4%) strong fine-tuned code LLMs baselines (CodeLlama, Starcoder) in a binary translation task. Recent  code LLMs are starting to consider the inclusion of low-level languages in pretraining datasets, previously typically neglected,. Starcoder2, a concurrent work to ours, is a step in the right direction, but the LLVM IR is not paired with the corresponding assembly, and the proportion of runtime-optimized assembly is tiny. We will add a discussion to the final paper.\n\n\n[1] Lee et al. Guess & Sketch: Language Model Guided Transpilation. https://openreview.net/forum?id=qPFsIbF3V6 \n[2] Lachaux et al. https://proceedings.neurips.cc/paper/2020/file/ed23fbf18c2cd35f8c7f8de44f85c08d-Paper.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The addressed problem is of practical significance and has the potential to become part of the software's foundational infrastructure.\n2. The experimental results are impressive, surpassing previous methods and the latest large language models, demonstrating the effectiveness of Forklift.", "reasons_to_reject": "1. There are certain limitations in the methodological contribution. Could some structured learning modules be introduced based on the representation of the intermediate language? Will the increase in data volume continue to improve effectiveness, i.e., is there a scaling law in the size of intermediate representation data?\n2. Experiments were not conducted on popular code LLMs. For instance, StarCoder2 was pre-trained on LLVM. Would this be beneficial for the task?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W99YAeN2u4", "reviewer": "Reviewer_GXRk", "summary": "This paper presents Forklift, a model designed to translate machine architecture-dependent assembly code into an intermediate representation that is architecture-independent, facilitating the migration of (legacy) programs to other machine architectures. Forklift is based on an encoder-decoder Transformer architecture and is trained using a parallel dataset created by the authors. The authors also incorporate the idea of incremental learning, initially training the model with x86 architecture and then fine-tuning it with another architecture. Experimental results demonstrate that Forklift outperforms other baselines, including a state-of-the-art hand-written model and GPT4, particularly in terms of passing unit tests. Furthermore, the authors present results from additional experiments to address interesting questions, such as the impact of incremental learning and error analysis.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. \n\n### Lack of parallel dataset details\n\nWe use the C programs and unit tests, as described in Exebench [1]. This is a publicly available open-source benchmark suite with about 4M compilable functions that we augment with additional assembly and LLVM IR (which we will release). The corresponding unit tests are also public. The input/output pairs for the Synth dataset are also public. In both datasets, there are 10 unit tests per function. In both cases, not only return values are tested, but also arrays modified in place. Additionally, for Exebench, global variables are also checked for potential mutations (Synth has no global variables).\n\nWe compile each program to 3 different ISAs (x86, ARM, and RISC-V)  with two compilers (GCC and LLVM)  and three optimization levels (Oz, O0, and O3).  We will make all data public and clarify. We will  also provide more statistics about the data in the paper itself.\n\n### Incremental learning order.\n\n\n\nWe experimented with x86 O3 => ARM O3 => RISC-V O3. The motivation for this order is that  x86 is the established/legacy architecture with ARM and then  RISC-V emerging later. We did not consider O0 as in reality, vendor released  binaries are  optimized. Furthermore, optimized binaries  have been shown  to be  more challenging to translate. We would expect to see similar results with learning strategies involving O0. Instead, we included compiler transfer results (Clang -> GCC) in section 5.3.\n\n### Tokenizer\n\nWe used Unigram rather than BPE as [2] showed it to perform marginally better. Additionally, we added custom regex split rules for the pre-tokenizer to avoid inconsistent digit/register tokenization and remove boilerplate characters from assembly/LLVM IR.  BPE is unlikely to make a significant difference. We will add a discussion to the paper.\n\n### Features\n\nEdit_sim means edit similarity, which is defined as 1- edit distance. Has_float means the code contains floating point operations. We will provide a full description of Table 8 entries in the paper, originally limited by the length limit.\n\n[1] Armengol-Estape et al. ExeBench: an ML-scale dataset of executable C functions. https://dl.acm.org/doi/abs/10.1145/3520312.3534867\n\n[2] Bostrom et al. Byte Pair Encoding is Suboptimal for Language Model Pretraining. https://aclanthology.org/2020.findings-emnlp.414/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces an interesting problem: mitigating legacy software formed by architecture-dependent assembly code, which could inspire further research by other researchers.\n\n- The paper presents a simple and effective method for solving the problem, providing a solid foundation for future research extensions.\n\n- The paper demonstrates various experimental results supporting the usefulness of Forklift.", "reasons_to_reject": "- Lack of detailed description of the parallel datasets and evaluation datasets, particularly unit tests. The number of data items and the difficulty level of unit tests are not specified, and it's unclear whether the authors have made the datasets open for reproducibility.\n\n- Some questions I have for the authors:", "questions_to_authors": "- The incremental learning idea is intriguing. Would training the model at the optimization level be feasible? For instance, the paper suggests the order: x86 => ARM => RISC-V. How about this alternative order: x86 O0 => x86 O3 => ARM O0 => ARM O3? Since the optimization level varies even with the same architecture, I'm curious about the potential results.\n\n- In my experience, tokenization poses a significant challenge in improving the performance of understanding assembly code. How does the model's performance differ when using Byte Pair Encoding (BPE) compared to a unigram tokenizer?\n\n- What are the features listed in Table 8? Could you provide more information about \"edit_sim\" and \"has_float\"?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Forklift for translating assembly to LLVM IR. They discussed their approach for data generation, model architecture, and other design choices. The incremental learning method is interesting: when adding new assembly languages, they can freeze the decoder and finetune for new languages. The neural architecture is an encoder-decoder Transformer based on BART. They demonstrate strong empirical results, where their model outperforms Lasagne and GPT-4.

The data and method from this work can be great contributions to the field of LM for software engineering, and thus I recommend acceptance. I recommend the authors to revise their work according to suggestions from reviewers. In particular, I think it is more convincing to add finetuning experiments with more recent coding language models, and it is helpful to show the empirical results on how the base model and pretraining affect the performance.
Decision: Accept