Unsupervised Binary Code Translation with Application to Code Clone Detection and Vulnerability Discovery
Authors: Iftakhar Ahmad
Lannan Luo
Venue: EMNLP 2023 Findings (2023)
Keywords: NLP
Neural Machine Translation
Binary Code Analysis
Vulnerability Discovery
Code Clone Detection

Abstract: Binary code analysis has immense importance in the research domain of software security. Today, software is very often compiled for various Instruction Set Architectures (ISAs). As a result, cross-architecture binary code analysis has become an emerging problem. Recently, deep learning-based binary analysis has shown promising success. It is widely known that training a deep learning model requires a massive amount of data. However, for some low-resource ISAs, an adequate amount of data is hard to find, preventing deep learning from being widely adopted for binary analysis. To overcome the data scarcity problem and facilitate cross-architecture binary code analysis, we propose to apply the ideas and techniques in Neural Machine Translation (NMT) to binary code analysis. Our insight is that a binary, after disassembly, is represented in some assembly language. Given a binary in a low-resource ISA, we translate it to a binary in a high-resource ISA (e.g., x86). Then we can use a model that has been trained on the high-resource ISA to test the translated binary. We have implemented the model called UNSUPERBINTRANS, and conducted experiments to evaluate its performance. Specifically, we conducted two downstream tasks, including code similarity detection and vulnerability discovery. In both tasks, we achieved high accuracies.

Unsupervised Binary Code Translation with Application to Code Similarity Detection and Vulnerability Discovery
Iftakhar Ahmad University of South Carolina iahmad@ema il. sc. edu
Lannan Luo
George Mason University
1luo4@gmu. edu
Abstract
to spread across binaries of diverse devices. As result, pross-architecture binary code analysis has become an emerging problem (Pewny et al., 2015; Eschweiler et al., 2016; Feng et al., 2016; Xu et al., 2017; Zuo et al., 2018). Analysis of binaries across ISAs, however, is non-trivial: such binaries dif- fer greatly in instruction sets; calling conventions; general- and special-purpose CPU register usages; and memory addressing modes.
Binary code analysis has immense importance the research domain of software security. To- day, software is very often compiled for vari- ous Instruction Set Architectures (ISAs). As a result, arosstarcite binary code anal- ysis has become emerging problem. Re- cently, deep learning- based binary analysis has shown promising success. It is widely known that training deep learning model requires massive amount data. However, for some low-resource ISAs, an adequate amount of data is hard to find, preventing deep learning from being widely adopted for binary analysis. overcome the data scarcity problem and fa- cilitate cross-architecture binary code analy- sis, we propose to apply the ideas and tech- niques in Neural Machine Translation (NMT) to binary code analysis. Our insight that binary, after disassembly, represented in some assembly language. Given binary in low-resource ISA, we translate it to a binary in high-resource ISA (e.g.. x86). Then can use a model that has been trained on the high-resouree ISA test the translated binary. We have implemented the model called UNSU- PERBINTRANS, and conducted experiments to evaluate its performance. Specifically, we con- ducted two downstream tasks, including code similarity detection and vulnerability discovery. In both tasks, we achieved high accuracies.
Recently, we have witnessed a surge in research efforts that leverage deep learning to tackle various binary code analysis tasks, including code clone detection (Luo et al., 2017a; Zuo et al., 2018), mal- ware classification (Raff et al., 2018; Cakir and Dogdu, 2018), vulnerability discovery (Lin et al., 2019; Wu et al., 2017), and function prototype in- ference (Chua al., 2017). Deep learning has demonstrated its strengths in code analysis, and shown noticeably better performances over tradi- tional program analysis-base based methods (Song et al.. 2013; BAP, 2011; Luo et al., 2014; Wang et al., 2009b) in terms of both accuracy and scalability. It is widely known that training a deep learn- ing model usually requires a massive amount data. As a result, most deep lcarning-basee binary code analysis methods have been dedicated to a high-resource ISA, such as x86, where large-scale labeled datasets exist for training their models. But for many other ISAS (e.g., ARM, PowerPC), there are few even no labeled datasets, resulting in negligent focus on these low-resource ISAs. More- over, it is labor-intensive and time-consuming collect data samples and manually label them to build datasets for such low-resource ISAs. In this work, we aim to overcome the challenge of data scarcity in low-resource ISAs and facilitate cross- architecture binary code analysis.
1 Introduction
software security research, binary code analy- sis plays a significant role. It can play a vital role in various tasks, such as vulnerability discovery, code plagiarism detection, and security auditing of software, etc., without accessing the source code. Today, software is very often compiled for various Instruction Set Architectures (ISAs). For exam- ple. 1oT vendors often use the same code base to compile firmware for different devices that oper- ate on varying ISAs (c.g., x86 and ARM), which causes a single vulnerability at source-code level
Our Insight. Given a pair of binaries in differ- ent ISAs, they, after being disassembled, are rep- resented in two sequences of instructions two assembly languages. In light of this insight, we

---

propose learn from Neural Machine Translation (NMT), an area that focuses on translating texts from one human language into another using ma- chine/deep learning techniques (Sutskever et al., 2014). There are two types of NMT systems. One is supervised and requires a large parallel corpus, which poses major practical problem. Another is unsupervised, which removes the need for parallel data. As unsupervised NMT has the advantage of requiring no cross-lingual signals, we adopt unsu- pervised NMT for binary code analysis.
int foo (int a)
r. if (a 1234)
int
(a ( r=
} else {
r=
r= 0
return
basic block
return
(a) Source code
(b) Control flow graph
Figure 1: Control flow graph and basic block.
This work proposes a new research direction binary code analysis, especially for low- resource ISAs. Based on it, for downstream binary analysis task, we only need to train a model high-resource ISA and transfer it to analyze binaries in other ISAs by trans- lating such binaries to the high-resource ISA. Through this, we can resolve the data scarcity issue in low-resource ISAs.
Our Approach. Drawing inspiration from the unsupervised neural machine translation model Undreamt (Artetxe et al., 2018b), we design an unsupervised binary code translation model called UNSUPERBINTRANS UNSUPERBINTRANT is trained in completely unsupervised manner, re- lying solely -nchttccci corpora. Once trained, UNSUPERBINTRANS can translate bina- ries from low-resource ISA to a high-resource ISA (e.g., x86). Consequently, when dealing with a binary in low-resource ISA, translating it into a high-resource ISA enables us to use a model trained on the high-resource ISA to test the translated bi- can be applied
Background
A control flow graph (CFG) is the graphical rep- resentation of control flow or computation during the execution of programs or applications. A CFG is directed graph in which each node represents basic block and each edge represents the flow of control between blocks. A basic block is quence of consecutive statements in which the flow of control enters at the beginning and leaves at the end without halt or branching except at the end.
nary code. UNSUPER to variety of applications, such as vulnerability discovery and code similarity detection.
We have implemented UNSUPERBINTRANS and evaluated its performance. Specifically, we conduct two critical binary analysis tasks, includ- ing code similarity detection and vulnerability dis- covery. In both tasks, we achieve high accuracies across different ISAs. For example, a model trained on x86 can successfully detect all vulnerable func- tions in ARM (even the model has not been posed to any vulnerable functions in ARM during its training), indicating the exceptional translation capabilitics UNSUPERBINTRANE Below we summarize our contributions:
As shown in Figure I(a), given a piece of source code, its corresponding CFG shown in Fig- ure 1(b), where each node is a basic block. Simi- larly, we can also generate the corresponding CFG for a piece of binary code. We here use the source code as example for simplicity.
3 Motivation and Model Design
This section first discusses the motivation behind our proposal for unsupervised binary code transla- tion, and then presents the model design.
We propose a novel unsupervised method for translating binary code from one ISA to an- other, such that the translated binary shares similar semantics as the original code.
3.1 Motivation
Consider the vulnerability discovery task, long- standing problem of binary code analysis, as an example, binaries in different ISAS are analyzed in detail to find vulnerabilities. Nowadays, apply- ing machine/deep learning binary analysis has drawn great attention due to the exceptional perfor- mance. However, it usually requires large amount of data for training. Unfortunately, many a time it
We have implemented the model UNSU- PERBINTRANS and conducted evaluations on two important binary analysis tasks. The re- sults show that UNSUPERBIN BINTRANS can suc- cessfully capture code semantics and effec- tively translate binaries across ISAs.

---

is difficult to collect such a large amount of training data, especially for low-resource ISAs.
(3) Other symbols are replaced with <TAG>, Take the code snippets below as an example, where the left one shows the original assembly code and the right one is the preprocessed code.
There are some ISAs that are commonly used (e.g.. x86) and it becomes easy to collect large training data for such ISAs. It would be a great advantage if the availability of such large training data of high-resource ISAs could facilitate the au- tomated analysis of binaries in low-resource ISAs, where sufficient training data not available.
MOV EDX, 11EH
MOV EDX, 11E1H
MOV ECX, OFFFFFFFFH JLE LOC [OC__8BAB FREE MOV RCX, C 2_5 MOV [RSP+VAR_ -AR1581, RDX
MOV ECX, <CONST>
JLE LOC_<TAG> CALL CRYPTO_
CALL CRYE
MOV RCX, <ADRA MOV [RSP+<VAR>], RDX
For example, suppose there a large training dataset for the ISA X, however, we need to analyze a binary b in the ISA Y, where the available training data is insufficient. As result, it is difficult to train a model for analyzing b in Y. In order to resolve the problem, our idea is to translate b from the ISA Y to X, resulting in translated binary b' in X. Then, we can leverage the large training dataset of X train model, which can be used perform prediction on the translated version b' in X.
Building Training Datasets. As we regard instruc- tions words and basic blocks as sentences, we use large number of basic blocks to train the in- struction embeddings. Specifically, for given ISA X, we first collect various opensource programs, and compile them in X. Note that given the wide availability of opensource code, this requires little effort. After that, we use IDA Pro (The IDA Pro Disassembler and Debugger) to disassemble all the binaries to generate the assembly code, from which we can extract the basic blocks. We use all the ba- sic blocks (after preprocessing) to build a training dataset for this ISA.
3.2 Learning Cross-Architecture Instruction Embeddings
A binary, after disassembled, represented as a se- quence of instructions. An instruction includes an opcode (specifying the operation to be performed) and zero or more operands (specifying registers, memory locations, or literal data). For example, mov eax, ebx is an instruction where mov is an opcode and both eax and ebx are operands.? 2 In NMT, words are usually converted into word em- beddings to facilitate further processing. Since we regard instructions as words, similarly we represent instructions as instruction embeddings.
Learning Process. Below we define two terms, mono-architecture instruction embeddings (MAIE) and cross-architecture instruction embeddings (CAIE).
Definition 1 (Mono-Architectture Instruction Embeddings) Mono-architecture instruction embeddings (MAIE) are architecture-speccfic. where similar instructions in the same ISA have close embeddings. MAIE of different ISAs are projected into different vector spaces.
Challenge. In NLP, if a trained word embedding model is used to convert a word that has never ap- peared during training, the word is called an out-of- vocabulary (OOV) word and the embedding gener- ation for such words will fail. This is well-known problem in NLP, and it exacerbates significantly in our case, as constants, address offsets, and labels are frequently used in instructions. How to deal with the OOV problem is a challenge.
Definition 2 (Cross-Architecture Instruction Em- instruction embed- dings (CATE) are architecture-agnosti where simi- lar instructions, regardless of their ISAs, have close embeddings. CAIE are projected into the same vec- tor space.
beddings)
We adopt fastText (Bojanowski et al., 2017) to learn MAIE for an ISA. The result an embed- ding matrix X E IRV xd where V is the number of unique instructions in the vocabulary, and d is the dimensionality of the instruction embeddings. The ith row X,, is the embedding of the ith instruction in the vocabulary.
Solution:
Preprocessing Instructions. To resolve the OOV problem, we propose to preprocess the instructions using the following rules: (1) Con- stants up to 4 digits are preserved because they usually contain useful semantic information; con- stants more than digits are replaced with <CONST>, (2) Memory addresses are replaced with <ADDR>.
Given two ISAS, where one is high-resource ISA and another a low-resource ISA, we follow the aforementioned steps to learn MAIE for each
'Assembly code in this paper adopts the Intel syntax, i.c., dst, ssc(s).
ISA. After that, we map MAIE of the low-resource

---

L1 decoder
x86
Shared encoder
Translating binaries low-resource ISA code
binary code
Cross-archilleclure instruction embeddings
Binary in low- resource ISA
(CAIE)
L2 decoder
CAIE
Binary anallysis in low-resource ISA
Figure 2: Architecture of UNNUPERRBNTTRAN
ISA into the vector space the high-resource ISA by adopting the unsupervised learning technique vecmap (Artetxe et al., 8a). Note that other un- supervised learning methods for generating cross- lingual word embeddings (Conneau et al., 2017; Ruder et al., 2019) also work.
parallel text corpora. Undreamt achieves very good results for translation due to the three main proper-
ties of its architecture. These properties are dual structure, shared encoder, and fixed embeddings in the encoder. First, the dual structure enables bidirectional translation capabilities to be achieved through training. Second, the shared encoder pro- duces language-independent representation of the input text corpus. Then the corresponding de- coders for a particular language can transform that representation into the required language. Finally, with the help of fixed embeddings in the encoder, Undreamt can easily learn how to compose word- level representations of language to build repre- sentations for bigger phrases. These embeddings pre-trained and kept fixed throughout the train- ing which makes the training procedure simpler.
3.3 Translating Basic Blocks Across ISAS
Drawing inspiration from the unsupervised neu- ral machine translation model Undreamt (Artetxe et al., 2018b), design unsupervised binary code translation model, named UNSUPERBIN- TRANS, translate piece of binary code (i.e., a basic block) from one ISA to another ISA. The model architecture of INsUPERBINTRANE is shown in Figure 2.
First, CAIE are generated for two ISAs, where one is high-resource ISA (x86) and another low-resource ISA (see Section 3.2). Then, we train UNSUPERBINTRR (the grey dashed box; the detailed process discussed below). The training only uses mono-architecture datasets and CAIE generated from the mono-architecture datasets. The mono-architecture dataset for each ISA con- tains a large number of basic blocks from this ISA. Finally, the trained model can translate basic block from the low-resource ISA to x86, and the translated basic block shares similar semanties as the original one. Through the translation of bina- ries from a low-resource ISA to x86, we can use a model trained on x86 to analyze the translated code. This capability allows UNSUPER BINTRAN to address the data scarcity issue in low-resource ISAS. UNSUPERBINTRANS can be applied to vari- ous binary analysis tasks, such as code similarity detection and vulnerability discovery.
The training procedure consists of two main underlying components. Denoising and On-the- fly backtranslation. Although the dual property of Undreamt helps gain preliminary translation knowledge, eventually such procedure results in simple copying task where the trained model may result in making word-by-word replacements of the source and target languages. Therefore, improve the translation capability of the trained model, Undreamt injects noise into input data by adding some random swaps of contiguous words. Then the model is asked to recover the original input text by denoising the corrupted data. This process of adding and removing the noise helps the model to learn the actual relationship between words in the sentences for the given languages.
We implement UNSUPER BINTRANS upon Undreamt. During the testing phase, when pro- vided with a basic block from ISA, we first represent
is designed based on Undreamt, below we briefly discuss how Undreamt works. Please refer to (Artetxe al., 2018b) for more details. Undreamt alleviates the major limi- tations of NMT, i.e., the requirement of very large
As UNNUPERBIN'TRA
sequence of CAIE, which is fed UNSUPER BINTRANS. UNSUPERBINTRANN ANS then translates it to a basic block in another ISA, which shares similar semantics as the source basic block.

---

Given two Table 1: Statistics of datascts.
dataset contains a of preprocessed basic blocks corresponding one ISA and one optimization level. Table shows the statistics of our datasets, including the number of functions, the number of unique instructions (i.e., vocabulary size), and the
Opt. Level
of Functions
of Unique Instructions
Total # of Instructions
ISA
ARM
80,065
100,515
6,669,266
00
x86
71,608
94,121
6,555.483
ARM
83,184
102,388
6,864,017
total number of instructions for each optimization 6,587,172
X86
70,350
89,893
level in terms of the two ISAs, x86 and ARM.
ARM
74,173
111,564
7,408,810
02
x86
70,678
91,139
6,626.772
Training UNSUPERBINTRANE
ARM
74,186
111,587
7,317,972
03
datasets corresponding to two different ISAs and the same optimization level, we first use fext to generate MAIE for each ISA, which are then mapped to a shared space to generate CAIE by vecmap. After that, we train PISSUPERBINTRAAE using the two datasets and CAIE to translate basic blocks from one ISA to another. Note that UNSU- PERBINTRANS is based on unsupervised learning; we do need to determine which basic block one ISA similar to basic block in another ISA.
x86
70,329
89,932
6,619.398
4
Evaluation
We conducted experiments to evaluate the transla- tion performance of UNSUPER We first describe the experimental settings, and then con- duct quantitative analysis by computing the BLEU scores (Bilingual Evaluation Understudy) (Pap- ineni et al., 2002). After that, we conduct two downstream tasks, including code similarity com- parison and vulnerability discovery. Finally, we evaluate the efficiency.
4.2 Quantitative and Qualitative Analysis
We first calculate the BLEU score to measure the quality of our translation. A BLEU score repre- sented by a number between zero and one. If the score is close to zero, the translation quality is poor. On the other hand, the translation quality is good when the score is close to one.
4.1 Experimental Settings
UNSUPERBINTRANS contains three compo- nents, including fastText (Bojanowski et al., 2017), vecmap (Artetxe et al., 2018a), and Undreamt (Artetxe et al., 2018b). The experiments were conducted on computer with Ubuntu 20.04, 64-bit 2.50 GHz 11th Gen lntel (TM) i7- 11700 CPU with 8 cores, an Nvidia GeForee RTX 3080 GPU, 64 GB RAM, and 2 TB HD. We set the embedding dimension to 200 and use the default settings of vecmap and Undreamt: the hyperparam- eters, such as learning rate, epoch, and batch size. etc., are set to their default values.
For each x86 function, we first find its simi- lar ARM function: following the datasct building method in InnerEye (Zuo et al., 2018), we con- sider two functions similar if they are compiled from the same piece of source code, and dissimilar if their source code is rather different. As result, for each ARM function, denoted as S, upon iden- tifying its similar x86 function R, we include this similar pair into our dataset. For each ARM func- tion in the dataset, we use UNNUPPRRBINTRAN translate each of its basic blocks into x86, result- ing in a translated function in x86, denoted as T. A fter that, we compute the BLEU score between T and Rt. Finally, we compute the average BLEU score for all functions in the dataset. We obtain 0.76, 0.77, 0.77, 0.76 BLEU scores, for the four different optimization levels, respectively, when using fast for instruction embedding genera- tion. Moreover, when we use word2vec, we obtain 0.77, 0.77, 0.77, 0.76 BLEU scores, for the four different optimization levels, respectively. Thus, our model achieves good translation.
Datasets. We consider two ISAs, x86 and ARM. We select set of widely used open-source pack- ages, include binutils, coreutils, findutils, inetutils, and openssl, to build two mono- architecture datasets (one for x86 and another for ARM). Specifically, for each program, we compile it into binaries in x86 and ARM. We consider four optimization levels, O0, O1, O2, and 03, when generating binaries. As result, for each program, we obtain four binaries for each ISA, each of which is for one optimization level.
to
We next disassemble these binaries using IDA Pro and extract basic blocks. Each instruction in basic blocks is then preprocessed. Finally, we build four mono-ane architcccit datasets for x86 and another four for ARM. Each mono-architecture
Table shows three randomly selected examples. Due to space limits, we use basic blocks as exam- ples. In each example, 1) the source ARM is the

---

Table 2: Three examples of assembly code translations.
Source
LDR RO, [R7, OXIO+<ADDR>1 MOV RI, <TAG>; STARTSWITH O:
ARM
MOV R3, RO; CMP R3, #0; BEQ _OC_<OTAG <TAG>
Reference
MOV RAX. [RBP+NAME]; LEA RSI, < TAG>; MOV RDI, RAX;
x86
CALL STARTSWITHLU O; TEST AL, AL; JZ SHORT LOC_ <TAG>
Translated
MOV RAX. [RBP+NAME]; LEA RSI, <TAG>; MOV RDI, RAX;
x86
CALL STRCMP: TEST EAX, EAX: JZ SHORT LOC <TAG>
Source
LDR R3, [R7, 0X18+H]; LDRB. W R3, IR3, #0X37]; AND. W R3, R3.
ARM
UXTB R3, R3; CMP R3, #0; BNE LOC_<TAG>
Reference
MOV RAX. [RBP+H]; MOVZX EAX, <BYTE PTR-|RAX+6BH]]
x86
AND EAX, 2; TEST AL, AL; JNZ SHORT LOC <TAG>
Translated
MOV RAX, [RBP4H]; MOVZX FAX, <BYTE_ <BYTE_PTR>> [RAX+6BH];
x86
AND EAX, 2; TEST AL, AL; INZ SHORT LOC <T<>>
Source
LDR R3, [R7, OXI8+INFO): LDRB R3, [R3, #6]; AND.W R3, R3, #4:
ARM
UXTB R3, R3; CMP R3, #0: BEQ LOC <TAG>
Reference
MOV RAX, [RBP+H]; MOVZX EAX, - [RAX+6CH];
x86
AND EAX. 20H: TEST AL. AL: JNZ SHORT LOC, <TAG>
Translated
MOV RAX, [RBP+INFO]: MOVZX EAX, <BYTE PTR>IRAX+0];
x86
AND EAX, 4; TEST AL, AL; JZ SHORT LOC <TAG>
original ARM basic block: 2) the reference x86 is the x86 basic block that is similar the original ARM basic block; and 3) the translated x86 the x86 basic block that is translated from the original ARM basic block by our model. By comparing the translated x86 block with the reference x86 block, we can see that UNSUPER BINTRANS (1) success- fully predicts all opcodes (i.e., the opcodes in the translated x86 and those in the reference x86 block are the same), and (2) small number of operands are not predicted correctly but reasonable. For ex- ample, in the first example, the fifth instruction in the reference x86 block is TEST AL, AL, while the predicted instruction in the translated x86 block is TEST, EAX, EAX. Opcodes, which determines the operation to be performed, captures more seman- ties compared to operands. Moreover, the random allocation of registers across different optimization levels diminishes their significance. We thus can conclude that AASUPERBINTR.CR consistently at- tains exceptionally high-quality translations. This results in superior performance in various down- stream tasks, as detailed in the following sections.
the ISA X a basic block in the ISA Y, resulting in function fí in Y. After that, we have two binary functions, Jí and fz, within the same ISA Y.
For each function, we generate its function em- bedding. which is computed as the weighted sum of the CAIE of all instructions this function. The weight each CAIE is calculated as the term- frequency (TF). We then compute the cosine simi- larity between their function embeddings to mea- sure their similarity.
Vulnerability Discovery. We consider three vulnerabilities: CVE-2014-0160 (or Heartbleed bug) (Banks, 2015), CVE-2014-3508, and CVE- 1791- We follow a similar way as above to compute the functions embeddings for the vulnera- ble functions and benign functions. We use Support Vector Machine (SVM), a widely used machine learning model to detect vulnerabilities.
Since there is usually only one vulnerable function (positive sample) for training SVM, the dataset becomes extremely imbalanced. To han- dle the imbalance issue, Random Oversampling (ROS) (Moreo et al., 2016) and Synthetic Minority Over-sampling Technique (SMOTE) (Chawla et al., 2002) are adopted. For each vulnerability case, we first train an SVM model using the vulnerable func- tions and benign functions in x86. We then use the trained model detect the vulnerability in ARM functions. During detection, we first translate each basic block in the ARM function to basic block in x86 and use the SVM model (that has been trained on x86) to test the translated function.
4.3 Downstream Applications
We consider two downstream tasks as follows.
Function Similarity Comparison. Given two bi- nary functions, fi and f2, in two different ISAs, X and Y. our goal is to measure their similarity. If they are similar, they may copy from each other (i.c., stem from the same piece of source code). To achieve this, we translate each basic block of fi in

---

Table 3: Accuracy of the function similarity task.
4.5 Vulnerability Discovery
Opt. Level
Tool for instruction embedding generation
We consider three case studies as follows.
word2vec
fastText
Case Study CVE-2014-0160 is the Heartbleed vulnerability in OpenSSL (Banks, 2015). allows remote attackers to get sensitive information via accessing process stack memory using manually crafted data packets which would trigger a buffer overread. We obtain the vulnerable binary function in x86 and ARM from OpenSSL v1.0.1d. 1d.
00
78.09%
81.24%
01
83.33%
91.11%
02
87.05%
95.30%
03
86.76%
94.97%
4.4
Function Similarity Comparison
For each optimization level, we first randomly se- lect 7, 418 x86 functions. For each selected x86 function, we search for its similar and dissimilar function in ARM. We consider two functions sim- ilar if they are compiled from the same piece of source code, and dissimilar if their source code is rather different. We assign the label "I" to each similar function pair, and the label "0" to each dis- similar function pair.
Case Study II. For CVE-2014-3508, context- dependent attackers are able to obtain sensitive in- formation from process stack memory via reading output from some specific functions. The vulnera- ble function is collected from OpenSSL v1.0.10 1d.
Case Study III. For CVE-2015-1791, race con- dition occurs when the vulnerable function is used for a multi-threaded client. A remote attacker can create denial of service attack (double free and application crash) when attempting to reuse a ticket that had been obtained earlier. The vulnerable func- tion is also collected from OpenSSL v1.0. 1d.
We next generate function embeddings for cach function. In NLP, many methods exist for compos- ing word embeddings to sentence/document embed- dings. We choose the following way. We calculate the tern-frequency (TF) for each instruction in function and then multiply it by the correspond- ing CAIE of that instruction. We next add all the weighted CAIE and use the sum as the function em- bedding. Note that for an ARM function, we first translate it to an x86 function using ENSUPERBIN. TRANS, and then compute its function embedding using the method discussed before. To measure the function similarity, we calculate the cosine simi- larity between their function embeddings and then calculate the accuracy.
Our Results. For each case study, we select 9,999 benign functions. However, only have one vulnerable sample. To address this unbal- anced sample issue, we use ROS (Moreo et al., 2016) and SMOTE (Chawla et al., 2002). We set the parameters k *_neighbors to 2 and samr pling_strateg, to 0.002. We try different values of sampling_straiee _trategy, including 0.5, 0,25, and 0.02, etc., and find that with the value of 0.002, we obtain enough oversampling data and yield good results. As set k_ neighhoors 2, duplicate the vulner- able function three times, so that the initial minority sample size is more than the value of k_ neighbors.. We then compute the function embedding for each function. We use the function embeddings of x86 vulnerable functions and benign functions to train SVM model. Then, the trained SVM model is transferred to test the translated code of each ARM function, without any modification. In this experiment, we use fastText to generate MAIE as it gives better results than word2vec.
We perform the experiment for all the optimiza- tion levels. Table 3 shows the results. As UN- SUPERBINTRANS needs to first generate MAIE (see Section 3.2) and then perform the transla- tion, we also explore different tools for generat- ing MAIE and assess which one yields better per- formance. Specifically, we use fastTtext and word2vec. The results demonstrate that we achieve reasonable accuracies for each optimization level. Moreover, fastText leads to better performance.
Note that in this task, we use weighted sum of instruction embeddings to generate function em- beddings. As there many advanced methods for composing word embeddings, including those us- ing machine/deep learning models (Lin et al., 2017; Kenter et al., 2016; Xing et al., 2018), we expect that by adopting these advanced methods, we can achieve even higher accuracies.
Baseline Method. Our work aims to address the data scarcity issue in low-resource ISAs. To achieve this, we introduce UNSUPERBINTRANN which can translate binaries from low-resource ISA to high-resource ISA. By applying UN- SUPER BINTRANS, we can use the SVM model trained on x86 to test a binary in ARM by translat- ing the ARM binary to x86.

---

Table 4: Performance of the vulnerability discovery task (%).
True Positive Rate
False Positive Rate
Precision
Fl-score
Opt. Level
Case Study
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
1.00
1.00
0.00
0.0001
1.00
0.95
1.00
0.98
00
II
1,00
1.00
0.0002
0,0001
0.83
0.95
0,91
1.00
III
1.00
1.00
0.00
0.0001
1.00
0.95
1.00
0.98
1.00
1.00
0.00
0.0001
1.00
0.95
1.00
0.98
01
II
1.00
1.00
0.0003
0.0002
0.77
0.83
0.87
0.91
III
1.00
1.00
0.00
0.0002
1.00
0.83
1.00
0.91
I
1.00
1.00
0.00
0.00
1.00
1.00
1.00
1.00
02
II 1.00
1.00
0.0001
0.00
0.91
1.00
0.95
1.00
1.00
1.00
0.00
0.00
1.00
1.00
1.00
I 1.00
1.00
0.00
0.00
1,00
1.00
1.00
1.00
03
II 1.00
1.00
0.0002
0.00
0.83
1.00
0.91
1.00
1.00
1.00
0.00
0.0001
1.00
0.95
1.00
0.98
To showcase the effectiveness of the translation, perform baseline comparison. The baseline model consider is SVM model trained and tested on ARM, without employing any translation. As one would expect, the model trained and tested on ARM is likely to outperform a model trained on x86 and tested on the translated code (from ARM to x86). That is, the baseline model represents the best-case scenario. Moreover, if the performance difference between the baseline and our model is small, it signifies effective translation.
5 Related Work
The related work can be divided into: traditional and machine/deep learning based. Each one can be further divided into: mono-architecture and cross- architecture based.
5.1 Traditional Code Similarity Comparison
Mono-Architecture Approaches. Most traditional approaches work on a single ISA. Some analyze source code (Kamiya et al., 2002; Jiang et al., 2007; Luo and Zeng, 2016). Others analyze bi- nary code (Luo, 2020; Zeng et al., 2019b; Luo et al., 2019a; Zeng et al., 2019a, 2018; Luo et al., 2016), e-g., using symbolic execution (Luo et al., 2014, 2021, 2017b), but are expensive. Dynamic approaches include API birthmark (Tamada et al., 2004), system call birthmark (Wang et al., 2009a), instruction birthmark (Tian et al., 2013), and core- value birthmark (Jhi et al., 2011). However, extend- ing them to other ISAs would be hard and tedious. Moreover, code coverage is another challenge.
Table shows the results. We can see that our model consistently demonstrates close proximity in most cases when comparing to the baseline. The across all
performance of UNSUPERBINTRANL performance metries is impressive. For example, UXSUPERBINTRANE has a 100% true positive rate, indicating identifies the ARM vulnerable func- tions accurately. Therefore, we can conclude that UNsuPERBinTe BINTR ANS has exceptional translation capabilities and can be applied to the vulnerability discovery task with high reliability.
Cross-Architecture Approaches. Recent works have applied traditional approaches to the cross- architecture scenario (Pewny al., 2015; Es- chweiler al., 2016; Chandramohan al., 2016; Feng et al.; David et al., 2018). Multi-MH and Multi-k-MH (Pewny et al., 2015) are the first two methods for comparing functions across ISAs, but their fuzzing- based basic-block similarity compar- ison and graph (i.e., CFG) matching matching-based algo- rithms expensive. discovRE (Eschweiler et al., 2016) uses pre-filtering to boost CFG-based match- ing process, but still expensive and unreliable. Esh (David al., 2016) uses data-flow slices of ba- sic blocks as comparable unit. uses a SMT solver to verify function similarity, which is unscalable.
4.6 Efficiency
We evaluate the training time of UNSUPERBIN- TRANS, which needs to first generate MAIE using fastText (Part 1) and then learn CAIE (Part IT). After that, we train UNSUPERBINTRANS using CAIE and two nono-architecture datasets (Part III). The total training time is the sum of the three parts. Part approximately around 20 minutes for MAIE generation of ARM and x86. Part takes around 15 minutes for CAIE generation. Part III takes approximately 16 hours for the training of each optimization level. Thus, the total training time is around 16.5 hours.

---

5.2 Machine/Deep Learning-based Code Similarity Comparison
to a high-resource ISA, our approach has several advantages. One advantage is that we can gener- ate function embeddings by directly summing the CATE of instructions within a function and use the function embeddings to measure similarity (as il- lustrated in the function similarly comparison task), This eliminates the need for downstream model training, which however is required by UniMap. An- other advantage is that can directly use the ex- isting downstream models that have already been trained on the high-resource ISA to test the trans- lated code (as illustrated the vulnerability dis- covery task). However, UniMap needs to retrain the existing models using the learned CAIE.
Mono-Architecture Approaches. Recent research has demonstrated the applicability of machine/deep learning techniques to code analysis (Han et al., 2017: Ding et al., 2019; Van Nguyen et al., 2017; Phan et al., 2017; Yan et al., 2019; Massarelli et al.. 2019). Lee et al. propose Instruction innvec for converting assembly instructions to vector repre- sentations (Lee et al., 2017). PalmTree (Li et al., 2021) generates instruction embeddings by adopt- ing the BERT model (Devlin al., 2018). However, all these approaches work on a single ISA.
Crost-Architecturee Approaches. A set of ap- proaches target cross-architecture binary analy- sis (Feng et al., 2016; Xu et al., 2017; Chandramo- han et al., 2016; Zuo et al., 2018; Redmond et al., 2019). Some exploit the code statistical. For ex- ample, Gemini (Xu al., 2017) use manually se- lected features (e.g., the number of constants and function calls) to represent basic blocks, but ig- nore the meaning of instructions and dependency between them, resulting in significant information loss. InnerEye (Zuo et al., 2018) uses LSTM to encode each basic block into embedding, but needs to train separate model for each ISA.
6 Discussion
Intermediate representation (IR) can be used to rep- resent code of different ISAs (angr Documentation). Given two pieces of binaries from different ISAs, which have been compiled from the same piece of source code, even if they are converted into a common IR, the resulting IR code still looks quite different (see Figure and 3 of (Pewny et al., 2015) and the discussion in (Wang et al., 2023)). As a result, existing works that leverage IR for analyz- ing binaries across ISAs have to perform further advanced analysis on the IR code (Pewny et al.. 2015; David et al., 2017; Luo et al., 2019b, 2023).
For these approaches, in order to train their mod- els, cross-architecture signals (i.e, labeled similar and dissimilar pairs of code samples across AS) are needed, which require a lot of engineering ef- forts. For example, InnerEye (Zuo et al., 2018) modifies the backends of various ISAs in the LLVM compiler (LLVM) generate similar and dissim- ilar basic block pairs in different ISAs, while the dataset collection is identified as one of the chal- lenges of this work. We instead build an unsuper- vised binary code translation model that does not require large parallel corpora. More importantly, our work can resolve the data scarcity problem in low-resource IS
Conclusion
Deep learning has been widely adopted for binary code analysis. However, the limited availability of data for low-resource ISAs hinders automated analysis. In this work, we proposed UNSuPERBIN- TRANS, which can leverage the easy availability of dataset in x86 to address the data scarcity problem in low-resource ISAs. We conducted experiments to evaluate the performance. For vulnerability dis- covery, UNSUPERBINTRANE is able detect all vulnerable functions across ISAs. Thus, UNSU- PER BINTRANS offers a viable solution for address- ing the data scarcity issue in low-resource ISAs. ALP-inspired binary code analysis is promis- ing research direction, but not all NLP techniques are applicable to binary code analysis. Therefore, works like ours that identify and study effective NLP techniques for binary code analysis are valu- able in advancing exploration along this direction. This work demonstrates how NI.P techniques can be leveraged binary code analysis and under- scores the practical utility of such approaches.
The concurrent work. UniMap (Wang et al., 2023), also aims to address the data scarcity issue in low-resource ISAs. We proposed entirely differ- ent approaches. UniMap learns cross-architecture instruction embeddings (CAIE) to achieve the goal. In contrast, work takes more progressive stride by translating code from low-resource ISAs a high-resource ISA. As shown in Figure UniMap stops the CATE learning stage, while our work moves beyond this point, concentrating on bi- nary code translation, As we translate binary code

---

Limitations
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- rion for Computational Linguistics.
the evaluation, we consider two ISAs, x86 and ARM. There are many other ISAs, such as MIPS and PowerPC. We leave the evaluation on whether our approach works for other ISAs as future work. In this work, we consider instructions as words, which is a natural choice and works well accord- ing our evaluation. However, there exist other choices. For instance, we may also consider raw bytes (Liu et al., 2018; Guo et al., 2019; Raff et al., 2018) or opcode and operands (Ding et al., 2019; Li et al., 2021) as words. An intriguing research work is to study which choice works the best and whether it depends on ISAs and downstream tasks.
Bugra Cakir and Erdogan Dogdu, 2018. Malware clas- sification using deep learning methods. In ACMSE.
Mahinthan Chandramohan, Yinxing Xue, Zhengzi Xu, Yang Liu, Chia Yuan Cho, and Hee Beng Kuan Tan. 2016. Bingo: cross-architecture cross-os binary search. In FSE.
Nitesh Chawla, Kevin Bowyer, Lawrence Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artift- cial intelligence research.
Given the large variety of binary analysis tasks and their complexity, proach be applied to all tasks. The fact that it works well for two important tasks (vulnerability discovery and code similarity detection). Demon- strating our approach for other applications of great use. Much research can be done for exploring and expanding the boundaries of the approach.
Zheng Leong Chua, Shiqi Shen, Prateek Saxena, and Zhenkai Liang. 2017. Neural nets can learn function type signatures from binaries. In USENIX Security.
do not claim that ap-
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04007.
Yaniv David, Nimrod Partush, and Eran Yahav. 2016. Statistical similarity of binaries. In PLDI.
Ethics Statement
Yaniv David, Nimrod Partush, and Fran Yahav. 2017. Similarity of binaries through re-optimization. ACM SIGPLAN Notices.
We state that this work does not violate any of the ethics and principles of scientific research in the field of computer science.
Yaniv David, Nimrod Partush, and Eran Yahav. 2018. Firmup: Precise static detection of common vulnera- bilities in firmware. In the 23rd International Con- ference on Architectural Support for Programming Languages and Operating Systems.
A Acknowledgments
This work was supported in part by the US Na- tional Science Foundation (NSF) under grants CNS-2304720. The authors would like to thank the anonymous reviewers for their valuable com- ments.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1...v.181.
References
Steven HH Ding, Benjamin CM Fung, and Philippe Charland. 2019. Asm2vec: Boosting static represen- tation robustness for binary clone search against code obfuscation and compiler optimization. In S&P.
angr Documentation. Intermediate representation. https: //docs. angr. witp///aaaee-ttopcc..ooooooo
Mikel Artetxe, Gorka Labaka. and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. arXiv preprint XXXv:1805.062:
Sebastian Fschweiler, Khaled Yakdan, and Elmar Gerhards-Padilla. 2016. discovRE: Efficient cross- architecture identification of bugs in binary code. In NDSS.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018b. Unsupervised neural ma- chine translation. In ICLR.
Qian Feng, Minghua Wang, Mu Zhang, Rundong Zhou, Andrew Henderson, and Heng Yin. Extracting con- ditional formulas for cross-platform bug search. In the 2017 ACM Asia Conference Computer and Communications Security.
James Banks. 2015. The heartbleed bug: Insecurity repackaged, rebranded and resold. Crime, Media, Culture.
Qian Feng. Rundong Zhou, Chengeheng Xu, Yao Cheng, Brian Testa, and Heng Yin. 2016. Scalable graph-based bug search for firmware images. In CCS.
BAP. 2011. Bap: Binary analysis platform. https: //gi thub. htm//Biinrrynalllssssaattfoooooo

---

Wenbo Guo, Dongliang Mu, Xinyu Xing. Min Du, and Dawn Song. 2019. DEEPVSA: Facilitating value-set analysis with deep learning for postmortem program analysis. In USENIX Security.
Lannan Luo, Jiang Ming, Dinghao Wu, Peng Liu, and Sencun Zhu. 2014. Semantics- obfuscation- resilient binary code similarity comparison with ap- plications software plagiarism detection. In FSE.
Zhuobing Han, Xiaohong Li, Zhenchang Xing, Hongtao Liu, and Zhiyong Feng. 2017. Learning to predict severity of software vulnerability using only vulnera- bility description. In IEEE International Conference on Software Maintenance and Evolution.
Lannan Luo, Jiang Ming, Dinghao Wu, Peng Liu, and Sencun Zhu. 2017a. Semantics-basee obfuscation- resilient binary code similarity comparison with ap- plications software and algorithm plagiarism detec- tion. IEEE Transactions on Software Engineering.
Yoon-Chan Jhi, Xinran Wang. Xiaoqi Jia, Sencun Zhu, Peng Liu, and Dinghao Wu. 2011. 1. Value-based pro- gram characterization and its application to software plagiarism detection. In ICSE.
Lannan Luo and Qiang Zeng. 2016. Solminer: min- ing distinct solutions in programs. In International Conference on Software Engineering Companion.
Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. 2007. Deckard: Scalable and ac- curate tree-based detection of code clones. In ICSE.
Lannan Luo, Qiang Zeng, Chen Cao, Kai Chen, Jian Liu, Limin Liu, Neng Gao, Min Yang. Xinyu Xing, and Peng Liu. 201 System service call-oriented symbolic execution of android framework with appli- cations to vulnerability discovery and exploit genera- tion. In MobiSys.
Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. 2002. CCFinder: multilinguistic token-based code clone detection system for large scale source code. IEEE Transactions on Software Engineering.
Lannan Luo, Qiang Zeng, Chen Cao, Kai Chen, Jian Liu, Limin Liu, Neng Gao. Min Yang. Xinyu Xing. and Peng Liu. 2019a. Tainting-assien assisted and context- migrated symbolic execution of android framework for vulnerability discovery and exploit generation. IEEE Transactions on Mobile Computing.
Tom Kenter, Alexey Borisov, and Maarten De Rijke. 2016. Siamese chow: Optimizing word embed- dings for sentence representations. arXiv preprint 1xXi:/6006006400.
Young Jun Lee, Sang-Hoon Choi, Chulwoo Kim, Seung- Ho Lim, and Ki- Ki-Woong Park. 2017. Learning binary code with deep learning to detect software weakness. In KSII the 9th International Conference Internet.
Lannan Luo, Qiang Zeng, Bokai Yang, Fei Zuo, and Junzhe Wang. 2021. Westworld: Fuzzing-assien assiisted remote dynamic symbolic execution of smart apps on iot cloud platforms. In ACSAC.
Xuezixiang Li. Qu Yu, and Heng Yin. 2021. Palmtree: Learning an assembly language model for instruction embedding. In CCS.
Zhenhao Luo, Baosheng Wang, Yong Tang, and Wei Xie. 2019b. comantic-basee representation binary clone detection for cross-architectues in the internet of things. Applied Sciences.
Guanjun Lin. Jun Zhang, Wei Luo, Lei Pan. Olivier De Vel, Paul Montague, and Yang Xiang. 2019. Software vulnerability discovery via learning multi- domain knowledge bases. IEÉE Transactions on Dependable and Secure Computing.
Zhenhao Luo, Pengfei Wang, Baosheng Wang, Yong Tang, Wei Xie, Xu Zhou, Danjun Liu, and Kai Lu. 2023. Vulhawk: Cross-architecture vulnerability de- tection with entropy-based binary code search. In NDSS.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos San- tos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In ICLR.
Luca Massarelli, Giuseppe Antonio Di Luna, Fabio Petroni. Roberto Baldoni, and Leonardo Querzoni. 2019. Safe: Self-attentive function embeddings for binary similarity. International Conference on De- tection of Intrusions and Malware, and Vulnerability Assessment. Springer.
Bingchang Liu, Wei Huo, Chao Zhang, Wenchao Li, Feng Li, Aihua Piao, and Wei Zou. 2018. adiff: cross-version binary code similarity detection with dnn. In ASE,
Alejandro Moreo, Andrea Esuli, and Fabrizio Sebastiani. 2016. Distributional random oversampling for imbal- anced text classification. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval.
LLVM. The LLVM compiler infrastructure. https: //1lvm. org.
Lannan Luo. 2020. Heap memory snapshot assisted pro- gram analysis for android permission specification. In IEEE 27th International Conference on Software Analysis. Evolution and Reengineering.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics.
Lannan Luo, Yu Fu, Dinghao Wu, Sencun Zhu, and Peng Liu. 2016. sopaakagge-rooofing android apps. In DSN.

---

Jannik Pewny, Behrad Garmany, Robert Gawlik, Chris- tian Rossow, and Thorsten Holz. 2015. Cross- architecture bug search in binary executables. In S&P.
Xinran Wang, Yoon-Chan Jhi, Sencun Zhu, and Peng Liu. 2009a. Behavior based software theft detec- tion. In the 16th ACM conference on Computer and communications security.
Anh Viet Phan, Minh Le Nguyen, and Lam Thu Bui.
Xinran Wang, Yoon-Chan Jhi, Sencun Zhu, and Peng Liu. 2009b. Detecting software theft via system call based birthmarks. In Computer Security Applications Conference.
2017. Convolutional neural networks over control flow graphs for software defect prediction. In the 29th International Conference Tools with Artificial Intelligence.
Fang Wu. Jigang Wang. Jiqiang Liu, and Wei Wang. 2017. Vulnerability detection with deep learning. In the 3rd IEEE international conference on computer and communications.
Edward Raff, Jon Barker, Jared Sylvester, Robert Bran- don, Bryan Catanzaro, and Charles Nicholas. 2018. Malware detection by eating a whole exe. In Work- shops at the 32 AAAI Conference on Artificial Intelli- gence.
Wenhui Xing, Xiaohui Yuan, Lin Li, Lun Hu, and Jing Peng. 2018. Phenotype extraction based on word em- bedding sentence embedding cascaded approach. IEEE trunsactions on nanobioscience.
Kimberly Redmond, Lannan Luo, and Qiang Zeng. 2019. crost-architecture instruction embedding model for natural language processing-inspiredd bi- nary code analysis. In NDSS Workshop Binary Analysis Research.
Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song. and Dawn Song. 2017. Neural network-based graph embedding for cross-ple pooss-platfomm binary code similarity detection. In CCS.
Sebastian Ruder, Anders Søgaard, and Ivan Vulié. 2019. Unsupervised cross-lingual representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts.
Jiaqi Yam, Guanhua Yan, and Dong Jin. 2019. Classify- ing malware represented as control flow graphs using deep graph convolutional neural network. In DSN.
Dawn Song. D Brumley, H Yin, J Caballero, I Jager, MG Kang, 7. Liang, J Newsome, P Poosankam, and P Saxena. 2013. Bitblaze: Binary analysis for com- puter security.
Qiang Zeng. Golam Kayas, Emil Mohammed, Lan- nan Luo, Xiaojiang Du and Junghwan Rhee. 2019a. Heaptherapy+: Efficient handling of (almost) all heap vulnerabilities using targeted calling-context encod- ing. In DSN.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Se- quence sequence learning with neural networks. In Advances in neural information processing systems.
Qiang Zeng. Lannan Luo, Zhiyun Qian, Xiaojiang Du, and Zhoujun Li. 2018. Resilient decentralized an- droid application repackaging detection using logic bombs. In International Symposium on Code Gener- ation and Optimization.
Haruaki Tamada, Keiji Okamoto, Masahide Nakamura, Akito Monden, and Ken-ichi Matsumoto. 2004. Dy- namic software birthmarks to detect the theft of win- dows applications. In International Symposium on Future Software Technology.
Qiang Zeng, Lannan Luo, Zhiyun Qian, Zhoujun Li, Chin- Huang, and Csilla Farkas. 2019b. Re- silient user-side android application repackaging and tampering detection using cryptographically obfus- cated logic bombs. IEEE Transactions on Depend- able and Secure Computing (TDSC).
The IDA Pro Disassembler and Debugger. http:// www. datarescue com/ idabase/.
Zhenzhou Tian, Qinghua Zheng, Ting Liu, and Ming Fan. 2013. Dkish: Dynamic key instruction sequence birthmark for software plagiarism detection. In 2013 IEEE 10th International Conference on High Per- formance Computing and Communications 2013 IEEE International Conference on Embedded and Ubiquitous Computing.
Fei Zuo, Xiaopeng Li, Zhexin Zhang, Patrick Young. Lannan Luo, and Qiang Zeng. 2018. Neural ma- chine translation inspired binary code similarity comparison beyond function pairs. arXiv preprint (NDSS' 19).
Thanh Van Nguyen, Anh Tuan Nguyen, Hung Dang Phan, Trong Due Nguyen, and Tien N Nguyen. 2017. Combining word2vec with revised vector space model for better code retrieval. In the 39th International Conference on Saftware Engineering Companion.
Junzhe Wang, Matthew Sharp. Chuxiong Wu. Qiang Zeng, and Lannan Luo. 2023. Can a deep learn- ing model for one architecture be used for others? Eeeargeted-Architeee
binary code analysis. In 32nd USENIX Security Symposium.


Reviews:
[{"review_id": "XrNgFB6wY5", "reviewer": "Reviewer_6b4d", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the insightful and valuable comments. Please find our response below.\n\n> 1. “*While the primary contribution is about the translation of low-resource architecture code, the paper lacks detailed information and analysis of these results. Section 5.2 only presents BLEU scores without any in-depth analysis or explanation. The reported scores don't seem to vary significantly across different optimizations and settings, which raises questions about how the model converged, among other things. These aspects need more elaboration.*”\n\n**Response**: In Section 5.2, we presented the BLEU scores when translating ARM to x86 in terms of four different optimization levels. The BLEU scores are around 0.76 and 0.77, which shows the good translation performance of our model (given that BLEU scores range from 0 to 1, with higher scores indicating better translation quality). \n\nWe then used a few randomly selected samples as examples to illustrate the robust translation capability of our model. We are open to further analysis and explanations. Please kindly apprise us of your thoughts, and we would be more than willing to include them into the revision. \n\nWe find ourselves uncertain regarding your question: “*The reported scores don't seem to vary significantly across different optimizations and settings, which raises questions about how the model converged, among other things.*” Our perspective contrasts here, as we observe minimal score variance, showing the convergence of our model. \n\n> 2. “*The code similarity and vulnerability discovery task seem simplistic. The definition of similarity is too dependent on same name and derived from same source code. What about the case when two functions have different names but they do the same task. No significant differences are discernible compared to the baseline models.*”\n\n**Response**: Besides function names, we used a method (n-gram) similar to that in InnerEye (Zuo et al., 2019) (see Section V.D.2) to double check whether two functions are indeed dissimilar.\n\n**No significant differences between our model and the baseline model are what we expected, which show the good translation capability of our approach**. Our work aims to address the data scarcity issue in low-resource ISAs for binary code analysis. To achieve this, we introduce UNSUPERBINTRANS, a model trained to translate assembly code in one ISA to assembly code in the data-rich ISA x86. Through the application of UNSUPERBINTRANS, we can resolve the data scarcity issue. Specifically, given a downstream binary analysis task, we can use the downstream model trained on x86 to test binary code in another ISA by translating the code to x86. \n\nTo demonstrate the translation effectiveness of our approach, we conducted a baseline comparison, wherein we trained a downstream model on ARM and tested the trained model on ARM, without employing any translation. As one would anticipate, the model trained and tested on ARM data should outperform a model trained on x86 and tested on translated ARM data. \n\nIf the performance discrepancy is small, it signifies effective translation. Our results in Table 4 show that our model consistently demonstrates close proximity in most cases when comparing to the baseline. Thus, our approach has good translation capabilities.\n\n> 3. “*There have been previous studies on intermediate representation-based binary analysis and rewriting systems. It's essential to highlight how your approach is different or improved compared to existing methods. I found no comparisons in your paper.*”\n\n**Response**: Existing studies that leverage intermediate representation (IR) cannot resolve the data scarcity issue in low-resource ISAs. First, given two pieces of binaries from different ISAs, which have been compiled from the same piece of source code, even if they are converted into a common IR, the resulting IR code still looks quite different. As a result, existing works that leverage IR for analyzing binaries across ISAs have to perform further advanced analysis (e.g., training deep learning models) on the IR code.\n\nSecond, for a given downstream task, existing works require the task-specific data from all ISAs considered to train their downstream models. In contrast, for a given downstream task, our approach only needs the task-specific data in a data-rich ISA (such as x86) to train a downstream model on x86. Then, when confronted with the need to analyze code in a low-resource ISA, we can employ our model UNSUPERBINTRANS to translate the code to x86, and use the downstream model (trained on x86) to test the translated code. \n\nIt should be highlighted that the training of UNSUPERBINTRANS only needs unlabelled data, which can be obtained conveniently via cross-compiling some source code. This is similar to the NLP works that utilize machine translation to deal with the data scarcity issue in low-resource languages for various NLP tasks such as cross-lingual document classification. \n\nWe will include the discussion in the revision of this paper. Moreover, existing works based on intermediate representation are also discussed in Section 4.3.1 in the recently published paper: \n\n* Wang, J., Sharp, M., Wu, C., Zeng, Q. and Luo, L., 2023. Can a Deep Learning Model for One Architecture Be Used for Others? Retargeted-Architecture Binary Code Analysis. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 7339-7356).\n\n> 4. “The paper should elaborate on how this model can be generalized to handle new coding styles, functions, and more. It would be helpful if you could elaborate on the model's scalability and adaptability.”\n\n**Response**: Great suggestion! We acknowledge that the models we trained are unlikely to generalize to all kinds of code. That is an ambitious goal, and the community is not there yet, although our work can be considered as a good effort towards it; that is, the generalizability across different architectures (for certain Linux code). \n\nWe agree that the ambitious goal can be attained in one shot. To mitigate the limitation, we may increase the diversity of the dataset by involving more kinds of code (with new coding styles, functions, and more). We will elaborate this in the revision of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The concept and framework of translating low-resource binaries to high-resource binaries is a strength of this paper. \n", "reasons_to_reject": "1) While the primary contribution is about the translation of low-resource architecture code, the paper lacks detailed information and analysis of these results. Section 5.2 only presents BLEU scores without any in-depth analysis or explanation. The reported scores don't seem to vary significantly across different optimizations and settings, which raises questions about how the model converged, among other things. These aspects need more elaboration. \n2) The code similarity and vulnerability discovery task seem simplistic. The definition of similarity is too dependent on same name and derived from same source code. What about the case when two functions have different names but they do the same task. No significant differences are discernible compared to the baseline models.  \n3) There have been previous studies on intermediate representation-based binary analysis and rewriting systems. It's essential to highlight how your approach is different or improved compared to existing methods. I found no comparisons in your paper. \n4) The paper should elaborate on how this model can be generalized to handle new coding styles, functions, and more. It would be helpful if you could elaborate on the model's scalability and adaptability.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors use the ideas and techniques of neural machine translation to translate binaries in low-resource architectures into binaries in high-resource architectures. They show the quality of the translation through two downstream tasks: code similarity detection and vulnerability discovery. \n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is clear overall. \n\nAuthors should add an introduction or overview before the first sub-sections in sections 2 and 4.\n\n"}, "extra_text": {}}, {"review_id": "AG77OnmAXf", "reviewer": "Reviewer_SuGi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the insightful and valuable comments. Please find our response below.\n\n> “*The claimed novel model is not actually novel as it is a straight combination of already existing tools.*”\n \n**Response:** We will tone down the claim and properly describe the novelty of this work. Our work is the first to explore the translation of binary code between different ISAs. We leverage the ideas and techniques in NMT to a quite different area, binary code analysis. NLP-inspired binary code analysis is a promising research direction, but not all NLP techniques are applicable to binary code analysis. Therefore, works like ours that identify and study effective NLP techniques for binary code analysis are valuable in advancing exploration along this direction. We adapt NMT techniques to binary code analysis, aiming to address the data scarcity issue in low-resource ISAs.  Our evaluation shows how the adaptation works and why it is useful through two downstream tasks, function similarity comparison and vulnerability discovery.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A novel task of binary translation and a novel dataset for evaluation in this task.\n", "reasons_to_reject": "The claimed novel model is not actually novel as it is a straight combination of already existing tools.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors in this work present a novel task of binary translation and a dataset for its evaluation. The authors also use combination of pre-existent tools to create a baseline for the task.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GjRIJ623fO", "reviewer": "Reviewer_qjhC", "summary": "", "questions": "- How does this approach compare against baselines with respect to your two downstream evaluations?\n    - Author rebuttal claims to promise such evaluations. \n- In what way is this an improvement over using intermediate representations like LLVM?  Especially considering that this paper depends on translating binaries to x86, is this not similar to converting to LLVM IR or similar?\n    - partially addressed in the rebuttal\n- In what way does this paper differ from the Usenix 2023 paper, \"Can a Deep Learning Model for One Architecture Be Used for Others?\nRetargeted-Architecture Binary Code Analysis\" ?  \n    - The authors clarified this point.  It sounds like there is substantial overlap, but still novel contributions in the current submission.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the insightful and valuable comments. Please find our response below.\n\n>  1. “*no comparison against baselines. The functionality similarity comparison study reports only accuracy across optimization levels of binaries, but no baselines are considered. This is a widely-understood binary analysis application and many papers have developed architecture-agnostic similarity comparison (or often reported as codesearch, which is a similar task).*”\n\n**Response**: Thanks for the suggestion. We will conduct a baseline comparison for the function similarity comparison task. Specifically, we will consider existing deep learning-based code similarity comparison models as the baseline methods, and train and test the models on ARM. The results of the baselines will be compared with those obtained by training the models on x86 and testing them on ARM (when testing on ARM, we will first use our model UNSUPERBINTRANS to translate ARM code to x86). Through this, we can evaluate the effectiveness of our approach’s translation capabilities: If the performance discrepancy is small, it indicates effective translation.\n\n> 2. “*in addition, the functionality similarity comparison methodology is questionable. The authors use cosine similarity with respect to embeddings, which to me makes the experiment rather circular. In contrast, I might have expected some type of dynamic analysis, testing, or some other reasoning to establish semantic similarity between code snippets.*”\n\n**Response**: We first translate code from ARM to x86 using our model UNSUPERBINTRANS. Through this, we have two pieces of code, both in x86. To measure the similarity between the two pieces of code, a complex way is to train a deep learning model. However, we opted for the simplest way, aiming to better demonstrate the good translation of our model. That is, we sum the instruction embeddings of all instructions within each function, and use the sum as the function embedding (this is similar to the way of computing sentence/document embeddings using word embeddings in NLP). Subsequently, the cosine similarity between the two function embeddings is computed to measure the function similarity. We believe this simplest way, compared to training a deep learning model (which can help capture code similarity), better highlights the good translation of our model. \n\nThanks for the great suggestion. The process of translating assembly code from one ISA into the data-rich x86 ISA opens the door to employing both static and dynamic analysis techniques for code similarity comparison. In this work, we focus on static analysis techniques, wherein we employ deep learning-based models to analyze the code. While dynamic analysis techniques represent a valid direction, they are beyond the scope of this work.\n\n> 3. “*vulnerability discovery methodology is also questionable. The authors consider a single vulnerability at a time, and while they acknowledge and address the data imbalance issue, I am not sure about the ecological validity of such a study. Previous work has considered multiple CVEs or CWEs at a time, and report whether or not the code contains any such vulnerability. Are the authors arguing that identifying one vulnerability at a time is an intended use case? In any case, the results are difficult to interpret (or are marginal improvements at best).*”\n\n**Response**: For the vulnerability discovery task, there are two common handlings in the literature. The first, as mentioned by the reviewer, considers multiple CVEs or CWEs at a time. The second one, which we adopted, considers one CVE at a time, such as Gemini (Xu et al., 2017). \n\nWe want to clarify that our interpretation of the results does not focus on performance improvements; rather, our focus lies on performance discrepancies, specifically performance decreases. When comparing our model's results to the baseline, we consider our model's translation capabilities to be effective if the observed decrease is minimal. For the details, please refer to our response to the Question numbered 6 (below).\n\n> 4. “*This paper is very similar to another accepted at Usenix 2023: Can a Deep Learning Model for One Architecture Be Used for Others? Retargeted-Architecture Binary Code Analysis. In comparison to that paper, I do not quite understand the novelty here, except perhaps for a slightly different evaluation/application domain. I certainly acknowledge that this submission was made slightly before the Usenix 2023 proceedings were made available, but I would still want to understand how this differs given the overall similarity in idea (building embeddings that help a model target a new ISA).*”\n\n**Response**: The USENIX 2023 work focuses on the same goal as ours (i.e., addressing the data scarcity issue in low-resource ISAs). However, we proposed entirely different approaches. The USENIX work learns cross-architecture instruction embeddings (CAIE) to achieve the goal. In contrast, our work takes a more progressive stride by translating code in low-resource ISAs to a data-rich ISA. A visual reference to these distinctions can be found in Figure 2, where the USENIX work stops at the CAIE learning stage, while our work moves beyond this point, concentrating on binary code translation. \n\nAs we translate binary code to a data-rich ISA, our approach has several advantages. One advantage is that we can directly use the existing models that have already been trained on the data-rich ISA to test the translated code (converted into the data-rich ISA by our model UNSUPERBINTRANS). However, the USENIX work needs to **retrain** the existing models using their learned CAIE for analyzing code in different ISAs. Another advantage is that our approach opens the door to utilize existing models, which are designed to analyze code for extracting features from binaries, to analyze the translated code. However, the USENIX work lacks support of this potential direction. \n\n> 5. “relatedly, only x86 and ARM appear to be considered in the evaluation (the authors discuss building datasets for these ISAs). There are other ISAs to consider (e.g., PPC), and validating the approach against other ISAs would be important if claiming to build models that generalize to across architectures.”\n\n**Response**: Thanks for the suggestion! We will include other ISAs (e.g., PPC) into our evaluation, and conduct the downstream tasks in terms of these ISAs.\n\n> 6. “How does this approach compare against baselines with respect to your two downstream evaluations?”\n\n**Response**: Our work aims to address the data scarcity issue in low-resource ISAs for binary code analysis. To achieve this, we introduce UNSUPERBINTRANS, a model trained to translate assembly code in one ISA to assembly code in the data-rich ISA x86. Through the application of UNSUPERBINTRANS, we can resolve the data scarcity issue. Specifically, given a downstream binary analysis task, we can use the downstream model trained on x86 to test binary code in another ISA by translating the code to x86.\n\nTo demonstrate the translation effectiveness of our approach, we conducted a baseline comparison, wherein we trained a downstream model on ARM and tested the trained model on ARM, without employing any translation. As one would anticipate, the model trained and tested on ARM data should outperform a model trained on x86 and tested on translated ARM data.\n\nIf the performance discrepancy is small, it signifies effective translation. Our results in Table 4 show that our model consistently demonstrates close proximity in most cases when comparing to the baseline. Thus, our approach has good translation capabilities. \n\n> 7. “In what way is this an improvement over using intermediate representations like LLVM? Especially considering that this paper depends on translating binaries to x86, is this not similar to converting to LLVM IR or similar?”\n\n**Response**: Our approach differs from the process of converting binaries to LLVM IR. Given two pieces of binaries from different ISAs, which have been compiled from the same piece of source code, even if they are converted into a common IR, the resulting IR code still looks quite different. Please see examples shown in Figure 1 and Figure 3 of the work (Pewny et al., 2015). Moreover, the recent USENIX 2023 work also discusses IR in Section 4.3.1. Thus, binary code similarity (which cannot be addressed through code lifting to IR) has motivated intensive studies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- important problem.  Binary analysis is challenging, and representing binaries for deep neural networks is quite appropriate given the structure and difficulty in comprehending binary or assembly instructions -- it is a very natural application. \n- cross architectural binary analysis is quite important given the increasing prevalence of embedded ISAs combined with the longstanding widespread availability of x86(-64)-based analyses.", "reasons_to_reject": "- no comparison against baselines.  The functionality similarity comparison study reports only accuracy across optimization levels of binaries, but no baselines are considered.  This is a widely-understood binary analysis application and many papers have developed architecture-agnostic similarity comparison (or often reported as codesearch, which is a similar task). \n    - rebuttal promises to add this evaluation\n- in addition, the functionality similarity comparison methodology is questionable.  The authors use cosine similarity with respect to embeddings, which to me makes the experiment rather circular.  In contrast, I might have expected some type of dynamic analysis, testing, or some other reasoning to establish semantic similarity between code snippets.\n    - rebuttal addresses this point. \n- vulnerability discovery methodology is also questionable.  The authors consider a single vulnerability at a time, and while they acknowledge and address the data imbalance issue, I am not sure about the ecological validity of such a study.  Previous work has considered multiple CVEs or CWEs at a time, and report whether or not the code contains any such vulnerability.  Are the authors arguing that identifying one vulnerability at a time is an intended use case?  In any case, the results are difficult to interpret (or are marginal improvements at best). \n    - addressed in rebuttal\n- This paper is very similar to another accepted at Usenix 2023: Can a Deep Learning Model for One Architecture Be Used for Others?\nRetargeted-Architecture Binary Code Analysis.  In comparison to that paper, I do not quite understand the novelty here, except perhaps for a slightly different evaluation/application domain.   I certainly acknowledge that this submission was made slightly before the Usenix 2023 proceedings were made available, but I would still want to understand how this differs given the overall similarity in idea (building embeddings that help a model target a new ISA).\n     - addressed in rebuttal\n- relatedly, only x86 and ARM appear to be considered in the evaluation (the authors discuss building datasets for these ISAs).  There are other ISAs to consider (e.g., PPC), and validating the approach against other ISAs would be important if claiming to build models that generalize to across architectures. \n    - author rebuttal promises a followup evaluation", "questions_to_authors": "- How does this approach compare against baselines with respect to your two downstream evaluations?\n    - Author rebuttal claims to promise such evaluations. \n- In what way is this an improvement over using intermediate representations like LLVM?  Especially considering that this paper depends on translating binaries to x86, is this not similar to converting to LLVM IR or similar?\n    - partially addressed in the rebuttal\n- In what way does this paper differ from the Usenix 2023 paper, \"Can a Deep Learning Model for One Architecture Be Used for Others?\nRetargeted-Architecture Binary Code Analysis\" ?  \n    - The authors clarified this point.  It sounds like there is substantial overlap, but still novel contributions in the current submission.", "paper_topic_and_main_contributions": "The authors develop an approach to retargeting a deep neural network used for binary analysis on one ISA to work with others.  The issue is that certain esoteric or more rarely-used architectures have a scarce amount of well-labeled data that preclude training large, deep neural networks.  The authors develop cross-architectural instruction embeddings that abstract commonalities between high-resource ISAs like x86 and a lower-resource ISA like ARM or PPC.  The CAIEs are generated by mapping instruction embeddings of one program across multiple ISAs into the same embedding space.  Ultimately, the technique involves translating programs represented in one ISA to another ISA.  \n\nThe technique translates a given low-resource ISA binary into another high-resource ISA (i.e., x86).  The translated binaries can then be used in subsequent downstream tasks.  The authors consider functional similarity comparison and vulnerability detection.  The authors provide relevant metrics (accuracy for functionality similarity comparison and F-1 for vulnerability detection).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Wang et al. Can a Deep Learning Model for One Architecture Be Used for Others? Retargeted-Architecture Binary Code Analysis.  Usenix 2023."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the main strength of the paper is the proposed conceptual framework for translating binaries, and the dataset created to evaluate it. That said, the paper does not compare against existing baselines for the function similarity comparison task. The paper over-claims some of the contributions and novelty, but based on rebuttal response, the authors intend to change it in the revision, including clearly contrasting the work and highlighting novelty over the Usenix 2023 paper.
Decision: Accept-Findings