Title: MAXIMAL DIVERGENCE SEQUENTIAL AUTO-ENCODER FOR BINARY SOFTWARE VULNERABILITY DETECTION *

Abstract: Due to the sharp increase in the severity of the threat imposed by software vulnerabilities, the detection of vulnerabilities in binary code has become an important concern in the software industry, such as the embedded systems industry, and in the field of computer security. However, most of the works in binary code vulnerability detection has relied on handcrafted features which are manually chosen by a select few domain experts. In this paper, we attempt to alleviate this severe binary vulnerability detection bottleneck by leveraging recent advances in deep learning representations and propose the Maximal Divergence Sequential Auto-Encoder. In particular, latent codes representing vulnerable and non-vulnerable binaries are encouraged to be maximally divergent, while still being able to maintain crucial information from the original binaries. We conducted extensive experiments to compare and contrast our proposed methods with the baselines, and the results indicate that our proposed methods outperform the baselines in all performance measures of interest.

1. INTRODUCTION:

Software vulnerabilities are specific flaws or oversights in a piece of software that allow attackers to perform malicious acts including exposing or altering sensitive information, disrupting or destroying a system, or taking control of a computer system or program (Dowd et al., 2006) . Due to the ubiquity of computer software, the growth and the diversity in its development process, a great amount of computer software potentially includes software vulnerabilities. This fact makes the problem of software security vulnerability identification an important concern in the software industry and in the field of computer security. Although a great effort has been made by the security community, the severity of the threat from software vulnerabilities has gradually increased over the years. Numerous exist of examples and incidents in the past two decades in which software vulnerabilities have imposed significant damages to companies and individuals (Ghaffarian & Shahriari, 2017) . For example, vulnerabilities in popular browser plugins have threatened the security and privacy of millions of Internet users (e.g., Adobe Flash Player (US-CERT 2015; Adobe Security Bulletin 2015) and Oracle Java (US-CERT 2013)), vulnerabilities in popular and fundamental open-source software have also threatened the security of thousands of companies and their customers around the globe (e.g., Heartbleed (Codenomicon 2014) and ShellShock (Symantec Security Response 2014).

Software vulnerability detection (SVD) can be categorized into source code and binary code vulnerability detection. Source code vulnerability detection has been widely studied in a variety of works (Shin et al., 2011; Neuhaus et al., 2007; Yamaguchi et al., 2011; Li et al., 2016; Kim et al., 2017; Li et al., 2018) . Most of the previous work in source code vulnerability detection (Neuhaus et al., 2007; Shin et al., 2011; Yamaguchi et al., 2011; Li et al., 2016; Kim et al., 2017) has been based on handcrafted features which are manually chosen by a limited number of domain experts. To mitigate the dependency on handcrafted features, the use of automatic features in SVD has been studied recently in (Dam et al., 2017; Li et al., 2018; Lin et al., 2018) . In particular, Dam et al. (2017) ; Lin et al. (2018) employed a Recurrent Neural Network (RNN) to transform sequences of code tokens to vectorial features, which are further fed to a separate classifier, while Li et al. (2018) combined learning the vector representation and the training of the classifier in a deep network.

Compared with source code vulnerability detection, binary code vulnerability detection is significantly more difficult because much of the syntactic and semantic information provided by high-level programming languages is lost during the compilation process. The existence of such syntactic and semantic information makes it easier to reason how data and inputs drive the paths of execution. Unfortunately, a software binary, such as proprietary binary code (with no access to source code) or embedded systems code, is generally all that is made available for code analysis (together perhaps with the processor architecture such as x86 etc.). The ability to detect the presence or absence of vulnerabilities in binary code, without getting access to source code, is therefore a major importance in the context of computer security. Some work has been proposed to detect vulnerabilities at the binary code level when source code is not available, notably work based on fuzzing, symbolic execution (Cadar & Sen, 2013; Avancini & Ceccato, 2013; Meng et al., 2016) , or techniques using handcrafted features extracted from dynamic analysis (Grieco et al., 2016; Cozzie et al., 2008; White & Lüttgen, 2013) . To the best of our knowledge, there has been no work studying the use of automatically extracted features for binary code vulnerability detection, though there has been some work using automatic features in conjunction with deep learning methods for malware detection, notably (Saxe & Berlin, 2015; Raff et al., 2017) . It is worth noting that binary code vulnerability detection and malware detection are two different tasks. In particular, binary code vulnerability detection aims to detect specific flaws or oversights in binary code, while malware detection aims to detect if a given binary is malicious or not. The former is arguably harder in the sense that vulnerable and non-vulnerable binaries might be only slightly different, while there might be a clearer difference in general between malware and benign binaries.

In addition, a significant constraint in research on binary code vulnerability detection is the lack of suitable binaries labeled as either vulnerable or non-vulnerable. Although we have some source code datasets for software vulnerability detection, to the best of our knowledge, there exists no large public binary dataset for the purpose of binary code vulnerability detection. The reason is that most source code in source code vulnerability detection datasets is not compilable due to incompleteness, and they have important pieces missing (e.g., variables, data types) and relevant libraries -making the code compilable take a large effort in fixing a vast volume of source code. This arises from the nature of the process that involves collecting and labeling source code wherein we start from security reports in CVE1 and navigate through relevant websites to obtain code snippets of vulnerable and non-vulnerable source code.

In this work, we leverage recent advances in deep learning to derive the automatic features of binary code for vulnerability detection. In particular, we view a given binary as a sequence of machine instructions and then use the theory of Variational Auto-Encoders (VAE) (Kingma & Welling, 2013) to develop the Maximal Divergence Sequential Auto-Encoder (MDSAE) that can work out representations of binary code in such a way that representations of vulnerable and nonvulnerable binaries are encouraged to be maximally different for vulnerability detection purposes, while still preserving crucial information inherent in the original binaries. In contrast to the original VAE wherein the data prior is kept fixed, we propose using two learnable Gaussian priors, one for each class. Based on the VAE principle, latent codes (i.e., data representations) are absorbed (or compressed) into the data prior distribution, we further propose maximizing a divergence (e.g., Wasserstein (WS) distance or Kullback-Leibler (KL) divergence) between these two priors to separate representations of vulnerable and non-vulnerable binaries. Our MDSAE can be used to produce data representations for another independent classifier (e.g., Support Vector Machine or Random Forest) or incorporated with a shallow feedforward neural network built on top of the latent codes for simultaneously training both the mechanism to generate data representations and the classifier. The former is named MDSAE-R and the latter is named MDSAE-C. We summarize our contributions in this paper as follows:

• We propose a novel method named Maximal Divergence Sequential Auto-Encoder (MDSAE) that leverages recent advances in deep learning representation (namely, VAE) for binary code vulnerability detection.

• One of our most significant contributions is to create a labeled dataset for use in binary code vulnerability detection. In particular, we used the source code in the published NDSS18 dataset used in (Li et al., 2018) and then extracted vulnerable and non-vulnerable functions. We developed a tool that can automatically detect the syntactical errors in a given piece of source code, fix them, and finally compile the fixed source code into binaries for various platforms (both Windows OS and Linux OS) and architectures (both x86 and x86-64 processors). Specifically, after preprocessing and filtering out identical functions from the NDSS18 source code dataset, we obtain 13, 000 functions of which 9, 000 are able to be fixed and compiled to binaries. By compiling the source code of these functions under the various platform and architecture options, we obtain 32, 281 binary functions including 17, 977 binaries for Windows and 14, 304 binaries for Linux.

• We conducted extensive experiments on the NDSS18 binary dataset. The experimental results show that the two variants MDSAE-R and MDSAE-C outperform the baselines in all performance measures of interest. It is not surprising that MDSAE-C achieves higher predictive performances compared with MDSAE-R, but the fact that MDSAE-R achieves good predictive performances confirms our hypothesis of encouraging the separation in representations of data in different classes so that a simple linear classifier subsequently trained on these data representations can obtain good predictive results.

2. RELATED BACKGROUND:



2.1. THE VARIATIONAL AUTO-ENCODER:

The Variational Auto-Encoder (VAE) (Kingma & Welling, 2013 ) is a probabilistic auto-encoder that takes into account both the reconstruction of true samples and generalization of samples generated from a latent space. The underlying idea is to learn a probabilistic decoder p θ (x | z) , z ∼ N (0, I) that can mimic the true data sample x 1 , . . . , x N drawn from an existing but unknown data distribution p d (x). VAE is developed based on the following lower bound:

log p θ (x) ≥ L (x; θ, φ) = E q φ (z|x) [log p θ (x | z)] -D KL (q φ (z | x) p (z))

where q φ (z | x) is the approximate posterior distribution.

We need to maximize the log likelihood at each training example x. Therefore the objective function is of the following form:

max θ,φ E x E q φ (z|x) [log p θ (x | z)] -D KL (q φ (z | x) p (z)) (1)

where x is drawn from the empirical data distribution.

To reduce the variance when using Monte Carlo (MC) estimation for tackling the above optimization problem, the reparameterization trick is employed. More specifically, assuming that q φ (z

| x) = N (z | µ φ (x) , diag (σ φ (x)

)), we can do reparameterization as:

z = µ φ (x) + diag (σ φ (x)) 1/2

where the source of randomness ∼ N (0, I) and µ φ (z) , σ φ (z) are two neural networks representing the mean and covariance matrix of the approximate Gaussian posterior.

The optimization problem in Eq. ( 1) can be equivalently rewritten as:

EQUATION

The first term in Eq. ( 2) is regarded as the reconstruction term and the second term in this equation is regarded as the regularization term. In this term, we minimize

E x [D KL (q φ (z | x) p (z))],

hence trying to compress and squash the latent codes z for each true example x into those sampled from the prior distribution p (z). This observation is the key ingredient for us to develop our proposed model.

2.2. THE KULLBACK-LEIBLER DIVERGENCE AND L2 WASSERSTEIN DISTANCE:

Given two distributions with the probability density functions p (z) and q (z) where z ∈ R d , the Kullback-Leibler (KL) divergence between these two distributions is defined as:

D KL (q p) = q (z) log q (z) p (z) dz Another divergence of interest is L2 Wasserstein (WS) distance with the cost function c (z 1 , z 2 ) = z 1 -z 2 2 2

. The L2 Wasserstein divergence between two distributions is defined as:

D W S (q p) = min π∈Π(q,p) E (z1,z2)∼π z 1 -z 2 2 2

where Π (q, p) specifies the set of all joint distributions over p, q which admits p, q as marginals.

If p, q are two Gaussians, i.e., p

(z) = N (z | µ 1 , Σ 1 ) and q (z) = N (z | µ 2 , Σ 2 ) then both KL

divergence and L2 WS distance can be computed in close forms as:

D KL (q p) = 1 2 log |Σ 1 | |Σ 2 | -dtr Σ -1 1 Σ 2 + (µ 1 -µ 2 ) T Σ -1 1 (µ 1 -µ 2 ) D W S (q p) = µ 1 -µ 2 2 2 + Σ 1/2 1 -Σ 1/2 2 2 F

where • F is the Frobenius norm and Σ 1 Σ 2 = Σ 2 Σ 1 .

3. THE MAXIMAL DIVERGENCE SEQUENTIAL AUTO-ENCODER (MDSAE):

FOR BINARY VULNERABILITY DETECTION For each machine instruction, we employ the Capstone2 binary disassembly framework to detect entire machine instructions. We then eliminate redundant prefixes to obtain the core parts that contain the opcode and other significant information.

Each core part in a machine instruction consists of two parts: the opcode and instruction information (i.e., memory location, registers, etc.). We embed both the opcode and instruction information into vectors and then concatenate them. To embed the opcode, we build a vocabulary of opcodes and then multiply the one-hot vector of the opcode with the corresponding embedding matrix. To embed the instruction information, we build the vocabulary over 256 hex-bytes from 00 to F F , then view the instruction information as a sequence of hex-bytes to construct the frequency vector of a size 256, and finally multiply this frequency vector with the corresponding embedding matrix. More specifically, the output embedding is e = e op e ii where e op = one-hot(op) × W op and e ii = freq (ii) × W ii with the opcode op, the instruction information ii and its frequency vector freq (ii), and the embedding matrices W op and W ii . The process of embedding machine instructions is presented in Figure 1 .

3.2. PROPOSED MODEL:

In this work, we view binary code x as a sequence of machine instructions, i.e., x = [x i ] i=1,...,m where each x i is a machine instruction. Our idea is to encode x to the latent code z in such a way that the latent codes of data in different classes are encouraged to be maximally divergent. Let us denote the distributions of vulnerable and non-vulnerable sequences by p 1 (x) and p 0 (x) respectively. Inspired by the Variational Auto-Encoder (Kingma & Welling, 2013) , we propose to use a probabilistic decoder p θ (x | z) such that for z ∼ p 0 (z), x drawn from p θ (x | z) can mimic those drawn from p 0 (x) and for z ∼ p 1 (z), x drawn from p θ (x | z) can mimic those drawn from p 1 (x). In other words, we aim to learn the probabilistic decoder p θ (x | z) satisfying:

p 0 (x) = p θ (x | z) p 0 (z) dz and p 1 (x) = p θ (x | z) p 1 (z) dz

For any approximate posterior q φ (z | x), we have the following lower bounds:

log p k (x) ≥ L k (x; θ, φ) = E q φ (z|x) [log p θ (x | z)] -D KL q φ (z | x) p k (z) , k = 0, 1

Figure 2 : Maximal divergence sequential auto-encoder. The latent codes of vulnerable and nonvulnerable are encouraged to be maximally divergent, while still maintaining crucial information from the original binaries. Note that we use the same network for q φ (z | x, y = 0) and q φ (z | x, y = 1) and they are discriminated by the source of data used to fit.

Using the architecture shown in Figure 2 , we consider the probabilistic decoder p θ (x | z) of the following parametric form

p θ (x | z) = L i=1 p θ (x i | x 1:i-1 , z) = L i=1 p θ (x i | h i-1 , z)

and hence we can further derive the lower bounds as:

L k (x; θ, φ) = E q φ (z|x) L i=1 log p θ (x i | h i-1 , z) -D KL q φ (z | h L ) p k (z) , k = 0, 1

We arrive at the following optimization problem: max θ,φ

E x:y=0 L 0 (x; θ, φ) + E x:y=1 L 1 (x; θ, φ)

It is worth noting that since we are minimizing:

E x:y=0 D KL q φ (z | h L ) p 0 (z) + E x:y=1 D KL q φ (z | h L ) p 1 (z)

the encoding z ∼ q φ (z | h L ) with y = 0 are absorbed (compressed) into the prior p 0 (z). Similarly, the encoding z ∼ q φ (z | h L ) with y = 1 are compressed into the prior p 1 (z). Therefore, to maximize the difference between the encodings of data in the two classes, we propose to maximize the divergence between p 0 (z) and p 1 (z) and arrive the following optimization problem:

max θ,φ E x:y=0 L 0 (x; θ, φ) + E x:y=1 L 1 (x; θ, φ) + αD p 0 (z) p 1 (z)

where α > 0 is a non-negative trade-off parameter and D p 0 (z) p 1 (z) is the divergence between the two priors.

To facilitate the evaluation, we endow these two priors with Gaussian distributions as follows:

p k (z) = N (z | µ k , Σ k ) , k = 0, 1.

We also propose using the Gaussian approximate posterior as:

q φ (z | h L ) = N (z | µ φ (h L ) , diag (σ φ (h L ))) which enables the reparameterization trick: z = µ φ (h L ) + diag (σ φ (h L )) 1/2

, where the source of randomness ∼ N (0, I) and µ φ (z) , σ φ (z) are two neural networks representing the mean and covariance matrix of the approximate Gaussian posterior.

Hence we come to the following optimization problem:

EQUATION

where we note that D p 0 (z) p 1 (z) is tractable for both the KL-divergence and L2 Wasserstein distance (See Section 2.2) and L 0 (x; θ, φ), L 1 (x; θ, φ) can be rewritten using the reparameterization trick as:

L k (x; θ, φ) = E ∼N (0,I) L i=1 log p θ (x i | h i-1 , z) -D KL q φ (z | h L ) p k (z) , k = 0, 1 with z = µ φ (h L ) + diag (σ φ (h L )) 1/2 .

To classify data, we can train a classifier C over the latent space either independently or simultaneously with the maximal divergence auto-encoder. If we train the classifier simultaneously, the final optimization problem is as follows:

max θ,φ,ψ,µ 0 ,Σ0,µ 1 ,Σ1 {f (θ, φ, ψ, µ 0 , Σ 0 , µ 1 , Σ 1 )} where f (θ, φ, ψ, µ 0 , Σ 0 , µ 1 , Σ 1 ) = E x:y=0 L 0 (x; θ, φ) + E x:y=1 L 1 (x; θ, φ) + αD p 0 (z) p 1 (z) + β (E x:y=0 [log (1 -C ψ (x))] + E x:y=1 [log C ψ (x)]

) where C ψ (x) stands for the probability to classify x as a vulnerable binary code (y = 1), and α, β > 0 are two non-negative trade-off parameters.

It is worth noting that to model the conditional distributions p θ (x i | h i-1 , z), we only take into account the opcode of the machine instruction x i . Since this opcode lies in a fixed vocabulary of the opcodes, we can use the softmax distribution to define the corresponding distribution p θ (x i | h i-1 , z). By this means, the reconstruction phase aims to reconstruct the opcodes of the machine instructions in a given binary rather than the whole machine instructions.

4. EXPERIMENTS:



4.1. EXPERIMENTAL DATASETS:

One of the most significant contributions of our work is to create a labeled binary dataset for binary code vulnerability detection. We first extracted the functions from the NDSS18 source code dataset. We then preprocessed and filtered out any identical functions to obtain 13, 000 functions, of which 9, 000 could be fixed to compile to binaries using our automatic tool. In addition, we developed a tool based on Joern3 to parse the semantic and syntactical relationships in a given piece of source code. In particular, our tool first used the compiler gcc/g++ (MinGW) to compile a given piece of source code, then captured the error messages, parsed these error messages, relied on Joern to be aware of the semantic and syntactical relationships of the error messages with respect to the source code, and finally fixed the corresponding error message. This process was repeated until the given source is error-free and ready to compile to a binary. By compiling the compilable function source code under various platforms and architectures, we obtained 32, 281 binary functions including 17, 977 binaries for Windows and 14, 304 binaries for Linux. The statistics of our binary dataset is given in Table 1 . Additionally, in order to obtain this binary dataset our tool fixed tens of thousands of errors of which many are strongly associated with specific source code. We compared our proposed methods MDSAE-R (for learning maximally divergent representations in conjunction with an independent linear classifier to classify vulnerable and non-vulnerable functions) and MDSAE-C (for learning maximally divergent representations incorporating a linear classifier) with the following baselines:

4.2. BASELINES:

• RNN-R: A Recurrent Neural Network (RNN) for learning representations and linear classifier independently trained on the resulting representations for classifying vulnerable and non-vulnerable functions. In addition, to learn representations in an unsupervised manner, we applied the method of language modeling whereby we trained the model to predict the opcode of the next machine instruction given the previous machine instructions. • RNN-C: A RNN with a linear classifier built on the top of the last hidden unit.

• Para2Vec: The paragraph-to-vector distributional similarity model proposed in (Le & Mikolov, 2014) . This work proposed to embed paragraphs including many words in a fixed vocabulary into a vector space. To apply this work in our context, we view a binary as a sequence of opcodes residing in the fixed vocabulary of the opcodes. • SeqVAE-C: Sequential VAE as in Section 3.2, but we set two priors to N (0, I) and kept fixed during training as in the original VAE. A linear classifier was built up on the top of the latent codes and trained simultaneously. With this setting, we aim to show that learning the priors produces more separable representations, hence boosts the performance. • VulDeePecker: proposed in (Li et al., 2018) for source code vulnerability detection.

This model employed a Bidirectional RNN (BRNN) to take sequential inputs and then concatenated hidden units to input to a feedforward neural net classifier. This method can inherently be applied to binaries wherein sequences of machine instructions are inputted to the BRNN.

In addition, we also inspected two variants of divergence (i.e., KL divergence and L2 WS distance (See Section 2.2)) for formulating the divergence D p 0 (z) p 1 (z) in the optimization problem in Eq. ( 3). Consequently, we have four variants of our proposed method, namely MDSAE-RKL, MDSAE-RWS, MDSAE-CKL, and MDSAE-CWS.

4.3. PARAMETER SETTING:

We split the data into 80% for training, 10% for validation, and the remaining 10% for testing.

We employed a dynamic RNN to tackle the variation in the number of machine instructions of the functions. For the RNN baselines and our models, the size of hidden unit was set to 256. For our model, the size of the latent space was set to 4,096, the trade-off parameters α, β were set to 2×10 -2 and 10 -4 respectively. We used the Adam optimizer (Kingma & Ba, 2014) with an initial learning rate equal to 0.0001. The minibatch size was set to 64 and the number of epochs was set to 100. We implemented our proposed method in Python using Tensorflow (Abadi et al., 2016) , an open-source software library for Machine Intelligence developed by the Google Brain Team. The source code, as well as the dataset, is available in our GitHub repository 4 . We ran our experiments on a computer with an Intel Xeon Processor E5-1660 which had 8 cores at 3.0 GHz and 128 GB of RAM.

4.4. EXPERIMENTAL RESULTS:



4.4.1. EXPERIMENTAL RESULTS ON THE NDSS18 BINARY DATASET:

We conducted the experiments on the subset of Windows binaries, the subset of Linux binaries, and the whole set of binaries to compare our methods with the baselines. The experimental results are shown in Table 2 . It can be seen that our proposed method outperforms the baselines in all performance measures of interest. Specifically, in the field of computer security, the recall is a very important measure of completeness since a higher recall value leads to fewer vulnerable functions being incorrectly classified as non-vulnerable, which can otherwise present an issue for code auditors when there can be a large imbalance in the number of non-vulnerable and vulnerable functions in real-world use. In addition, the fact that the resulting data representations of MDSAE-RKL and MDSAE-RWS work well with a linear classifier confirms our intuition and motivation of that the encouragement of data separation effectively supports the classifiers. Training In this experiment, we study i) the L2 WS distance between the two priors, ii) the Euclidean distance of two means of priors (i.e., µ 0 -µ 1 ), iii) the KL divergence of q φ (z | h L , y = 0) and p 0 (z) (i.e., D KL q φ (z | h L , y = 0) p 0 (z) ), iv) the KL divergence of q φ (z | h L , y = 1) and p 1 (z) (i.e., D KL q φ (z | h L , y = 1) p 1 (z) ), v) the Maximum Mean Discrepancy (MMD) distance (Gretton et al., 2012) of q φ (z | h L , y = 0) and q φ (z | h L , y = 1), and vi) the reconstruction loss across epochs of MDSAE-RWS-the variant of our proposed method for learning separable representations. As shown in Figure 3 , during the training process, two distributions p (z | y = 0) and p (z | y = 1) become consistently and gradually more distant with the increase in their MMD distance (Figure 3 , second row, middle), hence implying the gradually increasing separation of the corresponding latent codes. In addition, as we expect, the two priors become consistently and gradually more distant (Figure 3 , first row, left-hand side and Figure 3 , first row, middle) and the latent codes of vulnerable (y = 1) and non-vulnerable (y = 0) classes become more compressed into its priors respectively (Figure 3 , first row, right-hand side and Figure 3 , second row, left-hand side). Furthermore, the reconstruction error consistently decreases which implies that the latent codes maintain crucial information of the original binaries (Figure 3 , second row, right-hand side).

Figure 3 : The L2 WS distance between two priors (first row, left-hand side), ii) the Euclidean distance of two means of priors (i.e., µ 0 -µ 1 ) (first row, middle), the KL divergence between q φ (z | h L , y = 0) and p 0 (z) (i.e., D KL q φ (z | h L , y = 0) p 0 (z) ) (first row, right-hand side), the KL divergence of q φ (z | h L , y = 1) and p 1 (z) (i.e., D KL q φ (z | h L , y = 1) p 1 (z) ) (second row, left-hand side), the MMD distance of q φ (z | h L , y = 0) and q φ (z | h L , y = 1) (second row, middle), and the reconstruction loss (second row, right-hand side) across epochs.

Visualization of Latent Codes of Two Classes in The Latent Space In this experiment, we set the dimension of the latent space to 2 to visualize the latent codes of the two classes before and after training. As shown in Figure 4 , the latent codes of the two classes are intermingled before training, whereas, they become more separable and distinct after training. This shows that our proposed methods discover data representations that support the classification task.

Figure 4 : The 2D latent codes in the latent space before (left) and after (right) training. The green points are the means of two distributions q φ (z | h L , y = 0) and q φ (z | h L , y = 1).

5. CONCLUSION:

The detection of vulnerabilities in binary code is an important problem in the software industry and in the field of computer security. In this paper, we leverage recent advances in deep learning representation to propose the Maximal Divergence Sequential Auto-Encoder for binary vulnerability detection. Specifically, latent codes representing vulnerable and non-vulnerable binaries are encouraged to be maximally different, while still being able to maintain crucial information from the original binaries. To address the issue of limited labelled public binary datasets for this problem and to facilitate research in the application of machine learning and deep learning to the domain of binary vulnerability detection, we have created a labelled binary software dataset. Furthermore, our developed tool and approach can be reused to create other high-quality binary datasets. We conducted extensive experiments to compare our proposed methods with the baselines. The experimental results show that our proposed methods outperform the baselines in all performance measures of interest.

A APPENDIX: THE PROCESS TO OBTAIN BINARIES FROM SOURCE CODE

The process of compiling the VulDeePecker dataset into binaries is divided into three main stages: collecting functions' source code, detecting and fixing source code, and compiling source code to binary functions.

A.1 COLLECTING FUNCTIONS' SOURCE CODE:

The source code is collected from VulDeePecker GitHub5 . This source code involve two types of vulnerability in C/C++ programs: buffer error vulnerability CWE-119 (11,427 files) and resource management error vulnerability CWE-399 (2,088 files). Figure 8 provides an example of a source code file together with its highlighted buffer error vulnerability. We then use Joern's parser6 to identify the start and end points of each function in order to recognize the function scope. After this step, 19,009 non-vulnerable and 12,946 vulnerable functions are detected and obtained. However, there are a considerable number of functions which are identical to each other. They are either some common functions that are widely used or some unchanged functions in different versions of a particular source code file. To address this issue, these identical functions are removed. Eventually, the numbers of distinctive non-vulnerable and vulnerable functions are 6,412 and 6,592 functions respectively (See Figure 5 ). source code is fixed successfully and ready to compile when the compiler cannot issue any error messages in the process of detecting it.

The process of detecting errors and fixing source code also has its own challenges. As mentioned before, we collect the code snippets of the functions which are always incomplete. This leads to the missing of the declarations for some objects due to the lack of certain libraries to which those objects belong.

Figure 7 refers to a typical example of an uncompilable function. For that code, when our automatic tool starts detecting the error, the gcc/g++ (MinGW) compiler informs the following error message which needs to be fixed at line 23: 'CWE761 Free Pointer Not at Start of Buffer wchar t environment 34 unionType' has no member named 'unionFirst'. The error information is then sent to Joern's parser in order to analyze and find the appropriate solution to fix this error. Unfortunately, the parser is not confident enough to declare the 'unionFirst' variable as a member of the 'CWE761 Free Pointer Not at Start of Buffer wchar t environment 34 unionType' class. The reason arising from the fact that the 'myUnion.unionFirst' was assigned to the 'data' variable, but the parser cannot give any information about the data type of this variable. In this situation, the error description is logged into a log file, the execution of the function including this error is skipped, and our tool proceeds to the next function. After detecting and fixing errors process, we synthesize and do some statistics from the log file to know which errors account for the most popular quantities to upgrade promptly our tool. We also consider the complexity of functions and the priority order of errors to ensure errors fixed from easiest to hardest. After each upgrade, our automatic tool is very likely to detect and fix more complex errors to become more completed and stable.

The result we obtain from this stage is 8,991 fixed source code. It is noticeable that while an original function's source code has 40 errors on average, our automatic tool is able to detect and fix up to 22 and 28 general errors for each C and C++ source code respectively.

A.3 COMPILING SOURCE CODE TO BINARY CODE:

At the last stage, we compile 8,991 fixed functions which contain 4,501 non-vulnerable and 4,490 vulnerable functions into binaries under two platforms (Windows and Ubuntu) and architectures (x86/x64). The process of compilation raises a certain number of small errors due to the behavior inconsistency of gcc/g++ between two platforms which leads to the fact that some fixed functions cannot be compiled. The total number of binary functions are 32,281 wherein 15,954 functions are non-vulnerable binaries and 16,327 functions are vulnerable binaries. As the consequence, we utilize the Capstone software to disassemble these binaries into assembly code. Figure 9 shows the assembly code together with its highlighted buffer error vulnerability of the source code file 

:

https://cve.mitre.org/

www.capstone-engine.org

http://mlsec.org/joern/

https://github.com/dascimal-org/MDSeqVAE

https://github.com/CGCL-codes/VulDeePecker

http://mlsec.org/joern/

https://clang.llvm.org/docs/ClangFormat.html

Figures and Tables:

Figure 1: Machine instruction embedding.

Figure 5: Detailed steps of the process of compiling VulDeePecker dataset into binaries.

Figure 7: Example of an uncompilable function.

Figure 8: Example of a source code file from VulDeePecker dataset together with its highlighted buffer error vulnerability.

Figure 9: The vulnerability highlighted assembly code of the corresponding function's source code in Figure 8. 15

The statistics of our binary funtions dataset.

Table content: <table><tr><td/><td colspan="3">#Non-vulnerable #Vulnerable #Binaries</td></tr><tr><td>Windows</td><td>8, 999</td><td>8, 978</td><td>17, 977</td></tr><tr><td>Linux</td><td>6, 955</td><td>7, 349</td><td>14, 304</td></tr><tr><td>Whole</td><td>15, 954</td><td>16, 327</td><td>32, 281</td></tr></table>

The experimental results in percent (%) of the proposed methods compared with the baselines on the NDSS18 binary dataset. Acc, Rec, and Pre are shorthand for the performance measures accuracy, recall, and precision, respectively. 4.4.2 INSPECTIONS OF MODEL BEHAVIORS Distances between Two Priors, Distributions of Vulnerable, Non-vulnerable Classes During

Table content: <table><tr><td>Datasets</td><td/><td/><td>Windows</td><td/><td/><td/><td/><td>Linux</td><td/><td/><td/><td/><td>Whole</td><td/><td/></tr><tr><td>Methods</td><td>Acc</td><td>Rec</td><td>Pre</td><td>F1</td><td>AUC</td><td>Acc</td><td>Rec</td><td>Pre</td><td>F1</td><td>AUC</td><td>Acc</td><td>Rec</td><td>Pre</td><td>F1</td><td>AUC</td></tr><tr><td>RNN-R</td><td>54.1</td><td>92.6</td><td>52.6</td><td>67.0</td><td>53.8</td><td>55.3</td><td>93.5</td><td>53.3</td><td>67.9</td><td>54.9</td><td>56.3</td><td>93.9</td><td>53.9</td><td>68.5</td><td>55.8</td></tr><tr><td>Para2Vec</td><td>55.5</td><td>93.5</td><td>53.4</td><td>68.0</td><td>55.0</td><td>55.8</td><td>92.1</td><td>53.6</td><td>67.8</td><td>55.5</td><td>54.9</td><td>94.3</td><td>53.1</td><td>67.7</td><td>54.4</td></tr><tr><td>MD-RKL</td><td>80.8</td><td>86.9</td><td>77.6</td><td>82.0</td><td>80.7</td><td>82.7</td><td>81.3</td><td>83.9</td><td>82.6</td><td>82.7</td><td>75.3</td><td>87.8</td><td>70.5</td><td>78.2</td><td>75.1</td></tr><tr><td>MD-RWS</td><td>80.6</td><td>91.3</td><td>75.5</td><td>82.6</td><td>80.6</td><td>84.7</td><td>90.7</td><td>81.2</td><td>85.7</td><td>84.6</td><td>83.7</td><td>94.3</td><td>78.0</td><td>85.4</td><td>83.5</td></tr><tr><td>RNN-C</td><td>81.5</td><td>94.6</td><td>75.1</td><td>83.7</td><td>81.4</td><td>84.4</td><td>96.9</td><td>77.7</td><td>86.3</td><td>84.2</td><td>83.4</td><td>94.1</td><td>77.8</td><td>85.2</td><td>83.3</td></tr><tr><td>VulDeePeck</td><td>82.5</td><td>94.4</td><td>76.5</td><td>84.5</td><td>82.4</td><td>85.5</td><td>94.2</td><td>80.5</td><td>86.8</td><td>85.4</td><td>83.5</td><td>91.0</td><td>79.5</td><td>84.8</td><td>83.4</td></tr><tr><td>SeqVAE-C</td><td>80.8</td><td>91.4</td><td>75.7</td><td>82.8</td><td>80.7</td><td>83.0</td><td>93.7</td><td>77.5</td><td>84.8</td><td>82.9</td><td>78.5</td><td>89.4</td><td>73.6</td><td>80.7</td><td>78.4</td></tr><tr><td>MD-CKL</td><td>83.2</td><td>97.7</td><td>75.8</td><td>85.4</td><td>83.0</td><td>85.9</td><td>97.2</td><td>79.5</td><td>87.4</td><td>85.7</td><td>82.3</td><td>98.0</td><td>74.8</td><td>84.8</td><td>82.1</td></tr><tr><td>MD-CWS</td><td>84.5</td><td>97.2</td><td>77.7</td><td>86.4</td><td>84.4</td><td>86.9</td><td>97.8</td><td>80.6</td><td>88.3</td><td>86.8</td><td>85.3</td><td>98.1</td><td>78.4</td><td>87.1</td><td>85.2</td></tr></table>

Peer review (iclr2019):
 This paper sets out to classify source code snippets are “vulnerable” or “not vulnerable” using sequential auto-encoders with two latent distributions (corresponding to the output classes), regularized to maximize divergence between theses two distributions (named Maximal Divergence Sequential Auto-Encoder). The authors created a compiled subset of the NDSS18 vulnerable vs. non-vulnerable software dataset (which is listed as one of their primary contributions). The dataset construction required non-trivial effort since example code snippets are often incomplete and the authors needed to “fix” these code examples in order to compile them. The fixed code examples are then compiled against both Windows and Linux and both the x86 and x86-64 architectures. The inputs to all predictive models are the opcode sequence of the compiled programs. This paper compares against one previously published vulnerability detection method (VulDeePecker) which is a bidirectional RNN followed by a linear classifier. They also compare with a cascade of models with increasingly complex components: * RNN-R: A recurrent neural network trained in an unsupervised fashion (language modeling over opcode sequences), whose representations are then fed into an independent linear model. * RNN-C: End-to-end training of a recurrent model over opcodes, followed by a single dense layer. * Para2Vec: Encoding of the opcode sequence using the paragraph-to-vector architecture — I’m curious what they used as the paragraph boundaries in the compiled programs and whether the subsequent classifier was the same as RNN-C. * SeqVAE-C: Sequential variational auto encoder trained end-to-end with a final classification layer. * MDSAE-RKL: Maximal divergence sequential auto-encoder with KL divergence between the two class’s latent distributions, final classifier trained independently. * MDSAE-RWS: Maximal divergence sequential auto-encoder with L2/Wasserstein divergence between the two class’s latent distributions, final classifier trained independently. * MDSAE-CKL: Maximal divergence sequential auto-encoder with KL divergence between the two class’s latent distributions, final classifier included as the final layer of the whole model. * MDSAE-CWS: Maximal divergence sequential auto-encoder with L2/Wasserstein divergence between the two class’s latent distributions, final classifier included as the final layer of the whole model. The two MDSAE models using Wasserstein divergence vastly outperform the two equivalent models using KL divergence. Another generalization that can be drawn from the evaluation is that models which are trained with supervision end-to-end outperform those which train representation and classifier separately. Overall, I think this is an interesting and cool paper but I’m not sure I actually buy into the basic premise that it makes sense to model vulnerable vs. non-vulnerable code as two different latent spaces. Aren’t the changes to make a vulnerable function safe again rather small and/or subtle? I think that beyond visualizing the convergence of properties of the latent spaces it would greatly improve this paper to inspect which aspects of the source contribute to both the latent representation and final classification as vulnerable vs. non-vulnerable. Also, I wish the process of “fix”ing the input code was better described, since the failure of this procedure excluded 4k/13k of the programs/functions in their initial dataset and had the potential to introduces learnable biases in the source code. At the very least, the authors should list how many vulnerable vs. non-vulnerable samples required fixing vs. could be compiled in their original form. Lastly, the definition of "vulnerable" may be obvious to someone more familiar with the domain but seemed to me somewhat vague and never directly addressed. Typo: p3, need space in "obtain32, 281"