This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Access

Date of publication xxxx
Digital Object Identifier xxxx

Identifying Compiler and Optimization
Level in Binary Code from Multiple
Architectures
DAVIDE PIZZOLOTTO1 and KATSURO INOUE2
1
2

Osaka University, Osaka, Japan (e-mail: davidepi@ist.osaka-u.ac.jp)
Osaka University, Osaka, Japan (e-mail: inoue@ist.osaka-u.ac.jp)

Corresponding author: Davide Pizzolotto (e-mail: davidepi@ist.osaka-u.ac.jp).
This work was supported by JSPS KAKENHI Grant Number 18H04094.

ABSTRACT While compiling a native application, different compiler flags or optimization levels canbe configured. This choice depends on the different requirements. For example, if the application binary
is intended for final release, the flags and optimization settings should be set for execution speed andefficiency. Alternatively, if the application is to be used for debugging purposes, debug flags should be
configured accordingly, usually involving minor or no code optimization. However, this information cannot
be easily extracted from a compiled binary. Nonetheless, ensuring the same compiler and compilation flags
is particularly important when comparing different binary files, to avoid inaccurate or unreliable analyses.
Unfortunately, to understand which flags and optimizations have been used, a deep knowledge of the target
architecture and the compiler used is required. In this study, we present two deep learning models used todetect both compiler and optimization level in a compiled binary. The optimization levels we study are O0,
O1, O2, O3, and Os in the x86_64, AArch64, RISC-V, SPARC, PowerPC, MIPS, and ARM architectures.
In addition, for the x86_64 and AArch64 architectures, we also determine whether the compiler is GCC
or Clang. We created a dataset of more than 76000 binaries and used it for training. Our experiments
showed over 99.99% accuracy in detecting the compiler flags and between 92% to 98%, depending on
the architecture, in detecting the optimization level. Furthermore, we analyzed the change in accuracy when
the amount of data was extremely limited. Our study shows that it is possible to accurately detect bothcompiler flag settings and optimization levels with function-level granularity.
INDEX TERMS Compilers, Deep Learning, Static Code Analysis, Reverse Engineering

I. INTRODUCTION

During the software development life-cycle of a natively
compiled application, the process of converting source code
to binary code performed by a compiler occurs frequently.
During this transformation, the compiler is passed several
flags via the “build” commands and settings contained withina Makefile or equivalent. This informs the compiler of thedeveloper’s intent to retain or omit some information or tomodify the original code in an optimized version.
These flags can be used to optimize faster executions,
smaller sizes, and lower energy consumption [1]. However,the flags are not explicitly recorded in the binary itself, as
they are completely unnecessary for the machine to executethe binary code.
Moreover, the compiler itself is not easily identifiable.

There is no standard way to record this information, andalthough some compilers write a comment in the binary
itself, it is easily skipped or duplicated and not guaranteed
to be parsable. For example, if a file compiled with theClang compiler is linked with a library compiled with GNU
compiler collection (GCC), this comment will contain bothsignatures.
However, this information is extremely valuable for various applications, such as categorizing an older build, finding vulnerabilities [2], finding similarities in binaries [3],or providing accurate bug reports in case the compilation
environment cannot be controlled [4]. A simple example ofthe latter case could be a library that has incompatibilities
only with a specific compiler, in a product published by adifferent vendor than the library developer. Pallister et al.
1

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Accesshave shown that different optimization options can have asignificant impact on final energy consumption [5]. The useof precompiled libraries could be problematic in embedded
applications where energy consumption is a concern.
Knowing the compilation flags could even be helpful when
performing binary analysis. In their work, Pewny et al. reported that applying their analysis to different compilation
flags significantly affects accuracy [6]. In this case, ourwork will help to provide confidence in the analysis results,
because the differences between the optimization levels of
O2 and O3 are much less pronounced than those of O0 andO3.
Although there are several papers on detecting the compiler [4] and toolchain [7] used, these methods do not rely
on automatic learning approaches. Therefore, a significant
effort is required to detect the above information in different
architectures. The new architecture must be studied and understood to check if and how the information can be retrieved.
In contrast, with a machine learning-based approach, it is sufficient to provide new data and re-run the training to detect a
new compiler or flag. With the automated dataset generation
provided in our work, the time required to generate this data
is a couple of hours for each optimization level that we want
to classify.
In this study, we present our approach for recognizing
both compiler and optimization levels using a long-short term
memory network (LSTM) [8] and a convolutional neural network (CNN) [9] within different architectures. We analyzed
common optimization levels on Linux binaries, compiled
with either GCC or Clang in two different architectures, andwith GCC only in seven different architectures. We wantto identify optimization levels ranging from non-optimized
code (O0), code that is optimized for speed with different
levels of aggressiveness (O1, O2, O3) or code optimized
for small binary size (Os). Although we are not the first to
tackle this problem [10], the novelty of our research can besummarized as follows:
The creation of a huge dataset with automated replication scripts consisting of 76630 compiled files. Thesefiles come from seven different CPU architectures andtwo different compilers, with a combined total of 123
GB of stripped binary data.
• The implementation and tuning of a neural network
structure that outperforms existing work in flag detection.
• An analysis that examines the minimum possible number of raw bytes that are required to obtain accurate
predictions.

•This study is an extended version of our previous
work [11]. The main differences between our previous study
and the current study are as follows.
The number of optimization levels test was increased
from {O0, O2} to {O0, O1, O2, O3, Os}.
• The number of architectures tested have increased from
{x86_64} to {x86_64, AArch64, RISC-V, SPARC,•

PowerPC, MIPS, ARM32}. These additional architectures require a completely different cross-compilation
approach for generating the dataset, as described in
Section IV-A. We also made considerable efforts toautomate the generation of the dataset. In our previous
work, compilation was a manual task, while now weprovide several scripts to automate the process. The
main advantage of this automated approach is the ability
to generate a different dataset with different compilation
flags without the user having to do anything. Although
this does not work for additional architectures, it still
provides an effortless way to generate the dataset withadditional flags.
• The evaluation has been completely reworked, considering the expanded category set. Moreover, in thisextended version, we tested the feasibility of our study
in a scenario with function-level granularity. We also
examined the distribution of flags in the Ubuntu and
MacOS operating systems.
The rest of our paper proceeds as follows. Section II
presents the motivation for our work. Section III discusses
related work in the field of binary file analysis with machine learning. Section IV presents the problem and ourapproach and Section V presents the empirical evaluation.
Section VI compares our choices with another similar workin the field and discusses them in terms of the results obtained. Section VII describes the limitations of our study andSection VIII concludes the paper. In addition, Section IX
provides instructions to download our dataset and sourcecode.
II. MOTIVATION

After briefly introducing the motivation behind our workin Section I, we provide an example of the reasons behind
our work in this section. Although there are several reverse
engineering tools such as IDA1 or Ghidra2 , these usually
do not detect the flags used during compilation or at linktime. This is because a user is interested in extracting andcollecting information from single binary files as part ofreverse engineering. In this case, the compilation flags areof little to no interest, as the main goal is to understand the
behavior of a particular program.
In contrast, the decompilation scenario is slightly moreinteresting. According to Katz et al. [12], the presence of optimizations reduces the success rate in correctly decompilingan executable. Moreover, their results are based on recurrent
neural networks (RNNs), so we expect traditional approacheswith hand-crafted rules to be more susceptible to underlying
optimizations. With a higher level of optimization, we expect
the decompiler to output a higher amount of goto statements. If this is the case, knowing the optimization flags may
give an indication of the expected accuracy of a decompiler,
1 https://hex-rays.com/
2 https://ghidra-sre.org/

2

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Accessknowing that a heavily optimized code may decompile into
source code that is more difficult to analyze
Nevertheless, the focus of our study was on code analysis.
When comparing multiple binaries, knowledge of the compilation flags becomes much more useful, almost mandatory.
As reported in Section I, Pewny et al. in their analysis ofcross-architectural binary code devoted an entire researchquestion investigating the presence of false/true positives atdifferent optimization levels [6]. Ultimately, they concluded
that comparing non-optimized to optimized code results insignificantly lower accuracy. In our internal study, we came toa similar conclusion. Comparing control flow graphs (CFGs)
of functions, even within the same architecture but withdifferent compilers or optimization flags, is dominated byfalse positives. The impact is so great that the comparison isusually impractical when dealing with different compilers or
O0 versus O2, where it is perfectly fine when comparing O1
with O2 or O3. Thus, performing analyses or comparisons
between binaries may lead to results that are either goodor completely unusable, depending only on the compilation
flags used. These two contrasting results have led us to look
for a way to detect in advance the presence of optimizations.
This is a way to determine whether a comparison or analysis
between binaries is feasible, and if the results will be reliable.
In our previous study, we focused only on the presence/absence of optimizations, the most influential factor
in determining a poor comparison between two binaries. In
this extended version, we also tried to detect the degree of
optimization. In fact, we can expect a lesser loss of accuracy
when comparing, for example, an O1 optimized code to an
O3 optimized code.
III. PREVIOUS WORKS

Binary file analysis is widely used in the field of security.
Recently, machine learning techniques have been used toaid malware detection. Pascanu et al. used recurrent neuralnetworks (RNNs) [13] to extract malicious features froma binary file in an unsupervised manner, which was extended with convolutional neural networks by Athiwaratkun
et al. [14].
Related works dealing with compiler flags, instead, focuses on the effects of these flags rather than their detection.
Work performed by Triantafyllis et al. focused on exploring
optimal compiler flags [15] as did the work of Hoste et al.
[1]. In recent years, several machine learning-based techniques have been developed [16] [17]. Older techniques focus
on the use of machine learning to reduce the number of iterative compilations required to obtain a good set of flags [18]and to help approximate NP-hard problem efficiently, like
phase ordering [19]. Instead, more recent techniques use deeplearning to detect function boundaries, a work by Bao et
al. [20], which was then extended by Shin et al. [21]. Chua et
al. instead detected function types using RNNs [22] whereas
He et al. attempted to recover the debug symbols from astripped binary [23].
To the best of our knowledge, the only work attempting todetect flags in an existing binary, rather than optimizing them,is that of Chen et al. [10]. The main differences between theirstudy and our work are as follows:• We investigate the detection of not only flags but also
compilers.
• We investigate the detection in seven different architectures instead of only one.
• Our analysis aims not only to maximize the accuracy but
also to minimize the required input.
• Our dataset is more than 100 times larger [24], disproving some claims of previous research.
IV. APPROACH

The problem we are trying to solve is to identify the optimization level and possibly the original compiler used to compile
from source code to binary code, when only a portion of thebinary code is available. Specifically, given a sequence ofbytes v of arbitrary length coming from a binary, we want
to train a classifying function Mf lags capable of predicting
the compilation flags and a classifying function Mcompiler
predicting the compiler used.
With Mf lags , we try to classify the optimization level
used in the input binary. In our study, we target the commonly used optimization levels {O0, O1, O2, O3, Os} 3 . We
trained different Mf lags for each of the seven architectures
we studied, namely x86_64, AArch64, ARM32, MIPS,
PowerPC, RISC-V, and SPARC, and expect a user to selectthe prediction model according to the input architecture. The
architecture of a binary is easily recognizable by tools such as
file, so this fact is not a limitation and simplifies training.
Similarly, with Mcompiler we classify the compiler between gcc and clang, which means that the compiler
analysis is a binary classification. This is done only forthe natively generated architectures, such as x86_64 andAArch64, for reasons explained in Section IV-A2. Thus, wehave trained two different Mcompiler .
Our goal is not only to maximize accuracy, but also tokeep the sequence of bytes v as small as possible; as such,
we dedicate Section Section IV-B to explain how the binary
code is transformed into v (or several vs), the input expectedby our learning network. To compare the performance ofdifferent models, we trained all the aforementioned configurations using a feed-forward Convolutional Neural Network
MCN N and a Long-Short Term Memory Network MLST M ,
N
LST M
CN N
producing in total 7MCNf lags , 7 Mf lags , 2 Mcompiler and 2
LST M
Mcompiler . These networks are trained in several different
datasets, explained in detail in Section IV-A, and their prediction results are compared.
More details about the network models can be found in
Section IV-D.
A. DATASET

To train our networks, we must first collect the data. Our
networks perform supervised learning, so it is necessary to
3 Note that some applications might use additional “hand-picked” flags.
This limitation is discussed in Section VII

3

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Access

Source A

gcc -O0

A.o
Linker

Source B

gcc -O2

a.out

libB.aFIGURE 1. Linking an executable may include binary data compiled withdifferent flags

partition the binary code by optimization level and compiler
used.
Although this task may seem trivial, as there are plentyof open-source software programs that can be compiled withthe desired flags, several precautions are required during thelinking phase. This is because although we can choose boththe optimization level and compilers, we have no guarantees
about the environment that performs the compilation. There
are several libraries available to be statically linked, and weknow nothing about the compilation settings used for theselibraries. This problem is highlighted in Figure 1.
When a library is statically linked, its binary code is copied
inside the final executable during the linking phase. As such,in most build systems, pre-existing libraries could be linked
while generating the dataset. In Figure 1, this is shown in the
Source B path, where, a source originally compiled with an
optimization level of O2 is linked within the same binary with
an optimization level of O0. This implies that these libraries
could irreversibly contaminate our build, because we lackinformation about their creation.
Possible options for solving this problem would be:1) Use a dataset composed of object files prior to linking.
2) Edit the build script for the generated binary to exclude
static linking.
3) Cross-compile from a different architecture.
4) Create a system with only shared libraries. Then use
this system to build the dataset.
In our study, we chose options 3 and 4 for the following
reasons: Option number 1 would not provide a realistic casestudy, as several optimizations can also be performed at linktime [25]. Option 2, could be “too risky” as it involves
manually checking various compilation settings written indifferent languages and styles in order to ensure dynamic
linking and has an high risk of error. For example, in our
experiments, we found that some build scripts use hardcoded parameters, others use environment variables, and others recursively integrate files. Checking, understanding, andmodifying all these build scripts correctly is entirely possible
but carries a high risk of error. Option 3 solves the linking
problem. The compiler cannot statically link binary code fordifferent architectures. This ensures that every link actionuses the object files generated by us. However, the drawback
is that the build architecture and the target architecture must
be different. If this is not the case, the build tools would
probably prefer linking libraries in default paths, increasing
the likelihood of linking external files. Option 4 requires
the build architecture and the target architecture to match.
We want to use a system without static libraries as build

environment, which means that we need to run this system
natively.
Following the dualism of option numbers 3 and 4 regarding
native compilation and cross compilation, we can take twodifferent approaches depending on the target architecture.
The final dataset is publicly available on Zenodo at thefollowing link [24].
1) Dataset: Native compilation

The dataset we use for the native compilation contains all
software listed in the Linux from Scratch book4 , version
9.1-systemd, published on March 1, 2020. In addition, we
added to every dataset the LLVM suite version 10.0.0, whichis required for building Clang5 . To solve the static linking
problem within a native build system we performed thefollowing steps:
1) We built a toolchain with no particular compiler andoptimization level from the host machine. We ensured
that only shared libraries were built in this toolchain.
2) We created a chrooted environment containing only thetoolchain to isolate it from the original build system.
3) We built the actual dataset, with the desired compiler
and optimization level.
After building all software binaries, we then strip and use
each ELF file.
2) Dataset: Cross compilation

The cross-compilation approach we use is based on the presence of readily available toolchains in the Ubuntu Package
repository6 . For the following study, we need the gcc, g++and binutils packages for each architecture we want to target.
For cross-compiled architectures, we did not build a Clangdataset and limit our analysis to GCC only.
Unlike the native compilation presented in Subsection IV-A1, chrooting is not needed because the linker is notcapable of linking different binaries of different architectures.
Moreover, the lack of chrooting means that we can fully
automate the entire building process, which normally ends
with the execution of the chroot command. Pointers to thescript used to perform this automated building can be foundin Section IX.
However, not all software used in the native building supports cross-compiling. Despite heavily editing most scripts,
some software, such as Perl, fails to cross-compile. For thisreason, when cross-compiling, we use a slightly different
dataset and limit our study to the GCC compiler. This newdataset comprises all the software from LFS that supports
cross-compilation as well as some software coming from Beyond Linux from Scratch7 . Due to space limitations, we cannot list every piece of software used in this building process.
For a comprehensive list see the resources/scripts
folder in the repository listed in Section IX.
4 http://www.linuxfromscratch.org/lfs/index.html
5 https://releases.llvm.org/10.0.0/
6 https://packages.ubuntu.com
7 http://www.linuxfromscratch.org/blfs/index.html

4

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE AccessAs with native compilation, each ELF file generated isstripped and used in the dataset.
B. PREPROCESSING

In this section, we explain the additional preprocessing before the input vector is fed into the networks. In the evaluation, we prove that advanced encoding is unnecessary, but wereport it here anyway because previous research has used it.
The goal of this preprocessing is to transform a binary file,usually composed of a sequence of data and instructions, into
one or more input vectors for the automated learning step. We
are naturally interested in correctly classifying the smallest
possible input, with the finest grain being the function grain.
We compare two approaches: one using a stream of bytes
without prior knowledge of the underlying data and onethat requires disassembly and precise function boundaries.
The effectiveness of these two approaches is analyzed in
Section V-C.
The first approach, referred to as “raw bytes” throughout
the paper, has been shown to be effective for learning functions boundaries in previous research [20].
To generate this representation, we use readelf to dump
the .text section of the executable and divide it into fixedsize chunks. We chose a-priori 2048 bytes as the maximum
size of this chunk of bytes v. However, we also evaluatedthe precision of the networks in detecting the compilation
settings for these chunks as their size varied, in order tosimulate a case where we want to detect the optimization ofa single function.
One drawback of this representation is that we do notknow whether the raw data represents instructions or stackdata. In contrast, if we want to classify an entire executable,disassembly is not required, a step that usually requires
several minutes. Even when we want to classify at function
granularity, based on the research of Bao et al. [20], we
can extract function boundaries and perform classification
without disassembling the executable, which is not only slow,
but is also inherently difficult and prone to error in somearchitectures [26].
In the second representation, the one that requires disassembly, radare28 , is used to extract each function from theexecutable. The results are shown in Figure 2.
In the figure, the left column represents the raw bytes
written in the binary and the right column their translations inIntel Assembly syntax. The example is taken from x86_64
disassembled code. We can see many bytes specifying thatthe registers to be used should have a length of 64 bits,
represented by the red bytes in figure, preceding every
instruction involving rax, rsi, rsp registers. This is aproblem, because real functions can be of arbitrary length,
but our networks support fixed-length vectors as input. We
expect these extra bytes to contribute almost nothing to thefinal result and thus decide to remove them and keep only thebyte(s) representing the operations to be performed, without
8 https://rada.re/4889442418
31C0
4885FF
7423
488B4208
48893424
4889E64889442408
488B024889442410
E85A0E0000
4885C0
0F95C0

mov qword [rsp+0x18], rax
xor eax, eax
test rdi, rdi

je 0x25
mov rax, qword [rdx+0x8]

mov qword [rsp], rsi
mov rsi, rsp
mov qword [rsp+0x8], rax
mov rax, qword [rdx]mov qword [rsp+0x10], raxcall 0xE5F

test rax, rax
setne alFIGURE 2. Portion of a disassembled function

parameters. Unlike the previous research, we also remove theoperands in our representation, in order to save more space
and accomodate even more “valuable” instructions inside thelimited length vector [10]. Finally, only the blue bytes in
Figure 2 were encoded in our representation. Extra data are
pre-truncated, because we expect the most useful operations
are at the end of a function and not at the beginning, whichcontains the initialization. Insufficiently long functions are
pre-padded with zeroes, as pre-padding has been proved tobe better for LSTMs [27]. From now on, we will refer to thisrepresentation as “encoded”.
In both representations, we feed our data to the
networks as a time series, where each point in timeis actually a byte of data from the binary file.
For example, the first two instructions of Figure 2
would have this vector in the raw byte approach:[0x48,0x89,0x44,0x24,0x18,0x31,0xC0].
In
the encoded representation instead they would be[0x89,0x31].
C. PADDING

For the encoded representation presented in Section IV-B, theinput vector length is equal to the function length. This makes
it necessary to pad the data, as the function length is always
different. However, the raw data approach, requires a fixedamount of sequential data from the binary. As any data in any
part of the .text binary section can be used as an input, inprinciple, no padding is strictly required.
However, we determined that by always providing unpadded vectors, both CNN and LSTM are unable to deal
with padded data during evaluation. The experimental data in
Section V-D shows that when training with unpadded inputs,
the inference of a vector padded with zeroes by more than
60% of its length results in a 10% accuracy drop. This canbe detrimental in a real case, as it would be necessary to train
different models for different input sizes if we want to infer asmaller amount of data or just a portion of the executable.
To solve this problem, we truncate a random number of
5

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Access

f0 25 14 de af 8c 85 c3
85 bf 5b cf e0 f2 63 0b
92 af 97 0b 06 84 1d 5d
e3 14 bc ac a8 de 21 e7
73 11 27 9a ff 4f d9 73
03 d6 ce de 8b 0d af 46
74 37 35 f2 49 c3 e5 69
8c 47 4a 57 d2 cf 7e 46

00 f0 25 14 de af 8c 85
00 00 00 00 85 bf 5b cf
00 00 00 92 af 97 0b 06
00 00 00 00 00 00 e3 1400 00 00 00 00 73 11 27
00 00 03 d6 ce de 8b 0d00 00 00 74 37 35 f2 49
00 8c 47 4a 57 d2 cf 7e

FIGURE 3. Truncation of input sequences on the left and subsequent paddingon the right. The amount of truncated bytes in this Figure is symbolic[0x89, 0x31, 0x85, 0x74, ...]
[0x89, 0xE8, 0x89, 0x89, ...]...

Embeddings

out=128

LSTM

n=256

Tanh
Dense

n=#classes

bytes in the interval [0, α] where
α = len(v) − 32
The value 32 has been chosen so that the input chunk v to
be still classifiable: if we pad too much, we might get chunks
where the classification is impossible due to lack of enoughinformation. This number is also extremely conservative:
consider that in x86_64, for example, just calling an imported
function requires at least 11 bytes of data, and translates to asingle opcode.
The random amount is defined by an exponential distribution. Our intention is to use a distribution where 99% of
the values fall within the above interval, while clamping theoutliers to 32. In this case, the network would predominately
receive low-padded vectors, while occasionally encounteringa mostly-padded vector. With the exponential cumulative
distribution function as y = 1 − e−λx we fix y to 0.99 and x
to α to obtain:
2 ln 10λ=αWe then use this λ in the exponential distribution generating
the random number of bytes that should be truncated foreach input. After truncating the input, we prepad it by adding
zeroes.
An example of this can be seen in Figure 3, where each line
represents an input sequence before padding on the left blockand after padding on the right block. The red part represents
the amount of input data that will be truncated. The lengthof this part is decided randomly within the interval bounds
previously mentioned. On the right block we can see that thesame amount is replaced by prepending zeroes.
Evaluation of this padding is provided in Section V-D.
D. NETWORKS

For our analysis, we used two different networks: a LSTM [8]and a feed-forward CNN [9]. These networks have beenchosen due to their successful applications in natural language processing or image recognition. In fact, we model
our optimization recognition problem as a pattern recognition
problem: a particular optimization can be recognized by a
network as a pattern of opcodes in the input sequence ofbytes.
The first model is shown in Figure 4. This model depicts
a simple LSTM, given that we encoded our sequence ofSoftmax[0.96, 0.0, 0.04, 0.0]
FIGURE 4. LSTM model structure

bytes as a time series and the ability of LSTMs to perform
well on this type of problem. LSTMs, in fact, have special“memory” cells, that allows them to memorize a particular
input or pattern even in long sequences [8]. Our core idea
is to train this kind of model into memorizing a particularpattern, representing the compiler or the optimization level,
over a long sequence of bytes belonging to the binary.
As we can see from the figure the model is pretty straightforward. It is composed of an embedding layer with 256
as vocabulary size (because we use bytes in range 0x0 to0xFF) and 128 as dimension for the dense embedding. This
layer encodes positive integers into a dense vector of fixed
size, understandable by the LSTM. Then, the LSTM layer
with 256 units is used for the actual learning. This layer
uses a hyperbolic tangent (tanh) as the activation function.
The kernel is initialized by drawing samples from a uniform
1
1
distribution in [−64− 2 , 64− 2 ]. The last part of the network
is a dense layer with 1 output and Sigmoid activation forthe binary case, dense layer with 5 outputs and Softmax
activation for the multiclass case. The optimizer is Adam [28]with learning rate of 10−3 .
The second model, is based on the trend in image recognition and categorization [29] and is based on a convolutional
neural network. The idea is that a series of convolutions
is used to extract highly dimensional information from thesequence of raw bytes passed as input. The Structure isshown in Figure 5.
The first layer is identical to that of the LSTM version,
because its utility is the same. Then, three blocks of convolution, convolution, and pooling, with increasing numberof filters, were used. In the Figure, the label k3n32s1 for a
convolutional layer indicates a kernel size of 3, a number offilters of 32, and a stride of 1. In these blocks, the convolutions are used to extract features from the sequence of bytes,
and the pooling is used to make these features independent of
their position in the sequence.

6

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Access

Softmax

Dense

n=#classes

ReLU

Dense

MaxPool1D

n=1024

Leaky ReLU

Conv1D

Conv1D

MaxPool1D

k5n128s2
k3n128s1

Leaky ReLU

Conv1D

Conv1D

MaxPool1D

k5n64s2
k3n64s1

Leaky ReLU

Conv1D

k5n32s2
k3n32s1

Conv1D[0x89, 0x31, 0x85, 0x74, ...]
[0x89, 0xE8, 0x89, 0x89, ...]...

Embeddings

n=128[0.96,
0.0,
0.04,
0.0]

FIGURE 5. CNN model structure

The leaky ReLU [30] is used in place of the ReLU [31],because the latter suffers from the vanishing gradient problem. Although the ReLU function returns 0 for values lessthan 0, the leaky variant returns an , in our case 0.01, to
keep the neuron alive. Before output, the final fully connected
layer composed of 1024 neurons is used, followed by a
ReLU activation and the canonical dense and sigmoid forbinary classification or dense and softmax for multiclassclassification. Also in this case the optimizer is Adam withlearning rate of 10−3 .
All the models use binary cross-entropy as loss function
for the binary classification and categorical cross-entropy forthe multiclass classification [32].
The presented hyperparameters of both the LSTM and the
CNN were estimated using the Hyperband algorithm [33].
We used powers of two as space search in the interval[32, 1024] for most features, except kernel size and strides.
For the kernel size, the space search was the set {3, 5, 7}.
Instead, for the stride, the space search was {1, 2}.
V. EVALUATION

We performed experiments using an Nvidia Quadro
RTX5000 on the data presented in Section IV-A after the
preprocessing explained in Section IV-B. However, given thehigh amount of binary data, we obtained a number of input
vectors in the order of millions. Thus, we could safely split
the data into disjointed sets, while maintaining a high numberof samples for each set. Thus, we set training, validation, andtesting with split ratios of 50%, 25% and 25% respectively.
No augmentation was performed and no overlapping sequences were collected, with each sample being absolutely
unique between training validation and testing. Duplicate
samples were removed from each set. These are common
especially in the opcode-encoded datasets. Training was performed for 40 epochs using a batch size of 512 samples. An
early stopper was employed, which stopped the learning after
three epochs if the loss function in the validation dataset did
not improve by at least a factor 10−3 . In total, we trained 36
models in approximately 550h.
In this section we want to answer the following researchquestions:• RQaccuracy : Is it better to use a CNN or a LSTM? What
results can be expected from each network?

RQmin : What is the minimum number of bytes required
to have accurate predictions?
• RQencoding : Does extracting data with a disassembler
increase the accuracy of the predictions?
• RQpad : Does padding during training improve the performance of the networks?
• RQoccurrence : What are the most common optimizationlevels in real-case distributions?
RQaccuracy aims to investigate the advantages and disadvantages of using one network over the other for bothcompiler detection and optimization level detection. This
question is then expanded in RQmin to investigate howmuch the input size can be reduced while still maintaining
a sufficiently high accuracy. RQencoding was investigated to
explain our choice of training with raw data. In particular,
this contradicts the claim of previous work conducted by us
and Chen et al. [10]. RQpad , instead, serves the purpose ofjustifying why in Section IV-C, we claim that padding isnecessary while training with raw data. Finally, RQoccurrence
concludes our study by running our models in real-case code
to obtain statistics about the most used flags, proving theassumption of O2 being the most popular flag as not always
correct.
•

A. ACCURACY

To evaluate the accuracy of both the CNN and LSTM we
divided our dataset by architecture. The number of samples
we used for each architecture is listed in Table 1. It should
be noted that in the worst case we trained with at least 106
samples and tested on at least 6 · 105 samples.
DatasetTraining

Testing

Dx86_64
Daarch64
Driscv64
Dsparc64
Dpowerpc
Dmips
Darm

2.4 · 1061.2 · 106
1.1 · 106
7.0 · 105
9.8 · 105
1.1 · 1068.2 · 105
6.2 · 105

2.3 · 1061.3 · 1061.9 · 106
2.0 · 106
1.6 · 1061.2 · 106TABLE 1. Number of training and testing samples for each architecture. Eachsample is composed of 2048 bytes

The number of features for each sample are 2048 se7

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE AccessTime required to train the MIPS modelend of the first epoch. In addition, the CNN can complete
10 epochs in the time the LSTM is able to complete only
3. We measured similar speeds also during inference, witha CNN two to three times faster than the LSTM. This is not
limited to the single architecture we presented in Figure 6. As
we can see in Table 3, it applies to any architecture, with theworst case being x86_64 having an LSTM requiring morethan five times the corresponding CNN training time. Note,however, that these times were collected only once. As such,variations, even significant ones, are expected.

0.95

Accuracy

0.90
0.85
0.80
0.75
Accuracy CNN
Accuracy LSTM

0.70
0

200

400

600
800
Time (minutes)

1000

1200

FIGURE 6. Accuracy obtained in the validation dataset at the end of eachtraining epoch.

quential bytes from the specified architecture, categorizedby optimization level. In addition, x86_64 and AArch64
contains samples compiled with both GCC and Clang. We
trained a CNN and a LSTM model for each dataset and obtained the results shown in Table 2 regarding the optimizationlevel detection. Note that all results were obtained with raw
encoding and padded data unless otherwise stated.
Architecture

CNN Accuracy

LSTM Accuracy

x86_64
AArch64
RISC-VSPARC
PowerPC
MIPS
ARM

0.87810.91810.84270.93640.87020.9596
0.93800.9291
0.96870.92090.96820.92270.98370.9588TABLE 2. Accuracy for each architecture while detecting the optimization level

This table represents the categorical accuracy achieved
while performing supervised evaluation of our trained models in the testing data. Being the accuracy a metric definedfor binary classification, every time we use this term in a
multiclass context, we refer to the following formula:Pk
tpi +tni
accuracy =i tpi +tni +f pi +f nikwhere tp, tn, f p, f n are true positives, true negatives, false
positives, false negatives and k is the number of classes.
We can note how that the LSTM is always better than the
CNN, with the worst accuracy being recorded for x86_64,
RISC-V, and PowerPC in both networks. However, thedownside of using the LSTM is its extensive training time.
We can see this in Figure 6.
The figure shows the times obtained from the MIPS
dataset, one of the datasets with the fastest training times
owing to its size. It takes approximately seven epochs for
the LSTM to reach the same accuracy as the CNN at theArchitecture

CNN Time

LSTM Timex86_64
AArch64
RISC-VSPARC
PowerPC
MIPS
ARM

361 min
298 min
220 min
346 min
398 min
257 min
407 min1845 min1764 min1034 min1397 min1379 min1362 min
629 min

TABLE 3. Training time, in minutes, required for each network andarchitecture.

To further investigate the accuracy, Figure 7 shows all the
confusion matrices for each model trained with the CNN.
The problematic part, as seen from the Figure, is thedistinction between O2 and O3. In fact, O0 and O1 can bedetected with 99% accuracy in any architecture and Os isnever below 96%. O3, however, in the worst case has morewrong classifications than correct ones, as we can see inPowerPC and RISC-V.
This situation is slightly better when an LSTM is used. The
results are shown in Figure 8
In this Figure, we can see how the LSTM achieves highaccuracy in some architectures, namely AArch64, SPARC,
MIPS, and ARM. The architectures problematic for the CNN
remain problematic also for the LSTM, but to a much lesser
extent. In fact, no optimization level reports more wrong
classification than correct ones, and the worst case is a 70%
accuracy for PowerPC O2.
To mitigate this problem, we trained two additional
datasets: Dmerged and Dsplitted : the first containing all optimization flags but with O2 and O3 merged together, thesecond containing only O2 and O3.
Figure 9 reports this split dataset situation. We can notice how the CNN network performs slightly better after
separating O2 and O3 from the rest of the dataset, whereas
the LSTM performs slightly worse compared to the results
without separation.
Concerning the compiler detection, results are reported inTable 4.
The table shows how both networks in both architectures
perform excellently. Even in the x86_64 with CNN case,that performed quite poorly in the optimization detection,
the incorrect classifications were 587 compared to 1225822
correct classifications. Given the high accuracy of the CNN

8

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Access

SPARC
0.993 0.002 0.006 0.003 0.004

O0

O1
O2
O3
Os
Predicted optimization levelO0

0.000 0.996 0.002 0.000 0.002
0.001 0.002 0.950 0.050 0.004
0.004 0.002 0.617 0.273 0.004
0.006 0.001 0.004 0.001 0.977
O0O0
True optimization level
O3
O2
O1

0.996 0.001 0.004 0.001 0.002

0.000 0.001 0.732 0.130 0.005
0.002 0.001 0.008 0.002 0.983
O0

O1
O2
O3
Os
Predicted optimization level

0.989 0.000 0.001 0.002 0.002
0.000 0.998 0.001 0.001 0.001
0.000 0.001 0.869 0.146 0.006
0.000 0.001 0.024 0.966 0.005
0.001 0.001 0.002 0.004 0.993
O0

True optimization level
O3
O2
O1

True optimization level
O3
O2
O1

O1
O2
O3
Os
Predicted optimization level

0.000 0.001 0.892 0.120 0.006

MIPS

Os

Os

True optimization level
O3
O2
O1

O0

PowerPC

0.000 0.994 0.004 0.000 0.003

Os

0.000 0.002 0.014 0.009 0.963

Os

0.000 0.001 0.220 0.779 0.015

O1
O2
O3
Os
Predicted optimization level

0.001 0.987 0.009 0.003 0.001
0.001 0.003 0.950 0.050 0.002
0.001 0.002 0.250 0.708 0.002
0.004 0.001 0.011 0.006 0.968
O0

ARM

O0

0.000 0.001 0.002 0.006 0.985

0.000 0.001 0.825 0.150 0.023

True optimization level
O3
O2
O1

0.000 0.003 0.377 0.636 0.004

0.000 0.984 0.008 0.003 0.009

Os

0.000 0.004 0.754 0.228 0.004

True optimization level
O3
O2
O1

0.000 0.988 0.006 0.004 0.003

O0RISC-V
0.992 0.000 0.004 0.001 0.011

O0AArch64
0.998 0.000 0.002 0.001 0.000

O0

x86_64
0.998 0.000 0.002 0.001 0.001

Os

Os

True optimization level
O3
O2
O1

O0

Normalized confusion matrices (CNN)

O1
O2
O3
Os
Predicted optimization level

O1
O2
O3
Os
Predicted optimization level

0.996 0.001 0.004 0.002 0.001
0.001 0.994 0.002 0.003 0.002
0.000 0.001 0.933 0.076 0.001
0.000 0.001 0.235 0.724 0.001
0.005 0.001 0.007 0.004 0.977
O0

O1
O2
O3
Os
Predicted optimization level

FIGURE 7. Confusion matrices while detecting optimization level for each architecture. Results obtained with the CNN.

SPARC
0.993 0.001 0.001 0.002 0.012

O0

O1
O2
O3
Os
Predicted optimization levelO0

0.000 0.002 0.714 0.324 0.002
0.001 0.002 0.100 0.876 0.005
0.002 0.001 0.002 0.007 0.984
O0

O1
O2
O3
Os
Predicted optimization level

O0
True optimization level
O3
O2
O1

0.000 0.997 0.001 0.001 0.001

0.000 0.001 0.209 0.749 0.002
0.005 0.000 0.003 0.002 0.984
O0

0.990 0.000 0.000 0.000 0.003
0.000 0.999 0.000 0.000 0.001
0.000 0.000 0.976 0.025 0.002
0.000 0.000 0.036 0.955 0.002
0.001 0.000 0.002 0.001 0.997
O0

O1
O2
O3
Os
Predicted optimization level

O1
O2
O3
Os
Predicted optimization level

ARM

True optimization level
O3
O2
O1

True optimization level
O3
O2
O1

0.000 0.001 0.799 0.232 0.003

MIPS

Os

Os

True optimization level
O3
O2
O1

O0

PowerPC

0.993 0.001 0.001 0.006 0.007

O1
O2
O3
Os
Predicted optimization level

0.000 0.994 0.003 0.001 0.003

Os

0.000 0.002 0.011 0.004 0.976

Os

0.001 0.001 0.072 0.926 0.004

O00.001 0.001 0.001 0.002 0.992

0.001 0.002 0.929 0.060 0.008

True optimization level
O3
O2
O1

0.000 0.003 0.146 0.856 0.003

0.000 0.995 0.003 0.001 0.002

Os

0.001 0.004 0.784 0.199 0.003

True optimization level
O3
O2
O1

0.000 0.993 0.003 0.002 0.001

O0

RISC-V
0.995 0.000 0.001 0.001 0.009

O0AArch64
0.999 0.000 0.001 0.000 0.000

O0

x86_640.999 0.001 0.001 0.000 0.000

Os

Os

True optimization level
O3
O2
O1

O0

Normalized confusion matrices (LSTM)

0.001 0.994 0.001 0.002 0.003
0.001 0.001 0.947 0.055 0.004
0.001 0.001 0.090 0.891 0.004
0.003 0.001 0.003 0.002 0.987
O0

O1
O2
O3
Os
Predicted optimization level

0.993 0.000 0.001 0.001 0.011
0.001 0.997 0.001 0.000 0.001
0.000 0.001 0.902 0.110 0.002
0.000 0.001 0.102 0.877 0.002
0.002 0.001 0.002 0.002 0.992
O0

O1
O2
O3
Os
Predicted optimization level

FIGURE 8. Confusion matrices while detecting optimization level for each architecture. Results obtained with the LSTM.

9VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Access

Normalized confusion matrices (AArch64, 2-step evaluation)

0.001

0.000

0.000

0.986

0.004

0.009

0.000

0.003

0.987

0.032

0.000

0.002

0.012

0.962

O1
O2/O3
Os
Predicted optimization level

0.999

0.000

0.001

0.000

0.000

0.993

0.003

0.003

0.000

0.001

0.996

0.010

0.000

0.001

0.007

0.978

O0

1.0
0.8
Accuracy

0.000

True optimization level
O2/O3
O1
O0

0.998

O0

Accuracy variation in optimization detection
with increasing input length (CNN)

LSTM

Os

Os

True optimization level
O2/O3
O1
O0

CNN

Accuracy x86_64
Accuracy AArch64
Accuracy RISC-V
Accuracy SPARC
Accuracy PowerPC
Accuracy MIPS
Accuracy ARM

0.4

O1
O2/O3
Os
Predicted optimization level

CNN

0.6

LSTM

0.850

0.144

0.177

0.831

True optimization level
O3
O2

True optimization level
O3
O2

0.2

O2
O3
Predicted optimization level

0.825

0.167

0.094

0.910

0

x86_64
AArch64

0.99950.9996

0.99320.9996TABLE 4. Accuracy for each architecture while detecting the compiler

0.8
0.6
Accuracy x86_64
Accuracy AArch64
Accuracy RISC-V
Accuracy SPARC
Accuracy PowerPC
Accuracy MIPS
Accuracy ARM

0.4
0.2

and its faster speed compared to an LSTM, it should be thepreferred choice for compiler detection.
We can thus answer RQaccuracy as follows:
While detecting the optimization level, the LSTM canoffer higher accuracy at the price of slower train andinference. The accuracy range from a minimum of 92%
to a maximum of 98% depending on the architecture.
While detecting the compiler, however, both networks
perform well. In this case the CNN is the preferredchoice due to its speed advantage and an accuracy of
99.95%.
B. MINIMUM BYTESThis section investigates the possibility of detecting the optimization level and compiler with function granularity. To
this end, we performed our evaluation for each model whilefeeding a progressively increasing number of bytes. We thus
performed the initial evaluation with only 1 byte for eachsample; then, we performed a second evaluation with 2 bytes
and so on until we used the full vector length of 2048 bytes.
Figure 10 shows the results obtained using the CNN network. Figure 11, shows the same results but with the LSTM
network.
Note that every architecture follows the same detection
trend in the LSTM network. In the CNN one, however,

2000

1.0

Accuracy

LSTM Acc.

1500

Accuracy variation in optimization detection
with increasing input length (LSTM)

FIGURE 9. Confusion matrices whith split dataset. CNN on left, LSTM onright.

CNN Acc.

1000
Input bytes

FIGURE 10. Accuracy for the CNN in the optimization detection.

O2
O3
Predicted optimization level

Architecture

500

0

500

1000
Input bytes

1500

2000

FIGURE 11. Accuracy for the LSTM in the optimization detection.

the accuracy for x86_64, PowerPC, and RISC-V stops
increasing, unlike in the other architectures. These threearchitectures are the same architecture we classified as “problematic" in Section V-A. In addition, this does not happen for
the LSTM network. We can assume the CNN failed to learn
how to properly recognize O2 and O3 in these architectures,
given the strong similarity between these two optimizationlevels. As such, additional bytes do not help the network at
all, in contrast to the LSTM case.
Additionally, we can note how, with any number of bytes,
the LSTM performs definitely better than the CNN.
To highlight this, Figure 12 shows a direct comparison
between LSTM and CNN in a single architecture, x86_64.
This figure, shows how there is always approximately 5%
more accuracy in the predictions of an LSTM compared tothe predictions of a CNN.
Having analyzed how the overall accuracy varies when the
input length changes, we now want to check whether the
average function length is sufficient to achieve good accu-

10

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE AccessAccuracy variation in optimization detection
with increasing input length
0.9

Accuracy

0.8
0.7

Arch.

O0

O1

O2

O3

Os

x86_64
AArch64
RISC-VSPARC
PowerPC
MIPS
ARM

65
68
84
104
136
184
72

223
232
160
192
224
276
188

220
220
166
220
264
284
174

257
260
218
268
292
376
232

125
132
113
132
148
188
106

TABLE 5. Median number of bytes per function for each optimization level ineach architecture. Results collected over a total of 47 · 106 functions.

0.6
0.5
0.4

Accuracy CNN, x86_64
Accuracy LSTM, x86_64

0.3
0

500

1000
Input bytes

1500

2000

FIGURE 12. Comparison between CNN and LSTM in the x86_64
optimization detection.

of correctly predicting each optimization level. Instead, wewant to consider the accuracy of predicting each optimizationlevel at its own statistical median, that represents the typicallength of a function at that optimization level. As an example,
for architecture x86_64 we evaluate the accuracy for O0
with an input of 65 bytes, for O1 with an input of 223 bytes.
Table 6 shows the accuracy at these input lengths.

Function length for each optimization level

Function length (bytes)

10

3

212

210

268

Arch.

O0

O1

O2

O3

Os

x86_64
AArch64
RISC-VSPARC
PowerPC
MIPS
ARM

0.992
0.984
0.987
0.985
0.988
0.985
0.984

0.935
0.948
0.863
0.899
0.968
0.986
0.951

0.568
0.719
0.729
0.829
0.701
0.786
0.658

0.579
0.590
0.344
0.512
0.527
0.771
0.529

0.761
0.694
0.791
0.808
0.884
0.909
0.807

132
10

2

TABLE 6. Accuracy of each optimization level limiting input to the median
number of bytes per function at that optimization level. Results obtained with
the LSTM.

84

O0

O1

O2
Optimization level

O3

Os

FIGURE 13. Statistics about the length of 47 · 106 functions. The value insideeach box refers to the median.

racy. To do this, however, we need to gather statistical dataon binary files. The idea is to calculate the average function
length at each optimization level and check the accuracy ofthe network for that particular level at that particular length.
We disassembled every binary for every optimization level
and compiler and counted the number of bytes that compose
each function. The results are shown in Figure 13.
The reported number is the median, calculated over 47·106
functions across all architectures. We chose to show thisaverage metric, as opposed to the mean, as we want to draw
conclusions based on the “typical" function length. This isalso a much more conservative approach, given that the meanis influenced by some outliers with a very high numberof bytes, on the order of 106 . Although Figure 13 shows
the medians for all the architectures together, more precise
results for each architecture are reported in Table 5.
At this point, we calculated the accuracy for each of thelisted medians. Figure 10 and 11 show the overall accuracy

The Table confirms the results we obtained in Section V-A.
Optimization levels O0 and O1 are easy to detect even atfunction granularity. The same goes for Os, although witha lower accuracy. The problem is, again, distinguishing between O2 and O3, as the median length of each function
at that optimization level is not enough for an accurate
prediction.
Regarding the compiler detection, the accuracy plot forincreasing number of bytes can be seen in Figure 14. Unlike
the CNN and LSTM comparison for optimization level detection, in Figure 12, we can see very few differences between
the two networks, even with shorter inputs. Moreover, theaccuracy is high even when there is not much data available;
for example, with only 100 bytes, it is possible to have morethan 90% accuracy. This means that we can correctly predictthe compiler, even with function granularity.
Given these results, we can answer RQmin as follows:
When performing a function grained analysis, witha short input, it is generally possible to detect O0,
O1 and Os optimization level. O2 and O3, instead,requires as much bytes as possible, given their subtle
differences. In contrast, compiler detection does not
suffer this problem, achieving great accuracy even with
102 input bytes.
11

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE AccessAccuracy variation in compiler detection
with increasing input length

1.0

1.0

0.9

0.9

0.8

0.8

Accuracy

Accuracy

Accuracy variation in compiler detection
with increasing input length

0.7
0.6

Accuracy CNN, x86_64
Accuracy LSTM, x86_64
Accuracy CNN, AArch64
Accuracy LSTM, AArch64

0.5
101

100

Input bytes

102

103

0.8
Accuracy

0.7
0.6
0.5
0.4
Accuracy x86_64 CNN, Raw
Accuracy x86_64 LSTM, Raw
Accuracy x86_64 CNN, Encoded
Accuracy x86_64 LSTM, Encoded
500

1000
Input bytes

1500

101

Input bytes

102

103

towards the encoded variant, especially when we use a small
number of bytes. In fact, for any given number of bytes inthe encoded variant, we have more bytes available in thecorresponding raw one. We calculated this difference to be
an average of 186 bytes per function.

0.9

0

0.5

FIGURE 16. Accuracy in the compiler detection with encoded input data.

Accuracy variation in optimization detection
with increasing input length

0.2

Accuracy x86_64 CNN, Raw
Accuracy x86_64 LSTM, Raw
Accuracy x86_64 CNN, Encoded
Accuracy x86_64 LSTM, Encoded

0.6

100

FIGURE 14. Accuracy in the compiler detection.

0.3

0.7

2000

FIGURE 15. Accuracy in the optimization detection with encoded input data.

C. ENCODING

After showing how function-grained analysis is possible forsome flags in Section V-B, we want to explore a possible
improvement by removing redundant information from the
input array. This stems from the conclusion of Chen et al.
who found removing x86_64 prefixes increases the accuracy [10]. In this section, we compare the raw input withthe encoded variant which is explained in Section IV-B. The
result of this analysis is shown in Figure 15, depicting only
the x86_64 architecture.
We can note how the encoded variant reflects the samedifference between LSTM and CNN previously highlighted
in Section V-B, in particular in Figure 12. More interestingly,
the encoded variant reaches its maximum accuracy with an
input length of approximately 250 bytes. The raw input
variant, instead, as more bytes are supplied to it, steadily
improves in accuracy beyond this limit.
Moreover, the comparison we used in the Figure is biased

A similar situation can be seen when analyzing compiler
detection, as depicted in Figure 16. In this case, we cansee that the encoded variant reaches maximum accuracy
at approximately 100 bytes without further improvements.
In contrast, the raw data accuracy continues to increase,outperforming the encoded variant at 150 bytes and peaking
at 1000 bytes.
We decided, however, against extending this analysis toall seven architectures. In fact, in Section I, one of the
motivations of our study was to have an automated wayof detecting optimization flags that does not require deep
knowledge of the underlying architecture. To generate theencoded variant, however, it is necessary to possess a basic
knowledge of the target architecture, which contradicts ouroriginal motivation. This fact, in addition to the poor performance and the need for accurate disassembly prompted us toabandon the encoded variant study.
Before concluding this section, it is worth noting that the
study of Chen et al. used a dataset 100 times smaller anddetermined the encoding variant was remarkably better [10].
In our previous study, we used a dataset 10 times smaller thanour current dataset and determined the encoding variant to beon par with the raw variant [11]. We can easily assume thatwith a smaller dataset, the network is less capable of learningwhich information is useful and what is not in the raw data.
This would explain why in previous studies the encodedvariant, which provides data without useless prefixes, wasmore competitive. However, with a sufficiently large dataset,the encoded variant does not offer any advantages.
We can thus conclude RQencoding as follows:

12

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE AccessAccuracy variation considering padded values
during training (MIPS architecture)

1.0

Accuracy, no padding (CNN)
Accuracy with padding (CNN)
Accuracy, no padding (LSTM)
Accuracy with padding (LSTM)

0.8

Operating System
Ubuntu Server 20.04
macOS Catalina

0.30
0.25

0.6

Probability

Accuracy

Optimization levels of pre-compiled binaries

0.20
0.15

0.4
0.10

0.2

0.05

100

101

Input bytes

102

103

FIGURE 17. Accuracy variation in the optimization detection evaluation when
including padding data during training.

Disassembling and encoding the data does not provide additional benefits, requiring knowledge of theunderlying architecture and function disassembly foran overall lower accuracy.
D. PADDING

In Section IV-B we assert that our networks perform worse
if, during training, raw byte sequences are never padded
during training, and then padded sequences are predicted. In
this section we present RQpad , and investigate the difference
between padding during training and not padding. In thisexperiment, we trained two networks with the same dataset,
seed, and samples ordered in the same way. However, in one
case; the training values were always of 2048 bytes, in theother case, they were randomly cut in the interval [32, 2048],
following the distribution explained in Section IV-C.
We evaluated both CNN and LSTM over the MIPS architecture, which achieved the best results. The differences
between the padded and unpadded variants are shown in
Figure 17
From the figure, we can see how the absence of padding
during training is a problem when evaluating small input
vectors. For example, the LSTM is approximately 10% lessaccurate and reaches its counterpart trained with padding
only when the input vectors are longer than 1000 bytes. The
CNN results are even more extreme: with less than 100 bytes
the network trained without padding always predicts the same
output, and even at 125 bytes, there is a 60% accuracy gap
between the two variants.
Although not presented, we performed this analysis also in
the x86_64 architecture, and obtained similar results.
We can thus conclude RQpad as follow:
If inputs are never padded during training, networks
will have significantly lower accuracy while predicting
shorter sequences.

0.00

O0

O1

O2
Optimization level

O3

Os

FIGURE 18. Optimization level in pre-compiled binaries shipped with Ubuntu
server 20.04 and macOS Catalina.

E. OCCURRENCE

To conclude the evaluation, we would like to provide somedata on the optimization flag distribution in a real scenario.
To do so, we took every binary and library inside an Ubuntu
Linux 20.04 server and macOS 10.15.7 Catalina, both unmodified. For each binary, we predicted the optimizationlevel using our LSTM model for x86_64, reporting theresults.
To analyze each binary, we divided the binary into several
chunks of 2048 bytes, which is the same as our max input
length. We then performed the inference for each chunk,and calculated an average between all the chunks. For eachchunk, we weighted its contribution to the average by the
network accuracy achieved at the chunk’s input length.
Figure 18 shows the results of this analysis, performed
over 10254 and 1216 files, respectively, for Ubuntu andmacOS. Given the highly imbalanced number of input files,
the histogram was normalized.
We can note how the distribution of files in the Linuxsystem tends towards the O2 optimization level. This is not
surprising, as the O2 optimization level provides the highest
optimization without increasing the code size to the same
extent as O3. The latter, in fact, could generate a larger
code that does not fit in the instruction cache, resulting inoverall slower execution [34]. Therefore, O2 is the suggested
optimization level in some distributions, such as Gentoo
Linux9 . The macOS result, despite being more diverse, shows
how most of its core programs are optimized for code size.
As this is rather uncommon, we verified this results bymanually inspecting the publicly available build scripts forApple software10 . As a confirmation, in most Makefiles we
can find Os as the default optimization flag, explaining thehistogram results. However, no reason for this choice is givenin the build scripts.
9 https://wiki.gentoo.org/wiki/GCC_optimization
10 http://opensource.apple.com

13

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Access

Nonetheless, this final analysis is useful to prove the pointthat targeting binary analyses at the O2 optimization levelassuming it to be the default one [6] may be a completely
wrong assumption in some cases.
With this data, we can answer RQoccurrence as follows:
The most common optimization found in Ubuntu Linuxis O2. In contrast, on macOs, Os is more common
although not as dominant as O2 is for the Linux case.
This proves that expecting O2 to always be the mostcommon optimization level may be a false expectation.
VI. DISCUSSION

The results obtained in Section V provide a fast way ofdetecting the optimization level with multiple granularities.
One major change from previous studies is the lack of disassembly in our approach. Using a disassembler in order toretrieve function boundaries can be very time consuming.For
large binaries, if the function grain is not required, previous
approaches still require functions beginning and function
ending, where ours can select a random 2048 bytes fromthe .text section and be dominated by the inference time.
Unlike disassembly, dumping raw bytes from an executableor library is very fast, because it involves just reading the fileitself.
If a function grain is necessary, the function headers may
be retrieved by other means (i.e. Deep Learning). Given that
our method does not require to preprocessing of the input
data, we can skip disassembly also in this case, avoiding
again the slowest part of binary analysis. This allows our tool
to be used to check the compilation flags even at runtime,
without any noticeable performance impact.
This lack of disassembly is the result of our evaluationin Section V-C, which contradicts the claim of Chen et al.
that the encoded variant is better [10]. As mentioned in thesection, this could be due to our larger dataset, which allowed
the network to automatically learn which data is useful inthe input architecture and which is not, rendering manual
encoding useless.
In our study, given the larger size of our analysis, we
also focused on producing an automated script to generate
the dataset. In fact, manually compiling a matrix of five
optimization levels using seven architectures would haverequired an unmanageable amount of time. In addition, withthis automated generation, we can extend the study to additional flags with small changes in the scripts parameters. In
previous approaches, including ours, the entire dataset had to
be manually regenerated to add new flags, the most tediouspart of this entire study.
In addition, thanks to the small number of bytes requiredby our method, we can target very small portions of code,and thus, it can be used to check which portions of the binary
match the used compilation flags. This allows for better
categorization of the binary content and can help in binary
analysis. In fact, if a small portion of the file is found withdifferent flags or compiler than the rest of the file, there is ahigh probability that this portion belongs to a static library ora different compilation unit.
VII. LIMITATIONS AND FUTURE WORKS

The analysis we performed was limited to a pair of compilers and the most common optimization levels. In thisstudy, we have shown that detection results can vary greatly
between different architectures, and we have no guaranteethat this analysis can be extended to more architectures
without sacrificing accuracy. This is true even in the caseof different compilers. The main difficulty in our study was
distinguishing between O2 and O3 given their similarities.
However, optimization levels are compiler specific and wecannot assume compilers other than GCC or Clang provide
the same set of optimization levels.
Moreover, in this study, we focused entirely on optimization levels instead of specific flags. Although it would beeasy to consider optimization flags, given our automateddataset generation, the classification should probably change
from multiclass to multilabel. Furthermore, some flags would
be challenging if not impossible to detect, the “dead code
elimination" flag being one example.
Future work will involve assessing the feasibility of thismultilabel classification, especially in compilers other than
GCC or Clang.
VIII. CONCLUSION

In this paper, we have described two deep learning networks,
one based on a long short-term memory model and theother based on a convolutional neural network model. We
evaluated them in seven different architectures and showedthat they can achieve between 92% and 98% accuracy while
detecting between five different optimization levels and over
99.99% accuracy while detecting two different compilers.
We also provided an evaluation of the minimum number ofbytes needed for accurate predictions, combined with statistical data about the different architectures and their medianfunction length for each optimization level. Ultimately weproved that function grained optimization level detection ispossible unless we are not aiming to distinguish between O2
and O3.
The results obtained are consistent with the initial motivation for our study: when comparing the structure of different
binaries we reported the highest accuracy drop emerging in
the case of different compilers or O0 compared with anyother optimization level. These are also the values with thehighest detection accuracy in our study, suggesting that ourapproach may be useful in detecting accuracy drop whencomparing different binaries.
IX. REPLICATION

The dataset used in our study can be found on Zenodo at thefollowing url [24]. This dataset contains each binary, divided
by architecture, optimization level and compiler. Source codeand pre-trained models can be found publicly on GitHub 11 .
11 http://github.com/inoueke-n/optimization-detector

14

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3132950, IEEE Access

REFERENCES[1] K. Hoste and L. Eeckhout, “Cole: compiler optimization level exploration,” in Proceedings of the 6th annual IEEE/ACM international symposium on Code generation and optimization, 2008, pp. 165–174.
[2] M. Demertzi, M. Annavaram, and M. Hall, “Analyzing the effects of compiler optimizations on application reliability,” in 2011 IEEE International
Symposium on Workload Characterization (IISWC), 2011, pp. 184–193.
[3] Y. David, N. Partush, and E. Yahav, “Similarity of binaries through reoptimization,” in Proceedings of the 38th ACM SIGPLAN Conference on
Programming Language Design and Implementation, 2017, pp. 79–94.
[4] N. E. Rosenblum, B. P. Miller, and X. Zhu, “Extracting compiler provenance from program binaries,” in Proceedings of the 9th ACM SIGPLANSIGSOFT workshop on Program analysis for software tools and engineering, 2010, pp. 21–28.
[5] J. Pallister, S. J. Hollis, and J. Bennett, “Identifying compiler options tominimize energy consumption for embedded platforms,” The Computer
Journal, vol. 58, no. 1, pp. 95–109, 2015.
[6] J. Pewny, B. Garmany, R. Gawlik, C. Rossow, and T. Holz, “Crossarchitecture bug search in binary executables,” in 2015 IEEE Symposium
on Security and Privacy. IEEE, 2015, pp. 709–724.
[7] N. Rosenblum, B. P. Miller, and X. Zhu, “Recovering the toolchain
provenance of binary code,” in Proceedings of the 2011 International
Symposium on Software Testing and Analysis, 2011, pp. 100–110.
[8] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neuralcomputation, vol. 9, pp. 1735–80, 12 1997.
[9] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[10] Y. Chen, Z. Shi, H. Li, W. Zhao, Y. Liu, and Y. Qiao, “Himalia: Recoveringcompiler optimization levels from binaries by deep learning,” in Intelligent
Systems and Applications, K. Arai, S. Kapoor, and R. Bhatia, Eds. Cham:
Springer International Publishing, 2019, pp. 35–47.
[11] D. Pizzolotto and K. Inoue, “Identifying compiler and optimization options from binary code using deep learning approaches,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME).
IEEE, 2020, pp. 232–242.
[12] O. Katz, Y. Olshaker, Y. Goldberg, and E. Yahav, “Towards neural decompilation,” arXiv preprint arXiv:1905.08325, 2019.
[13] R. Pascanu, J. W. Stokes, H. Sanossian, M. Marinescu, and A. Thomas,“Malware classification with recurrent networks,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE, 2015, pp. 1916–1920.
[14] B. Athiwaratkun and J. W. Stokes, “Malware classification with lstm andgru language models and a character-level cnn,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE, 2017, pp. 2482–2486.
[15] S. Triantafyllis, M. Vachharajani, N. Vachharajani, and D. I. August,“Compiler optimization-space exploration,” in International Symposiumon Code Generation and Optimization, 2003. CGO 2003. IEEE, 2003,
pp. 204–215.
[16] Z. Wang and M. O’Boyle, “Machine learning in compiler optimization,”
Proceedings of the IEEE, vol. 106, no. 11, pp. 1879–1901, 2018.
[17] A. H. Ashouri, W. Killian, J. Cavazos, G. Palermo, and C. Silvano, “A
survey on compiler autotuning using machine learning,” ACM Computing
Surveys (CSUR), vol. 51, no. 5, pp. 1–42, 2018.
[18] F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M. F. O’Boyle,
J. Thomson, M. Toussaint, and C. K. Williams, “Using machine learning
to focus iterative optimization,” in International Symposium on Code
Generation and Optimization (CGO’06). IEEE, 2006, pp. 11–pp.
[19] M. Stephenson, S. Amarasinghe, M. Martin, and U.-M. O’Reilly, “Metaoptimization: Improving compiler heuristics with machine learning,” ACM
sigplan notices, vol. 38, no. 5, pp. 77–90, 2003.
[20] T. Bao, J. Burket, M. Woo, R. Turner, and D. Brumley,“{BYTEWEIGHT}: Learning to recognize functions in binary code,” in23rd {USENIX} Security Symposium ({USENIX} Security 14), 2014, pp.
845–860.
[21] E. C. R. Shin, D. Song, and R. Moazzezi, “Recognizing functions inbinaries with neural networks,” in 24th {USENIX} Security Symposium({USENIX} Security 15), 2015, pp. 611–626.
[22] Z. L. Chua, S. Shen, P. Saxena, and Z. Liang, “Neural nets can learn
function type signatures from binaries,” in 26th {USENIX} Security
Symposium ({USENIX} Security 17), 2017, pp. 99–116.

[23] J. He, P. Ivanov, P. Tsankov, V. Raychev, and M. Vechev, “Debin: Predicting debug information in stripped binaries,” in Proceedings of the 2018
ACM SIGSAC Conference on Computer and Communications Security,
2018, pp. 1667–1680.
[24] D. Pizzolotto and K. Inoue, “Binary software compiled for different
architectures with different optimization levels,” Apr. 2021. [Online].
Available: https://doi.org/10.5281/zenodo.4659370[25] A. V. Aho, R. Sethi, and J. D. Ullman, “Compilers, principles, techniques,”
Addison wesley, vol. 7, no. 8, p. 9, 1986.
[26] D. Andriesse, X. Chen, V. Van Der Veen, A. Slowinska, and H. Bos, “Anin-depth analysis of disassembly on full-scale x86/x64 binaries,” in 25th
{USENIX} Security Symposium ({USENIX} Security 16), 2016, pp. 583–
600.
[27] M. Dwarampudi and N. V. S. Reddy, “Effects of padding on lstms
and cnns,” CoRR, vol. abs/1903.07288, 2019. [Online]. Available:
http://arxiv.org/abs/1903.07288[28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[29] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[30] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improveneural network acoustic models,” in Proc. icml, vol. 30, no. 1, 2013, p. 3.
[31] V. Nair and G. E. Hinton, “Rectified linear units improve restrictedboltzmann machines,” in Proceedings of the 27th international conferenceon machine learning (ICML-10), 2010, pp. 807–814.
[32] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press,
2016, http://www.deeplearningbook.org.
[33] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,”
The Journal of Machine Learning Research, vol. 18, no. 1, pp. 6765–6816,
2017.
[34] M. T. Jones, “Optimization in gcc,” Linux journal, vol. 2005, no. 131,
p. 11, 2005.

DAVIDE PIZZOLOTTO received his B.S. degree
in computer science and M.S. degree in computerscience from University of Trento in 2015 and2019 respectively. He previously was Research
Assistant at Fondazione Bruno Kessler (FBK) andhe is currently a Ph.D candidate in the Graduate
School of Information Science and Technology at
Osaka University. His research interests include
code obfuscation, binary code analysis and sourcecode analysis and transformation.

KATSURO INOUE received his Ph.D. from Osaka University in 1984. He was an associate professor of University of Hawaii at Manoa from1984 to 1986. After becoming an assistant professor of Osaka University in 1986, he has beena professor since 1995. His research interests include software engineering, especially software
maintenance, software reuse, empirical approach,program analysis, code clone detection, and software license/copyright analysis.

15

VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/