SASIMI 2019 Proceedings

R3-13

Detection of Vulnerability Guard Elimination by Compiler Optimization
Based on Binary Code Comparison
Yuka AZUMA

Nagisa ISHIURA

School of Science and Technology, Kwansei Gakuin University
2-1 Gakuen, Sanda, Hyogo, 669–1337, Japan

Abstract—It is known that guards against vulnerabilities in C
programs might be eliminated by compiler optimization if they
are not written properly. This paper proposes a method to detect such ﬂaws in software by binary code comparison. Given asource code, a pair of binary codes are generated, one with standard optimization and the other with problematic optimization
suppressed. Since simple comparison of the binary codes end up
with an unacceptable amount of false positives, call instructions
in each function are collated to detect discrepancies. In a preliminary experiment on 7 programs, our method successfully detected
2 instances of guard losses with only one false positive.

I. Introduction
Cyber attacks targeting software vulnerability are criticalthreats to our today’s society. To keep software free from such
ﬂaws, collective eﬀorts are being made. Security critical codes
are heavily audited and formal techniques may be applied insome cases.
However, even if thorough countermeasures are taken insource code level, there is no guarantee that the security iskept all the way down to the machine code. It is pointed outthat compiler optimization, even if it is correctly implemented,
can violate security guarantees incorporated in source code [1].
Examples of security violations induced by compiler optimization are elimination of codes to delete secret information, elimination of codes to guard vulnerabilities, destruction of codearrangement against side channel attacks, etc.
In this paper, we focus on guard elimination by compiler
optimization, where carelessly written guards to protect vulnerable code sections may be deleted by optimization utilizing
undeﬁned behavior in the C language standard. We propose
a method to detect the loss of guards by comparing a pair ofbinary codes, with and without problematic optimization. We
also propose a code comparison method focusing on call instructions, which yields fewer false positives. A preliminary
experiment demonstrates that guard extinction is detected byour method with very few false positives.
II. Elimination of Guards by Compiler Optimization
To protect the memory space from attacks, operations thatmay destroy the memory integrity must be guarded. For example, in Fig. 1 (a), dynamic memory allocation is attempted inline 6, for which the value of size must be checked if it is in
the valid range. The code in line 5 is a guard which is intended
to reject the case where overﬂow occurs on size.
Who wrote this code might expect that the value of size
would rap around on overﬂow. However, the C language speciﬁcation stipulates that signed overﬂow triggers undeﬁned behavior, where any result is valid for the program, including
continuing computation with random values or aborting thecomputation. C compilers, such as GCC and LLVM, makeSOF.c
1: #define BASE 1024
2:...
3: char *alloc buff (int n){
4:int size = BASE + n;
5:
if (size < n) error();6:
char* p = malloc(size);
7:
...
8: }

NPD.c
1: typedef struct{
2:int a[A SIZE];
3:
int x;
4: } str t;
5: ...
6: int sub(str t p){
7:...
8:
int y = p -> x;
9:
if (p==NULL) error();
10:p -> a[k] = z;
11: }12: ...

(a) Signed overﬂow (SOF)(b) NULL pointer dereference (NPD)

Fig. 1. Guard elimination by compiler optimization.

full use of the undeﬁned behavior in a source program to generate a better code. When overﬂow does not occur in line 4of the example, the comparison in the guard is always false
and hence the guard is not necessary. When overﬂow occurs,
any behavior is valid for this program, since it is undeﬁned behavior. Taking the both cases in consideration, compilers candelete the guard in line 5, so that the memory allocation in line
6 will be exposed without protect1 .
Fig. 1 (b) shows another example. The guard in line 9 rejects accesses to a struct by a NULL pointer. However, in
this code, the guard was accidentally misplaced; it should havebeen placed before line 8. When p is not NULL, the condition of the guard in line 9 is always false, and the guard isunnecessary. When p is NULL, the dereference of p in line
8 triggers undeﬁned behavior, then any behavior is valid forthis program. Taking both cases into account, compilers may
delete the guard in line 9. Most compilers do not generated thecode to abort program on NULL pointer dereference only; if
the eﬀective is valid, the access may be granted. This led to avulnerability in Linux kernel [2].
As a ﬁrst aid ﬁx to this problem, GCC has implemented
compiler options to suppress problematic optimization. For
example, -fwrapv option generates codes that precisely wrap
round on integer overﬂow. However, these options are not always recommended because they slow down the codes substantially. Moreover, not all the compilers do not support
all the options. GCC also has some options to warn oncode elimination utilizing undeﬁned behavior. For example, -Wstrict-overflow option turns on warning when code
elimination utilizing signed overﬂow is performed. However,this option does not always work and there is no option for
optimization regarding null pointer dereference. Static codeanalysis may detect this kind of ﬂaw, but it may yield many
irrelevant false positives.
1 The guard would not be eliminated if the predicate were written as(INT MAX-BASE<n). Unsigned integers cause no problem because they do
not overﬂow but wrap round.

- 229 -source

binaryTABLE I
Experimental result.
SOF
NPD
#line
program(×1000)#diﬀ#detect#diﬀ#detect
echo
4.9
0
000
mcc
1.2
1
100
libpng
4.6
1000
zlib
13.4
0000
libvorbis
19.2
0000pngquant
0.70000
pngquant’
0.7
1100

suppress
problematic
optimization

standard
optimizationcomparison

binary’Fig. 2. Flow of the proposed method.
binarybinary’...
<main>
callq <fprintf>
callq <memcpy>
callq <__assert_fail>callq <strcpy>...
<main>
callq <memcpy>
callq <fprintf>
callq <exit>
callq <strcpy>
callq <__assert_fail><sub1>...

<sub1>...

1:...
2:if ( x->type == token STAR) {3: int count = 0;
4: while (x -> type == token STAR ) {5:count ++;6:...
7: }
8: if (count == 1) { ... }9: else if (count > 1) { ... }10: else { assert (0); }11: ...

Fig. 3. Code comparison focusing on call instructions.

Detection of Guard Elimination by Compiler
Optimization
This paper proposes a method to detect guard elimination bycompiler optimization. We assume that a set of source codes
for building a binary executable code is given. Then it is tested
if any guard is lost during compilation.
Fig. 2 illustrates the ﬂow of the proposed method. A pairof binary codes are generated from given source codes. One
is with standard optimizing options. The other is without
optimization that is known to delete guards. By comparing the binary codes (the assembly codes obtained by disassembling), loss of guards can be detected. For example, in the case of GCC, optimization tasks utilizing undeﬁned behavior regarding signed overﬂow and null pointerdereference are suppressed by compiler options -fwrapv and-fno-delete-null-pointer-checks, respectively.
However, these suppressors change resulting codes morelargely than we expect, all over the whole code. This is because many optimization passes work in combination and dependent on each other. As a result, a simple code diﬀerencer
produces an impractical amount of false positives.
To solve this problem, we compare a pair of binaries focusing on subroutine calls. This scheme works because the guards
almost always have calls error handling routines for failed assertions. In our method, call instructions in each subroutine
are collated. Fig. 3 illustrates the procedure. The codes aredissected into subroutines and then call instructions (callq)
in each function are matched by the operands. Reordering ofthe instructions are allowed within the subroutine. If a lonely
call is found, it is reported as a potential guard loss.

Fig. 4. Code fragment of mcc.

III.

IV. Preliminary Experiment
A guard loss detector based on the proposed method is implemented in Perl5. It runs on Linux (Ubuntu 18.04.2).
The result of preliminary experiment is summarized in Table I. The compiler used was GCC-7.4.0 and the default
optimizing option was -O2. “SOF” and “NPD” are results
when -fwrapv and -fno-delete-null-pointer-checks
were applied which suppressed optimization regarding signed
overﬂow and null pointer dereference, respectively. “#diﬀ”
shows the number of reported code diﬀerences and “#detect”
the number of actual guard losses which were conﬁrmed by human inspection. Two guard losses regarding signed overﬂow

were detected with only one false positives.
Fig. 4 shows a code fragment of mcc. It was found that
assert statement in line 10 was optimized away. Note that nowarning was issued even with -Wstrict-overflow option,
so that the programmer had had no chance to know the guardloss. The diﬀerence reported on “libpng” was a false positive.
This was due to the fact that -fwrapv option somehow suppressed function inlining of a certain function which made thenumber of call instructions diﬀerent. Program “quant’“ wasa clone of “quant” but one of the guards was intentionallyrewritten so that the predicate was dependent on wrap roundon signed overﬂow. The loss of guard from this code ﬂaw wassuccessfully detected.
Our method just detects that guards are eliminated, which
do not directly lead to vulnerabilities. However, we consider itstill important to detect and remove potential vulnerabilities.
V. Conclusion
This paper has proposed a method of detecting guard elimination by compiler optimization. Loss of guards have beensuccessfully detected in a preliminary experiment.
Currently, localization and conﬁrmation of guard losses aredone by manual inspection, which should be automated tosome extent. Moreover, experiments on much more programs
should be done. We will be working on these issues.
Acknowledgments—Authors would like to express their appreciation
to the members of Ishiura Lab. of Kwansei Gakuin Univ. for theircooperation.

References[1] Vijay D’Silva, et al.: “The Correctness-Security Gap in
Compiler Optimization,” in Proc. IEEE Security and Privacy Workshops, pp. 73–87 (May 2015).
[2] Cve-2009-1897, http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1897 (June 2009).
[3] “Bug 30475: assert(int+100>int) optimized away,”
https://gcc.gnu.org/bugzilla/show_bug.cgi?
id=30475 (Jan. 2007).

- 230 -