Capturing 0day Exploits with PERFectly Placed Hardware Traps
Cody Pierce, Endgame Matthew Spisak, Endgame
Kenneth Fitch, Endgame
Abstract
Advanced software exploitation is a rapidly changing
field of study. In recent years, clever ways to bypass
existing exploit defenses have become mainstream.
Reactive defensive solutions based on known exploitation techniques have been proven ineffective, and easily
circumvented. In this paper, we discuss a new systemfor early detection and prevention of unknown exploits.
Our system uses Performance Monitoring Unit hardware to enforce coarse-grained Control Flow Integrity(CFI). By using hardware features that exist in modern
processor architectures, and real-time CFI policy enforcement, we hope to prove that our approach is effective, suitable for practical use, while staying resistant to
bypass.

1. Introduction
The PMU and CFI have been discussed and studied inacademic literature for several years. Outside of academia, vendors are also hard at work to enforce theproper behavior of programs through control flow integrity. In most cases this enforcement is performed byvalidation logic inserted at compile time. While compile time instrumentation is powerful, it takes significant effort to leverage these features in commercial
software.
Our research furthers the state-of-the-art in PMU-based
defensive applications by introducing the concept ofreal-time control flow integrity, with the goal of developing a system that could be adapted to real-world scenarios such as web browsers and document readers.
Additionally we demonstrate how the use of the BranchPrediction Unit and, more specifically branch mispredictions, is a strong candidate for implementing
Control Flow Integrity policies. We intend to demonstrate that our hardware-assisted CFI (HA-CFI) system
has a low performance impact and measurable prevention success against 0day exploits and previously unknown exploitation techniques.

in most microprocessor architectures that provides useful performance measuring facilities for developers.
Most features of the unit are intended to count hardware
level events during program execution to aid in program optimization and debugging.
In their paper, Yuan et al [YUAN11] introduced thenovel application of these events to exploit detection forsoftware security. Their researched focused on using
PMU events along with the Branch Trace Store messages to correlate and detect code-injection and codereuse attacks without source code. Xia et al exploredthe idea further in their paper CFIMon [XIA12]. By
combining precise event context gathering with the
BTS and PEBS to enforce real-time control-flow integrity. In addition to these foundational papers, others
have pursued variations on the idea to specifically target exploit techniques such as Return-OrientedProgramming.
Alternatively, just-in-time CFI solutions have been proposed using dynamically instrumented frameworks
such as PIN [PIN12] or DynamoRIO [DYN16]. Theseframeworks dynamically interpret code as it is executed
while providing instrumentation functionality to developers. Applying control flow policies with a framework
like PIN allows for the flexible and reliable checking ofcode. However, it often incurs a significant CPU overhead, in the area of 10 to 100x, making it unusable in
the enterprise.
Our research into dynamic run-time CFI included parameters we feel would make this approach relevant toenterprise security, while also providing significant
detection and prevention assurances. We established thefollowing functional requirements:
1.
2.
3.

1.1. Prior Art and Operational Constraints

Our work builds on previous research that identified the
Performance Monitoring Unit of microprocessors as agood candidate for enforcing Control Flow Integrity.
The Performance Monitoring Unit is a specialized unit

4.

Implementation must work on 32 and 64bit
Operating Systems.
CFI policies must be applied without software
recompilation or access to source code.
The system must not have prior knowledge ofthe program, or preprocessing of the programin any way.
Performance overhead of the solution should
be minimal.

To maintain resiliency against bypass we included three
additional parameters to scope the work:
1.

2.
3.

Additional code will not be added to the running program in the form of “hooks” or validation logic.
It must work in the Kernel.
The system must be able to detect and prevent
an exploit in real-time

valid code pointer addresses as each .dll/.so is loaded
into protected processes. When a counter overflows, theInterrupt Service Routine (ISR) called is then able tocompare the mispredicted branch to a whitelist, anddetermine if the branch is anomalous.

2. Approach
HA-CFI uses PMU-based traps in order to apply
coarse-grained CFI on indirect calls on the x86 architecture. The system uses the PMU to count and trap
mispredicted indirect branches in order to validate
branch destinations in real-time. In addition to gaining
assistance from a carefully tuned PMU, a practical implementation of this approach requires support fromIntel’s Last Branch Record (LBR) feature, and a method for tracking thread context switching in a given OS.
It also requires an algorithm for validating branch destination addresses, all while keeping performance overhead to a minimum. After more than a year of finetuning these hardware features, we have proven ourmodel is capable of generically detecting control-flowhijacks in real-time with acceptable performance overhead on both Windows and Linux.
Because control-flow hijack attacks often stem from a
corrupted or modified VTable, many CFI designs focus
on validating all indirect branches. For example, oncean attacker is able to layout memory appropriately witha fake VTable with custom virtual function pointer(s), a
use-after-free or type confusion bug then exercises codethat invokes the virtual function in this hijacked VTable
providing the initial code execution. These call sites areindirect branches. Because these call sites have neverbefore jumped to the attacker controlled address, thisindirect call is almost always mispredicted by thebranch prediction unit. Therefore, by only focusing on
mispredicted indirect call sites we greatly limit the
number of places that a CFI check is necessary.
HA-CFI configures the Intel PMU on each core tocount and generate an interrupt on every mispredictedindirect branch. The PMU is capable of delivering aninterrupt any time an event counter overflows, and thus
HA-CFI sets the initial counter value to -1 and resets
the counter to -1 from the interrupt service routine togenerate a trap for every occurrence of the event. In thisway, the HA-CFI interrupt service routine becomes our
CFI component capable of validating each mispredictedcall and determining whether it is the result of malicious behavior. To validate target indirect branch addresses, HA-CFI builds a comprehensive whitelist of

Figure 1: High level design of HA-CFI using the PMU to validate
mispredicted branches

3. Design
To ensure we minimized the overhead of HA-CFI while
maintaining an extremely low false-positive rate, several key design decisions had to be made, and are described below.
3.1. The Indirect Branch

On the Intel x86 architecture, an indirect branch canoccur at both a CALL or JMP instruction. This paper
focuses exclusively on the CALL instruction for several
reasons. First, indirect JMP branch locations were
found most often to be utilized for switch statements, inwhich the jump table data would rarely if ever be corrupted or utilized in a control-flow hijack attack. In our
experimentation on Linux, we found roughly 12% ofhijacked indirect branches occurred as part of an indirect JMP, but occurred even less frequently on Windows. In fact, in one ActiveX Flash binary on Windows, 98% of all indirect branches were found to be
CALL instructions. Secondly, ignoring mispredicted
JMP instructions further reduces the overhead of HACFI. Therefore, we opted to omit mispredicted JMP
branches during this research, which can be achieved
with settings on the PMU and LBR.

Figure 2: A breakdown of hijackable indirect JMP vs CALL instructions found in Windows and Linux x64 binaries

The Intel PMU supports hundreds of events for counting, many of which provide conditional filters and“unit-masks”, offering numerous additional combinations and finer granularity for these events. We evaluated two similar events for counting mispredicted indirect
CALL
instructions:
BR_MISP_RETIRED.NEAR_CALL
andBR_MISP_EXEC.TAKEN_INDIRECT_NEAR_CALL. The main difference between the two events is that the
BR_MISP_RETIRED event can be utilized as a precise
event for PEBS. We verified that some speculative mispredicts counted by the BR_MISP_EXEC event occasionally included branches that were not actually mispredicted.
While
the
BR_MISP_RETIRED.NEAR_CALL technically includes both direct and indirect CALLs, rarely, if ever, isa direct call actually mispredicted. As a result we chose
this event as it causes fewer interrupts due to the excluEvent Name

Code

Mask

Description

BR_MISP_RETIRED.
NEAR_CALL

0xC5

0x02

Direct andindirect
mispredicted near call
instructions
retired.

BR_MISP_EXEC.TA
KEN_INDIRECT_NE
AR_CALL

0x89

0xA0

Taken
speculativeand retired
mispredicted indirect
calls

sion of speculative events.

Intel branch misprediction event codes

3.2. Added Precision with the LBR

CFIMon [XIA12] chose to validate branches in a delayed manner using the Intel Branch Trace Store (BTS),
a completely separate hardware feature from the PMU.
One of the reasons they focused on Intel BTS is because branch traces are recorded in a precise mannerwith the tradeoff that analysis of the trace data cannot
be performed in real-time, thus only offering a detection solution. HA-CFI offers both detection and prevention of control-flow hijacks since it’s operating in realtime, validating each mispredicted branch as it occurs.
The interrupt instruction skid is one of the greatest challenges to our approach. This term represents the number of instructions that execute past the instruction that
actually triggered the counter overflow in the PMU.
Since our ISR is responsible for validating each mispredicted branch destination address, we can’t rely on
the instruction pointer within the context of the interrupted thread to actually represent the precise branch

destination. In fact, [AMD07] states that this skid canbe up to 72 instructions. Frequently the skid is a single
instruction or just a handful of instructions. Regardless,to precisely resolve the exact branch that caused the
PMU to generate an interrupt, we make use of Intel’s
Last Branch Record (LBR) stack.
The Intel LBR stack size varies across different Intelmicroarchitectures: 16 entries for Haswell, and 32 entries for Skylake. A powerful feature of the LBR is theability to filter the types of branches that are recorded.
For example, returns, indirect calls, indirect jumps, andconditional branches can all be included or excluded.
With this in mind, we can configure the LBR to only
record indirect call branches occurring in user mode.
With this filter in place, our ISR queries the top of the
LBR stack knowing that it was the likely source for the
PMC overflow. In this way, the LBR offers a level ofprecision in determining the exact branch destination
address, and is thus a requirement for our system tooperate.
Additionally, the most significant bit of the LBR branch
FROM address indicates whether the branch was actually mispredicted. As a result, this provides a quick
filter for the ISR to ignore the branch if it was predicted
correctly. It’s important to note that we are not iteratingover the entire LBR stack, only the most recently inserted branch.
3.3. On-Demand PMU-Assisted CFI

HA-CFI is focused on protecting commonly exploited
applications such as browsers, mail clients, and Flash.
As such, the PMU and LBR are both configured to only
operate on mispredicted indirect calls occurring in user
mode, ignoring those that occur in ring-0. Moreover, bymonitoring thread context switches in both Windows
and Linux, we can turn the entire PMU on and off depending upon which applications are being protected.
For example, when a thread as part of Internet Explorer
is schedule in, we enable the PMU to resume trapping
mispredicted indirect call instructions. However, assoon as a thread not being monitored, such as a threadin Calc.exe, is scheduled in we can immediately disable
the PMU for that core. This design decision is perhaps
the most critical element in keeping our performance
overhead at an acceptable level.
3.4. Runtime Whitelist Generation

The final component to the HA-CFI system is the actual
integrity check that involves querying a whitelist data
structure containing valid destination addresses for indirect calls. Whitelist generation is performed at run-timefor each image loaded into a protected process.

We first focused on post-processing captured branchdata. In this setup, we had our kernel driver and ISR indata collection mode only, buffering each mispredictedindirect call and loaded image base addresses, andsending it to a user-space client. After running numerous benchmarking tools we started with a working data
set of over 150 million mispredicted branches. We used
this data to iteratively explore and add new static analysis algorithms for the identification of valid indirect calldestinations. Eventually, we generated a whitelist suchthat all branches from our dataset could be verified in a
hashtable leaving zero unknown captured branches.
The key to proper whitelist generation for indirect calladdresses is to exhaustively find all code pointer addresses present in each loaded image. While the specific implementation is slightly different between PE andELF binaries, the approach is the same. When loaded
into memory, PE/ELF binaries are scanned in search ofcode pointer addresses, addresses that point into the.text section, using the following checks:-

Exports: All function symbols are enumerated
and addresses retrieved
Relocations: Relocations section is scanned
searching for code pointers and recording addresses that point to .text
Callbacks: The .text section is scanned using
simple pattern matching to identify code
pointer addresses that we label callback functions

4. Implementation Challenges
Throughout the course of our research, we encounterednumerous hurdles to meet our original goal of lowoverhead, high detection stats. Some of these challenges such as registering for PMU interrupts on Windows,
and properly tracking thread context switches are highlighted in this section.
4.1. Windows PMU and APIC Programming

We first configured the performance management unit
(PMU) and the programmable interrupt controller(APIC) to use a supplied function pointer as the interrupt handler for counter overflows. Our initial prototype
was developed under Linux. Programming the interrupt
controller to handle PMU overflows was relatively
straightforward, but porting the same techniques toWindows proved problematic. We were aware that directly modifying the interrupt descriptor table (IDT)
would violate Windows’ Kernel Patch Protection[MIC16], aka PatchGuard, causing a system halt when
the operating system detected our changes. After significant research, we discovered an undocumented option

in the Windows Hardware Abstraction Layer (HAL)
that registers a driver supplied interrupt handler for
PMU interrupts. Calling the HalpSetSystemInformation
routine with the HalProfileSourceInterruptHandler
information class allowed us to register a callback for
performance interrupts supported by Windows.
NTSTATUS status;
PVOID buffer[1];buffer[0] = profileSourceInterruptHandler;
status = HalpSetSystemInformation(HalProfileSourceInterruptHandler,
sizeof(PVOID),
buffer);Registering a PMU Interrupt Handler on Microsoft
Windows

4.2. Thread Tracking

The largest issue with implementing CFI on Windows
was restricting PMU monitoring to a single process or
thread. The PMU isn’t natively aware of thread context
and will continuously count and generate interrupts. In
our Linux prototype, we leveraged a kernel API for
registering callbacks whenever a thread was swappedout on a given processor. To our knowledge, Windows
has no such functionality.
The technique we ultimately arrived at makes use of the
Asynchronous Procedure Call (APC) mechanism. Windows allows developers to register APC routines for agiven thread, which are then added to a queue to be
executed at certain points. This can be applied to scenarios requiring a thread to be interrupted when an
asynchronous event occurs, such as the fulfillment of an
IO request. However, we leveraged the fact that the
APC routine list is drained opportunistically when athread resumes execution after being swapped out. By
maintaining an APC registered on all threads that weseek to monitor at all times, we are notified that athread has resumed execution when our routine executes. The routine re-enables the PMU counter if necessary and updates various tracking metrics. We detect
when a processor swaps out a thread and begins executing another when we receive an interrupt in a different
thread context. We can then disable the PMU counters,
if needed.

5. Results
To evaluate our system, we measured success both interms of performance overhead added by HA-CFI as
well as detection statistics when tested against various
exploits in common client applications, including themost common web browsers, as well as Microsoft Office and Adobe Flash. We sourced exploits from
Metasploit modules for testing, as well as numerous
live samples from popular Exploit Kits found in thewild.

5.1. Performance

After completing our prototype, we were concerned
about the overhead of monitoring with HA-CFI and its
impact on system performance and usability. Since eachmispredicted branch in a monitored process would
cause an interrupt, there was the potential for a very
high number of interrupts to be generated. We subjected our prototype implementations to several tests tomeasure overhead. Monitoring Internet Explorer 11
while running a JavaScript performance test suite, thedriver detected approximately 83,000 interrupts per
second on average. In contrast, monitoring an “idle” IE
resulted in roughly 1,000 interrupts per second. Our
performance analysis revealed that overhead is highly
dependent upon the process being protected. For example, with Firefox we saw around 10% overhead whilerunning Dromaeo [DRO16] JavaScript benchmarks,
with PassMark benchmarking tool we saw 8-10% overhead, but with Internet Explorer under heavy usage thisnumber was above 10% We have deployed HA-CFI on
systems in daily use monitoring web browsing, andobserved little impact on performance or usability.

CODE EXECUTION TECHNIQUE# SAMPLES

HA-CFI
DETECTIONRATEEMET DETECTION
RATE

ROP

37

95%

100%

ROPless Technique A

1

100%

0%

ROPless Technique B

10

100%

0%

Results of VirusTotal Sample Testing, by Exploitation Technique

BUG CLASS

#
CVEs
3

# SAMPLES
6

HA-CFI DETECTION RATE
83%

Out-Of-Bounds
Write
Buffer Overflow
3
6
83%
Integer Overflow
2
6
100%
Use-After-Free
4
14
100%
Double Free2
4100%
Type Confusion
3
6
100%
Race Condition
1
4
100%
Uninitialized
1
1
100%
MemoryResults of VirusTotal Sample Testing, by Bug Class

5.2. Exploit Detection

We extensively tested HA-CFI against a variety of exploits to determine its efficacy against as many bugclasses and exploitation techniques as possible, with anemphasis on recent samples using approaches intended
to bypass other mitigation measures. We ran one set oftests against more than 15 Metasploit exploits targeting
Adobe Flash Player, Internet Explorer, and Microsoft
Word. HA-CFI detected and prevented exploitation foreach of the tested modules, with an overall detection
rate greater than 98%. We attribute the 2% false negative rate to instruction skid, as described above.
We found the Metasploit results to be encouraging, but
came to the conclusion that they did not provide sufficient diversity in exploitation techniques needed tocomprehensively test HA-CFI. We used the VirusTotal
service to download a set of samples used in real-world
exploit kit campaigns from several widespread kits
[KAF16]. In total, we tested forty-eight samples comprising twenty unique CVE vulnerabilities. We analyzed the samples to verify that they employed a varied
set of both Return-Oriented Programming (ROP) and“ROPless” techniques. HA-CFI succeeded in detecting
all 48 samples, with an overall detection rate of 96% in
a multiple trial consistency test.

5.3. Case Studies

The first case study is CVE-2015-2419, a double-free injscript9 and this particular vulnerability is being exploited in most of the popular exploit kits. This specificexample depicted below was taken from the Magnitude
exploit kit. We chose this sample since the exploit authors used the traditional ROP approach to codeexecution, where the initial control-flow hijack jumps
directly to stack pivot gadget, and then eventually agadget that returns into a call to VirtualProtect. Our
HA-CFI system was able to reliably detect and prevent
this initial control-flow hijack branch jumping to thestack pivot gadget. In comparison, Microsoft’s Enhanced Mitigation Experience Toolkit (EMET)[EME16] is only able to detect the stack pivot once the
VirtualProtect gadget is invoked. The figure below
shows where HA-CFI and EMET detect this particular
exploit found in the wild.

Figure 3: CVE-2015-2419 exploit execution prevention

A second case study to highlight is CVE-2014-0515, a
heap overflow in Adobe Flash. This bug is also commonly found in popular exploit kits, and while the bug
and sample itself are almost two years old, the significance of this bug is that the exploit authors implemented a technique that is “ROP-less”. In other words, a
DEP bypass is implemented without using ROP gadgets. Instead, the authors hijack a virtual function call for
FileReference.cancel() in ActionScript, but re-use afunction within the Flash binary that invokes VirtualProtect marking a region of memory as executable.
The exploit then searches for this VirtualProtect wrapper function, and points the virtual function pointer forcancel() to the wrapper function. ActionScript code isthen able to mark the region of memory containing
shellcode as executable, the process is repeated to hijack the same call to then jump straight to the shellcode
address. This control-flow hijack effectively bypasses
anti-ROP mitigations such as Microsoft EMET, while
HA-CFI is able to capture and prevent the initial hijack
branch since the wrapper function is not a normal
branch for indirect call sites. The figure below depicts
this example found in the SweetOrange exploit kit.

Figure 4: CVE-2015-0515 exploit execution prevention

5.4. Future Work

While the PMU is largely supported in current virtualization products, most hypervisors don’t emulate Intel’s
Last Branch Record (LBR) feature. Since HA-CFI uses
the LBR for precise lookups of the offending mispredicted branch, HA-CFI is currently unable to run invirtualized environments. We have performed some
initial investigations into modifying Xen [XEN16] tosupport the required MSR’s and determined it would bereasonable to modify the hypervisor to add the needed
LBR support and filtering, making our approach potentially viable on Xen guest operating systems.
Additionally, our current whitelist approach encounterschallenges identifying legitimate branches into just-intime (JIT) compiled code. We are continuing our research in this area with hopes of properly identifying
legitimate just-in-time code pages.

6. Conclusion
Modern exploitation techniques are rapidly changing. A
new approach to exploit detection that can work withcomplex applications is needed. In this paper wedemonstrated a practical and novel approach to exploit
detection that uses the Performance Monitoring Unit toenforce control flow integrity on branch mispredictions. Using a run-time generated whitelist we
can determine the validity of indirect calls to locationsclassified as malicious. This approach greatly reduces
the overhead of the instrumentation by moving the policy enforcement to a “coarse-grained” verifier on mispredicted indirect branch targets. The data provided
also shows the efficacy of such a system on samples
captured in-the-wild. These samples, from popular exploit kits, allow us to measure against unknown threatsfurther validating its application. As exploits advance,
we also need advanced exploit detection. Using HACFI we have advanced the state-of-the-art to give enterprise-scale security software an upper hand in thedetection of exploitation.

7. References[YUAN11] L. Yuan, W. Xing, H. Chen, B. Zang, “Security Breaches as PMU Deviation: Detecting and Identifying Security Attacks Using Performance Counters”,
APSys’11, July 11-12, 2011.
[PIN12] PIN: A Dynamic Binary Instrumentation Tool.
https://software.intel.com/en-us/articles/pin-a-dynamicbinary-instrumentation-tool[DYN16] DynamoRIO: Dynamic Instrumentation Tool
Platform.
http://www.dynamorio.org/[XIA12] Y. Xia, Y. Liu, H. Chen, and B. Zang,“CFIMon: Detecting violation of control flow integrity
using performance counters,” in Proceedings of the2012 42nd Annual IEEE/IFIP International Conferenceon Dependable Systems and Networks (DSN), pp. 1–12,
IEEE Computer Society, 2012.
[LI14] X. Li and M. Crouse, “Transparent ROP Detection using CPU Performance Counters.” https: / /www.trailofbits.com / threads / 2014 / transparent rop
detection using cpu perfcounters.pdf. Threads 2014.
[AMD07] P. Drongowski, “Instruction-Based Sampling: A New Performance Analysis Technique for
AMD Family 10h Processors”, AMD, Nov 2007.
[DRO16] Dromaeo JavaScript Benchmark.
http://www.dromaeo.com[EME16] The Enhanced Mitigation Experience
Toolkit.
https://support.microsoft.com/en-us/kb/2458544[KAF16]
Kafeine.
ExploitKit
Samples.
http://malware.dontneedcoffee.com/

[MIC16] Kernel patch protection for x64-based operating systems.
https://technet.microsoft.com/enus/library/cc759759(v=ws.10).aspx[XEN16] Xen Project
http://www.xenproject.org/