Using Undocumented CPU Behaviour to See into Kernel
Mode and Break KASLR in the Process
Anders Fogh and Daniel Gruss

August 4, 2016

1 / 64

About this presentation

This talk is about a class of microarchitectorial attacks
I Not about software bugs
I It is about CPU design as an attack vector
I But not about Instruction Set Architecture
I Focus on Intel x86-64 - applies to other architectures too

2 / 64

Take aways

Take aways
I CPU design is security relevant
I Prefetch instructions leak information

3 / 64

Take aways

Take aways
I CPU design is security relevant
I Prefetch instructions leak information

Exploit this to:
I Locate a driver in kernel = defeat KASLRI Translate Virtual to physical addresses for other attacks

3 / 64

Introduction
Memory subsystem
Kernel Address-space Layout Randomization (KASLR)
Prefetch Side Channel
Prefetching the Kernel
Case study: Defeating Windows 7 KASLR
Case study: Exploiting direct-physical maps
Bonus material
4 / 64

Introduction

5 / 64

The chamber of secrets

NOTE

6 / 64

The chamber of secrets

NOTE
Using the PREFETCH instruction is recommended only if data does not fit incache.

6 / 64

The chamber of secrets

NOTE
Using the PREFETCH instruction is recommended only if data does not fit incache. Use of software prefetch should be limited to memory addresses that aremanaged or owned within the application context.

6 / 64

The chamber of secrets

NOTE
Using the PREFETCH instruction is recommended only if data does not fit incache. Use of software prefetch should be limited to memory addresses that aremanaged or owned within the application context. Prefetching to addresses thatare not mapped to physical pages can experience non-deterministic performance
penalty.

6 / 64

The chamber of secrets

NOTE
Using the PREFETCH instruction is recommended only if data does not fit incache. Use of software prefetch should be limited to memory addresses that aremanaged or owned within the application context. Prefetching to addresses thatare not mapped to physical pages can experience non-deterministic performance
penalty. For example specifying a NULL pointer (0L) as address for a prefetch
can cause long delays.

6 / 64

The chamber of secrets (translation)

PLEASE

7 / 64

The chamber of secrets (translation)

PLEASE,
only use prefetch as Intel intends

7 / 64

The chamber of secrets (translation)

PLEASE,
only use prefetch as Intel intends,
or else Intel will be angry

7 / 64

The chamber of secrets (translation)

PLEASE,
only use prefetch as Intel intends,
or else Intel will be angry,
and there is no reason why anyone would measure the execution time.

7 / 64

The chamber of secrets (translation)

7 / 64

WhoamiI Anders Fogh
I Principal Security Researcher, GDATA Advanced Analytics
I Playing with malware since 1992
I Twitter: @anders_fogh
I Email: anders.fogh@gdata-adan.de

8 / 64

WhoamiI Daniel GrussI PhD Student, Graz University of Technology
I Currently intern at Microsoft Research Cambridge
I Twitter: @lavados
I Email: daniel.gruss@iaik.tugraz.at

9 / 64

And the teamThe research team
I Clémentine Maurice
I Moritz Lipp
I Stefan Mangard

from Graz University of Technology

10 / 64

Memory subsystem

11 / 64

CPU Caches

Memory (DRAM) is slow compared to the CPU
I buffer frequently used memory for the CPU
I every memory reference goes through the cache
I transparent to OS and programs

12 / 64

Memory Access Latency
cache hitscache misses

Number of accesses

107
106
105
104
103
102
101
100
50

100

150

200

250

300

350

400

Access time in cycles
13 / 64

Data Caches

core 0core 1core 2

core 3

L1

L1

L1

L1

L2

L2

L2

L2

Last-level cache:
I shared memory shared is inring bus

cache, across cores!
→ physically indexed
I need physical address to

LLC
slice 0LLC
slice 1

LLC
slice 2

LLC
slice 3manipulate
I only one cache entry perphysical address

14 / 64

Unprivileged cache maintainanceUser programs can optimize cache usage:
I prefetch: suggest CPU to load data into cache
I clflush: throw out data from all caches... based on virtual addresses

15 / 64

Caches: Software control

There are 5 prefetch instructions:
I prefetcht0: suggest CPU to load data into L1
I prefetcht1: suggest CPU to load data into L2
I prefetcht2: suggest CPU to load data into L3
I prefetchnta: suggest CPU to load data for non-temporal access
I prefetchw: suggest CPU to load data with intention to write

actual behaviour varies between CPU models

16 / 64

Caches: Software control

The prefetch instructions are somewhat unusual
I Hints – can be ignored by the CPU
I Do not check privileges or cause exceptions

17 / 64

Virtual and physical addressing

Why address translation: Run multiple processes securely on a single CPU
I Let applications run in their own virtual address space
I Create exchangeable map from “virtual memory” to “physical memory”
I Privileges are checked on memory accesses
I Managed by the operating system kernel

18 / 64Address translation on x86-64
CR3

PML4
PML4E 0PML4E 1···
#PML4I···PML4E 511

PDPT
PDPTE 0PDPTE 1···#PDPTI···PDPTE 511

Page Directory
PDE 0
PDE 1
··
·
PDE #PDI···
PDE 511

PML4I (9 b) PDPTI (9 b) PDI (9 b) PTI (9 b)48-bit virtual address

Page Table
PTE 0
PTE 1
··
·
PTE #PTI···
PTE 511

Offset (12 b)

4 KiB Page
Byte 0
Byte 1···
Offset···Byte 4095
19 / 64Address Translation Caches

Problem: translation tables are stored in physical memory

20 / 64

Solution: Address Translation CachesCore 0DTLB
PDE cache
PDPTE cache

Lookup direction

ITLB

Core 1ITLB

PML4E cache

DTLB
PDE cache

PDPTE cache
PML4E cache

Page table structures insystem memory (DRAM)

21 / 64

Kernel Address-space Layout Randomization (KASLR)

22 / 64

Kernel is mapped in every process

Today’s operating systems:
Shared address space
User memory

Kernel memory−10context switch

23 / 64Address-Space Layout Randomization (ASLR)I Kernel and drivers at randomized offsets in virtual memory
I Mitigates code reuse attacks e.g. return-oriented-programming
I Attacks based on read primitives or write primitives

24 / 64Address-Space Layout Randomization (ASLR)I Kernel and drivers at randomized offsets in virtual memory
I Mitigates code reuse attacks e.g. return-oriented-programming
I Attacks based on read primitives or write primitives
I But: leaking kernel/driver addresses defeats ASLR

24 / 64

Kernel direct-physical map
Physical memory0max. phys.

re
di

0

User

247

−247

Kernel

m
ct

ap

−1

Virtual address space

25 / 64

Kernel direct-physical map
Physical memory0max. phys.

re
di

0

User

247

−247

Kernel

m
ct

ap

−1

Virtual address space

Available on many operating systems / hypervisors
I OS XI Linux
I BSD
I Xen PVM (Amazon EC2)
But not on Windows!
25 / 64

Prefetch Side Channel

26 / 64

Summary
1.
I

The kernel is mapped in every process

27 / 64

Summary
1.
I

The kernel is mapped in every process2.
The prefetch instruction takes a virtual address as input
To manipulate L3 a physical address is needed= The prefetch instruction must translate
II

27 / 64

Summary
1.
I

The kernel is mapped in every process2.
The prefetch instruction takes a virtual address as input
To manipulate L3 a physical address is needed= The prefetch instruction must translate
I
I

3.
The prefetch instruction does not check privileges= Any address can be prefetchedI

27 / 64

Summary
1.
I

The kernel is mapped in every process2.
The prefetch instruction takes a virtual address as input
To manipulate L3 a physical address is needed= The prefetch instruction must translate
I
I

3.
The prefetch instruction does not check privileges= Any address can be prefetched
I

4.
I
I

Translation is cached
Lookup searches caches in a fixed order

27 / 64

Summary
1.
I

The kernel is mapped in every process2.
The prefetch instruction takes a virtual address as input
To manipulate L3 a physical address is needed= The prefetch instruction must translate
I
I

3.
The prefetch instruction does not check privileges= Any address can be prefetched
I

4.
I
I

Translation is cached
Lookup searches caches in a fixed order

Can we measure a time difference?
27 / 64

Execution time in cycles

There is a timing difference!

400

383

300
230

246
222

200

181
PDPT

PD

PT
Mapping level

cached P.

uncached P.

Idea: Would this also work on inaccessible kernel memory?

28 / 64

Prefetching the Kernel

29 / 64

A translation oracle

Definition of our translation oracle
I Timing the prefetch instruction on an arbitrary address will recover thetranslation level

30 / 64

Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses

31 / 64

Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
2. Recover mappings in PDPT by using the translation oracle on kernel
addresses

31 / 64

Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
2. Recover mappings in PDPT by using the translation oracle on kernel
addresses
3. Recover mappings in PD by using the translation oracle on kernel addresses

31 / 64

Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
2. Recover mappings in PDPT by using the translation oracle on kernel
addresses
3. Recover mappings in PD by using the translation oracle on kernel addresses
4. Recover mappings in PT by using the translation oracle on kernel addresses

31 / 64

Recovering a map of the kernel
Recovering /proc/pid/pagemap in 4 steps:
1. Recover mappings in PML4 by using the translation oracle on kernel
addresses
2. Recover mappings in PDPT by using the translation oracle on kernel
addresses
3. Recover mappings in PD by using the translation oracle on kernel addresses
4. Recover mappings in PT by using the translation oracle on kernel addresses
Complete process takes from seconds to hours depending on how pages are
actually mapped by the operating system.

31 / 64

The address-translation oracle

The address-translation oracle tell us whether virtual address p and p̄ map to thesame physical address
1. Use clflush to remove p from cache

32 / 64

The address-translation oracle

The address-translation oracle tell us whether virtual address p and p̄ map to thesame physical address
1. Use clflush to remove p from cache
2. Use prefetch to load p̄ into cache

32 / 64

The address-translation oracle

The address-translation oracle tell us whether virtual address p and p̄ map to thesame physical address
1. Use clflush to remove p from cache
2. Use prefetch to load p̄ into cache
3. time access of p. If fast it was cached: p: maps to the same physical memory
as p̄
Beware! prefetch is a hint!

32 / 64

Timing instructions
The CPU may reorder instructions – a look at rdtscp
instruction 1

rdtscpinstruction 2

rdtscpinstruction 3

33 / 64

Timing instructions
The CPU may reorder instructions – a look at rdtscp
instruction 1

rdtscp3
instruction 2
3
rdtscp3
instruction 3

33 / 64

Timing instructions
The CPU may reorder instructions – a look at rdtscp
instruction 1

rdtscp7

instruction 2

rdtscp7

instruction 3

33 / 64

Timing instructions
The CPU may reorder instructions – a look at mfence
instruction 1

mfenceinstruction 2

mfenceinstruction 3

34 / 64

Timing instructions
The CPU may reorder instructions – a look at mfence
instruction 1

mfence3
instruction 2
3
mfence3
instruction 3

34 / 64

Timing instructions
The CPU may reorder instructions – a look at mfence
instruction 1
7mfenceinstruction 2
7
mfenceinstruction 3

34 / 64

Timing instructions
The CPU may reorder instructions
instruction 1

cpuid

instruction 2

cpuid

instruction 3

but not over cpuid!
35 / 64

Timing the prefetch instruction
The CPU may reorder prefetch instruction – a look at rdtscp
prefetch

rdtscpprefetch

rdtscpprefetch

36 / 64

Timing the prefetch instruction
The CPU may reorder prefetch instruction – a look at rdtscp
prefetch

rdtscp3
prefetch3
rdtscp3
prefetch

36 / 64

Timing the prefetch instruction
The CPU may reorder prefetch instruction – a look at rdtscp
prefetch

rdtscp3

prefetch

rdtscp3

prefetch

36 / 64

Timing the prefetch instruction
The CPU may reorder instructions – a look at mfence
prefetchmfenceprefetchmfenceprefetch

37 / 64

Timing the prefetch instruction
The CPU may reorder instructions – a look at mfence
prefetchmfence3
prefetch3
mfence3
prefetch

37 / 64

Timing the prefetch instruction
The CPU may reorder instructions – a look at mfence
prefetchmfence3

prefetchmfence3

prefetch

37 / 64

Timing instructions

Combine instructions in clever procedures to
I prevent reordering of prefetch if necessary
I avoid noise from cpuid if possible

38 / 64

Timing instructions

Combine instructions in clever procedures to
I prevent reordering of prefetch if necessary= mfence rdtscp cpuid target instruction cpuid rdtscp mfence
I avoid noise from cpuid if possible

38 / 64

Timing instructions

Combine instructions in clever procedures to
I prevent reordering of prefetch if necessary= mfence rdtscp cpuid target instruction cpuid rdtscp mfence
I avoid noise from cpuid if possible= mfence cpuid rdtscp target instruction rdtscp cpuid mfence

38 / 64

Timing instructions

Combine instructions in clever procedures to
I prevent reordering of prefetch if necessary= mfence rdtscp cpuid target instruction cpuid rdtscp mfence
I avoid noise from cpuid if possible= mfence cpuid rdtscp target instruction rdtscp cpuid mfence

We use either the prefetchnta or prefetcht2 instructions and a mov instruction
for memory access

38 / 64

Case study: Defeating Windows 7 KASLR

39 / 64Windows 7 Memory layout

I HAL and kernel located bewetween
I
I

start: 0xffff f800 0000 0000
end : 0xffff f87f ffff ffff

I Kernel drivers
I
I

start: 0xffff f880 0000 0000
end : 0xffff f8ff ffff ffff

40 / 64Windows 7 Breaking KASLR

1. Map the drivers address space using translation recovery attack

41 / 64Windows 7 Breaking KASLR

1. Map the drivers address space using translation recovery attack
2. Evict the page translation caches: Sleep() and/or access large memory
buffer41 / 64Windows 7 Breaking KASLR

1. Map the drivers address space using translation recovery attack
2. Evict the page translation caches: Sleep() and/or access large memory
buffer
3. Perform a syscall to targeted driver

41 / 64Windows 7 Breaking KASLR

1. Map the drivers address space using translation recovery attack
2. Evict the page translation caches: Sleep() and/or access large memory
buffer
3. Perform a syscall to targeted driver
4. Time prefetch(PageAddress)

41 / 64Windows 7 Breaking KASLR

1. Map the drivers address space using translation recovery attack
2. Evict the page translation caches: Sleep() and/or access large memory
buffer
3. Perform a syscall to targeted driver
4. Time prefetch(PageAddress)
5. Repeat 2,3,4 for all pages found in 1
Fastest average access time is right address.

41 / 64

Avg. execution time in cycles

Locate Kernel Driver (defeat KASLR)
120
110
100
90
0

4,000
8,000
12,000Page offset in kernel driver region

42 / 64

Case study: Exploiting direct-physical maps

43 / 64

Kernel exploits

I Overwrite return address→ jump to userspace code
I Overwrite stack pointer→ switch to userspace stack

44 / 64

Mitigating kernel exploits

I Jump to userspace code? Nope! Hardware prevents that.

= Supervisor-mode execution prevention (SMEP)I Switch to userspace stack? Nope! Hardware prevents that.

= Supervisor-mode access prevention (SMAP)

45 / 64

Kernel direct-physical map

Physical memory0max. phys.

ct

ap

m

re
di

0

User

247

−247

Kernel−1

Virtual address space

46 / 64

Evading the mitigation

I Get direct-physical-map address of userspace address→ jump/switch there
Known as “ret2dir” attacks Kemerlis et al. 2014

47 / 64

Mitigating the evasion

I Getting rid of direct-physical map?

48 / 64

Mitigating the evasion

I Getting rid of direct-physical map? Apparently not.

→ Do not leak physical addresses to user

48 / 64

Circumventing the mitigation

Prefetching via direct-physical map
I use known address or translation recovery attack to find the

direct-physical-map
I find user mode address in with direct-physical map using

address-translation oracle

49 / 64

Circumventing the mitigation

Min. access latency in cycles

Prefetching via direct-physical map

250
200
150
100

0

20

40

60 80 100 120 140 160 180 200 220 240
Page offset in direct-physical map

50 / 64

Prefetching via direct-physical map

I immediately leaks a direct-physical map address→ no information leak necessary (compared to ret2dir)
I if direct-physical map offset is known→ leaks physical address

51 / 64

Prefetching via direct-physical map

I works on LinuxI works on OSX
I works on Xen PVM (on Amazon EC2)
I does not work on WindowsI

no direct-physical map ;)52 / 64

Cache side-channel attacks

Powerful side channel in the cloudI infer user input
I crypto key recovery
I cross-VM, cross-core, even cross-CPU
I any architecture

53 / 64

Cache mapping

35

physical address

17

6

tag

set

0

oﬀset

30
H

I slice function H unknown

11
2

I reverse-engineered by Hund et al.

2013; Maurice et al. 2015; Inci et al.
2015; Yarom et al. 2015

→ we need the physical addressline
slice 0slice 1slice 2slice 354 / 64

Rowhammer

Rowhammer: yet another attack requiring physical address information
I Rowhammer: bit flip at a random location in DRAM
I exploitable → gain root privileges Seaborn and Dullien 2015

55 / 64

Rowhammer“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice

DRAM bank11111111111111
11111111111111
11111111111111
11111111111111...
11111111111111

row buffer

56 / 64

Rowhammer“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice

DRAM bank11111111111111
activate

11111111111111
11111111111111
11111111111111
...

copy

11111111111111

row buffer

56 / 64

Rowhammer“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice

DRAM bank11111111111111
11111111111111
11111111111111
activate

11111111111111...
11111111111111

copy

row buffer

56 / 64

Rowhammer“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice

DRAM bank11111111111111
activate

11111111111111
11111111111111
11111111111111
...

copy

11111111111111

row buffer

56 / 64

Rowhammer“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice

DRAM bank11111111111111
11111111111111
11111111111111
activate

11111111111111...
11111111111111

copy

row buffer

56 / 64

Rowhammer“It’s like breaking into an apartment by repeatedly slamming a neighbor’s door until the vibrations
open the door you were after” – Motherboard Vice

DRAM bank11111111111111

bit flips in row 2!

11111111111111
10111110101111
11111111111111...
11111111111111

row buffer

56 / 64How is DRAM organized?

57 / 64How is DRAM organized?

channel 0channel 1

57 / 64How is DRAM organized?

back of DIMM: rank 1channel 0
front of DIMM:
rank 0channel 1

57 / 64How is DRAM organized?

back of DIMM: rank 1channel 0
front of DIMM:
rank 0chip
channel 1

57 / 64

DRAM organization example

bank 0I bits in cells in rows

chip
row 0
row 1
row 2
...
row 32767

I access: activate row,

copy to row buffer
I cells leak → refreshnecessary
I cells leak faster uponrow buffer

proximate accesses

58 / 64

DRAM mapping functions on a DDR4 system

BG0
BG1
Rank
BA0
BA1
... 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 ...
Ch.

Again: based on physical addresses

59 / 64

Summary

I Defeat SMAP/SMEP through direct-physical map
I Leak physical addresses
I
I

Perform cache attacks
Perform Rowhammer attacks

60 / 64

Black Hat Sound Bytes.

I CPU Design is security relevant (prefetch leaks significant information)
I We can locate a driver in the kernel and thus break KASLRI We can break SMAP/SMEP and get physical addresses to assist other attacks

61 / 64

Using Undocumented CPU Behaviour to See into Kernel
Mode and Break KASLR in the Process
Anders Fogh and Daniel Gruss

August 4, 201662 / 64

Bonus material

63 / 64

Stronger Kernel Isolation
Today’s operating systems:
Shared address space
User memory

Kernel memory−10context switch

Stronger kernel isolation:
User address space
User memory

context switch

Not mapped0

Not mappedswitchaddr. space0−1Interrupt
dispatcher

Kernel memory

Kernel address space−1

64 / 64

Bibliography I

Hund, Ralf et al. (2013). “Practical Timing Side Channel Attacks against Kernel Space ASLR”. In:
S&P’13.
Inci, Mehmet Sinan et al. (2015). “Seriously, get off my cloud! Cross-VM RSA Key Recovery in a
Public Cloud”. In: Cryptology ePrint Archive, Report 2015/898, pp. 1–15.
Kemerlis, Vasileios P et al. (2014). “ret2dir: Rethinking kernel isolation”. In: USENIX Security
Symposium, pp. 957–972.
Maurice, Clémentine et al. (2015). “Reverse Engineering Intel Complex Addressing Using
Performance Counters”. In: RAID.
Seaborn, Mark and Thomas Dullien (2015). “Exploiting the DRAM rowhammer bug to gain kernel
privileges”. In: Black Hat 2015 Briefings.
Yarom, Yuval et al. (2015). “Mapping the Intel Last-Level Cache”. In: Cryptology ePrint Archive,
Report 2015/905, pp. 1–12.

65 / 64