Xenpwn – Breaking Paravirtualized Devices
Felix Wilhelm
1#whoamio Security Researcher @ ERNW Researcho Application and Virtualization Securityo Recent Research
Security Appliances (Palo Alto, FireEye)o Hypervisors
oo @_fel1x on twitter

2

Agendao Device Virtualization & Paravirtualized Devices
o Double Fetch Vulnerabilities
o Xenpwn: Architecture and Designo Results
o Case Study: Exploiting xen-pciback3

Device Virtualizationo Virtualized systems need access to virtualdevices
oDisk, Network, Serial, ...

o Traditionally: Device emulation
Emulate old and well supported hardwaredevices
o Guest OS does not need special drivers
o Installation with standard installation sources
supportedo4

Paravirtualized Devices
o Most important downsides of emulated devices:
Hard to implement securely and correctlyo Slow performanceo No support for advanced features
oo Solution: Paravirtualized Devices
Specialized device drivers for use in virtualized systems
o Idea: Emulated devices are only used as fallback mechanism
o Used by all major hypervisorso Not the same as Xen paravirtualized domains!
o5Paravirtualized Devices
o Split Driver Model
Frontend runs in Guest systemo Backend in Host/Management domain
oo Terminology differs between hypervisors
VSC / VSP in Hyper-V
o Virtio devices and drivers
oo Implementations are quite similar
6

Paravirtualized Devices
o PV devices are implemented on top of sharedmemoryGreat Performanceo Easy to implemento Zero copy algorithms possible
oo Message protocols implemented on topoXen, Hyper-V and KVM all use ring bufferso Shared memory mappings can be constant orcreated on demand7

Security of PV Devices
o Backend runs in privileged context è Communication between
frontend and backend is trust boundaryo Low level code + Protocol parsing è Bugso Examples
ooHeap based buffer overflow in KVM disk backend (CVE-2011-1750)
Unspecified BO in Hyper-V storage backend (CVE-2015- 2361)

o Not as scrutinized as emulated devices
Device and hypervisor specific protocols
o Harder to fuzzo8Very Interesting Target
o Device emulation often done in user space ßà PV backend often
in kernel for higher performanceoCompromise of kernel backend is instant win Jo PV devices are becoming more importantMore device types (USB, PCI pass-through, touch screens, 3Dacceleration)o More features, optimizations
oo Future development: Removal of emulated devices
osee Hyper-V Gen2 VMs
9Research Goalo ”Efficient vulnerability discovery in Paravirtualized Devices”
o Core Idea: No published research on the use of shared memoryin the context of PV deviceso Bug class that only affect shared memory? è Double fetches!

10

Double Fetch Vulnerabilities
o Special type of TOCTTOU bug affecting shared memory.
o Simple definition: Same memory address is accessed multiple
times with validation of the accessed data missing on at least one
accesso Can introduce all kinds of vulnerabilities
Arbitrary Write/Reado Buffer overflows
o Direct RIP control Jo11

Double Fetch Vulnerabilities
o Term “double fetch” was coined by Fermin J. Serna in 2008
oBut bug class was well known before thato Some interesting research published in 2007/2008
Usenix 2007 “Exploiting Concurrency Vulnerabilities in System CallWrappers” - Robert N. M. Watson
o CCC 2007: “From RING 0 to UID 0” and Phrack #64 file 6 – twiz,sgrakkyu
oo First example I could find is sendmsg() linux bug reported in 2005
oHappy to hear about more J
12

Example:sendmsg()

13

Bochspwno “Identifying and Exploiting Windows Kernel
Race Conditions via Memory AccessPatterns” (2013)
oby j00ru and Gynvael Coldwind

o Uses extended version of Bochs CPU
emulator to trace all memory access fromkernel to user space.

14

Bochspwno Resulted in significant number of Windows bugs (and a welldeserved Pwnie)
obut not much published follow-up researcho Whitepaper contains detailed analysis on exploitability of doublefetches
oOn multi core system even extremely short races are exploitableo Main inspiration for this research.
15

Example:Bochspwn

16

Xenpwno Adapt memory access tracing approach used by Bochspwn foranalyzing PV device communication.
o Why not simply use Bochspwn?
Extremely slow
o Passive overhead (no targeted tracing)o Compatibility issueso Dumping traces to text files does not scale
oo Idea: Implement memory access tracing on top of hardware
assisted virtualization

17

18

Xenpwn Architectureo Nested virtualizationoTarget hypervisor (L1) runs on top of base
hypervisor (L0)o Analysis components run in user space of L1
management domain.
No modification to hypervisor requiredo Bugs in these components do not crash wholesystem
oo L0 hypervisor is Xen
19

libVMIo Great library for virtual machine
introspection (VMI)
Hypervisor agnostic (Xen and KVM)
o User-space wrapper around hypervisor APIs
oo Allows access to and manipulation of guest
state (memory, CPU registers)o Xen version supports memory events
20

libVMI Memory Eventso Trap on access to a guest physical addresso Implemented on top of Extended Page Tables(EPT)
Disallow access to GPA
o Access triggers EPT violation and VM exito VM exit is forwarded to libvmi handlero21Memory Access Tracing withlibvmi
1.
2.
3.
4.

Find shared memory pages
Register memory event handlers
Analyze memory event, extract needed
information and store in trace storage.
Run analysis algorithms (can happen muchlater)

22

Trace Collectoro Use libvmi to inspect memory and identify shared memory pages
Target specific code.
o Identify data structures used by PV frontend/backend and addressesof shared pages
oo Registers memory event handlerso Main work is done in callback handler
oDisassemble instructions using Capstone23

Callback handler24

Trace Storageo Storage needs to be fast and persistent
Minimize tracing overheado Allow for offline analysis
oo Nice to have: Efficient compression
oAllows for very long traceso Tool that fulfills all these requirements: Simutrace
osimutrace.org

25

Simutraceo Open source project by the Operation System
Group at the Karlsruhe Institute of Technology
o Designed for full system memory tracing
oAll memory accesses including their content

oHighly efficient communication over sharedmemory pages

o C++ daemon + client libraryo Uses specialized compression algorithm
optimized for memory tracesoHigh compression rate + high speedo Highly recommended!

26

Trace Entries
For every memory access:For every unique instruction:27

Double Fetch
Algorithm

Simplified version

28

Advantages & Limitations
o Good:Low passive overheado Largely target independento only Trace collector requires adaptiono Easy to extend and develop
oo BadHigh active overheado VM exits are expensiveo Reliance on nested virtualizationo29

Nested Virtualization on Xen
o Xen Doku: Nested HVM on Intel CPUs, as of Xen 4.4, is considered"tech preview". For many common cases, it should work reliably
and with low overheado Reality:
Xen on Xen works
o KVM on Xen works (most of the time)o Hyper-V on Xen does not work Lo30

Results
o KVM: no security critical double fetches
Main reason seems to be endian independent memory wrappers
o .. but discovered other interesting issues while reading the virtio code;)
oo bhyve: one very interesting result
oOngoing disclosure process

o Xen: Three interesting double fetches
31

QEMU xen_disk
Normally not
exploitable thanks
to compiler
optimizations
32

xen-blkback
OOB Read/Write

33

xen-pciback34

xen-pciback: xen_pcibk_do_op35

xen-pcibacko switch statement is compiled into jump tableo op->cmd == $r13+0x4
o Points into shared memoryo Range check and jump use two different
memory accesseso Valid compiler optimization
oop is not marked as volatile36

Exploiting pcibacko Race is very small: 2 Instructions
oBut can be reliably won if guest VM has
multiple coreso Lost race does not have any negative sideeffects
oInfinite retries possibleo Simple to trigger
oSend PCI requests while flipping value using
XOR
37

Exploiting pcibacko Indirect jump è No immediate RIP control
oNeed to find reliable offset to function pointero Load address of xen-pciback.ko is randomo Virtual address of backend mapping also not knowno A lot of similarities to a remote kernel exploito Chosen approach: Trigger type confusion to get write primitiv

38

Type Confusiono Second jump table generated for xen-pciback
oAlmost directly behind the jump table
generated for vulnerable functiono XenbusStateInitialized uses value of r13
register as first argument
Should be a pointer to a xen_pcibk_device
structureo Is a pointer to the start of the shared memory
page Jo39

Getting a write primitveo xen_pcibk_attach first tries to lock thedev_lock mutex of referenced structure.
o Gives us the possibility to call mutex_lockwith a fake mutex structure
o mutex_lock
Fastpath: Switch lock count from 1 -> 0o Slowpath: Triggered when lock count != 1o40

Getting a write primitive:
mutex_lock slowpath
1.

mutex_optimistic_spin needs to fail.
oCan be achieved by setting lock->owner to areadable zero page2.

If lock count still not 1, mutex_waiterstructure is created and stored on stack
3. mutex_waiter structure is added to lock>wait_list and kernel thread goes to sleep
till wake up.
è Pointer to waiter is written to attacker
controlled location.
41

Write Primitiveo write-where but not write-what
Pointer to pointer to attacker controlled datao Can‘t simply overwrite function pointers
oo One shot
opciback is locked due to xen_pcibk_do_op
never returningo Idea: Add faked entries to a global linked list.
oRequires known kernel version + no KASLR orinfoleak
42

fake_prev
controlled data

list_head.next

entry1.next

entry2.next

list_head.prev

entry1.prev

entry2.prev

43

fake_prev
controlled data
next
prev
waiterlist_head.next
list_head.prev
44

Overwrite target
o Global data structure
oNeed to know address of list_heado No new elements should be attached during run time
olist_head.prev is not changed, new entry might be added directly
behind list_heado Needs to survive one “junk“ entryoNo full control over waiter structure / stack frame

45

46

fs/exec.c: formats
o formats linked list contains entries for different file formats
supported by exec
ELF
o #! shell scripts
o a.out format
oo Walked every time exec* syscall is called to load input file.
o waiter entry is skipped because try_module_get function fails
47

Getting Code Execution
o Set address of load_binary pointer to stack pivot
o ROP chain to allocate executable memory and copy shellcode
ovmalloc_exec + memcpy

o Restore original formats listo $shellcode
o Return to user space48

Demo49

Open Source
o Xenpwn open source release:ohttps://github.com/felixwilhelm/xenpwn

o Whitepaper contains a lot more technical details
Implementation detailso Performance evaluation
o ...
o50

Future Worko Use Xenpwn against Hyper-V and VMWareoRequires improved support for nested virtualizationo Identify and analyze other shared memory trust boundaries
oSandboxes?

o What types of bugs can we find with full memory traces?

51Thanks for your Attention!

Q&A
52