kvm: the Linux Virtual Machine Monitor
Avi Kivity
Qumranet

Yaniv Kamay
Qumranet

Dor Laor
Qumranet

avi@qumranet.com

yaniv@qumranet.com

dor.laor@qumranet.com

Uri Lublin
Qumranet

Anthony Liguori
IBMuril@qumranet.com

aliguori@us.ibm.com

Abstract
Virtualization is a hot topic in operating systems thesedays. It is useful in many scenarios: server consolidation, virtual test environments, and for Linux enthusiasts
who still can not decide which distribution is best. Recently, hardware vendors of commodity x86 processors
have added virtualization extensions to the instruction
set that can be utilized to write relatively simple virtual
machine monitors.

when executed in user mode, e.g. popf. Some privileged
state is difficult to hide, e.g. the current privilege level,
or cpl.
Recognizing the importance of virtualization, hardware
vendors [Intel][AMD] have added extensions to the x86
architecture that make virtualization much easier. Whilethese extensions are incompatible with each other, they
are essentially similar, consisting of:

The Kernel-based Virtual Machine, or kvm, is a newLinux subsystem which leverages these virtualization
extensions to add a virtual machine monitor (or hypervisor) capability to Linux. Using kvm, one can create
and run multiple virtual machines. These virtual machines appear as normal Linux processes and integrate
seamlessly with the rest of the system.

1• Hardware state switch – when switching to guest
mode and back, the hardware switches the control
registers that affect processor operation modes, aswell as the segment registers that are difficult toswitch, and the instruction pointer so that a control
transfer can take effect.

Background

Virtualization has been around almost as long as computers. The idea of using a computer system to emulate
another, similar, computer system was early recognized
as useful for testing and resource utilization purposes.
As with many computer technologies, IBM led the way
with their VM system. In the last decade, VMware’s
software-only virtual machine monitor has been quite
successful. More recently, the Xen [xen] open-source
hypervisor brought virtualization to the open source
world, first with a variant termed paravirtualization andas hardware became available, full virtualization.

2• A new guest operating mode – the processor canswitch into a guest mode, which has all the regular privilege levels of the normal operating modes,
except that system software can selectively request that certain instructions, or certain register
accesses, be trapped.

x86 Hardware Virtualization Extensions

x86 hardware is notoriously difficult to virtualize. Some
instructions that expose privileged state do not trap

• Exit reason reporting – when a switch from guest
mode back to host mode occurs, the hardware reports the reason for the switch so that software cantake the appropriate action.

3

General kvm Architecture

Under kvm, virtual machines are created by opening adevice node (/dev/kvm.) A guest has its own memory,
separate from the userspace process that created it. A
virtual cpu is not scheduled on its own, however.

• 225 •

226 • kvm: the Linux Virtual Machine Monitor
User address space

Guest address space
User mode

Guest memory
mapped to user
process

Kernel mode

Guest mode

Issue Guest
Execution ioctl

Enter
Guest
Mode
Execute natively
in Guest Mode

HandleExit

KernelMemoryassigned
to Guest

Figure 1: kvm Memory Map
3.1/dev/kvm Device Node

kvm is structured as a fairly typical Linux character device. It exposes a /dev/kvm device node which canbe used by userspace to create and run virtual machines
through a set of ioctl()s.
The operations provided by /dev/kvm include:• Creation of a new virtual machine.
• Allocation of memory to a virtual machine.
• Reading and writing virtual cpu registers.
• Injecting an interrupt into a virtual cpu.
• Running a virtual cpu.
Figure 1 shows how guest memory is arranged. Like
user memory in Linux, the kernel allocates discontiguous pages to form the guest address space. In addition,
userspace can mmap() guest memory to obtain directaccess. This is useful for emulating dma-capable devices.
Running a virtual cpu deserves some further elaboration. In effect, a new execution mode, guest mode isadded to Linux, joining the existing kernel mode anduser mode.
Guest execution is performed in a triply-nested loop:

Yes

I/O?
No

Handle I/O

Yes

SignalPending?
No

Figure 2: Guest Execution Loop
• At the outermost level, userspace calls the kernel
to execute guest code until it encounters an I/O instruction, or until an external event such as arrivalof a network packet or a timeout occurs. External
events are represented by signals.
• At the kernel level, the kernel causes the hardware
to enter guest mode. If the processor exits guest
mode due to an event such as an external interrupt
or a shadow page table fault, the kernel performs
the necessary handling and resumes guest execution. If the exit reason is due to an I/O instruction
or a signal queued to the process, then the kernel
exits to userspace.
• At the hardware level, the processor executes guest
code until it encounters an instruction that needs
assistance, a fault, or an external interrupt.

Refer to Figure 2 for a flowchart-like representation of
the guest execution loop.

2007 Linux Symposium, Volume One • 227
3.2

Reconciling Instruction Set Architecture Differences

Unlike most of the x86 instruction set, where instruction set extensions introduced by one vendor are adopted
by the others, hardware virtualization extensions are notstandardized. Intel and AMD processors have different
instructions, different semantics, and different capabilities.
kvm handles this difference in the traditional Linuxway of introducing a function pointer vector, kvm_
arch_ops, and calling one of the functions it defines whenever an architecture-dependent operation is
to be performed. Base kvm functionality is placed in amodule, kvm.ko, while the architecture-specific functionality is placed in the two arch-specific modules,
kvm-intel.ko and kvm-amd.ko.

4

Virtualizing the MMU

As with all modern processors, x86 provides a virtual
memory system which translates user-visible virtual addresses to physical addresses that are used to access thebus. This translation is performed by the memory management unit, or mmu. The mmu consists of:• A radix tree ,the page table, encoding the virtualto-physical translation. This tree is provided bysystem software on physical memory, but is rooted
in a hardware register (the cr3 register)• A mechanism to notify system software of missing
translations (page faults)• An on-chip cache (the translation lookaside buffer,or tlb) that accelerates lookups of the page table• Instructions for switching the translation root in order to provide independent address spaces• Instructions for managing the tlb
The hardware support for mmu virtualization provides
hooks to all of these components, but does not fully virtualize them. The principal problem is that the mmu
provides for one level of translation (guestvirtual !
guest physical) but does not account for the second level required by virtualization (guest physical !
host physical.)

The classical solution is to use the hardware virtualization capabilities to present the real mmu with a separate page table that encodes the combined translation (guestvirtual ! host physical) while emulating the
hardware’s interaction with the original page table provided by the guest. The shadow page table is built incrementally; it starts out empty, and as translation failures
are reported to the host, missing entries are added.
A major problem with shadow page tables is keeping
the guest page table and the shadow page table synchronized. Whenever the guest writes to a page table, thecorresponding change must also be performed on the
shadow page table. This is difficult as the guest page table resides in ordinary memory and thus is not normallytrapped on access.
4.1

Virtual TLB Implementation

The initial version of shadow page tables algorithm in
kvm used a straightforward approach that reduces theamount of bugs in the code while sacrificing performance. It relies on the fact that the guest must use the tlb
management instructions to synchronize the tlb with its
page tables. We trap these instructions and apply theireffect to the shadow page table in addition to the normal
effect on the tlb.
Unfortunately, the most common tlb management instruction is the context switch, which effectively invalidates the entire tlb.1 This means that workloads
with multiple processes suffer greatly, as rebuilding the
shadow page table is much more expensive than refilling
the tlb.
4.2

Caching Virtual MMU

In order to improve guest performance, the virtual mmu
implementation was enhanced to allow page tables tobe cached across context switches. This greatly increases performance at the expense of much increased
code complexity.
As related earlier, the problem is that guest writes to theguest page tables are not ordinarily trapped by the virtualization hardware. In order to receive notifications ofsuch guest writes, we write-protect guest memory pages
that are shadowed by kvm. Unfortunately, this causes a
chain reaction of additional requirements:
1 Actually, kernel mappings can be spared from this flush; but theperformance impact is nevertheless great.

228 • kvm: the Linux Virtual Machine Monitor• To write protect a guest page, we need to know
which translations the guest can use to write to thepage. This means we need to keep a reverse mapping of all writable translations that point to each
guest page.
• When a write to a guest page table is trapped, weneed to emulate the access using an x86 instruction
interpreter so that we know precisely the effect on
both guest memory and the shadow page table.
• The guest may recycle a page table page into a normal page without a way for kvm to know. This cancause a significant slowdown as writes to that page
will be emulated instead of proceeding at native
speeds. kvm has heuristics that determine whensuch an event has occurred and decache the corresponding shadow page table, eliminating the needto write-protect the page.

In kvm, I/O virtualization is performed by userspace.
All pio and mmio accesses are forwarded to userspace,which feeds them into a device model in order to simulate their behavior, and possibly trigger real I/O such astransmitting a packet on an Ethernet interface. kvm also
provides a mechanism for userspace to inject interrupts
into the guest.
5.2

kvm also provides interrupt injection facilities touserspace. Means exist to determine when the guest
is ready to accept an interrupt, for example, the interrupt flag must be set, and to actually inject the interrupt
when the guest is ready. This allows kvm to emulate the
APIC/PIC/IOAPIC complex found on x86-based systems.
5.3

At the expense of considerable complexity, these requirements have been implemented and kvm context
switch performance is now reasonable.

5

I/O Virtualization• Bandwidth – framebuffers typically see very highbandwidth transfers. This is in contrast to typical devices which use mmio for control, but transfer the bulk of the data with direct memory access(dma).
• Memory equivalence – framebuffers are mostly justmemory: reading from a framebuffers returns thedata last written, and writing data does not cause
an action to take place.

Virtualizing Guest-Initiated I/O Instructions

Trapping pio is quite straightforward as the hardware
provides traps for pio instructions and partially decodes
the operands. Trapping mmio, on the other hand, is quite
complex, as the same instructions are used for regularmemory accesses and mmio:• The kvm mmu does not create a shadow page tabletranslation when an mmio page is accessed• Instead, the x86 emulator executes the faulting instruction, yielding the direction, size, address, andvalue of the transfer.

Virtualizing Framebuffers

An important category of memory-mapped I/O devices
are framebuffers, or graphics adapters. These have characteristics that are quite distinct from other typical mmio
devices:

Software uses programmed I/O (pio) and memorymapped I/O (mmio) to communicate with hardware devices. In addition, hardware can issue interrupts to request service by system software. A virtual machinemonitor must be able to trap and emulate pio and mmio
requests, and to simulate interrupts from virtual hardware.
5.1

Host-Initiated Virtual Interrupts

In order to efficiently support framebuffers, kvm allows
mapping non-mmio memory at arbitrary addresses suchas the pci region. Support is included for the VGA windows which allow physically aliasing memory regions,and for reporting changes in the content of the framebuffer so that the display window can be updated incrementally.

6

Linux Integration

Being tightly integrated into Linux confers some important benefits to kvm:

2007 Linux Symposium, Volume One • 229• On the developer level, there are many opportunities for reusing existing functionality within the
kernel, for example, the scheduler, NUMA support,
and high-resolution timers.
• On the user level, one can reuse the existing Linuxprocess management infrastructure, e.g., top(1)to look at cpu usage and taskset(1) to pin
virtual machines to specific cpus. Users can use
kill(1) to pause or terminate their virtual machines.

7

Live Migration

One of the most compelling reasons to use virtualization is live migration, or the ability to transport a virtual
machine from one host to another without interrupting
guest execution for more than a few tens of milliseconds. This facility allows virtual machines to be relocated to different hosts to suit varying load and performance requirements.
Live migration works by copying guest memory to thetarget host in parallel with normal guest execution. If aguest page has been modified after it has been copied,
it must be copied again. To that end, kvm provides adirty page log facility, which provides userspace with abitmap of modified pages since the last call. Internally,
kvm maps guest pages as read-only, and only maps themfor write after the first write access, which provides ahook point to update the bitmap.
Live migration is an iterative process: as each passcopies memory to the remote host, the guest generates
more memory to copy. In order to ensure that the process converges, we set the following termination criteria:• Two, not necessarily consecutive, passes weremade which had an increase in the amount of memory copied compared to previous pass, or,8

8.1

Guest SMP Support

Demanding workloads require multiple processing
cores, and virtualization workloads are no exception.
While kvm readily supports SMP hosts, it does not yet
support SMP guests.
In the same way that a virtual machine maps to a hostprocess under kvm, a virtual cpu in an SMP guest maps
to a host thread. This keeps the simplicity of the kvm
model and requires remarkably few changes to implement.
8.2

Paravirtualization

I/O is notoriously slow in virtualization solutions. Thisis because emulating an I/O access requires exiting
guest mode, which is a fairly expensive operation compared to real hardware.
A common solution is to introduce paravirtualized devices, or virtual “hardware” that is explicitly designed
for virtualized environments. Since it is designedwith the performance characteristics of virtualization inmind, it can minimize the slow operations to improve
performance.
8.3

Memory Management Integration

Linux provides a vast array of memory management
features: demand paging, large pages (hugetlbfs), andmemory-mapped files. We plan to allow a kvm guest
address space to directly use these features; this can enable paging of idle guest memory to disk, or loading aguest memory image from disk by demand paging.

• Thirty iterations have elapsed.

8.4

Future Directions

Currently, the Linux scheduler has no knowledge that it
is scheduling a virtual cpu instead of a regular thread.
We plan to add this knowledge to the scheduler so thatit can take into account the higher costs of moving avirtual cpu from one core to another, as compared to aregular task.

While already quite usable for many workloads, manythings remain to be done for kvm. Here we describe themajor features missing; some of them are already workin-progress.

Scheduler Integration

230 • kvm: the Linux Virtual Machine Monitor
8.5

New Hardware Virtualization Features

Virtualization hardware is constantly being enhancedwith new capabilities, for example, full mmu virtualization, a.k.a. nested page tables or extended page tables,or allowing a guest to securely access a physical device[VT-d]. We plan to integrate these features into kvm inorder to gain the performance and functionality benefits.
8.6

Additional Architectures

kvm is currently only implemented for the i386 andx86-64 architectures. However, other architectures
such as powerpc and ia64 support virtualization, andkvm could be enhanced to support these architectures as
well.

9

Conclusions

kvm brings an easy-to-use, fully featured integrated virtualization solution for Linux. Its simplicity makes extending it fairly easy, while its integration into Linux
allows it to leverage the large Linux feature set and thetremendous pace at which Linux is evolving.

10

References[qemu] Bellard, F. (2005). Qemu, a Fast and Portable
Dynamic Translator. In Usenix annual technicalconference.
[xen] Barham P., et al. Xen and the art of virtualization.
In Proc. SOSP 2003. Bolton Landing, New York,U.S.A. Oct 19-22, 2003.
[Intel] Intel Corp. IA-32 Intel R Architecture Software
Developer’s Manual, Volume 3B: System Programming
Guide, Part 2. Order number 25366919.
[AMD] AMD Inc. AMD64 Architecture Programmer’s
Manual Volume 2: System Programming.
[VT-d] Abramson, D.; Jackson, J.; Muthrasanallur, S.;
Neiger, G.; Regnier, G.; Sankaran, R.; Schoinas, I.;
Uhlig, R.; Vembu, B.; Wiegert, J. Intel R Virtualization
Technology for Directed I/O. Intel Technology Journal.
http://www.intel.com/technology/itj/
2006/v10i3/ (August 2006).

Proceedings of theLinux Symposium
Volume One

June 27th–30th, 2007
Ottawa, Ontario
CanadaConference Organizers
Andrew J. Hutton, Steamballoon, Inc., Linux Symposium,Thin Lines Mountaineering
C. Craig Ross, Linux Symposium

Review Committee
Andrew J. Hutton, Steamballoon, Inc., Linux Symposium,Thin Lines Mountaineering
Dirk Hohndel, Intel
Martin Bligh, Google
Gerrit Huizenga, IBM
Dave Jones, Red Hat, Inc.
C. Craig Ross, Linux Symposium

Proceedings Formatting TeamJohn W. Lockhart, Red Hat, Inc.
Gurhan Ozen, Red Hat, Inc.
John Feeney, Red Hat, Inc.
Len DiMaggio, Red Hat, Inc.
John Poelstra, Red Hat, Inc.