The bhyve Operator's Manual
Michael Dexter
AsiaBSDCon 2013
OVERVIEW
bhyve is a legacy-free Type-2 hypervisor for FreeBSD that was imported into the mainline
FreeBSD development repository in January of 2013 with svn revision r245652. A hypervisor
allow for the operation of one or more guest operating systems within a host operating system.
As a legacy-free hypervisor, a bhyve host requires the Extended Page Tables (EPT) featurefound on "Nehalem" and newer generations of Intel processors. This requirement eliminates theneed for memory management routines that are traditionally implemented in software and yields
virtually bare metal guest performance. A bhyve guest requires VirtIO network and blockdevices, which were already available in FreeBSD 8-STABLE, 9-STABLE and 10-CURRENT
at the time of bhyve's import. If these two requirements are satisfied, the bhyve host and guests
will operate in the established FreeBSD manner.
HARDWARE REQUIREMENTS
The presence of the Extended Page Table (EPT) feature can be determined by examining thehost's demesg(8) output for the presence of the POPCNT (POP Count) feature as the two arecoupled but not related. Established dynamic memory and storage requirements apply otherwise
with the caveat that there is a 1:1 relationship between the deduction of dynamic memory fromthe host and its allocation to guests.
SOFTWARE REQUIREMENTS
A FreeBSD 10-CURRENT system from svn revision r245652 onward will include all of thenecessary bhyve host components: the vmm(4) kernel module, the libvmmapi library and thebhyveload(8), bhvye(8) and bhyvectl(8) utilities.
A suitable FreeBSD 8-STABLE, 9-STABLE or 10-CURRENT guest can exist in a disk image or
any valid storage device and only requires a modified /etc/ttys entry to work. All otheroptions can be specified at runtime at the loader prompt. Permanent configuration changes
however are generally desired and will be demonstrated.
Permanent /etc/ttys configuration (can be appended):console "/usr/libexec/getty std.9600"

vt100

on secureBoot time or permanent /etc/fstab configuration for a MBR-partitioned device:# Device/dev/vtbd0s1a

Mountpoint/FStypeufs

OptionsrwDump1

Pass#1

FStypeufsOptionsrwDump
1

Pass#1

or for a GPT-partitioned device:# Device/dev/vtbd0p1

Mountpoint/Example runtime or permanent /etc/rc.conf networking configuration:
ifconfig_vtnet0="DHCP"

Depending on how your guest is built, the /boot/loader.conf may require:virtio_load="YES"
if_vtnet_load="YES"
virtio_pci_load="YES"
virtio_blk_load="YES"BHYVE OPERATION
At a minimum, a bhyve host requires that the vmm.ko kernel module be loaded:su kldload vmm

To enable guest networking, load the if_tap.ko kernel module and create a tap(4) interface(tap0 on em0 as an example):su kldload if_tap
su ifconfig tap0 createsu ifconfig bridge0 addm tap0 addm em0 up
su ifconfig tap0 up

The host and guest are now configured for bhyve operation. The bhyve guest bootstrap process isa two-stage operation not unlike that of the host. The bhyveload(8) utility loads the guest
kernel into memory and the bhyve(8) utility executes it. These commands should be run in tightsequence with identical memory, storage device and guest name parameters.
All configuration parameters forward will be highly site-specific.

su /usr/sbin/bhyveload -m 256 -M 0 -d mydiskimage myguestname
su /usr/sbin/bhyve -a -A -m 256 -M 0 -I -H -g 0 \-s 0:0,hostbridge \-s 1:0,virtio-net,tap0 \-s 2:0,virtio-blk,mydiskimage \-S 31,uart,stdio \myguestname

Assuming a successful boot, the guest should behave like a standard FreeBSD system and'shutdown -p now' will cleanly shut it down.
If you wish to terminate a guest at the loader prompt, simply type 'quit'.
Executing guests can be viewed by name on the host system with:su ls /dev/vmm

The above bootstrap procedure will not free the memory allocated to the guest but it can be freed
using bhyvectl(8):su /usr/sbin/bhyvectl --vm=myguestname --destroy

PRAGMATIC BHYVE GUEST OPERATION
The above procedures do not address the subjective matter of populating and configuring theguest userland or automating the guest operation lifecycle.
Two efforts exist to automate the building and operation of bhyve guests.
bhyve developer Neel Natu provides a script named 'vmrun.sh' that works in conjunction with aFreeBSD 10-CURRENT installation iso image named 'release.iso' to automate the creationof a bootable disk image named 'diskdev' that can be populated using the standard FreeBSD
installer. This script provides an 8G disk image and 512M of dynamic memory by default but
these parameters can be easily modified at runtime or by modifying the script.
The 'vmrun.sh' script requires only a guest name such as 'vm1':
su vmrun.sh vm1

Mr. Natu's script and instructions can be obtained from:http://people.freebsd.org/~neel/bhyve/vmrun.sh
http://people.freebsd.org/~neel/bhyve/bhyve_instructions.txt

A suitable 10-CURRENT 'release.iso' image can be retrieved from:http://ftp4.us.freebsd.org/pub/FreeBSD/snapshots/amd64/amd64/ISO-IMAGES/10.0/

Note that the versioning prefix must be removed for the iso to work.
Alternatively, the bhyve.org web site provides a series of scripts that facilitate the creation andoperation of bhyve guests in a highly-customizable manner.
While evolving, at the time of writing these scripts include:
0-make-softdevice.sh
1-format-device.sh
1-format-zvol-gpt.sh
2-install-guest.sh
3-host-prep.sh4-boot-guest.sh
5-cleanup-guests.sh
mount-diskdev.shCreate and mount a disk image or zvol
Format the disk image or zvol
Format a zfs zvol with a GPT layout
Populate the disk image with the OS
Prepare the host for bhyve and networking
Boot the guest
Clean up after the guest
Mount a specified disk image

In addition to the basic functionality provided by 'vmrun.sh', these scripts create and format diskimages in MBR and GPT format, format zvol, iSCSI and hardware devices, populate FreeBSD
8-STABLE, 9-STABLE and 10-CURRENT userlands from the official mirrors and facilitateguest operation.
Any and all jail(8) and NanoBSD configuration strategies should be applicable to bhyve andthe creation of "thick" and "thin" bhyve hosts and guests.
THE BHYVE DEVELOPMENT ENVIRONMENT
As a native FreeBSD 8-STABLE and later on FreeBSD 10-CURRENT computing environment,
bhyve provides the ideal infrastructure for all categories of FreeBSD development. The hardpartitioning provided by the vmm virtual machine manager allows the "unstable" FreeBSD 10CURRENT environment to be used for "stable" FreeBSD 8 and 9 development with far fewer
potential risks than emulated or jailed environments. Common development nuisances such as
library incompatibilities are eliminated by the virtual machine divide.
The inclusion of dtrace(1M) in FreeBSD 10-CURRENT provides an unprecedented level ofsystem introspection that should prove invaluable to FreeBSD, bhyve and future foreign guest
operating systems.

Some rudimentary DTrace examples:su kldload dtraceallsu dtrace -n 'io:::start /execname == "bhyve"/ { @[ustack()] = count(); }'libc.so.7`0x800ed3fda
bhyve`pci_vtblk_qnotify+0x59
bhyve`pci_vtblk_write+0x220
bhyve`pci_emul_io_handler+0x16e
bhyve`emulate_inout+0x197
bhyve`vmexit_inout+0x155
bhyve`vm_loop+0x118
bhyve`fbsdrun_start_thread+0x87
libthr.so.3`0x800c53413
505

The sysutils/DTraceToolkit port provides FreeBSD-specific DTrace scripts.
su /usr/share/dtrace/toolkit/hotuser -p `pgrep -n bhyve`
Sampling... Hit Ctrl-C to end.
^CFUNCTION
bhyve`vm_loop
libvmmapi.so.5`0x800838920
libthr.so.3`0x800c59491
libthr.so.3`0x800c58df1
libvmmapi.so.5`0x800838950
libthr.so.3`0x800c5992f
libc.so.7`0x800f83bb3
bhyve`emulate_inout
libc.so.7`0x800ed3fd0
libthr.so.3`0x800c594c1
bhyve`pci_vtblk_write
bhyve`pci_vtblk_proc
libvmmapi.so.5`0x800838a78
libvmmapi.so.5`0x800838a2ebhyve`vmexit_inout
libc.so.7`0x800f8875a

COUNT
11111111111
12
5631

PCNT
1.8%
1.8%
1.8%
1.8%
1.8%
1.8%
1.8%
1.8%
1.8%
1.8%
1.8%
1.8%
3.6%
8.9%
10.7%
55.4%

su /usr/share/dtrace/toolkit/hotkernel
Sampling... Hit Ctrl-C to end.
^CFUNCTION
vmm.ko`vm_guest_msrs
vmm.ko`vcpu_set_state
vmm.ko`vmx_setreg
vmm.ko`vm_nmi_pending
vmm.ko`vm_get_registervmm.ko`lapic_intr_accepted
vmm.ko`vm_lapic
vmm.ko`vlapic_op_mem_read
vmm.ko`lapic_mmio_write
vmm.ko`vlapic_intr_accepted
vmm.ko`vm_set_registervmm.ko`lapic_pending_intr
vmm.ko`vmm_fetch_instruction
vmm.ko`vmmdev_ioctl
vmm.ko`vmmdev_ioctl
vmm.ko`vm_exitinfo
vmm.ko`vcpu_stats
vmm.ko`vmm_emulate_instruction
vmm.ko`vmx_getreg
vmm.ko`lapic_timer_tick
vmm.ko`vm_gpa2hpa
vmm.ko`vlapic_pending_intr
vmm.ko`vlapic_op_mem_write
vmm.ko`vmm_decode_instruction
vmm.ko`vmcs_read
vmm.ko`ept_vmmmap_get
vmm.ko`vlapic_timer_tick
vmm.ko`vm_run
vmm.ko`restore_guest_msrs
vmm.ko`restore_host_msrs
vmm.ko`vlapic_update_ppr
vmm.ko`vmx_run
kernel`acpi_cpu_c1

COUNT
1
1
1
1
1
1
1
2
2
2
2
3
3
3
3
3
3
4
4
5
7
7
7
19
20
21
2930
32
42142
33275
168750

PCNT
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.1%
16.3%
82.6%

In addition to the DTrace suite, the bhyvectl(8) command provides extensive bhyve-specificprofiling information:su bhyvectl --get-lowmem --vm=guest0
lowmem0x0000000000000000/268435456
su bhyvectl --get-stats --vm=guest0
vcpu0
number of ticks vcpu was idle
number of NMIs delivered to vcpu
vcpu migration across host cpus
vm exits due to external interrupt
number of times hlt was ignored
number of times hlt was intercepted
vcpu total runtime

524409701186676
222974202158532
288974908299

BHYVE REFERENCE
The following options are available to the bhyveload(8) and bhyve(8) commands:
Usage: bhyveload [-d <disk image path>] [-h <host filesystem path>] [-m<lowmem>] [-M <highmem>] <vmname>
Usage: bhyve [-aehABHIP] [-g <gdb port>] [-z <hz>] [-s <pci>] [-S <pci>][-p pincpu] [-n <pci>] [-m lowmem] [-M highmem] <vm>
-a: local apic is in XAPIC mode (default is X2APIC)-A: create an ACPI table-g: gdb port (default is 6466 and 0 means don't open)-c: # cpus (default 1)-p: pin vcpu 'n' to host cpu 'pincpu + n'
-B: inject breakpoint exception on vm entry-H: vmexit from the guest on hlt-I: present an ioapic to the guest-P: vmexit from the guest on pause-e: exit on unhandled i/o access-h: help-z: guest hz (default is 100)-s: <slot,driver,configinfo> PCI slot config-S: <slot,driver,configinfo> legacy PCI slot config-m: lowmem in MB-M: highmem in MB-x: mux vcpus to 1 hcpu
-t: mux vcpu timeslice hz (default 200)

FUTURE DEVELOPMENTS
bhyve contains experimental PCI device pass-through support and is scheduled to include:• AMD Virtualization Extensions• Foreign Guest Operating System Support• ACPI Guest Suspend and Resume• Thin Provisioning of Memory• Generalization of CPUID Features for Guest Migratability• Sparse Image Support such as QCOW, VDI and VMDK• Porting to other Host Operating Systems
bhyve is arguably the most compelling FreeBSD development since jail(8) and continues
FreeBSD's tradition of providing innovative multiplicity options to operators and developers.
jail(8) and bhyve are by no means mutually-exclusive technologies and should provide value
when used in conjunction with one another in parallel or via encapsulation. bhyve also promises
to make the most of recent FreeBSD features such as the DTrace, the ZFS filesystem andFreeBSD’s virtual network stack. The fundamental point to remember about bhyve is that aFreeBSD bhyve system is simply a FreeBSD system that will fully leverage your FreeBSD
administration and development experience.