Partitioning a hypervisor into virtual hypervisors

In an example, a computer system includes a hardware platform and a hypervisor executing on the hardware platform. The hypervisor includes a kernel and a plurality of user-space instances within a user-space above the kernel. Each user-space instance is isolated from each other user-space instance through namespaces. Each user-space instance includes resources confined by hierarchical resource groups. The computer system includes a plurality of virtual hypervisors, where each virtual hypervisor executes in a respective user-space instance of the plurality of user-space instances.

BACKGROUND

Computer virtualization is a technique that involves encapsulating a physical computing machine platform into virtual machine(s) executing under control of virtualization software on a hardware computing platform or “host.” A virtual machine provides virtual hardware abstractions for processor, memory, storage, and the like to a guest operating system. The virtualization software, also referred to as a “hypervisor,” includes one or more virtual machine monitors (VMMs) to provide execution environment(s) for the virtual machine(s). As physical hosts have grown larger, with greater processor core counts and terabyte memory sizes, virtualization has become key to the economic utilization of available hardware.

Conventional hypervisors are designed to be managed by a single entity. While such hypervisors support multiple users with restricted permissions, they do not support isolated management by multiple independent entities. Users with sufficient privileges can observe host-wide state and all resources are visible to any connected client. While significant effort has focused on ensuring isolation among virtual machines on a physical host, there are few options today for isolating management within the hypervisor on the physical host.

SUMMARY

One or more embodiments provide techniques for partitioning a hypervisor into virtual hypervisors. In an embodiment, a computer system includes a hardware platform and a hypervisor executing on the hardware platform. The hypervisor includes a kernel and a plurality of user-space instances within a user-space above the kernel. Each user-space instance is isolated from each of the other user-space instances through namespaces. Each user-space instance includes resources confined by hierarchical resource groups. The computer system includes a plurality of virtual hypervisors, where each virtual hypervisor executes in a respective user-space instance of the plurality of user-space instances.

A method of creating a tenant in a multi-tenant hypervisor executing on a hardware platform of a host includes creating a user-space instance within a user-space above a kernel of the multi-tenant hypervisor, the user-space instance being isolated from one or more other user-space instances through namespaces, the user-space instance having resources confined by hierarchical resource groups. The method further includes creating a virtual hypervisor within the user-space instance.

Further embodiments include a non-transitory computer-readable storage medium comprising instructions that cause a computer system to carry out the above method.

DETAILED DESCRIPTION

FIG. 1is a block diagram of a computing system100in which one or more embodiments of the present disclosure may be utilized. Computing system100includes a data center150and computing devices122. Computing devices122communicate with data center150through a network130. Computing devices122are controlled or managed by different entities, such as entities132-1through132-N (collectively “entities132”), where N is an integer greater than or equal to one. Computing devices122include conventional components, such as one or more processors, system memory, network interfaces, storage systems, and other input/output (I/O) devices such as, for example, a mouse and keyboard (not shown).

In one application, data center150is controlled and administrated by a particular enterprise or business organization, and entities132are different logical divisions of the enterprise or business organization (e.g., different departments, subsidiaries, etc.). In such an application, data center150can be all or part of a “private cloud.” In another application, data center150is operated by a cloud computing service provider and exposed as a service available to account holders, such as other enterprises and business organizations. In such an application, data center150can be all or part of a “public cloud.” As used herein, an internal cloud or “private” cloud is a cloud in which entities and the cloud operator are part of the same enterprise or business organization. An external or “public” cloud is a cloud in which entities are separate from the cloud operator and are customers or clients of the cloud operator.

Data center150includes one or more host computer systems (“hosts104”), network systems152, and storage systems154. Network systems152can include gateways, routers, firewalls, switches, local area networks (LANs), and the like. Storage systems154can include storage area networks (SANs), network attached storage (NAS), fibre channel (FC) networks, and the like. Hosts104may be constructed on a server grade hardware platform106, such as an x86 architecture platform. Hosts104can be coupled to network systems152and storage systems154. Hosts104can be further coupled to network130either directly or through network systems152.

In some examples, data center150can also include cloud components156. Cloud components156can include cloud directors, cloud orchestration components, and like type components that are configured to dynamically provide an entity with one or more virtual data centers in which a user may provision VMs, deploy multi-tier applications on VMs, and/or execute workloads. Cloud components156can be coupled to hosts104for implementing virtual data centers. One example of a cloud director is the VMware vCloud Director® produced by VMware, Inc. Cloud components156do not typically expose hosts104directly to entities132and, as such, entities132cannot directly access hypervisor116through cloud components156. Instead, cloud components156expose an application programming interface (API) to entities132for VM and virtual data center management, such as a REST (Representational State Transfer) API (Application Programming Interface) or any other client-server communication protocol. Some cloud components156may make available direct host management, but the smallest supported entity is a host104. Cloud components156do not provide sub-host granularity with direct management interfaces, in contrast with the multi-tenant hypervisor discussed below.

As shown, hardware platform106of each host104may include conventional components of a computing device, such as one or more processors (CPUs)108, system memory110, a network interface112, storage system114, and other I/O devices such as, for example, a mouse and keyboard (not shown). CPU108is configured to execute instructions, for example, executable instructions that perform one or more operations described herein and may be stored in memory110and in local storage. Memory110is a device allowing information, such as executable instructions and data to be stored and retrieved. Memory110may include, for example, one or more random access memory (RAM) modules. Network interface112enables host104to communicate with another device via a communication medium, such as network systems152and network130. Network interface112may be one or more network adapters, also referred to as a Network Interface Card (NIC). Storage system114represents local storage devices (e.g., one or more hard disks, flash memory modules, solid state disks, and optical disks) and/or a storage interface that enables host104to communicate with one or more network data storage systems, such as storage systems154. Examples of a storage interface are a host bus adapter (HBA) that couples host104to one or more storage arrays, such as a SAN or a NAS, as well as other network data storage systems.

Each host104is configured to provide a virtualization layer that abstracts processor, memory, storage, and networking resources of hardware platform106into multiple virtual machines (VMs)120that run concurrently on the same hosts. VMs120run on top of a software interface layer, referred to herein as a hypervisor116, which enables sharing of the hardware resources of host104by VMs120. One example of hypervisor116that may be configured and used in embodiments described herein is a VMware ESXi™ hypervisor provided as part of the VMware vSphere® solution made commercially available from VMware, Inc. of Palo Alto, Calif. (although it should be recognized that any other virtualization technologies, including Xen® and Microsoft Hyper-V® virtualization technologies may be utilized consistent with the teachings herein). Hypervisor116may run on top of an operating system of host104or directly on hardware components of host104.

Hypervisor116includes a file system, referred to herein as the “global file system.” In an embodiment, the global file system is backed by storage system114(e.g., a local disk). In another embodiment, the global file system is backed by memory110. For example, a system image for hypervisor116may be stored in storage system114. The system image can include installation files that are used during boot to create the global file system within memory110. Hypervisor state can be maintained within storage system114by updating the system image. Thus, in general, the global file system of hypervisor116can be backed by persistent storage, non-persistent storage, or both.

In an embodiment, hypervisor116is a multi-tenant hypervisor configured to support independent and isolated management by multiple entities132. Hypervisor116employs operating system (OS)-level virtualization to define a plurality of user-space instances. Each user-space instance is isolated from each other user-space instance through namespaces defined by hypervisor116, such as process namespaces, network namespaces, storage namespaces, user namespaces, and the like. Further, each user-space instance includes resources allocated by hypervisor116using hierarchical resource groups. In this manner, each user-space instance provides an isolated execution environment for a virtual hypervisor. A “virtual hypervisor” is an isolated instance of a hypervisor management plane comprising management daemons, associated processes, and resources confined by the underlying user-space instance. Thus, a virtual hypervisor acts as an isolated virtualized host capable of being managed by an entity in a manner similar to that of a physical host. Hypervisor116supports a plurality of virtual hypervisors, e.g., virtual hypervisors118-1through118-N (collectively “virtual hypervisors118”). Together, a virtual hypervisor and its underlying user-space instance may be referred to herein as a “tenant.” Thus, hypervisor116supports multiple tenants.

Each virtual hypervisor118-1through118-N is managed by a respective one of entities132-1through132-N through an exposed management API. For example, computing devices122can include one or more of a virtualization manager124, a virtualization client126, or other management software128. Virtualization manager124is a computer program that resides and executes on a computing device (e.g., a server) either directly or as a virtual machine. One example of a virtualization manager is the VMware vCenter Server™ product made available from VMware, Inc. Virtualization manager124is configured to carry out administrative tasks, including managing hosts, managing VMs running within each host, provisioning VMs, migrating VMs from one host to another host, load balancing between hosts, and the like. Virtualization client126is a computer program that resides and executes on a computing device (e.g., a workstation) and is configured to manage an individual hypervisor through its API. In general, various other management software128can be configured to manage one or more hypervisors through the exposed management API. Each of virtualization manager124, virtualization client, and other management software128can manage one or more of virtual hypervisors118transparently as if virtual hypervisors118were hypervisors executing on physical hosts.

Operator of data center150can manage creation, deletion, etc. of tenants on hypervisor116through a tenant manager140executing as a computer program within a computing device138coupled to network130. Tenant manager140can cooperate with a service executing within hypervisor116, or an API exposed by hypervisor116, to create user-space instances, provision virtual hypervisors within such user-space instances, and power-on such virtual hypervisors. Tenant manager140can cooperate with hypervisor116to power-down virtual hypervisors, remove such virtual hypervisors, and remove associated user-space instances. In some examples, tenant manger140can cooperate with hypervisors116to perform other operations, such as migrating tenants to another hypervisor executing on another physical host.

By enabling multi-tenancy in hypervisor116, the granularity of provisioning in data center150can be decreased. This allows the operator of data center150to fully utilize and partition large physical hosts for use by multiple distinct entities. Further, sub-host provisioning is not limited to cloud services provided through cloud components156. Some entities may desire more control over resource management afforded by direct management of a hypervisor as opposed to procuring more opaque cloud services. In such a case, the operator of data center150is not limited to provisioning the entire host104to an entity and can instead enable multi-tenancy in hypervisor116. Entities are provided direct access to a hypervisor in the form of a virtual hypervisor118, and the operator of data center150more efficiently employs the resources of host104.

FIG. 2is a block diagram depicting a software stack200implemented by hypervisor116according to an embodiment. At the bottom, software stack200includes a kernel202. Logical operating spaces are disposed above kernel202, such as a user-space203and a virtual machine monitor (VMM)-space205. VMs120are disposed above VMM-space205.

Kernel202includes device drivers204, a storage stack206, a network stack208, file system modules210, network modules212, logical file system modules214, a resource scheduler module216, a user-space instance module218, and interfaces222. Interfaces222include a system call interface224and a virtual file system interface226. Kernel202supports a plurality of logical operating spaces, including user-space203and VMM-space220. VMM-space205includes one or more VMMs220executing therein. User-space203includes a plurality of user-space instances, e.g., user-space instances228-1,228-2, and229. User-space instance229includes a tenant management service230executing therein.

Kernel202can be a portable operating system interface (POSIX)-style OS designed to operate as a hypervisor that supports execution of virtual machines. Device drivers204include a collection of modules configured to manage the hardware of the physical host on which hypervisor116is installed (e.g., host104). Storage stack206includes a collection of modules providing various layers that create and map logical storage to physical storage managed by device drivers204. For example, device drivers204can manage physical storage interfaces, such as FC, FC over Ethernet (FCoE), Internet Small Computer System Interface (iSCSI), SCSI, Integrated Drive Electronics (IDE), Serial AT Attachment (SATA), Serial Attached SCSI (SAS), and like interfaces. Storage stack206provides higher-level logical interfaces to such physical storage, such as a block-level interfaces, logical volume interfaces, SCSI subsystems, and the like. File system modules210impose file system structures on the logical devices provided by storage stack206, such as file allocation table (FAT), New Technology File System (NTFS), extended file systems (e.g., ext2, ext3, ext4), Virtual Machine File System (VMFS), network file system (NFS), and the like.

Network stack208includes a collection of modules that implement various layers to provide a logical interface to physical network devices managed by device drivers204. For example, device drivers204can manage physical NICs, while network stack208can include a link layer (e.g., Ethernet), a network layer (e.g., Internet Protocol (IP)), a transport layer (e.g., Transmission Control Protocol (TCP)), application session layer (e.g., Secure Socket Layer (SSL)), and the like. Network modules212provide various logical network devices supported by network stack208, such as virtual switches, virtual NICs, and the like. In particular, network modules212can define one or more virtual switches. Each virtual switch can have one or more uplink ports associated with one or more physical NICs. Network modules212can define one or more virtual NICs to be coupled to ports of each virtual switch. Virtual NICs can be provisioned to virtual machines. In this manner, network modules212allow a plurality of virtual machines to share the physical NIC(s).

Kernel202includes various logical file system modules214, such as a device file system module (e.g., devfs), a process file system module (e.g., procfs), a volume cache file system module (e.g., vcfs) and the like. Kernel202includes a resource scheduler module216configured to allocate resources among different executing processes, such as CPU resources, memory resources, storage resources, network resources, and the like. System call interface224includes a plurality of system calls that processes use to interface with kernel202. System call interface224can include standard POSIX system calls, as well as specialized system calls that extend the core set of POSIX system calls (e.g., system calls added by kernel modules or proprietary to the kernel itself). Virtual file system interface226includes a versioned and typed hierarchical namespace that provides an interface to kernel202. Virtual file system interface226can be used to communicate hardware and kernel state and for manipulating kernel modules.

VMM-space205provides an execution environment and appropriate privileges for the execution of VMMs220. VMMs220in turn provide execution environments for guest operating systems of VMs120. VMMs220expose virtualized hardware to the guest operating systems, such as a virtual BIOS, virtual CPUs, virtual RAM, virtual disks, virtual NICs, and the like. The guest operating systems execute on the virtual hardware.

Each user-space instance provides an execution environment and appropriate privileges for the execution of processes. That is, the management plane of hypervisor116is implemented within user-space203. A hypervisor management plane includes management daemons and associated processes that provide an API for managing the hypervisor (e.g., setting network parameters, setting storage parameters, setting users, passwords, and permissions, creating, deleting, or otherwise managing virtual machines, managing resource pools, managing features such as migration, disaster recovery, and high-availability). If hypervisor116is configured as a single-tenant hypervisor, then the management plane executes directly within a single user-space instance (e.g., the user-space instance229). When hypervisor116is configured as multi-tenant hypervisor, then multiple isolated management plane instances execute within multiple user-space instances (e.g., user-space instances228-1and228-2, as well as user-space instance229).

Tenant management service230is configured to implement multi-tenancy within hypervisor116. Tenant management service230can be a process executing within user-space instance229. An administrator can interact with tenant management service230(e.g., using tenant manager140) to create, delete, and otherwise manage tenants. In particular, tenant management service230can interact with user-space instance module218through interfaces222to create user-space instances228. Each user-space instance228provides an isolated execution environment for a virtual hypervisor118. User-space instances228achieve isolation through provisioning of namespaces and hierarchical resource groups. Each virtual hypervisor118includes a management plane instance, e.g., virtual hypervisor118-1includes management plane232-1and virtual hypervisor118-2includes management plane232-2. Management plane232of each virtual hypervisor118includes management daemons, associated processes, and resources confined by a respective user-space instance228. Each management plane instance exposes an API234. Users can interact with API234to manage a virtual hypervisor118, as well as create, delete, or otherwise manage virtual machines supported by VMMs220in VMM-space205. VMMs220provide isolation among virtual machines, while user-space instances228provide isolation among hypervisor management planes. To the users, each virtual hypervisor118acts as a hypervisor executing on an isolated host.

User-space instance module218is configured to interact with other kernel modules to support host-level multi-tenancy through creation and management of tenants (e.g., user-space instances228and virtual hypervisors118). User-space instance module218can add system calls to system call interface224and/or nodes to virtual file system interface226to allow process(es) to create tenants (e.g., tenant management service230). While user-space instance module218is described as a single module, it is to be understood that some portions of the program code necessary to create and manage tenants can be distributed throughout other kernel modules (e.g., storage stack206, network stack208, logical file system modules214, etc.). Thus, references to user-space instance module218herein also encompass such distributed code in other kernel modules.

In an embodiment, user-space instance module218is configured to sanitize interfaces222to prevent user-space instances228from accessing and manipulating host state. For example, user-space instance module218can hide or provide stubs for certain nodes of virtual file system interface226. A hidden node is not accessible (readable or writeable) by user-space instances228. A stubbed node may be accessible, but user-space instance module218intercepts any access (read or write) by user-space instances228and performs custom processing. For example, certain node-trees in virtual file system interface226related to the hardware of host104can be hidden from user-space instances228so that user-space instances cannot access or manipulate host-hardware state. Other node-trees in virtual file system interface228related to resource groups can be stubbed to show only a subset of information to each user-space instance228. This prevents a user-space instance228from observing or manipulating resource groups assigned to other user-space instances228. In an embodiment, user-space instance module218implements an opt-in approach for node visibility. In such an approach, nodes are hidden from user-space instances228unless specifically opting-in. Virtual file system interface226can include flags or other information that indicates whether a given node-tree opts in to be visible to user-space instances228.

Likewise, user-space instance module218can hide or provide stubs for certain system calls of system call interface224. For example, system calls that provide for module loading, CPU microcode updates, and device scanning can be hidden or stubbed. In another example, system calls that manipulate kernel202or query hardware state can be hidden from user-space instances228. In an embodiment, user-space instance module218implements an opt-in approach for presenting system calls to user-space instances228. In such an approach, system calls are not presented to user-space instance228unless specifically opting-in. System call interface224can include flags or other information that indicates whether a given system call opts in to be visible to user-space instances228.

As described below, user-space instances228include separate storage namespaces. User-space instance module218cooperates with file system modules210to implement mount points and/or reparse points in the global file system of hypervisor116. Mount points allow a file system (whether logical or backed by a physical storage device) to be attached to a particular sub-tree of the global file system. Reparse points allow for a sub-tree of the global file system to be aliased at an arbitrary location. For example, user-space instance module218can cooperate with logical file system modules214to provide reparse points for instances of the logical file systems (e.g., devfs, procfs, vcfs, etc.). Each of the logical file system modules can be configured to support multiple user-space instances. For example, the volume cache file system (vcfs) provides access to storage volumes accessible by hypervisor116. The vfcs module can maintain per-user-space instance data structure(s) identifying storage volumes associated with particular tenants. In another example, the device file system (devfs) provides access to various block, character, and special devices recognized by kernel202. The devfs module can be configured to filter-out various devices from being accessible by tenants for purposes of isolating the tenants from direct access to host devices.

As described below, user-space instances228include separate network namespaces. User-space instance module218cooperates with network stack208to implement separate network stacks for user-space instances228. User-space instance module218also cooperates with network modules212to create simulated physical NICs for user-space instances228. Each simulated physical NIC is coupled to a port of a virtual switch managed by hypervisor116, which is in turn coupled to a physical NIC. A virtual hypervisor118executing in a user-space instance228can create its own vSwitch having its uplink port coupled to an simulated physical NIC assigned to the underlying user-space instance228.

FIG. 3is a block diagram showing a tenant300within a multi-tenant hypervisor according to an embodiment. As shown inFIG. 3, user-space instance228includes resources302, a process namespace310, a user namespace312, a storage namespace314, and a network namespace318. Resources302include a share of resources allocated to user-space instance228by resource scheduler module216and user-space instance module218in kernel202. Example resources include persistent storage304, non-persistent storage305, simulated physical NIC(s)306, and compute resources308(e.g., CPU and memory allocations). Process namespace310provides a separate process tree for user-space instance228. All processes executing within user-space instance228are confined to process namespace310. User namespace312can provide separate users, passwords, permissions, etc. for user-space instance228. Network namespace318can include a separate network stack for user-space instance228.

Storage namespace314provides a separate root file system for user-space instance228. All processes are confined to use the separate root file system (e.g., a chroot environment). In an embodiment, storage namespace314is placed in the global file system hierarchy of hypervisor116. For example, the global file system of hypervisor116can include a directory “/tenants” that will hold storage namespaces of tenants. Storage namespace314for user-space instance228can be placed in the global file system hierarchy at “/tenants/name,” where name is an identifier associated with user-space instance228. Storage namespace314can be rooted at “/tenants/name” (e.g., /tenants/name can be mounted as/(root file system) for user-space instance228). As such, user-space instance228will have no visibility outside of the /tenants/name sub-tree in the global file system of hypervisor116.

Storage namespace314can be backed by persistent storage304, non-persistent storage305, or both. For example, the /tenants/name sub-tree can be backed by non-persistent storage305(e.g., a RAM-based file system backed by RAM). Re-parse points can be created in the /tenants/name sub-tree for various logical file systems, such as devfs and procfs.

Tenant state, such as log files, configuration files, VMs, and the like, can be stored on a particular storage volume that is part of persistent storage304(e.g., either local storage or remote storage). For example, a tenant having name can be associated with a storage volume labeled name-state. The storage volume name-state would be mounted by the vfcs module in the global file system of hypervisor, e.g., at /vmfs/volumes/name-state. Storage namespace314has no visibility into the /vmfs sub-tree of the global file system, but a re-parse point can be created at /tenants/name/persistent that is backed by /vmfs/volumes/name-state in the global file system.

User-space instance228can store persistent data at /persistent within storage namespace314. Parts of file system for user-space instance228that need to be persistently stored can be symbolically linked (symlinked) into the /persistent directory. For example, the /etc and /var directories within storage namespace314can be symlinked into /persistent/etc and /persistent/var. The storage volume labeled name-state can also store a datastore having VM files for the tenant. For example, the datastore can be accessible at /persistent/datastore1 within storage namespace314. Some management processes may expect the datastore to be mounted at a particular location within storage namespace314, such as in /vmfs/volumes. As such, a re-parse point can be created at /vmfs/volumes/datastore1 within storage namespace314backed by /persistent/datastore1. The above-described construction of storage namespace314is merely one example and various other file system structures can be used having the same or different mount points and/or re-parse points.

Virtual hypervisor118includes binaries322, configuration files324, VM files326, virtual network devices328, management daemons330, and processes332. Binaries322, configuration files324, and VM files326are stored within storage namespace314(e.g., binaries322in /bin, configuration files324in /etc, and VM files326in /persistent/datastore1). Virtual network devices328include virtual NICs, virtual switches, and the like. Each virtual switch of virtual hypervisor118includes uplink port(s) coupled to simulated physical NIC(s)306. Virtual NICs can be provisioned among the VMs. Management daemons330include various management processes that provide the API for virtual hypervisor118. Processes332include various other processes, such as virtual machine helper processes, logging processes, and the like. Management daemons330and processes332execute within process namespace310.

FIG. 4is a flow diagram depicting a method400of creating a tenant in a multi-tenant hypervisor according to an embodiment. Method400can be performed by tenant management service230in cooperate with user-space instance module218in kernel202. An administrator can initiate and control execution of method400by tenant management service230using tenant manager140.

Method400begins at step402, where tenant management service230creates a user-space instance for the tenant. In an embodiment, tenant management service230provisions persistent storage for hold tenant state (404), provision a storage namespace (406), provision a network namespace (410), and provision any additional namespaces (414). At step404, tenant management service230provisions persistent storage for storing tenant state on a storage volume, which can be stored in storage114or in storage systems154. In an embodiment, the storage volume is implemented as a VMFS file system on a virtual disk. Implementing the storage volume as a virtual disk allows for control and management of persistent storage provisioned for each tenant (e.g., a tenant cannot consume more space than allocated to the virtual disk). Alternative implementations are possible, including other types of file systems on a virtual disk, as well as VMFS or other types of file systems directly on storage114or storage systems154.

At step406, tenant management service230provides the storage namespace. In an embodiment, tenant management service230creates a root file system linked with the global file system of hypervisor116(e.g., /tenants/name sub-tree), and establishes various mount points, re-parse points, and/or symlinks (408). For example, the root file system can include a re-parse point for accessing the storage volume provisioned at step404(e.g., a re-parse point at /tenants/name/persistent backed by /vmfs/volumes/name-state). The root file system can include re-parse points for accessing logical file systems (e.g., devfs, procfs, etc.). The root file system can include symlinks to directories on the storage volume for directories requiring persistent storage (e.g., /var, /etc/, and the like).

At step410, tenant management service230provisions the network namespace. In an embodiment, tenant management service230creates a network stack instance and simulated physical NIC(s) (412). At step414, tenant management service230provisions one or more other namespaces, such as a process namespace, user namespace, and the like.

At step416, tenant management service230creates a virtual hypervisor within the user-space instance. In an embodiment, tenant management service230provisions an instance of a management plane within the namespaces (418), store configuration as part of the tenant state (424), and startup the management plane (426).

At step418, tenant management service230provisions the instance of the management plane within the namespaces defined for the user-space instance. In an embodiment, tenant management service230populates the root file system with binaries, scripts, configuration files, and other files for the management daemons and associated processes of the management plane (420). In an embodiment, tenant management service230copies files from the global file system of hypervisor116. In another embodiment, tenant management service230copies files from installable packages in a system image. This allows the virtual hypervisor to be a different version than hypervisor116(e.g., storage114can store multiple system images for different hypervisor versions). In an embodiment, tenant management service230creates various virtual network device(s), such as virtual switch(es) and virtual NIC(s) (422).

At step424, tenant management service424stores a configuration of virtual hypervisor (state information) as part of the tenant state in storage volume provisioned at step404. At step426, tenant management service424starts up the management plane. Thereafter, a user can interact with the management plane of the virtual hypervisor to manipulate the virtual hypervisor, create or otherwise manage virtual machines, and the like.