Managing lifecycle of virtualization software in a virtualized computing system

An example method of managing a lifecycle of virtualization software in a host is described. The method includes: obtaining, by an initiator in a current version of the virtualization software, a software installation bundle (SIB) from an image repository for a target version of the virtualization software, the SIB including a patcher; verifying, by the initiator, authenticity of the SIB; mounting at least one payload of the SIB in a root filesystem of the virtualization software, and initiating, by the initiator, the patcher in the at least one payload as mounted to perform at least one check operation.

BACKGROUND

In many virtualization computing systems, virtualization software is installed on a cluster of hosts using an ISO image that is created from a flat list of software installation bundles (SIBs). An SIB is the smallest unit of software that can be shipped and installed, and these SIBs make up, for example, a base hypervisor image (hereinafter also referred to as “base image”) from a virtualization software provider, as well as drivers, agents, and other software components from an OEM (original equipment manufacturer) and other vendors of hardware. In a typical installation, hundreds of these SIBs are packaged as one or more ISO images and installed in the hosts.

After installation, lifecycle management of the virtualization software becomes cumbersome and error-prone. Although different software developers create new versions or updates to the SIBs, the new versions or updates cannot be released independently. The releases have to be tightly controlled because it is likely that one SIB has a dependency to another SIB. As a result, new releases are made in the form of bulletins, which are a collection of software installation bundles, or as a new ISO image in which new SIBs from the virtualization software provider, the OEM, and other software vendors are packaged. Because of the inter-dependencies and the integration of the newly developed SIBs with other SIBs, it is difficult to make piecemeal changes to the virtualization software for easy consumption by an end user during the lifecycle of the virtualization software.

SUMMARY

One or more embodiments provide a method of managing a lifecycle of virtualization software in a host. The method includes: obtaining, by an initiator in a current version of the virtualization software, a software installation bundle (SIB) from an image repository for a target version of the virtualization software, the SIB including a patcher; verifying, by the initiator, authenticity of the SIB; mounting at least one payload of the SIB in a root filesystem of the virtualization software; and initiating, by the initiator, the patcher in the at least one payload as mounted to perform at least one check operation.

DETAILED DESCRIPTION

Managing lifecycle of virtualization software in a virtualized computing system is described. In embodiments described herein, a virtualized computing system includes a software-defined datacenter (SDDC) comprising a server virtualization platform integrated with a logical network platform. The server virtualization platform includes clusters of physical servers (“hosts”) referred to as “host clusters.” Each host cluster includes a virtualization layer, executing on host hardware platforms of the hosts, which supports execution of virtual machines (VMs) A virtualization management server manages host clusters, the virtualization layers, and the VMs executing thereon. The virtualized computing system utilizes a “patch the patcher” process to manage the lifecycle of the virtualization software on each host, such as patching or upgrading the virtualization software. In the patch the patcher process, lifecycle software obtains a software installation bundle (SIB) for the target image that includes a new patcher for the target version. An old patcher of the current version is used to verifying the SIB, securely mount its payloads, and initiation execution of scripts/binaries of the new patcher. The scripts/binaries can perform various check operations, such as pre-checks, post-checks, etc., as well as patch or upgrade operations. The new patcher executes instead of the old patcher and can be removed in case a patch or upgrade operation is not performed, leaving the old patcher in place. These and further advantages are discussed below with respect to the drawings.

FIG. 1is a block diagram of a virtualized computing system100in which embodiments described herein may be implemented. System100includes a cluster of hosts120(“host cluster118”) that may be constructed on server-grade hardware platforms such as an x86 architecture platforms. For purposes of clarity, only one host cluster118is shown. However, virtualized computing system100can include many of such host clusters118. As shown, a hardware platform122of each host120includes conventional components of a computing device, such as one or more central processing units (CPUs)160, system memory (e.g., random access memory (RAM)162), one or more network interface controllers (NICs)164, and optionally local storage163. CPUs160are configured to execute instructions, for example, executable instructions that perform one or more operations described herein, which may be stored in RAM162. NICs164enable host120to communicate with other devices through a physical network180. Physical network180enables communication between hosts120and between other components and hosts120(other components discussed further herein). Physical network180can include a plurality of virtual local area networks (VLANs) to provide external network virtualization as described further herein. Hardware platform122can further include firmware165and a trusted platform module (TPM)166, described further herein.

In the embodiment illustrated inFIG. 1, hosts120access shared storage170by using NICs164to connect to network180. In another embodiment, each host120contains a host bus adapter (HBA) through which input/output operations (IOs) are sent to shared storage170over a separate network (e.g., a fibre channel (FC) network). Shared storage170include one or more storage arrays, such as a storage area network (SAN), network attached storage (NAS), or the like. Shared storage170may comprise magnetic disks, solid-state disks, flash memory, tape and the like as well as combinations thereof. In some embodiments, hosts120include local storage163(e.g., hard disk drives, solid-state drives, etc.). Local storage163in each host120can be aggregated and provisioned as part of a virtual SAN, which is another form of shared storage170.

A software platform124of each host120provides a virtualization layer, referred to herein as a hypervisor150, which directly executes on hardware platform122. In an embodiment, there is no intervening software, such as a host operating system (OS), between hypervisor150and hardware platform122. Thus, hypervisor150is a Type-1 hypervisor (also known as a “bare-metal” hypervisor). As a result, the virtualization layer in host cluster118(collectively hypervisors150) is a bare-metal virtualization layer executing directly on host hardware platforms. Hypervisor150abstracts processor, memory, storage, and network resources of hardware platform122to provide a virtual machine execution space within which multiple virtual machines (VM) may be concurrently instantiated and executed. One example of hypervisor150that may be configured and used in embodiments described herein is a VMware ESXi™ hypervisor provided as part of the VMware vSphere® solution made commercially available by VMware, Inc. of Palo Alto, Calif. Hypervisor150manages virtual machines (VMs)140executing thereon. VMs140support applications deployed onto host cluster118, which can include containerized applications or applications executing directly on guest operating systems (non-containerized).

Virtualization management server116is a physical or virtual server that manages host cluster118and the virtualization layer therein. Virtualization management server116installs agent(s)152in hypervisor150to add a host120as a managed entity. Virtualization management server116logically groups hosts120into host cluster118to provide cluster-level functions to hosts120, such as VM migration between hosts120(e.g., for load balancing), distributed power management, dynamic VM placement according to affinity and anti-affinity rules, and high-availability. The number of hosts120in host cluster118may be one or many. Virtualization management server116can manage more than one host cluster118.

In an embodiment, system100further includes an image repository190. As described herein, image repository190can store image profiles and software installation bundles (SIBs) for hypervisor software. The profiles and SIBs can be downloaded to hosts120and used to patch, update, upgrade, etc. hypervisor150as described further herein.

Virtualization management server116comprises a virtual infrastructure (VI) control plane113of virtualized computing system100. Virtualization management server116can include VI services108and lifecycle service111. VI services108include various virtualization management services, such as a distributed resource scheduler (DRS), high-availability (HA) service, single sign-on (SSO) service, virtualization management daemon, and the like. DRS is configured to aggregate the resources of host cluster118to provide resource pools and enforce resource allocation policies. DRS also provides resource management in the form of load balancing, power management, VM placement, and the like. HA service is configured to pool VMs and hosts into a monitored cluster and, in the event of a failure, restart VMs on alternate hosts in the cluster. A single host is elected as a master, which communicates with the HA service and monitors the state of protected VMs on subordinate hosts. The HA service uses admission control to ensure enough resources are reserved in the cluster for VM recovery when a host fails. SSO service comprises security token service, administration server, directory service, identity management service, and the like configured to implement an SSO platform for authenticating users. The virtualization management daemon is configured to manage objects, such as data centers, clusters, hosts, VMs, resource pools, datastores, and the like. Lifecycle service111cooperates with an agent152in hypervisor150to patch, update, upgrade, etc. the hypervisor software as described further herein.

A VI admin can interact with virtualization management server116through a VM management client106. Through VM management client106, a VI admin commands virtualization management server116to form host cluster118, configure resource pools, resource allocation policies, and other cluster-level functions, configure storage and networking, patch, update, or upgrade hypervisor software on hosts120, and the like.

FIG. 2is a block diagram depicting software platform124according an embodiment. As described above, software platform124of host120includes hypervisor150that supports execution of VMs140. Each VM140can include an operating system (OS)204that supports applications202. In an embodiment, hypervisor150includes a VM management daemon213, a host daemon214, lifecycle agent216, SIBs218, and a root filesystem (FS)220. VM management daemon213is an agent152installed by virtualization management server116. VM management daemon213provides an interface to host daemon214for virtualization management server116. Host daemon214is configured to create, configure, and remove VMs140.

Lifecycle agent216cooperates with lifecycle service111to manage hypervisor software for hypervisor150. Lifecycle agent216can download SIBs218from image repository190upon command by lifecycle service111. Each SIB218includes one or more payloads220. A payload can be a compressed collection of files in a file system structure (e.g., a tardisk or the like). Root FS221includes files222of hypervisor150, such as the hypervisor runtime (e.g., kernel, virtual machine monitors, and the like). Root FS221is constructed by mounting various payloads220from SIBs218during boot.

FIGS. 3A and 3Bdepict a process for installing a software installation bundle in a hypervisor according to an embodiment. As shown inFIG. 3A, image repository190includes image metadata302and SIB files304. Image metadata302includes information as to which SIB files304comprise a target image, which a user has selected. Image repository190can include image metadata302that provides many choices to form different target images. Each target image can include a specific set of SIB files304. A SIB file304includes a SIB218. SIB218includes metadata306, a signature308, and one or more payloads220. Metadata306describes SIB218. Signature308is a cryptographic function of metadata306(e.g., a hash or the like) that is authenticated by a trusted source (e.g., using a certificate). Payloads220include images of files in a file structure (e.g., a tardisk or the like). Root FS221includes the files of hypervisor150, including runtime310and mounted payload(s)312. Runtime310includes the kernel, virtual machine monitors, etc. of hypervisor150. Mounted payload(s)312include those mounted from a SIB218in this example installation process.

FIG. 3Bshows a flow diagram depicting a method300of installing a software installation bundle in a hypervisor according to an embodiment. Method300can be understood with reference toFIG. 3A. Method300begins a step314, where lifecycle software (e.g., lifecycle agent216and initiator software in hypervisor150) looks up a link to an SIB218in image metadata302for a target image. In examples described below, an example SIB318is a new patcher SIB that will be used to execute pre-checks, post-checks, upgrade operations, etc. on hypervisor150. The new patcher SIB will be used to replace an old patcher SIB that is part of the current version of hypervisor150as part of the patching/upgrade process where the patcher is being upgraded.

At step316, lifecycle software downloads SIB218to host120. At step318, lifecycle software verifies signature308of SIB218to ensure its authenticity. Lifecycle software can utilize SecureBoot in firmware165and/or TPM166to provide additional verification as noted below. SecureBoot is a firmware standard that enforces loading of only trusted binaries. TPM is an industry standard for authenticating and attesting the state of software running on a host120. At step320, lifecycle software verifies a checksum of each payload220. The checksums for payloads can be included in metadata306. This ensures the payloads220are not corrupted. At step322, lifecycle software mounts payloads220in root FS221. At step324, lifecycle software initiates execution of scripts and/or binaries in the mounted payloads220. Such execution can be used to perform various functions, such as software pre-checks, software post-checks, patches, upgrades, and the like.

As part of the hypervisor patching/upgrade process, in several stages, formalized prechecks are executed to assure the desired state (image/config) can be applied, or that a remediation step (such as enter maintenance mode) can be performed. Examples of precheck include but are not limited to: Is the host currently in a healthy operating state? Is the host ready to receive the new desired state: image, config and hardware compatibility, VSAN health, and network status/health. Can the host enter maintenance mode, check for DRS status and recommendation, HA health status, VSAN resource status. Can the host exit maintenance mode: VSAN health post-check? Similarly, post remediation verifications can be performed: Is the ESX host currently in a healthy operating state. Is the desired state successfully remediated: verify desired image and config are applied successfully? The pre/post-check framework features the following characteristics/advantages. Check items are detached from the main remediation process, being held in different SIB payloads, and can be added in a flexible manner without any change to the initiator software. In a precheck, check items are securely “extracted” from the desired image (FIGS. 3A-3B) and be executed on the current running system that is pending remediation. The mechanism provides a uniform way to report check result and form user messages including errors, warnings and info.

Aside from signature/security checksum verifications as described above, additional security guarantees can be present when SecureBoot and/or TPM are enabled: only verified SIB can have their payloads mounted (no user override can be given), once mounted the payload will not be able to be altered, including all files/executables enclosed, no untrusted checks will be able to launch even if runtime filesystem is tampered with, etc.

FIG. 4is a block diagram depicting a state of hypervisor software according to an embodiment. In embodiments, SIBs218can include both an old patcher402and a new patcher404. A patcher is software (e.g., binaries, libraries, etc.) configured to patch/upgrade hypervisor150. When hypervisor150is first installed or last patched/upgraded, hypervisor software includes old patcher402, which is part of the current version of the hypervisor software. In contrast, new patcher404is part of a new version of hypervisor software. Lifecycle agent216can download new patcher404from image repository190prior to patching/upgrading hypervisor software to the new version. New patcher404can then be executed to perform the pre-checks, post-checks, patches, upgrades, etc. This is desirable, since there may be many differences between the current version of hypervisor150and the new version. In such case, old patcher402may not have sufficient capability of performing pre-checks/post-checks since it is based on the older version of hypervisor150(e.g., requirements may have changed, new hardware compatibility may have been added, existing hardware capability may have been deprecated, etc.). Lifecycle software can mount payload(s) of new patcher404in root FS221using the process described above inFIGS. 3A-3B. Mounted new patcher406includes scripts/binaries408that are to be executed to perform the checks, patch process, etc. Mounted old patcher405can be used as the initiator for verifying new patcher404, mounting new patcher404, and initiating execution of scripts/binaries408.

This process is referred to herein as “patch the patcher,” since new patcher404is being layered on top of old patcher405. However, old patcher402is not removed or changed. Thus, new patcher404can be removed in case patching/upgrading is not performed (e.g., the user only desired to perform a compatibility check). The design of the patch the patcher procedure has the following features: Ability to upgrade the system with a new patcher from the target system image; trivial overhead of initiating the upgrade process with patch the patcher; not requiring upgrading just the patcher separately in the upgrade process; and the running system will be unchanged in case a failure occurs, a retry can be attempted without a system reset. Security guarantees of an upgrade process with patch the patcher include: patch the patcher logic, as part of the running system, is trusted and measured; the new patcher is enclosed in a payload whose authenticity/integrity is guaranteed as the current running patcher verifies its signature and security checksum and mounts it in its entirely; the new patcher is mounted as a whole, and individual binaries/scripts therein cannot be altered. New system payloads (SIBs) that are downloaded and installed by the new patcher go through signature verification and security checksum verification to ensure their authenticity. When booting the new image, all running binaries in the booted upgraded system are trusted when loaded, and only measured binaries can execute.

The security mechanisms described above can assist these guarantees: SecureBoot/TPM: when used in conjunction, they guarantee payloads loaded during the boot process can be anchored to a trusted root, all binaries running in the system are measured and trusted, and no untrusted binaries/scripts can execute. This ensures the authenticity of the upgrade initiator that starts patch the patcher, and also that the new system image is not corrupted/tampered with during the new system boot after upgrade. Secure Payload Enclosure and Binary Execution: all files/binaries/scripts can be installed via a payload mount only, any payload mount must go through a check to make sure the payload belongs to a signed SIB and is not corrupted/tampered, and modification to contents in a payload would invalidate its eligibility of trusted execution.

FIG. 5is a flow diagram depicting a method500of initiating a patch the patcher procedure for hypervisor software according to an embodiment. Method500begins at step502, where old patcher405functions as an initiator and performs signature and checksum verification of new patcher404. Old patcher405can also leverage SecureBoot and/or TPM if available to ensure authenticity and integrity of new patcher404. At step504, old patcher405mounts payload(s) of new patcher SIB in the secure fashion described above. At step506, old patcher405initiates new patcher404to execute scripts/binaries408therein in order to perform a requested process (e.g., pre-check, post-check, patch, upgrade, etc.).

FIGS. 6 and 7depict a process of performing a software compliance check for hosts executing hypervisor software according to an embodiment. The process can be used to verify a host is ready to have its hypervisor software patched/upgraded from a current version to a target version.FIG. 6is a signal flow diagram showing the communication between the user interface (UI) of virtualization management server116, lifecycle service111, lifecycle agent216, and image repository190.FIG. 7is a flow diagram depicting a method700of performing the compliance check. Steps inFIG. 7that correspond to signals inFIG. 6are designated with identical reference numerals.

Method700begins at step602, where the UI requests a scan of a host120. Note that method700is described with respect to a single host120by way of example, but can be extended for verification of host cluster118. At step604, lifecycle service111commands lifecycle agent216to perform a scan in response to the request. At step606, lifecycle agent216invokes the patch the patcher process described above. Namely, at step608, lifecycle agent216cooperates with initiator (e.g., old patcher402) to download new patcher SIB for the target image. At step612, old patcher402verifies the signature of the new patcher SIB and securely mounts its payloads as described above. Old patcher402then initiates the new patcher. At step614, new patcher gets the image profile for the target image from image repository190. Step618includes the following actions performed by the new patcher. At step720, the new patcher creates a software specification from the image profile for the target image. That is, the new patcher parses the image profile to extract the software specification for the target image. At step722, the new patcher determines the difference between the current software specification and the new software specification. At step706, the new patcher determines if a reboot is required to apply the patches/updates. At step707, the new patcher determines if the host hardware is compatible with the target image. At step708, the new patcher computes a compliance result. At step620, the new patcher returns the compliance result to lifecycle service111(e.g., through lifecycle agent216). At step622, the old patcher unmounts the payloads of the new patcher. At step624, lifecycle service111presents the compliance result to the user through the UI.

The embodiments described herein may employ various computer-implemented operations involving data stored in computer systems. For example, these operations may require physical manipulation of physical quantities. Usually, though not necessarily, these quantities may take the form of electrical or magnetic signals, where the quantities or representations of the quantities can be stored, transferred, combined, compared, or otherwise manipulated. Such manipulations are often referred to in terms such as producing, identifying, determining, or comparing. Any operations described herein that form part of one or more embodiments may be useful machine operations.

One or more embodiments of the invention also relate to a device or an apparatus for performing these operations. The apparatus may be specially constructed for required purposes, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. Various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

One or more embodiments of the present invention may be implemented as one or more computer programs or as one or more computer program modules embodied in computer readable media. The term computer readable medium refers to any data storage device that can store data which can thereafter be input to a computer system. Computer readable media may be based on any existing or subsequently developed technology that embodies computer programs in a manner that enables a computer to read the programs. Examples of computer readable media are hard drives, NAS systems, read-only memory (ROM), RAM, compact disks (CDs), digital versatile disks (DVDs), magnetic tapes, and other optical and non-optical data storage devices. A computer readable medium can also be distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

Virtualization systems in accordance with the various embodiments may be implemented as hosted embodiments, non-hosted embodiments, or as embodiments that blur distinctions between the two. Furthermore, various virtualization operations may be wholly or partially implemented in hardware. For example, a hardware implementation may employ a look-up table for modification of storage access requests to secure non-disk data.

Many variations, additions, and improvements are possible, regardless of the degree of virtualization. The virtualization software can therefore include components of a host, console, or guest OS that perform virtualization functions.

Plural instances may be provided for components, operations, or structures described herein as a single instance. Boundaries between components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention. In general, structures and functionalities presented as separate components in exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionalities presented as a single component may be implemented as separate components. These and other variations, additions, and improvements may fall within the scope of the appended claims.