Patent Description:
A hypervisor is a software layer that provides the environment (i.e., a virtualized hardware partition) in which the virtual machines operate. A hypervisor sits between the physical resources on a physical hardware computing system and VMs. Without a hypervisor, an operating system communicates directly with the hardware beneath it, i.e., disk operations would go directly to the disk subsystem, and memory calls would be fetched directly from the physical memory. When multiple operating systems of multiple VMs are running simultaneously on a single machine, a hypervisor manages the interactions between each VM and the shared hardware, such that simultaneous control of the shared hardware by multiple VMs is prevented.

When a hypervisor resets or restarts (e.g., due to a software upgrade), all the VMs are running on top of the hypervisor are interrupted (e.g., restarted suspended, etc.), which can significantly impact the workloads and services operating on the VMs.

<NPL> relates to fast and live hypervisor replacement. In Spoorti Doddamani Et Al, a new technique, called HyperFresh, is presented, that transparently replaced a hypervisor with a new updated instance without disrupting any running virtual machines. A thin shim layer, called the hyperplexor, performs live hypervisor replacement by remapping guest memory to a new updated hypervisor on the same machine. It is the object of the present application to provide a system that enables replacing a first hypervisor with a second hypervisor with minimal to imperceptible interruption to guest virtual machines. This object is solved by the subject matter of the independent claim.

This Summary is provided to introduce a selection of concepts in a simplified form that is further described below in the Detailed Description.

The embodiments described herein are related to hypervisor hot restart, which replaces a first hypervisor with a second hypervisor with little interference to the currently running guest partitions. The embodiments described herein are implemented at a computing system. First, the computing system executes a first hypervisor on the computing system. The first hypervisor is configured to create one or more guest partitions. During a hypervisor soft restart, a service partition is created. A second hypervisor is initialized within the service partition. The service partition is created with an identity mapping between its guest physical address (GPA) space to host physical address (HPA) space. Optionally, it may be bestowed additional privileges over a traditional partition/virtual machine to facilitate its initialization. Any execution environment that can satisfy platform's boot and security requirement is capable of creating and/or initializing the service partition. For example, a component in trusted computing base (TCB) of the computing system, such as the first hypervisor, can be tasked to create or initialize the second hypervisor. During initialization, at least a portion of the state of the first hypervisor is initialized to the second hypervisor. The state of the first hypervisor includes, but is not limited to, (<NUM>) static system state, (<NUM>) dynamic system state, and (<NUM>) logical software state. The static system state includes, but is not limited to, system topology, memory map, and memory layout of the first hypervisor. The dynamic system state includes guest visible hardware architecture state, including but not limited to, general purpose registers, mode control registers. The dynamic system state also includes virtualization instruction set architecture (ISA) specific hardware state, including but not limited to, state of guest VMs, such as virtual central processing unit (CPU) state, second level page tables (also known as nested page tables), and a list of assigned hardware devices (e.g., network card). The logical software state includes, but is not limited to, page frame number (PFN) databases. Since the dynamic system state and logical software state may change constantly during runtime, these states are also called runtime state. Finally, the second hypervisor is devirtualized from the service partition to replace the first hypervisor, and the state of each VM is restored by the second hypervisor.

In some embodiments, at least one of the one or more guest partitions comprises a privileged parent partition, and the parent partition operates a host operating system that includes an orchestrator that is configured to orchestrate the initialization and synchronization of the second hypervisor. In some embodiments, the first hypervisor allows and supports the orchestrator to register and complete certain requests (e.g., hypercalls intercepts, register intercepts) received from the second hypervisor. Since these requests (sent from the second hypervisor to the first hypervisor) are intercepted (e.g., registered and completed) by the orchestrator, such requests are also called "intercepts. " In some embodiments, in response to receiving the intercepts, the orchestrator issues "inverse hypercalls" to the second hypervisor following the hypercall application binary interface (ABI) to migrate relevant state. A hypercall is a call from a guest partition to a hypervisor, and an "inverse hypercall" is a call from a privileged software or hardware component (e.g., the orchestrator) to the second hypervisor in a service partition. In some embodiments, the orchestrator may share a portion of a memory page with the first hypervisor and the second hypervisor for communication purposes (e.g., hypercalls, intercepts, and/or inverse hypercalls).

Although the example embodiments are implemented in a special partition (e.g., a service partition) for servicing a hypervisor, the overall techniques of pre-initializing data-structures in a virtual machine and/or migrating/synchronizing runtime state inside a VM/partition, to reduce down-time, offer better uptime and continuity for services, may be used for any portion of OS of any VM.

The embodiments described herein are related to hypervisor hot restart, which replaces a first hypervisor with a second hypervisor with little interference to the currently running guest partitions. The embodiments described herein are implemented at a computing system. First, the computing system executes a first hypervisor on the computing system. The first hypervisor is configured to create one or more virtual machines/partitions, each of which hosts a guest operating system. During a hypervisor soft restart, a service partition with identity mapped second level page tables is created. A second hypervisor is then initialized within the service partition, and synchronizes with at least a portion of the state of the first hypervisor. Any execution environment that can satisfy platform's boot and security requirements is capable of creating and/or initializing the second hypervisor. For example, a component in the trusted computing base (TCB), such as the first hypervisor, may be tasked to create and/or initialize the service partition.

In some embodiments, the state of the first hypervisor includes, but is not limited to, (<NUM>) static system state, (<NUM>) dynamic system state, and (<NUM>) logical software state. The static system state includes, but is not limited to, system topology, memory map, and memory layout of the first hypervisor. The dynamic system state includes guest visible hardware architecture state, including but not limited to, general purpose registers, mode control registers. The dynamic system state also includes virtualization instruction set architecture (ISA) specific hardware state, including but not limited to, state of guest VMs, such as virtual central processing unit (CPU) state, second level page tables (also known as nested page tables), and a list of assigned hardware devices (e.g., network card). The logical software state includes, but is not limited to, page frame number (PFN) databases. Since the dynamic system state and logical software state may change constantly during runtime, these states are also called runtime state. Finally, the second hypervisor is devirtualized from the service partition to replace the first hypervisor, and the state of each VM is restored by the second hypervisor.

In some embodiments, at least one of the one or more guest partitions comprises a privileged parent partition, and the parent partition operates a host operating system that includes an orchestrator that is configured to orchestrate the initialization and synchronization of the second hypervisor. In some embodiments, the first hypervisor is part of the system's trusted computing base (TCB), and the orchestrator is executed in a trusted execution environment (TEE) to uphold the trustworthiness of inputs generated for the second hypervisor. In some embodiments, the first hypervisor allows and supports the orchestrator to register and complete certain requests received from the second hypervisor. Since these requests (sent from the second hypervisor to the first hypervisor) are intercepted (e.g., registered and completed) by the orchestrator, such requests are also called "intercepts. " In some embodiments, in response to receiving the intercepts, the orchestrator issues "inverse hypercalls" to the second hypervisor following the hypercall application binary interface (ABI) to migrate relevant state. A hypercall is typically a call from a guest partition to a hypervisor requesting an operation of appropriate privilege. The switch from guest partition context to the hypervisor context is achieved using a platform specific instruction. On completion of the privileged operation, the control is returned to the guest context from the hypervisor context, against using a platform specific instruction/mechanism. An "inverse hypercall" requests an operation from a privileged component (e.g., the orchestrator) to a hypervisor in a service partition using the hypercall ABI. The switch from the orchestrator to the second hypervisor context is achieved by permitting execution of the service partition's virtual processor with the registers formatted as required by hypercall ABI. Upon completion of the inverse hypercall, the second hypervisor generates a very specific intercept causing a deliberate context switch from the second hypervisor in the service partition back to the orchestrator. In some embodiments, the orchestrator may share one or more memory pages with the first hypervisor and the second hypervisor for communication purposes (e.g., hypercalls, intercepts, and/or inverse hypercalls).

In some embodiments, initializing the second hypervisor includes the first hypervisor generating a loader block for the second hypervisor. The loader block contains a logical construct describing some of static system state for initialization. The static system state includes various system invariants, such as system topology, memory map and layout. The memory map includes an identity map of substantially all system memory. In some embodiments, the identity map includes at least the memory visible to the first hypervisor that is being replaced. For example, in some cases, the first (old) and second (new) hypervisors both know about all of the memory in the system. In some cases, at least one of the first and/or the second hypervisors does not know about all memory, e.g., when some RAM, containing errors, needs to be offlined, or the second (new) hypervisor will need to know about new memory that is hot-added during a hot restart. The orchestrator obtains one or more system invariants of the computing system, which includes at least one or more features supported by the hardware resources of the computing system (<NUM>) from the hardware or system resource(s) directly, when the specific resource is not virtualized by the hypervisor, and/or (<NUM>) from the first hypervisor, when the orchestrator's access to the underlying physical hardware resource is virtualized by the first hypervisor. In some embodiments, the orchestrator migrates relevant system invariants via inverse hypercalls. The second hypervisor is then initialized based on the shared one or more system invariants. In some embodiments, initializing the second hypervisor may also include offering the second hypervisor read-only specific access to certain physical resources.

After the second hypervisor is initialized with the static system state in the service partition, the computing system then migrates the runtime state of the first hypervisor to the second hypervisor and keeps it synchronized. The runtime state includes at least dynamic system state and logical software state. The synchronizing the runtime state of the first hypervisor may include the orchestrator gathering runtime state of the first hypervisor and sharing/migrating the runtime state with the second hypervisor over inverse hypercalls. As mentioned above, the dynamic system state includes guest visible hardware architecture state, including but not limited to, general purpose registers, mode control registers. The dynamic system state also includes virtualization instruction set architecture ISA specific hardware state, including but not limited to, state of guest VMs, such as virtual central processing unit (CPU) state, second level page tables (also known as nested page tables), and a list of assigned hardware devices (e.g., network card). The logical software state includes, but is not limited to, page frame number (PFN) databases. The second hypervisor then duplicates the shared second level memory page tables and/or PFN databases and leaves them dormant until it eventually devirtualizes.

Additionally, during this synchronization, the guest partitions running atop the first hypervisor continue to operate and issue hypercalls to the first hypervisor. Accordingly, after the initial synchronization of computationally intensive state, additional incoming hypercalls to the first hypervisor that impact previously transmitted state may also need to be synchronized to the second hypervisor. In some embodiments, such state changes are also transmitted using inverse hypercalls.

Once the first hypervisor and the second hypervisor are completely or substantially synchronized, the second hypervisor is then devirtualized. In some embodiments, the devirtualization includes the first hypervisor "trampolining" to the second hypervisor and ceding all physical hardware control to the second hypervisor. "Trampolines" are sometimes referred to memory locations holding addresses pointing to interrupt service routines, I/O routines, etc. Here, "trampolining" is referred to a transfer of control of all physical hardware from the first hypervisor to the second hypervisor. In some embodiments, trampolining may be achieved by reusing an existing control transfer mechanism of the computing system for Kernel Soft-Reboot. In some embodiments, no other system or user software, barring system firmware, is permitted to execute during the trampoline process. However, direct memory accesses (DMAs) programmed prior to devirtualization may continue to be in flight and complete. In some embodiments, the second hypervisor may undergo additional initialization, validate hardware state, re-initialize hardware, and/or initialize new hardware not previously programmed by the first hypervisor. The devirtualization may also include freezing each of the guest partitions currently running atop the first hypervisor and transmitting a final state of the first hypervisor to the second hypervisor. Thereafter, each of the guest partitions is switched onto the second hypervisor. The second hypervisor then thaws each of the guest partitions. The memory foot print of the first hypervisor may be eagerly or lazily reclaimed by the computing system, subject to overall memory pressure.

Because the embodiments described herein are implemented in a VM environment that is capable of supporting multiple guest partitions simultaneously, each running its own operating system and associated application programs, some introductory discussion of virtualization and hypervisors will be described with respect to <FIG>.

Hypervisor-based virtualization often allows a privileged host operating system (operating in a "parent" partition), as well as multiple guest operating systems (operating in "child" partitions), to share access to hardware of a single computing system simultaneously, with each operating system being given the illusion of having access to a full set of system resources. In order to create this illusion, in some embodiments, a hypervisor at a computing system creates multiple partitions that operate as virtual hardware machines (i.e., VMs), each running its own operating system and associated application programs. Each operating system controls and manages a set of virtualized hardware resources.

<FIG> illustrates an example computing system 100A that includes a hypervisor 140A that hosts one or more VMs 110A, 120A. The computing system 100A includes various hardware devices <NUM>, such as one or more processors <NUM>, one or more storage devices <NUM>, and/or one or more peripheral devices <NUM>. The peripheral devices <NUM> may be configured to perform input and output (I/O) for the computing system 100A. The ellipsis <NUM> represents that there may be different or additional hardware devices included in the computing system 100A. Hereinafter, the physical hardware <NUM> running the hypervisor 140A is also referred to as the "host computing system" 100A (also referred to as a "host"), and the VMs 110A, 120A hosted on the hypervisor 140A are also referred to as the "guest partitions" (also referred to as a "guest").

As illustrated in <FIG>, the hypervisor 140A hosts one or more guest partitions (e.g., VM A <NUM>, VM B <NUM>). The ellipsis <NUM> represents that there may be any number of guest partitions contained at the computing system 100A. The hypervisor 140A allocates to each of the guest partition 110A, 120A a portion of the computing system 100A's physical hardware resources <NUM> and gives the guest partitions 110A, 120A an illusion of owning the resources 114A, 124A. These "illusive" resources are called "virtual" resources. Each of the VMs 110A, 120A uses its own virtual resources 114A, 124A to run its own operating system (OS) 113A, 123A. The operating system 113A, 123A can then allocate the virtual resources 114A, 124A to its various user applications 112A, 122A. For example, guest partition 110A runs OS 113A and user applications 112A. If the OS 113A is a Windows® operating system, the user applications 112A would be Windows® applications. As another example, VM 120A runs OS 123A and user applications 122A. If the OS 123A is a Linux operating system, the user applications 122A would be Linux applications.

Each virtual hardware resource 114A, 124A may or may not have a corresponding physical hardware resource <NUM>. When a corresponding physical hardware resource <NUM> is available, the hypervisor 140A decides how access will be provided to guest partitions 110A, 120A requiring its use. For example, the resource <NUM> may be partitioned or may be time shared. When a virtual hardware resource 114A, 124A does not have a matching physical hardware resource, the hypervisor 140A may emulate the action of the desired hardware resource, typically via a combination of software and other hardware resources that are physically available on the host computing system 100A.

As illustrated in <FIG>, in some embodiments, the hypervisor 140A may be the only software that executes in the highest privilege level defined by the system architecture. Such a system is called a native VM system. Conceptually, in a native VM system, the hypervisor 140A is first installed on the bare hardware, and the guest partitions VM A 110A and VM B 120A are then installed on top of the hypervisor 140A. The guest operating systems 113A, 123A and other less privileged applications 112A, 122A run in levels of privilege lower than that of the hypervisor. This typically means that the privilege level of the guest OS 113A, 123A is likely emulated by the hypervisor 140A.

Alternatively, in some embodiments, a hypervisor is installed on a host platform that is already running an existing OS. Such a system is called a hosted VM system. In a hosted VM system, the hypervisor utilizes the functions already available on the host OS to control and manage resources desired by each of the guest partitions. In a hosted VM system, the hypervisor may be implemented at a user level or a privilege level similar to the host operating system. Alternatively, a portion of the hypervisor is implemented at a user level, and another portion of the hypervisor is implemented at the privileged level.

In some embodiments, one of the guest partitions running on the same computing system may be considered more privileged than the others. <FIG> illustrates such an example embodiment, in which the more privileged guest partition may be called a parent partition 110B, and the rest of the partitions are called child partitions 120B. In some embodiments, the parent partition 110B includes an operating system 113B and/or a virtualization service module 115B. Each of the operating system. 113B and/or the virtualization service module 115B may have direct access to the hardware devices <NUM> via device drivers 114B. Thus, the operating system 113B may also be called a host operating system. Further, the virtualization service module 115B may be capable of creating child partitions 120B using hypercalls. Depending on configuration, the virtualization service module 115B may expose a subset of the hardware resources to each child partition 140B via a virtualization client module 125B. The child partition 120B generally does not have direct access to the physical processor and/or cannot handle real interrupts. Instead, the child partition 120B may use the virtualization client module 125B to obtain a virtual view of the virtual hardware resources 124B.

In some embodiments, the parent partition 110B may also include a VM management service application 112B, which may allow users (e.g., a system administrator) to view and modify the configurations of the virtualization service module 115B. For example, the hypervisor 140B may be a Microsoft® Hyper-V hypervisor, and the parent partition may run a Windows® server. The user interface of the parent partition may provide a window displaying the full user interface of the child partition(s) 120B. Interactions with applications running on a child partition120B may take place within the window. When the host operating system 113B is Windows®, a graphical window could be established on the desktop interface to interact with a child partition 120B on the same platform. The elements 113B, 122B, 123B, and/or 124B of <FIG> are similar to the elements 113A, 122A, 123A, and/or 124A of <FIG>, therefore, will not be further discussed.

Regardless of whether it is a native VM system or a hosted VM system, the relationship between the hypervisor and guest partitions is generally analogous to the relationship between an operating system and application programs in a conventional computing system. In a conventional computing system, the operating system typically works in a privileged level higher than that of the applications, e.g., in kernel mode versus user mode. Similarly, in the VM environment, the hypervisor also operates in a privileged mode higher than the mode of the guest partitions. When a guest partition needs to perform a privileged operation, such as updating page tables, the guest partition will use a hypercall to request such an operation, just as a system call in a conventional operation.

Accordingly, the inventive embodiments described in are applicable to both native VM systems and hosted VM systems, and the term "hypervisor" herein is referred to a hypervisor implemented in any type of VM systems.

To further understand how a hypervisor operates, we also need to understand how the hypervisor maintains each guest partition's state. In a computing system, the architected state of the computing system is contained within, and maintained by, the hardware resources of the computing system. There is usually an architected hierarchy of state resources, ranging from registers at one end of the hierarchy to secondary storage (e.g., hard drives) at the other end of the hierarchy.

In a VM environment, each guest partition has its own architected state information, and there may or may not be adequate physical resources in the host computing system to map each element of a guest's state to its natural level in the host's memory hierarchy. For example, a guest's register state may actually be held in the main memory of the host platform as part of a register context block.

In normal operations, the hypervisor 140A periodically switches control amongst the guest partitions 110A, 120A. As operations on the guest's state are performed, the state maintained on the host computing system 100A is modified just as it would be on the guest operating system 113A, 123A. In some embodiments, the hypervisor 140A constructs an isomorphism that maps a state of virtual guest operating system 113A, 123A to a state of physical host computing system 100A.

<FIG> illustrates a schematic diagram of an isomorphism <NUM> between a guest partition <NUM> and a host computing system <NUM>. The isomorphism <NUM> maps the guest state(s) <NUM>, <NUM> to the host state(s) <NUM>, <NUM> using function <NUM>, i.e., V(State A) = State A', and V(State B) = State B'. For a sequence of operations in the guest partition <NUM> that modifies the state of the guest partition <NUM> from State A <NUM> to State B <NUM>, there is a corresponding sequence of operations in the host computing system <NUM> that modifies the state of the host computing system from State A' <NUM> to State B' <NUM>. The isomorphism <NUM> between a guest partition <NUM> and the host computing system <NUM> is managed by a hypervisor (e.g., 140A, 140B).

In embodiments, there are two basic ways of managing guest states so that this VM isomorphism is accomplished. One way is to employ a level of indirection, by holding the state for each guest in fixed locations in the host computing system's memory hierarchy with a hypervisor managed pointer indicating the guest state that is currently active. As the hypervisor switches among guest partitions, it changes the pointer to match the current guest. <FIG> illustrates such an example embodiment 300A of state management by indirection. Referring to <FIG>, the host computing system's memory <NUM> managed by hypervisor stores register value for each VMs A and B in the register context block <NUM>, <NUM>. The register block point 311A of the processor 310A points to the register context block <NUM> of the currently active guest partitions (e.g., VM B). When a different guest partition is activated, the hypervisor changes the pointer 311A stored at the processor to point to the register context block <NUM>, <NUM> of the activated guest partition, and loads program counter to point to the activated VM program and start execution.

Another way of managing guest states is to copy a guest's state information to its natural level in the memory hierarchy whenever it is activated by the hypervisor and to copy it back when a different guest is activated. <FIG> illustrates such an example embodiment 300B of guest state management by copying. As illustrated in <FIG>, the host memory <NUM> managed by hypervisor similarly stores register value for each guest partitions VMs A and B in the register context block <NUM>, <NUM>. However, unlike that in <FIG>, the hypervisor here copies the entire guest register contents <NUM> into the processor 310B's register file 311B at the time the VM B is activated (after saving the registers of the previous guest back to the memory <NUM>).

The choice between indirection and copying can depend on, for example, the frequency of use and whether the guest state being managed by the hypervisor is held in a different type of hardware resources than on a native system. For frequently used state information, such as the general-purpose registers, it may be preferable to swap the state of the virtual machine to the corresponding physical resource each time the virtual machine is activated. However, as illustrated in <FIG> and <FIG>, in either case, the register value <NUM>, <NUM> of each VM is often held in the main memory <NUM> of the host platform as part of a register context block.

In addition to VMs' state management, memory management is also worth discussing. In a VM environment, each of the guest partitions has its own set of virtual memory tables, which are also called "first level" memory page tables. Address translation in each of the first level memory page tables transforms addresses in its virtual address space to locations in guest physical memory. The guest physical memory here does not correspond to the host physical memory on the host computing system. Instead, the guest physical address (GPA) undergoes a further mapping to determine the address in the physical memory of the host hardware, also called host physical address (HPA). This GPA-to-HPA mapping is performed by another set of virtual memory tables of the host computing system, which are also called "second level" or nested memory page tables. Note that the combined total size of the guest physical memory of all the guests could be bigger than the actual physical memory on the system. In embodiments, the hypervisor maintains its own swap space, distinct from the swap space of each of the guests, and hypervisor manages physical memory by swapping guest physical pages into and out of its own swap spaces. Further, the states of every virtually or physically allocated page, and its corresponding attributes, is stored in a list, called page frame number (PFN) list. A track of virtually or physically allocated pages are stored in a database, called PFN database.

<FIG> illustrates an example embodiment <NUM> of memory virtualization using multiple-stage memory page tables <NUM> and <NUM>. Each entry in the first level memory table <NUM> maps a location (e.g., a PFN) in the PFN database <NUM> of virtual memory to a location (e.g., a PFN) in a PFN database <NUM> of guest physical memory. As illustrated in <FIG>, the portion of PFN database <NUM> tracks the virtually allocated pages of the programs running on VM A; and the portion of PFN database <NUM> tracks the guest VM's physical memory of VM A. Further, in order to convert a GPA to a HPA, the hypervisor also maintains a second level memory page table <NUM> mapping the guest physical page to host physical pages. There is also a PFN database <NUM> that tracks the physical memory of the host computing system.

As illustrated in <FIG>, the physical page frame numbered <NUM> is assigned to a guest physical page frame numbered <NUM>, which is then assigned to a virtual memory page frame numbered <NUM>. Similarly, a physical page frame <NUM> is assigned to a guest physical page frame numbered <NUM>, which is then assigned to a virtual memory page frame numbered <NUM>. The remaining physical memory pages may be allocated either to other VMs or to the hypervisor itself. These remaining physical memory pages, including the ones allocated to the hypervisor itself to record register values of each VM (e.g., memory <NUM>), are also tracked by the PFN database <NUM>.

<FIG> is merely a schematic diagram for illustrating a simplified concept of memory management using multi-stage page tables. Additional mechanisms may be implemented to achieve the same or similar memory management purposes. For example, in some embodiments, page translation is supported by a combination of a page table and a translation lookaside buffer (TLB).

Having described virtual environments and how a hypervisor manages and virtualizes various hardware resources with reference to <FIG>, specific embodiments of hypervisor hot restart will now be explained with respect to <FIG> and <FIG>.

<FIG> illustrates an example computing system 500A, in which a hypervisor hot restart is enabled. The computing system 500A may contain a native VM system as illustrated in <FIG>, in which the hypervisor 520A is the only software that executes in the highest privilege level defined by the system architecture. Alternatively, the computing system 500A may contain a hosted VM system, in which the hypervisor 520A is installed on a computing system that is already running a host OS.

Regardless of whether the computing system 500A contains a native VM system or a hosted VM system, during a hypervisor hot restart, a new partition, called a service partition <NUM>, is generated. In embodiments, the service partition <NUM> is treated differently from the other partition types. At the time the service partition <NUM> is created, hardware resources, including at least some processor resource(s) and memory resource(s), are allocated to the service partition <NUM>. In some embodiments, the allocation of hardware resources may be based on user input(s). In alternate embodiments, the hypervisor 520A or the component created the service partition <NUM> automatically allocates a predetermined portion of processor resources and/or memory resources to the service partition <NUM>.

The allocation of processor resource may specify a total amount of processing power the service partition <NUM> needs and leave the allocation of available processors to workload management software, or the service partition <NUM> or the hypervisor 520A may specify that specific processors in the system be dedicated for the service partition <NUM>'s use. The service partition <NUM> or the hypervisor 520A can specify that the service partition <NUM> needs a certain number of processors but that the service partition <NUM> is willing to share these processors with other partitions. For example, if a total of eight processing units is needed by the service partition <NUM>, the service partition <NUM> or the hypervisor 520A can specify either that the service partition <NUM> needs eight processors dedicated to itself or that the service partition <NUM> needs sixteen processors, but only half the available compute power in each of the processors. The allocation of memory (including RAM and/or hard disk) may specify an amount of memory in chunks of a particular granularity, e.g., 1MB.

The service partition <NUM> is then initialized. Any component in the trusted computing base (TCB) can be tasked to create and/or initialize the service partition <NUM>. In some embodiments, the first hypervisor 520A is part of the TCB, and the first hypervisor 520A generates and/or initializes the service partition <NUM>. The initialization process includes bootstrapping, which involves a series of actions, each action bringing to life a capability that allows the next action to be performed, until eventually the entire system is brought to life. In some embodiments, the hypervisor 520A builds a loader block for a second hypervisor <NUM>. The execution of the initializing code allows other aspects of the service partition <NUM> to be initialized. As illustrated in <FIG>, unlike other regular guest partitions 540A, 550A, in which a conventional OS 541A, 551A is loaded, a hypervisor <NUM> is loaded into the service partition <NUM>. For clarity, hereinafter, the hypervisors 520A and <NUM> are called a first hypervisor 520A and a second hypervisor <NUM>. In some embodiments, initializing the second hypervisor <NUM> may also include offering the second hypervisor read-only specific access to certain physical resources <NUM>.

The purpose of hypervisor hot restart is to eventually replace the first hypervisor 520A with the second hypervisor <NUM> with minimal to imperceptible interruption to guest virtual machines. Before the second hypervisor <NUM> replaces the first hypervisor 520A, the first hypervisor 520A initializes the second hypervisor <NUM> with the available system invariants (e.g., features that the hardware resource(s) <NUM> support), and then synchronizes runtime state <NUM> with the second hypervisor <NUM>. In some embodiments, the runtime state <NUM> is stored in a memory (e.g., a RAM) that is managed by the hypervisor 520A. As described with respect to <FIG>, <FIG>, and <FIG>, runtime state <NUM> may include hardware architectural state like register value <NUM>, <NUM> of each guest partition (e.g., guest A 540A, 550A), virtualization hardware state like second level memory page table(s) <NUM> of each guest partition (e.g., guest A 540A, 550A), and/or software defined state like PFN database <NUM> associated with first hypervisor 520A.

In some embodiments, the communications between the first hypervisor 520A and the second hypervisor <NUM> during the initialization and synchronization are orchestrated via an orchestrator 530A. The first hypervisor 520A allows and supports the orchestrator 530A to register and complete certain requests received from the second hypervisor <NUM>. The orchestrator 530A is a software component of the host computing system 500A that is configured to arrange communications between the first hypervisor 520A and the second hypervisor <NUM>. In some embodiments, the first hypervisor 520A is part of the system's trusted computing base (TCB), and the orchestrator 530A is executed in a trusted execution environment (TEE) to uphold the trustworthiness of inputs generated for the second hypervisor.

In some embodiments, the orchestrator 530A may use inverse hypercalls to transmit state to second hypervisor <NUM>, and/or the second hypervisor <NUM> may use intercepts to request from the orchestrator 530A some service (e.g., property of a physical resource that the second hypervisor <NUM> does not have access to). In some embodiments, the orchestrator 530A shares a portion of a memory page <NUM> with the second hypervisor <NUM>, such that data can be efficiently passed amongst the orchestrator 530A, the first hypervisor 520A, and the second hypervisor <NUM>. In some embodiments, the orchestrator 530A issues inverse hypercalls to the second hypervisor following the hypercall application binary interface (ABI) to migrate relevant state.

As described above, once the service partition <NUM> is created and loaded, the second hypervisor <NUM> first needs to obtain various system invariants, such as the features of the hardware resources <NUM>. In some embodiments, there is a strict coupling between the features that the first hypervisor 520A sees and the features the second hypervisor <NUM> will see.

Alternatively, in other embodiments, there is loose coupling between the first and second hypervisors. The orchestrator 530A may query the first hypervisor via hypercalls and/or query the hardware resource(s) directly to obtain features and properties of the hardware resource(s) pertinent to initialization of the second hypervisor <NUM>. For example, the second hypervisor <NUM> may try to read a CPUID or MSR value. However, being in a virtual machine itself, it may only have access to the virtualized values, not the physical values of the underlying physical computing system. In such a case, the orchestrator 530A may invoke a hypercall to the first hypervisor 520A to obtain the corresponding physical value of the corresponding physical resource. For example, the second hypervisor <NUM> may want to query processor support for XSAVE features and instructions. If the first hypervisor 520A, doesn't support virtualizing XSAVE, the orchestrator 530A may query the properties of the underlying processor and determine that the processor supports XSAVE. The obtained query results may also be stored in the shared memory page <NUM>, such that the second hypervisor <NUM> would have access to the query results. In some cases, the second hypervisor <NUM> may not be able to obtain every feature of the hardware resources <NUM>. In some embodiments, such features will be left as unknown during the initialization, and be obtained later after devirtualization.

In addition to obtaining the system invariants, the runtime state of the first hypervisor 520A is also synchronized with the second hypervisor <NUM>. The runtime state includes at least dynamic system state and logical software state. The dynamic system state includes guest visible hardware architecture state, including but not limited to, general purpose registers, and control registers. The dynamic system state also includes virtualization instruction set architecture ISA specific hardware state, including but not limited to, state of guest VMs, such as virtual central processing unit (CPU) state, second level page tables (also known as nested page tables), and a list of assigned hardware devices (e.g., network card). The synchronizing the first hypervisor's runtime state <NUM> includes at least synchronizing one or more second level memory page tables and/or one or more PFN databases to the second hypervisor <NUM>. The second level memory tables are the memory page tables that map the GPA of each guest partition to the HPA (e.g., the second level memory page table <NUM> of <FIG>). The synchronized PFN database may include the PFN databases that track physical memory of the host computing system (e.g., PFN database <NUM>).

However, after the initial synchronization of runtime state between the two hypervisors 520A and <NUM>, and before the second hypervisor <NUM> is devirtualized, the guest partitions 540A, 550A are still running on top of the first hypervisor 520A and those guest partitions 540A, 550A still may invoke hypercalls. Once an incoming hypercall is serviced by the first hypervisor 520A, the state of the guest partition 540A or 550A that invoked the hypercall changes, and the second hypervisor <NUM>'s previously synced state is no longer accurate. To solve this problem, the incoming hypercalls (before the devirtualization of the second hypervisor) also need to be recorded and synchronized with the second hypervisor <NUM>.

In some embodiments, the orchestrator 530A is also tasked with recording and synchronizing each incoming hypercall. For example, when the first hypervisor 520A receives and services a hypercall invoked by a guest partition 540A, 540B, the orchestrator 530A records the hypercall and a log of actions that occurred during the service of the hypercall by the first hypervisor 520A. At the same time, the orchestrator 530A feeds the hypercall to the second hypervisor <NUM> via an inverse hypercall. In some embodiments, an instruction point of the second hypervisor <NUM> is in a hypercall dispatch loop, such that when an inverse hypercall is fed to the second hypervisor <NUM>, the second hypervisor <NUM> processes it. Receiving the inverse hypercall, the second hypervisor <NUM> switches the context of the guest partition 540A, 540B that invoked the hypercall and processes the inverse hypercall to rebuild necessary software state and/or dormant hardware state. Upon completion of the inverse hypercall, the second hypervisor <NUM> then notifies the orchestrator 530A the completion.

In some cases, a sequence of hypercalls are serviced within a short time period, and the orchestrator 530A may only send to the status associated with the last or relevant operation of the sequence of hypercalls. In such a case, the second hypervisor <NUM> may only replay partial logs of actions, i.e., perform a condensed replay. This process may repeat as many times as necessary until the second hypervisor <NUM> is completely or at least substantially synchronized with the first hypervisor 520A. Thereafter, the second hypervisor <NUM> is devirtualized from the service partition <NUM> to replace the first hypervisor 520A. The devirtualization includes the first hypervisor 520A trampolining to the second hypervisor and ceding all physical hardware control to the second hypervisor <NUM>. In some embodiments, the trampoline may be achieved by reusing an existing control transfer mechanism of the computing system 500A for Kernel Soft Reboot.

In some embodiments, the second hypervisor <NUM> may undergo additional initialization, validate hardware state, re-initialize hardware, and/or initialize new hardware not previously programmed by the first hypervisor. In some embodiments, the devirtualization may include the first hypervisor 520A freezing all the guest partition(s) 540A, 550A, sending the final state details to the second hypervisor <NUM>, and transferring the guest partition(s) onto the second hypervisor. The second hypervisor then thaws each of the guest partition(s). Once the second hypervisor <NUM> begins devirtualizing, the first hypervisor 520A is effectively terminated. No other system software, user software, and/or system firmware is permitted to execute during devirtualization. However, DMAs programmed prior to devirtualization may continue to be in flight and complete.

For example, when a guest partition (e.g., child A 540A, 550A) is provided access to a physical device, the guest partition may initiate DMAs with its GPAs as the source or target of the DMA operation. A second level page table (which may be programmed into an input-output memory management unit (IOMMU) by the first hypervisor) translates the GPA to HPA in addition to performing necessary permission checks and provides an HPA to the DMA engine. As noted earlier, the first hypervisor transmitted/synchronized architectural guest state as well as architectural virtualization state to the second hypervisor. The architectural virtual state, among other things, includes second level page tables for the CPU, IOMMU and/or device domain information. Thus, when the second hypervisor devirtualizes and reinitializes the hardware, the second hypervisor carefully programs the hardware with new page-table constructed by itself during the earlier synchronization stage. Since the effective address translation and permission of the new page-table are identical despite being two different instances, all new translation requests from the DMA engine can continue to be in flight and use the same page-table successfully with no loss of fidelity.

Additionally, in some embodiments, the memory foot print of the first hypervisor 520A may be eagerly or lazily reclaimed by the computing system, subject to overall memory pressure.

<FIG> illustrates another example computing system 500B, in which a hypervisor hot restart is enabled. The computing system 500B corresponds to the computing system 100B of <FIG>, in which one of the guest partitions (e.g., a parent partition 110B, 530B is more privileged than the rest of the guest partitions (e.g., child partitions 120B, 540B, 550B). In some embodiments, the parent partition 530B includes a host operating system and/or a virtualization service module 531B, either one of which may be configured to create and manage the child partitions 540B and 550B and handle various system management functions and device drivers. Similar to the embodiment illustrated in <FIG>, a service partition <NUM> is created, a second hypervisor <NUM> is initialized within the service partition <NUM>, and eventually, the first hypervisor 520B is replaced with the second hypervisor <NUM> to complete the hypervisor hot restart.

In some embodiments, in the computing system 500B, since the parent partition 530B has a higher level privilege than the child partitions 540B, 550B, an orchestrator 532B may be implemented in the parent partition 530B as part of the host operating system or a virtualization service module 531B of the parent partition 530B. The orchestrator 532B functions, similarly to the orchestrator 530B, to orchestrate the communications between the first hypervisor 520B and the second hypervisor <NUM>. The elements 541B, 551B of <FIG> are similar to the elements 541A, 551A of <FIG>, therefore, will not be further discussed.

Additionally, in some embodiments, the hot restart of the hypervisor 520A, 520B may also be accompanied with a kernel soft reboot or reset of a guest partition 540A, 550A, 530B, 540B, 550B. In a kernel soft reboot, a guest partition 540A, 550A, 530B, 540B, or 550B may be recreated and initialized as a new partition, and the corresponding guest partition's runtime state is synchronized with the new partition. Once all the runtime state of the guest partition is synchronized with the new partition, the new partition can replace the corresponding guest partition to complete the kernel soft reboot of the corresponding guest partition. In some embodiments, only the parent partition 530B is hot restarted accompanying a hot restart of hypervisor 520B. Alternatively, or in addition, each of the guest partitions 540A, 550A and/or child partitions 540B, 550B is hot restarted with a hot restart of hypervisor 520A, 520B.

Some embodiments restart VM-related components within the parent partition 530B, without restarting the host operating system within the parent partition 530B. For example, some embodiments restart the virtualization service module 531B in connection with a hypervisor hot restart, without restarting a host operating system 113B. In this way, it is possible to upgrade and restart a hypervisor, together with its operating-system level management components, without restarting the host operating system 113B.

The hypervisor hot restart described herein substantially reduces the interruption time caused to the running guest VMs, unlike conventional hypervisor restarts. Unlike a regular hypervisor restart, which could take several minutes, depending on the number of VMs being hosted and the amount of hardware resources being managed, the second hypervisor <NUM> described herein is initialized and synchronized while the first hypervisor 520A or 520B was still running; thus, only a short period of freeze (e.g., less than a second or a few seconds) would occur at each running guest partition, which may not even be noticed by users.

<FIG> illustrates a flowchart of an example method <NUM> for hypervisor hot restart. The method <NUM> is implemented on a computing system that may correspond to the computing system 100A, 100B or 500A, 500B. The method <NUM> includes executing a first hypervisor (<NUM>). The first hypervisor, in turn, creates one or more guest partitions (<NUM>), each of which may host a guest operating system. The purpose of a hypervisor hot restart is to replace the first hypervisor with a new hypervisor. When a hypervisor hot restart is performed, the computing system creates a service partition (<NUM>) and initializes a second hypervisor (<NUM>) within the service partition. Next, at least a portion of runtime state of the first hypervisor is synchronized with the second hypervisor (<NUM>). Once the synchronization is completed, or substantially completed, the computing system devirtualizes the second hypervisor to replace the first hypervisor with the second hypervisor (<NUM>).

<FIG> illustrates a flowchart of an example method <NUM> for initializing the service partition, which corresponds to the step <NUM> of <FIG>. The method <NUM> includes generating, by the fist hypervisor, a loader block for the second hypervisor (<NUM>). The loader block is a logical construct describing some of static system properties and resources for initialization. The method <NUM> also includes obtaining one or more system invariants (<NUM>) and sharing the one or more system invariants with the second hypervisor (<NUM>). The second hypervisor is then initialized based on the one or more system invariants (<NUM>). In some embodiments, the system invariants include one or more features of the hardware resource(s) of the computing system (<NUM>). In some embodiments, there is strict coupling between the first hypervisor and the second hypervisor (<NUM>). In some embodiments, there is loose coupling between the first and second hypervisors. In such a case, the one or more feature(s) of the hardware resource(s) may be obtained from the first hypervisor via hypercalls (<NUM>) or from the hardware resource(s) directly via system call(s) (<NUM>).

<FIG> illustrates a flowchart of an example method <NUM> for synchronizing a portion of runtime state of the first hypervisor with the second hypervisor, which corresponds to the step <NUM> of <FIG>. The method <NUM> may be performed by the orchestrator 530A, 532B of <FIG> or <FIG>. The method <NUM> includes gathering runtime state of the first hypervisor (<NUM>). The method <NUM> also includes sharing the runtime state with the second hypervisor (<NUM>), which may be performed by the orchestrator 530A, 532B via an inverse hypercall" which may or may not additionally rely on a shared memory page to facilitate signaling and/or message/data passing. The shared runtime state of the first hypervisor is then duplicated by the second hypervisor (<NUM>). Before the second hypervisor is devirtualized, the guest partitions running on top of the first hypervisor continues to operate and still can make hypercalls, the gathered runtime state will be inaccurate soon. Thus, this process may repeat several times until the second hypervisor and the first hypervisor are completely or substantially in sync.

<FIG> illustrates a flowchart of an example method <NUM> for gathering runtime state of the first hypervisor, which corresponds to the step <NUM> of <FIG>. The method <NUM> includes gathering current data related to one or more second level memory page tables and one or more PFN(s) (<NUM>-A). The method <NUM> also includes gathering data related to incoming hypercalls <NUM>-B. In particular, when a hypercall is invoked by a guest partition, the first hypervisor will receive (<NUM>-C) and service (<NUM>-D) the received hypercall. The computing system (e.g., the orchestrator 530A or 532B) then records the received hypercall and a log of actions occurred during the service of the hypercall (<NUM>-E).

<FIG> illustrates a flowchart of an example method <NUM> for sharing runtime state of the first hypervisor with the second hypervisor, which corresponds to the step <NUM> of <FIG>. The method <NUM> includes feeding one or more second level page table(s) and one or more PFN(s) to the second hypervisor (<NUM>-A). The method <NUM> also includes feeding the incoming hypercall to the second hypervisor (<NUM>-B) and feeding the log of actions occurred during the service of the hypercall (by the first hypervisor) to the second hypervisor (<NUM>-C).

<FIG> illustrates a flowchart of an example method <NUM> for duplicating runtime state of the first hypervisor with the second hypervisor, which corresponds to the step <NUM> of <FIG>. The method <NUM> includes receiving an inverse hypercall from the orchestrator in the second hypervisor (<NUM>-A). The second hypervisor then switches the context of the guest partition that invoked the hypercall (<NUM>-B) and processes the inverse hypercall to rebuild necessary software state and/or dormant hardware state (<NUM>-C). upon completion of the processing of the inverse hypercall, the second hypervisor notifies the orchestrator (<NUM>-D).

<FIG> illustrates a flowchart of an example method <NUM> for devirtualizing the second hypervisor, which corresponds to the step <NUM> of <FIG>. The method <NUM> includes freezing each guest partition that is currently running on top of the first hypervisor (<NUM>). The final state of the first hypervisor is then transmitted to the second hypervisor (<NUM>). Next, each of the guest partitions is switched onto the second hypervisor (<NUM>), and the second hypervisor thaws each of the guest partitions (<NUM>). Finally, the first hypervisor is terminated (<NUM>).

Note, even though the example embodiments described above are implemented in a special partition (e.g., a service partition) for servicing a hypervisor, the overall techniques of pre-initializing data structures and/or migrating/synchronizing runtime state to save time may be used for any portion of any VM.

Finally, because the principles described herein are implemented in the context of a computing system (e.g., computing system 100A, 100B of <FIG>, <FIG> and/or 500A, 500B of <FIG> or <FIG>) some introductory discussion of a computing system will be described with respect to <FIG>.

As illustrated in <FIG>, in its most basic configuration, a computing system <NUM> typically includes at least one hardware processing unit <NUM> and memory <NUM>. The processing unit <NUM> may include a general-purpose processor and may also include a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit. The memory <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage, such as physical storage media. If the computing system is distributed, the processing, memory and/or storage capability may be distributed as well.

The computing system <NUM> also has thereon multiple structures often referred to as an "executable component. " For instance, memory <NUM> of the computing system <NUM> is illustrated as including executable component <NUM>. The term "executable component" is the name for a structure that is well understood to one of ordinary skill in the art in the field of computing as being a structure that can be software, hardware, or a combination thereof. For instance, when implemented in software, one of ordinary skill in the art would understand that the structure of an executable component may include software objects, routines, methods, and so forth, that may be executed on the computing system, whether such an executable component exists in the heap of a computing system, or whether the executable component exists on computer-readable storage media.

In such a case, one of ordinary skill in the art will recognize that the structure of the executable component exists on a computer-readable medium such that, when interpreted by one or more processors of a computing system (e.g., by a processor thread), the computing system is caused to perform a function. Such a structure may be computer-readable directly by the processors (as is the case if the executable component were binary). Alternatively, the structure may be structured to be interpretable and/or compiled (whether in a single stage or in multiple stages) so as to generate such binary that is directly interpretable by the processors. Such an understanding of example structures of an executable component is well within the understanding of one of ordinary skill in the art of computing when using the term "executable component.

The term "executable component" is also well understood by one of ordinary skill as including structures, such as hardcoded or hard-wired logic gates, that are implemented exclusively or near-exclusively in hardware, such as within a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit. In this description, the terms "component," "agent," "manager," "service," "engine," "module," "virtual machine," or the like may also be used.

In the above description, embodiments are described with reference to acts that are performed by one or more computing systems. For example, such computer-executable instructions may be embodied in one or more computer-readable media that form a computer program product. If such acts are implemented exclusively or near-exclusively in hardware, such as within an FPGA or an ASIC, the computer-executable instructions may be hardcoded or hard-wired logic gates.

While not all computing systems require a user interface, in some embodiments, the computing system <NUM> includes a user interface system <NUM> for use in interfacing with a user. The user interface system <NUM> may include output mechanisms 712A as well as input mechanisms 712B. The principles described herein are not limited to the precise output mechanisms 712A or input mechanisms 712B as such will depend on the nature of the device. However, output mechanisms 712A might include, for instance, speakers, displays, tactile output, holograms, and so forth. Examples of input mechanisms 712B might include, for instance, microphones, touchscreens, holograms, cameras, keyboards, mouse or other pointer input, sensors of any type, and so forth.

Embodiments described herein may comprise or utilize a special purpose or general-purpose computing system, including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below.

Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computing system.

Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general-purpose computing system, special purpose computing system, or special purpose processing device to perform a certain function or group of functions. Alternatively or in addition, the computer-executable instructions may configure the computing system to perform a certain function or group of functions. The computer-executable instructions may be, for example, binaries or even instructions that undergo some translation (such as compilation) before direct execution by the processors, such as intermediate format instructions such as assembly language, or even source code.

Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computing system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, pagers, routers, switches, data centers, wearables (such as glasses) and the like.

The remaining figures may discuss various computing system which may correspond to the computing system <NUM> previously described. The computing systems of the remaining figures include various components or functional blocks that may implement the various embodiments disclosed herein as will be explained. The various components or functional blocks may be implemented on a local computing system or may be implemented on a distributed computing system that includes elements resident in the cloud or that implement aspect of cloud computing. The various components or functional blocks may be implemented as software, hardware, or a combination of software and hardware. The computing systems of the remaining figures may include more or less than the components illustrated in the figures and some of the components may be combined as circumstances warrant. Although not necessarily illustrated, the various components of the computing systems may access and/or utilize a processor and memory, such as processor <NUM> and memory <NUM>, as needed to perform their various functions.

Claim 1:
A computing system (100A, 500A) comprising:
one or more processors (<NUM>); and
one or more computer-readable media having stored thereon computer-executable instructions that are structured such that, when executed by the one or more processors (<NUM>), the computer-executable instructions cause the computing system (100A, 500A) to perform the following:
execute (<NUM>) a first hypervisor (520A) that creates (<NUM>) one or more guest partitions (540A, 550A);
create (<NUM>) a service partition (<NUM>);
initialize (<NUM>) a second hypervisor (<NUM>) within the service partition (<NUM>) with static system state of the first hypervisor (520A) by an orchestrator (530A), wherein the orchestrator (530A) is a software component configured to arrange communications between the first hypervisor (520A) and the second hypervisor (<NUM>);
synchronize (<NUM>) at least a portion of runtime state (<NUM>) of the first hypervisor (520A) to the second hypervisor (<NUM>), including the orchestrator (530A) configured to gather (<NUM>) the at least a portion of the runtime state (<NUM>) of the first hypervisor (520A) and to share (<NUM>) the at least a portion of the runtime state (<NUM>) with the second hypervisor (<NUM>) over inverse hypercalls, calls from the orchestrator (530A), a privileged component, to the second hypervisor (<NUM>); and
devirtualize (<NUM>) the second hypervisor (<NUM>) from the service partition (<NUM>) to replace the first hypervisor (520A), including the first hypervisor (520A) trampolining to the second hypervisor (<NUM>) by ceding all physical hardware control to the second hypervisor (<NUM>).