Patent Description:
The updating of host operating systems traditionally requires a reboot and therefore workload downtime. For virtualization hosts running virtual machine (VM) instances, downtime also implies downtime of the workloads in hosted virtual machines. Virtualization technology often allows running VM contexts to be preserved by pausing execution of VMs and writing their associated RAM contexts to disk. However, such operations are typically relatively slow and result in detectable outages, especially where the storage subsystem is slow or the amount of memory to be preserved and therefore the amount of IO required would be very large.

In some update operations, rather than writing VM memory contents to disk, the host memory pages used for VM memory are catalogued such that the virtualization software can reassemble the state of VMs across the update operation. During the update operation, the current operating system catalogues the VM artifacts, tears down the VM, prepares the new software for execution, unloads the current software, and loads the updated software which reads the meta-data and reconstructs the VMs using the same memory pages as previously used. While this can be faster than writing memory contents to disk, the duration of the operation and therefore the window of downtime is still negatively affected by several factors, including: <NUM>) the quantity and continuity of memory to be preserved <NUM>) the destruction (including un-mapping) and re-construction (including re-mapping) of VM artifacts, especially the guest to host page mappings. In addition, VM instances which have assigned devices cannot participate in the update operation because assigned device cannot be paused or quiesced across the operation nor can their state be saved and restored.

<CIT> discloses a memory sharing method of virtual machines through the combination of kernel same page merging (KSM) and pass-through, including: a virtual machine manager judging whether operating systems of guests use an input/output memory management unit (IOMMU), if not, not participating in shared mapping of a KSM technology; if yes, judging memory pages of each guest to confirm whether the pages are mapping pages, if yes, remain the mapping pages into a host; and if not, on the premise of keeping the properties of Pass-through, using the KSM technology for all non-mapping pages to merge the memory pages with same contents among various virtual machines and perform write protection processing simultaneously. The guest memory pages are divided into those special for direct access to memory (DMA) and those for non-DMA purpose, then the KSM technology is only selectively applied to the non-DMA pages.

<NPL> discloses a survey of GPU virtualization techniques and their scheduling methods. A range of virtualization techniques implemented at the GPU library, driver, and hardware levels is reviewed. Furthermore, GPU scheduling methods that address performance and fairness issues between multiple virtual machines sharing GPUs are reviewed.

<CIT> discloses techniques for updating a host operating system on a server while maintaining virtual machines running on the server. An updated host operating system is copied to the server. The currently active host operating system freezes the virtual machines but leaves them resident in RAM. The allocations and state for each virtual machine is copied to RAM or local storage. The active host operating system is shut down. Instead of issuing a command to reboot the server after it finishes shutting down, the active host operating system transfers execution to a loader. The loader reads the kernel of the updated host operating system into RAM along with an allocation map for the virtual machines and instructions to resume the virtual machines. The loader transfers execution to the updated host operating system entry point, and the updated host operating system loads the states of the virtual machines and resumes them.

<NPL> discloses an OS update mechanism that uses a userspace checkpoint-and-restart mechanism, by using a data structure for checkpointing and a memory persistence mechanism across update, combined with an in-place kernel switch.

It is therefore the object of the present invention to provide an improved computing system for suspending and resuming an operation of a virtual machine, a corresponding method and a computer readable hardware storage device having stored thereon computer-executable instructions which, when executed, cause a computing system to perform a corresponding method.

At least one embodiment disclosed herein is related to computing systems and methods for performing a servicing operation on a virtual machine (VM). A computing system has first virtual machine components and second VM components that remain loaded in computing system physical hardware during the servicing operation. An operation of a VM running the first and second VM components is suspended so that the servicing operation for the VM can be performed. The VM has devices that are directly attached to it. A state of the first VM components is saved. An identification pointer for the second VM components is saved in a portion of the computing system physical memory without removing any underlying data structures of the second VM components from the computing system physical hardware. The directly attached devices remain configured as attached to the VM and configured to be in communication with the VM while the VM is suspended and while the servicing operation is performed. The first VM components are shut down and then restored at the completion of the servicing operation using the saved state. The restored first VM components are reconnected to the second VM components using the identification pointers. The operation of the VM is restored.

The embodiments disclosed herein provide improvements for updating computing system running VMs over previous update methods.

At least one embodiment disclosed herein is related to computing systems and methods for performing a servicing operation on a virtual machine (VM). A computing system has first virtual machine components and second VM components that remain loaded in computing system physical hardware during the servicing operation. An operation of a VM running the first and second VM components is suspended so that the servicing operation for the VM can be performed. The VM has devices that are directly attached to it. A state of the first VM components is saved. An identification pointer for the second VM components is saved in a portion of the computing system physical memory without removing any underlying data structures of the second VM components from the computing system physical hardware. The directly attached devices remain configured as attached to the VM and configured to re in communication with the VM while the VM is suspended and while the servicing operation is performed. The first VM components are shut down and then restored at the completion of the servicing operation using the saved state. The restored first VM components are reconnected to the second VM components using the identification pointers. The operation of the VM is restored.

There are various technical effects and benefits that can be achieved by implementing aspects of the disclosed embodiments. By way of example, at least some of the embodiments the operation is not affected by the quantity and continuity of the guest to host memory mappings as will be explained in more detail to follow. This is advantageously an improvement on current methods where the memory stack must be walked to determine guest to host mappings. As will be appreciated, it may be timely and take computing system resources to walk the memory stack when there is a large number of discontinuous memory page mappings.

It is additionally an improvement on traditional methods as in the disclosed embodiments some of the VM components have their underlying data structures remain loaded in the computing system physical resources such as memory and processing resources. This saves on time as these components do not have to be rebuilt. In traditional methods, these components are destroyed and rebuilt, thus taking more time.

Further, the embodiments disclosed herein provide the technical improvement of allowing devices to remain directly attached to a VM during an update process. This ability is not present in traditional update methods.

Further, the technical effects related to the disclosed embodiments can also include reduced power consumption, and can also include efficiency gains, as the quicker time will require less processing resources, which can in turn be used by other process of the computing system.

Some introductory discussion of a computing system will be described with respect to <FIG>. Computing systems may, for example, be handheld devices, appliances, laptop computers, desktop computers, mainframes, distributed computing systems, datacenters, or even devices that have not conventionally been considered a computing system, such as wearables (e.g., glasses). In this description and in the claims, the term "computing system" is defined broadly as including any device or system (or combination thereof) that includes at least one physical and tangible processor, and a physical and tangible memory capable of having thereon computer-executable instructions that may be executed by a processor.

As illustrated in <FIG>, in its most basic configuration, a computing system <NUM> typically includes at least one hardware processing unit <NUM> and memory <NUM>. The memory <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computing system is distributed, the processing, memory and/or storage capability may be distributed as well.

The computing system <NUM> also has thereon multiple structures often referred to as an "executable component". For instance, the memory <NUM> of the computing system <NUM> is illustrated as including executable component <NUM>. The term "executable component" is the name for a structure that is well understood to one of ordinary skill in the art in the field of computing as being a structure that can be software, hardware, or a combination thereof. For instance, when implemented in software, one of ordinary skill in the art would understand that the structure of an executable component may include software objects, routines, methods, and so forth, that may be executed on the computing system, whether such an executable component exists in the heap of a computing system, or whether the executable component exists on computer-readable storage media.

In such a case, one of ordinary skill in the art will recognize that the structure of the executable component exists on a computer-readable medium such that, when interpreted by one or more processors of a computing system (e.g., by a processor thread), the computing system is caused to perform a function. Such structure may be computer-readable directly by the processors (as is the case if the executable component were binary). Alternatively, the structure may be structured to be interpretable and/or compiled (whether in a single stage or in multiple stages) so as to generate such binary that is directly interpretable by the processors. Such an understanding of example structures of an executable component is well within the understanding of one of ordinary skill in the art of computing when using the term "executable component".

The term "executable component" is also well understood by one of ordinary skill as including structures that are implemented exclusively or near-exclusively in hardware, such as within a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other specialized circuit.

While not all computing systems require a user interface, in some embodiments, the computing system <NUM> includes a user interface system <NUM> for use in interfacing with a user. The user interface system <NUM> may include output mechanisms 112A as well as input mechanisms 112B. The principles described herein are not limited to the precise output mechanisms 112A or input mechanisms 112B as such will depend on the nature of the device. However, output mechanisms 112A might include, for instance, speakers, displays, tactile output, holograms and so forth. Examples of input mechanisms 112B might include, for instance, microphones, touchscreens, holograms, cameras, keyboards, mouse of other pointer input, sensors of any type, and so forth.

Embodiments described herein may comprise or utilize a special purpose or general-purpose computing system including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments described herein may also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computing system.

Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other physical and tangible storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computing system.

Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computing system.

Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general purpose computing system, special purpose computing system, or special purpose processing device to perform a certain function or group of functions. Alternatively or in addition, the computer-executable instructions may configure the computing system to perform a certain function or group of functions.

The invention may also be practiced in distributed system environments where local and remote computing systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks.

In embodiments, the computing system <NUM> may be implemented as a host computing system that is capable of running one or more, and potentially many, virtual machines (VM). For instance, <FIG> abstractly illustrates a host <NUM> in further detail. In the case of <FIG>, the host <NUM> is illustrated as operating three virtual machines <NUM> including virtual machines 210A, 210B and 210C. However, the ellipses 210D once again represents that the principles described herein are not limited to the number of virtual machines running on the host <NUM>. There may be as few as zero virtual machines running on the host with the only upper limit being defined by the physical capabilities of the host <NUM>.

During operation, the virtual machines emulate a fully operational computing system including an at least an operating system, and perhaps one or more other applications as well. Each virtual machine is assigned to a particular client computer, and is responsible to support the desktop environment for that client. As the user interacts with the desktop at the client, the user inputs are transmitted from the client to the virtual machine. The virtual machine processes the user inputs and, if appropriate, changes the desktop state. If such change in desktop state is to cause a change in the rendered desktop, then the virtual machine alters the image or rendering instructions, if appropriate, and transmits the altered image or rendered instructions to the client computing system for appropriate rendering. From the prospective of the user, it is as though the client computing system is itself performing the desktop processing.

The host <NUM> includes a hypervisor <NUM> that emulates virtual resources for the virtual machines <NUM> using physical resources <NUM> that are abstracted from view of the virtual machines <NUM>. The hypervisor <NUM> also provides proper isolation between the virtual machines <NUM>. Thus, from the perspective of any given virtual machine, the hypervisor <NUM> provides the illusion that the virtual machine is interfacing with a physical resource, even though the virtual machine only interfaces with the appearance (e.g., a virtual resource) of a physical resource, and not with a physical resource directly. In <FIG>, the physical resources <NUM> are abstractly represented as including resources 221A through 221F. Examples of physical resources <NUM> include processing capacity, memory, disk space, network bandwidth, media drives, and so forth. The host <NUM> may operate a host agent <NUM> that monitors the performance of the host, and performs other operations that manage the host. Furthermore, the host <NUM> may include other components <NUM>.

Attention is now given to <FIG>, which illustrate an embodiment of a host <NUM>, which may correspond to the host <NUM> previously described. The host <NUM> may include various components or functional blocks that may implement the various embodiments disclosed herein as will be explained. The various components or functional blocks of host <NUM> may be implemented on a local computing system or may be implemented on a distributed computing system that includes elements resident in the cloud or that implement aspects of cloud computing. The various components or functional blocks of the host <NUM> may be implemented as software, hardware, or a combination of software and hardware. The host <NUM> may include more or less than the components illustrated in <FIG>, and some of the components may be combined as circumstances warrant.

As illustrated, the host <NUM> may include or otherwise have access to physical memory <NUM>, which may correspond to memory <NUM> or one of the physical resources <NUM> previously described. In the embodiments, the physical memory <NUM> represents the physical memory resources, both volatile and non-volatile, of the host <NUM> that may be utilized by one or more virtual machines as will be explained in more detail to follow. Accordingly, any reference to computing system physical memory in this disclosure or in the claims may refer to volatile RAM memory, persistent RAM memory, disk memory, storage memory, or any type of volatile and non-volatile memory source. Thus, the embodiments and claims disclosed herein are not limited by the particular type of computing system physical memory that is implemented when practicing the embodiments disclosed herein.

The host <NUM> may also include or otherwise have access to a physical processor <NUM>, which may correspond to the physical processor <NUM> or one of the physical resources <NUM> previously described. In the embodiments, the physical processor <NUM> represents the physical processing resources of the host <NUM> that may be utilized by one or more virtual machines as will be explained in more detail to follow.

As illustrated, the host <NUM> may operate a virtual machine (VM) <NUM> instance that may correspond to the one of the VMs <NUM> previously described. Although only one VM is shown as being operated by the host <NUM>, this is for ease of explanation only and accordingly the embodiments disclosed herein contemplate that the host <NUM> may operate any number of additional VMs as circumstances warrant. The host <NUM> may also include a host operating system (OS) <NUM> that may be able to control the operation and function of the VM <NUM>. The VM <NUM> and host OS <NUM> will be explained in further detail to follow.

As mentioned previously in relation to <FIG>, the host may include a hypervisor such as the hypervisor <NUM>. Accordingly, the embodiments disclosed herein contemplate a hypervisor associated with the host <NUM> and the VM <NUM>. However, for ease of explanation, <FIG> does not show a hypervisor. Rather, <FIG> shows a hypervisor partition <NUM> that is generated by the hypervisor for the VM <NUM>. As illustrated, the hypervisor partition <NUM> includes a first virtual processor <NUM> and second virtual processor <NUM>. It will be noted that the hypervisor partition <NUM> may include additional virtual processors as circumstances warrant. The virtual processors <NUM> and <NUM> are seen by the VM <NUM> as being actual processors. However, as illustrated by the dotted lines in the physical processor <NUM>, the virtual processors use the hypervisor to access the processing resources of the physical processor <NUM>.

The hypervisor partition <NUM> also includes VM memory <NUM> with corresponding Guest Page Address (GPA) space that is used to map portions of the physical memory <NUM> to VM memory <NUM>. For example, as illustrated in <FIG> a portion of physical memory 305A having an address range <NUM> and a portion of physical memory 305B having an address range <NUM> may be mapped by the hypervisor to portion <NUM> of VM memory having an address range 331A. Likewise, a portion of physical memory 305C with an address range <NUM> may be mapped by the hypervisor to a portion <NUM> of the VM memory having an address range of 332A. In other words, the hypervisor is able to create and maintain a page table of the mappings between the GPA space and the physical memory space.

The host OS <NUM> may include a VM worker process instance <NUM> that is instantiated for running or configuring the VM <NUM>. As illustrated, the VM worker process <NUM> may include a virtual device (Vdev) <NUM> and a virtual device (Vdev) <NUM>. It will be noted that additional virtual devices may be included as illustrated by the ellipses <NUM>. In embodiments, the virtual devices may be configured to emulate various devices associated with the VM <NUM>. In embodiments, the VM worker process <NUM> may include additional components as also represented by the ellipses <NUM>. The additional components may include one or more of a state machine, a virtual motherboard, an IC proxy, or a RDP encoder. In operation, the VM worker process <NUM> may direct the hypervisor to create the mapping between the VM memory <NUM> and the physical memory <NUM> previously described.

In embodiments, the host OS <NUM> may include a driver <NUM>. In one embodiment, the driver <NUM> may be a virtual PCI driver that allows for the control of one or more devices that are directly assigned or attached to the VM <NUM> such as a Graphical Processing Unit (GPU) <NUM> and/or a Field Programmable Gate Array (FPGA) <NUM>. The directly attached devices will be explained in more detail to follow. As illustrated by the ellipses <NUM>, additional drivers may also be included in the host OS as circumstances warrant. In embodiments, the driver <NUM> may be able to, independent of the hypervisor, map a portion <NUM> of VM memory <NUM> to a portion 305D of physical memory <NUM> for the use of the attached devices. In such embodiments, the driver <NUM> may maintain any page tables and the like as needed.

As mentioned, the VM <NUM> may have various devices that are directly assigned or attached to the VM. For example, one directly assigned or attached device may be the GPU <NUM>. In some embodiments, the GPU <NUM> may include an input-output memory management unit (IOMMU) <NUM> or be communicatively coupled to an external IOMMU. Although not illustrated, in other embodiments the IOMMU <NUM> may be part of the physical processor <NUM>. The IOMMU <NUM> may have access to a corresponding address space <NUM> of the GPU <NUM>. In operation, the IOMMU <NUM> may interface with an attach module <NUM> of the hypervisor partition <NUM>. The attach module <NUM> represents the various functionality of the hypervisor that allows the GPU <NUM> to directly attach to the VM <NUM>. The IOMMU <NUM> is then able to directly access the GPA space of the VM memory <NUM> so that a mapping between the GPA space and the device address space may be made, which allows the GPU <NUM> to directly access the physical memory <NUM> via the VM memory <NUM>, The GPU <NUM> may thus have access to the memory pages mapped to the VM memory <NUM>.

In addition, the attach module <NUM> may allow the GPU <NUM> to directly access the virtual processors <NUM> and <NUM> for needed processing resources. Said another way, the GPU <NUM> views at least one of the virtual processors <NUM> and <NUM> as its assigned processor.

In like manner, another directly assigned or attached device may be the FPGA <NUM>. As illustrated, the FPGA <NUM> may interface with the attach module <NUM>. As illustrated, in some embodiments, the FPGA <NUM> may include or otherwise have access to an IOMMU <NUM>. Although not illustrated, in other embodiments the IOMMU <NUM> may be part of the physical processor <NUM>. The IOMMU <NUM> may perform a mapping between an address space <NUM> of the FPGA <NUM> and the GPA space of the VM memory <NUM> in the manner previously described for the IOMMU <NUM>. In addition, the FPGA <NUM> may directly access the virtual processors <NUM> and <NUM> for needed processing resources.

The ellipses <NUM> represent that any number of additional devices may be directly-assigned or attached to the VM <NUM>. Examples of the additional devices <NUM> include, but are not limited to, Universal Serial Bus (USB) including USB3 controllers, storage controllers, Peripheral Component Interconnect Express (PCIe) devices, and Non-Volatile Memory Express (NVMe) storage devices. The additional devices <NUM> may also include various network cards and the like that expose registers at locations in the computer's physical memory space that may be mapped to the GPA space of the VM memory <NUM>. Accordingly, the embodiments disclosed herein are not limited by the number or type of devices that are directly assigned or attached to the VM <NUM>. It will be noted that for ease of explanation, the embodiments disclosed herein are described having the GPU <NUM> or the FPGA <NUM> directly attached to the VM <NUM>. Accordingly, any discussion relating to the operation and the like of the GPU <NUM> and the FPGA <NUM> will also apply to any of the other directly assigned devices <NUM>.

The host OS <NUM> may also include a management module <NUM>. It will be noted that the management module <NUM> is used for ease of explanation and may represent kernel mode processes such as a Virtualization Infrastructure Driver (VID) and user mode processes such as Virtual Machine Management Service (VMMS). Accordingly, the embodiments disclosed herein are not limited by the actual type of the management module <NUM>.

In operation, the management module <NUM> may include storage resources that are backed by portions the physical memory <NUM>, such as the portion 305E. In embodiments, the storage resources may be a device extension or the like of the management module <NUM>. In embodiments, the management module may include a partition for each VM of the host <NUM>. However, the memory resources (i.e., the device extension) may remain even if the partitions are removed as long as the memory management module remains loaded in the host <NUM>.

In some embodiments, however, it may be possible to save the storage resources in a portion of the physical memory that has been designated to be persisted even during a host OS <NUM> shut down. In such embodiments, the management module <NUM> may be able to be unloaded while still persisting any information saved in the storage resources.

As shown in <FIG>, in embodiments the VM worker process <NUM> may direct the management module <NUM> to store various information in the memory resources of the management module <NUM> for the partition corresponding to the VM <NUM>. For example, the VM worker process <NUM> may direct the management module <NUM> to store a data block <NUM> that corresponds to the array of memory pages mapped to the portion <NUM> of VM memory <NUM> and to store the corresponding GPA address range 331A as shown at <NUM>. Likewise, the VM worker process <NUM> may direct the management module <NUM> to store a data block <NUM> that corresponds to the array of memory pages mapped to the portion <NUM> of VM memory <NUM> and to store the corresponding GPA address range 332A as shown at <NUM>.

The VM worker process <NUM> may also direct the management module <NUM> to generate a state file <NUM> that records the current state of the VM worker process <NUM>. The current state of the various virtual devices <NUM> and <NUM> may also be recorded in the state file <NUM>. This file may be used to persist the state of the VM worker process and the virtual devices as will be explained in more detail to follow.

In embodiments, the driver <NUM> may also direct the management module <NUM> to record information. For example, the driver <NUM> may direct the management module <NUM> to record a data block <NUM> that corresponds to the array of memory pages mapped to the portion <NUM> of VM memory <NUM>. The corresponding GPA range may also be recorded. It will be noted that any of the additional drivers <NUM> may also direct the management module <NUM> to record information as circumstances warrant. Accordingly, the embodiments disclosed herein contemplate both the VM worker process <NUM> and various drivers directing the management module <NUM> to record information as needed.

In embodiments, it may be desirable to service one or more of the components of the VM <NUM> and/or the host OS <NUM>. The servicing may allow for software updates and the like to happen to the relevant components. In the embodiments disclosed herein, the servicing operation may occur without the need to reboot the host OS <NUM>. In addition, some of the components will not have their underlying data structures removed from the computing system physical hardware (i.e., the physical memory <NUM> and the physical processor <NUM>) during the servicing operation. Thus, these components will typically not be subject to any update during the servicing operation.

In embodiments, the VM worker process <NUM> may initiate the VM servicing operation by directing the management module <NUM> to store the various information in its storage resources (i.e., device extension) in the manner previously described. For example, the management module <NUM> may save the data blocks <NUM> and <NUM> and the corresponding GPA ranges <NUM> and <NUM>. In addition, the driver <NUM> may direct the management module <NUM> to store the data block <NUM> and corresponding GPA range.

In response to storing the information, the management module <NUM> generates a store <NUM> for storing persistent identity pointers for components of the VM that are not to have the servicing operation performed on them, but that are to have their underlying data structures remain loaded on the computing system physical hardware. For example, an identity pointer <NUM> for the hypervisor partition <NUM> including the virtual processors may be stored in the store <NUM>. In addition, identity pointers <NUM>-<NUM> for the memory block <NUM>, the GPA range <NUM>, the memory block <NUM>, the GPA range <NUM>, and the memory block <NUM> respectively may be stored in the store <NUM>. The store <NUM> is written into a dedicated portion 305F of the physical memory <NUM>. It will be noted that the identity pointers <NUM>-<NUM> may be considered artifacts that point to the underlying data structures in the computing system physical hardware (i.e., the physical memory <NUM> and the physical processor <NUM>) for the components corresponding to the identity pointers.

It will be noted that, in at least some implementations, only the data blocks in the management module <NUM> are persisted. In these implementations, there is no need to access the physical memory <NUM> to determine the addresses of the data blocks to persist. This may result in a time savings if there is a large number of data blocks to persist or if the data blocks are non-contiguous with each other. In other words, there is no need to walk the memory stack to determine the data blocks to persist.

The state of the VM worker process and virtual devices may also be stored in the state file <NUM>. This state may then be stored in the memory resources of the management module <NUM> or in the portion 305E of the physical memory <NUM>.

The operation of the VM <NUM> may then be suspended so that the servicing operation may occur. <FIG> represents the VM <NUM> during the servicing operation. For ease of illustration, some of the elements shown in <FIG> are not included in <FIG>.

As shown in <FIG>, VM worker process <NUM> and the virtual devices <NUM> and <NUM> have been shut down and thus are not present in the figure. In other words, the VM worker process <NUM> and the virtual devices <NUM> and <NUM> have been removed so that any of their underlying data structures have been removed from the computing system physical hardware. As mentioned previously, however, the state of VM worker process <NUM> and the virtual devices <NUM> and <NUM> prior to being shut down was saved in the state file <NUM> and may be used to restore these components as will be discussed in more detail to follow.

In <FIG>, the hypervisor partition <NUM> and its components including the virtual processors <NUM> and <NUM> and the virtual memory <NUM> and its associated mappings with the physical memory <NUM> (i.e., <NUM> and <NUM>) are shown as being dotted. The mapping <NUM> of the driver <NUM> is also shown as being dotted. The dots represent that although the operation of the VM <NUM> has been suspended, the underlying data structures for these components have not been removed from the computing system physical hardware. In other words, even while the operation of the VM <NUM> has been suspended and the servicing operation is being performed, the hypervisor partition <NUM> and the GPA mapping remain loaded or programed in the computing system physical hardware. It will be noted, however, that the operation of the components of the hypervisor partition is suspended while the operation of the VM <NUM> is suspended.

It will also be noted that the host OS <NUM> is not shut down during the servicing operation, but also remains loaded along with the management module <NUM>, although the specific partition in the memory management module for the VM <NUM> has been removed. Thus, those components that are not shut down are typically not able to be serviced by the servicing operation as this typically requires that a component be shut down so that any changes to the components during the servicing operation may be implemented.

<FIG> also shows that the devices that are directly attached to the VM <NUM> including the GPU <NUM> and the FPGA <NUM> remain attached or are kept active while the operation of the VM <NUM> has been suspended. That is, from the perspective of the directly attached devices that are kept active, the VM <NUM> is still in active operation and the devices and the VM <NUM> are able to normally function in relationship with each other. Thus, the attach module <NUM> allows the mapping between the GPA space of the VM memory <NUM> and the IOMMU previously described to remain intact. Since the GPA mapping has not been removed as previously described, the mapping between the device and the VM <NUM> is able to be maintained.

The devices that are directly attached to the VM <NUM> including the GPU <NUM> and the FPGA <NUM> also retain access to processing resources of the virtual processors <NUM> and <NUM>. As may be appreciated, since the devices that are directly attached to the VM <NUM> including the GPU <NUM> and the FPGA <NUM> may continue to operate while the operation of the VM <NUM> is suspended, there may be instances where the directly attached device initiates an interrupt to be handled by one of the virtual processors. For example, the GPU <NUM> may initiate an interrupt <NUM> and the FPGA <NUM> may initiate an interrupt <NUM>. To handle such instances, the attach module or some other component of the VM <NUM> may include a queue file <NUM> that is able to queue the interrupts while the virtual processors are suspended. For example, in embodiments where the directly attached device uses Message Signal Interrupts (MSI), the device has at most <NUM> interrupts and an address that the device writes to indicate a pending interrupt. Where that address is mapped to the VM, the directly attached device is allowed to write to that address, even when the operation of the VM is suspended. The same is true for MSI-X, except that MSI-X has up to <NUM> interrupts.

As will be explained in more detail, once the operation of the virtual processors is restored, the virtual processors are able to handle the interrupts in the queue file <NUM>. Thus, from the perspective of the directly attached devices this is only seen as a small delay in processing time.

<FIG> shows the process of restoring the VM <NUM> to operation. As shown, <FIG> includes the host OS <NUM> and the store <NUM>. After the servicing operation of the VM <NUM> is completed, which may result in one or more components of the VM or host OS being updated or the like, the VM worker process <NUM> and the virtual devices <NUM> and <NUM> may be restored by the host computing system. As previously described, the state of the VM worker process <NUM> and the virtual devices <NUM> and <NUM> was saved before the operation of the VM <NUM> was suspended. This state may then be used to reload or program the underlying data structures in the computing system hardware to thereby restore the VM worker process <NUM> and the virtual devices <NUM> and <NUM>.

Once the VM worker process <NUM> is restored, the VM worker process may direct the management module <NUM> to rebuild the partition for the VM <NUM> and to build a data object for memory blocks and corresponding GPA ranges and may also request that a hypervisor partition be generated. In such case, the management module <NUM> as shown at <NUM> accesses the identity pointers stored in the store <NUM>. The identity pointers then allow the management module <NUM> to access the underlying data structures for the various components stored in the store <NUM>. The management module <NUM> may then populate the data objects <NUM>-<NUM> and <NUM> with the recovered underlying data structures as shown at <NUM>. This has the effect of reconnecting the hypervisor partition <NUM>, virtual processors <NUM> and <NUM>, and the mapping of the virtual memory <NUM> to the restored VM worker process <NUM>. That is, the VM worker process <NUM> is now able to use these components without the need for the system to actually rebuild these components in the computing system physical hardware. Said another way, since the hypervisor partition <NUM>, virtual processors <NUM> and <NUM>, and the mapping of the virtual memory <NUM> to the physical memory <NUM> was not removed during the servicing operation, the management module <NUM> only needs to reconnect the VM worker process <NUM> to these components.

The reconnecting of the VM worker process to the hypervisor partition <NUM>, virtual processors <NUM> and <NUM>, and the mapping of the virtual memory <NUM> to the physical memory <NUM> may return the VM to the state shown in <FIG> prior to the generation of the store <NUM>. That is, the VM <NUM> may resume its normal operation.

Upon the resumption of the operation of the VM <NUM>, any interrupts such as interrupts <NUM> and <NUM> stored in the queue file <NUM> may be handled by the virtual processors <NUM> and <NUM>. The directly attached devices such as the GPU <NUM> and the FPGA <NUM> may then continue to normally communicate with and operate in relation to the resumed VM <NUM>.

<FIG> illustrates a flow chart of an example computerized method <NUM> for servicing components of a VM while second components remain loaded in the computing system physical hardware during the servicing operation. The method <NUM> will be described with respect to one or more of <FIG> discussed previously.

The method <NUM> includes suspending an operation of a VM running one or more first and second VM components so that a servicing operation for the VM may be performed (<NUM>). In some embodiment the VM may have one or more devices that are directly attached to it. For example, as previously described the VM <NUM> may run or have associated with it the VM worker process <NUM> and the virtual devices <NUM> and <NUM>, which may be examples of a first VM component. In addition, the VM <NUM> may run or have associated with it the hypervisor partition <NUM>, virtual processors <NUM> and <NUM>, and the virtual memory <NUM> that has a GPA space that is mapped to the physical memory <NUM>. These may be considered as examples of a second VM component.

As previously described in some embodiments the VM <NUM> may have one or more devices that are directly attached to it. For example, the GPU <NUM> and the FPGA <NUM> may be directly attached to the VM <NUM>.

The method <NUM> includes saving a state of the one or more first VM components (<NUM>). For example, as previously described the state of the VM worker process <NUM> and the virtual devices <NUM> and <NUM> may be stored in the state file <NUM>.

The method <NUM> includes saving an identification pointer for the one or more second VM components in a portion of the computing system physical memory without removing any underlying data structures of the one or more second VM components from the computing system physical hardware (<NUM>). For example, as previously described the identification pointers <NUM>-<NUM> for the hypervisor partition, the memory blocks <NUM> and <NUM>, and GPA ranges <NUM> and <NUM>, as well as the memory block <NUM> may be stored in the store <NUM>, which may correspond to the portion 305F of the physical memory <NUM>.

As previously described, the one or more directly attached devices remain attached to the VM and remain configured to communicate with the VM while the VM is suspended and while the servicing operation is performed since the underlying data structures of the one or more second VM components are not removed. For example, the GPU <NUM> and the FPGA <NUM> remain directly attached to and remain configured to communicate with the VM <NUM> while the operation of the VM <NUM> is suspended. This happens because the underlying data structures of the hypervisor partition <NUM>, virtual processors <NUM> and <NUM>, and the virtual memory <NUM> that has a GPA space that is mapped to the physical memory <NUM> remain in the physical hardware of the computing system while the VM is suspended.

The method <NUM> includes shutting down the one or more first VM components by removing any underlying data structures for the one or more first VM components from the computing system physical hardware (<NUM>). As previously described, the VM work process <NUM> and the virtual devices <NUM> and <NUM> are shut down by having their underlying data structures removed from the physical hardware of the computing system.

The method <NUM> includes restoring at the completion of the servicing operation the one or more first VM components (<NUM>). For example, as previously described the underlying data structures in the computing system physical hardware of VM work process <NUM> and the virtual devices <NUM> and <NUM> are rebuilt using the saved state from the state file <NUM>.

The method <NUM> may include reconnecting the restored one or more first VM components to the one or more second VM components using the identification pointers (<NUM>). For example, as previously described the identity pointers <NUM>-<NUM> may be used to access the underlying data structures of the hypervisor partition <NUM>, virtual processors <NUM> and <NUM>, and the virtual memory <NUM> that has a GPA space that is mapped to the physical memory <NUM>. These components may then be reconnected to the VM work process <NUM> and the virtual devices <NUM> and <NUM> in the manner previously described.

Claim 1:
A computing system (<NUM>) for suspending and resuming an operation of a virtual machine, VM, running one or more first VM components and one or more second VM components, the VM having one or more devices attached to it, the computing system being configured to:
suspend (<NUM>) the operation of the VM, wherein at least one device remains configured as attached to the VM and remains configured to communicate with the VM while the VM is suspended;
save (<NUM>) a state of the one or more first VM components;
save (<NUM>) one or more identification pointers pointing to underlying data structures in computing system physical hardware of the one or more second VM components in a portion of the computing system physical memory while said underlying data structures of the one or more second VM components remain loaded in the computing system physical hardware;
shut down (<NUM>) the one or more first VM components by removing any underlying data structures for the one or more first VM components from the computing system physical hardware;
restore (<NUM>) the one or more first VM components using the saved state of the one or more first VM components;
reconnect (<NUM>) the restored one or more first VM components to the one or more second VM components using the identification pointers; and
resume (<NUM>) the operation of the VM.