Patent Publication Number: US-10318329-B2

Title: Driver switch for live migration with an assigned device

Description:
BACKGROUND 
     Computer systems may employ virtual machines including one or more guest operating systems (OS). A virtual machine (VM) is a software implementation of a computer that executes programs in a way similar to a physical machine. The virtual machine may share underlying physical hardware resources between different virtual machines. Each virtual machine may be associated with a physical device, such as a hardware device and/or an assigned device. A virtual machine may include a virtual device that serves as a pass-through device to the physical device, and the virtual machine may perform tasks associated with the functions of the physical device and/or related to the associated physical device. Additionally, virtual machines may be stopped and resumed and/or migrated to various physical machines with differing configurations of physical hardware resources. 
     SUMMARY 
     The present disclosure provides new and innovative systems and methods of live migration with an assigned device. In an example embodiment, a system includes a memory, one or more physical processors in communication with the memory, a first device, a second device, a first hypervisor, a second hypervisor, a first virtual machine, and a second virtual machine. The first hypervisor executes on the one or more physical processors and is located at a migration source location. The second hypervisor executes on the one or more physical processors and is located at a migration destination location. Before migration, the first virtual machine includes a guest OS executing on the first hypervisor and a guest driver in the guest OS executing on the first hypervisor. After migration, the second virtual machine includes the guest OS executing on the second hypervisor and the guest driver in the guest OS executing on the second hypervisor. The first hypervisor executes on the one or more physical processors to send a request to save a device state to the guest driver in the guest OS executing on the first hypervisor. The device state is associated with the first device on the first virtual machine. The second hypervisor executes on the one or more physical processors to send a migration notification to the guest OS executing on the second hypervisor. The guest driver in the guest OS executing on the first hypervisor receives the request from the first hypervisor and save a state signature in the memory. The state signature includes a device signature of the first device attached to the device state of the first device. The guest driver in the guest OS executing on the second hypervisor determines a status of the state signature as one of matching the second device and mismatching the second device. 
     In an example embodiment, a method includes sending, before migration, by a first hypervisor, a request to save a device state to a guest driver in a guest OS executing on the first hypervisor. The device state is associated with a first device on a first virtual machine, and the first virtual machine is located at a migration source location. Before migration, the guest driver in the guest OS executing on the first hypervisor receives the request from the first hypervisor and saves a state signature in a first memory. The state signature includes a device signature of the first device attached to the device state. After migration, the second hypervisor sends a migration notification to a guest OS executing on the second hypervisor. Additionally, after migration, the guest OS executing on the second hypervisor receives the migration notification and the guest driver in the guest OS executing on the second hypervisor determines whether the state signature is a match or a mismatch with a second device on a second virtual machine. 
     In an example embodiment, a non-transitory machine readable medium stores a program, which when executed by a processor, causes a guest OS including a guest driver to receive, before migration, by a guest driver in the guest OS executing on a first hypervisor, a request from the first hypervisor to save a device state. The device state is associated with a first device on the first virtual machine, and the first virtual machine is located at a migration source location. The guest driver in the guest OS executing on the first hypervisor saves a state signature of the first device in a first memory before migration. The state signature includes a device signature attached to the device state. After migration, the guest OS executes on a second hypervisor to receive a migration notification from the second hypervisor, and the guest driver in the guest OS executing on the second hypervisor determines a status of the state signature as one of matching and mismatching with a second device on a second virtual machine. The second virtual machine is located at a migration destination location. Responsive to determining the status of the state signature as matching, the guest driver in the guest OS executing on the second hypervisor loads, after migration, the device state to the second device, and responsive to determining the status of the state signature as mismatching, the guest OS executing on the second hypervisor reinitializes, after migration, the second device on the second virtual machine at the migration destination. 
     Additional features and advantages of the disclosed method and apparatus are described in, and will be apparent from, the following Detailed Description and the Figures. The features and advantages described herein are not all-inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the figures and description. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and not to limit the scope of the inventive subject matter. 
    
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
         FIG. 1A  illustrates a block diagram of an example migration system at a migration source location according to an example embodiment of the present disclosure. 
         FIG. 1B  illustrates a block diagram of an example migration system at a migration destination location according to an example embodiment of the present disclosure. 
         FIG. 2  illustrates a flowchart of an example process for virtual machine migration according to an example embodiment of the present disclosure. 
         FIG. 3  illustrates a flowchart of an example process for virtual machine migration according to an example embodiment of the present disclosure. 
         FIG. 4  illustrates a flow diagram of an example process for virtual machine migration according to an example embodiment of the present disclosure. 
         FIG. 5  illustrates a block diagram of an example migration system according to an example embodiment of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS 
     Techniques are disclosed for providing a driver switch for live migration with an assigned device. Assigned devices have varying hardware, firmware, memory sizes, etc. Current techniques for live migration with an assigned device typically involve fully removing any assigned devices from the guest OS and discarding the driver and/or state (e.g., device state and device signature). Then, the device state is recreated after the device is re-added after migration. However, these techniques are generally expensive and result in additional downtime. For example, device profiles may need to be recreated after migration before the system resumes operating as it was before migration. To reduce overhead and downtime, the device state may be stored in memory such that if the virtual machine is migrated to another identical host OS, the device state may be restored from memory at the destination location. However, often times the host OS at the migration destination location is not identical to the host OS at the migration source location. Additionally, devices may typically vary slightly by way of build versions, vendors, and/or updates. For example, in ten PCI cards, each PCI card may have slightly different hardware and firmware versions such that a saved device state on one device and/or host OS may not work well on another device and/or host OS. 
     As discussed in the various example embodiments disclosed herein, to decrease downtime and improve efficiency of migrating varying devices on host OSes that may not be identical to the host OS at the migration source location, a guest device driver may be used to detect a mismatch in the device state and enable the guest OS to recover (e.g., reinitialize the device) to ensure that the system functions properly. This advantageously allows for efficient migration and prevents issues that may arise due to varying differences between devices and/or host OSes. For example, loading the device state from memory without checking for a match may result in wasted resources in each case of a mismatch, as the device may need to be reinitialized for a successful migration. Specifically, in the case of a mismatching devices on the migration source location and the migration destination location, the guest OS may crash. On the other hand, reinitializing every device after migration consumes additional resources in each case of a match because the device state may have been loaded from memory resulting in a quicker and more efficient migration. Thus, the techniques described herein advantageously enables migration for both matched and mismatched devices while avoiding crashing the guest OS and the additional overhead associated with recreating a device state or reinitializing a device for each instance of migration. 
       FIG. 1A  depicts a high-level component diagram of an example migration system  100  at a migration source location in accordance with one or more aspects of the present disclosure. The migration system  100  may include a memory (e.g., MD  130 A-E), one or more physical processors in communication with the memory (e.g., CPU  120 A-C), one or more virtual machines (e.g., VM  170 A-C), and a hypervisor (e.g., hypervisor  180 A). 
     The virtual machines  170 A-C may include a guest OS, guest memory, a virtual CPU (VCPU), virtual memory devices (VMD), and virtual input/output devices (VI/O). For example, virtual machine  170 A may include Guest OS  196 , guest memory  195 A, a virtual CPU  190 A, a virtual memory devices  192 A, and virtual input/output device  194 A. In an example embodiment, a first virtual machine (e.g., VM  170 A) may include a first device  174 A. For example, the first device  174 A may be a pass-through device that is virtualized or emulated by the hypervisor  180 A. The first device  174 A may be associated with an assigned device  171 A. The first device  174 A may include a device state  192 A and a device signature  191 A. Guest memory (e.g., Guest Memory  195 A) may include one or more memory pages. 
     As noted above, migration system  100  may run multiple virtual machines (e.g., VM  170 A-C), by executing a software layer (e.g., hypervisor  180 A) above the hardware and below the virtual machines  170 A-C, as schematically shown in  FIG. 1 . In an example embodiment, the hypervisor  180 A may be a component of the host operating system  186 A executed by the migration system  100 . In another example embodiment, the hypervisor  180 A may be provided by an application running on the operating system  186 A, or may run directly on the migration system  100  without an operating system beneath it. The hypervisor  180 A may virtualize the physical layer, including processors, memory, and I/O devices, and present this virtualization to virtual machines  170 A-C as devices, including virtual processors (e.g., VCPU  190 A), virtual memory devices (e.g., VMD  193 A), and/or virtual I/O devices (e.g., VI/O  194 A). It should be appreciated that VM  170 B and VM  170 C may include one or more VCPUs, VMDs, and/or VI/O devices. 
     In an example embodiment, a virtual machine  170 A may execute a guest operating system  196  which may utilize the underlying VCPU  190 A, VMD  193 A, and VI/O device  194 A. One or more applications  198 A-B may be running on a virtual machine  170 A under the respective guest operating system  196 . A virtual machine (e.g., VM  170 A-F, as illustrated in  FIGS. 1A and 1B ) may run on any type of dependent, independent, compatible, and/or incompatible applications on the underlying hardware and OS (e.g., host OS  186 A-B). In an example embodiment, applications (e.g., App  198 A-B) run on a virtual machine  170 A may be dependent on the underlying hardware and/or OS  186 A. In another example embodiment, applications  198 A-B run on a virtual machine  170 A may be independent of the underlying hardware and/or OS  186 . For example, applications  198 A-B run on a first virtual machine  170 A may be dependent on the underlying hardware and/or OS  186  while applications run on a second virtual machine (e.g., VM  170 B) are independent of the underlying hardware and/or OS  186 A. Additionally, applications  198 A-B run on a virtual machine  170 A may be compatible with the underlying hardware and/or OS  186 A. In an example embodiment, applications  198 A-B run on a virtual machine  170 A may be incompatible with the underlying hardware and/or OS  186 A. For example, applications  198 A-B run on one virtual machine  170 A may be compatible with the underlying hardware and/or OS  186 A while applications run on another virtual machine  170 B are incompatible with the underlying hardware and/or OS  186 A. In an example embodiment, a device may be implemented as a virtual machine (e.g., virtual machine  170 A-F). 
     The hypervisor  180 A may manage host memory  184 A for the host operating system  186 A as well as memory allocated to the virtual machines  170 A-B and guest operating systems  196  such as guest memory  195 A provided to guest OS  196 . Host memory  184 A and guest memory  195 A may be divided into a plurality of memory pages that are managed by the hypervisor  180 A. Guest memory  195 A allocated to the guest OS  196  may be mapped from host memory  184 A such that when a guest application  198 A-B uses or accesses a memory page of guest memory  195 A it is actually using or accessing host memory  184 A. 
     The migration system  100 , at the migration source location, may include one or more interconnected nodes  110 A-D. Each node  110 A-B may in turn include one or more physical processors (e.g., CPU  120 A-C) communicatively coupled to memory devices (e.g., MD  130 A-C) and input/output devices (e.g., I/O  140 A-B). Nodes  110 C-D may include a device such as an assigned device  171 A or a hardware device  150 A. In an example embodiment, a hardware device (e.g.,  150 A) and/or an assigned device  171 A may include a network device (e.g., a network adapter or any other component that connects a computer to a computer network), a peripheral component interconnect (PCI) device, storage devices, disk drives, sound or video adaptors, photo/video cameras, printer devices, keyboards, displays, etc. The Nodes  110 C-D may include one ore more physical processors communicatively coupled to memory devices (e.g., MD  130 D-E) and input/output devices (e.g., I/O  140 C). Additionally, Node  110 C may be an assigned device  171 A with a device state  192 B. For example, device states  192 A-B may both be associated with the first device  174 A because the first device  174 A may serve as a pass-through device that may be associated with assigned device  171 A. 
       FIG. 1B  depicts a high-level component diagram of the example migration system  100  at a migration destination location in accordance with one or more aspects of the present disclosure. The migration system  100  may include a memory (e.g., MD  130 F-H), one or more physical processors in communication with the memory (e.g., CPU  120 D-F), one or more virtual machines (e.g., VM  170 D-F), and a hypervisor (e.g., hypervisor  180 B). 
     The virtual machines  170 D-F may include a guest OS, guest memory, a virtual CPU (VCPU), virtual memory devices (VMD), and virtual input/output devices (VI/O). For example, virtual machine  170 D may include Guest OS  196 ′ (e.g., the migrated Guest OS  196 ), guest memory  195 B, a virtual CPU  190 B, a virtual memory devices  192 B, and virtual input/output device  194 B. In an example embodiment, a second virtual machine (e.g., VM  170 D) may include a second device  174 B. For example, the second device  174 B may be a pass-through device that is virtualized or emulated by the second hypervisor  180 B. The second device  174 B may be associated with assigned device  171 B. The second device  174 B may include a device state  192 C and a device signature  191 B. Guest memory (e.g., Guest Memory  195 B) may include one or more memory pages. 
     As noted above, migration system  100  may run multiple virtual machines (e.g., VM  170 D-F), by executing a software layer (e.g., hypervisor  180 B) above the hardware and below the virtual machines  170 D-F, as schematically shown in  FIG. 1B . In an example embodiment, the hypervisor  180 B may be a component of the host operating system  186 B executed by the migration system  100 . In another example embodiment, the hypervisor  180 B may be provided by an application running on the operating system  186 B, or may run directly on the migration system  100  without an operating system beneath it. The hypervisor  180 B may virtualize the physical layer, including processors, memory, and I/O devices, and present this virtualization to virtual machines  170 D-F as devices, including virtual processors (e.g., VCPU  190 B), virtual memory devices (e.g., VMD  193 B), and/or virtual I/O devices (e.g., VI/O  194 B). It should be appreciated that VM  170 E and VM  170 F may include one or more VCPUs, VMDs, and/or VI/O devices. 
     In an example embodiment, a virtual machine  170 D (hereinafter, virtual machine  170 ) may execute a guest operating system  196 ′ which may utilize the underlying VCPU  190 B, VMD  193 B, and VI/O device  194 B. One or more applications  198 C-D may be running on a virtual machine  170 D under the respective guest operating system  196 ′. The hypervisor  180 B may manage host memory  184 B for the host operating system  186 B as well as memory allocated to the virtual machines  170 D-F and guest operating systems  196 ′ such as guest memory  195 B provided to guest OS  196 ′. Host memory  184 B and guest memory  195 B may be divided into a plurality of memory pages that are managed by the hypervisor  180 B. Guest memory  195 B allocated to the guest OS  196 ′ may be mapped from host memory  184 B such that when a guest application  198 C-D uses or accesses a memory page of guest memory  195 B it is actually using or accessing host memory  184 B. 
     The migration system  100 , at the migration destination location, may include one or more interconnected nodes  110 E-F. Each node  110 E-F may in turn include one or more physical processors (e.g., CPU  120 D-F) communicatively coupled to memory devices (e.g., MD  130 F-H) and input/output devices (e.g., I/O  140 D-E). Nodes  110 G-H may include a device such as an assigned device  171 B or a hardware device  150 B. In an example embodiment, a hardware device (e.g.,  150 B) and/or an assigned device  171 B may include a network device (e.g., a network adapter or any other component that connects a computer to a computer network), a peripheral component interconnect (PCI) device, storage devices, disk drives, sound or video adaptors, photo/video cameras, printer devices, keyboards, displays, etc. The Nodes  110 G-H may include one ore more physical processors communicatively coupled to memory devices (e.g., MD  1301 -J) and input/output devices (e.g., I/O  140 F). Additionally, Node  110 G may be an assigned device  171 B with a device state  192 D. For example, device states  192 C-D may both be associated with the second device  174 B because the second device  174 B may serve as a pass-through device that may be associated with assigned device  171 B. 
     As used herein, a physical processor or a processor  120 A-F refers to a device capable of executing instructions encoding arithmetic, logical, and/or I/O operations. In one illustrative example, a processor may follow Von Neumann architectural model and may include an arithmetic logic unit (ALU), a control unit, and a plurality of registers. In a further aspect, a processor may be a single core processor which is typically capable of executing one instruction at a time (or process a single pipeline of instructions), or a multi-core processor which may simultaneously execute multiple instructions. In another aspect, a processor may be implemented as a single integrated circuit, two or more integrated circuits, or may be a component of a multi-chip module (e.g., in which individual microprocessor dies are included in a single integrated circuit package and hence share a single socket). A processor may also be referred to as a central processing unit (CPU). 
     As discussed herein, a memory device  130 A-H refers to a volatile or non-volatile memory device, such as RAM, ROM, EEPROM, or any other device capable of storing data. As discussed herein, I/O device  140 A-E refers to a device capable of providing an interface between one or more processor pins and an external device capable of inputting and/or outputting binary data. 
     Processors  120 A-F may be interconnected using a variety of techniques, ranging from a point-to-point processor interconnect, to a system area network, such as an Ethernet-based network. Local connections within each node and/or between nodes, including the connections between a processor  120 A-F and a memory device  130 A-J and between a processor  120 A-F and an I/O device  140 A-F, may be provided by one or more local buses of suitable architecture, for example, peripheral component interconnect (PCI). 
       FIG. 2  illustrates a flowchart of an example method  200  for virtual machine migration in accordance with an example embodiment of the present disclosure. Although the example method  200  is described with reference to the flowchart illustrated in  FIG. 2 , it will be appreciated that many other methods of performing the acts associated with the method  200  may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional. The method  200  may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. 
     The example method  200  includes receiving a request from a first hypervisor to save a device state associated with a first device before migration (block  202 ). For example, a guest OS  196  executing on a first hypervisor  180 A may receive a request from the first hypervisor  180 A to save a device state  192 A associated with a first device  174 A before migration. The first device  174 A may serve as a pass-through device that may be associated with assigned device  171 A and the device state  192 A may be the same as the device state  192 B. In an example embodiment, a first guest driver  188 A in a guest OS  196  executing on a first hypervisor  180 A may receive a request from the first hypervisor  180 A to save a device state  192 A associated with a first device  174 A before migration. Then, example method  200  saves a state signature of the first device in a first memory before migration (block  204 ). For example, the guest OS  196  executing on the first hypervisor  180 A may save a state signature of the first device  174 A in a first memory (e.g., guest memory  195 A) before migration. In an example embodiment, the first guest driver  188 A in the guest OS  196  executing on a first hypervisor  180 A may save a state signature of the first device  174 A in a first memory before migration. 
     Then, example method  200  receives a migration notification from the second hypervisor after migration (block  206 ). For example, the guest OS  196 ′ executing on a second hypervisor  180 B may receive a migration notification from the second hypervisor  180 B after migration. In an example embodiment, the hypervisor  180 B may send the guest OS  196 ′ a message including instructions related to migrating the associated virtual machine (e.g., virtual machine  170 A) from a migration source location to a migration destination location. In another example embodiment, the guest OS  196 ′ may unbind the guest driver  188 A′ in the guest OS  196 ′ executing on the second hypervisor  180 B from the first device  174 A. For example, the guest driver  188 A′ may be associated with the first device  174 A and the assigned device  171 A. Once the system migrates the virtual machine  170 A to the migration destination location, the assigned device  171 A may no longer have reason to be bound to guest driver  188 A′. 
     Then, example method  200  determines a status of the state signature as one of matching and mismatching a second device after migration (block  208 ). For example, the guest OS  196 ′ executing on the second hypervisor  180 B may determine a status of the state signature as one of matching and mismatching a second device  174 B after migration. In an example embodiment, guest driver  188 A′ in the guest OS  196 ′ executing on the second hypervisor  180 B may determine the status of the state signature as one of matching and mismatching the second device  174 B after migration. Additionally, the guest driver  188 A′ may notify the guest OS  196 ′ of the status of the state signature as mismatching. Responsive to determining the status of the state signature as matching, example method  200  loads the device state to the second device (block  210 ). For example, the guest OS  196 ′ executing on the second hypervisor  180 B may load the device state  192 A to the second device  174 B. The device state  192 A may be the same as device state  192 B or may be similar to device state  192 B. In an example embodiment, the guest driver  188 A′ in the guest OS  196 ′ executing on the second hypervisor  180 B may load the device state  192 A to the second device  174 B. For example, the second device  174 B and the associated assigned device  171 B may have the same device state as the first device  174 A (e.g., device states  174 A-D are the same). Responsive to determining the status of the state signature as mismatching, example method  200  reinitializes the second device on the second virtual machine at the migration destination (block  212 ). For example, the guest OS  196 ′ may reinitialize the second device  174 B on the second virtual machine  170 D at the migration destination. The second device  174 B may be reinitialized such that the guest OS  196 ′ may resume operation with the newly loaded device states  192 C-D. 
     In an example embodiment, the guest OS  196 ′ executing on the second hypervisor  170 D may bind the guest driver  188 A′ to the second device  174 B. For example, in the case of matching, the guest OS  196 ′ may bind guest driver  188 A′ to the second device  174 A (associated with assigned device  171 A) because the guest driver  188 A′ (e.g., migrated driver  188 A) is configured to work with the second devices (e.g., second device  174 B and assigned device  171 B) because they have matching state signatures with the original devices (e.g., first device  174 A and assigned device  171 A). Additionally, the guest OS  196 ′ executing on the second hypervisor  170 D may bind a different driver (e.g., driver  188 B) to the second device  174 A. For example, the guest OS  196 ′ may decide to use a driver (e.g., driver  188 B) that is compatible with multiple devices. In another example embodiment, the guest OS  196 ′ may bind a different driver (e.g., driver  188 B) in the case of mismatching to ensure that the proper guest driver is used with the second device  174 B and the associated assigned device  171 B. 
     In another example embodiment, the guest OS  196 ′ executing on the second hypervisor  170 D may unload the guest driver  188 A′ from memory (e.g., guest memory  195 B). For example, on mismatch, a different guest driver may be used and the guest OS  196 ′ may load the different guest driver in the memory (e.g., guest memory  195 B). Then, the guest driver may be used with the second device  174 B and the associated assigned device  171 B. 
       FIG. 3  illustrates a flowchart of an example method  300  for virtual machine migration in accordance with an example embodiment of the present disclosure. Although the example method  300  is described with reference to the flowchart illustrated in  FIG. 3 , it will be appreciated that many other methods of performing the acts associated with the method  300  may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional. The method  300  may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. 
     In the illustrated embodiment, a first hypervisor  170 A may send a request to save a device state  192 A and/or device state  192 B to a guest driver  188 A in a guest OS  196  executing on the first hypervisor before migration (block  302 ). In an example embodiment, the device state  192 A of the first device  174 A may be associated with an assigned device  171 A. For example, the first device  174 A may be a virtual device exposed to the guest OS  196  and may be a pass-through device associated with the assigned device  171 A on the host OS  186 A. Responsive to the first hypervisor  170 A sending the request, the guest driver  188 A in the guest OS  196  executing on the first hypervisor  170 A may receive the request from the first hypervisor  170 A before migration (block  304 ). In an example embodiment, the request may be sent directly to the guest driver  188 A. In another example embodiment, the request may be sent to the guest OS  196 , which may then forward the request to the guest driver  188 A. Then, the guest driver  188 A in the guest OS  196  executing on the first hypervisor  170 A may save a state signature of the first device  174 A in a first memory before migration (block  306 ). For example, the state signature of the first device  174 A may include the device signature  191 A attached to the device state  192 A of the first device  174 A. In an example embodiment, the state signature and/or the device signature  191 A of the first device  174 A may include a hardware device ID, a firmware device ID, a vendor ID and/or a version. Additionally, the state signature may include a hash function, a list of loaded firmware and/or a memory size of the first device  174 A and/or the associated assigned device  171 A. 
     A second hypervisor  180 B may send a migration notification to the guest OS  196 ′ executing on the second hypervisor  180 B after migration (block  308 ). For example, the guest OS  196  may migrate from a migration source location to a migration destination location (illustrated as guest OS  196 ′ in  FIG. 1B ). The migrated guest OS  196 ′ is depicted in virtual machine  170 D. In an example embodiment, virtual machine  170 D may be a newly loaded virtual machine or may be the migrated virtual machine  170 A. Additionally, the second hypervisor  180 B may send the migration notification to the guest OS  196 ′ as soon as the guest OS  196  has migrated to the migration destination location. For example, the second hypervisor  180 B may detect that VM  170 A and the guest OS  196  have migrated to the migration destination location and then may send the migration notification. Then, the guest OS  196 ′ executing on the second hypervisor  180 B may receive the migration notification after migration (block  310 ). In an example embodiment, the migration notification may include instructions to perform additional operations after migration. 
     The guest driver  188 A′ in the guest OS  196 ′ executing on the second hypervisor  180 B may determine whether the state signature is a match or a mismatch with a second device  174 B on a second virtual machine  170 D after migration (block  312 ). For example, the guest driver  188 A′ may check the state signature of the second device  174 B against the state signature of the first device  174 A (e.g., device signature  191 A and device state  192 A). In an example embodiment, a pointer may be used to pass the state signature of the first device  174 A and/or second device  174 B to the guest driver  188 A′. For example, the pointer may point to the location in memory of the saved state signature of the first device  174 A. Additionally, the state signature of the second device  174 B may also be saved in memory. Additionally, the guest driver  188 A′ may notify the guest OS  196 ′ of the status of the state signature as mismatching. 
     Then, responsive to determining the status of the state signature as matching, the guest driver  188 A′ in the guest OS  196 ′ executing on the second hypervisor  180 B may load the device state  192 A to the second device  174 B after migration. For example, the device state  192 A may be loaded to the second device  174 B such that the second device  174 B has the same device state as the first device  174 A (e.g., device states  192 A-D are the same). Then, the guest OS  196 ′ may start using the second device  174 B, which advantageously provides migration with minimum downtime when the assigned device  171 B is unchanged (e.g., assigned device  171 A and  171 B are the same). 
     Conversely, responsive to determining the status of the state signature as mismatching, the guest OS  196 ′ executing on the second hypervisor  180 B may reinitialize the second device  174 B on the second virtual machine  170 D at the migration destination after migration. The state signature may mismatch between the first device  174 A and the second device  174 B if the devices have varying hardware and/or firmware versions. For example, the assigned device  171 A associated with the first device  174 A may be a PCI card with a firmware version of 10.1BB and the assigned device  171 B may be a PCI card with a firmware version of 10.1A, in which case the state signatures between the first device  174 A and the second device  174 B may not match. To account for this mismatch, the second device  174 B may be reinitialized so that the migrated system runs properly and the guest OS  196 ′ may start using the second device  174 B at the migration destination location, which advantageously prevents the guest OS  196 ′ from crashing and provides a full functionality migration with an assigned device  171 B (at the migration destination location) that differs from the assigned device  171 A (at the migration source location). 
       FIG. 4  depicts a flow diagram illustrating an example method  400  for virtual machine migration according to an example embodiment of the present disclosure. Although the example method  400  is described with reference to the flow diagram illustrated in  FIG. 4 , it will be appreciated that many other methods of performing the acts associated with the method may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional. The method may be performed by processing logic that may comprise (e.g., circuity, dedicated logic, etc.), software, or a combination of both. 
     In the illustrated example embodiment, a first device  174 A may include a device signature  191 A and a device state  192 A (block  402 ). Device state  192 A may be associated with or may be the same as the device state  192 B of the assigned device  171 A. In an example embodiment, the device signature  191 A may include a unique identifier such as a random number, a token, or a text string. For example, the device signature  191 A may be 0x1e9702 or may be a MAC Address. Additionally, the device signature  191 A may include a hardware device ID, a firmware device ID, a vendor ID and/or a version. The device signature may also include a hash function, a list of loaded firmware, memory size, etc. A hypervisor  180 A may send a request to save the device state  192 A to a guest OS  196  (blocks  404  and  406 ). For example, the hypervisor  180 A at a migration source location may send a request to save the device state  192 A from a first device  174 A. In an example embodiment, a device state may include information about the device such as the CPU state, registers, non-pageable memory used by the device, where the device is located, etc. Responsive to the hypervisor  180 A sending the request, the guest OS  196  may receive the request to save the device state  192 A (block  408 ). In an example embodiment, the request to save the device state  192 A may be a message and/or instructions received by the guest OS  196 . For example, the guest OS  196  may receive a request to save the device state  192 A of the first device  174 A associated with the assigned device  171 A. In another example embodiment, the guest OS  196  may receive a request and/or multiple requests to save device states for multiple pass-through devices associated (e.g., multiple pass-through devices in a virtual machine). 
     Then, the guest OS  196  may save a state signature of the first device  174 A in memory (e.g., guest memory  195 A) (block  410 ). In an example embodiment, the state signature may be the device signature  191 A attached to the device state  192 A and/or device state  192 B. For example, the state signature may include information associated with the device&#39;s memory, storage, and network connectivity. Additionally, the state signature may include hardware, firmware, vendor and/or version information. Then, the guest OS  196  may migrate from the source to the destination (blocks  412  to  414 ). For example, the guest OS  196  may move to a different physical machine, at the migration destination location. The migrated virtual machine  170 D may include guest OS  196 ′ (e.g., guest OS at the migration location) and may run on the underlying host OS  186 B. In an example embodiment, the host OS  186 B may be similar to host OS  186 A. In another example embodiment, the host OS  186 B may be identical to host OS  186 A. In yet another example embodiment, the host OS  186 B may be the same as host OS  186 A (e.g., same host on the same physical machine). 
     The hypervisor  180 B may send a migration notification to the guest OS  196 ′ (blocks  420  and  422 ). In an example embodiment, the hypervisor  180 B may detect that a virtual machine (e.g., virtual machine  170 A) and/or a guest OS (e.g., guest OS  196 ) has migrated to the migration destination location. For example virtual machine  170 D may be the migrated version of virtual machine  170 A (e.g., the virtual machine resumed on the migration destination location). In an example embodiment, the guest OS  196 ′ may unbind the guest driver  188 A′ in the guest OS  196 ′ executing on the second hypervisor  180 B from the first device  174 A. For example, the guest driver  188 A′ may be associated with the first device  174 A and the assigned device  171 A. Once the system migrates the virtual machine  170 A to the migration destination location, the assigned device  171 A may no longer have reason to be bound to guest driver  188 A′. Responsive to the hypervisor  180 B sending the migration notification, the guest OS  196 ′ may receive the migration notification (block  424 ). For example, the guest OS  196 ′ may receive the notification or the guest driver  188 A′ (e.g., migrated guest driver) may receive the notification. In an example embodiment, the migration notification may include instructions to perform additional operations after migration. 
     A second device  174 B with a device signature  191 B and a device state  192 C may be located at the migration destination (block  426 ). For example, the second device  174 B may be a virtual device associated with assigned device  171 B. The second device  174 B may serve as a pass-through device for the hypervisor  180 B. The device state  192 C may be the same as the device state  192 D. Then, the guest OS  196 ′ may determine whether the state signature of the first device  174 A matches the second device  174 B (block  428 ). In another example embodiment, the guest driver  188 A′ may determine whether the state signature of the first device  174 A matches the second device  174 B. For example, the guest driver  188 A′ may check the state signature of the second device  174 B against the state signature of the first device  174 A (e.g., device signature  191 A and device state  192 A). In an example embodiment, a pointer may be used to pass the state signature of the first device  174 A and/or second device  174 B to the guest driver  188 A′. For example, the pointer may point to the location in memory of the saved state signature of the first device  174 A. Additionally, the state signature of the second device  174 B may also be saved in memory. 
     If the state signature of the first device  174 A matches the second device  174 B, the guest OS  196 ′ may load the state signature to the second device  174 B (blocks  430  and  432 ). For example, the guest driver  188 A′ may load the state signature to the second device  174 B. Then, the second device  174 B may receive the loaded state signature (block  434 ). For example, the device state  192 A may be loaded to the second device  174 B such that the second device  174 B has the same device state as the first device  174 A (e.g., devices signatures  192 A-D are the same). Then, the guest OS  196 ′ may start using the second device  174 B, which advantageously provides migration with minimum downtime when the assigned device  171 B is unchanged (e.g., assigned device  171 A and  171 B are the same). In an example embodiment, the guest OS  196 ′ executing on the second hypervisor  170 D may bind the guest driver  188 A′ to the second device  174 B. For example, in the case of matching, the guest OS  196 ′ may bind guest driver  188 A′ to the second device  174 A (associated with assigned device  171 A) because the guest driver  188 A′ (e.g., migrated driver  188 A) is configured to work with the second devices (e.g., second device  174 B and assigned device  171 B) because they have matching state signatures with the original devices (e.g., first device  174 A and assigned device  171 A). Additionally, the guest OS  196 ′ executing on the second hypervisor  170 D may bind a different driver to the second device  174 A. For example, the guest OS  196 ′ may decide to use a driver that is compatible with multiple devices. 
     If the state signature of the first device  174 A does not match the second device  174 B, the guest OS  196 ′ may reinitialize the second device (block  436 ). The state signature may mismatch between the first device  174 A and the second device  174 B if the devices have varying hardware and/or firmware versions. For example, the assigned device  171 A associated with the first device  174 A may be a PCI card with a firmware version of 10.1BB and the assigned device  171 B may be a PCI card with a firmware version of 10.1A, in which case the state signatures between the first device  174 A and the second device  174 B may not match. To account for this mismatch, the second device  174 B may be reinitialized to prevent the guest OS  196 ′ from crashing and to ensure that the migrated system runs properly. The guest OS  196 ′ may start using the second device  174 B at the migration destination location, which advantageously allows a full functionality migration with an assigned device  171 B (at the migration destination location) that differs from the assigned device  171 A (at the migration source location). 
     In an example embodiment, the guest OS  196 ′ may bind a different driver in the case of mismatching to ensure that the proper guest driver is used with the second device  174 B and the associated assigned device  171 B. In another example embodiment, the guest OS  196 ′ executing on the second hypervisor  170 D may unload the guest driver  188 A′ from memory (e.g., guest memory  195 B). For example, on mismatch, a different guest driver may be used and the guest OS  196 ′ may load the different guest driver in the memory (e.g., guest memory  195 B). Then, the guest driver may be used with the second device  174 B and the associated assigned device  171 B. 
       FIG. 5  is a block diagram of an example migration system  500  according to an example embodiment of the present disclosure. The migration system  500  may include a memory  510  and one or more physical processors (e.g., processor  520  and  522 ) in communication with the memory  510 . The migration system  500  may include a first device  580  and a second device  582 . The migration system  500  may include a first hypervisor  530  executing on the one or more physical processors (e.g., processor  520 ). The first hypervisor  530  may be located at a migration source location  540 . The migration system  500  may further include a second hypervisor  532  executing on the one or more physical processors (e.g., processor  522 ). The second hypervisor  532  may be located at a migration destination location  542 . The migration system  500  may include a first virtual machine  550 . The first virtual machine  550  may include, before migration, a guest OS  560  executing on the first hypervisor  530 . The first virtual machine  550  may also include a guest driver  570  in the guest OS  560  executing on the first hypervisor  530 . The migration system  500  may further include a second virtual machine  552 . The second virtual machine  552  may include, after migration, the guest OS  560  executing on the second hypervisor  532 . The second virtual machine  552  may also include the guest driver  570  in the guest OS  560  executing on the second hypervisor  532 . The first hypervisor  530  may execute on the one or more physical processors (e.g., processor  520 ) to send a request  596  to save a device state  586  to the guest driver  570  in the guest OS  560  executing on the first hypervisor  530 . The device state  586  may be associated with the first device  580  on the first virtual machine  550 . The second hypervisor  532  may execute on the one or more processors (e.g., processor  522 ) to send a migration notification  594  to the guest OS  560  executing on the second hypervisor  532 . The guest driver  570  in the guest OS  560  executing on the first hypervisor  530  may receive the request  596  from the first hypervisor  530  and may save a state signature  598  in the memory  510 . The state signature  598  may include a device signature  590  attached to the device state  586  of the first device  580 . The guest driver  570  in the guest OS  560  executing on the second hypervisor  532  may determine a status  588  of the state signature  598  as one of matching the second device  582  and mismatching the second device  582 . 
     It will be appreciated that all of the disclosed methods and procedures described herein can be implemented using one or more computer programs or components. These components may be provided as a series of computer instructions on any conventional computer readable medium or machine readable medium, including volatile or non-volatile memory, such as RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. The instructions may be provided as software or firmware, and/or may be implemented in whole or in part in hardware components such as ASICs, FPGAs, DSPs or any other similar devices. The instructions may be configured to be executed by one or more processors, which when executing the series of computer instructions, performs or facilitates the performance of all or part of the disclosed methods and procedures. 
     It should be understood that various changes and modifications to the example embodiments described herein will be apparent to those skilled in the art. Such changes and modifications can be made without departing from the spirit and scope of the present subject matter and without diminishing its intended advantages. It is therefore intended that such changes and modifications be covered by the appended claims.