Method and system for VM-granular SSD/FLASH cache live migration

The instant disclosure describes embodiments of a system and method for migrating virtual machine (VM)-specific content cached in a solid state drive (SSD) attached to an original host. During operation, the original host receives event indicating an upcoming migration of a VM to a destination host. In response, the original host transmits a set of metadata associated with the SSD cache to the destination host. The metadata indicates a number of data blocks stored in the SSD cache, thereby allowing the destination host to pre-fetch data blocks specified in the metadata from a storage shared by the original host and the destination host. Subsequently, the original host receives a power-off event for the VM, and transmits a dirty block list to the destination. The dirty block list specifies one or more data blocks that have changed since the transmission of the metadata.

RELATED APPLICATION

The present application is related to U.S. patent application Ser. No. 13/658,567, entitled “METHOD AND SYSTEM FOR VM-GRANULAR I/O CACHING,” by inventors Li Zhou, Samdeep Nayak, and Sandeep Uttamchandani, filed 23 Oct. 2012, which is incorporated by reference in its entirety herein.

BACKGROUND

Flash memory is a non-volatile storage chip that is electrically erasable and programmable. Negated AND (NAND) type flash memories exist in a wide range of portable storage devices, such as memory cards and USB drives. In recent years, the cost of NAND flash has dropped enough to make solid state drives (SSD) a viable option as primary storage devices for computer systems, often as a replacement for conventional hard disks.

Compared with traditional hard disks, flash/SSD devices (hereinafter, also generally referred to as an “SSD” device) have superior I/O performance. In addition, the cost of SSD devices has been continuously decreasing. These two factors make it increasingly popular to use SSD devices as a “second-level” cache which resides between the main memory (e.g., RAM) and a primary persistent storage such as a hard disk (as opposed to “first-level” or buffer cache which needs to utilize a portion of the main memory). Such a “second level” cache can be used by an operating system to cache “hot” I/O blocks (e.g., I/O blocks that may be frequently accessed) to improve I/O latency and throughput. Typically, such second-level caching involves a filter driver in the OS kernel I/O stack, which can intercept all I/Os, identify hot blocks, and dispatch I/Os to a cache or persistent storage.

In a virtualized environment, however, using a SSD device as a second-level cache brings up new challenges. For example, because a host computer system (e.g., a “host”) supports the running of multiple virtual machines (VMs) in a virtualization environment, the host computer system would need to manage the second-level cache on a per-VM basis and maintain VM-specific caching policies. In addition, VM live migration, also referred to as “VMotion,” is a technology that enables moving running VMs between different hosts without service interruption, and with complete transaction integrity. During VMotion, the hypervisor moves a VM's memory to the new host over a network connection (such as Ethernet) first, and then quickly suspends the VM on the original host an resumes it on the new host. It is often critical to keep the migration latency low in order to guarantee continuous service availability for VMotion. When a host uses an SSD device as a second-level cache, how to take advantage of the SSD cache during VMotion presents a challenging problem.

SUMMARY

The instant disclosure describes embodiments of a system and method for migrating virtual machine (VM)-specific content cached in a solid state drive (SSD) attached to an original host. During operation, the original host receives event indicating an upcoming migration of a VM to a destination host. In response, the original host transmits a set of metadata associated with the SSD cache to the destination host. The metadata indicates a number of data blocks stored in the SSD cache, thereby allowing the destination host to pre-fetch data blocks specified in the metadata from a storage shared by the original host and the destination host. Subsequently, the original host receives a power-off event for the VM, and transmits a dirty block list to the destination. The dirty block list specifies one or more data blocks that have changed since the transmission of the metadata.

In a variation on this embodiment, the original host maintains the dirty block list subsequent to transmitting the metadata associated with the SSD cache.

In a variation on this embodiment, the original host suspends the transmission of the metadata associated with the SSD cache in response to receiving the power-off event for the VM.

In a further variation, the original host resumes transmission of the metadata associated with the SSD subsequent to transmitting the dirty block list.

In a further embodiment, during operation, the destination host receives an event indicating an upcoming migration of a VM to a destination host. The destination host then receives a set of metadata associated with the SSD cache from the original host. In response, the destination host pre-fetches data blocks specified in the metadata from a storage shared by the original host and the destination host, and stores the pre-fetched data blocks in an SSD cache attached to the destination host.

In a variation on this embodiment, the destination host receives a power-on event for the VM at the destination host. In response to the power-on event, the destination host receives a dirty block list from the original host, wherein the dirty block list indicate a number of blocks that have changed on the SSD cache attached to the original host. Next, the destination host invalidates one or more pre-fetched blocks based on the dirty block list.

In a further variation, subsequent to invalidating the one or more pre-fetched blocks, the destination host places the SSD cache attached to the destination host in service for the newly migrated VM.

In a further variation, subsequent to receiving the dirty block list, the destination host resumes the receiving of the metadata from the original host.

In a variation on this embodiment, the destination host receives multiple sets of SSD cache metadata which are associated with a number of migrating VMs, and pre-fetches data blocks specified by the multiple sets of SSD cache metadata based on each set of metadata's corresponding cache hit rate.

In a variation on this embodiment, the destination host discontinues pre-fetching the data blocks in response to a timeout event or a size limit being reached.

DETAILED DESCRIPTION

Certain embodiments described herein facilitate live migration of a VM-granular, SSD-based, “second-level” I/O cache by transmitting, upon a VM's VMotion, a set of metadata of the SSD cache on the original host to the destination host. This metadata indicates main storage blocks for the VM (e.g., a disk provided at a storage array and accessible via a high-speed network connection, such as Fibre Channel (FC)). Based on this metadata, the destination host can pre-fetch these blocks and populate its local SSD cache. Essentially, the content of the SSD cache does not migrate from the original host to the destination host. Instead, the destination host obtains the content (or a portion thereof) from the main storage disk, taking advantage of the high speed links between the destination host and the disk, which can be significantly faster than the network link between the original host and the destination host.

FIG. 1illustrates an exemplary system architecture of a virtualized data center which supports live migration of a VM-granular second-level I/O cache, in accordance with one embodiment. The system architecture ofFIG. 1introduces at least three components to the virtualized data center: (i) a global caching policy manager126, which, for example, may be functionality that is embedded (e.g., as a “plug-in”) into a more general VM data center management platform, such as vCenter Server from VMware, Inc. (“VMware”), (ii) a cache module110that is embedded into an I/O stack of a hypervisor108of each host122and124, serves as a communication channel with an SSD device106(or SSD devices) installed in each of hosts122and124, respectively, and manages SSD device106as a “transparent” cache for VMs (e.g., unknown to the VMs themselves) running on the hosts, and (iii) management agent120running (e.g., running in “user world,” as discussed above) on each of hosts122and124that serves as a communication intermediary between global caching policy manager126and cache module110.

Global caching policy manager126maintains VM-specific caching policies such that the policies are accessible to a number of hosts in a data center or managed cluster (e.g., such as the2host cluster comprising hosts122and124). By maintaining a VM's caching policy in global caching policy manager126, such a VM can be migrated (e.g., via live migration technologies in certain embodiments, such as VMware's VMotion migration technology) from one host to another host while ensuring that the host to which the VM will be migrated can comply with the VM's caching policy once it is migrated. For example, hypervisor108on host122may be requested (e.g., by a VM data center management platform such as vCenter Server) to run a VM118, upon which, management agent120on host122may receive a notification from hypervisor108(e.g., by registering with hypervisor108to request notification of certain VM related events, such as VM startup, migration, shutdown and the like) of the intention to start VM118, and accordingly communicates with global policy manager126to obtain the caching policy for VM118. Management agent120may communicate such a caching policy to cache module110to determine whether SSD106of host122can support such a cache requirement. In one embodiment, the communication channel between management agent120and cache module110may be implemented as I/O control (“IOCTL”) system calls by management agent120into hypervisor108. If cache module110determines that SSD device106of host122can support the caching policy of VM118, then cache module110may allocate a portion of SSD device106to be used as a cache for VM118when VM118, for example, performs I/O in order to read or write data to a virtual disk file that resides in a shared disk array102. Similar to allocating space in SSD device106to startup VM118on host122, cache module110also deallocates or otherwise invalidates “stale” cache or space in SSD device106, for example, when a VM is shutdown or migrated to a different host (or for any other cache management purposes, such as cache time-outs, full cache space, and the like). In one embodiment, when a VM data center management platform such as vCenter Server instructs hypervisor108to shutdown VM118or migrate VM118to host124, hypervisor108transmits a corresponding notification to management agent120, which, in turn, communicates with cache module110(via IOCTL) to invalidate or deallocate that portion of SSD device106which had been allocated to VM118. In an alternative embodiment, global caching policy manager, as part of a VM data center management platform such as vCenter Server, may directly communicate an intention to shutdown or migrate VM118to management agent120in host122, which, in turn communicates the intention to cache module110.

In addition to allocating and invalidating cache space in SSD device106, cache module110also facilitates the per-VM caching of data in SSD device106when VMs are performing I/O. In certain embodiments that utilize a system architecture similar to that ofFIG. 1, cache module110is implemented within a multi-pathing (MPIO) layer104of hypervisor108. As depicted inFIG. 1, MPIO layer104sits within an I/O stack of hypervisor108that includes (i) a virtual SCSI (vSCSI) layer that, for example, presents to the guest OS of VM118a virtual disk file stored in a file system in disk array102as a logical storage device (e.g., SCSI block device), and (ii) a logical device I/O scheduler116that schedules block I/O to the logical storage devices for the VMs running in a hypervisor and, for example, enforces the fair sharing of logical storage devices between VMs.

MPIO layer104receives block I/Os from logical device I/O scheduler116and, to increase fault tolerance and enhance I/O performance, appropriately selects one of a multiple of paths (e.g., one port of a number of available ports in a number of host bus adapters in host122that can route I/O to disk array102, etc.) from host122to disk array102to route I/O. The particular embodiment of MPIO layer104ofFIG. 1further implements an MPIO framework114that enables third party disk array vendors (e.g., such as the third party vendor of disk array102) to provide “plug-in” modules, such as array plugin112, that can be dynamically loaded into MPIO layer104in order to dispatch I/Os received by MPIO framework115to, for example, disk array102along a selected path and in accordance proprietary policies and techniques of the vendor. In such an embodiment, cache module110can also be inserted as a “plug-in” supported by MPIO framework114. However, as further discussed below, cache module110selectively routes block I/Os received from MPIO framework115to SSD106instead of disk array102, utilizing SSD106as a local “second level” cache.

Cache module110manages SSD device106as a cache to VMs running on host122on a “per VM” basis. In order to perform caching of I/O on a per-VM basis, cache module110associates block I/O requests that it receives from the I/O stack of hypervisor108with the particular VMs that are performing such I/O requests. In one embodiment, cache module110obtains a unique identifier for VM118from management agent120which, for example, receives the unique identifier of the VM upon receipt of a VM startup notification, as previously discussed. In certain of such embodiments, such a unique identifier may be a “universally unique identifier” (UUID) that, for example, is based on an identifier of the physical host in which a VM has been instantiated and a pathname of the location of the VM's configuration file (e.g. a .vmx file in certain VMware technology based embodiments) that is stored in shared disk array102. Cache module110may associate a VM's allocated cache space in SSD device106with this unique identifier.

Management agent120may be further configured to request and receive from hypervisor108(e.g., upon receiving a VM start notification, etc.) the identifiers or IDs of all processes associated with a VM running on hypervisor108. For example, in certain embodiments, such as embodiments using VMware's vSphere Hypervisor as hypervisor108, hypervisor108spawns a separate process or thread for each virtual CPU (vCPU) of a running VM—the process ID (PID) of each such vCPU process is sometimes referred to as a “vCPU ID” or “world ID.” In one embodiment, such as an embodiment using VMware's vSphere virtualization technologies, management agent120is able to obtain the vCPU IDs corresponding to a VM through an API provided by hypervisor108(e.g., in vSphere, referred to as the VMkernel Sysinfo Interface or VSI) and forward the vCPU IDs to cache module110in association with the VM's unique identifier. Cache module110then maintains a mapping between the unique identifier of the VM (which also identifies the area of SSD device106) which serves as a cache for such VM and the PIDs or vCPU IDs of the vCPU processes running in the VM. When a VM transmits an I/O read or write request, the PID or vCPU ID of the VM's vCPU that is requesting the I/O is also passed through the various layers of the I/O stack of hypervisor108, along with the I/O request itself, such that an I/O response can be ultimately be returned to the requesting PID or vCPU ID. As such, when the I/O request reaches cache module110, which is embedded in the I/O stack of hypervisor108, cache module110is able obtain the PID or vCPU ID associated with the I/O request and map it to the unique identifier of the VM. In doing so, cache module110is thus able to access the cache in SSD device106that is associated with the unique identify of the VM to see if the I/O request can be handled by the cache.

In virtualized data centers, VMs can move automatically between physical hosts, enabled by technologies like VMotion. When a VM migrates to a different host, there are two approaches to handle the local SSD cache on the original host: (1) discard the cache data on the original host; or (2) migrate the cache data from the original host to the destination host. The first approach simplifies design and adds little extra latency to the VMotion process. However, the tradeoff is that SSD cache “re-warm” is necessary after the VM migration. That is, the SSD cache on the destination host will need to fetch the content based on new I/O access by the migrated VM. For a large SSD cache, it could take a long time to re-warm the SSD cache, which can reduce I/O performance for the migrated VM. Particularly, in the case of a virtualized data center adopting dynamic VM migration based on resource availability, where VMs are moved automatically to hosts with more resources in order to improve VM's performance, I/O performance degradation after VMotion could confuse the management system and cause unexpected consequences.

Hence, it is more desirable to migrate the SSD cache content along with the VM during VMotion to avoid temporary performance degradation. Ideally, the SSD cache content migration should incur minimal latency, because VMotion typically demands very low latency so that the end users will not notice any interruption of service. However, a low-latency migration of SSD cache is not a trivial task because traditionally VMotion only moves the VM's memory, which is only a fraction of the size of the VM's SSD cache. How to move such a big amount of data without causing noticeable latency to VMotion presents a challenge. Since a VM needs to continue its operation during the process of live migration, new WRITEs could be issued to the virtual disk during this process. How to guarantee data integrity is another challenge. Furthermore, when there are multiple VMs migrating simultaneously to the same destination host, it is important that the destination host can properly schedule the transfer of the cached blocks to minimize the impact on I/O performance of the migrated VMs.

During VMotion, the original host and destination host transfers the VM memory over a VMotion network, which is typically a Gigabit Ethernet link. Since the size of a VM's SSD cache is much larger than its memory size, it would take a long time to transfer the image of the VM's SSD cache over the VMotion network. In addition, such cache image transfer would compete with VMotion traffic for bandwidth, which could potentially cause VMotion to fail due to timeout.

Since the actual data blocks are already available on the main storage disk, which is accessible via high-speed I/O channels from both the original host and destination host, embodiments of the system described herein address the aforementioned challenges by transferring only the SSD cache metadata over the VMotion network. Based on the transferred metadata, the destination host can rebuild the SSD cache locally by pre-fetching the data blocks from the main storage disk into the local SSD cache through the fast I/O channel, instead of obtaining the image of the SSD cache from the original host over a conventional network connection. The benefit of doing so is that the cached content transfers from the main storage disk to the destination host's over a data link (e.g., a FC link, which ca) that is much faster than a conventional Ethernet link.

With reference to the example illustrated inFIG. 1, assume that VM118is to migrate from host122to host124. First, host124creates a shadow VM. Next, host122copies each memory page of VM118to host124via a VMotion network, a process known as “preCopy.” After the first round of preCopy, host122performs another pass over VM188's memory and copies any page that has changed since the last round of preCopy. Host122continues this iterative memory copying process until there is no changed page left since the previous round of preCopy. Subsequently, a VMotion manager powers off, or “stuns,” VM118on host122, and resumes the shadow VM on host124, which uses the copied memory pages of VM118. Consequently, VM118resumes operation on host124.

To locally rebuild the SSD cache image for VM118on host124, when the first round of preCopy occurs, cache module110provides the metadata to management agent120through IOCTL. In one embodiment, the SSD cache metadata can indicate VM118's identity (e.g., VM118's UUID), the total number of cached blocks for VM118in SSD106, and the identities of each cached block. Subsequently, management agent120transmits a copy of the SSD cache metadata to management agent121on host124. As management agent121receives the metadata, management agent121forwards the metadata to cache module109, which can start pre-fetching data blocks indicated in the metadata from disk array102. Cache module109may pre-fetch the blocks by issuing READs to disk array102, and optionally coalesce the READs to big blocks when possible to improve I/O performance. Note that the pre-fetching can start before the transmission of metadata is complete, and well before VM118is stunned.

The pre-fetching process on host124can occur simultaneously as the preCopy process on host122. Since VM118is still running during the preCopy process, it is possible that VM118might issue WRITE commands to some of the blocks previously cached in SSD106. Since SSD106is a read-only cache, these WRITE commands result in updated data blocks in disk array102as well as SSD106. Because the update to the data blocks occur after the transmission of the SSD cache metadata, if host124has already pre-fetched such data blocks into SSD107, these blocks are no longer valid in SSD108. To resolve this problem, cache module110maintains a list of “dirty blocks,” which are data blocks that have been updated after host112has started transmitting the SSD cache metadata. After the preCopy process is complete for VM118, and when VM118is stunned, management agent120temporarily suspends the transmission of the SSD cache metadata if such transmission is not complete yet, and transmits this dirty block list to management agent121. Upon receiving the dirty block list, management agent121instructs cache module109to invalidate any pre-fetched block in SSD107that is on the dirty block list. At approximately the same time, VM118restarts on host124, and the corresponding cache in SSD107makes all the pre-fetched blocks available for caching service for VM118. Subsequently, transmission of the rest of SSD cache metadata (if any) resumes, and cache module109continues to pre-fetch the rest of the blocks specified in the SSD cache metadata and the dirty block list.

In general, the VM-granular SSD cache migration includes three phases, as illustrated inFIG. 2A. During phase 1, the original host enters into a pre-VMotion synchronization phase (operation202), which occurs at the same time as the preCopy process on the original host. The original host begins transmitting the SSD cache data to the destination host. The destination host can start pre-fetching blocks specified in the SSD cache metadata as soon as the metadata starts arriving at the destination host.

Phase 2 begins when the preCopy process is complete and when the VM on the original host is stunned. During phase 2, the original host and destination host synchronize the dirty blocks, which are the blocks that have been written to by the migrating VM during phase 1 (operation204). The destination host invalidates pre-fetched blocks that are on the dirty block list. At the same time, the destination host powers on the migrated VM.

After the destination host synchronizes all the dirty blocks, which is a short process since the dirty block list typically is a small file, the SSD cache on the destination host starts serving cache to the migrated VM. Subsequently, the destination host enters into phase 3, during which the SSD cache resumes pre-fetching blocks specified in the received SSD cache metadata as well as the dirty block list (operation206).

FIG. 2Bpresents a time-space diagram illustrating the process of migrating a VM-granular second-level I/O cache, in accordance with one embodiment. During operation, cache module110on the original host122first receives a pre-VMotion alert for VM118(operation212). In response, cache module110provides VM118's SSD cache metadata to management agent120, which transmits the metadata to management agent121on destination host124(phase 1). Cache module109can immediately start pre-fetching the blocks from disk array102for SSD cache107(operation213).

In one embodiment, management agent120only sends the metadata to management agent121after management agent121notifies management agent120that it is ready, as illustrated inFIG. 3A. As illustrated inFIG. 3B, if management agent121is not ready, management agent121can send management agent120a “wait” signal, in response to which management agent120temporarily suspends transmission of the metadata. When management agent121is ready to receive more metadata, management agent121sends a “ready” signal to management agent120, upon which management agent120continues to transmit the metadata. As illustrated inFIG. 3C, when management agent121determines to abandon the metadata transmission (for example, due to timeout or a cache migration policy), management agent121sends an “abandon” signal to management agent120. In response, management agent120terminates the transmission of metadata.

Referring back toFIG. 2B, note that VM118can issue new WRITEs between the beginning of the pre-VMotion process (operation212) and VM power off (operation214). In one embodiment, host122maintains a dirty block list which records the blocks updated by these new WRITEs. The existence of the dirty blocks means that host124cannot start serving the cached content immediately, even after VM118is resumed on host124. Host124must wait until cache module109receives the list for all dirty blocks.

When VM118powers off on host122(operation214), VM118will issue no more WRITEs to disk array102, and there can be no more dirty blocks. At this point, cache module110can start transferring the metadata of the dirty blocks to host124(phase 2). If the original SSD cache data has not been completely transmitted to host124, the transmission will be temporarily stopped. The reason for doing so is that host124cannot start using the cache unless the metadata for all dirty blocks are available. Therefore, as soon as VM118powers off on host122(operation214), management agent120transfers the dirty block list to management agent121. Meanwhile, VM118can start on host124before host124receives the dirty block list (operation215).

Note that SSD cache107may not start serving the migrated VM118before cache module109receives the dirty block list. If a READ by VM118to a dirty block occurs on host124, the MPIO framework on host124just passes the READ to disk array102. In the mean time, cache module109continues to pre-fetch data blocks from disk array102to build up the cache.

After receiving the dirty block list, cache module109invalidates any dirty pre-fetched block in SSD108, and starts the SSD cache service for the migrated VM118(operation216). Meanwhile, host122resumes transmission of the remaining SSD cache metadata in phase 3 (operation218). In addition, cache module109continues to traverse the SSD cache metadata and to pre-fetch blocks for the local SSD cache, until all blocks specified in the metadata are fetched, or until a timeout event occurs (operation220).

The total size of the SSD cache for a high performance VM could reach hundreds of GBs. It would take many minutes to transfer such a big dataset even over multiple 8 Gb FC or 10 Gb FCoE connections. Moreover, when an original host is experiencing resource shortage, multiple VMs could be VMotioned to other hosts simultaneously, which further increases the total size of cache data to be migrated. In one embodiment, the cached blocks are migrated in order of priority to allow blocks with the highest cache hit rate to be available the earliest on the destination host. Therefore, when lower priority blocks are moved to the destination host, they might not be “hot” blocks anymore and should have already been evicted from the cache. Hence, it may be desirable to only move blocks with higher hit rate when the cache size surpasses a threshold, and to set a timeout to the cache migration process.

In some embodiments, a cache module can use the size limit and timeout value together to avoid moving low priority blocks unnecessarily. In one embodiment, the size limit can be 10× of the VM's memory size by default, and is tunable by user on a per VM basis. The timeout value can be set to for example 30 sec by default, and again is tunable on a per VM basis by user. The destination host will stop pre-fetching blocks from the disk when either the size limit is reached or timeout occurs.

A VM often has multiple virtual disks (VMDKs). Since the SSD cache provides caching services on a per VMDK basis, and each VMDK has a different cache hit rate on its cache, it is desirable to schedule the cache migration I/O (i.e., pre-fetching) based on a weight corresponding to the cache hit rate. This can also apply to the case where multiple VMs are VMotioned simultaneously to the same destination host. In some embodiments, the caching layer in the destination host does global I/O scheduling for cache migration, with a weight set on a per VMDK basis. The pre-fetching I/Os commands for VMDK with heavier weight receive higher priority and will be scheduled more frequently. In one embodiment, the weight is equal to the cache hit rate percentage value of the VMDK. For example, if a VMDK's cache hit rate is 30%, its weight can be set at 30. The I/O scheduler iterates among the VMDKs to-be-VMotioned, and does N (N=weight value) I/Os for the VMDK each round. Therefore the VMDK with higher hit rate gets more blocks pre-fetched each round. The number of blocks fetched from disk in each round is proportional to the hit rate. In one embodiment, the hit rate of a VM-granular SSD cache is stored as a part of the cache metadata.

FIG. 4presents a state diagram for an original host that supports live migration of a VM-granular second-level I/O cache, in accordance with one embodiment. During operation, assuming the VM at issue on the original host is initially powered off. Correspondingly, the caching management agent is in an inactive state402. When the hypervisor on the original host receives a “VM POWER ON” signal, the management agent enters into an active state404. Note that if the VM receives a “VM POWER OFF” signal the management agent will return to inactive state402. While the management agent is in active state404, if the hypervisor receives a pre-VMotion alert, the management agent enters a pre-VMotion synchronization state406, in which the management agent transmits the VM's SSD cache metadata to the management agent in the destination host. If there is more metadata to transmit, or if the VM is still in a powered on state, the management agent continues to transmit the SSD cache metadata. If the management agent receives an “ABANDON” or “TIMEOUT” alert from the destination host, the management agent returns to inactive state402. If all the data has been sent or the VM has been powered off (i.e., stunned), the management agent enters into a dirty data synchronization state408, in which the management agent transmits to the destination host the dirty block list. If there is more data for the dirty block list to transmit, the management agent stays in state408. If at this time the management agent receives an “ABANDON” or “TIMEOUT” alert from the destination host, the management agent returns to inactive state402. Once all the data for the dirty block list is sent, the management agent enters a VMotion synchronization state410to resume transmission of any SSD cache metadata that has been left off when the management agent stopped to transmit the dirty block list in state408. After all the data is sent, or in response to receiving an “ABANDON” or “TIMEOUT” alert, the management agent returns to inactive state402.

FIG. 5presents a state diagram for a destination host that supports live migration of a VM-granular second-level I/O cache, in accordance with one embodiment. During operation, assuming that the hypervisor on the destination host has created an image for the VM at issue and the VM is initially power off. Corresponding, the caching management agent is in an inactive state505. When the hypervisor on the destination host receives a “VM POWER ON” signal when the VMotion is complete, the caching management agent enters into an active state502. While in active state502, the management agent can enter back into inactive state505if the hypervisor receives a “VM POWER OFF” signal. While the management agent is in inactive state505, if the hypervisor receives a pre-VMotion alert, the management agent enters a pre-VMotion synchronization state506, in which the management agent receives the VM's SSD cache metadata from the management agent in the original host. If there is more metadata to receive, the management agent continues to receive the SSD cache metadata. If the management agent receives an “ABANDON” or “TIMEOUT” alert, the management agent returns to active state502. If all the data has been received or the VM has been powered on (i.e., VMotion completed), the management agent enters into a dirty data synchronization state508, in which the management agent receives from the original host the dirty block list. If there is more data for the dirty block list to receive, the management agent stays in state508. If at this time the management agent receives an “ABANDONMENT” or “TIMEOUT” alert, the management agent returns to active state502. If all data for the dirty block list is received, the management agent enters a VMotion synchronization state510to resume receiving any SSD cache metadata that has been left off when the management agent stopped to receive the dirty block list in state508. After all the data is received, or in response to receiving an “ABANDON” or “TIMEOUT” alert, the management agent returns to active502.

FIG. 6illustrates an exemplary computer system that facilitates live migration of a VM-granular I/O cache, in accordance with one embodiment and as generally previously discussed. In the system illustrated inFIG. 6, a host605includes a set of system hardware604, which includes a set of CPUs620, memory622, an SSD616serving as a second-level cache and a high speed interface619to a shared disk array. Host602also includes a hypervisor606, which is responsible for starting, managing, and shutting down VMs. Hypervisor606manages a number of VMs614,616, and618. Furthermore, in the embodiment ofFIG. 6, hypervisor606includes an MPIO layer608, which, as previously discussed couples to a cache plugin610(e.g., corresponding to cache module110) and an array plugin612. Cache plugin610includes a world ID-to-UUID mapping table or database613, a set of per-VM caching policies614, and a per-VM cache metadata module615. Host602also includes a software management agent620running in the “user world,” as previously discussed. Management agent620includes an event handling module621, a query module622, a communication module624, and an I/O control module626.

During operation, event handling module621monitors VM related events, such as power-on, shut-down and migrate. Upon a VM's power-on, event handling module621receives the corresponding power-on event and passes the VM's UUID to query module622. Query module622in turn queries hypervisor606with the UUID and obtains all the world IDs for that VM. I/O control module626subsequently communicates the VM world ID-to-UUID mapping to cache plugin610(e.g., via IOCTL), which saves the mapping relationship in its world ID-to-UUID mapping database613. In addition, communication module624in management agent620communicates with a global caching policy manager and obtains VM-specific caching policies that specify, for example, an amount of space in SSD616to allocate as a cache for a VM. Management agent620then communicates such policies to cache plugin610(e.g., via IOCTL). As a result, cache plugin610maintains the per-VM caching policies614. Cache plugin610performs the I/O operations, such as the processes depicted inFIGS. 4 and 5.

During a VMotion process (say, for VM618), if host602is the original host, upon receiving a pre-VMotion alert, per-VM cache metadata module615provides the current set of SSD cache metadata for VM618to management agent620. Management agent620in turn sends this SSD cache data to the destination host via communication module624. Meanwhile, per-VM cache metadata module615maintains a dirty block list. Upon receiving a VM POWER OFF signal (i.e., when VM618is stunned), communication module624temporarily suspends the transmission of VM618's SSD cache metadata, and transmits the dirty block list to the destination host. After the dirty block list is successfully transmitted, communication module624resumes transmission of the rest of the SSD cache metadata, if there is any left.

If host602is the destination host, upon receiving a pre-VMotion alert, communication module624starts receiving the SSD cache metadata sent by the original host. Management agent620subsequently passes this metadata to per-VM cache metadata module615via IOCTL module626. Meanwhile, based on the received metadata, cache plugin610starts pre-fetching the data blocks from shared disk array via high speed interface619and array plugin612, and stores the pre-fetched data blocks in SSD616. When management agent620receives a POWER ON event for VM618, which indicates that the VMotion is complete, communication module624stops receiving the SSD cache metadata, and starts receiving the dirty block list. After the dirty block list is received, cache plugin610invalidates the pre-fetched blocks in SSD616based on the dirty block list. Subsequently, cache plugin610places the cache in SSD616in service for VM618. Furthermore, communication module624continues to receive the rest of the SSD cache metadata, if there is any that is left from the prior transmission.

In summary, embodiments of the present invention provide a system and method for facilitating live migration of SSD based VM-granular I/O cache. During operation, when an original host receives a pre-VMotion alert, the original host transmits a copy of the metadata for the VM-granular SSD cache. This metadata indicates the data blocks stored in the SSD cache for the VM that is being migrated. When receiving this metadata, the destination host can start pre-fetching the data blocks specified in the metadata from a shared storage disk via high speed links to the shared storage. When the VMotion is complete and VM on the destination host is powered on, the original host transmits a dirty block list to the destination host. This dirty block list indicates the data blocks in the SSD cache that has changed since the transmission of the metadata. In response, the destination host invalidates the pre-fetched blocks specified by the dirty block list, and places the local SSD cache in service for the newly migrated VM. Furthermore, the destination host can continue to receive the SSD cache metadata and pre-fetching the blocks based on the metadata, if transmission of the metadata could not be completed when the VM is powered on at the destination host.

The methods and processes described herein can be embodied as code and/or data, which can be stored in a computer-readable non-transitory storage medium. When a computer system reads and executes the code and/or data stored on the computer-readable non-transitory storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the medium.

The foregoing descriptions of embodiments of the present invention have been presented only for purposes of illustration and description. They are not intended to be exhaustive or to limit this disclosure. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. For example, while certain functions have been described as taking place within a cache module or management agent, it should be recognized that any of these functions may be implemented in either of the cache module or the management agent in alternative embodiments and that even other embodiments may implement all the functions within one component or several other components. The scope of the present invention is defined by the appended claims.