Abstract:
A method of managing remote direct memory access (RDMA) to a virtual computing instance includes suspending locally initiated RDMA operations of the virtual computing instance executing on a first host prior to a migration of the virtual computing instance to a second host. The first host includes a first hypervisor and the second host includes a second hypervisor. The method further includes requesting a peer to suspend remotely initiated RDMA operations that target the virtual computing instance through a first channel, establishing after the migration, a second channel between the peer and the second hypervisor that supports execution of the virtual computing instance on the second host, configuring a virtual object of the second hypervisor on the second host to use the second channel for the locally initiated RDMA operations, and requesting the peer to resume the remotely initiated RDMA operations using the second channel.

Description:
BACKGROUND 
       [0001]    Computer virtualization is a technique that involves encapsulating a physical computing machine platform into virtual machine(s) executing under control of virtualization software on a hardware computing platform or “host”. A virtual machine provides virtual hardware abstractions for processor, memory, storage, and the like to a guest operating system. The virtualization software, also referred to as a “hypervisor,” includes one or more virtual machine monitors (VMMs) to provide execution environment(s) for the virtual machine(s). As physical hosts have grown larger, with greater processor core counts and terabyte memory sizes, virtualization has become key to the economic utilization of available hardware. 
         [0002]    Remote direct memory access (RDMA) is a technique that enables communication with higher throughput and lower latencies by allowing devices to read and write directly to an application&#39;s memory. RDMA uses processor-offloading and operating system-bypass methods such that the protocol processing is offloaded to RDMA peripheral devices. These capabilities have driven the popularity of RDMA in high-performance computing (HPC) applications. It is desirable to support RDMA communication between virtual machines in a virtualized computing environment to provide better performance to applications deployed in VMs. 
       SUMMARY 
       [0003]    One or more embodiments include a method of managing remote direct memory access (RDMA) to a virtual computing instance including suspending locally initiated RDMA operations of the virtual computing instance executing on a first host prior to a migration of the virtual computing instance to a second host. The first host includes a first hypervisor and the second host includes a second hypervisor. The method further includes requesting a peer to suspend remotely initiated RDMA operations that target the virtual computing instance through a first channel having an RDMA transport. The method further includes establishing, after the migration, a second channel between the peer and the second hypervisor that supports execution of the virtual computing instance on the second host. The method further includes configuring a virtual object of the second hypervisor on the second host to use the second channel for the locally initiated RDMA operations. The method further includes requesting the peer to resume the remotely initiated RDMA operations using the second channel. 
         [0004]    Further embodiments include a non-transitory computer-readable storage medium comprising instructions that cause a computer system to carry out the above method, and a computer system configured to carry out the above method. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0005]      FIG. 1  is a block diagram depicting a computing system in which one or more embodiments of the present disclosure may be utilized. 
           [0006]      FIG. 2  is a block diagram depicting a host in which one or more embodiments of the present disclosure may be utilized. 
           [0007]      FIG. 3  is a flow diagram depicting a method of managing remote direct memory access (RDMA) operations in response to migration of a virtual machine (VM) according to an embodiment. 
           [0008]      FIG. 4  is a block diagram depicting communication between VMs prior to migration. 
           [0009]      FIG. 5  is a block diagram depicting communication between VMs after migration. 
           [0010]      FIG. 6  is a flow diagram depicting a method of resuming RDMA communication between a migrated VM and its peer over a TCP transport. 
           [0011]      FIG. 7  is a block diagram depicting communication between VMs after migration of a VM to a host that does not include RDMA capabilities. 
           [0012]      FIG. 8  is a flow diagram depicting a method of resuming RDMA communication between a migrated VM and its peer using memory copying. 
           [0013]      FIG. 9  is a block diagram depicting communication between VMs after migration to a common host. 
       
    
    
       [0014]    To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation. 
       DETAILED DESCRIPTION 
       [0015]      FIG. 1  is a block diagram depicting a computing system  100  in which one or more embodiments of the present disclosure may be utilized. Computing system  100  includes host computers (“hosts”)  102 A,  102 B, and  102 C (collectively “hosts  102 ”), a network  108 , and a virtualization manager  116 . While three hosts  102  are shown in the example, computing system  100  can include a plurality of hosts. Each host  102  includes a network interface  107  coupled to network  108 . Network  108  includes routers, switches, and the like that facilitate data link connections among hosts  102  and virtualization manager  116 . The data link layer of network  108  supports a plurality of transports, including a remote direct memory access (RDMA) transport  110  and/or a transmission control protocol (TCP) transport  112 . The data link layer of network  108  can support various other known transports, such as user datagram protocol (UDP) and the like. In an embodiment, the data link layer of network  108  is Ethernet. In another embodiment, RDMA transport  110  is RDMA over Converged Ethernet (RoCE). Those skilled in the art will appreciate that embodiments described herein can be used with other types of RDMA transports, such as internet Wide Area RDMA Protocol (iWARP). Network interface  107  includes functionality to support RDMA transport  110  in addition to other transports, such as TCP transport  112 . 
         [0016]    Each host  102  comprises a computer, appliance, or the like that includes a hardware platform supporting execution of virtualization software that abstracts processor, memory, storage, and networking resources for use by one or more virtual machines (VMs)  104 . An example configuration of a host  102  is shown in  FIG. 2  and described below. Each host  102  further supports execution of virtual RDMA (vRDMA) software  106 . vRDMA software  106  includes one or more components that abstract the RDMA functionality of network interface  107  for use by VMs  104 . vRDMA software  106  allows VMs  104  to share the RDMA functionality of network interface  107 , enabling virtualization of distributed workloads requiring low latency and high bandwidth provided by RDMA, such as High-Performance Computing (HPC), databases, trading systems, “big data” systems, and the like. As described herein, vRDMA software  106  exhibits low overhead while maintaining the consolidation and isolation afforded by virtualization. 
         [0017]    Virtualization manager  116  comprises a computer, appliance, or the like that includes a hardware platform supporting execution of software for managing hosts  102  and VMs  104 . Virtualization manager  116  can perform various functions, including orchestrating the migration of VMs from one host to another. vRDMA software  106  is configured to support migration of VMs  104  between hosts  102 . Following migration, the connectivity between the migrated VM and its peer(s) may have changed. vRDMA software  106  is configured to reestablish RDMA connections to each peer and ensure that the migration is transparent to the RDMA-based applications of the migrated VM. In an embodiment, vRDMA software  106  reestablishes RDMA connections using the best available transport, which may include transports other than RDMA transport  110 . For example, a VM may be migrated to a host that does not have a network interface with RDMA functionality. In such case, vRDMA software  106  can reestablish RDMA connections between a migrated VM and its peers using TCP transport  112 . In another example, a migrated VM and a peer may be executing on the same host. In such case, vRDMA software  106  can reestablish an RDMA connection between the migrated VM and its peer using direct memory copying, which may be more efficient than RDMA communication. 
         [0018]      FIG. 2  is a block diagram depicting a host  102  in which one or more embodiments of the present disclosure may be utilized. Host  102  may be constructed on a server grade hardware platform  206 , such as an x86 architecture platform or the like. As shown, hardware platform  206  includes conventional components of a computing device, such as one or more processors (CPUs)  208 , system memory  210  (also referred to as “memory  210 ”), a network interface (referred to as a host channel adapter (HCA)  212 ), storage system  214 , and other I/O devices. CPU  208  is configured to execute instructions, for example, executable instructions that perform one or more operations described herein and may be stored in memory  210  and in local storage. Memory  210  is a device allowing information, such as executable instructions and data to be stored and retrieved. Memory  210  may include, for example, one or more random access memory (RAM) modules. HCA  212  enables host  102  to communicate with another device via a communication medium, such as network  108 . HCA  212  includes functionality for communication using various transports, such as an RDMA transport and a TCP transport. Storage system  214  represents local storage devices (e.g., one or more hard disks, flash memory modules, solid state disks, and optical disks) and/or a storage interface that enables host  102  to communicate with one or more network data storage systems. Examples of a storage interface are a host bus adapter (HBA) that couples host  102  to one or more storage arrays, such as a SAN or a NAS, as well as other network data storage systems. 
         [0019]    Host  102  executes virtualization software that abstracts processor, memory, storage, and networking resources of hardware platform  206  into multiple VMs  104  that run concurrently on host  102 . VMs  104  run on top of virtualization software, shown as a hypervisor  216 , which implements platform virtualization and enables sharing of the hardware resources of host  102  by VMs  104 . One example of hypervisor  216  that may be configured and used in embodiments described herein is a VMware ESXi™ hypervisor provided as part of the VMware vSphere® solution made commercially available from VMware, Inc. of Palo Alto, Calif. (although it should be recognized that any other virtualization technologies, including Xen® and Microsoft Hyper-V® virtualization technologies may be utilized consistent with the teachings herein). Each VM  104  supports execution of a guest operating system (OS)  231 . Guest OS  231  can be any commodity operating system known in the art, such as Linux®, Microsoft Windows®, Mac OS®, or the like. 
         [0020]    In the example shown, hypervisor  216  is a Type-1 hypervisor (also referred to as a “bare-metal hypervisor”) that executes directly on hardware platform  206 . In other embodiments, host  102  can include a Type-2 hypervisor (also referred to as a “hosted hypervisor”) that executes on an operating system. One example of a Type-2 hypervisor that may be configured and used in embodiments described herein is VMware Workstation Pro™ made commercially available from VMware, Inc. (although it should be recognized that any other hosted hypervisor can be used consistent with the teachings herein, such as VirtualBox® or the like). The term “hypervisor” as used herein encompasses both Type-1 and Type-2 hypervisors, as well as hybrids thereof (e.g., a Kernel-based Virtual Machine (KVM) infrastructure operating on a Linux® kernel). 
         [0021]    Hypervisor  216  includes a kernel  232  and vRDMA device emulator  222 . Kernel  232  is a Portable Operating System Interface (POSIX) or POSIX-like kernel that supports execution of drivers, modules, services, and the like that virtualize hardware platform  206  for execution of VMs  104 . Kernel  232  includes HCA device driver  218  that provides a software interface to HCA  212 . Kernel  232  also includes RDMA stack  220  that exposes an RDMA verbs application programming interface (API). RDMA stack  220  includes an industry standard OpenFabrics Enterprise Distribution (OFED) RDMA software stack or the like. 
         [0022]    RDMA verbs include setup verbs and datapath verbs. Applications use setup verbs to create control channels and to establish communication channels between endpoints. Applications use datapath verbs to initiate transfers between endpoints and to send and receive data to and from memory. The verbs operate on a set of RDMA primitives managed by HCA  212 , including queue pairs (QPs), completion queues (CQs), and memory regions (MRs). HCA  212  manages physical objects  224 , which include QPs, CQs, and MRs. HCA  212  also includes address translations  226  for accessing MRs. 
         [0023]    QPs represent the fundamental RDMA communication channel between two endpoints. QPs include a send queue (SQ) and a receive queue (RQ). Unreliable Datagram (UD) QPs are analogous to UDP sockets, and Reliable Connection (RC) QPs are analogous to TCP sockets. Applications transfer data by enqueuing Work Requests (WRs) on QPs using datapath verbs. QPs are identified by a QP number (QPN). CQs provide a mechanism for indicating completion of WRs. HCA  212  enqueues completions when data is sent or received. An application can poll a CQ via a datapath verb or the application can optionally request for a completion to be signaled. Similar to QPs, CQs are identified by a CQ number (CQN). MRs represent the regions of memory, or buffers, that an application wishes to use for DMA. MRs are registered directly with HCA  212 , which maintains address translations  226 . MRs are identified by a local key (lkey) and a remote key (rkey). 
         [0024]    RDMA stack  220  is not directly exposed to applications running in VMs  104 . Rather, vRDMA device emulator  222  virtualizes RDMA stack  220  for each of VMs  104 . vRDMA device emulator  222  provides RDMA verb support to each VM  104 , including support for both setup and datapath verbs. vRDMA device emulator  222  maintains a set of virtual objects  228  for each vRDMA NIC  238  of VMs  104  (e.g., virtual QPs, virtual CQs, virtual MRs) and mappings between virtual objects  228  and physical objects  224  (“virtual-to-physical mappings  230 ”). As discussed below, each VM  104  can include one or more vRDMA NICs  238 , each of which is part of the VM state. vRDMA device emulator  222  maintains separate instances for each vRDMA NIC  238  in each VM  104 . vRDMA device emulator  222  forwards verbs from VMs  104  to RDMA stack  220 , performing the necessary translations between virtual objects  228  and physical objects  224  (e.g., virtual QPNs to physical QPNs). When HCA  212  completes the operation of a verb, vRDMA device emulator  222  receives the response and re-translates the completion information for the VM (e.g., physical QPN to virtual QPN). 
         [0025]    In addition, vRDMA device emulator  222  can emulate verb operations. When a host does not include an HCA, vRDMA device emulator  222  emulates RDMA verbs and primitives using another transport, such as a TCP transport. This allows applications to continue to communicate even if a physical RDMA link is unavailable. If applications are in VMs executing on the same host, vRDMA device emulator  222  can emulate RDMA verbs and primitives using memory copying operations (memcpy). In general, vRDMA device emulator  222  selects between three different transports based on the endpoints: vRDMA-over-memcpy for host-local peers; vRDMA-over-RDMA for peers on a physical RDMA network; and vRDMA-over-TCP for remaining peers. 
         [0026]    Each VM  104  includes a guest OS  231 , one or more applications  240 , and one or more vRDMA network interface cards (NICs)  238 . vRDMA NIC  238  is a virtual network device provided to VM  104  by hypervisor  216 . In an embodiment, vRDMA NIC  238  is a virtual Peripheral Component Interface Express (PCIe) device. vRDMA NIC  238  interfaces with vRDMA device emulator  222 . Guest OS  231  can be any commodity operating system, such as Linux®, Microsoft Windows®, or the like. For some RDMA operations, application(s)  240  interface with guest OS  231 , which in turn interfaces with vRDMA NIC  238  (e.g., setup operations). For other RDMA operations, such as datapath operations, application(s)  240  directly interface with vRDMA NIC  238 . Application(s)  240  establish buffers  244  to store data to be transferred and data to be received. 
         [0027]    Guest OS  231  includes libraries  238 , vRDMA driver  234 , and the standard OFED RDMA stack  236 . Libraries  238  can include user-space and kernel-space libraries that cooperate with RDMA stack  236  to expose the RDMA verbs API to application(s)  240  and guest OS  231 . vRDMA driver  234  provides a software interface to vRDMA NIC  238 . vRDMA driver  234  interacts with vRDMA NIC  238  at the virtual hardware level via Base Address Registers (BARs) and memory mapped input/output (MMIO) pages (e.g., MMIO pages  242 ) to communicate with vRDMA device emulator  222 . 
         [0028]    The vRDMA software implements both OS-bypass and zero-copy features. For OS-bypass, each application  240  has direct access to vRDMA NIC  238  for datapath verbs using its own MMIO page(s)  242 . Each MMIO page  242  can include doorbells, which are specific areas in the page that indicate the different datapath verbs. When application  240  invokes a given datapath verb, libraries  238  write to the corresponding doorbell in the MMIO page allocated to the application. vRDMA device emulator  222  traps writes to the doorbells to receive and process the datapath verbs. Thus, RDMA datapath operations bypass guest OS  231  and RDMA stack  236 . 
         [0029]    Buffers  244  are assigned virtual addresses within the address space of application(s)  240  (e.g., guest virtual addresses). HCA  212  transfers data to/from buffers  244  using DMA. HCA  212  includes address translations  226  in order to translate between the virtual addresses and machine memory addresses in order to perform the DMA operations. RDMA stack  220  exposes an API to vRDMA device emulator  222  for registering guest virtual addresses directly with HCA  212 . 
         [0030]      FIG. 3  is a flow diagram depicting a method  300  of managing RDMA operations in response to migration of a VM according to an embodiment. Method  300  includes steps performed by a plurality of hypervisors, in particular, a plurality of instances of vRDMA device emulator  222 . In the present example, method  300  relates a first VM (VM 1 ) in a first host (host 1 ) communicating with a second VM (VM 2 ) in a second host (host 2 ) as an RDMA peer. VM 1  is being migrated from host 1  to a third host (host 3 ).  FIG. 4  is a block diagram depicting communication between VM 1  and VM 2  prior to migration of VM 1  and  FIG. 5  is a block diagram depicting communication between VM 1  and VM 2  after migration of VM 1 . 
         [0031]    As shown in  FIG. 4 , VM 1  is executing in a host  450  (host 1 ) and VM 2  is executing in a host  452  (host 2 ). Host  450  includes a hypervisor configured with a vRDMA device emulator  404  and host  452  includes a hypervisor configured with a vRDMA device emulator  414 . Host  450  includes an HCA  408  and host  452  includes an HCA  418 . HCA  408  is managing physical objects (e.g., QPs, CQs, MRs) having a protocol state  410 . HCA  418  is managing physical objects having a protocol state  420 . Protocol state  410 ,  420  can include various RDMA protocol state information that is exposed to the VMs, such as packet sequence numbers (PSNs). vRDMA device emulator  404  includes mappings  406  of virtual objects to physical objects, and vRDMA device emulator  414  includes mappings  416  of virtual objects to physical objects. vRDMA device emulator  404  communicates with vRDMA device emulator  414  over a control channel  422  (e.g., a TCP channel). HCA  408  communicates with HCA  418  over an RDMA channel  424 . As shown in  FIG. 5 , VM 2  is executing in host  452  as described above in  FIG. 4 . VM 1  is migrated to a host  454  having a vRDMA device emulator  510  and an HCA  514 . 
         [0032]    Returning to  FIG. 3 , at step  301 , vRDMA device emulator  404  suspends RDMA operations for VM 1 . In order to migrate VM 1  without loss of information, both sides of RDMA channel  424  must be brought into a consistent transport-independent state. At any given time, there can be a number of RDMA operations queued by HCA  408 . The state of these operations is not visible to VM 1 , so vRDMA device emulator  404  suspends new operations and allows pending operations to complete. At step  302 , vRDMA device emulator  404  sends a request to vRDMA device emulator  414  to have VM 2  quiesce RDMA operations in response to a pending migration operation. 
         [0033]    At step  303 , vRDMA device emulator  414  suspends RDMA operations of VM 2  that target VM 1 . vRDMA device emulator  414  suspends new operations from VM 2  that target VM 1  and allows pending operations to complete. At step  304 , once all pending operations have completed, vRDMA device emulator  414  saves protocol state  420  and closes connections to VM 1  through RDMA channel  424 . At step  306 , vRDMA device emulator  414  acknowledges the quiescence to vRDMA device emulator  404 . At step  310 , vRDMA device emulator  414  closes connections to vRDMA device emulator  404  over control channel  422 . 
         [0034]    At step  308 , vRDMA device emulator  404  saves protocol state  410  and closes connections over RDMA channel  424 . At step  312 , vRDMA device emulator  404  signals for the migration to proceed. 
         [0035]    At step  314 , after migration, vRDMA device emulator  510  recreates its virtual objects  228  ( FIG. 2 ), as well as physical objects on HCA  514 , and restores protocol state  410  to the virtual objects. However, when recreating the physical objects, the newly created objects (e.g., QPs, CQs, MRs) can have different identifiers than prior to the migration. Thus, vRDMA device emulator  510  will have new mappings  512  that map virtual objects to physical objects. At step  316 , vRDMA device emulator  510  recreates a control channel  506  with vRDMA device emulator  414 . At step  318 , vRDMA device emulator  510  sends mappings  512  to vRDMA device emulator  414 . 
         [0036]    At step  320 , vRDMA device emulator  414  recreates its virtual objects  228  ( FIG. 2 ), as well as the physical objects on HCA  418 , and restores protocol state  420  to the virtual objects and physical objects. In addition, vRDMA device emulator  414  creates new mappings  502  that map virtual objects to physical objects. At step  322 , vRDMA device emulator  414  sends mappings  502  to vRDMA device emulator  510 . At step  324 , vRDMA device emulator  510  updates mappings  512  and restores protocol state  410  to the physical objects on HCA  514 . Once the protocol state  410 ,  420  is restored by both vRDMA device emulators  414 ,  510  the RDMA channel  508  is established between HCA  514  and HCA  418 . At step  326 , vRDMA device emulator  510  requests that vRDMA device emulator  414  unquiesce. At step  328 , vRDMA device emulator  414  acknowledges the unquiesce to vRDMA device emulator  510 . At step  332 , vRDMA device emulator  510  directs VM 1  to resume RDMA operations. At step  330 , vRDMA device emulator  414  directs VM 2  to resume RDMA operations. In this manner, VM 1  and VM 2  will continue their RDMA operations with little or no interruption and without data loss. From the perspective of RDMA operations, neither VM 1  nor VM 2  will be aware of the migration. 
         [0037]      FIG. 3  shows the case where VM 1  is being migrated to a host that includes an HCA capable of RDMA communication. In some cases, VM 1  may be migrated to a host that does not have an HCA capable of RDMA communication.  FIG. 6  is a flow diagram depicting a method  600  of resuming RDMA communication between a migrated VM and its peer over a TCP transport.  FIG. 7  is a block diagram depicting communication between VM 1  and VM 2  after migration of VM 1  to a host  708  that does not include an HCA capable of RDMA communication. 
         [0038]    Referring to  FIGS. 6-7 , method  300  is performed through step  312 . Method  600  begins after step  312  at step  602 , where vRDMA device emulator  710  in host  708  emulates physical objects of an HCA and restores protocol state  410  thereto (emulated objects  704 ). Since vRDMA device emulator  710  is emulating the physical objects, vRDMA device emulator  710  can maintain the same identifiers used prior to migration (e.g., the same QPNs, CQNs, MR keys). Thus, vRDMA device emulator  710  does not need to create or change any virtual-to-physical object mappings. At step  604 , vRDMA device emulator  710  recreates a control channel  706  (TCP channel  706 ) with vRDMA device emulator  414 . At step  606 , vRDMA device emulator  710  requests vRDMA device emulator  414  to unquiesce and notifies vRDMA device emulator  414  of the physical object emulation. 
         [0039]    At step  608 , vRDMA device emulator  414  emulates physical objects of HCA  418  and restores protocol state  420  (from  FIG. 4 ) thereto. vRDMA device emulator  414  can maintain the same identifiers used prior to migration (e.g., the same QPNs, CQNs, MR keys). Thus, vRDMA device emulator  414  does not need to update virtual-to-physical object mappings. At step  610 , vRDMA device emulator  414  acknowledges the unquiescence to vRDMA device emulator  710 . At step  614 , vRDMA device emulator  710  directs VM 1  to resume RDMA operations, which will occur over TCP channel  706 . At step  612 , vRDMA device emulator  414  directs VM 2  to resume RDMA operations, which will occur over TCP channel  706 . The use of TCP channel  706  for RDMA communications is transparent to VM 1  and VM 2 . In this manner, even if VM 1  is migrated to a host that is not capable of RDMA communications, the RDMA operations between VM 1  and VM 2  will resume after the migration. 
         [0040]    In some cases, VM 1  may be migrated to the same host as its peer.  FIG. 8  is a flow diagram depicting a method  800  of resuming RDMA communication between a migrated VM and its peer using memory copy.  FIG. 9  is a block diagram depicting communication between VM 1  and VM 2  after migration of VM 1  to host  452 . Referring to  FIGS. 8-9 , method  300  is performed through step  312 . Method  800  begins after step  312  at step  802 , where an instance of vRDMA device emulator created in host  452  for a vRDMA NIC in VM 1  (referred to as vRDMA device emulator instance  414 B) emulates physical objects of an HCA and restores protocol state  410  thereto (emulated objects  904 ). Since vRDMA device emulator instance  414 B is emulating the physical objects, vRDMA device emulator instance  414 B can maintain the same identifiers used prior to migration (e.g., the same QPNs, CQNs, MR keys). Thus, vRDMA device emulator instance  414 B does not need to create or change any virtual-to-physical object mappings. At step  804 , vRDMA device emulator instance  414 B recreates a control channel  906 . At step  806 , vRDMA device emulator instance  414 B requests an instance of vRDMA device emulator created in host  452  for a vRDMA NIC in VM 2  (referred to as vRDMA device emulator instance  414 A) to unquiesce and notifies vRDMA device emulator instance  414 B of the physical object emulation. 
         [0041]    At step  808 , vRDMA device emulator instance  414 A emulates physical objects of HCA  418  and restores protocol state  420  (from  FIG. 4 ) thereto. vRDMA device emulator instance  414 A can maintain the same identifiers used prior to migration (e.g., the same QPNs, CQNs, MR keys). Thus, vRDMA device emulator instance  414 A does not need to update virtual-to-physical object mappings. At step  810 , vRDMA device emulator instance  414 A acknowledges the unquiescence. At step  814 , vRDMA device emulator instance  414 B directs VM 1  to resume RDMA operations, which will occur using memory copying. At step  812 , vRDMA device emulator instance  414 A directs VM 2  to resume RDMA operations, which will occur using memory copying. The use of memory copying for RDMA communications is transparent to VM 1  and VM 2 . In this manner, the efficiency of communication between VM 1  and VM 2  can be increased in such cases by using intra-memory data transfers rather than RDMA transport. 
         [0042]    In the example method  300  of  FIG. 3 , a first VM (VM 1 ) executes in a first host (host 1 ) and communicates with a second VM (VM 2 ) in a second host (host 2 ) as an RDMA peer. In another embodiment, prior to the migration of VM 1  from host 1  to the third host (host 3 ), VM 2  can be executing on the first host with VM 1 . In such an embodiment, there is no second host (host 2 ) and only the first host (host 1 ) and the third host (host 3 ). Also, in such an embodiment, the RDMA operations between VM 1  and VM 2  can occur using memory copy operations, as described above with respect to  FIGS. 8 and 9 . Thus, prior to migration, the RDMA operations between VM 1  and VM 2  occur using memory copy operations in host 1 . After migration, the RDMA operations between VM 1  and VM 2  occur over an RDMA channel or over a TCP channel, as described above in  FIGS. 3 and 6 . 
         [0043]    Certain embodiments as described above involve a hardware abstraction layer on top of a host computer. The hardware abstraction layer allows multiple contexts to share the hardware resource. In one embodiment, these contexts are isolated from each other, each having at least a user application running therein. The hardware abstraction layer thus provides benefits of resource isolation and allocation among the contexts. In the foregoing embodiments, virtual machines are used as an example for the contexts and hypervisors as an example for the hardware abstraction layer. As described above, each virtual machine includes a guest operating system in which at least one application runs. It should be noted that these embodiments may also apply to other examples of contexts, such as containers not including a guest operating system, referred to herein as “OS-less containers” (see, e.g., www.docker.com). OS-less containers implement operating system-level virtualization, wherein an abstraction layer is provided on top of the kernel of an operating system on a host computer. The abstraction layer supports multiple OS-less containers each including an application and its dependencies. Each OS-less container runs as an isolated process in user-space on the host operating system and shares the kernel with other containers. The OS-less container relies on the kernel&#39;s functionality to make use of resource isolation (CPU, memory, block I/O, network, etc.) and separate namespaces and to completely isolate the application&#39;s view of the operating environments. By using OS-less containers, resources can be isolated, services restricted, and processes provisioned to have a private view of the operating system with their own process ID space, file system structure, and network interfaces. Multiple containers can share the same kernel, but each container can be constrained to only use a defined amount of resources such as CPU, memory and I/O. The term “virtualized computing instance” as used herein is meant to encompass both VMs and OS-less containers. 
         [0044]    The various embodiments described herein may employ various computer-implemented operations involving data stored in computer systems. For example, these operations may require physical manipulation of physical quantities—usually, though not necessarily, these quantities may take the form of electrical or magnetic signals, where they or representations of them are capable of being stored, transferred, combined, compared, or otherwise manipulated. Further, such manipulations are often referred to in terms, such as producing, identifying, determining, or comparing. Any operations described herein that form part of one or more embodiments of the invention may be useful machine operations. In addition, one or more embodiments of the invention also relate to a device or an apparatus for performing these operations. The apparatus may be specially constructed for specific required purposes, or it may be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations. 
         [0045]    The various embodiments described herein may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. 
         [0046]    One or more embodiments of the present invention may be implemented as one or more computer programs or as one or more computer program modules embodied in one or more computer readable media. The term computer readable medium refers to any data storage device that can store data that can thereafter be input to a computer system—computer readable media may be based on any existing or subsequently developed technology for embodying computer programs in a manner that enables them to be read by a computer. Examples of a computer readable medium include a hard drive, network attached storage (NAS), read-only memory, random-access memory (e.g., a flash memory device), a CD (Compact Discs)—CD-ROM, a CD-R, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion. 
         [0047]    Although one or more embodiments of the present invention have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be made within the scope of the claims. Accordingly, the described embodiments are to be considered as illustrative and not restrictive, and the scope of the claims is not to be limited to details given herein, but may be modified within the scope and equivalents of the claims. In the claims, elements and/or steps do not imply any particular order of operation, unless explicitly stated in the claims. 
         [0048]    Virtualization systems in accordance with the various embodiments may be implemented as hosted embodiments, non-hosted embodiments or as embodiments that tend to blur distinctions between the two, are all envisioned. Furthermore, various virtualization operations may be wholly or partially implemented in hardware. For example, a hardware implementation may employ a look-up table for modification of storage access requests to secure non-disk data. 
         [0049]    Many variations, modifications, additions, and improvements are possible, regardless the degree of virtualization. The virtualization software can therefore include components of a host, console, or guest operating system that performs virtualization functions. Plural instances may be provided for components, operations or structures described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention(s). In general, structures and functionality presented as separate components in exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements may fall within the scope of the appended claim(s).