Patent Publication Number: US-11036530-B2

Title: Application continuous high availability solution

Description:
BACKGROUND 
     Virtualization is the process of creating a software-based (or virtual) representation of something, including virtual computer hardware platforms, operating systems, storage devices, and computer network resources. Virtualization can apply to applications, servers, storage, and networks and is an effective way to reduce IT expenses while boosting efficiency and agility for all size businesses. 
     Virtualization can increase IT agility, flexibility, and scalability while creating significant cost savings. Workloads get deployed faster, performance and availability increases and operations become automated, resulting in IT that is simpler to manage and less costly to own and operate. However, for mission critical services, while existing virtualization platforms may support safeguards to ensure data integrity, detection of and recovery from unexpected crashes still require non-trivial amount of time, resulting in a perceived interruption in such services. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates an example cluster of hosts. 
         FIG. 2  illustrates a primary virtual machine and a secondary virtual machine in a fault tolerance pair. 
         FIG. 3  shows a virtual machine (VM) system in examples of the present disclosure. 
         FIG. 4  illustrates a method for the system of  FIG. 3  to provide application-level continuous availability with close to zero downtime in examples of the present disclosure. 
         FIG. 5  is a flowchart of a method for primary and secondary fault tolerance (FT) agents in the system of  FIG. 3  to perform in examples of the present disclosure. 
         FIG. 6  is a flowchart of a method for an application on a primary host, a primary high availability (HA) agent, and a secondary HA agent in the system of  FIG. 3  to perform in examples of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein. 
     VMware vSphere is an example platform for virtualization and cloud infrastructure. VMware vSphere manages collections of infrastructure, such as, without limitation, central processing units (CPUs), storage, and networking, as a seamless and dynamic operating environment, and also manages the complexity of a datacenter. VMware vSphere may include various components, such as, without limitation, VMware ESXi, an ESXi host, and a vCenter server. VMware ESXi is a hypervisor (e.g., a virtualization layer) that abstracts processor, memory, storage, and other physical resources into multiple virtual machines (VMs), and an ESXi host generally refers to a physical server that runs VMware ESXi. A vCenter server generally refers to a central point for configuring, provisioning, and managing virtualized information technology environments. 
       FIG. 1  illustrates an example vSphere cluster  100 , which includes a group of ESXi hosts administered collectively by a vCenter server. vSphere cluster  100  may be enabled for vSphere High Availability (HA) (hereafter referred to as “vSphere HA cluster”). vSphere HA protects against an ESXi host failure by restarting its virtual machines on other ESXi hosts within the cluster. vSphere HA also protects against application failure by continuously monitoring a virtual machine or an application on the virtual machine and resetting the virtual machine in the event that a failure is detected. In some embodiments, there are three components that form the foundation for HA: Fault Domain Manager (FDM), HOSTD, and vCenter server. 
     FDM is the HA agent in ESXi that is responsible for tasks including communicating host resource information, virtual machine states, and HA properties to other ESXi hosts in the cluster. FDM also handles heartbeat mechanisms, virtual machine placement, virtual machine restarts, and logging. 
     HOSTD is the agent in ESXi responsible for tasks including powering on virtual machines. FDM communicates directly with HOSTD and vCenter server. FDM relies on HOSTD for information about the virtual machines that are registered to the ESXi host, and manages the virtual machines using HOSTD APIs. 
     vCenter server in vSphere cluster  100  is responsible for tasks including deploying and configuring FDM agents, communication of cluster configuration changes, and protection of virtual machines. vCenter server is responsible for pushing out the FDM agent to the ESXi hosts. vCenter server is also responsible for communicating configuration changes in the cluster to the host that is elected as the master. 
     When a user adds an ESXi host to vSphere HA cluster  100 , vCenter server uploads an FDM agent to the ESXi host and configures it to communicate with other FDM agents in the vSphere HA cluster. Each ESXi host in vSphere HA cluster  100  functions as a master host or a slave host. 
     When HA is first enabled in vSphere HA cluster  100 , all active hosts (those not in standby or maintenance mode, or not disconnected) participate in an election to choose the cluster&#39;s master host. Only one master host exists per cluster and all other hosts are slave hosts. If the master host fails, is shut down, or is removed from vSphere HA cluster  100 , the slave hosts hold a new election. 
     The master host has a number of responsibilities. The master host monitors the state of slave hosts. If a slave host fails or becomes unreachable, the master host identifies which virtual machines need to be restarted. 
     The master host monitors the power state of all protected virtual machines. If one virtual machine fails, the master host ensures that it is restarted. Using a local placement engine, the master host also determines where the restart should be done. 
     The master host manages the lists of cluster hosts and protected virtual machines. 
     The master host acts as vCenter server management interface to the cluster and reporting the cluster health state. 
     The slave hosts contribute to vSphere HA cluster  100  by running virtual machines locally, monitoring their runtime states, and reporting state updates to the master host. 
     The master host monitors the liveness of the slave hosts in the cluster. This communication is done through the periodic exchange of network heartbeats. When the master host stops receiving these heartbeats from a slave host, it checks for host liveness before declaring the host to have failed. The master host performs the liveness check by determining whether the slave host is exchanging heartbeats with one or more shared datastores. 
     HA also provides virtual machine and application monitoring, which is performed by the FDM agent on each host. If the FDM agent does not receive heartbeats for a specific (and configurable) amount of time from a virtual machine or an application, the FDM agent restarts the virtual machine or the application. 
       FIG. 2  illustrates vSphere Fault Tolerance (FT) with a primary virtual machine and a secondary virtual machine in a fault tolerance pair. vSphere FT provides continuous availability to virtual machines, eliminating downtime and disruption—even in the event of a complete host failure. 
     vSphere FT works by continuously replicating an entire running virtual machine from one ESXi host to another. The result is that an FT-protected virtual machine has two replicas: the primary virtual machine and the secondary virtual machine, each running on distinct ESXi hosts. These replicas are logically identical—they represent a single virtual machine state and a single network identity, but they are physically distinct. Each replica has its own set of virtual machine files (including VMX and VMDK files), which vSphere FT automatically keeps in sync. When an ESXi host fails, one of the replicas will resume execution, and the virtual machine state, the network identity, and all active network connections for the virtual machine will be identical, ensuring a seamless failover process. vSphere FT is implemented by FT agent in the ESXi using four underlying technologies: storage, runtime state, network, and transparent failover. 
     vSphere FT ensures the storage of the primary and secondary virtual machines is always kept in sync. When vSphere FT protection is enabled, an initial synchronization of the virtual machine disks (VMDKs) occurs to ensure the primary and secondary virtual machines have the exact same disk state. 
     This initial synchronization happens whenever FT is turned on while the virtual machine is running, whenever FT protection is re-established after a failover occurs, or whenever a powered-off FT virtual machine powers on. 
     After this initial synchronization, vSphere FT will mirror VMDK write operations between the primary and secondary virtual machines over the FT network to ensure the storage of the replicas continues to be identical. 
     vSphere FT ensures the runtime state of the two replicas is always identical. It does this by continuously capturing the active memory and precise execution state of the virtual machine, and rapidly transferring them over a high-speed network, allowing the virtual machine to instantaneously switch from running on the primary ESXi host to the secondary ESXi host whenever a failure occurs. 
     The networks used by the virtual machine are also virtualized by the underlying ESXi host, ensuring that even after a failover, the virtual machine identity and network connections are preserved. vSphere FT manages the virtual MAC address as part of the process. If the secondary virtual machine is activated, the secondary ESXi host sends a gratuitous ARP so the network switch is aware of the new physical location of the virtual MAC address. Since vSphere FT preserves the storage, the precise execution state, the network identity, and the active network connections, the result is zero downtime and no disruption to users should an ESXi host failure occur. 
     vSphere FT ensures that the primary virtual machine always agrees with the secondary virtual machine. This is achieved by holding externally visible output from the primary virtual machine, and only releasing it when an acknowledgement is made from the secondary virtual machine affirming that the state of the two virtual machines is consistent (for the purposes of vSphere FT, externally visible output is network transmissions). 
     As described, vSphere HA provides hardware and virtual hardware layer protection for virtual machines. For guest OS and application layers, virtual machine and application monitoring provides rapid recovery from failures. The downtime from restarting virtual machines or applications ranges from seconds to minutes. This downtime, though quite small, is unacceptable for customers that provides business critical services. 
     For example, telecom operators provide high degree of service ability with their business critical applications. Even a few seconds of downtime can be fatal for their services. Similar or same situation applies to customers like banks and public cloud providers. 
     vSphere FT provides continuous protection with zero downtime and no loss of state or interruption in service. However, vSphere FT cannot detect guest OS or application level failures. If an ESXi host in a vSphere HA cluster hangs or crashes due to guest OS or application level errors, the secondary virtual machine would nonetheless copy the state of the primary virtual machine and also hang or crash. 
       FIG. 3  shows a virtual machine (VM) system  300  in examples of the present disclosure. System  300  includes virtualization host computers  302 - 1 ,  302 - 2 , and  302 - 3  (collectively as “hosts  302 ” or generically as an individual “host  302 ”) coupled by a network  304 . Host  302 - 1  includes physical memory, processor, local storage, and network interface cards (NICs). Host  302 - 1  runs a hypervisor  306 - 1  to create and run a virtual machine  308 - 1 . Hypervisor  306 - 1  includes a HA agent  312 - 1  and a FT agent  314 - 1 . Virtual machine  308 - 1  runs a guest OS  316 - 1  to run an application  318 - 1 . Host  302 - 1  is coupled to a datastore  320 - 1 , which stores the virtual machine disks for virtual machines  308 - 1 . Hosts  302 - 2  and  302 - 3  are similarly configured. 
     A virtualization manager  322  centrally provisions and manages virtual and physical objects in VM system  300 , such as virtual machines, clusters, and hosts. Virtualization manager  322  may run on one of hosts  302  or a dedicated host (not shown) coupled by network  304  to hosts  302 . Together hypervisors  306 - 1 ,  306 - 2 ,  306 - 3  and virtualization manager  322  provide a virtualization platform that can implement information technology services such as web services, database services, and data processing services. Hypervisor  306 - 1 ,  306 - 2 , and  306 - 3  may be VMware vSphere ESXi hypervisors, and virtualization manager  322  may be a VMware vCenter server. 
       FIG. 4  illustrates a method for VM system  300  to provide application-level continuous availability with close to zero downtime in examples of the present disclosure. For simplicity and clarity, some elements of VM system  300  are omitted. 
     Virtual machine  308 - 1  is a primary virtual machine and virtual machine  308 - 2  is a secondary virtual machine in a fault tolerance pair. Secondary virtual machine  308 - 2  runs a guest OS  316 - 2  and an application  318 - 2  that are identical to guest OS  316 - 1  and application  318 - 1  on primary virtual machine  308 - 1 . 
     FT agent  314 - 1  on primary host  302 - 1  (hereafter “primary FT agent  314 - 1 ”) records and transmits activities  402  of primary virtual machine  308 - 1  to secondary host  302 - 2 . Instead of immediately replaying activities  402  to secondary virtual machine  308 - 2 , FT agent  314 - 2  on secondary host  302 - 2  (hereafter “secondary FT agent  314 - 2 ”) buffers them. Secondary FT agent  314 - 2  waits to receive a notification that the buffered activities  402  are safe before replaying them to secondary virtual machine  308 - 2 . 
     Application  318 - 1  on primary host  302 - 1  sends heartbeats to HA agent  312 - 1  on primary host  302 - 1  (hereafter “primary HA agent  312 - 1 ”) to indicate it is healthy. Primary HA agent  312 - 1  forwards the heartbeats to HA agent  312 - 2  on secondary host  302 - 2  (hereafter “secondary HA agent  312 - 2 ”). When primary HA agent  312 - 1  does not receive a heartbeat from application  318 - 1  for a specific (and configurable) time interval, the primary HA agent  312 - 1  declares to primary FT agent  314 - 1  that primary virtual machine  308 - 1  has failed. 
     When secondary HA agent  312 - 2  receives a heartbeat within the specific time interval, the secondary HA agent sends the notification to secondary FT agent  314 - 2  that the buffered activities  402  are safe to replay to secondary virtual machine  308 - 2 . When secondary HA agent  312 - 2  does not receive a heartbeat within the specific time interval, the secondary HA agent initiates failover by secondary FT agent  314 - 2  for secondary virtual machine  308 - 2  to become the new primary virtual machine and have application  318 - 2  take over services provided by the failed application  318 - 1 . 
       FIG. 5  is a flowchart of a method  500  for primary FT agent  314 - 1  and secondary FT agent  314 - 2  ( FIG. 3 ) in examples of the present disclosure. Method  500  may be performed in response to user input enabling fault tolerance for virtual machine  308 - 1 , thereby making it the primary virtual machine. Blocks  502  and  504  represent actions performed by primary FT agent  314 - 1 , and blocks  506  to  518  represent actions performed by secondary FT agent  314 - 2 . 
     In block  502 , primary FT agent  314 - 1  initially synchronizes primary virtual machine  308 - 1  ( FIG. 3 ) to secondary virtual machine  308 - 2  ( FIG. 3 ). Block  502  may be followed by block  504 . 
     In block  504 , primary FT agent  314 - 1  records activities  402  ( FIG. 4 ) of primary virtual machine  308 - 1  and sends them to secondary FT agent  314 - 2 . Block  504  may loop back to itself to record new activities of primary virtual machine  308 - 1  and sends them to secondary FT agent  314 - 2 . 
     In block  506 , secondary FT agent  314 - 2  creates secondary virtual machine  308 - 2  matched with primary virtual machine  308 - 1  in a fault tolerance pair. Block  506  may be followed by block  508 . 
     In block  508 , secondary FT agent  314 - 2  initially synchronizes secondary virtual machine  308 - 2  to primary virtual machine  308 - 1 . Block  508  corresponds to block  502  performed by primary FT agent  314 - 1 . Block  508  may be followed by block  510 . 
     In block  510 , secondary FT agent  314 - 2  receives activities  402  of the primary virtual machine  308 - 1  from primary FT agent  314 - 1  and buffers them. For example, secondary FT agent  314 - 2  saves activities  402  in a back buffer  404  of a double buffer  406  ( FIG. 4 ). Block  510  corresponds to block  504  performed by primary FT agent  314 - 1 . Block  510  may be followed by block  512 . 
     In block  512 , secondary FT agent  314 - 2  determines if it has received a notification from HA agent  312 - 2  that the buffered activities  402  are safe to replay to secondary virtual machine  308 - 2 . If so, block  512  may be followed by block  514 . Otherwise, block  512  may be followed by block  516 . 
     In block  514 , secondary FT agent  314 - 2  flips double buffer  406  and replays the buffered activities  402  from a front buffer  408  ( FIG. 4 ) to secondary virtual machine  308 - 2 . Block  514  may loop back to block  510  to save new activities of primary virtual machine  308 - 1  received from primary FT agent  314 - 1  in back buffer  404 . 
     In block  516 , secondary FT agent  314 - 2  discards the (faulty) buffered activities  402  in back buffer  404 . Block  516  may be followed by block  518 . 
     In block  518 , secondary FT agent  314 - 2  sets secondary virtual machine  308 - 2  as a new primary virtual machine, which takes over the services for the failed primary virtual machine  308 - 1 . As part of this process, the new primary FT agent  314 - 2  selects a new secondary host to create a new secondary virtual machine (e.g., a new secondary virtual machine  308 - 3  on a new secondary host  302 - 3  in  FIG. 3 ). The new primary FT agent  314 - 2  then performs the actions described in blocks  502  and  504 , while the new secondary FT agent  314 - 3  performs the actions described in blocks  506  to  518 . 
       FIG. 6  is a flowchart of a method  600  for application  318 - 1  on primary host  302 - 1  ( FIG. 3 ), primary HA agent  312 - 1  ( FIG. 3 ), and secondary HA agent  312 - 2  ( FIG. 3 ) in examples of the present disclosure. Blocks  602  to  608  represent actions performed by application  318 - 1 , blocks  610  to  616  represent actions performed by primary HA agent  312 - 1 , and blocks  620  to  628  represent actions performed by secondary HA agent  312 - 2 . 
     In block  602 , application  318 - 1  enables monitoring of its heartbeats by primary HA agent  312 - 1 . Application  318 - 1  may enable heartbeat monitoring when it starts. Block  602  may be followed by optional block  604 . 
     In optional block  604 , application  318 - 1  sets the heartbeat interval for sending heartbeats. Blocks  602  and  604  may be implemented by application  318 - 1  sending an interrupt VMAppFt_Enable_V 1  (string appID, int HBInterval) to primary HA agent  312 - 1 , wherein appID is an unique identification of the application and HBInterval is the heartbeat interval. Optional block  604  may be followed by block  606 . 
     In block  606 , application  318 - 1  periodically sends its heartbeat to primary HA agent  312 - 1  at the heartbeat interval. Application  318 - 1  may send an interrupt VMAppFt_HB_V 1  (string appID) to HA agent  312 - 1 . Block  606  may be followed by optional block  608 . 
     In block  608 , application  318 - 1  disables its heartbeat monitoring by primary HA agent  312 - 1 . Application  318 - 1  may send an interrupt VMAppFt_Disable_V 1  (string appID) to HA agent  312 - 1 . Application  318 - 1  disables its heartbeat monitoring when it ends. Block  608  may end the actions of application  318 - 1 . 
     In block  610 , primary HA agent  312 - 1  starts monitoring the heartbeats of application  318 - 1 . Block  610  corresponds to block  602  and optional block  604 . Primary HA agent  312 - 1  also instructs secondary HA agent  312 - 2  to start monitoring the heartbeats of application  318 - 1 . Block  610  may be followed by block  612 . 
     In block  612 , primary HA agent  312 - 1  determines if it has received a heartbeat from application  318 - 1  in a specific (and configurable) time interval. If so, block  612  may be followed by block  614 . Otherwise, block  612  may be followed by block  616 . Note the time interval may be set greater than the heartbeat interval so a certain number of heartbeats may be missed. 
     In block  614 , primary HA agent  312 - 1  forwards the received heartbeat of application  318 - 1  to secondary HA agent  312 - 2 . Block  614  may loop back to block  612  to continue heartbeat monitoring. 
     In block  616 , primary HA agent  312 - 1  declares to primary FT agent  314 - 1  that primary virtual machine  308 - 1  has failed. Optionally, primary HA agent  312 - 1  notifies secondary HA agent  312 - 2  that primary virtual machine  308 - 1  has failed so the secondary HA agent can immediately initiate failover to secondary virtual machine  308 - 2 . Block  616  may end the actions of primary HA agent  312 - 1 . 
     In block  620 , secondary HA agent  312 - 2  starts monitoring the heartbeats of application  318 - 1 . Block  620  corresponds to block  610 . Block  620  may be followed by block  622 . 
     In block  622 , secondary HA agent  312 - 2  determines if it has received, via primary HA agent  312 - 1 , a heartbeat from application  318 - 1  in the specific (and configurable) time interval. If so, block  622  may be followed by block  624 . Otherwise, block  622  may be followed by block  626 . Note the time interval may be set greater than the heartbeat interval so a certain number of heartbeats may be missed. As described above, primary HA agent  312 - 1  may immediately notify secondary HA agent  312 - 2  that primary virtual machine  308 - 1  has failed so the secondary HA agent does not wait for the entire time interval before starting failover to secondary virtual machine  308 - 2 . 
     In block  624 , secondary HA agent  312 - 2  notifies secondary FT agent  314 - 2  that the buffered activities  402  are safe. Block  628  corresponds to block  512  ( FIG. 5 ). Block  624  may loop back to block  622  to continue heartbeat monitoring. 
     In block  626 , secondary HA agent  312 - 2  instructs secondary FT agent  314 - 2  ( FIG. 3 ) to discard the (faulty) buffered activities  402  ( FIG. 4 ) of primary virtual machine  308 - 1  in back buffer  404  of double buffer  406  ( FIG. 4 ). Block  626  corresponds to block  512  ( FIG. 5 ). Block  626  may be followed by block  628 . 
     In block  628 , secondary HA agent  312 - 2  declares to secondary FT agent  314 - 2  that secondary virtual machine  308 - 2  is the new primary virtual machine. Block  626  correspond to block  518  ( FIG. 5 ), which causes the new primary FT agent  314 - 2  to set the secondary virtual machine  308 - 2  as the new primary virtual machine and select a new secondary host to create a new secondary virtual machine (e.g., a new secondary virtual machine  308 - 3  on a new secondary host  302 - 3  in  FIG. 3 ). The new primary HA agent  312 - 2  then performs the actions described in blocks  610  to  616 , while the new HA agent  312 - 3  performs the actions described in blocks  620  to  628 . 
     From the foregoing, it will be appreciated that various embodiments of the present disclosure have been described herein for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the present disclosure. Accordingly, the various embodiments disclosed herein are not intended to be limiting, with the true scope and spirit being indicated by the following claims.