Patent Publication Number: US-7213246-B1

Title: Failing over a virtual machine

Description:
BACKGROUND OF THE INVENTION 
   1. Field of the Invention 
   This invention is related to the field of highly available computer systems and, more particularly, to the failing over of applications in computer systems, including clustered computer systems. 
   2. Description of the Related Art 
   Certain applications are often required to be available virtually uninterrupted, either 24 hours a day or at least during working hours. Various efforts have been undertaken to provide high availability services to support the high availability of such applications. Such highly-available applications may include email servers, web servers, database servers, etc. 
   Typically, efforts to provide high availability for a given application have focused on detecting that the application has failed and getting the application re-started. An application may fail due to an internal coding error in the application, an error in the operating system on which the application is running, an error in the hardware of the computer system on which the application is running, or a combination of any of the above errors. The errors may cause the application, or the operating system, to cease executing (e.g. a crash) or to stop functioning (e.g. a hang). 
   In some cases, each application for which high availability is desired may be assigned to a separate computer system. In this configuration, a failure of one application may not affect the operation of the other applications on the other computer systems. Additionally, this configuration allows for variations in the operating system on which the applications are run (e.g. different versions of the same operating system, or different operating systems). The cost of obtaining and maintaining separate computer systems for each application may be considerable. 
   Another method is to cluster a group of computer systems using specialized software (referred to as a cluster server) to control the group of computer systems. A given application may be executed on a first computer system of the group. The cluster server monitors the operation of the application and, if the cluster server detects that the application has failed, the cluster server may close the application on the first computer system and restart the application on another computer system. Typically, such cluster servers involve identifying, for each application supported by the cluster server, all of the state in the computer system that is needed to restart the application. In practice, such identification may be problematic and frequently involves making use of undocumented features of the application. Additionally, some applications may not function correctly when restarted on another machine. For example, the Exchange2000 application from Microsoft Corporation may not access a mailbox database used when the application was executing on another machine because Microsoft&#39;s Active Directory may identify that other machine as the owner of the database. 
   SUMMARY OF THE INVENTION 
   A computer system, carrier medium, and method for failing over a virtual machine are provided. A first computer system may be configured to execute a first application in a first virtual machine. A second computer system may be coupled to the first computer system. In response to a failure, the first computer system is configured to failover the first virtual machine to the second computer system. In one embodiment, failing over the first virtual machine may be accomplished using instructions comprising a program carried on a carrier medium. In one specific implementation, the program may include instructions which cause an image of a first virtual machine in which a first application is executing to be stored to a storage accessible to two or more computer systems in response to a failure of the first application on a first computer system of the two or more computer systems. Additionally, the instructions may activate a second virtual machine on the first computer system from the storage in response to receiving a message to failover the second virtual machine from another one of the computer systems. 
   In one particular implementation, cluster server software may be executing on the computer systems. The first virtual machine may be defined as a resource monitored by the cluster server software. The first virtual machine resource may be failed over in response to detection of a failure. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The following detailed description makes reference to the accompanying drawings, which are now briefly described. 
       FIG. 1  is a block diagram of one embodiment of a cluster of computer systems. 
       FIG. 2  is a block diagram of another embodiment of a cluster of computer systems. 
       FIG. 3  is a state machine diagram illustrating operation of one embodiment of certain components shown in  FIG. 2 . 
       FIGS. 4–10  illustrate one embodiment of the cluster shown in  FIG. 2  at various points during a failover of an application. 
       FIG. 11  is a block diagram of a carrier medium. 
   

   While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims. 
   DETAILED DESCRIPTION OF EMBODIMENTS 
   Turning now to  FIG. 1 , a block diagram is shown illustrating one embodiment of a cluster of computer systems. Other embodiments are possible and contemplated. In the embodiment of  FIG. 1 , the cluster includes computer systems  10 A– 10 N, a network  12 , and a shared storage device  14 . The computer systems  10 A– 10 N are each coupled to the network  12 , and to the shared storage device  14 . Each of the computer systems  10 A– 10 N includes one or more virtual machines (e.g. virtual machines  16 A– 16 B on the computer system  10 A, virtual machines  16 C– 16 D on the computer system  10 B, and virtual machine  16 E on the computer system  10 N). The virtual machines are controlled by a virtual machine (VM) kernel (e.g. kernels  18 A,  18 B, and  18 N in the computer systems  10 A,  10 B, and  10 N, respectively). The virtual machines  16 A– 16 E and the VM kernels  18 A– 18 N may comprise software and/or data structures. The software may be executed on the underlying hardware in the computer systems  10 A– 10 N (e.g. the hardware  20 A,  20 B, and  20 N in the computer systems  10 A,  10 B, and  10 N). The hardware may include any desired circuitry. For example, the hardware  20 A is shown to include a processor, or central processing unit (CPU)  22 , a storage  24 , and input/output (I/O) circuitry  26 . In some embodiments, a computer system may include multiple CPUs  22 . Similarly, a virtual machine may comprise multiple virtual CPUs, in some embodiments. 
   As shown in  FIG. 1 , each application executing on the computer systems  10 A– 10 N executes within a virtual machine  16 A– 16 E. Generally, a virtual machine comprises any combination of software, one or more data structures in memory, and/or one or more files stored on the shared storage device  14 . The virtual machine mimics the hardware used during execution of a given application. For example, in the virtual machine  16 A, an application  28  is shown. The application  28  is designed to execute within the operating system (O/S)  30 . Both the application  28  and the O/S  30  are coded with instructions executed by the virtual CPU  32 . Additionally, the application  28  and/or the O/S  30  may make use of various virtual storage and virtual I/O devices  34 . The virtual storage may represent any type of storage, such as memory, disk storage, tape storage, etc. The disk storage may be any type of disk (e.g. fixed disk, removable disk, compact disc read-only memory (CD-ROM), rewriteable or read/write CD, digital versatile disk (DVD) ROM, etc.). Each disk storage in the virtual machine may be mapped to a file on the shared storage device  14 , to a file on any storage device accessible to the hardware  20 A, or directly to a storage device accessible to the hardware  20 A. A storage device may be accessible to the hardware  20 A if the storage device is connected to the hardware  20 A or if the storage device is coupled to a network to which the hardware  20 A is coupled. For example, storage devices attached to a network may include network attached storage (NAS) or storage area network (SAN) devices, or IP-based storage devices of any type may be used. The virtual I/O devices may include any type of I/O devices, including modems, audio devices, video devices, network interface cards (NICs), universal serial bus (USB) ports, firewire (IEEE 1394) ports, serial ports, parallel ports, etc. Generally, each virtual I/O device may be mapped to a corresponding I/O device in the underlying hardware (e.g. the I/O circuitry  26 ) or may be emulated in software if no corresponding I/O device is included in the I/O circuitry  26 . 
   The virtual machine in which an application is executing encompasses the entire system state associated with an application. Generally, when a virtual machine is active (i.e. the application within the virtual machine is executing), the virtual machine may be stored in the memory of the computer system on which the virtual machine is executing (although the VM kernel may support a paging system in which various pages of the memory storing the virtual machine may be paged out to local storage in the computer system) and in the files on the shared storage device  14  which are mapped to the virtual storage devices in the virtual machine. The VM kernel may support a command to suspend the virtual machine. In response to the command, the VM kernel may write an image of the virtual machine to the shared storage device  14 , thus capturing the current state of the executing application. The image may include one or more files written in response to the suspend command, capturing the state of the virtual machine that was in memory in the computer system, as well as the files stored on the shared storage device  14  that represent the virtual disks included in the virtual machine. The state may include not only files written by the application, but uncommitted changes to files which may still be in the memory within the virtual machine, the state of the hardware (including the processor  32 , the memory in the virtual machine, etc.) within the virtual machine, etc. Thus, the image may be a snapshot of the state of the executing application. 
   A suspended virtual machine may be resumed using a resume command supported by the VM kernel. In response to the resume command, the VM kernel may read the image of the suspended virtual machine from disk and may activate the virtual machine in the computer system. 
   An application may be failed over in the cluster of computer systems  10 A– 10 N. If an application is detected as failing, the suspend command may be used to suspend the virtual machine in which the application is executing. Thus, the image of the virtual machine may be written to the shared storage device  14 . Another computer system  10 A– 10 N in the cluster may be selected to execute the application. The resume command may be used in the selected computer system  10 A– 10 N to resume the virtual machine from the image on the shared storage device  14 . In this manner, the application may begin executing on the selected computer system  10 A– 10 N after experiencing a failure on the previous computer system  10 A– 10 N. In this context, an application may fail due to an internal coding error in the application, an error in the operating system on which the application is running, an error in the virtual machine or the VM kernel on which the virtual machine is executing, an error in the hardware of the computer system on which the application is running, or a combination of any of the above errors. The errors may cause the application, or the operating system, to cease executing (e.g. a crash) or to stop functioning (e.g. a hang). It is noted that, in some embodiments, the computer systems  10 A– 10 N may be configured to periodically checkpoint the virtual machines executing thereon, thus providing a set of possible images from which to resume when a failover occurs. 
   Since the state of the virtual machine is failed over from one computer system to another, the process of identifying which machine state is to be failed over with a given application may be avoided. Additionally, applications which may be dependent on the particular computer system on which they are executing (e.g. applications using Microsoft&#39;s Active Directory) may be failed over. Since the virtual machine in which the application executes is the same on any computer system on which it executes, the application appears to be executing on the same computer system. This appearance is independent of which actual computer system  10 A– 10 N is executing the virtual machine including the application. 
   In the illustrated embodiment, cluster server software is used to manage the cluster and to failover applications (by failing over virtual machines). The cluster server software may be installed on each computer system  10 A– 10 N in the cluster. For example, the cluster server software  36 A may be installed on the computer system  10 A, the cluster server software  36 B may be installed on the computer system  10 B, and the cluster server software  36 N may be installed on the computer system  10 N. The cluster server software on each computer system  10 A– 10 N communicates with the other cluster server software on the other machines to manage the cluster, select computer systems to execute applications, monitor operation of the applications and underlying hardware to detect failures, and failover applications when a failure is detected. In one embodiment, the cluster server software also executes in a virtual machine. For example, in the embodiment of  FIG. 1 , the cluster server  36 A executes in the virtual machine  16 B, the cluster server  36 B executes in the virtual machine  16 C, and the cluster server  36 N executes in the virtual machine  16 E. In one implementation, the cluster server may operate similar to the Veritas Cluster Server™ product available from Veritas Software Corporation (Mountain View, Calif.). 
   The cluster server software may define various resources that it monitors and fails over a resource (or a group of dependent resources) if one of the resources are detected as failing. Resources may include software (e.g. applications, operating system software components, operating system services, etc.) and hardware (e.g. disk drives, network interface circuitry, etc.). In such embodiments, the virtual machine may be considered as a resource which may be failed over by the cluster server software. The number of resources associated with a given application may be reduced, since the virtual machine encompasses many software and hardware components that may have otherwise been tracked as separate resources by the cluster server software. The failing over of applications within virtual machines may be simpler than tracking the various resources separately. 
   The virtual hardware in the virtual machine  16 A (and other virtual machines such as virtual machines  16 B– 16 E) may be similar to the hardware  20 A included in the computer system  10 A. For example, the virtual CPU  32  may implement the same instruction set architecture as the processor  22 . In such cases, the virtual CPU  32  may be one or more data structures storing the processor state for the virtual machine  16 A. The application and O/S software instructions may execute on the CPU  22  when the virtual machine  16 A is scheduled for execution by the VM kernel  18 A. When the VM kernel  18 A schedules another virtual machine for execution (e.g. the virtual machine  16 B), the VM kernel  18 A may write the state of the processor into the virtual CPU  32  data structure. Alternatively, the virtual CPU  32  may be different from the CPU  22 . For example, the virtual CPU  32  may comprise software coded using instructions from the instruction set supported by the CPU  22  to emulate instruction execution according to the instruction set architecture of the virtual CPU  32 . Alternatively, the VM kernel  18 A may emulate the operation of the hardware in the virtual machine. Similarly, other virtual hardware may be emulated in software if not included in the hardware  20 A. 
   Different virtual machines which execute on the same computer system  10 A may differ. For example, the O/S  30  included in each virtual machine may differ. Different virtual machines may employ different versions of the same O/S (e.g. Microsoft Windows NT with different service packs installed), different versions of the same O/S family (e.g. Microsoft Windows NT and Microsoft Windows2000), or different O/Ss (e.g. Microsoft Windows NT, Linux, Sun Solaris, etc.). 
   Generally, the VM kernel may be responsible for managing the virtual machines on a given computer system. The VM kernel may schedule virtual machines for execution on the underlying hardware, using any scheduling scheme. For example, a time division multiplexed scheme may be used to assign time slots to each virtual machine. Additionally, the VM kernel may handle the suspending and resuming of virtual machines responsive to suspend and resume commands. The commands may be received from a virtual machine (e.g. the virtual machine in which the cluster server is executing). In one embodiment, the VM kernel may be the ESX product available from VMWare, Inc. (Palo Alto, Calif.). 
   In the illustrated embodiment, the VM kernel may execute directly on the underlying hardware (i.e. without an underlying operating system). In other embodiments, the VM kernel may be designed to execute within an operating system. For example, the GSX product available from VMWare, Inc. may execute under various versions of Microsoft&#39;s Windows operating system and/or the Linux operating systems. 
   The network  12  may comprise any network technology in various embodiments. The network  12  may be a local area network, wide area network, intranet network, Internet network, or any other type of network. The network  12  may be designed to be continuously available (although network outages may occur), or may be intermittent (e.g. a modem connection made between a computer system in a user&#39;s home and a computer system in a user&#39;s workplace). Any network protocol may be used. For example, the network  12  may be an Ethernet network. Alternatively, the network may be a token ring network, etc. Additionally, the network  12  may comprise shared storage or shared memory for communicating between the computer systems, in some embodiments. 
   The shared storage device  14  may be any type of storage device to which the computer systems  10 A– 10 N may be coupled. For example, the shared storage device  14  may comprise one or more small computer system interface (SCSI) drives attached to a SCSI bus shared by the computer systems  10 A– 10 N. Alternatively, the shared storage device  14  may couple to the network  12  (e.g. network attached storage (NAS) or storage area network (SAN) technologies may be used). The shared storage device may also comprise memory. Generally, the shared storage device  14  may be any device capable of storing data. 
   Turning now to  FIG. 2 , a block diagram of a second embodiment of a cluster of computer systems is shown. Other embodiments are possible and contemplated. In the embodiment of  FIG. 2 , the cluster includes computer systems  10 A and  10 B. Other computer systems may be included, as desired, in the cluster. The cluster shown in  FIG. 2  will be used as an example to highlight the failover of an application. As used herein, a failover refers to the starting of application execution on a different computer system in response to a failure of the application on a given computer system. In the illustrated embodiment, the computer system  10 A includes the virtual machines  16 A and  16 B, the VM kernel  18 A, and the hardware  20 A, similar to the embodiment of  FIG. 1 . The virtual machine  16 A includes the application  28 , the O/S  30 , and storage  34 A (part of the storage and I/O  34 ). The storage  34 A is mapped, by the VM kernel  18 A, to the shared storage device  14 . Additionally, the virtual machine  16 A includes a proxy agent  40  configured to communicate with the cluster server  36 A in the virtual machine  16 B. The cluster server  36 A may be designed to execute within an O/S  30 B. Additionally, the virtual machine  16 B may include a pair of virtual NICs  42 A– 42 B. In the illustrated embodiment, the hardware  20 A includes NICs  44 A– 44 E. The virtual machine  16 B is configured such that each of the virtual NICs  42 A– 42 B are mapped by the VM kernel  18 A to one of the NICs  44 A– 44 E (e.g. the NICs  44 D– 44 E, respectively, in this embodiment). The computer system  10 B includes the virtual machine  16 C, the virtual machine  16 A with the same components as the computer system  10 A includes, the VM kernel  18 B, and the hardware  20 B. The hardware  20 B includes NICs  44 F– 44 J. The virtual machine  16 C includes the cluster server  36 B, an O/S  30 C, and virtual NICs  42 C– 42 D which are mapped to two of the NICs  44 F– 44 J (e.g. the NICs  44 I and  44 J, in this embodiment). The NICs  44 E and  44 J are coupled to communicate, as are the NICs  44 D and  44 I and the NICS  44 C and  44 H. 
   The virtual machine  16 A may be active at any given time in only one of the computer systems  10 A– 10 B. Since the image of the virtual machine  16 A is stored on the shared storage device  14  in response to a suspend command, the virtual machine  16 A may be executed on either computer system  10 A– 10 B. If, for example, the virtual machine  16 A is executing on the computer system  10 A and a failure is detected, the virtual machine  16 A may be failed over to the computer system  10 B. Likewise, if the virtual machine  16 A is executing on the computer system  10 B and a failure is detected, the virtual machine  16 A may be failed over to the computer system  10 A. As mentioned previously, in some embodiments, multiple suspended images of the virtual machine may be maintained on the shared storage  14 , and one of the checkpoint images may be selected for resuming the virtual machine in a failover. Additionally, in some embodiments, a failover may occur to the same computer system  10 A– 10 B on which the failure occurred, if desired. 
   While the embodiment shown in  FIG. 2  includes two computer systems, other embodiments may include more than two computer systems. If more than two computer systems are included, a computer system may be selected to be the receiver of a given failover. Various selection criteria may be used. For example, the cluster server may monitor the load on each computer system and select the computer system with the lowest current load for the failover. Alternatively, a fixed priority scheme or a round-robin scheme may be used. 
   The proxy agent  40  may be used to monitor the application  28 , to detect any errors that may occur in the execution of the application. The proxy agent  40  may monitor the processes comprising the application  28  to see that the processes are active and consuming/releasing system resources properly. The proxy agent  40  may be configured to periodically make requests to the application  28  to determine if the application  28  responds properly. The proxy agent  40  may be configured to connect to defined ports of the application  28  to request status information on the application  28 . In embodiments running on Microsoft&#39;s Windows operating system as the O/S  30 , the proxy agent  40  may replicate the registry entries corresponding to the application  28  to the shared storage device  14 . Alternatively, the registry entries may be copied from the virtual storage device of the failed virtual machine after the virtual machine is failed over. 
   The cluster server  36 A may be configured to monitor the virtual machine  16 A to detect any failures in the virtual machine as a whole. Particularly, in one embodiment, the cluster server  36 A may ensure that the virtual machine  16 A is active and may ping the IP address of the virtual machine  16 A (e.g. the IP address of a virtual NIC in the virtual machine  16 A). If the ping is successful, the cluster server  36 A may connect to the proxy agent  40  to collect status information regarding the application  28 . In some embodiments, the cluster server  36 A may also be configured to reserve access to the shared storage device  14  when such access is desired. 
   While the illustrated embodiment includes the proxy agent  40  running within the virtual machine  16 A, other embodiments may not include the proxy agent  40 . For example, an application may be designed to communicate status information to the cluster server  36 A directly. 
   The cluster server  36 A and the cluster server  36 B communicate using the virtual NICs  42 A– 42 D. Two NICs may be used for each cluster server to provide redundancy, to allow the cluster servers to distinguish between a network error between the computer systems and an error within one of the computer systems. Other embodiments may use one NIC for each cluster server if redundancy is not desired. The communication between the cluster servers  36 A– 36 B may include a “heartbeat” communication indicating that the sending computer system  10 A– 10 B is still running. The heartbeat may be sent and checked at predetermined intervals. Alternatively, heartbeat information may be stored by each cluster server  36 A– 36 B on a shared storage device or in shared memory, and the heartbeat information may be accessed from the shared storage device or memory as desired. Any mechanism for communicating heartbeat information among cluster servers  36 A– 36 B may be used. Additionally, the communication may include status indicating the state of the system. A failover message may also be communicated, requesting the failover of a virtual machine in which an application experiences a failure. 
   The cluster server software in each computer system may monitor for the heartbeat communications from each other computer system. If the heartbeat communications from a given computer system cease, the cluster server software may conclude that the given computer system has failed. The cluster server software may attempt to suspend the virtual machines executing on the given computer system remotely. If successful, the suspended virtual machines may be resumed on another computer system. If not successful, the virtual machines on the given computer system may be restarted on other computer systems from a clean boot (or from one of the checkpoint images, in embodiments in which multiple images are maintained for a virtual machine on the shared storage device  14 ). While some state may be lost in such a case, the applications which experienced failure on the given computer system may be available for use. 
   The NICs  44 C and  44 H may be used for other communications between the computer systems  10 A– 10 B. The remaining NICS  44 A– 44 B and  44 F– 44 G may be provided for use by the applications executing on the virtual machines within a given computer system  10 A– 10 B. For example, an email server like Microsoft&#39;s Exchange2000 may use one or more NICs to receive requests to check email accounts. 
   Turning next to  FIG. 3 , a state machine diagram is shown illustrating various states of one embodiment of the software illustrated in  FIG. 2 . Other embodiments are possible and contemplated. Thick lines (e.g. reference numeral  50 ) represent transitions between states. Thin lines (e.g. reference numeral  52 ) represent communication between software. Dashed lines (e.g. reference numeral  54 ) indicate state changes due to an external event or a state change in the source of the dashed line. The state machine will be described below with reference to the virtual machine  16 A, the proxy agent  40 , the application  28 , and the cluster server  36 A on the computer system  10 A. Similar state machines may be used in other computer systems in the cluster. The virtual machine  16 A has two states in the illustrated diagram: a powered on state  56  and a powered off state  58 . The proxy agent  40  has three states in the illustrated diagram: a monitor state  60 , a shutdown state  62 , and a tell-CS state  64 . The application  28  has three states in the illustrated diagram: a running state  66 , a failed state  68 , and a closed state  70 . The cluster server  36 A has four states in the illustrated diagram: an online state  72 , a cleared state  74 , a faulted state  76 , and an offline state  78 . 
   During normal operation, the virtual machine  16 A is in the powered on state  56 , the proxy agent  40  is in the monitor state  60 , and the application  28  is in the running state  66 . Generally, when the virtual machine  16 A transitions to the powered on state, the proxy agent  40  is initialized in the monitor state  60  and the application is started (the running state  66 ). In the monitor state  60 , the proxy agent monitors the application  28  to detect failures. 
   The application  28  transitions from the running state  66  to the failed state  68  in response to the occurrence of a failure. In the failed state  68 , the application  28  is no longer running correctly. The application may be hung, may have crashed, or may have one or more services that have ceased operating. The proxy agent  40  detects the failure in the application  28 , and transitions to the shutdown state  62 . 
   In the shutdown state  62 , the proxy agent  40  transmits a close message to the application  28 . The application  28  transitions to the closed state  70  in response to the close message. Additionally, the proxy agent  40  transitions to the tell-CS state  64 . The proxy agent remains in the tell-CS state  64  until contacted by the cluster server  36 A (which is in the online state  72 ). The proxy agent  40  transmits a status of faulted to the cluster server  36 A. The status may optionally include information identifying the fault, if desired. 
   In response to the faulted status from the proxy agent  40 , the cluster server  36 A transmits a power off message to the virtual machine  16 A, causing the virtual machine  16 A to transition from the powered on state  56  to the powered off state  58 . The virtual machine  16 A may be suspended (writing the image of the virtual machine  16 A to the shared storage device  14 ) to allow failover of the virtual machine  16 A to another computer system. Additionally, the cluster server  36 A transitions to the cleared state  74 . 
   In the cleared state  74 , the cluster server  36 A transmits a failover message to the cluster server in the computer system selected to execute the failed application (the “new node” in  FIG. 3 ). The failover message may include the name of the virtual machine to be failed over, which may indicate the file or files on the shared storage device  14  that are used to establish the virtual machine in the new node. The cluster server  36 A then transitions to the faulted state  76 . While the cluster server  36 A is in the faulted state  76 , the state of the computer system  10 A may remain frozen. New applications may not be started on the computer system  10 A through the cluster server  36 A. If the computer system  10 A itself is the source of the failure, preventing new applications from running on the computer system  10 A may prevent additional failures until the failure can be corrected. Additionally, by freezing the state of the computer system  10 A, the state may be analyzed to determine the failure and if any correcting measures may be indicated. 
   The cluster server  36 A transitions from the faulted state  76  to the offline state  78  in response to an external clear message. The clear message may be transmitted by a user, for example, after examining the computer system  10 A to determine the cause of the failure. Alternatively, the clear message may be transmitted by the user without first examining the computer system to bring the computer system  10 A back into eligibility for executing applications within the cluster. 
   In the offline state  78 , the cluster server  36 A may receive a failover message from another computer system in which a failure of an application has occurred (the “old node” in  FIG. 3 ). In response to the failover message, the cluster server  36 A transitions to the online state  72 . From the online state  72 , the cluster server transmits a power on message to the virtual machine  16 A to cause the virtual machine  16 A to be resumed. The power on may include transmitting the resume command to the VM kernel  18 A. The virtual machine  16 A powers up and is loaded with the state from the image of the virtual machine  16 A on the shared storage device  14  (or from a selected image, if multiple checkpoint images are maintained). The proxy agent  40  is initialized to the monitor state  60 . The application  28  is started (the running state  66 ). 
   It is noted that, while a state machine is shown in  FIG. 3  with various states for various software included in the computer system  10 A, the software may not formally be divided into states. Instead, the software may have various code sequences, delimited by branches which test for the causes of transitions shown in  FIG. 3 , and may progress to the next code sequence based on the results of the branches. Alternatively, each state may be a different routine with the code for executing that state and determining state transitions. Still further, the states of the application  28  may not describe different sections of the application  28 , but rather its state of execution. 
   Turning now to  FIGS. 4–10 , a failover of an application from the computer system  10 A to the computer system  10 B is shown at various points in time during the failover. For each piece of software, the name is listed followed by its state (for the embodiment of  FIG. 3 ) in parentheses. 
   In  FIG. 4 , the computer system  10 A is executing the virtual machine  16 A including the application  28  and the proxy agent  40 . The application  28  is in the running state  66 , the proxy agent  40  is in the monitor state  60 , and the virtual machine  16 A is in the powered on state  56 . The virtual machines  16 B and  16 C are in the powered on state as well. The cluster server  36 A, executing within the virtual machine  16 B, is in the online state  72 . The cluster server  36 B, executing within the virtual machine  16 C, is in the offline state  78 . 
   As illustrated in  FIG. 5 , the application  28  fails on the computer system  10 A. Thus, the application  28  is in the failed state  68 , and the proxy agent  40  detects the failure. In response to the failure, the proxy agent  40  transitions to the shutdown state  62  ( FIG. 6 ). The proxy agent  40  transmits a close message to the application  28 . As shown in  FIG. 7 , the application  28  transitions to the closed state  70  in response to the close message and the proxy agent  40  transitions to the tell-CS state  64 . The cluster server  36 A receives the faulted status from the proxy agent  40 , and transmits a power off message to the virtual machine  16 A in response. 
     FIG. 8  illustrates the virtual machine  16 A powered off, and an image  80  of the virtual machine  16 A stored on the shared storage device  14 . The cluster server  36 A has transitioned to the cleared state, and transmits a failover message to the cluster server  36 B. The cluster server  36 A then transitions to the faulted state  76  ( FIG. 9 ), and the cluster server  36 B transitions to the online state  72 . The cluster server  36 B transmits a power on message to the virtual machine  16 A (which is shown in a powered off state in  FIG. 9 , but may generally not exist in the computer system  10 B yet). As part of powering on the virtual machine  16 A on the computer system  10 B, the computer system  10 B reads the image  80  of the virtual machine  16 A from the shared storage device  14  ( FIG. 10 ) and resumes the virtual machine  16 A. As mentioned previously, the image  80  may be selected from one of several images of the virtual machine  16 A that may be stored on the shared storage device  14 , in some embodiments. The virtual machine  16 A activates on the computer system  10 B, with the application  28  in the running state  66  and the proxy agent  40  in the monitor state  60 . 
   It is noted that, while the above description has referred to a cluster of computer systems managed by cluster server software, other embodiments may failover virtual machines containing applications from one computer system to another without configuring the computer systems into clusters. For example, software may be executed on a first computer system including a first virtual machine in which a first application executes. The software may monitor the first application and the first virtual machine to detect failures, and may cause the failover to another computer system. Alternatively, the software may executed on a second computer system separate from the first computer system. Still further, the failover may be a manual process performed by a user. 
   It is further noted that, while the above examples have shown one application executing in each virtual machine, a given virtual machine may include one or more applications, as desired. 
   Turning next to  FIG. 11 , a block diagram of a carrier medium  300  including one or pieces of software described above is shown. Generally speaking, a carrier medium may include storage media such as magnetic or optical media, e.g., disk or CD-ROM, volatile or non-volatile memory media such as RAM (e.g. SDRAM, RDRAM, SRAM, etc.), ROM, etc., as well as transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link. 
   As illustrated in  FIG. 11 , the carrier medium  300  may store one or more of the cluster server  36 A, the VM kernel  18 A, or the proxy agent  40 . In other embodiments, the application  28  may also be stored, as may be the O/S  30  and any other software. Generally, as used herein, software comprises one or more instructions which, when executed, performs the function described for the software. The instructions may be machine level instructions from an instruction set implemented in the virtual CPU  32  or the CPU  22 , or may be higher level instructions (e.g. shell scripts, interpretive languages, etc.). 
   Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.