Patent Publication Number: US-10310754-B1

Title: Method and system for providing storage checkpointing to a group of independent computer applications

Description:
NOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION 
     A portion of the material in this patent document is subject to copyright protection under the copyright laws of the United States and of other countries. The owner of the copyright rights has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the United States Patent and Trademark Office publicly available file or records, but otherwise reserves all copyright rights whatsoever. The copyright owner does not hereby waive any of its rights to have this patent document maintained in secrecy, including without limitation its rights pursuant to 37 C.F.R. § 1.14. 
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     This invention pertains generally to enterprise computer systems, computer networks, embedded computer systems, wireless devices such as cell phones, computer systems, and more particularly to methods, systems and procedures (i.e., programming) for providing high-availability, virtualization and checkpointing services for a group of computer applications. 
     2. Description of Related Art 
     Enterprise and wireless systems operating today are subject to continuous program execution that is 24 hours a day and 7 days a week. There is no longer the concept of “overnight” or “planned downtime.” All programs and data must be available at any point during the day and night. Any outages or deteriorated service can result in loss of revenue as customers simply take their business elsewhere, and the enterprise stops to function on a global scale. Traditionally, achieving extremely high degrees of availability has been accomplished with customized applications running on custom hardware, all of which is expensive and proprietary. Furthermore, application services being utilized today are no longer run as single applications or processes; instead, they are built from a collection of individual programs jointly providing the service. Traditionally, no mechanisms have existed for protecting such multi-application services. This problem is compounded by the fact that the individual applications comprising the service are typically provided by different vendors and may get loaded at different times. Furthermore, distributed storage systems contain much of the applications data and may need to be included. 
     Storage checkpointing operating at the block level of the storage subsystem are well known in the art and widely deployed. Commercial products are available from Symantec/Veritas in the form of “Veritas Storage Foundation.” Similar technologies are available from StorageTek under the Sun Microsystems brand. All of those technologies operate at the level of the storage device. If the storage device gets restored to an earlier checkpoint, all applications on that disk are affected; including applications unrelated to the restore event. The present invention breaks this fundamental constraint, and only checkpoints storage related to individual applications. This means that one application can do a storage checkpoint restore without affecting any other applications on the server. 
     Two references provide a background for understanding aspects of the current invention. The first reference is U.S. patent application Ser. No. 11/213,678 filed on Aug. 26, 2005, incorporated above in its entirety, which describes how to provide transparent and automatic high availability for applications where all the application processes run on one node. The second reference is U.S. Pat. No. 7,293,200 filed on Aug. 26, 2005 which describes how to transparently provide checkpointing of multi-process applications, where all processes are running on the same node and are launched from one binary. The present invention is related to applications comprised of one or more independent applications, where the independent applications dynamically join and leave the application group over time and where the applications may operate off of files located either locally or on the network. 
     BRIEF SUMMARY OF THE INVENTION 
     A method, system, apparatus and/or computer program are described for achieving checkpointing, restoration, virtualization and loss-less migration of application groups including their associated storage The invention provides transparent migration and fail-over of application groups while ensuring that connected clients remain unaware of the migration. The client&#39;s connection and session are transparently transferred from the primary to the backup server without any client involvement. 
     One aspect of the present invention relates to a system for storage checkpointing to a group of independent computer applications. The system has a storage disk that stores files; a storage access interface to access the storage disk; and a server. The server runs the group of independent computer applications and utilizes the files stored on the storage disk. A file system on the server accesses the files stored on the storage disk. An operating system and at least one device driver can be called by the file system, and at least one buffer buffers first data written to the storage disk, and second data read from the storage disk. 
     Another aspect of the present invention relates to a computer readable medium comprising instructions for storage checkpointing to a group of independent computer applications. The instructions are for storing files on a storage disk; accessing the storage disk via a storage access interface; running the group of independent computer applications on a server, wherein the group of independent computer applications utilizes the files stored on the storage disk; accessing the files stored on the storage disk via a file system on the server; calling an operating system via the file system; calling at least one device driver via the file system; and buffering first data written to the storage disk, and second data read from the storage disk in at least one buffer. 
     Yet another aspect of the present invention relates to a method for storage checkpointing to a group of independent computer applications. The method includes storing files on a storage disk; accessing the storage disk via a storage access interface; 
     running the group of independent computer applications on a server, wherein the group of independent computer applications utilizes the files stored on the storage disk; 
     accessing the files stored on the storage disk via a file system on the server; calling an operating system via the file system; calling at least one device driver via the file system; and buffering first data written to the storage disk, and second data read from the storage disk in at least one buffer. 
     The term “checkpointing” and “checkpointing service” is utilized herein interchangeably to designate a set of services which capture the entire state of an application group and stores all or some of the application group state locally or remotely. The checkpointing services run (execute) on all nodes where one or more of the application group&#39;s applications run (execute) or can fail over to. 
     The term “node” is utilized herein to designate one or more processors running a single instance of an operating system. A virtual machine, such as VMWare or XEN VM instance, is also considered a “node.” Using VM technology, it is possible to have multiple nodes on one physical server. 
     The term “application group” is utilized herein to describe a set of independent applications that jointly provide a service. The term “independent” is utilized herein to mean that the applications need no prior knowledge of each other. An application group is simply a logical grouping of one or more applications that together or independently provide some service. The independent applications do not need to be running at the same time. A member of the application group can also load, perform work and exit, essentially joining and leaving the group. 
     The terms “application” and “independent application” are utilized interchangeably to designate each of the applications in an application group. Each independent application can consist of one or more processes and be single threaded or multi-threaded. Operating systems generally launch an application by creating the application&#39;s initial process and letting that initial process run/execute. In the following teachings we often identify the application at launch time with that initial process and then describe how to handle creation of new processes via fork and/or exec. 
     In the following we use commonly known terms including but not limited to “process,” “process ID (PID),” “thread,” “thread ID (TID),” “files,” “disk,” “CPU,” “storage,” “memory,” “address space,” “semaphore,” “System V, SysV,” “Windows,” “Microsoft Windows” and “signal.” These terms are well known in the art and thus will not be described in detail herein. 
     The term “coordinator” is utilized for designating a special control process running as an element of the invention. The coordinator is generally responsible for sending out coordination events, managing application group registration and coordinating activities across all applications in an application group. For the sake of simplicity, the coordinator is often depicted as running on the same node as the application-group, however this is not a requirement as the coordinator can run on any node. 
     The term “transport” is utilized to designate the connection, mechanism and/or protocols used for communicating across the distributed application. Examples of transport include TCP/IP, Message Passing Interface (MPI), Myrinet, Fibre Channel, ATM, shared memory, DMA, RDMA, system buses, and custom backplanes. In the following, the term “transport driver” is utilized to designate the implementation of the transport. By way of example, the transport driver for TCP/IP would be the local TCP/IP stack running on the host. 
     The term “fork( )” is used to designate the operating system mechanism used to create a new running process. On Linux, Solaris, and other UNIX variants, a family of fork( ) calls is provided. On Windows, one of the equivalent calls is “CreateProcess( )”. Throughout the rest of this document we use the term “fork” to designate the functionality across all operating systems, not just on Linux/Unix. In general fork( ) makes a copy of the process making the fork( ) call. This means that the newly created process has a copy of the entire address space, including all variables, I/O etc of the parent process. 
     The term “exec( )” is used to designate the operating system mechanism used to overlay a new image on top of an already existing process. On Linux, Solaris, and other UNIX a family of exec( ) calls is provided. On Windows, the equivalent functionality is provided by e.g. “CreateProcess( )” via parameters. Throughout the rest of this document we use the term “exec” to designate the functionality across all operating systems, not just Linux/Unix. In general, exec( ) overwrites the entire address space of the process calling exec( ). A new process is not created and data, heap and stacks of the calling process are replaced by those of the new process. A few elements are preserved, including but not limited to process-ID, UID, open file descriptors and user-limits. 
     The term “shell script” and “shell” is used to designate the operating system mechanism to run a series of commands and applications. On Linux, Solaris, and other Unix variants, a common shell is called ‘bash’. On Windows equivalent functionality is provided by “cmd.exe” and .bat files or Windows PowerShell. Examples of cross-platform scripting technologies include JavaScript, Perl, Python, and PHP. Throughout the rest of this document we use the term “shell” and “shell script” to designate the functionality across all operating systems and languages, not just Linux/Unix. 
     The term “interception” is used to designate the mechanism by which an application re-directs a system call or library call to a new implementation. On Linux and other UNIX variants interception is generally achieved by a combination of LD_PRELOAD, wrapper functions, identically named functions resolved earlier in the load process, and changes to the kernel sys call table. On Windows, interception can be achieved by modifying a process&#39; Import Address Table and creating Trampoline functions, as documented by “Detours: Binary Interception of Win32 Functions” by Galen Hunt and Doug Brubacher, Microsoft Research July 1999.” Throughout the rest of this document we use the term to designate the functionality across all operating systems. 
     The term “Barrier” and “Barrier Synchronization” is used herein to designate a type of synchronization method. A Barrier for a group of processes and threads is a point in the execution where all threads and processes must stop at before being allowed to proceed. Barriers are typically implemented using semaphores, mutexes, Locks, Event Objects, or other equivalent system functionality. Barriers are well known in the art and will not be described further here. 
     In the following descriptions, the product name “Duration” is utilized in referring to a system as described in the first and second references cited previously. It should be appreciated, however, that the teachings herein are applicable to other similarly configured systems. 
     By way of example, consider an e-Commerce service consisting of a WebLogic AppServer and an Oracle Database. In this case WebLogic and Oracle would be the independent applications, and the application group would consist of WebLogic and the Oracle database. 
     By way of example, consider a cell phone with an address book and built-in navigation system. In this case the address book and the navigation system would be the independent applications, and the application group would consist of the address book and the navigation application. 
     By way of example, consider a shell-script running a series of applications and other scripts. In this case the script and all applications and scripts launched by the script comprise the application group, and all the individual applications and other scripts called within the script are the independent applications. 
     The two references cited above cover the cases where the multi-process applications are created starting with one binary. As described in U.S. Pat. No. 7,293,200 this is generally accomplished by the application using a series of fork( ) calls to create new sub-processes. The present invention broadens the checkpointing services to cover all types of multi process applications, including those that exec( ). 
     In at least one embodiment, a method of checkpointing single process application groups and multi-process application groups is provided. The method may include creating at least one full checkpoint for each application process in an application group, and may include creating at least one incremental checkpoint for each application process in the application group. Further, the method may automatically merge each of the at least one available incremental application checkpoint against a corresponding full application checkpoint, and synchronize checkpointing across all applications in the application group. Each application may use both fork( ) and exec( ) in any combination. 
     In at least one embodiment a special mechanism is provided to handle exec-only calls. With exec essentially overwriting the entire address space of the calling process, all registration and checkpointing information is lost. Special care needs to be taken to preserve this information across the exec call. One example embodiment of the present invention provides a mechanism to preserve such information using a combination of shared memory and environment variables. 
     In at least one embodiment, checkpointing services are configured for automatically performing a number of application services, including: injecting registration code into all applications in the application group during launch, registering the group&#39;s application as they launch, detecting execution failures, and executing from backup nodes in response to application group failure, application failure or node failure. The services can be integrated transparently into the system in that they are implemented on the system without the need of modifying or recompiling the application program, without the need of a custom loader, or without the need for a custom operating system kernel. In another embodiment, a custom loader is used. 
     In at least one embodiment the checkpointing services are configured to support fork( ) and exec( ) in any combination. Exec( ) without a prior fork( ) overwrites the entire address space of the application, including all registration with the coordinator, fault detectors etc. The present invention provides techniques to handle the fact that all memory and registration information is being overwritten during exec( ). 
     In at least one embodiment the checkpointing services support shell scripts, where the core shell script application launches (using fork( ) exec( ) and overlays (using exec( ) new independent applications in any order. 
     The present invention comprises a set of checkpointing services for application groups. The checkpointing services run on every node where the group application can run. One embodiment of the invention generally functions as an extension of the operating system and runs on all nodes. A coordination mechanism is utilized to ensure that the execution of the independent applications are coordinated at certain points. 
     By way of example, and not of limitation, the present invention implements checkpointing services for stateless applications (e.g., sendmail), stateful applications (e.g., Voice over IP (VOIP)), multi-tier enterprise applications (e.g., Apache, WebLogic and Oracle Database combined), wireless devices, such as cell phones, pages and PDAs, and large distributed applications, for example those found in High Performance Computing (HPC), such as seismic exploration and financial modeling. 
     According to one aspect of the invention, the application group runs on a node, with one or more of the independent applications running at any point in time Each independent application is running independently, but is protected and checkpointed together with all other independent applications in the application group. 
     According to one aspect of the invention the application group has one or more backup nodes ready to execute the independent application in the place of the original in the event of a fault. The protection of the application group is thus coordinated and guaranteed to be consistent across fault recovery. 
     An application group can be configured according to the invention with any number of independent applications. Each independent application runs on the primary node while the backup node for the applications stands ready to take over in the event of a fault and subsequent recovery. The primary and backup can be different nodes or the primary and backup can be the same node, in which case the fault recovery is local. 
     The invention provides layered checkpointing services for application groups, with checkpointing services provided both at the application group level and at the individual independent application level. High availability, including fault detection and recovery, for the individual independent application is provided by Duration&#39;s existing stateful High Availability Services. The invention layers a distributed fault detection and recovery mechanism on top of the local fault detection and ensures that fault detection and recovery is consistent across the entire grid. 
     According to one aspect of the invention, a coordinator provides general coordination and synchronization for the individual independent applications of the group applications. By way of example, and not limitation, the coordinator is shown running on the same node as the independent applications to simplify the following teachings. It should be appreciated, however, that this is not a requirement as the coordinator can run on any node in the system. 
     By way of example, and not of limitation, the invention implements stateless or stateful recovery of application groups by recovering each independent application and ensuring all independent applications are recovered in a consistent state. The recovery is automatic without any application group or independent application involvement. 
     According to an aspect of the invention, there is a clean separation of the application logic from the checkpointing services code. This allows application programmers to focus on writing their application code, rather than on writing checkpointing code. An administrator can make applications highly available by simply configuring the desired settings, such as by using a graphical configuration tool implemented according to the invention. The result is that high availability applications are developed easily and deployed quickly without the necessity of custom coding. 
     According to another aspect of the invention, protection is provided against node faults, network faults and process faults. The present invention provides user-controlled system management, automatic availability management, and publish/subscribe event management, including notification of faults and alarms. 
     In various embodiments of the invention, features are provided that are useful for application groups that must be highly available, including but not limited to the following: 
     (a) Stateful high availability and checkpointing for application groups, scripts, including high performance computing, financial modeling, enterprise applications, web servers, application servers, databases, Voice Over IP (VOIP), Session Initiation Protocol (SIP), streaming media, Service Oriented Architectures (SOA), wireless devices, such as cell phones, and PDA 
     (b) Coordinated Restart and stateful restore for applications groups. 
     (c) Coordinated and transparent checkpointing of application groups 
     (d) Coordinated full and incremental checkpointing for applications groups 
     (e) Checkpoints stored on local disks, shared disks, or memories. 
     (f) Automatic and transparent fault detection for application groups 
     (g) Node fault detection. 
     (h) Process fault detection. 
     (i) Application group deadlock and hang protection through external health checks. 
     (j) Coordinated automatic and transparent recovery of applications groups. 
     (k) Auto-startup of applications groups 
     (l) Script support of starting, stopping, or restarting. 
     (m) Dynamic policy updates. 
     (n) User-controllable migration of distributed applications. 
     The invention can be practiced according to various aspects and embodiments, including, but not limited to, those described in the following aspects and embodiments which are described using phraseology which is generally similar to the claim language. 
     According to an aspect of the invention a method for achieving transparent integration of an application group program with a high-availability protection program comprises: (a) injecting registration code, transparently and automatically, into all independent applications when they launch, without the need of modifying or recompiling the application program and without the need of a custom loader; (b) registering the independent applications automatically with the high-availability protection program; (c) detecting a failure in the execution of the application group or any independent application within the group; and (d) executing the application group with application group being executed from their respective backup servers automatically in response to the failure. The high-availability protection program is preferably configured as an extension of the operating system wherein recovery of application groups can be performed without modifying programming within said application programs. The high-availability protection can be configured for protecting against node faults, network faults, and process faults. 
     According to another aspect of the invention, a method, system, improvement or computer program is provided for performing loss-less migration of an application group, including loss-less migration of all independent applications from their respective primary nodes to their backup nodes and while being transparent to a client connected to the primary node over a TCP/IP, MPI, system bus or other transport. The transport, i.e. TCP/IP, MPI, or system bus will optionally be flushed and/or halted during checkpointing. 
     According to another aspect of the invention, a method, system, improvement or computer program performs loss-less migration of an application group, comprising: 
     (a) migrating the independent applications within an application, without loss, from their respective primary nodes to at least one backup node; (b) maintaining transparency to a client connected to the primary node over a transport connection; (c) optionally flushing and halting the transport connection during the taking of checkpoints; and (d) restoring the application group, including all independent applications, from the checkpoints in response to initiating recovery of the application. The execution transparency to the client is maintained by a high-availability protection program configured to automatically coordinate transparent recovery of distributed applications. Transparency is maintained by a high-availability protection program to said one or more independent applications running on a primary node while at least one backup node stands ready in the event of a fault and subsequent recovery. 
     According to another aspect of the invention, a method, system, improvement or computer program performs fault protection for applications distributed across multiple computer nodes, comprising: (a) providing high-availability application services for transparently loading applications, registering applications for protection, detecting faults in applications, and initiating recovery of applications; (b) taking checkpoints of independent applications within applications groups; (c) restoring the independent applications from the checkpoints in response to initiating recovery of one or more the applications; (d) wherein said high-availability application services are provided to the independent applications running on a primary node, while at least one backup node stands ready in the event of a fault and subsequent recovery; and (e) coordinating execution of individual independent applications within a coordinator program which is executed on a node accessible to the multiple computer nodes. 
     According to another aspect of the invention, a method, system, improvement or computer program performs loss-less migration of an application group, comprising: (a) a high-availability services module configured for execution in conjunction with an operating system upon which at least one application can be executed on one or more computer nodes of a distributed system; and (b) programming within the high-availability services module executable on the computer nodes for loss-less migration of independent applications, (b)(i) checkpointing of all state in the transport connection, (b)(ii) coordinating checkpointing of the state of the transport connection across the application group (b)(iii) restoring all states in the transport connection to the state they were in at the last checkpoint, (b)(iv) coordinating recovery within a restore procedure that is coupled to the transport connection. 
     According to another aspect of the invention, there is described a method, system, improvement and/or computer program for maintaining all transport connection across a fault. Transport connections will be automatically restored using Duration&#39;s virtual IP addressing mechanisms. 
     Another aspect of the invention is a method, system, improvement and/or computer program that provides a mechanism to ensure that the independent applications are launched in the proper order and with the proper timing constraints during recovery. In one embodiment, a mechanism is also provided to ensure that application programs are recovered in the proper order. 
     Another aspect of the invention is a method, system, computer program, computer executable program, or improvement wherein user controllable launch of independent applications for the application group is provided. 
     Another aspect of the invention is a method, system, computer program, computer executable program, or improvement wherein user controllable stop of independent applications and application group is provided. 
     Further aspects of the invention will be brought out in the following portions of the specification, wherein the detailed description is for the purpose of fully disclosing preferred embodiments of the invention without placing limitations thereon. 
    
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S) 
       The invention will be more fully understood by reference to the following drawings which are for illustrative purposes only: 
         FIG. 1  is a block diagram of how the coordinator launches an independent application and the events and mechanism of installing the interceptor and handling fork( ). 
         FIG. 2  is a block diagram of how the coordinator launches an independent application and the events and mechanism of installing the interceptor and handling an exec-only( ) call. 
         FIG. 3  is a block diagram illustrating the preservation of registration and checkpointing information across an exec( ) call. 
         FIG. 4  is a block diagram illustrating incremental checkpointing of application groups using both fork( ) and exec( ). 
         FIG. 5  is a block diagram illustrating incremental checkpointing of application groups, where the applications are launched independently. 
         FIG. 6  is a block diagram illustrating launch and registration of independently launched applications. 
         FIG. 7  is a block diagram illustrating restoration of an application group from a checkpoint. 
         FIG. 8  is a block diagram illustrating incremental checkpointing of memory pages written from kernel space. 
         FIG. 9  is a block diagram illustrating typical deployment scenarios. 
         FIG. 10  is a block diagram illustrating devices and computers running the invention. 
         FIG. 11  is a block diagram illustrating storage checkpointing application groups. 
         FIG. 12  is a block diagram illustrating File Operations Databasing. 
         FIG. 13  is a block diagram illustrating storage checkpointing with concurrent file operations. 
         FIG. 14  is a block diagram illustrating storage checkpointing over Network Attached Storage. 
         FIG. 15  is a block diagram illustrating storage checkpointing over Storage Area Networks. 
         FIG. 16  is a block diagram illustrating a checkpointing algorithm with respect to barrier operation for storage checkpointing. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Referring more specifically to the drawings, for illustrative purposes the present invention will be described in relation to  FIG. 1  through  FIG. 16 . It will be appreciated that the system and apparatus of the invention may vary as to configuration and as to details of the constituent components, and that the method may vary as to the specific steps and sequence, without departing from the basic concepts as disclosed herein. 
     1. Introduction 
     The context in which this invention is described is an application group consisting of any number of independent applications. Each independent application runs on the primary node and can be supported by one or more designated backup nodes. Without affecting the general case of multiple backups, the following describes scenarios where each independent application has one primary node and one backup node. Multiple backups are handled in a similar manner as a single backup. 
     The mechanisms for transparently loading applications, transparently registering applications for protection, preloading libraries, transparently detecting faults, and transparently initiating recovery are described in the first reference above which was incorporated by reference. The mechanisms for taking checkpoints of multi-process, multi-threaded processes including processes using fork, and restoring from those checkpoints are described in the second reference above which was incorporated by reference. The mechanism for launching the coordinator, which in turn launches the application, is described in the first and second references, which were incorporated by reference. The mechanism used by the “Duration AM” to launch any process, including the coordinator, is described in the first and second reference and incorporated by reference. All applications in this invention are launched by the Duration AM, through either a coordinator or directly. 
     2. Checkpointing Across Fork and Exec 
       FIG. 1  illustrates, by way of example embodiment  10 , an independent application  12  being launched by the coordinator  11 . The coordinator  11  installs the interceptors  24  for fork and exec  14  and then takes the application through registration  16  with the coordinator. The interceptors  24  are not called at this point; they are loaded into memory and ready to take over when the application calls fork or exec. All preparation is now complete, and the application proceeds to run  18 . If the application  20  issues a fork call, the control passes to the interceptor  24 . The interceptor calls the operating system fork( )  26 , which in turn creates the new application process  28  and passes control back to the interceptors  24 . The interceptors takes the new process  28  through the same configuration and registration as the parent process ( 14 , 15 , 18 , 20 ), and updates the process information for the parent process  20 . The parent process  20  resumes execution with the instruction following fork( )  27 , and the child process  28  also resumes execution at the instructions following the return of fork( )  29 . Application processes  20  and  28  are now both executing. As each of the processes terminates  22 ,  30  they unregister, and the independent application terminates  32 . 
       FIG. 2  illustrates by way of example embodiment  40  an independent application  42  being launched by the Coordinator  41 . The coordinator  41  installs the interceptors  54  for fork and exec and then takes the application through registration  46  with the coordinator. The interceptors  54  are not called at this point; they are loaded into memory and ready to take over when the application calls fork or exec. All preparation is now complete, and the application proceeds to run  48 . If the application  50  issues an exec call, the control passes to the interceptor  54 . The mechanism by which the interceptor keeps track of key application state across exec is described along with  FIG. 3  below. The interceptor calls the operating system exec  56 , which then in turn overlays the new image onto the existing application process  58 . The checkpointer preload library takes the newly created image through full initialization, including registration with the coordinator and restoration of all internal state from shared memory. As described below, the new process image  58  is now fully initialized and begins executing. The original process image  50  no longer exists as exec overwrote its address space. An environment variable CPENV_EXEC is used to store the number of times the process has exec&#39;ed and to retrieve the information from shared memory, as described below. Eventually the application process  58  terminates and unregisters  60 . 
       FIG. 3  illustrates by way of example embodiment  70  how the exec interceptor preserves its internal state across an exec call.  FIG. 3  describes the state preservation across exec-calls as previously described in the exec interceptor  54  on  FIG. 2 . As previously described the Coordinator  71  launches the application and installs the interceptors  73 . 
     Furthermore, the invention always stores the following global application state  75  to shared memory, so it is therefore available at all times:
         Checkpoint barrier info, including barrier semaphore ID   Virtual PID table   Pipe table   Semaphore ID table for non-checkpointer semaphores   SysV shared memory segment ID table (non-checkpointer segments)
 
After attaching to the global state in shared memory, the application resumes execution  77 . The exec interceptor  72  is called when the main application calls exec. The interceptor  74  proceeds to capture all process data that must be preserved across exec. The example embodiment  70  preserves the following data using shared memory:
   Registration Info   Fifo to communicate to coordinator   Checkpointer policies from parent   File info for files that don&#39;t close-on-exec (descriptors, creation attributes, flags, dup info, etc.)   Dynamic priority and scheduling policy/parameters   Signal mask   Virtualized resource limits   Virtualized IP info   Virtualized SysV shared memory segment IDs for segments the process is attached to (non-checkpointer segments)   Application group logical name (HA_APPLICATION)   Coordinator process ID   Defunct children info
 
In this context “virtualized” is utilized to mean the resource abstraction and remapping described in the two references cited above. When all data has been assembled  76 , it&#39;s written to shared memory  82 . The shared memory is identified by a shared memory ID. In an example embodiment using POSIX shared memory, the shared memory ID can be constructed directly from the process ID of the process and the HA_APPLICATION name, so it is not necessary to save it to the environment. The exec-counter CPENV_EXEC is stored in the local environment  84 , and the interceptor preserves it across the exec call. The shared memory is external to the process and remains unaffected by exec. With the exec-count stored in the local environment  84  and the state preserved in shared memory  82 , the checkpointer library, using the exec-count and data retrieved from shared memory, takes the newly exec&#39;ed process  80  through initialization as described under  FIG. 2 .
       

     In another embodiment, the shared memory ID and the CPENV_EXEC count are both written to the environment and used for correct re-initialization. 
     3. Incremental Checkpointing of Application Groups Started from One Application 
     The mechanisms for taking checkpoints of multi-process, multi-threaded processes launched from one binary and restoring from those checkpoints are described in the second reference above which was incorporated by reference.  FIG. 4  illustrates by way of example embodiment ( 100 ), how an application group that uses both fork/exec and exec is incrementally checkpointed. The coordinator  101  launched the application  102 , and then installs interceptors and registers the process as described previously. Upon completion of the initialization the application  104  is ready and starts running  106 . The first checkpoint  108  is a full checkpoint as there are no prior checkpoints. The 2nd checkpoint  110  is incremental and only contains the memory pages changed since the first checkpoint. The application now calls fork and creates a new process  120 , which registers and installs interceptors. The 3rd checkpoint  112  is a bit more involved: both the original process  106  and the new process  120  are checkpointed incrementally. Following fork, both parent and child have identical address spaces, page tables, and identical lists of dirty pages. As each process  106 ,  120  resumes running, each becomes independent, but still has incremental information against the same full checkpoint; they can therefore both be checkpointed incrementally and merged against the pre-fork full checkpoint. If the child process  120  forks another process, the same description applies. The 4th checkpoint  114  is incremental for both processes  106  and  120 . The process  106  now calls exec and overlays a new image. Following the procedure described under  FIG. 2  and  FIG. 3  checkpointer infrastructure is preserved and the checkpointing continues to operate across the complete replacement of the address space. The 5th checkpoint  116  is now a full checkpoint for process  106  while it continues to be incremental for 120. The 6th checkpoint  118  is incremental for both processes  106  and  120 . Upon termination of both processes  122 ,  124  the application terminates  126 . 
     4. Incremental Checkpointing of Application Groups 
     Up until now we&#39;ve considered checkpointing of application groups where the independent applications are created using fork( ) and exec( ) from one application. We now turn to the general scenario of application groups consisting of multiple independent applications launched independently at different times.  FIG. 5  illustrates by way of an example embodiment  140  how the coordinator  141  first launches application  142  and then installs interceptors and registers  142  with the coordinator. Application  142  is ready to run  143  and proceeds to run  144 . In the meantime, the Duration AM  161  launches a second independent application  162  and passes the coordinator  141  process ID and HA_APPLICATION name in the environment. Using the Coordinator PID and the HA_APPLICATION name, the application  162  registers with the coordinator  141 . The second application is ready to run  164  and proceeds to run  166 . While  FIG. 5  looks similar to  FIG. 4  there is one very significant difference: in  FIG. 4 , the second application  120  is created by fork( ) from the first application  102 , while in  FIG. 5  the second application  162  is launched independently from the first application  142 . The mechanism by which application  162  joins an already running coordinator and checkpoint barrier is described in  FIG. 6 . 
     The first checkpoint  146  is taken as a full checkpoint of application process  144 . This is followed by an incremental checkpoint  148 . The third checkpoint  150  includes the second independent application  166 , and contains an incremental checkpoint for application  144  and a full checkpoint of application process  166 . The fourth checkpoint  152  is incremental for both applications  144  and  166 . The embodiment in  FIG. 5  shows applications  144  and  166  without any use of fork( ) and exec( ). 
     It is readily apparent to someone skilled in the art, that application  144 , 166  could use fork( ) and/or exec( ) and combined with the teachings above application groups containing any number of independent applications, launched independently or via fork/exec can be checkpointed using the present invention. 
     5. Launching Independent Applications 
     In order to let any independent application, join an existing coordinator and application group, that new application needs to be able to find and communicate with the coordinator.  FIG. 6  is an example embodiment  180  of how that can be achieved. The coordinator  181  launches the first application  182  and, as previously described, takes it through registration  182  and proceeds to let it run  184 . At a later time, the Duration AM  186  launches a second application  188  and passes the coordinator  181  PID and HA_APPLICATION name via the environment. As described in the second reference, checkpointing is coordinated using a checkpointer semaphore. As described above the checkpointer semaphore is always stored in shared memory, and can be accessed via the shared memory ID constructed from the coordinator PID and HA_APPLICATION name, both of which were provided to the application  188  via the environment. The coordinator  181  is unaware of the second application  188  until registration, and could conceivably trigger a checkpoint during the registration process. To prevent checkpointing of partially launched applications, the second application  188  first acquires the checkpointer semaphore  190 , which prevents the coordinator  181  from triggering checkpoints. This is followed by registration  192  with the coordinator  181  and followed by the release of the checkpointer semaphore  194 . The mechanism for obtaining and releasing semaphores is well known in the art and will not be described further here. The new application  188  is now ready to run  196 . 
     It&#39;s readily apparent to anyone skilled in the art that the launch mechanism described here combines with the previous teaching and completes the support for coordinated checkpointing of application groups to include both programmatic creation of processes with fork( ) and external loading of new processes with the AM. The teachings also support loading the applications at different times, as just described above. 
     6. Restoring an Application Group 
     The mechanisms for restoring multi-process, multi-threaded applications launched from one binary are described in the second reference above which was incorporated by reference. The checkpoints for the application groups contain all the process and thread tree hierarchy information, the environmental information needed to register independent applications and checkpoint across exec.  FIG. 7  illustrates an example embodiment  200  of restoring an application group. As described in the second reference, the coordinator  201  is initially launched as a place holder for all processes to be restored. The coordinator reads the process tables  202  from the checkpoint and creates the process hierarchy  206 ,  212  for the entire application group. For the first process  206  the image is restored from the checkpoint and the environment variables  204 . After the process hierarchy has been recreated each process exec its binary image the same number of times it previously exec&#39;ed using checkpoint and environment variables. The second process  212  is similarly restored from checkpoint and environment variables  214 , and each process exec as described for the first process. Interceptors for both application processes  206  and  212  are also installed at this point. The independent applications  208 ,  216  are now ready to run and proceed to execute as of the restored checkpoints  210 ,  218 . Both independent applications  210 ,  218  now run and are checkpointed  220  using the techniques previously taught. 
     7. Incremental Checkpointing of Memory Pages Written from Kernel Space 
     The mechanism for incremental checkpointing and how to mark/clear dirty pages written from user-space is described in reference two and incorporated by reference. The mechanism relies on interception of SIGSEGV signals as described. However, attempts to write to read-only use-space pages in memory from kernel-mode, i.e. from a system call, do not trigger SIGSEGV; rather they return EFAULT as an error code. Systems calls in general return an EFAULT error instead of triggering the SIGSEGV, should they write to read-only application memory. The present invention adds full support for EFAULT from system calls, in addition to SIGSEGV. It should be noted that in the example embodiment system library functions can also return EFAULT. Since the system library EFAULTs originate outside kernel-mode, the previous teachings above apply; here we&#39;re only concerned with pages written from kernel space, i.e. system calls.  FIG. 8  illustrates an example embodiment  220  of how the coordinator  221  initializes  222  and launches the application or application group  226  as previously described. In one embodiment of the invention, a customized system library  228  is used. The customized system library  228  contains predefined pre-system-call and post-system-call function-calls to the checkpointer library. 
     By way of example, we consider the case where the application  226  calls a system-library call “library_callX( )” located in the system library  228 . Initially the entry point library_callX( )  237  is called. Before reaching the system call  236  it executes the pre-call callback  234  and registers information with the checkpointer  230 , then the system call  236  named “system_callA( )” by way of example is run. The system call reaches the kernel  232  and system_callA( ) runs and returns potentially with an EFAULT error condition. The post-call callback  238  processes the error codes, if any, and updates via the callbacks  230  the page tables maintained by the checkpointer. Finally, control returns  239  to the application  226  and execution continues. 
     In another embodiment the standard system library is used, and the pre-system-call and post-system-call callbacks are installed dynamically by the coordinator as part of application initialization. 
     8. Handling of Efault 
     As described in reference two and incorporated by reference, processing a SIGSEGV fault is done by updating the page table and making the page writable. We now proceed to describe the handling of EFAULT is more detail. Continuing with the example embodiment  220  in  FIG. 8 , if the system call “system_callA( )” safely can be called again, the pre/post callbacks operate as follows: 
     1. pre-call callback  234  does nothing. 
     2. post-call callback  238  determines if EFAULT was returned. If EFAULT was returned due to the checkpointer write-protecting one of more of system_callA( )&#39;s call-arguments memory pages, the pages are marked as writable, the checkpointers page table is updated, and the system_callA( ) is called again. 
     If system_callA( ) cannot be safely called again, the present invention proceeds as follows: 
     1. the pre-call callback  234  marks memory pages belonging to the calls arguments as dirty and disables write-protection for the duration of the system call. 
     2. let the call to system_callA( ) go through  236 . 
     3. the post-call callback  238  then re-enables write protection for the affected pages. 
     The terms “call-arguments memory pages” and “memory pages belonging to call argument” are utilized to mean the following. By way of example, a function might have a number of parameters, some of which are pointers to memory locations. The aforementioned “memory pages” are the memory pages referenced, or pointed to, by pointers in the argument list. 
     In another embodiment all EFAULT handling is done in a kernel module sitting under the system library. 
     9. Loss-Less Migration of Application Groups 
     Referring once again to  FIG. 2  for illustrative purposes, the case of migrating the distributed application from one set of nodes to another set of nodes is considered. Migration of live applications is preferably utilized in responding to the anticipation of faults, such as detecting that a CPU is overheating, a server is running out of memory, and the like, when the administrator wants to re-configure the servers or when the servers currently being used have to be freed up for some reason. 
     Building on the disclosures above, a loss-less migration is achieved by: first checkpointing the application group, including all independent applications and optionally the local transports, then restoring all independent applications and optionally the local transports from the checkpoints on the backup nodes. The migration is loss-less, which means that no data or processing is lost. 
     10. Virtualization and Live Migration of Application Groups 
     Loss-less migration of application groups can be viewed differently. The ability to checkpoint and migrate entire application groups makes the application location-independent. The application groups can be moved, started and stopped on any server at any point in time. The present teaching therefore shows how to de-couple a live running instance of an application from the underlying operating system and hardware. The application execution has therefore been virtualized and enables live migration, i.e., a migration of a running application, without any application involvement or even knowledge. 
     11. Deployment Scenarios 
       FIG. 9  illustrates by way of example embodiment  240  a variety of ways the invention can be configured to operate. In one embodiment, the invention is configured to protect a database  242 , in another it is configured to protect a pair of application servers  244 ,  246 . In a third embodiment the invention is configured to protect a LAN  248  connected PC  252  together with the application servers  244 ,  246 . In a fourth embodiment the invention is configured to protect applications on a cell phone  250 , which is wirelessly connected  258  to the Internet  256 , the application servers  244 , 246  and the database  242 . A fifth embodiment has a home-PC  254  connected via the internet  256  to the application servers  244 , 246  and the LAN PC  252 . The invention runs on one or more of the devices, can be distributed across two or more of these elements, and allows for running the invention on any number of the devices ( 242 , 244 , 246 , 250 , 252 , 254 ) at the same time providing either a joint service or any number of independent services. 
     12. System Diagram 
       FIG. 10  illustrates by way of example embodiment  260  a typical system  262  where the invention, as described previously, can run. The system memory  264  can store the invention  270  as well as any run application  266 ,  268  being protected. The system libraries  272  and operating system  274  provide the necessary support. Local or remote storage  276  provides persistent storage of and for the invention. The invention is generally loaded from storage  276  into memory  264  as part of normal operation. One or more CPUs  282  performs these functions, and may use the network devices  278  to access the network  284 , and Input/Output devices  280 . 
     13. Storage Checkpointing of Application Groups—Consistency 
       FIG. 11  illustrates by way of example embodiment  300  a typical server/computer  308  with attached storage  316 . The storage  316  can be built into the server as Direct Attached Storage (DAS), or be remote and accessed via either Network Attached Storage (NAS) or a Storage Area Network (SAN). Each of those topologies will be addressed specifically below; at this point all that is assumed is that the storage  316  is accessible over some storage access interface  302 . By way of example and not limitation, the storage access interface  302  is PCI, ATA, SAS, SCSI for DAS, Ethernet, Fibre Channel and Infiniband for NAS, and SCSI, Fibre Channel, ATA over Ethernet, and HyperSCSI for SAN. 
     The application group  310  runs on the server  308  and utilizes files. All access to the files stored on disk  316  goes through the file system  312 , which in turn calls the operating system  314  and device drivers  315 . By way of example and not limitation, we show storage and networking device drivers in  315  and in the following diagrams; this is not a limitation—all device drivers are included. Ultimately the storage device driver is responsible for reading and writing data to disk  316  or transmitting the data to the disk in case of NAS and SAN. When writing data to the disk, data is buffered in both the file system  320  and the operating system  314  and device drivers  325 . Likewise, on retrieving data from the disk  316 , data is buffered both in the device drivers  327 , operating system  326  and the file system  322 . Finally, commands such as “seek” or “delete file” may be buffered as well. Depending on file system and operating system, a read operation may be filled fully from one or more of the buffers without ever accessing the disk. Depending on file system and operating system, the file system may report a file as having been written, even though the data still is in one of the buffers and not fully on disk yet. For storage checkpointing to be synchronized with the memory checkpoints, steps must be taken to ensure that the data has been fully written to disk, fully retrieved from disk, and commands completed, as part of the checkpointing process. The buffers  320 , 322 , 324 , 326 ,  325 ,  327  are also referred to as caches. 
     The actual number of buffers used varies by operating system, file system, and storage device. By way of example  FIG. 11  illustrates the use of separate buffers for file system, operating system and device drivers. It is readily apparent to someone skilled in the art that each component may use zero, one or more buffers without affecting the teachings. It is generally harder to ensure consistency with more buffers, so the following example diagrams continue to show buffers at all components; file system, operating system and device drivers. 
     As previously taught, the present invention checkpoints the application group  310  and captures all relevant application state. The application group checkpoint includes all state data related to the application group, including select file info such as path, size, ownership, but does not include the file content, the state of the file system or the disk itself. The only exception is memory mapped files, which are in memory, and therefore included in the checkpoint. The following teachings detail how to make sure all disk operations are in a state consistent with the memory checkpoint. 
       FIG. 12  illustrates by way of example embodiment  340  the mechanism used to ensure that file operations have completed and checkpoints are consistent. By way of example and not limitation, we describe the case of a single application. It&#39;s readily apparent to anyone skilled in the art that the following teachings extend to any number of individual applications. First the coordinator launches the application  342  and installs the exec/fork interceptors as previously disclosed. Additionally, the coordinator installs  344  interceptors for all file operations  356 . The application registers with the coordinator  346  and is ready to run  348 . The application proceeds to run  350 . 
     Upon encountering a file operation, the file operations interceptor  356  is called. The interceptor  356  stores the file event in a memory resident File Operations Database (FODB)  358  for further processing later. The FODB is incorporated into the checkpoint and therefore available at restore time. After storing the file operation, the call is passed to the file system  360 , the operating system  362 , device drivers  363 , and finally the disk  364  via the storage interface  365 . Upon completion control returns to the interceptor  356 . The interceptor proceeds to verify that the file-operation actually completed. Verification of the file operations is covered below. 
     14. Concurrent File Operations 
     File Systems guarantee that serially issued operations will access data in the order the operations were submitted. By way of example, if an application thread first writes data to the disk, then reads data from the disk, the file system ensures that the initial write operation has completed before returning to the thread and letting the following read operation instruction proceed. The file system only guarantees that the sequence of operations is strictly maintained, it does not guarantee that that write-operation data actually has been written to the disk. With many layers of caching, it is very likely that the written data still sits in one of the buffers between the file system and the physical disk. A common file system optimization is to handle writing of data to the physical disk, a.k.a. flushing the buffers or flushing the caches, in the background after the write operation has returned to the calling application thread. By way of example, if the application thread issues a series of write operations, all data might still be sitting in a variety of buffers, but as soon as the first read operation is issued, all the buffers will be brought in sync. Issuing a read operation from the application thread essentially forces all caches into a consistent state. The present invention writes and reads checkpoint tokens as a way to ensure cache consistency by forcing data on and off the disk. This is covered in detail below. 
     In general applications are multi-threaded and may have multiple overlapping storage operations. Each thread is guaranteed serial consistency as described above.  FIG. 13  illustrates by way of example embodiment  380  the operation of the file operations interceptor  390  for an application with ‘n’ threads: Thread 1   382 , Thread 2   384  and Thread-n  386 . Each thread has ongoing file operations. As described above, each threads file operations are guaranteed to be serially consistent. With multi-threaded and multi-process applications, it&#39;s the application&#39;s responsibility to ensure that access to files is coordinated using for instance semaphores or mutexes. Arbitration of shared resources using semaphores or mutexes is well known in the art, and will not be described further herein. By way of example, if two threads simultaneously write to the same file without coordination through e.g. a semaphore, the results are unpredictable. The preferred implementation of this invention relies on the application correctly arbitrating file access using semaphores or mutexes. With full arbitration at the application level, and each thread being serially consistent, no further coordination is needed across threads while accessing the FODB  392 . The FODB  392  maintains separate events for each thread. By way of example, the FODB  392  maintains a list of pending events  393  for thread  1 , list of pending events  395  for thread  2 , and list of pending events  397  for thread ‘n’. If the application relies on file-level locking, such as ‘FileLock’ on Windows and ‘fcntl’ on Linux, the invention falls back on the alternate implementation described next. 
     In an alternate implementation the requirement for application file access arbitration is removed. In this case the FODB  392  needs to ensure atomic access for the file operations and locking, and uses a semaphore for each file to coordinate file operations for a particular application group. Use of the semaphore only ensures that overlapping and conflicting file operations are serialized; it does not eliminate the lack of application level resource arbitration. 
     15. Cache Consistency During Checkpointing 
     For each thread, every time a file operation arrives at the interceptor  390 , the details of the file operation are logged in the FODB  392 . Upon successful completion of the file operation, the pending event is removed from the FODB  392 . Referring to the previous teachings; at the time the file operation completes and control returns to the interceptor, all we know is that the data has been exchanged with the buffers; there are no guarantee that any data has reached or been retrieved from the disk. At any point in time, the FODB  392  contains all storage operations that have been issued by the application thread, but not completed. At the time of a checkpoint, the checkpointer needs to bring all pending operations in the FODB  392  in sync with the applications memory image. By way of example, if the application thread has issued a write operation prior to the checkpoint, but the write operation has not completed, the interceptor needs to bring the file system and all caches in sync, and make sure that the write completes. By way of example, if the application has issued a read operation, the interceptor needs to ensure that the read brings the data into the applications address space and that all caches are in sync. To ensure consistency, triggering of checkpoints is disabled between adding a file operation to the FODB and the beginning of the file operation. The detailed sequence of how memory checkpointing is combined with FODB synchronization is detailed below. 
     For each individual thread the FODB  392  processes pending file operations as follows: The FODB waits for the operation to complete. Return values are captured by the interceptor, including both success and failure of the operation. The return values will be returned to the application after the checkpointing process completes. The pending operation is removed from the FODB  392 . At this point, the application thread and the file system have identical view of thread data written to and read from the file system, and the interceptor for the application thread contains the return values and data for the operation. The interceptor waits for the interceptor barrier to complete, as described below, before resuming 
     At checkpointing time, the individual threads are handled as just described. All threads are coordinated using the barrier as described in reference two and incorporated by reference. The barrier ensures that all pending operations for all threads complete. When all threads have completed their File Operations processing described above, the main checkpointing thread optionally flushes all buffers in the file system and the kernel belonging to the application group processes using the standard library functions. This global file system and kernel flush forces all caches data onto the disk in the case of DAS, or onto the storage subsystem in the case of NAS and SAN. 
     In an alternate implementation on the Linux operating system, checkpoints are triggered using signals, as described in reference two and included by reference. Checkpointing runs on the signal handler thread, and cannot call application functions, including issuing calls to wait for the FODB threads to complete. This particular limitation is addressed by using a slightly different flushing mechanism. The FODB has a list of all open file descriptors, and calls fsync( ) on each of the open file descriptors. The call to fsync( ) forces all currently queued I/O operations for the file descriptor to completion. Once flushing is complete the corresponding entries are removed from the FODB. This is functionally equivalent to the general sequence described above. If checkpointing was triggered in the middle of a file operation, the result of the file operation would still be in sync with the memory image of the application after the fsync( ) and the appropriate error values and/or data will be returned to the application. 
     16. Storage Checkpointing of Application Groups Running Over Nas Storage 
       FIG. 14  illustrates by way of example embodiment  420  a typical Network Attached Storage (NAS)  422 ,  434  configuration. An application group  424  is running on its host system  422 . NAS presents the storage subsystem using a network file system, which is mounted locally  426 . Example network file systems include Network File Systems (NFS), Server Message Block (SMB), and the older Common Internet File System (CIFS). The network file system  426  utilizes the underlying operating system  428  and device drivers  430  to communicate over a data network  432  to the NAS device  434 . The present invention has no control over the NAS device; all it can do is to operate on files using the Network File System  426 . 
     To ensure consistency between the application group&#39;s checkpoint and the application group&#39;s files, one additional step can be taken. Also referring to  FIG. 13  for illustrative purposes: when all threads in the interceptor  390  have completed processing as described above, they all additionally write a checkpoint token file  406  to the Network File System  426 , followed by flush commands to the Network File System  426  and the Operating System  428 . This is followed by reading back the checkpoint token file  406 . This write-commit-read cycle forces data out of the local server  422 , onto the network  432  and onto the NAS device  434 , and forces a consistency flush at the NAS device. From the local host&#39;s  422  and the application group&#39;s  424  perspective there is now consistency between the application groups view of its files and what has been committed to the remote NAS device. 
     17. Storage Checkpointing of Application Groups Running Over San Storage 
       FIG. 15  illustrates by way of example embodiment  440  a typical Storage Area Network (SAN) configuration. An application group  444  is running on its host system  442 . SAN uses standard file systems  446 , but uses a specialized storage network  452  and associated device drivers  450 . Example storage networks include Fibre Channel, iSCCI, and Gigabit Ethernet. SAN makes the remote SAN device  454  appear local to the application group  444  and the local operating system  446 . Even though the mechanism of SAN is very different than NAS, the NAS teachings apply directly. The present invention makes no assumptions about the nature of the remote storage, only that it can be accessed via a file system that offers standard read, write, and flush operations. 
     18. Taking Storage Checkpoints 
     As described in reference two, which was incorporated by reference, and augmented by the teachings above, the checkpointer uses a barrier to get the application group into a consistent state. While in the barrier, the techniques taught above are used to ensure cache and file consistency between the application group and its associated local or remote storage.  FIG. 16  illustrates by way of example embodiment  460  the checkpointing algorithm. First the main thread claims the barrier semaphore  462 , and waits for all threads and processes to join  464 . When all processes and threads have entered the barrier, storage buffers are flushed  466 , followed by memory checkpointing  468  and finally the storage checkpoint  470 . Upon completion of the storage checkpoint, the barrier is release  472 , and the application group resumes execution  474 . 
     The storage checkpoint consists of a copy of all files modified by the application groups since last storage checkpoint. The list of files that have been modified since last checkpoint is readily available as the interceptor for file operations ( 356  on  FIGS. 12 and 390  on  FIG. 13 ) has processed all file commands. For each thread, the interceptor simply keeps an in-memory list of all files modified. 
     Taking a storage checkpoint  470  breaks down as follows: 
     a. Obtain list of modified files from the file-operations interceptor 
     b. Copy all files to the backup location 
     c. Clear list of modified files in the file-operations interceptor 
     As part of configuring the present invention, the administrator provides either a pre-defined location to be used for storage backup, or the system uses the default temporary directory. 
     The aspect of storage checkpointing where modified files are being copied can be further optimized. The direct approach is to use the operating system provided copy( ) command. This works across all file systems so it is the default mode of operation. More advanced storage systems offer a “device copy,” where the storage device, typically NAS and SAN, does all the copying without any host operating system involvement. For a given storage system, if the device copy is available, that is the preferred implementation. 
     19. Double Buffering of Storage Checkpoints 
     For reliability, all storage checkpoints need to be double buffered. At any given point in time, the invention maintains the most recent successful storage checkpoint, in addition to the current storage checkpoint being created. If anything fails while taking a storage checkpoint, the invention can fall back on the previous during storage checkpoint and use that, combined with its associated memory checkpoint for restoration. Upon successful creation of a storage checkpoint, the previous one is deleted. 
     20. Restoring a Storage Checkpoints for an Application Group 
     Restoring a storage checkpoint only requires copying all files from the storage checkpoint backup directory back to their original locations. This is followed restoring the application group&#39;s memory image from the associated checkpoint. The application group&#39;s memory and storage are now consistent and the application group can resume execution. 
     21. Conclusion 
     In the embodiments described herein, an example programming environment was described for which an embodiment of programming according to the invention was taught. It should be appreciated that the present invention can be implemented by one of ordinary skill in the art using different program organizations and structures, different data structures, and of course any desired naming conventions without departing from the teachings herein. In addition, the invention can be ported, or otherwise configured for, use across a wide-range of operating system environments. 
     Although the description above contains many details, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the exemplary embodiments of this invention. Therefore, it will be appreciated that the scope of the present invention fully encompasses other embodiments which may become obvious to those skilled in the art, and that the scope of the present invention is accordingly to be limited by nothing other than the appended claims, in which reference to an element in the singular is not intended to mean “one and only one” unless explicitly so stated, but rather “one or more.” All structural and functional equivalents to the elements of the above-described preferred embodiment that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Moreover, it is not necessary for a device or method to address each and every problem sought to be solved by the present invention, for it to be encompassed by the present claims. Furthermore, no element, component, or method step in the present disclosure is intended to be dedicated to the public regardless of whether the element, component, or method step is explicitly recited in the claims. No claim element herein is to be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited using the phrase “means for.”