Abstract:
Systems and methods operating over extended distances provide for recovery of data and operational continuity of computer applications accessing data within an information technology system if an event occurs effecting access to the data. In one embodiment, an extended distance data recovery system ( 100 ) includes first, second and third data storage devices ( 112, 122, 132 ) located at respective first, second and third sites ( 110, 120, 130 ). The second and third sites ( 120, 130 ) are remotely located from the first site ( 110 ) with the second site ( 120 ) being nearby the first site ( 110 ). The first data storage device ( 112 ) has data ( 116 ) stored thereon. A computer executable control process ( 150 ) directs synchronous replication of the data ( 116 ), either at the storage level or at the application level, onto the second data storage device ( 122 ). The control process ( 150 ) also directs asynchronous replication of the data ( 116 ) from the second data storage device ( 122 ) onto the third data storage device ( 132 ) and coordinates among the three sites ( 110, 120, 130 ) the state of application servers, storage replication, network address changes, and other prerequisite aspects of the IT infrastructure required to allow the application servers to successfully start at the disaster recovery site.

Description:
RELATED APPLICATION INFORMATION 
   This application claims priority from U.S. Provisional Application Ser. No. 60/722,369, entitled “NO DATA LOSS IT DISASTER RECOVERY OVER EXTENDED DISTANCES” filed on Sep. 30, 2005, which is incorporated by reference herein in its entirety. 

   FIELD OF THE INVENTION 
   The present invention relates generally to information technology systems, and more particularly to providing zero loss data protection and automated data recovery over extended distances. 
   BACKGROUND OF THE INVENTION 
   In an information technology (IT) system, it is often desirable to replicate data stored at one location at another location so that if an event occurs (e.g., an equipment failure, a power failure, a natural disaster, or a terrorist attack or other man-made event) that damages or otherwise renders the data at the first location inaccessible, the data can be recovered from the second location. The first location may be referred to as the primary site, the second location may be referred to as the disaster recovery site, and such an occurrence may be referred to as a disaster event. In order to provide sufficient assurance that the disaster recovery site will not be effected by the disaster event effecting the primary site, the disaster recovery site must be geographically separated from the primary site by a sufficient distance. 
   When replicating the data at the disaster recovery site, it may be desirable to do so in a synchronous manner such that when data is created, updated or stored at the primary site, such data is replicated to the disaster recovery site first and only after receiving an acknowledgement from the disaster recovery site of successful replication is the data write considered complete and successful at the primary site. This allows for no lost data if the primary site goes down. However, excessive roundtrip packet delays resulting from long distances and other network conditions between the primary and disaster recovery sites prevents synchronous replication of data from the primary site to the disaster recovery site. This difficulty is present regardless of whether data replication takes place in the storage, middleware, or application layer of the IT system. Excessive roundtrip packet delays can be unacceptable since operation of applications creating, updating or accessing the data will be delayed while awaiting confirmation that the data has been replicated at the disaster recovery site. 
   SUMMARY OF THE INVENTION 
   Accordingly, the present invention introduces an intermediary site between the primary site and the disaster recovery site. The primary site may be referred to herein as the first site, the intermediary site may be referred to herein as the nearby safe site or the second site, and the disaster recovery site may be referred to herein as the third site. The data is replicated from the primary site to the nearby safe site and then subsequently replicated from the nearby safe site to the disaster recovery site. The nearby safe site may be geographically remote from the primary site such that the nearby safe site would be expected to survive a disaster event effecting the primary site for some period of time after the primary site goes down, but not necessarily survive indefinitely. However, the nearby safe site is located close enough to the primary site such that synchronous data replication is possible between the primary site and the nearby safe site without encountering unacceptable roundtrip packet delays. The disaster recovery site is located far enough from the primary site that the disaster recovery site would be expected to survive the disaster event effecting the primary site. Because the data has already been replicated to the nearby safe site, asynchronous replication of the data from the nearby safe site to the disaster recovery site is acceptable. 
   The nearby safe site includes a number of desirable characteristics. For example, the nearby safe site is close enough to the primary site that roundtrip packet delays between the nearby safe site and the primary site would not be detrimental to the user software applications. A typical maximum distance is, for example, one-hundred kilometers or even two-hundred kilometers, although actual allowable distances depend on the actual network paths traversed, roundtrip packet delays encountered, and the software application sensitivity to roundtrip packet delays. The nearby safe site can be run in a “lights out” configuration with no regular staffing requirements and is expected to operate in case of chemical, biological, or nuclear contamination. The primary hardware components at the nearby safe site are data storage devices as well as application servers. The data on the data storage devices could optionally be encrypted allowing the nearby site and its assets to be shared among multiple customers at different physical locations. The nearby site is physically protected to continue to operate for a relatively short period of time after the primary site has sustained a disaster, thereby allowing for the completion of the data transfer that has been buffered at this site to the disaster recovery site. 
   According to one aspect of the present invention, an extended distance data recovery system includes a first data storage device located at a first site, a second data storage device located at a second site, a third data storage device located at a third site, and a computer executable control process coordinating all three sites. The first data storage device has data stored thereon. The second data storage device is communicatively connected with the first data storage device, and the third data storage device is communicatively connected with the second data storage device. The computer executable control process is executable to direct synchronous replication of the data onto the second data storage device. The control process is also executable to direct asynchronous replication of the data from the second data storage device onto the third data storage device. The computer executable control process is executable to coordinate among the three sites the state of application servers, storage replication, network address changes, and other prerequisite aspects of the IT infrastructure required to allow the application servers to successfully start at the disaster recovery site. 
   According to another aspect of the present invention, a method for providing recovery of data and operational continuity of computer applications accessing the data if an event occurs effecting access to the data on an information technology system is provided includes the step of storing the data at a first site. The data is synchronously replicated from the first site to a second site under the direction of a computer executable control process. The data is asynchronously replicated from the second site onto a third site, also under the direction of the control process. The operational status of the first site is monitored, and, upon occurrence of the event, location identifying information associated with the data is updated so that computer applications access the data from the second site and/or the third site. 
   These and other aspects and advantages of the present invention will be apparent upon review of the following Detailed Description when taken in conjunction with the accompanying figures. 

   
     DESCRIPTION OF THE DRAWINGS 
     For a more complete understanding of the present invention and further advantages thereof, reference is now made to the following Detailed Description, taken in conjunction with the drawings, in which: 
       FIG. 1  is block diagram showing one embodiment of an extended distance data recovery system and the operation thereof in accordance with the present invention; 
       FIG. 2  is block diagram showing another embodiment of an extended distance data recovery system and the operation thereof in accordance with the present invention; and 
       FIG. 3  is block diagram showing one more embodiment of an extended distance data recovery system and the operation thereof in accordance with the present invention. 
   

   DETAILED DESCRIPTION 
     FIG. 1  illustrates one embodiment of an extended distance data recovery system  100 . The system  100  includes a first data storage device  112  located at a first site  110  (the primary site  110 ), a second data storage device  122  located at a second site  120  (the nearby safe site  120  or intermediary site  120 ), and a third data storage device  132  located at a third site  130  (the disaster recovery site  130 ). The data storage devices  112 ,  122 ,  132  may, for example, be storage area network (SAN) devices each including a group of networked data storage devices (e.g., hard drives, CD or DVD drives, tape drives, flash memory devices, etc.). In other embodiments one or more of the first, second and third data storage devices  112 ,  122 ,  132  may, for example, be devices other than a storage area network device such as, for example, an individual hard drive. In this regard, data replication may need to be handled at a middleware or application layer level. 
   The primary site  110  may be geographically remote from the other sites such that in the event of an occurrence (e.g., an equipment failure, a power failure, a natural disaster, a terrorist attack or other man-made event) that causes loss of data access at the primary site  110 , conditions effecting the operation of the primary site  110  may not necessarily effect the operations at the other two sites  120 ,  130 . In this regard, the primary and nearby safe sites  110 ,  120  may, for example, be located in different buildings or in different towns as long as the primary and nearby safe sites  110 ,  120  are close enough to one another that a roundtrip packet delay time between the primary and nearby safe sites  110 ,  120  is within an acceptable range, and the primary and disaster recovery sites  110 ,  130  may, for example, be located in different towns, in different states, or even in different countries without regard to a roundtrip packet delay time therebetween. Regardless of the location of each site  110 ,  120 ,  130  relative to each other, they are enabled for communication therebetween via a suitable data network so that data created and/or stored at one site can be communicated to and replicated at another site. The data network may include various private and/or publicly shared wired and/or wireless portions. 
   One or more user applications  114  are executable by one or more computer processors or the like at the primary site  110 . The user application(s)  114  create, update, and/or access data  116  that is stored, via a data input/output (I/O) interface  118  on the first data storage device  112 . 
   The extended distance data recovery system  100  also includes a virtual integration console  150  (VIC  150 ). VIC  150  may also be referred to herein the control process  150  or control application  150 . In one embodiment, VIC  150  is implemented in software executable by a computer processor, and there can be instances of VIC  150  executing on computer systems at each of the primary site  110 , nearby safe site  120  and the disaster recovery site  130 . Each instance of VIC  150  interfaces with the other instance of VIC  150 , and in  FIG. 1  all three instances of VIC  150  are represented as a single block. 
   VIC  150  directs the replication of the data  116  from the primary site  110  to the nearby safe site  120 . In the present embodiment, VIC  150  directs replication of the data  116  from the primary site  110  to the nearby safe site  120  to take place in a synchronous manner at the storage level. In this regard, as packets of the data  116  are written to the first data storage device  112 , the packets of the data  116  are also written to the second data storage device  122  and confirmation that the data replication operation has been completed is provided by the second data storage device  122  to the first data storage device  112  at the primary site  110 . Although the roundtrip packet delay time between the primary and nearby safe sites  110 ,  120  depends on a number of factors including the type of communication network equipment used and overall network traffic, the primary site  110  and the nearby safe site  120  should, in general, be sufficiently proximate to one another such that the roundtrip packet delay time between the primary and nearby safe sites  110 ,  120  is not so long in duration that normal operation of the user application(s) is unacceptably impacted. While the geographic distance between the primary and nearby safe sites  110 ,  120  providing such an acceptable roundtrip packet delay time can vary greatly from one situation to another, geographic distances in the range of 100 kilometers up to 200 kilometers may be possible. 
   In addition to directing replication of the data  116  to the second data storage device  122  at the nearby safe site, VIC  150  also directs replication of the data  116  from the second data storage device  122  to the third data storage device  132  at the disaster recovery site  130 . In the present embodiment, VIC  150  directs replication of the data  116  from the nearby safe site  120  to the disaster recovery site  130  to take place in an asynchronous manner. In this regard, the data  116  is replicated from the second data storage device  122  to the third data storage device  132  when resources at the nearby safe and disaster sites  120 ,  130  are available to copy the data  116 . In this manner, the data  116  is initially replicated at the nearby safe site  120  and thereafter replicated to the disaster recovery site  130 . The asynchronous data replication may occur periodically (e.g., every day, every hour, every minute, or as fast as the communication throughput between nearby safe site  120  and disaster recovery site  130  allow), or may occur in response to certain predefined events. The asynchronous data transfer protocol allows the primary site  110  to effectively be decoupled from the disaster recovery site  130 . Thus, there is no need for the user application(s)  114  to wait for the data to be replicated to the disaster recovery site  130  before continuing with their operations. This configuration permits the disaster recovery site  130  to be located at a distance from the primary site  110  that is much greater than the distance between the nearby safe site  120  and the primary site  110 . For example, the disaster recovery site  130  may be located in a different state or in a different country than the primary site  110 . This provides even greater protection of the data  116  and continuity of the user application(s)  114  from the occurrence of a disaster event. 
   VIC  150  also monitors the operational status of the primary site  110 . If a failure is detected and the application(s)  114  at the primary site  110  are not operating or are not able to access the data  116  from the first data storage device  112 , VIC  150  makes the data  116  available from the nearby safe site  120  and/or the disaster recovery site  130  as appropriate. In this regard, if asynchronous replication of the data  116  from the second data storage device  122  to the third data storage device  132  is complete, then VIC  150  directs resources to access the data  116  from the third data storage device  132  instead of from the first data storage device  112 . For example, one or more user application(s)  134  executable on computer systems located at the disaster recovery site  130  may access the data  116  from the third data storage device  132  via a data I/O interface  138 . The disaster recovery site  130  user application(s)  134  may be the same as and/or provide the same functionality as the user application(s)  114  at the primary site  110  in order to provide continuity of operations formerly accomplished at the primary site  110 . If, however, asynchronous replication of the data  116  from the second data storage device  122  to the third data storage device  132  has not been completed, then VIC  150  directs completion of the asynchronous data replication process. In the meantime, VIC  150  may direct resources to access the data  116  from the second data storage device  122  until the asynchronous data replication process is complete. 
   The user application(s)  134  at the disaster recovery site  130  do not operate while the primary site  110  user application(s)  114  operate, but when the primary site  110  goes down, VIC  150  activates the application(s)  134  at the disaster recovery site. In this regard, the primary site  110  user application(s)  114  are considered active, the disaster recovery site  130  user application(s)  134  are considered passive, and therefore the present embodiment may be referred to as an active/passive extended distance data recovery system  100 . 
   Redirection of resources accessing the data  116  to the nearby safe site and/or disaster recovery site may be accomplished by VIC  150  in a number of manners. One manner is by providing updated IP address information associated with the data  116  to one or more domain name servers  170  (DNSs  170 ). 
     FIG. 2  shows another embodiment of an extended distance data recovery system  200 . The extended distance data recovery system  200  of  FIG. 2  includes a number of elements in common with the system  100  of  FIG. 1 , and corresponding elements are referenced using the same numerals. In the system  200  of  FIG. 2 , VIC  150  directs replication of the data  116  from the primary site  110  to the nearby safe site  120  to take place in a synchronous manner at the application level rather than the storage level. In this regard, as packets of the data  116  are created or updated by the user application(s)  114  at the primary site  110 , the packets of the data  116  are also created and/or updated by user application(s)  124  executable by one or more computer processors at the nearby safe site  120 . The user application(s)  124  at the nearby safe site  120  store the data  116  via a data I/O interface  128  on the second data storage device  122  thereby achieving replication of the data  116  at the nearby safe site  120 . Additionally as packets of the data  116  are created or updated by the user application(s)  124  at the nearby safe site  120 , the packets of the data  116  are also created and/or updated by user application(s)  114  executable by one or more computer processors at the primary site  110 . The user application(s)  114  at the primary site  110  store the data  116  via a data I/O interface  118  on the primary data storage device  112  thereby achieving replication of the data  116  at the primary site  110 . This is true since user application(s)  114  and  124  are both active simultaneously and potentially in a load balanced architecture. 
   The user application(s)  114  at the primary site  110  and the user application(s)  124  at the nearby safe site  120  may be corresponding instances of the same application(s). Since the user application(s)  124  are operating at the nearby safe site  120  simultaneously with the user application(s)  114  at the primary site  110 , both the primary site  110  user application(s)  114  and the nearby safe site  120  user application(s)  124  are considered active and the present embodiment may be referred to as an active/active extended distance data recovery system  200 . 
   As with the system  100  of  FIG. 1 , VIC  150  monitors operation of the primary site  110 , and upon detection of a problem, redirects resources accessing the data  116  to the second data storage device  122  or the third data storage device  132  as appropriate depending upon whether the asynchronous data replication process between nearby safe site  120  and the disaster recovery site  110  has been completed. In this embodiment, upon failure of user application(s)  114 , since user application(s)  124  is(are) also active user communications will be directed solely to user application(s)  124  through normal load balancing mechanisms. Users would not experience an outage as user application(s)  124  provides(provide) dynamic redundancy. At a convenient time which provides a minimal amount of operational impact, user application(s)  124  is(are) stopped by VIC  150  and user application(s)  134  is(are) restarted by VIC  150  after it directs and verifies storage device  122  at nearby safe site  120  to complete replicating all of its changed data to storage device  132  at disaster recovery site  130 . 
     FIG. 3  shows another embodiment of an extended distance data recovery system  300 . The extended distance data recovery system  300  of  FIG. 3  includes a number of elements in common with the systems  100 ,  200  of  FIGS. 1 and 2 , and corresponding elements are referenced using the same numerals. The primary site  110  in the system  300  of  FIG. 3  includes a plurality of first data storage devices  312 A- 312 H each having data  316 A- 316 H stored thereon. The data  316 A- 316 H may be created, updated, and/or accessed by one or more user applications (not shown). The data  316 A- 316 H is synchronously replicated onto the second data storage device(s)  122  at the nearby safe site  120 . In this regard, the data  316 A- 316 H may be synchronously replicated at the application level and/or the storage level as previously described in connection with the system  200  of  FIG. 2  or the system  100  of  FIG. 1 . Regardless of the manner in which the data is replicated from the primary site  110  to the nearby safe site  120 , data replication is directed by VIC  150 . 
   The first data storage devices  312 A- 312 H may be organized into groups. For example, a first group may include first data storage devices  312 A- 312 D and a second group may include first data storage devices  312 E- 312 H. When the data  316 A- 316 H is asynchronously replicated from the nearby safe site  120 , the data  316 A- 316 H may be asynchronously replicated to more than one disaster recovery site. For example, system  300  includes two disaster recovery sites  330 A- 330 B. The data  316 A- 316 D originating from the first group of first data storage devices  312 A- 312 D is replicated to the first disaster recovery site  330 A, and the data  316 E- 316 H originating from the second group of first data storage devices  312 E- 312 H is replicated to the second disaster recovery site  330 B. In this regard, the first disaster recovery site  330 A may include a number of third data storage devices  332 A- 332 D on which the data  316 A- 316 D is replicated, and the second disaster recovery site  330 B may include a number of third data storage devices  332 E- 332 H on which the data  316 E- 316 H is replicated. Regardless of the manner in which the data is replicated from the nearby safe site  120  to the disaster recovery sites  330 A- 330 B, data replication is directed by VIC  150 . 
   As with the systems  100 ,  200  of  FIGS. 1 and 2 , VIC  150  monitors operation of the primary site  110 , and upon detection of a problem, redirects resources accessing the data  316 A- 316 H to the second data storage device  122  or the third data storage devices  332 A- 332 H as appropriate depending upon whether the asynchronous data replication process between the nearby safe site  120  and the disaster recovery sites  330 A- 330 B has been completed. 
   While various embodiments of the present invention have been described in detail, further modifications and adaptations of the invention may occur to those skilled in the art. However, it is to be expressly understood that such modifications and adaptations are within the spirit and scope of the present invention.