Patent Publication Number: US-7711712-B2

Title: System and method for managing consistency among volumes based on application information

Description:
DESCRIPTION OF THE INVENTION 
     1. Field of the Invention 
     This invention generally relates to data storage systems and, more specifically, to data storage systems using Continues Data Protection (CDP) technology. 
     2. Description of the Related Art 
     Continuous Data Protection (CDP) technology protects user data anytime by journaling every write input/output (IO) operation from an application. The journaled IO data is stored in storage, which is independent from the primary storage associated with the host computer system executing the subject application. The modern CDP can record various application events such as checkpoints associated with a database (DB) application, as well as the timing of the execution of the file closing operations. 
     Basic CDP technology is described in U.S. Patent Application No. US20040268067 A1, “Method and apparatus for backup and recovery system using storage based journaling,” which is incorporated herein by reference. The described CDP methodology provides copy on write journaling capabilities and keeps unique sequence number for journal and snapshot. 
     There are also several available commercial CDP products. One exemplary enterprise product is REVIVO CPS 1200i, This product operates to mirror I/O operations on host, and provides an appliance for receiving mirrored I/O operations and storing those operations as journal. The system also indexes the stored information for subsequent restore. 
     Another CDP&#39;s product is able to automatically determine the behavior of the protected application. This product is XOSoft&#39;s Enterprise Rewinder. This product can automatically determine behavior of various applications, such as Microsoft® Exchange®, including application&#39;s checkpoints. 
     U.S. patent application Ser. No. 11/404,190 described system and method for processing a plurality types of event markers of a continuous data protection system. Also, U.S. patent application Ser. No. 11/471,105 describes system and method for managing a consistency among multiple volumes in a continuous data protection system. The disclosure of these patent applications is incorporated by reference herein in their entirety. 
     On the other hand, there are numerous database (DB) applications, which also require data protection. The modern CDP systems can provide data protection for solo DB applications using crash recovery. However, currently there is no technology permitting recovery of multiple DB applications using CDP crash recovery mechanism, which is capable of performing roll forward type recovery for transactions distributed among several database applications. 
     Thus, what is needed is a technology for creating a federated application environment protected by both of CDP and other data protection methods. 
     SUMMARY OF THE INVENTION 
     The inventive methodology is directed to methods and systems that substantially obviate one or more of the above and other problems associated with conventional techniques for continuous data protection. 
     In accordance with one aspect of the inventive methodology, there is provided a method for data recovery. The inventive method involves collecting consistency points from at least two database applications; collecting journal terms associated with continuous data protection for each of the at least two database applications; collecting log terms associated with each of the at least two database applications; displaying to a user a timeline for a protected term and consistency points for the at least two database applications; receiving recovery time selection from a user; receiving recovery database application selection from the user; receiving baseline database application selection from the user; performing crash recovery for each database application; and performing recovery of data based on transactions associated with the at least two database applications. 
     In accordance with another aspect of the inventive methodology, there is provided a computerized data recovery system. The inventive system includes a storage subsystem and a backup server operatively coupled to backup storage medium. The storage subsystem includes a continuous data protection system, while the backup server executes a recovery manager operable to: collect consistency points from at least two database applications executing on at least one host; collect journal terms of the continuous data protection system for each of the at least two database applications; collect log terms associated with each of the at least two database applications; display to a user a timeline for a protected term and consistency points for the at least two database applications; receive recovery time selection from a user; receive recovery database application selection from the user; receive baseline database application selection from the user; perform crash recovery for each database application; and perform recovery of data based on transactions associated with the at least two database applications. 
     In accordance with yet another aspect of the inventive methodology, there is provided a computer-readable medium embodying a set of computer instructions, the instructions, when executed by one or more processors causing the one or more processors to: collect consistency points from at least two database applications; collect journal terms associated with continuous data protection for each of the at least two database applications; collect log terms associated with each of the at least two database applications; display to a user a timeline for a protected term and consistency points for the at least two database applications; receive recovery time selection from a user; receive recovery database application selection from the user; receive baseline database application selection from the user; perform crash recovery for a database application; and perform recovery of data based on transactions associated with the at least two database applications. 
     Additional aspects related to the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Aspects of the invention may be realized and attained by means of the elements and combinations of various elements and aspects particularly pointed out in the following detailed description and the appended claims. 
     It is to be understood that both the foregoing and the following descriptions are exemplary and explanatory only and are not intended to limit the claimed invention or application thereof in any manner whatsoever. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The accompanying drawings, which are incorporated in and constitute a part of this specification exemplify the embodiments of the present invention and, together with the description, serve to explain and illustrate principles of the inventive technique. Specifically: 
         FIG. 1  shows an exemplary diagram illustrating the hardware components and interconnections among components; 
         FIG. 2  illustrates an exemplary logical configuration of one embodiment including software components and illustrating their interoperation; 
         FIG. 3  illustrates an exemplary volume configuration structure  100 ; 
         FIG. 4  illustrates exemplary mapping table between a logical unit (LU) and a volume (a.k.a. LDEV); 
         FIG. 5  illustrates an exemplary table associated with a source database; 
         FIG. 6  illustrates an exemplary procedure for creating a GUI for recovery of a set of data; 
         FIG. 7  illustrates exemplary media manager consistent points; 
         FIG. 8  illustrates exemplary media manager terms; 
         FIG. 9  illustrates an exemplary embodiment of a graphical user interface (GUI). 
         FIG. 10  illustrates an exemplary time based restore operation; 
         FIG. 11  illustrates an exemplary embodiment of a recovery operation; 
         FIG. 12  illustrates an exemplary procedure for selecting a recovery transaction; 
         FIG. 13  illustrates an exemplary procedure for alignment of transactions; 
         FIG. 14  illustrates an exemplary operating sequence of an embodiment of the inventive concept; 
         FIG. 15  illustrates an exemplary embodiment of a mapping information; 
         FIG. 16  illustrates a second exemplary embodiment of the inventive concept; 
         FIG. 17  illustrates an exemplary operating sequence of the second exemplary embodiment of the inventive concept; and 
         FIG. 18  illustrates an exemplary embodiment of a recovery method selection process. 
     
    
    
     DETAILED DESCRIPTION 
     In the following detailed description, reference will be made to the accompanying drawing(s), in which identical functional elements are designated with like numerals. The aforementioned accompanying drawings show by way of illustration, and not by way of limitation, specific embodiments and implementations consistent with principles of the present invention. These implementations are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other implementations may be utilized and that structural changes and/or substitutions of various elements may be made without departing from the scope and spirit of present invention. The following detailed description is, therefore, not to be construed in a limited sense. Additionally, the various embodiments of the invention as described may be implemented in the form of a software running on a general purpose computer, in the form of a specialized hardware, or combination of software and hardware. 
     Three exemplary embodiments of the inventive concept will be described in detail hereinbelow. These three embodiments differ from each other in the aspects of the configuration of the respective implementations. However, the general concept of the present invention remains the same and it is implemented in all the described embodiments. Specifically, all three described embodiments are capable of keeping data consistency among several databases. 
     Embodiment 1 
     Recovery Under CDP&#39;s Single Consistency Group and Other Point in Time Backup Data 
     The first exemplary embodiment described herein uses a CDP&#39;s single consistency group for DB application and other point-in-time data such as mirror copies, snapshots, and tapes. 
     1. Physical Configuration 
       FIG. 1  shows a diagram illustrating various hardware components of the system in accordance with the first exemplary embodiment and interconnections among those hardware components. 
     The aforesaid system includes at least a host operatively coupled to several storage subsystems. Each component will be described in detail below: 
     Specifically, Host  10  and Backup Server  60  incorporate Operating System running on specialized or generic hardware, such as standard workstation or personal computer. 
     The host  10  incorporates CPU  11 , memory (mem)  12 , internal disc  13  and NIC  15  (Network interface card). 
     The host  10  includes Host Bus Adapter (HBA)  14  provided for connecting the host  10  to a generic FC (Fibre Channel) switch, a generic Ethernet switch  80  or a directly attached storage subsystem. 
     The backup server also includes Host Bus Adapter (HBA)  14  to enable connection to a generic FC switch, a generic Ethernet switch  80  or a directly attached tape library  70 . 
     Each host stores the associated data on a Logical Unit (LU) provided by the storage subsystem. 
     Storage Subsystem  30  has the features and capabilities described below: 
     Specifically, the storage subsystem has capabilities to store data using SCSI-2 or SCSI-3 command sets within the associated LUs. In addition, the storage subsystem may incorporate several RAID controllers (CTL)  20  and several Discs  31 . 
     The RAID controller  20  includes one or more processors, memory, NIC such as Ethernet and an FC port, which is accessible from SAN or to Discs to process SCSI I/O operations and consists of RAID configuration using several Discs residing in the storage subsystem (Not depicted in figures). 
     The controller preferably includes non-volatile random access memory (NVRAM) and is capable of storing data to the aforesaid NVRAM, utilizing it as a cache memory having a protection from a power failure. 
     The controller additionally provides ports capable of being identified by WWN (World Wide Name), which specifies the target ID as SCSI world from the host, and which consists of a logical unit number (LUN) on a Fibre Channel (FC) port. 
     The storage subsystem may include an internal management console  23 , which is connected to the storage subsystem internally and is accessible from any generic console, such as a generic web-based PC or a workstation provided for managing the storage subsystem. The console (not depicted in  FIG. 1 ) of the storage administrator may be located externally and can be accessible via a generic IP protocol enabled network switch, such Ethernet hub, switch and/or IP router. 
     A Tape Library  70  is equipped with SCSI-3 media changer command set or equivalent protocol/API to change media and may include several Tape drives for accessing data on tapes. The tapes could be loaded into the Tape Library  70  using one of several slots provided therein. 
     2. Logical Configuration 
       FIG. 2  illustrates logical configuration of the system of the first embodiment, including various software components and their interoperation. Each shown component will be discussed in detail below. 
     LAN  83 : LAN (Local Area Network) provides communicating among hosts and backup server. The communication will use TCP/IP protocol. LAN consists of Ethernet switch/Hub, FDDI, Token-ring, ATM or other kind of IP transferable network. 
     SAN  82 : SAN (Storage Area Network) provides block access capable logical network connection, such as SCSI, FC-SCSI (Fibre Channel SCSI), ESCON or other. In describing this embodiment, for purposes of simplifying the description, FC-SCSI will mainly be used. The SAN provides a logical connection between the Host and the Storage Subsystem via a port. This connectivity is provided mainly by the fibre channel (FC) switch or hub, or, alternatively, by Ethernet Switch or hub or the like. 
     Host  10 : Host  10  incorporates OS (Not depicted), database (DB)  16 , DB application  15 , SCSI driver (Not depicted), Raid Manager (RM)  18  and backup agent (Agent)  17 . The OS may be UNIX, Microsoft Windows, Solaris, Z/OS, AIX, Linux or any other kind of known or later developed operating system. 
     The application may be a transaction type application, which uses the DB. DB may include an online log as well as datafiles for tablespaces stored on storage subsystem. Archive log may be stored on the host or within another storage subsystem. 
     To control the mirror or CDP operation, the host  20  may incorporate raid manager (RM), provided as an in-band control mechanism. The RM communicates with the storage subsystem. With respect to the communication method with the storage subsystem, an embodiment of the inventive system utilizes a technology, which controls the storage device using SCSI command sets. The aforesaid technology for controlling the storage subsystem  30  using SCSI commands is described in detail in European patent application No. EP1246050, which is incorporated herein by reference in its entirety. It should be noted that the agent  19  of the embodiment of the inventive system shown in  FIG. 2  (RM) is generally equivalent to the RMLIB, while the command device  27  (CMD DEV) is generally equivalent to the CM described in the EP1246050. 
     An exemplary implementation of command line interface (CLI) operable to control CDP disposed within a storage system is illustrated in  FIG. 3  of the aforesaid U.S. patent application Ser. No. 11/404,190, incorporated by reference herein. 
     Backup agent  17  is operable to extract backup data/restore data to/from the backup server providing for data consistency in the DB application which stores the associated data in DB and may also store it in the filessystem. 
     Storage subsystem: In one embodiment, the modules of storage subsystem are enabled in microcode which is executed on the controller (CTL) and is provided as a program code installing form optical media, FD, and other removal devices. The microcode consists of parity group manager, volume manager (VOL Manager) that creates a volume to provide a volume to host and journal manager, which will be discussed hereinbelow. Each of the aforesaid microcode modules will be described in detail below. 
     Parity group manager (not depicted in  FIG. 2 ): This module is a part of microcode and may consist of a parity group including discs utilizing RAID0/1/2/3/4/5/6 technology. RAID 6 is based on the RAID 5 technology, well known to persons of skill in the art, but including a dual parity protection. The created parity group is listed in VOL Config structure  100 , shown in  FIG. 3 , and is provided with parity group number  101  to identify the parity group within the storage subsystem, usable capacity size  102  created from the RAID technology, RAID configuration  103 , and the utilized Discs  104  which are identify from discs  31 . 
     Volume Manager (Vol Manager) (not depicted in  FIG. 2 ): The Volume Manager manages volume&#39;s structure and the behavior with respect to I/O operations directed to the LUs. The volume is a portion of a parity group which exports toward LU to read and write data issued by the host. Administrator defines and initially formats the region of the volume added the volume number by volume manager. The mapping between the volume and the parity group is stored in the VOL Config  100 , shown in  FIG. 3 . On a parity group number  101 , a record in VOL Config has volume number  105  to identify the logical device (LDEV, a.k.a volume) within the storage subsystem, start Logical Block Address (LBA)  106  representing volume&#39;s start address on the parity group, end LBA  107  presenting volume, address on the parity group, and size  108  to present the size of VOL. 
     Port  22  (a.k.a  111 ): Ports enable the associated volumes to be accessed from SAN via logical unit (LU) on a WWN.  FIG. 4  shows an exemplary mapping table between an LU and a volume. Each value in the hardware port  111  column of the table shown in  FIG. 4  corresponds to one of the ports shown in  FIG. 1  or  FIG. 2 . Each port has its own WWN  112 , which is specified from host to access LUs via port. A plurality of different LUs can be assigned to the port. An LU is specified by a pair of WWN in column  112  and LUN in column  113 . Based on the FC-SCSI protocol specifications, the maximum number of logical units that can be associated with a port is  256 . Further, each LU is mapped to a volume, which is used to store data from the hosts  10 . Based on this mapping information, the controller  20  receives SCSI commands from the port and converts the set of WWN of column  112  and LUN of column  113  into the volume identified of column  114  to access the appropriate volumes. Also stored in the table shown in  FIG. 4  is a command device (CMD DEV) entry  115  used to control the storage subsystem from host&#39;s RM (Raid Manager)  18 . 
     With respect to each LU, storage system administrator can configure LU as a Virtual LU to make the LU visible to the host even if the LU does not have the corresponding LDEV or the restore image. When the administrator designates LU as a Virtual LU, the storage subsystem turns on the VLU bit in column  116 . In this mode, storage subsystem always makes the LU available to host regardless of assignment of LDEV to the LU. 
     Mirror Manager (not depicted in  FIG. 2 ): The Mirror Manager controls mirror operations based on the RM commands. The mirror operations include “create pair” operation between P-VOL and S-VOL, as well as “resync”, “split”, “reverse resync”, and “delete pair” operations. After the execution of the split after resync operation, a point-in-time (PIT) image volumes can be created using mirror or snapshot technology. 
     JNL Manager  25 : The JNL Manager obtains I/O journal stored on the JNL volumes  33  and  35  for P-VOL and stores marker information on the JNL, thereby creating a server specified point in time. The JNL Manager also creates an I/O consistency group and creates I/O serialization among volumes. During the restore operation, volumes  38  and  39  are restored, which store a copy of the base volumes  34 ,  36  by applying journal data up to time or up to marker. After the restore, each restore volume is attached to the LU of the P-VOL instead of the P-VOL. In an embodiment of the invention, the restore volume may be attached on another LU, which is exported to another host. In this configuration, a user needs to configure the host  10  in a similar manner in order to check data on restored volumes. The behavior of the Journal Manager  25  is the same as the operation of the JNL manager  24 , described in the aforesaid U.S. patent application Ser. No. 11/404,190. 
     Backup Server  60 : The Backup Server  60  incorporates of an OS (not depicted in  FIG. 2 ), Backup Software ( 61 ), HBA driver (not depicted in  FIG. 2 ). The Backup Software includes Media Manager (Media)  62  provided for managing the existing consistent backup data, and Recovery Manager ( 64 ) for restoring and recovering a set of DB. The Consistent  65  (also referred to as consistency manager) module of the Recovery Manager  64  makes an alignment of transactions among several DB applications. Media Manager  62  manages the consistent backup data, which are in a form of point-in-time images created by mirror&#39;s resync and split operation after the pair or snapshots, Tape or CDP&#39;s marker after creating application consistency point using application&#39;s hot and cold backup, as well as date and time. The Backup Software  61  is operable to communicate with the backup agent  17  to make a consistent state for application data, and to perform backup of the data via the LAN, and restore of the data via the LAN. The backup agent may work with Microsoft&#39;s Volume Shadow Copy Service (VSS) to make a consistent state for application data as one of examples. 
     3. Operations 
     On the backup server, user executes recovery operation for a set of DB applications. Before the details of the operation will be discussed, an initial setup and transaction operations for DB and DB application as dependent environment will be described. 
     3.1 Initial Setup 
     First, storage administrator creates LU for selected DB applications. The created LUs are configured to be protected by the CDP on storage subsystem. Host administrator then executes procedure to discover the created LUs and then creates the corresponding device files in the operating system environment. After that, the DB administrator creates DB data, log and related data sets on the created LUs. DB administrator may also create DB table data as data files on filesystem after the host administrator creates a filesystem on the LUs. DB application administrator runs the DB application on the DB using JDBC, ODBC or other types of DB access method. 
     3.2 Transactions in DB Application 
     To find the last transaction for other DB applications, a DB application leaves a record of the last transaction for other communicated DB&#39;s application after their transaction, which is communicated with other DB&#39;s application. 
     After commit of transaction, target DB&#39;s application gets a current active transaction ID like XID in Oracle-type database and System Change Number (SCN) for the transaction using V$TRANSACTION information in Oracle-type Database and store the information on source DB&#39;s table as last transaction for the DB application. Other DB vendors have similar information for the active transactions. 
       FIG. 5  illustrates an exemplary embodiment of a table  120  in DB 1 , which acts as a source DB. The shown table includes a communicated Host IP address  121  from source Host ( 125 ), application name  122 , Last updated Transaction ID ( 123 ), System change number (SCN) ( 124 ) and Completion Recovery ( 126 ) to indicate processed recovery operation using “Yes” as finish or “No” as unprocessed. Other databases, including DB 2 , also include tables  130  that are similar to the table  120  of DB 1 . 
     3.3. GUI for Recovery 
     When a failure occurs, which affects several DBs, it is necessary to restore and recover the DB data. Recovery manager provides a GUI for this purpose. During the recovery operation, the recovery manager creates a GUI for recovering a set of data among several DB applications. The corresponding procedure is described below and illustrated in  FIG. 6 . 
     The procedure starts with step  141 . 
     Step  141 : Recovery manager collects consistency points of several DB applications. The collected consistency points may include Mirrored or snapshot Point-In-Time image, as well as tape and CDP&#39;s markers from the media manager  62 . 
     The consistency point of DB applications includes backup data which may be created using either hot backup or cold backup procedure after the backup software coordinated with application via the backup agent. 
     As an example of media manager&#39;s consistent points, there is a table  150  in  FIG. 7 . The table consists of backup ID  151  serialized by backup software, data  152 , time  153  taken backup, application name  154  taken backup, Method of keeping consistent point  155  which are tape, PIT Mirror, PIT snapshot, CDP marker, and etc, Device number or identifier of device  156  like serial number of device, Media number  157  which is tape ID, LDEV number on storage subsystem, or marker number. 
     Step  142 : Recovery manager collects terms of CDP&#39;s journal terms for each CDP protected application accessing journal&#39;s first and end header information via RM or out-band I/F via LAN. 
     As an example of media manager&#39;s terms of those, there is a table  160  in  FIG. 8 . The table consists of DB&#39;s application name  161 , start  164  and end  165  data &amp; time in CDP journal. 
     Step  143 : Recovery manager collects terms of DB&#39;s logs (a.k.a. redo log and archive log) for each DB&#39;s application accessing DB&#39;s SQL I/F. 
     As an example of media manager&#39;s terms of those, there is a table  160  in  FIG. 8 . The table consists of DB&#39;s application name  161 , start  162  and end  163  data &amp; time in archive log. 
     Step  144 : Recovery manager shows timeline for protected term and consistency points on all of application (Discuss in section 3.3.1). 
     Step  145 : User selects a recovery time considering “consistency point based recovery” or “time based recovery” as restore, and then user selects target DB&#39;s applications to restore data and specifies baseline DB&#39;s application for this recovery (Discuss in section 3.3.1). 
     Step  146 : Recovery manager initiates CDP to restore data for requested time. If the restore is consistency point based, then recovery manager restores last consistency point data before the selected recovery time for CDP protected volumes. If the restore is time based, then recovery manager restores data based on the specified time. 
     Step  147 : Consistency Manager in Recovery manager initiates to execute recovery data considering transaction among DB&#39;s applications. Whereupon, the procedure terminates. 
     3.3.1 GUI 
     At Step  144 , recovery manager creates recovery GUI. An exemplary embodiment of the GUI  170  is shown in  FIG. 9 . 
     GUI  170  consists of terms of DB&#39;s Logs (which is online log and archive log) and CDP and consistent point indicated by small circle for each DB application on each timeline. User selects target recovery DB&#39;s applications using check box  175 , selects a baseline application using option button ( 176 ) and then select restore time using dash bar ( 174 ). The dash bar which indicates user wanted restore point among multiple DB&#39;s applications can slide only between time of earliest DB&#39;s log or CDP journal and one of latest DB&#39;s log or CDP journal. 
     As a recovery option, a user may choose a DB mountable restore method for data which are time based restore or nearest consistent point ( 177 ) for each of the DB&#39;s application volumes. 
     In case of time based restore ( FIG. 10 ), recovery manager  64  initiates CDP to restore data by time; Recovery manager restores data by time using CDP ( 182 ) and executes crash recovery for DB ( 183 ) and if LU is protected by CDP and the recovery time is between the times of the first and the last entries on CDP&#39;s JNL ( 181 ). If there are several applications under a consistency group, recovery manager initiates CDP to restore a set of volumes at a time using restore a set of volumes under the consistency group by time. To find a mapping between application and volume, recovery manager discoveries the mapping information via backup agent and RM. The mapping information  200  ( FIG. 15 ) consists of host address  226 , application name  227 , device files within OS  228 , port  229  on storage subsystem and LU ( 230 ). And to find whether the application on a consistency group or not, storage subsystem provides consistency group information referring table, which is illustrated in  FIG. 6  of the aforesaid U.S. patent application Ser. No. 11/404,190, incorporated by reference herein. This table can be used to find out which LUs are on a consistency group. Recovery manager retrieves this information from storage subsystem and uses this information. 
     The recovery manager initiates the restore data operation ( 185 ), which has a consistent image using each restore method like CDP&#39;s marker restore and SI/QS&#39;s reverse restore and other. The volumes for a DB may be protected by several types of copy technology, such as CDP with SI, QS, or using any other disk base backup method. In that case, we may choose a faster DB&#39;s recovery method. For example, we assume that applying time of DB&#39;s log and JNL&#39;s log is same and that copy rate from CDP&#39;s baseline volume to primary volume and one from SI, QS or other are same. The difference is differential copy of CDP from baseline to primary volume before applying JNL and one of SI from secondary volume to primary volume due to be difference of amount between CDP and SI The copy amount of differential copy is small near current time and is large far from current time. In CDP case, the amount of differential copy is small when CDP&#39;s baseline volume is located near from current like before 12 hours. In SI case, the amount of it is large when the elapsed time of creation of SI&#39;s secondary volume is past from CDP baseline. In this situation,  FIG. 18  shows an example of selection of recovery method. 
     Step  291 : Recovery Manager checks whether Recovery time is within CDP JNL term or not. If it is Yes, the procedure goes to Step  292 . If it is No, the procedure goes to Step  294 . 
     Step  292 : Recovery Manager checks whether elapsed time of SI/QS&#39;s PIT is without CDP JNL term or not. If it is yes, the procedure goes Step  293 , If it is no, the procedure goes to Step  294 . 
     Step  293 : Recovery manager chooses CDP as recovery and terminate their process. 
     Step  294 : Recovery manager chooses PIT&#39;s restore with DB&#39;s Log as recovery and terminate their process. 
     Based on this process, we can say following. In case that recovery time is within CDP JNL&#39;s term and elapsed time for SI&#39;s or QS&#39;s PIT is within CDP, recovery manager chooses recovery of SI, or QS&#39;s PIT with DB&#39;s log because the amount of differential copy on SI or QS&#39;s PIT with DB&#39;s log recovery is smaller than one on CDP recovery. In case that recovery time is within CDP JNL&#39;s term and elapsed time for SI or QS is without CDP, recovery manager chooses CDP recovery because the amount of differential copy is smaller than SI/QS&#39;s one. In case that recovery time is without CDP JNL&#39;s term and elapsed time for SI or QS is within CDP, recovery manager chooses to recovery SI, or QS&#39;s PIT with DB&#39;s log. In case that recovery time is without CDP JNL&#39;s term and elapsed time for SI or QS is within CDP, recovery manager chooses SI, or QS&#39;s PIT with DB&#39;s log recovery; Therefore, recovery manager may have a capability to select recovery method based on recovery time and a relation between a position CDP baseline volume and one of SI/QS PIT. If recovery manager uses other PIT technology or tape instead of SI/QS, recovery manager needs consider amount of data copy to estimate faster recovery time based on copy and applying DB&#39;s log. 
     In case of nearest consistent point restore, recovery manager restores data which is near consistent point from user specified time; CDP restores data by latest marker. Recovery manager restores data to the latest consistent point like tape, PIT mirror, or snapshot. Also, volumes for a DB may be protected by several types of copy technology, including CDP with SI, QS, or using any other disk base backup method. In that case, we may choose a faster DB&#39;s recovery method as well, as in Step  185 . 
     Those two options can work in Step  146 . 
     3.3.2 Recovery Operation 
     After the DB mountable restore operation in Step  146 , it is necessary to perform an alignment of multiple transactions in Step  147 , as shown in  FIG. 11 . The procedure is following. 
     The procedure starts with step  191 . 
     Step  191 : Consistency Manager in recovery manager gets a baseline DB&#39;s application which user specified in restore operation. 
     Step  192 : Consistency Manager retrieves transaction information  121 ,  122 ,  123 ,  124  for source DB ( 125 )&#39;s application related DB&#39;s application from user specified baseline DB&#39;s application. 
     Step  193 : Consistency Manager selects a un-processed DB&#39;s application which is “No” indicated in  126  from the listed DB&#39;s applications. 
     Step  194 : Consistency Manager checks if there is un-processed DB&#39;s application in the list; all of entries in the list  120  and  130  were processed. If there is (Y), it goes to Step  195 . If there is no (N), it goes to end of procedure. 
     Step  195 : Consistency Manager selects transaction ID on recovery for the target DB&#39;s application (Detail is discussed in Section 3.3.2.1). 
     Step  196 : Consistency Manager checks current transaction ID on the DB&#39;s application. If the ID is the same as source DB&#39;s application recorded information (Y), the process goes to Step  199 . If not, the Consistency Manager goes to Step  197 . And we may use SCN instead of transaction ID when performing recovery for autonomy distributed database like Oracle. 
     Step  197 : Consistency Manager initiates DB to execute alignment of transactions (Detail is discussed in Section 3.3.2.2). 
     Step  198 : Consistency Manager executes same process for other data base to recover other database using this procedure as sub-routine after collecting related DB applications&#39; information from aligned DB&#39;s application. When a Consistency Manager finish to recovery the selected database, consistency manager marks “Yes” in  126  to indicate finish of recovery for the selected database. End of the procedure. 
     3.3.2.1 Selecting a Recovery Transaction 
     To select a recovery transaction ID, Consistency Manager retrieves transaction ID based on location of application. The procedure is illustrated in  FIG. 12  and described in detail below. 
     Step  201 : Consistency Manager checks if the recovery target DB&#39;s application is user specified baseline application or not. If it is, the procedure goes to Step  202 . If it is not, the procedure goes to Step  203 . 
     Step  202 : Consistency Manager uses transaction ID for target DB&#39;s application retrieved from the baseline application&#39;s one. 
     Step  203 : Consistency Manager checks if the target DB&#39;s application is listed on baseline DB&#39;s application&#39;s list. If it is, there is a relation between baseline DB&#39;s application and target DB&#39;s application, and the procedure proceeds to Step  202 . If it is not, there is no such relation and the procedure proceeds to Step  204 . 
     Step  204 : Consistency Manager uses transaction ID for target application on target DB&#39;s application because Consistency Manager determines the target application to be independent from the baseline application. 
     Using this procedure, Consistency Manager uses transaction ID on baseline&#39;s application if there are linkages between baseline application and other application. If there is no, they uses own transaction ID for current target recovering application. 
     And we may use SCN instead of transaction ID when we recovery for autonomy distributed database like Oracle. 
     3.3.2.2 Executing Alignment of Transactions 
     To make an alignment of transactions, Consistency Manager executes rollback and roll forward operation on target DB&#39;s application if necessary for target DB. 
     The details of this procedure are described below and illustrated in  FIG. 13 . 
     Step  221 : Consistency manager checks if current transaction ID on database table is below of a selected transaction in section 3.3.2.1. If it is, the procedure proceeds with Step  223 . If it is not, the procedure goes to Step  222 . 
     Step  222 : Consistency manager executes roll forward operation up to next SCN for target DB. 
     Step  223 : Consistency manager executes roll back operation based on selected transaction ID number. 
     Step  224 : Consistency Manager checks the result. If there is any error for the rollback operation, Consistency Manager returns error (Step  225 ). If there is not, the procedure goes to end of procedure. Whereupon, the procedure terminates. 
     In case of autonomy distributed database environment like oracle, we may use recovered baseline DB&#39;s SCN on each host&#39;s change number recovery instead of procedure  FIG. 13  as alignment of transactions. 
       FIG. 14  illustrates an exemplary operation sequence of the described embodiment. 
     CDP protects DB 1  and DB 2  volumes under single consistency group. There is PIT image created by mirror technology for DB 3 . First, the user specified a recovery time, such as 8:30. Thereafter, the recovery manager restores DB 1  and DB 2 &#39;s image using CDP. Recovery manager initiates crash recovery for DB 1  and DB 2  ( 1 . a  and  2 . a  in  FIG. 14 ). The recovery manager restores DB 3 &#39;s PIT image using mirror or using a swap from primary volume to secondary volume and initiates an operation whereby DB performs roll forward up to user specified restore time ( 3 . a ). 
     It should be noted that storage and Host each have their own time clocks. Therefore, there may be differences in the recovered time. 
     After the crash recovery for CDP protected volumes, Consistency Manager in recovery manager executes an operation to make an alignment of transaction among DBs. DB 1  is the baseline. Consistency Manager checks DB 2 &#39;s last transaction ID and DB 2  is rollback to a specified transaction ID ( 2 . b ). Also, DB 1  checks DB 3 &#39;s last transaction ID on DB 1 , if there is a communication capability. Consistency Manager initiates a procedure whereby DB 3  executes roll forward ( 3 . a ) and rollback ( 3 . b ) using the procedure of  FIG. 11 . 
     In this embodiment, we assume the JNL manager is located on storage subsystem. Instead of storage subsystem, we may run JNL manager on switch based virtualization hardware like CISCO&#39;s MDS Series and EMC&#39;s invista, PC appliance based CDP protection hardware like Revivo&#39;s CPS series, software based CDP like CA&#39;s EnterpriseRewinder. 
     Embodiment 2 
     Recovery Under CDP&#39;s Several Consistency Group and Other Point in Time Backup Data 
     This embodiment uses the system configuration having multiple JNL groups on storage subsystems configured to protect data. 
     The physical system configuration of the second embodiment is similar to the one of the first embodiment. Logical configuration of the system of the second embodiment is similar to the logical configuration shown in  FIG. 2 . The difference is that each primary volumes is protected by a separate CDP journal manager, as shown in FIG.  16 . Therefore, each CDP journal manager has own  10  serialization; each CDP journal manager serializes IOs for each volume on each JNL volume and it restore data from each serialized journal by specified point time or marker. 
     On restore and recovery, Recovery Manager restores and Consistency Manager recovers data by user specified time on baseline application&#39;s CDP protected volume under a consistency group. However regarding other volumes including other CDP&#39;s consistency groups, Consistency Manager uses last consistency point&#39;s backup data before the user specified time and executes roll forward up to the specified time using capability. 
     Therefore time-based recovery operation in GUI has one step added, as shown in  FIG. 17 , to enable crash recovery for only baseline application&#39;s volumes. 
     Specifically, we have added a step for checking if the target application is baseline under CDP journal volume or not as Step  188 . If it is, the procedure goes to Step  182 . If it is not, goes to Step  184  to restore last consistent point&#39;s marker before user specified time. 
     Other operations performed by the system of the second exemplary embodiment of the inventive concept are the same as the corresponding procedures of the first embodiment. 
     Embodiment 3 
     Recovery Having Consistency Between DB and File Using Version Information for DB 
     Aforesaid U.S. patent application Ser. No. 11/471,105 discloses a technique for achieving a consistency between DB and file. This technique is disclosed in the third embodiment described therein, the description of which is incorporated by reference herein in its entirety. Specifically, the version information for a file, which is related to a DB is stored in the DB. On the other hand, the exemplary embodiments of the present invention create a consistency among multiple DBs on recovery. However the key concept is the same as the aforesaid third embodiment described in U.S. patent application Ser. No. 11/471,105. Specifically, in the present invention, each DB stores transaction ID communicated with other DBs instead of storing the version information. 
     Moreover, other implementations of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. Various aspects and/or components of the described embodiments may be used singly or in any combination in the computerized storage system with data protection functionality. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.