Patent Publication Number: US-9886359-B2

Title: Redundant system, redundancy method, and computer-readable recording medium

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2014-127730, filed on Jun. 20, 2014, the entire contents of which are incorporated herein by reference. 
     FIELD 
     The embodiments discussed herein are related to a redundant system, a redundancy method, and a redundancy program. 
     BACKGROUND 
     It is common that various types of nodes such as a web server, an application server, and a DB (DataBase) server are installed in a data center and each node is made redundant for disasters and failures. 
     For example, there is known a redundancy technique in which, for a node in a data center, a node in a primary system and a node in a secondary system are prepared, and when the node in the primary system has broken down, the node in the secondary system takes over the processes of the broken-down node in the primary system to continue the processes, instead of the broken-down node in the primary system. 
     In addition, there is known a technique in which a data center in a secondary system for a backup to a data center in a primary system is provided, and when the data center in the primary system has experienced a disaster, the data center in the secondary system for a backup takes over processes, by which processes performed by the data center in the primary system continue. 
     [Patent Document 1] Japanese Laid-open Patent Publication No. 2008-134986 
     However, when a node that transfers logs from the data center in the primary system to the data center in the secondary system has broken down, logs are not transferred to the data center in the secondary system until the node recovers. When the data center in the primary system has experienced a disaster during a period before the broken-down node recovers, data lost occurs. 
     To inhibit data lost upon the occurrence of both of a breakdown in the node and a data center&#39;s experience of a disaster, it is considered to provide a plurality of paths through which logs are transferred between the data centers. However, in that case, a node that receives logs through two paths is present in the data center in the secondary system. In this case, unless the node that receives logs through the two paths appropriately selects which one of the logs through the two paths is to be reflected, an increase in the amount of data lost is caused. 
     SUMMARY 
     According to an aspect of the embodiments, a redundant system includes: a primary system including: a first node; and a second node that backs up the first node; and a secondary system including: a third node; and a fourth node that backs up the third node. The first node in the primary system includes: a processor that executes a first process including inserting, when transmitting data update information to the second node and the third node, one or a plurality of pieces of delimiter information into both of transmit data, the data update information being generated in response to a data update in the first node, and the delimiter information indicating a boundary between update processing units. The fourth node in the secondary system includes: a processor that executes a second process including: specifying, based on the delimiter information, the data update information including update information whose process has progressed further from among the data update information obtained from the second node and the data update information obtained through the third node, and reflecting the specified update information to stored data of the fourth node. 
     The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  is a diagram illustrating an exemplary overall configuration of a redundant system according to a first embodiment; 
         FIG. 2  is a functional block diagram illustrating the functional configurations of nodes in a primary center; 
         FIG. 3  is a diagram illustrating an example of a user log; 
         FIG. 4  is a diagram illustrating an example of a control log; 
         FIG. 5  is a diagram illustrating an example of a recovery point log; 
         FIG. 6  is a diagram illustrating an example of an update file transmitted by communication between systems; 
         FIG. 7  is a functional block diagram illustrating the functional configurations of nodes in a secondary center; 
         FIG. 8  is a flowchart illustrating the flow of a notification process from a primary master node to a primary mirror node; 
         FIG. 9  is a flowchart illustrating the flow of a notification process from a primary master node to a secondary master node; 
         FIG. 10  is a flowchart illustrating the flow of an update process performed by the primary mirror node; 
         FIG. 11  is a flowchart illustrating the flow of a notification process performed by the primary mirror node; 
         FIG. 12  is a flowchart illustrating the flow of update and notification processes performed by a secondary master node; 
         FIG. 13  is a flowchart illustrating the flow of an update process performed by the secondary mirror node; 
         FIG. 14  is a diagram describing an example of the occurrence of a failure in a redundant system according to a second embodiment; 
         FIG. 15  is a diagram describing an example of system switching of the redundant system according to the second embodiment; 
         FIG. 16  is a flowchart illustrating the flow of an update process performed by a secondary mirror node according to the second embodiment; 
         FIG. 17  is a flowchart illustrating the flow of a system switching process performed by the secondary mirror node according to the second embodiment; and 
         FIG. 18  is a diagram describing an exemplary hardware configuration. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     Preferred embodiments of the present invention will be explained with reference to accompanying drawings. Note that the invention is not limited by the embodiments. 
     [a] First Embodiment 
     Exemplary Overall Configuration 
       FIG. 1  is a diagram illustrating an exemplary overall configuration of a redundant system according to a first embodiment. As illustrated in  FIG. 1 , the system is a redundant system in which a data center is mirrored by performing a DB quad-redundancy function, i.e., a DB quadruplication function, and includes a primary center  1  and a secondary center  5  which are data centers. 
     The primary center  1  is a data center including a primary master node  10  and a primary mirror node  20 , and has a redundant configuration for performing DB mirroring. Likewise, the secondary center  5  is a data center including a secondary master node  50  and a secondary mirror node  60 , and has a redundant configuration for performing DB mirroring. The secondary center  5  functions as a backup to the primary center  1 . Note that each node is an example of a DB server, a storage system, etc. 
     The primary master node  10  is an example of a first node having an active DB  12  to which updates by business operations are made, and is activated as a primary node at normal operation. When the primary master node  10  updates the active DB  12  by a business operation application, etc., the primary master node  10  extracts update information indicating, for example, differences between before and after the update. For example, the primary master node  10  transmits an update log indicating an updated content, to the primary mirror node  20  in synchronization with the update to the active DB  12 . In addition, the primary master node  10  creates an update file including a plurality of update logs, and transmits the update file to the secondary master node  50  at predetermined intervals. 
     The primary mirror node  20  is an example of a second node having a standby DB  22  which is updated in synchronization with the active DB  12 , and functions as a backup to the primary master node  10  at normal operation. When the primary mirror node  20  receives an update log as update information from the primary master node  10 , the primary mirror node  20  updates the standby DB  22  using the received update log. Thereafter, the primary mirror node  20  creates an update file including update logs received from the primary master node  10 , and transmits the update file to the secondary mirror node  60  at predetermined intervals. 
     The secondary master node  50  is an example of a third node having a standby DB  52  that stores information equivalent to the active DB  12 . At normal operation, the secondary master node  50  functions as a master node in a secondary system, as primary center  1 &#39;s measures against disasters, etc. When the secondary master node  50  receives an update file as update information from the primary master node  10 , the secondary master node  50  extracts update logs from the received update file and updates the standby DB  52  using the extracted update logs. Thereafter, the secondary master node  50  creates an update file including the plurality of update logs received from the primary master node  10 , and transmits the update file to the secondary mirror node  60  at predetermined intervals. 
     The secondary mirror node  60  is an example of a fourth node having a standby DB  62  that stores information equivalent to the active DB  12 . At normal operation, the secondary mirror node  60  functions as a mirror node in the secondary system, as primary center  1 &#39;s measures against disasters, etc. The secondary mirror node  60  receives an update file as update information from the primary mirror node  20  and receives an update log from the secondary master node  50 . Then, the secondary mirror node  60  updates the standby DB  62  using either one of the pieces of received update information. 
     In such a state, when the primary master node  10  transmits data update information which is generated in response to a data update in the primary master node  10  to the primary mirror node  20  and the secondary master node  50 , the primary master node  10  inserts one or a plurality of pieces of delimiter information indicating a boundary between update processing units, into both of transmit data. 
     Then, the secondary mirror node  60  identifies, based on the delimiter information, update information including update information whose processes have progressed further from among update information obtained from the primary mirror node  20  and update information obtained through the secondary master node  50 , and reflects the update information in stored data of the standby DB  62 . 
     That is, the secondary mirror node  60  receives update information transmitted from the primary master node  10  from both of the primary mirror node  20  and the secondary master node  50 , and reflects further progressed update information in the DB  62 , according to a checkpoint inserted into the update information. Therefore, the secondary mirror node  60  can suppress an increase in data lost in reflection of update information. 
     Functional Configurations of the Nodes 
     Next, the functional configurations of the nodes illustrated in  FIG. 1  will be described. Although here, as an example, functional configurations in the state of  FIG. 1  will be described, the functional configurations are not limited thereto, and the nodes can also have the same functional configuration. 
     Functional Configuration of the Primary Center 
       FIG. 2  is a functional block diagram illustrating the functional configurations of the nodes in the primary center. Here, the primary master node  10  and the primary mirror node  20  included in the primary center  1  will be described. 
     Functional Configuration of the Primary Master Node 
     As illustrated in  FIG. 2 , the primary master node  10  includes a communication control unit  11 , a DB  12 , and a control unit  13 . 
     The communication control unit  11  is a processor that controls communication with the primary mirror node  20  and with the secondary master node  50 , and is a network interface, for example. For example, the communication control unit  11  transmits update information of the DB  12  to the primary mirror node  20  and the secondary master node  50 . 
     The DB  12  is a database that stores business operation information, etc., and corresponds to the active DB  12  illustrated in  FIG. 1 . Updates by business operations are made to the DB  12 . The DB  12  is provided in a storage apparatus, e.g., a hard disk. The DB  12  corresponds to the active DB  12 . 
     The control unit  13  is a processor that controls the whole process of the primary master node  10 , and is an example of a processor, for example. The control unit  13  performs the function of implementing a DB redundant system between the primary master node  10  and the primary mirror node  20 , and performs the function of implementing a DB redundant system between the primary master node  10  and the secondary master node  50 . Namely, the control unit  13  executes an application that implements a DB dual-redundancy function in the primary center  1 , and an application that implements DB quad-redundancy function across the centers. 
     The control unit  13  includes a DB updating unit  14 , an intra-center notifying unit  15 , an inserting unit  16 , and a center-to-center notifying unit  17 . These processors are an example of processes performed by an electronic circuit included in the processor or by the processor. 
     The DB updating unit  14  is a processor that updates the DB  12 . For example, the DB updating unit  14  updates stored data in the DB  12  along with execution of an application, for example. 
     The intra-center notifying unit  15  is a processor that transmits update information of the DB  12  to the primary mirror node  20  in the same system, in synchronization with an update to the DB  12 . Specifically, when the DB  12  is updated, the intra-center notifying unit  15  extracts a difference from information obtained before and after the update. Then, the intra-center notifying unit  15  transmits an update log indicating differential information, as update information to the primary mirror node  20 . 
     Now, an example of an update log will be described.  FIG. 3  is a diagram illustrating an example of a user log.  FIG. 4  is a diagram illustrating an example of a control log. As illustrated in  FIG. 3 , the user log which is an example of an update log is a log indicating DB update information and consists of “a header, a user log display, a variable-length part, a variable-length part  2 , and BC key information”. 
     In the “header”, information indicating an update log, a creation date and time, etc., are set. In the “user log display”, information indicating that it is a user log is set. The “variable-length part” and the “variable-length part  2 ” are information indicating a DB update content, e.g., a specific record position, data before and after the update, and differential information. In the “BC key information”, information about DB dual-redundancy between the primary master node  10  and the primary mirror node  20  is set. For example, checksum information or the serial number of the log is set. 
     As illustrated in  FIG. 4 , the control log which is an example of an update log is a log indicating a DB control process, e.g., a rollback process, and consists of “a header, a control log display, and COMMIT specification”. In the “header”, information indicating an update log, a creation date and time, etc., are set. In the “control log display”, information indicating that it is a control log is set. In the “COMMIT specification”, information indicating a specific control process, e.g., transaction information, is set. 
     As described above, when the DB  12  is updated, the intra-center notifying unit  15  creates an update log, such as the above-described user log or control log, according to updated information. Then, the intra-center notifying unit  15  transmits the created update log to the primary mirror node  20 . In addition, the intra-center notifying unit  15  notifies the center-to-center notifying unit  17  of the created update log. Namely, the intra-center notifying unit  15  notifies of DB update information in synchronization with the update to the DB  12 , within the same center. 
     The inserting unit  16  is a processor that inserts, when update information generated in response to a DB data update in the primary master node  10  is transmitted to the primary mirror node  20  and the secondary master node  50 , one or a plurality of pieces of delimiter information indicating a boundary between update processing units, into both of transmit data. 
     Specifically, the inserting unit  16  periodically generates a checkpoint which is common determination information between the nodes and which determines an update log arrival status. Then, the inserting unit  16  transmits the checkpoints generated periodically to the primary mirror node  20  and notifies the center-to-center notifying unit  17  of the checkpoints. Note that, for the checkpoint as used herein, a recovery point log which is an example of an update log is used. Note also that, as an example of “periodically”, for example, five seconds can be set, but the time interval for generating checkpoints may be changed as appropriate. 
       FIG. 5  is a diagram illustrating an example of a recovery point log. As illustrated in  FIG. 5 , the recovery point log is a log indicating a checkpoint that determines an update log arrival status, and consists of “a header, a control log display, and RP information”. In the “header”, information indicating an update log, a creation date and time, etc., are set. In the “control log display”, information indicating that it is a recovery point log is set. The “RP information” is information identifying a recovery point and includes an “identifier” and a “serial number”. The “identifier” is information identifying that it is checkpoint information of the DB quad-redundancy function. The “serial number” is a 23-byte fixed positive number and is a unique serial number in the DB quad-redundant system. 
     The center-to-center notifying unit  17  is a processor that puts together pieces of update information of the DB  12  and periodically transmits the update information to the secondary master node  50  in a different system. Specifically, the center-to-center notifying unit  17  creates an update file in which update logs obtained from the intra-center notifying unit  15  and recovery point logs obtained from the inserting unit  16  are put together in chronological order, at 10-second intervals, for example, and transmits the update file to the secondary master node  50 . Namely, between different centers, the center-to-center notifying unit  17  generates update information in which DB update information and checkpoints are periodically put together, and notifies of the update information asynchronously with an update to the DB  12 . 
       FIG. 6  is a diagram illustrating an example of an update file transmitted by communication between the systems. As illustrated in  FIG. 6 , the update file consists of update logs and a recovery point log. In the example of  FIG. 6 , the update file includes an update log  1 , an update log  2 , a recovery point log  1 , etc., and indicates that the logs are created in this order. Note that the update log  1  and the update log  2  correspond to the above-described user log or control log, and the recovery point log  1  corresponds to the above-described recovery point log. 
     Functional Configuration of the Primary Mirror Node 
     As illustrated in  FIG. 2 , the primary mirror node  20  includes a communication control unit  21 , a DB  22 , and a control unit  23 . 
     The communication control unit  21  is a processor that controls communication with the primary master node  10  and with the secondary mirror node  60 , and is a network interface, for example. For example, the communication control unit  21  receives DB update information from the primary master node  10  and transmits the DB update information to the secondary mirror node  60 . 
     The DB  22  is a database that stores, for example, the same business operation information as that in the DB  12  of the primary master node  10 , and corresponds to the standby DB  22  illustrated in  FIG. 1 . The DB  22  is updated in synchronization with the DB  12 . Note that the DB  22  is provided in a storage apparatus, e.g., a hard disk. The DB  22  corresponds to the standby DB  22 . 
     The control unit  23  is a processor that controls the whole process of the primary mirror node  20 , and is an example of a processor, for example. The control unit  23  performs the function of implementing a DB redundant system between the primary master node  10  and the primary mirror node  20 , and performs the function of implementing a DB redundant system between the primary mirror node  20  and the secondary mirror node  60 . Namely, the control unit  23  executes an application that implements a DB dual-redundancy function in the primary center  1 , and an application that implements DB quad-redundancy function across the centers. 
     The control unit  23  includes a receiving unit  24 , a DB updating unit  25 , and a center-to-center notifying unit  26 . These processors are an example of processes performed by an electronic circuit included in the processor or by the processor. 
     The receiving unit  24  is a processor that receives update information of the DB  12  from the primary master node  10 . Specifically, the receiving unit  24  receives an update log which is synchronized with an update to the DB  12  of the primary master node  10 , and notifies the DB updating unit  25  and the center-to-center notifying unit  26  of the update log. In addition, when the receiving unit  24  receives a recovery point log, the receiving unit  24  notifies the center-to-center notifying unit  26  of the recovery point log. 
     The DB updating unit  25  is a processor that updates the DB  22  using data update information notified from the primary master node  10 . For example, the DB updating unit  25  extracts, for example, a record to be updated and updated data from a variable-length part, etc., in a received update log, and updates the DB  22  according to the extracted information. The DB updating unit  25  updates the DB  22  every time an update log is received. As a result, the DB  22  can be synchronized with the DB  12  of the primary master node  10  and functions as a mirroring DB. 
     The center-to-center notifying unit  26  is a processor that puts together pieces of update information of the DB  22  and periodically transmits the update information to the secondary mirror node  60  in a different system. Specifically, the center-to-center notifying unit  26  creates an update file in which update logs and recovery point logs received from the primary master node  10  are put together in chronological order, at 10-second intervals, for example, and transmits the update file to the secondary mirror node  60 . For example, the center-to-center notifying unit  26  creates an update file illustrated in  FIG. 6  and transmits the update file to the secondary mirror node  60 . 
     Functional Configuration of the Secondary Center 
       FIG. 7  is a functional block diagram illustrating the functional configurations of the nodes in the secondary center. Here, the secondary master node  50  and the secondary mirror node  60  included in the secondary center  5  will be described. 
     Functional Configuration of the Secondary Master Node 
     As illustrated in  FIG. 7 , the secondary master node  50  includes a communication control unit  51 , a DB  52 , and a control unit  53 . 
     The communication control unit  51  is a processor that controls communication with the primary master node  10  and with the secondary mirror node  60 , and is a network interface, for example. For example, the communication control unit  51  receives an update file including various types of update logs, as update information of the DB  12  of the primary master node  10 , from the primary master node  10 . In addition, the communication control unit  51  transmits the update logs of the DB  12  of the primary master node  10  to the secondary mirror node  60 . 
     The DB  52  is a database that stores business operation information, etc., and corresponds to the standby DB  52  illustrated in  FIG. 1 . The DB  52  is updated asynchronously with an update to the DB  12 , using update information notified from the primary master node  10 . Note that the DB  52  is provided in a storage apparatus, e.g., a hard disk. The DB  52  corresponds to the standby DB  52 . 
     The control unit  53  is a processor that controls the whole process of the secondary master node  50 , and is an example of a processor, for example. The control unit  53  executes an application that implements the entire DB quad-redundant system across the centers illustrated in  FIG. 1 , and executes an application that implements the DB dual-redundancy function in the secondary center  5 . 
     The control unit  53  includes a receiving unit  54 , a DB updating unit  55 , and an intra-center notifying unit  56 . These processors are an example of processes performed by an electronic circuit included in the processor or by the processor. 
     The receiving unit  54  is a processor that receives update information of the DB  12  from the primary master node  10 . Specifically, the receiving unit  54  receives an update file including update logs at predetermined intervals. Then, the receiving unit  54  outputs the received update file to the DB updating unit  55 . 
     The DB updating unit  55  is a processor that updates the DB  52  according to data update information notified from the primary master node  10 . For example, the DB updating unit  55  extracts, from an update file received by the receiving unit  54 , various types of logs included in the update file. 
     Then, the DB updating unit  55  identifies user logs and control logs among the extracted logs. Thereafter, the DB updating unit  55  reflects data updates identified by the respective logs, in the DB  52  in chronological order in which the logs are created. In addition, when the DB updating unit  55  extracts various types of logs from the update file, the DB updating unit  55  outputs the extracted logs to the intra-center notifying unit  56  in chronological order. 
     For example, assuming that an update file of  FIG. 6  is received, the DB updating unit  55  extracts an update log  1 , an update log  2 , and a recovery point log  1  from the update file. Then, the DB updating unit  55  first reflects a data update identified by the update log  1  in the DB  52  and then reflects a data update identified by the update log  2  in the DB  52 . Meanwhile, the DB updating unit  55  outputs the extracted update log  1 , update log  2 , and recovery point log  1  to the intra-center notifying unit  56 . 
     The intra-center notifying unit  56  is a processor that transmits update information of data reflected in the DB  52  to the secondary mirror node  60 . Specifically, the intra-center notifying unit  56  transmits update logs and recovery point logs which are received from the primary master node  10 , to the secondary mirror node  60  in chronological order in which the logs are created. 
     Describing using the above-described example, the intra-center notifying unit  56  receives an update log  1 , an update log  2 , and a recovery point log  1  in turn from the DB updating unit  55 . Then, the intra-center notifying unit  56  first transmits the update log  1  to the secondary mirror node  60 , and then transmits the update log  2  to the secondary mirror node  60 , and finally transmits the recovery point log  1  to the secondary mirror node  60 . 
     Functional Configuration of the Secondary Mirror Node 
     As illustrated in  FIG. 7 , the secondary mirror node  60  includes a communication control unit  61 , a DB  62 , a buffer  63 , and a control unit  64 . 
     The communication control unit  61  is a processor that controls communication with the primary mirror node  20  and with the secondary master node  50 , and is a network interface, for example. For example, the communication control unit  61  receives data update information from both of the primary mirror node  20  and the secondary master node  50 . 
     The DB  62  is a database that stores business operation information, etc., and corresponds to the standby DB  62  illustrated in  FIG. 1 . The DB  62  is updated asynchronously with an update to the DB  12  of the primary master node  10 , using update information notified from the primary master node  10 . On the other hand, the DB  62  is updated in synchronization with an update to the DB  52  of the secondary master node  50 . Note that the DB  62  is provided in a storage apparatus, e.g., a hard disk. The DB  62  corresponds to the standby DB  62 . 
     The buffer  63  is a storage area that temporarily stores update information received from the primary mirror node  20  by communication between the centers and update information received from the secondary master node  50  by communication within the center. Note that the buffer  63  is provided in a storage apparatus, e.g., a hard disk or a memory. 
     The control unit  64  is a processor that controls the whole process of the secondary mirror node  60 , and is an example of a processor, for example. The control unit  64  executes an application that implements the entire DB quad-redundant system across the centers illustrated in  FIG. 1 , and executes an application that implements the DB dual-redundancy function in the secondary center  5 . 
     The control unit  64  includes an intra-center receiving unit  65 , a center-to-center receiving unit  66 , an identifying unit  67 , and a DB updating unit  68 . These processors are an example of processes performed by an electronic circuit included in the processor or by the processor. 
     The intra-center receiving unit  65  is a processor that receives data update information from the secondary master node  50 . Specifically, the intra-center receiving unit  65  receives update logs and recovery point logs from the secondary master node  50  and stores the update logs and the recovery point logs in the buffer  63  in chronological order of log creation date and time. Describing using the above-described example, the intra-center receiving unit  65  receives an update log  1 , an update log  2 , and a recovery point log  1  in turn from the secondary master node  50  and stores the update log  1 , the update log  2 , and the recovery point log  1  in the buffer  63  in the order of reception. 
     The center-to-center receiving unit  66  is a processor that receives data update information from the primary mirror node  20 . Specifically, the center-to-center receiving unit  66  receives an update file including update logs and recovery point logs, from the primary mirror node  20 . Then, the center-to-center receiving unit  66  extracts various types of logs from the update file and stores the logs in the buffer  63  in the order of creation date and time. 
     The identifying unit  67  is a processor that identifies, based on the recovery point logs, update information including update information whose processes have progressed further from among update information of the DB  12  obtained through the secondary master node  50  and update information of the DB  12  obtained from the primary mirror node  20 . 
     Specifically, the identifying unit  67  extracts the latest recovery point log from among the logs which are stored in the buffer  63  by the intra-center receiving unit  65 , and further extracts the latest recovery point log from among the logs which are stored in the buffer  63  by the center-to-center receiving unit  66 . Then, the identifying unit  67  identifies which one of the two recovery point logs is the latest one, using the creation dates and times of the recovery point logs, the serial numbers included in the recovery point logs, etc. For example, when serial numbers are assigned in ascending order in creation of recovery point logs, the recovery point logs with a larger serial number value is determined to be the latest recovery point. Alternatively, the recovery point logs with a later creation date and time of a recovery point log is determined to be the latest recovery point. 
     Then, the identifying unit  67  notifies the DB updating unit  68  of the identified latest recovery point. For example, when the identifying unit  67  determines that the recovery point logs received by the intra-center receiving unit  65  is the latest one, the identifying unit  67  determines that the update logs received from the intra-center receiving unit  65  are update information whose processes have progressed further. Then, the identifying unit  67  transmits an instruction to use the update logs received from the intra-center receiving unit  65 , to the DB updating unit  68 . 
     The DB updating unit  68  is a processor that updates the DB  62 , using further progressed data update information. Specifically, the DB updating unit  68  extracts data update information from the update logs notified from the identifying unit  67 , and reflects the data update information in the DB  62 . In the case of the above-described example, the DB updating unit  68  sequentially reads update logs received by the intra-center receiving unit  65  from the buffer  63  in chronological order, and sequentially reflects data updates identified by the respective logs, in the DB  62 . 
     Flow of Processes 
     Next, processes performed by each node will be described. Here, a DB update process and an update information notification process which are performed by each node will be described. Note that although here, as an example, an example is described in which a checkpoint (recovery point log) is created after a DB update, the configuration is not limited thereto. For example, a DB update process and a checkpoint creation process can be performed in parallel or can also be performed using different flowcharts. 
     Notification Process from the Primary Master Node  10  to the Primary Mirror Node  20   
       FIG. 8  is a flowchart illustrating the flow of a notification process from the primary master node to the primary mirror node. 
     As illustrated in  FIG. 8 , if an update to the DB  12  occurs (S 101 : Yes), the DB updating unit  14  of the primary master node  10  updates the DB  12  (S 102 ). Subsequently, the intra-center notifying unit  15  extracts a difference between before and after the update to the updated DB  12  (S 103 ), creates an update log, and transmits the update log to the primary mirror node  20  (S 104 ). 
     On the other hand, if it is checkpoint creation timing (S 105 : Yes), the inserting unit  16  creates a recovery point log including the serial number of a checkpoint (S 106 ) and transmits the recovery point log to the primary mirror node  20  (S 107 ). Note that if it is not checkpoint creation timing (S 105 : No), processing returns to S 101  and the processes at and after S 101  are performed. 
     Notification Process from the Primary Master Node  10  to the Secondary Master Node  50   
       FIG. 9  is a flowchart illustrating the flow of a notification process from the primary master node to the secondary master node. 
     As illustrated in  FIG. 9 , if an update to the DB  12  occurs (S 201 : Yes), the DB updating unit  14  of the primary master node  10  updates the DB  12  (S 202 ). Thereafter, the center-to-center notifying unit  17  extracts and accumulates a difference between before and after the update to the updated DB  12  (S 203 ). 
     Meanwhile, if it is checkpoint creation timing (S 204 : Yes), the inserting unit  16  creates and accumulates a recovery point log including the serial number of a checkpoint (S 205 ). Note that if it is not checkpoint creation timing (S 204 : No), S 205  is not performed but S 206  is performed. 
     Thereafter, if notification timing to the secondary master node  50  has reached (S 206 : Yes), the center-to-center notifying unit  17  creates an update file in which accumulated update logs and recovery point logs are described in the order of creation (S 207 ). Then, the center-to-center notifying unit  17  transmits the created update file to the secondary master node  50  (S 208 ). Note that if it is not checkpoint creation timing (S 206 : No), processing returns to S 201  and the processes at and after S 201  are performed. 
     Update Process of the Primary Mirror Node  20   
       FIG. 10  is a flowchart illustrating the flow of an update process performed by the primary mirror node. As illustrated in  FIG. 10 , if the receiving unit  24  of the primary mirror node  20  receives information from the primary master node  10  (S 301 : Yes), the receiving unit  24  determines whether the received information is an update log (S 302 ). 
     Subsequently, if the received information is an update log (S 302 : Yes), the DB updating unit  25  updates the DB  22  according to the received update log (S 303 ) and accumulates the update log used for the update (S 304 ). 
     On the other hand, if the received information is not an update log but is a recovery point log (S 302 : No), the DB updating unit  25  accumulates the received recovery point log in a storage unit or the like (S 305 ). 
     Notification Process of the Primary Mirror Node  20   
       FIG. 11  is a flowchart illustrating the flow of a notification process performed by the primary mirror node. As illustrated in  FIG. 11 , if notification timing has reached (S 401 : Yes), the center-to-center notifying unit  26  of the primary mirror node  20  reads accumulated update logs and recovery point logs (S 402 ). 
     Thereafter, the center-to-center notifying unit  26  rearranges the read logs in the order of creation to create an update file (S 403 ) and transmits the created update file to the secondary mirror node  60  in the secondary center  5  (S 404 ). 
     Processes of the Secondary Master Node  50   
       FIG. 12  is a flowchart illustrating the flow of update and notification processes performed by the secondary master node. As illustrated in  FIG. 12 , if an update file is received (S 501 : Yes), the DB updating unit  55  of the secondary master node  50  extracts logs included in the update file (S 502 ). 
     Subsequently, the DB updating unit  55  sequentially reflects update logs among the obtained logs, in the DB  52  in chronological order of creation time (S 503 ), and repeats such a process until reflection of all of the extracted update logs has been completed (S 504 : No). 
     Then, if reflection of all of the extracted update logs has been completed (S 504 : Yes), the intra-center notifying unit  56  sequentially transmits the update logs and recovery point logs which are obtained from the update file, in chronological order of creation time to the secondary mirror node  60  (S 505 ). Thereafter, the intra-center notifying unit  56  repeats the process until transmission of all of the logs obtained from the update file has been completed (S 506 : No). If transmission of all of the logs has been completed (S 506 : Yes), processing returns to S 501  and the processes at and after S 501  are repeated. 
     Processes of the Secondary Mirror Node  60   
       FIG. 13  is a flowchart illustrating the flow of an update process performed by the secondary mirror node. As illustrated in  FIG. 13 , if the intra-center receiving unit  65  of the secondary mirror node  60  receives an update log or a recovery point log from the secondary master node  50  (S 601 : Yes), the intra-center receiving unit  65  accumulates the received update log or recovery point log in the buffer  63  in the order of reception (S 602 ). 
     On the other hand, if the center-to-center receiving unit  66  receives an update file (S 603 : Yes) instead of an update log or a recovery point log (S 601 : No), the center-to-center receiving unit  66  extracts logs from the update file (S 604 ). Subsequently, the center-to-center receiving unit  66  accumulates the obtained update logs and recovery point logs in the buffer  63  in chronological order of log creation time (S 605 ). 
     Thereafter, until the timing of reflection in the DB  62  has reached, processing returns to S 601  and the processes at and after S 601  are performed (S 606 : No). Then, if the timing of reflection in the DB  62  has reached (S 606 : Yes), the identifying unit  67  compares the accumulated recovery point logs to identify newer update logs (S 607 ). 
     For example, the identifying unit  67  compares the latest recovery point log received by the intra-center receiving unit  65  with the latest recovery point log received by the center-to-center receiving unit  66 . Then, the identifying unit  67  identifies which one, the update logs received from the secondary master node  50  or the update logs received from the primary mirror node  20 , is update information whose processes have progressed further. 
     Thereafter, the DB updating unit  68  reflects each of the update logs whose processes have progressed further and which are identified by the identifying unit  67 , in the DB  62  in the order in which the update logs are created (S 608 ). 
     As described above, when the primary master node  10  transmits update information of the DB  12 , the primary master node  10  periodically transmits a checkpoint. The secondary mirror node  60  receives update information and checkpoints through two routes. Then, the secondary mirror node  60  can update the DB  62  using update information received through a route with a newer checkpoint. As a result, the secondary mirror node  60  can suppress data lost of update information for updating the DB  62 . 
     In addition, when the primary master node  10  has updated the DB  12 , the primary master node  10  transmits an update log to the primary mirror node  20 . As a result, the primary mirror node  20  can allow the state of the DB  22  to be synchronized with that of the DB  12 . In addition, the primary master node  10  periodically transmits an update file in which logs having been used to update the DB  12  are put together, to the secondary master node  50 . As a result, the secondary master node  50  can update the DB  22  with reduced update time of the DB  12  and a reduced time lag. 
     Therefore, even when switching occurs within a system due to the occurrence of a failure, business operations can continue without delay, using a standby DB. In addition, even when switching occurs between the systems due to the occurrence of a failure in the primary center, business operations can continue without delay, using the DBs in the secondary center. 
     [b] Second Embodiment 
     Next, a second embodiment describes system switching performed when a failure occurs in a primary master node  10  in a DB quad-redundant system described in the first embodiment. 
     In addition, although the first embodiment describes an example in which a secondary mirror node  60  accumulates update logs received through each route and identifies and reflects, at predetermined timing, update information whose processes have progressed further, the method of reflection in a DB is not limited thereto. 
     For example, the secondary mirror node  60  accumulates update information received from a primary mirror node  20 , and reflects update information received from a secondary master node  50 , in a DB. Then, when system switching occurs, the secondary mirror node  60  determines whether the update information received from the primary mirror node  20  is the latest information. If the information is the latest one, the secondary mirror node  60  reflects the update information and then can perform system switching. 
     Hence, the second embodiment describes a DB update process of the secondary mirror node which is different than that of the first embodiment, and a system switching process. 
     Overall Configuration 
       FIG. 14  is a diagram describing an example of the occurrence of a failure in a redundant system according to the second embodiment. As illustrated in  FIG. 14 , the overall configuration is the same as that of  FIG. 1  and thus a detailed description thereof is omitted. 
     In addition, dotted lines illustrated in  FIG. 14  indicate the flow of update logs and recovery point logs. Specifically, as in the first embodiment, the primary master node  10  transmits update logs and recovery point logs to both of the primary mirror node  20  and the secondary master node  50 . The primary mirror node  20  and the secondary master node  50  transmit the update logs and recovery point logs which are received from the primary master node  10 , to the secondary mirror node  60 . The secondary mirror node  60  receives the update logs and recovery point logs of the primary master node  10  through two routes. 
     In such a state, when, as illustrated in  FIG. 14 , the primary master node  10  is stopped due to a failure, etc., system switching occurs. Note that, for a trigger for the occurrence of system switching, an administrator&#39;s terminal may notify each node, or the primary master node  10  may notify the primary mirror node  20  of the occurrence of a failure. As another example, it is also possible that the primary mirror node  20  performs alive monitoring by periodically issuing ping (Packet Internet Groper) to the primary master node  10 , and when the primary mirror node  20  has not been able to detect a response, system switching is performed. 
       FIG. 15  is a diagram describing an example of system switching of the redundant system according to the second embodiment. When, as illustrated in  FIG. 15 , the primary master node  10  is stopped, the primary mirror node  20  is promoted to a master node in a primary system, the secondary mirror node  60  is promoted to a master node in a secondary system, and the secondary master node  50  is demoted to a mirror node in the secondary system. As a result, a standby DB  22  of the primary mirror node  20  is promoted to an active DB, and thus, the DB  22  is updated along with the execution of an application. 
     Then, when the standby DB  22  is updated, the primary mirror node  20  transmits an update file including update logs and recovery point logs to the secondary mirror node  60 . The secondary mirror node  60  updates a standby DB  62  according to the update logs included in the received update file. Then, the secondary mirror node  60  transmits the update logs and recovery point logs included in the received update file to the secondary master node  50 . The secondary master node  50  updates a standby DB  52  according to the received update logs. 
     Update Process of the Secondary Mirror Node 
       FIG. 16  is a flowchart illustrating the flow of an update process performed by the secondary mirror node according to the second embodiment. Note that the process described here is an update process performed before system switching. 
     As illustrated in  FIG. 16 , if an intra-center receiving unit  65  receives information from the secondary master node  50  (S 701 : Yes), a DB updating unit  68  of the secondary mirror node  60  determines whether the received information is an update log (S 702 ). 
     Then, if the received information is an update log (S 702 : Yes), the DB updating unit  68  reflects the received update log in the DB  62  and thereby updates the DB  62  (S 703 ). On the other hand, if the received information is not an update log but is a recovery point log (S 702 : No), the DB updating unit  68  accumulates the recovery point log in a buffer  63  or the like (S 704 ). 
     In addition, if, at S 701 , a center-to-center receiving unit  66  receives an update file from the primary mirror node  20  instead of from the secondary master node  50  (S 701 : No and S 705 : Yes), the center-to-center receiving unit  66  extracts logs from the update file (S 706 ). Then, the center-to-center receiving unit  66  accumulates the obtained update logs and recovery point logs in the buffer  63  in chronological order of log creation time (S 707 ). 
     Thereafter, the DB updating unit  68  compares, at pre-specified intervals, the recovery point logs stored in the buffer  63  (S 708 ). If the recovery point log from the secondary master node  50  is the latest one (S 709 : Yes), the DB updating unit  68  performs S 710 . Specifically, the DB updating unit  68  deletes the update logs and recovery point logs received from the primary mirror node  20 , from the buffer  63  (S 710 ). 
     On the other hand, if the recovery point log from the secondary master node  50  is not the latest one, but the recovery point log from the primary mirror node  20  is the latest one (S 709 : No), the DB updating unit  68  repeats S 701  and subsequent steps, with the update logs and recovery point logs received from the primary mirror node  20  remaining in the buffer  63 . 
     System Switching Process of the Secondary Mirror Node 
       FIG. 17  is a flowchart illustrating the flow of a system switching process performed by the secondary mirror node according to the second embodiment. As illustrated in  FIG. 17 , in the DB updating unit  68  of the secondary mirror node  60 , if system switching occurs (S 801 : Yes), the buffer  63  extracts the latest recovery point log transmitted from the secondary master node  50  and the latest recovery point log transmitted from the primary mirror node  20  (S 802  and S 803 ). 
     Giving an example, the DB updating unit  68  of the secondary mirror node  60  detects the occurrence of system switching by receiving a switching notification from the primary mirror node  20  having detected a stop of the primary master node  10  or by receiving a switching instruction from the administrator&#39;s terminal. 
     Then, if the recovery point log from the secondary master node  50  is the latest one (S 804 : Yes), the DB updating unit  68  deletes the update logs and recovery point logs received from the primary mirror node  20 , from the buffer  63  (S 805 ). 
     On the other hand, if the recovery point log from the secondary master node  50  is not the latest one but the recovery point log from the primary mirror node  20  is the latest one (S 804 : No), the DB updating unit  68  performs S 806 . 
     Specifically, the DB updating unit  68  reflects the update logs received from the primary mirror node  20  and accumulated in the buffer  63 , in the DB  62  and thereby updates the DB  62  (S 806 ), and repeatedly performs such a process until reflection of all of the accumulated update logs has been completed (S 807 : No). 
     Thereafter, if reflection of all of the accumulated update logs at S 807  has been completed (S 807 : Yes) or when performance of S 805  has been completed, the DB updating unit  68  performs system switching (S 808 ). That is, the DB updating unit  68  performs the same processes as those performed by the secondary master node  50  which are described in the first embodiment, and behaves as a master node in the secondary system. 
     At this time, the DB updating unit  68  of the secondary mirror node  60  notifies the secondary master node  50  of the occurrence of system switching, and the secondary master node  50  performs the functions of a mirror node in the secondary system. That is, the secondary master node  50  behaves as a general DB dual-redundant mirror DB. 
     As such, in the DB quad-redundant system, even when the primary master node  10  is stopped, since each node can operate by automatically changing its role, business operations can continue, leading to an improvement in reliability. 
     In addition, since the secondary mirror node  60  can sequentially reflect update logs received from the secondary master node  50  until system switching occurs, the state of the DB  62  can be maintained to be more latest compared to the first embodiment. Therefore, even when the secondary master node  50  is stopped, the secondary mirror node  60  can maintain the functions as a backup while suppressing data loss. 
     [c] Third Embodiment 
     Although the embodiments of the present invention have been described so far, in addition to the above-described embodiments, the present invention may be implemented in various different modes. 
     Checkpoint 
     Although the above-described embodiments describe an example in which a recovery point log including a unique serial number in a system is used as a checkpoint, the configuration is not limited thereto. For example, a recovery point log including a date and time, a time, etc., can also be used. That is, various information can be used as long as the information can specify a unique order such as an ascending order or a descending order in the system. 
     System 
     In addition, of the processes described in the embodiments, all or some of those processes described as being performed automatically can also be performed manually. Alternatively, all or some of those processes described as being performed manually can also be performed automatically by publicly known methods. In addition, information including processing procedures, control procedures, specific names, various types of data, and parameters illustrated in the above-described document and drawings can be arbitrarily changed unless otherwise indicated. 
     In addition, each component of each apparatus illustrated in the drawings is functionally conceptual and thus does not always physically configured as illustrated in the drawings. Namely, a specific mode of separation or integration of each apparatus is not limited to that illustrated in the drawings. That is, all or some of the components can be configured by separating or integrating them functionally or physically in any unit, according to various types of loads, the status of use, etc. Furthermore, all or arbitrary ones of processing functions performed by each apparatus can be implemented by a CPU and a program analyzed and executed by the CPU or implemented by wired logic hardware. 
     Hardware 
       FIG. 18  is a diagram describing an exemplary hardware configuration. Since the nodes illustrated in  FIG. 1  have the same hardware configuration, here, as an example, a description is made using the primary master node  10  as an example. 
     As illustrated in  FIG. 18 , the primary master node  10  includes an HDD (Hard Disk Drive)  10   a , a communication interface  10   b , a memory  10   c , and a CPU (Central Processing Unit)  10   d . In addition, the components illustrated in  FIG. 18  are connected to each other by a bus or the like. Note that the hardware illustrated here is an example and thus the primary master node  10  may include other hardware, e.g., a graphics interface and a mouse. 
     The HDD  10   a  stores a program that causes the functions illustrated in  FIG. 2 , etc., to operate, and a DB. The communication interface  10   b  is an interface that controls communication with other apparatuses and is a network interface card, for example. 
     The CPU  10   d  reads a program that performs the same processes as those of the processors illustrated in  FIG. 2 , etc., from the HDD  10   a  or the like and expands the program in the memory  10   c , and thereby allows processes that perform the functions described in  FIG. 2 , etc., to operate. 
     Namely, the processes perform the same functions as those of the processors included in the primary master node  10 . Specifically, the CPU  10   d  reads a program having the same functions as the DB updating unit  14 , the intra-center notifying unit  15 , the inserting unit  16 , the center-to-center notifying unit  17 , etc., from the HDD  10   a  or the like. Then, the CPU  10   d  performs processes that perform the same processes as those of the DB updating unit  14 , the intra-center notifying unit  15 , the inserting unit  16 , and the center-to-center notifying unit  17 . 
     As such, the primary master node  10  operates as an information processing apparatus that performs a redundancy creation method by reading and executing a program. In addition, the primary master node  10  can also implement the same functions as those in the above-described embodiments, by reading the above-described program from a recording medium by a medium reading apparatus and executing the read program. Note that a program referred to in other embodiments is not limited to being executed by the primary master node  10 . For example, when another computer or server executes a program or when the computer and the server execute a program in cooperation with each other, too, the present invention can be applied in the same manner. 
     In one aspect, an increase in data lost in reflection of update information can be suppressed. 
     All examples and conditional language recited herein are intended for pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.