Abstract:
A method or apparatus for cooperative data replication. The method in one embodiment can be performed by a computer system or several computer systems executing software instructions. The method may include modifying data in n data blocks of a data volume to create n modified data blocks. A copy of each of the n modified data blocks is created for subsequent transfer to other nodes. A first computer system transmits the n modified data block copies to n nodes, respectively, wherein each of the n nodes comprises a second computer system and a replica of the data volume prior to the modification of data in the n data blocks. Thereafter, one of the n second computer systems transmits a copy of the modified data block copy it receives to another of the n second computer systems.

Description:
BACKGROUND OF THE INVENTION 
     Many businesses such as insurance companies, banks, brokerage firms, etc., rely heavily on data processing systems for storing and processing business critical data. Businesses seek reliable mechanisms to protect their data from natural disasters, acts of terrorism, computer hardware failure, or computer software failure. 
     Replication is one mechanism used by many businesses to ensure reliable access to data. Data replication is well known in the art. Essentially, replication is a process of creating at remotely located sites one or more real or near real-time copies (replicas) of a data volume.  FIG. 1  shows (in block diagram form) a data processing system  10  in which data replication is employed to create and maintain three replicas of a data volume. More particularly,  FIG. 1  shows a primary node P coupled to secondary nodes S 1 -S 3  via a communication network and data links  12 - 18 . The communication network may include a LAN or a WAN. For purposes of explanation, the communication network is the Internet, it being understood that the term “communication network” should not be limited thereto. 
     Primary node P consists of a primary computer system  22  coupled to a data storage system  24  via data link  26 . Data storage system  24  includes a memory  28  which includes several hard disks for storing data of a primary data volume V. Secondary node S 1  includes a secondary computer system  32  coupled to data storage system  34  via data link  36 . Data storage system  34  includes memory  38  consisting of several hard disks for storing a first replica R 1  of primary volume V. Secondary node S 2  includes secondary computer system  42  coupled to data storage system  44  via data link  46 . Data storage system  44  includes memory  48  consisting of several hard disks for storing a second replica R 2  of the primary volume V. Lastly, secondary node S 3  includes a secondary computer system  52  coupled to data storage system  54  via data link  56 . Data storage system  54  includes a memory  58  consisting of several hard disks that store data of a third replica R 3  of primary volume V. 
       FIG. 2  illustrates in block diagram form primary volume V and replicas R 1 -R 3 . Each of volumes V and R 1 -R 3  consists of n max  data blocks. While it is said that each of these blocks contain data, it is to be understood that the data is physically stored within hard disk memory blocks allocated thereto. Thus, data of blocks  1 - n   max  of primary volume V are stored in distributed fashion within hard disks of memory  28 . Further, data within blocks  1 - n   max  of replicas R 1 -R 3  are stored in distributed fashion across hard disks in memories  38 ,  48 , and  58 , respectively, allocated thereto. Each of replicas R 1 -R 3  is maintained as a real or near real-time copy of primary volume V. Thus, data of block n in primary volume V should be identical to data of blocks n in replicas R 1 -R 3 . 
     Primary computer system  22  is configured to receive requests from client computer systems (not shown) to read data from or write data to primary data volume V. In response to these requests, primary computer system  22  generates input/output (IO) transactions to read data from or write data to hard disks of memory  28 . In the event of failure of primary node P, requests from client computer systems can be redirected to and serviced by one of the secondary nodes S 1 -S 3 . For example, suppose a client computer system generates a request to read data block  4  in the primary volume V after primary computer system  22  is rendered inoperable as a result of hardware failure. The read request can be redirected to secondary node S 1  using mechanisms well known in the art. In response to receiving the read request, secondary computer system generates an IO transaction that accesses and reads data from a hard disk in memory  38  allocated to store block  4  data of replica R 1 . Data returned from memory  38  is subsequently forwarded by secondary computer system  32  to the client computer system that originally requested block  4  data. Since data of blocks n of the primary data volume V and replica R 1  are or should be identical, valid data should be returned to the client computer system even though primary computer system  22  has failed. 
     Replicas R 1 -R 3  can be maintained as a real-time or near real-time copy of primary volume V using one of several replication techniques including synchronous, asynchronous, and periodic replication. In each of these techniques, when a data block n of the primary data volume V is modified according to an IO transaction, the primary node P transmits a copy of the modified data block to each of the secondary nodes that store a replica. Each of the secondary nodes, in turn, overwrites its existing data block n with the copy received from the primary node P. In synchronous replication, the IO transaction that modified block n data of the primary data volume V is not considered complete until one or all of the secondary nodes acknowledge receipt of the copy of the modified data block n. In asynchronous replication, primary node P logs a copy of each data block of the primary data volume V that is modified by an IO transaction. Eventually, copies of the logged, modified data blocks are transmitted asynchronously to each of the secondary nodes S 1 -S 3 . The IO transaction that modifies data block n of the primary data volume V is considered complete when a copy of the modified block n is logged for subsequent transmission to secondary nodes S 1 -S 3 . Asynchronous replication requires ordering of dependent data modifications to ensure consistency between replicas R 1 -R 3  and primary volume V. Synchronous replication does not require ordering. Periodic replication is yet another technique for maintaining replicas R 1 -R 3 . U.S. patent application Ser. No. 10/436,354 entitled, “Method and System of Providing Periodic Replication” (filed on May 12, 2003, incorporated herein by reference in its entirety) describes relevant aspects of periodic replication. Like synchronous and asynchronous replication, periodic replication requires that primary node P transmit a copy of a modified data block n of the primary data volume V to each of the secondary nodes S 1 -S 3 . 
     Modified data blocks of the primary data volume V can be transmitted from primary node P to each of the secondary nodes S 1 -S 3  in separate transactions via the data link  12  and communication network. Each of the transactions transmitted to the secondary nodes S 1 -S 3  may include a single modified data block or multiple modified data blocks of the primary data volume V. Either way, each of the secondary nodes S 1 -S 3  receives, directly from primary node P, a copy of each data block n modified in primary data volume V. 
     The time needed for secondary nodes S 1 -S 3  to update replicas R 1 -R 3  with modified data blocks depends on the bandwidth of data link  12 . The higher the bandwidth of data link  12 , the faster transactions containing modified data blocks of the primary data volume V can be transmitted to secondary nodes S 1 -S 3  for subsequent storage in replicas R 1 -R 3 , respectively. The cost of data link  12 , however, is dependent on the bandwidth thereof. Table I below shows an example of how the cost of data link  12  can increase with bandwidth. 
     
       
         
               
             
               
               
               
               
             
               
               
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 Cost of Data Link Bandwidth 
               
             
          
           
               
                   
                   
                 Bandwidth 
                   
               
               
                   
                 Type of Link 
                 (Mbps) 
                 Approximate Cost (per month) 
               
               
                   
                   
               
             
          
           
               
                   
                 T1 
                 1.544 
                 $900-$1200 
               
               
                   
                 E1 
                 2.048 
                 $2000 
               
               
                   
                 T3 
                 44.736 
                 $10,000 + local loops + setup 
               
               
                   
                   
                   
                 ($4000 each) 
               
               
                   
                 OC-3 
                 155.5 
                 &gt;$40,000 + local loops + setup 
               
               
                   
                   
                   
                 (&gt;$10,000 each) 
               
               
                   
                 OC-12 
                 622.08 
                 &gt;$400,000 + local loops + setup 
               
               
                   
                   
                   
                 (&gt;$100,000 each) 
               
               
                   
                 OC-48 
                 2488 
                 &gt;$2,000,000 + local loops + 
               
               
                   
                   
                   
                 setup (&gt;$200,000 each) 
               
               
                   
                   
               
             
          
         
       
     
     SUMMARY OF THE INVENTION 
     In contrast to the prior art described within the background section, cooperative replication according to one or more of the embodiments described above reduces the total amount of modified data transmitted directly between the primary node and secondary nodes. The method in one embodiment can be performed by a computer system or several computer systems executing software instructions. The method may include modifying data in n data blocks of a data volume to create n modified data blocks. A copy of each of the n modified data blocks is created for subsequent transfer to other nodes. A first computer system transmits the n modified data block copies to n nodes, respectively, wherein each of the n nodes comprises a second computer system and a replica of the data volume prior to the modification of data in the n data blocks. Thereafter, one of the n second computer systems transmits a copy of the modified data block copy it receives to another of the n second computer systems 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention may be better understood, and its numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings. 
         FIG. 1  is a block diagram of a data processing system employing storage subsystems; 
         FIG. 2  shows block diagrams illustrating data contents of volumes in memory structure of storage systems shown in  FIG. 2 ; 
         FIG. 3  is a block diagram of a data processing system employing one embodiment of the present invention; 
         FIG. 4  shows block diagrams illustrating data contents of volumes in memory structure of storage systems shown in  FIG. 3 ; 
         FIG. 5  shows the data processing system of  FIG. 3  redrawn to more easily explain the present invention. 
     
    
    
     The use of the same reference symbols in different drawings indicates similar or identical items. 
     DETAILED DESCRIPTION 
     The present invention relates to a method or apparatus for maintaining several replica copies of a primary data volume. Using the present invention, secondary nodes that store replicas of the primary data volume cooperate with each other to maintain consistency between the replicas and the primary data volume. 
       FIG. 3  is a block diagram illustrating a data processing system  60  employing one embodiment of the present invention. Processing system  60  includes a primary node P coupled to secondary nodes S 1 -S 3  via a communication network (e.g. the Internet) and data links  62 - 68 . Primary node P stores a primary data volume V while secondary nodes S 1 -S 3  store real or near real-time replicas R 1 -R 3 , respectively, of the primary volume V. The present invention should not be limited to a system which includes three replicas of a primary data volume; the present invention could be employed in a processing system that has fewer or more than three replicas of a primary data volume. 
     For purposes of explanation, each of the primary node P and the secondary nodes S 1 -S 3  will be described below as containing a computer system (e.g., a server computer system) and a data storage system for storing the primary data volume V or one of the replicas R 1 -R 3 . The present invention should not be limited thereto. The primary node P and/or the secondary nodes S 1 -S 3  may take form in fiber channel switches, storage appliances or storage arrays. In any embodiment, the primary node P is capable of sending copies of modified volume V data blocks to the secondary nodes S 1 -S 3  in addition to instructions for the secondary nodes S 1 -S 3  to distribute among each other copies of modified data blocks received from the primary node P. 
     It is further noted that the present invention need not be limited to an environment in which the primary data volume V is stored at a single node or in which each of the replicas R 1 -R 3  are stored at a respective, single node. The present invention could be applied to hardware environment in which the data contents of the primary data volume V is distributed across several nodes each one of which contains a computer system (or similar device for processing data according to software instructions) and a data storage system. Further, the data contents of each of the replicas R 1 -R 3  may be distributed across two or more nodes each containing a computer system (or similar device for processing data according to software instructions) and a data storage system. 
     Primary node P consists of a primary computer system  72  coupled to a data storage system  74  via data link  76 . Data storage system  74  includes a memory  78  that includes one or more memory devices for storing data of a primary data volume V. Secondary node S 1  includes a secondary computer system  82  coupled to data storage system  84  via data link  86 . Data storage system  84  includes memory  88  that includes one or more memory devices for storing a first replica R 1  of primary volume V. Secondary node S 2  includes secondary computer system  92  coupled to data storage system  94  via data link  96 . Data storage system  94  includes memory  98  that includes one or more memory devices for storing a second replica R 2  of the primary volume V. Lastly, secondary node S 3  includes a secondary computer system  102  coupled to data storage system  104  via data link  106 . Data storage system  104  includes a memory  108  that includes one or more memory devices for storing data of a third replica R 3  of primary volume V. For purposes of explanation, each of the memories  78 ,  88 ,  98  and  108  include several magnetic and/or optical hard disks for storing data, it being understood that the term “memory” should not be limited thereto. 
     Primary node P and secondary nodes S 1 -S 3  each may execute a system for managing the distribution of data of a volume across one or more memory devices. Volume Manager™ provided by VERITAS Software Corporation of Mountain View, Calif., is an exemplary system for managing the distribution of data of a volume across one or more memory devices. Volume Manager™ virtualizes the memory devices (e.g., hard disks) of one or more data storage systems to form a large virtual disk that stores volume data. Volume and disk management products from other product software companies also provide a system for managing the distribution of volume data across multiple memory devices. Hardware RAID adapter cards and RAID firmware built into computer systems can also provide this function. In the embodiment where the data contents of the primary data volume V is distributed across several nodes each one of which contains a computer system (or similar device for processing data according to software instructions) and one or more data storage system, a distributed system such as Cluster Volume Manager™ provided by VERITAS Software Corporation of Mountain View, Calif. can be employed to manage the distribution of the volume data. Cluster Volume Manager™ can also be applied to manage the distribution of data in replicas R 1 -R 3  when the data contents of each of the replicas R 1 -R 3  is distributed across two or more nodes each containing a computer system (or similar device for processing data according to software instructions) and a data storage system. Virtualization of the hard disks may also be performed outside of the computer systems of primary node P and secondary nodes S 1 -S 3 . For example, virtualization of hard disks may occur in a SAN switch which connects a computer system to multiple data storage systems in primary node P. SAN Volume Manager™ provided by VERITAS Software Corporation is an exemplary system that can be employed in a SAN switch. 
       FIG. 4  illustrates in block diagram form primary volume V and replicas R 1 -R 3 . Each of volumes V and R 1 -R 3  consists of n max  data blocks. While it is said that each of the blocks contain data, it is to be understood that the data is physically stored within one or more hard disk memory blocks allocated thereto. Thus, data of blocks  1 - n   max  of primary volume V are stored in distributed fashion within hard disks of memory  78 . Further, data within blocks  1 - n   max  of replicas R 1 -R 3  are stored in distributed fashion across hard disks in memories  88 ,  98 , and  108 , respectively, allocated thereto. Each of replicas R 1 -R 3  is maintained as a real or near real-time copy of primary volume V. Thus, data of block n in primary volume V should be identical to data of blocks n in replicas R 1 -R 3 . 
     Primary computer system  72  is configured to receive requests from client computer systems (not shown) to read data from or write data to primary data volume V. In response to these requests, primary computer system  72  generates IO transactions to read data from or write data to hard disks of memory  78 . In the event of failure of primary node P, requests from client computer systems can be redirected to and serviced by any one of the secondary nodes S 1 -S 3  using respective replicas R 1 -R 3 . 
     Each of replicas R 1 -R 3  can be maintained as a real-time or near real-time copy of primary volume V using one of several replication techniques. To maintain consistency between primary data volume V and its replicas R 1 -R 3 , the replicas should be updated as soon as possible whenever a data block n of the primary data volume V is modified by, for example, an IO transaction generated by primary computer system  72 . In contrast to the data processing system described in the background section above, primary node P does not directly send a copy of each modified block n of the primary volume V to each of the secondary nodes S 1 -S 3 . However, replicas R 1 -R 3  are maintained as real or near real-time copies of primary data volume V using cooperative replication according to the present invention. For example, in one embodiment described below, primary node P sends a copy of three recently modified data blocks w 1 -w 3  of the primary V to secondary nodes S 1 -S 3 , respectively. Each of the secondary nodes S 1 -S 3  may forward a copy of the modified data block it receives from the primary node P to the other secondary nodes so that each of the secondary nodes S 1 -S 3  eventually receives a copy of each of the modified data blocks w 1 -w 3 . Using this embodiment or other embodiments of the present invention, the bandwidth requirement of at least data link  62  can be reduced while retaining the speed at which replicas R 1 -R 3  are maintained if a data link  62  of higher bandwidth is employed. A reduction in the bandwidth of data link  62  should reduce the costs of implementing and operating data processing system  60 . 
     For ease in explaining operative aspects of the present invention,  FIG. 5  shows the data processing system  60  of  FIG. 3  redrawn so that primary node P is coupled to secondary nodes S 1 -S 3  via communication links  112 ,  114 , and  116 , respectively; secondary node S 2  coupled to secondary node S 1  and S 3  via communication links  122  and  124 , respectively; and, secondary node S 1  is coupled to secondary node S 3  via communication link  126 . Each of the communication links  112 - 116  and  122 - 126  consist of the communication network  70  and two of the data links  62 - 68  shown within  FIG. 3 . More particularly, communication link  112  consists of data link  62 , communication network  70 , and data link  64 ; communication link  114  consists of data link  62 , communication network  70  and data link  66 ; communication link  116  consists of data link  62 , communication network  70  and data link  68 ; communication link  122  consists of data link  64 , communication network  70  and data link  66 ; communication link  124  consists of data link  66 , communication network  70 , and data link  68 ; and communication link  126  consists of data link  64 , communication network  70 , and data link  68 . 
     Primary computer system  72  may contain a log (not shown) that temporarily stores copies of recently modified data blocks w x  of primary data volume V. For purposes of explanation, each of the data blocks temporarily stored in the log contain data modified by a respective IO transaction, it being understood that two or more of the data blocks stored in the log may have been modified by a single IO transaction generated by primary computer system  72 . In another embodiment some other tracking mechanism like bitmap or extent map can be used to identify the modified blocks. 
     When the log contains copies of at least three recently modified data blocks w 1 -w 3  of the primary data volume V, primary computer system  72  transmits the modified blocks w 1 -w 3  to secondary nodes S 1 -S 3 , respectively, via communication links  112 - 116 , respectively. It is noted that the data blocks w 1 -w 3  may be of different sizes. Primary computer system  72  may also transmit meta data items i 1 -i 3 , to secondary nodes S 1 -S 3 , respectively, via communication links  112 - 116 , respectively. Meta data items i 1 -i 3  may be contained in the same transactions to secondary nodes S 1 -S 3  that contain modified data blocks w 1 -w 3 . Thus, primary computer system  72  may generate and send first, second, and third transactions to secondary nodes S 1 -S 3 , respectively, via communication links  112 - 116 . The first, second, and third transactions contain modified data blocks w 1 -w 3 , respectively, and meta data items i 1 -i 3 , respectively, it being understood that the present should not be limited thereto. 
     Each of the meta data items i 1 -i 3  may consist of an instruction or a copy of one or more of the modified data blocks w 1 -w 3 . The instruction may be a Send instruction that directs the secondary node receiving the Send instruction to send a copy of one or more modified data blocks identified by a list to one or more other secondary nodes, or the instruction may be a Recv instruction that directs the secondary node receiving the Recv instruction to receive a copy of a modified data block w x  from another secondary node. Primary computer system  72  generates meta data item i x  according to one of several different algorithms. Equations 1-3 below identify meta data i 1 -i 3  generated by primary computer system  72  according to one algorithm.
 
 i   1 =[Send( w   1   ,{S   2   ,S   3 })+Recv( w   2   ,{S   2 })+Recv( w   3   ,{S   3 })]  (1)
 
 i   2 =[Send( w   2   ,{S   1   ,S   3 })+Recv( w   1   ,{S   1 })+Recv( w   3   ,{S   3 })]  (2)
 
 i   3 =[Send( w   3   ,{S   1   ,S   2 })+Recv( w   1   ,{S   1 })+Recv( w   2   ,{S   2 })]  (3)
 
Send(w 1 ,{S 2 ,S 3 }) of item i 1  instructs node S 1  to send a copy of w 1  it received from primary computer system  72  to secondary nodes S 2  and S 3 , while Recv(w 2 ,{S 2 }) and Recv(w 3 ,{S 3 }) instructs node S 1  to receive from nodes S 2  and S 3  copies of w 2  and w 3 , respectively, they received from primary computer system  72 . Send(w 2 ,{S 1 ,S 3 }) of item i 2  instructs node S 2  to send a copy of w 2  it received from primary computer system  72  to secondary nodes S 1  and S 3 , while Recv(w 1 ,{S 1 }) and Recv(w 3 ,{S 3 }) instructs node S 2  to receive from nodes S 1  and S 3  copies of w 1  and w 3 , respectively, they received from primary computer system  72 . Send(w 3 ,{S 1 ,S 2 }) of item i 3  instructs node S 3  to send a copy of w 3  it received from primary computer system  72  to secondary nodes S 1  and S 2 , while Recv(w 1 ,{S 1 }) and Recv(w 2 ,{S 2 }) instructs node S 3  to receive from nodes S 1  and S 2  copies of w 1  and w 2 , respectively, they received from primary computer system  72 . Once instructions of items i 1 -i 3  are successfully performed by computer systems  82 ,  92 , and  102 , respectively, computer systems  82 ,  92 , and  102  can overwrite existing data blocks of replicas R 1 -R 3 , respectively with w 1 -w 3  to put replicas into a consistent state with primary data volume V. Copies of data blocks w 1 -w 3  in the log of primary computer system can be deleted or marked as sent. In another embodiment where log is not used, the bitmap or extent map can be modified to indicate that the data has been applied by the secondaries.
 
     Meta data items may be generated by primary computer system  72  according to the following algorithms:
 
 i   1   =[w   2 +Send( w   1   ,{S   2 })+Recv( w   3   ,{S   3 })]  (4)
 
 i   2   =[w   3 +Send( w   2   ,{S   3 })+Recv( w   1   ,{S   1 })]  (5)
 
 i   3   =[w   1 +Send( w   3   ,{S   1 })+Recv( w   2   ,{S   2 })]  (6)
 
Each of meta data items i 1 -i 3  defined by equations 4-6 above includes a copy of data block in addition to a pair of instructions. Thus, in addition to sending copies of blocks w 1 -w 3  to secondary nodes S 1 -S 3 , respectively, primary computer system  72  sends a copy of blocks w 2 , w 3 , and w 1 , to secondary nodes S 1 -S 3 , respectively, via items i 1 -i 3 , respectively. In other words, each of the secondary nodes S 1 -S 3  receives copies of two of the modified data block w 1 -w 3  directly from primary computer system  72  in this embodiment. Send(w 1 ,{S 2 }) of item i 1  instructs node S 1  to send a copy of w 1  it received from primary computer system  72  to secondary node S 2 , while Recv(w 3 ,{S 3 }) instructs node S 1  to receive from node S 3  a copy of w 3  it received from primary computer system  72 . Send(w 2 ,{S 3 }) of item i 2  instructs node S 2  to send a copy of w 2  it received from primary computer system  72  to secondary node S 3 , while Recv(w 1 ,{S 1 }) instructs node S 2  to receive from node S 1  a copy of w 1  it received from primary computer system  72 . Send(w 3 ,{S 1 }) of item i 3  instructs node S 3  to send a copy of w 3  it received from primary computer system  72  to secondary node S 1 , while Recv(w 2 ,{S 2 }) instructs node S 3  to receive from node S 2  a copy of w 2  it received from primary computer system  72 . Once instructions of items i 1 -i 3  are successfully performed by computer systems  82 ,  92 , and  102 , respectively, computer systems  82 ,  92 , and  102  can overwrite existing data blocks of replicas R 1 -R 3 , respectively with w 1 -w 3  to place replicas into a consistent state with primary data volume V. Copies of data blocks w 1 -w 3  in the log of primary computer system can be deleted. The embodiment involving equations (4)-(6) is useful when links  122 - 126  between secondary nodes S 1 -S 3  have limited bandwidth or other insufficiencies when compared to links  112 - 116 .
 
     In yet another embodiment, primary computer system  72  generates meta data items i 1 -i 3  according to the equations below:
 
 i   1 =[Send( w   1   ,{S   2   ,S   3 })+Recv( w   2   ,{S   2 })+Recv( w   3   ,{S   2 })]  (7)
 
 i   2   =[w   3 +Send( w   2   ,{S   1   ,S   3 })+Send( w   3   ,{S   1   ,S   3 })+Recv( w   1   ,{S   1 })]  (8)
 
 i   3   =[−w   3 +Recv( w   1   ,{S   1 })+Recv( w   2 ,{S 2 })+Recv( w   3   ,{S   2 })]  (9)
 
In this embodiment, not all secondary nodes S 1 -S 3  will receive a copy of w 1 -w 3 , respectively, directly from the primary computer system. The −w 3  in i 3  indicates that secondary node S 3  does not receive a copy of w 3  directly from primary computer system  72 . The Send and Recv instructions of equations (7)-(9) operate similar to that described above. In this embodiment, node S 3  will receive copies of w 1 -w 3  from secondary nodes S 1  and S 2  when the instructions in items i 1  and i 2  are completed. Once instructions of items i 1 -i 3  are successfully performed by computer systems  82 ,  92 , and  102 , respectively, computer systems  82 ,  92 , and  102  can overwrite existing data blocks of replicas R 1 -R 3 , respectively with w 1 -w 3  to place replicas into a consistent state with primary data volume V. Copies of data blocks w 1 -w 3  in the log of primary computer system can be deleted. This embodiment can be useful when the communication links  122 - 126  between the secondary nodes S 1 -S 3  are in a better condition (e.g., have a higher bandwidth) than the communication link between primary node P and one or more secondary nodes S 1 -S 3 .
 
     The meta data items i 1 -i 3  could be generated based upon many different types of statistics. For example, primary computer system  72  can take into account differences in data transmission rates between links  122 - 126 . The equations below illustrate how primary computer system  72  can generate items i 1 -i 3  in a situation where the communications links  122  and  126  have higher bandwidth when compared to the communication link  124 .
 
 i   1   =[w   2   +w   3 +Send( w   1   ,{S   2   ,S   3 })+Send( w   2   ,{S   3 })+Send( w   3   ,{S   2 })]  (10)
 
 i   2 =[Recv( w   1   ,{S   1 })+Recv( w   3   ,{S   1 })]  (11)
 
 i   3 =[Recv( w   1   ,{S   1 })+Recv( w   2   ,{S   1 })]  (12)
 
     Normally, modification of the data block according to a write data  10  transaction to the primary data volume V is considered complete in asynchronous replication when a copy of the modified data block is stored in the log of primary computer system  72 . Further, modification of the data block according to a write data  10  transaction to the primary data volume V is considered complete in synchronous replication when the secondary node acknowledges that its replica has been updated. In the present invention, modified data is multiplexed to the secondary nodes, and the secondary nodes in turn cooperate with each other to update their respective replicas accordingly. The system  60  shown with  FIG. 5  can use at least two alternatives in declaring when a write data  10  transaction has completed at the primary node P. In the first alternative called soft mode, a data modification to the primary data volume V is considered complete as soon as the secondary nodes S 1 -S 3  receive [w 1 +i 1 ], [w 2 +i 2 ], and [w 3 +i 3 ], respectively, from primary computer system  72 . This means the primary nodes do not wait for the secondary nodes to update their respective replicas. In the alternative designated hard mode, the primary node waits for all secondaries S 1 -S 3  to update their replicas R 1 -R 3 , respectively, and become consistent with the primary data volume V. Here the write data IO transaction at the primary node P will be declared complete when all the secondaries S 1 -S 3  receive all modified data blocks w 1 -w 3  from the primary node P and/or each other, and have acknowledged back to the primary node P that the replicas R 1 -R 3  have been updated. 
     For synchronous replication the soft mode may only be suitable if the user is willing to keep the secondaries incomplete for a short duration. For periodic replication, the soft mode may be a bit complex to implement since refresh and restore operations on the primary data volume V and the replicas R 1 -R 3  have to be synchronous. Hard mode may be more useful to asynchronous and periodic replication modes. 
     In contrast to the prior art described within the background section, cooperative replication according to one or more of the embodiments described above reduces the total amount of modified data transmitted directly between the primary node P and secondary nodes S 1 -S 3 . To illustrate, if each of the modified data blocks w 1 -w 3  consist of 64K bits of data, the primary node P will send (3*3*64)K for a total 576K worth of data to the secondary nodes S 1 -S 3 . But with cooperative replication using one of the embodiments described above, primary node P may send only (3*64)K or 192K worth of data (plus some meta data items consisting of instructions having insignificant bit lengths) to the secondaries S 1 -S 3 . As such, cooperative replication may reduce the bandwidth requirements between the primary node P and the secondary nodes S 1 -S 3 . Further, using an embodiment of the present invention, the buffer/memory requirements for transmitting data between the primary and the secondary nodes is reduced. Cooperative replication of the present invention also utilizes links  122 - 126  more when compared to the prior art. Lastly, cooperative replication, according to one or more of the embodiments described above, may reduce the processing bandwidth of primary computer system  72  since primary computer system  72  need not send multiple copies of modified data blocks to each of the secondary nodes S 1 -S 3 . 
     Although the present invention has been described in connection with several embodiments, the invention is not intended to be limited to the specific forms set forth herein. On the contrary, it is intended to cover such alternatives, modifications, and equivalents as can be reasonably included within the scope of the invention as defined by the appended claims.