Abstract:
A storage apparatus capable of copying data to a destination apparatus includes storages including a plurality of volumes for storing data, memories for temporarily storing data to be copied to the destination apparatus, and processors for controlling to copy the data, each processor being configured to manage the memories and parts of the volumes, respectively, and wherein one processor executes storing received write data to one memory and the part, detecting each data amounts stored in the memories when an data amount of one memory is greater than a predetermined amount, allocating management of a part of the part managed by the one processor to other processor when the amount of the data stored in the one memory is greater than an amount calculated by using the data stored in the memories, and transmitting the data stored in the memories to the destination apparatus.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2010-017829, filed on Jan. 29, 2010 the entire contents of which are incorporated herein by reference. 
       FIELD 
       [0002]    The embodiments discussed herein are related to a technique for copying data between storage apparatuses. 
       BACKGROUND 
       [0003]    A storage system records data on a plurality of disks in order to improve the performance of data transfer or the data retention reliability thereof. Even if writing to a disk is stopped suddenly owing to, for example, a crash of an operating system (OS) of a server or the like, the storage system can, after restoration, recover data recorded on a plurality of disks before the stoppage. In order to enable the data recovery, the storage system controls the order in which data is written to a disk. In addition, the storage system may perform data mirroring between a plurality of storage apparatuses. In data mirroring, a source storage apparatus having a volume in which data to be mirrored is stored and a destination storage apparatus having a volume in which mirrored data is to be stored control the order in which data is written to a disk so as to make it possible to recover the data in a shorter period of time if an abnormal condition occurs during a process for copying the data. That is, the storage system copies data in the same order as in writing of the data. In a synchronous copying mode, in which copying is performed in synchronization with writing, a writing process and a copying process are executed in the same order in a storage apparatus. However, there may be a case in which the source storage apparatus and the destination storage apparatus are located at distant places. It is unsuitable to use the synchronous copying mode for mirroring between distant places because the mirroring is affected by delay caused in a transmission line. Therefore, when performing mirroring between distant places, the storage system uses an asynchronous copying mode, which is less affected by the effect of delay. 
         [0004]    In the asynchronous copying mode, in which the order of reading and writing is guaranteed, data written to the source apparatus by a host apparatus is stored in a dedicated buffer (Remote Equivalent Copy (REC) buffer) in a cache of the source apparatus. The dedicated buffer is used for an asynchronous copying function that guarantees the order of reading and writing. In a storage system of a distributed cache memory type, in order to simplify the configuration, a group of redundant arrays of inexpensive disks (RAID) and a volume therein are allocated to one of the controller modules in a storage apparatus and each controller module controls a volume allocated thereto. For this reason, a dedicated buffer is provided for each control module. The area of the dedicated buffer for each controller module is divided into a plurality of generations in a chronological manner. In addition, in order to guarantee the order of reading and writing in asynchronous copying, the generations of each controller module are organized over a plurality of controller modules into a unit called a “buffer set” in advance. The storage system executes a copying process using a buffer set as a unit. When a use ratio of a portion of the dedicated buffer in the cache of the source apparatus to an entire storage area of the dedicated buffer exceeds a predetermined value (including a case in which the entire storage area is used), or when a predetermined period of time has elapsed since data storage began, the controller modules synchronize buffer data in the cache of the source apparatus in order to collectively transmit the buffer data to the destination apparatus. Upon having received all the data that was synchronized and transmitted by the controller modules, the destination apparatus collectively expands the received data in a storage medium therein. 
         [0005]    However, since copying between storage apparatuses uses a buffer set as a unit, volumes in the source apparatus may be intensively associated with a single controller module or repeated updating of data may occur in volumes controlled by a single controller module in the source apparatus. In these cases, data to be copied is intensively stored in a REC buffer of the controller module and accordingly, even when the REC buffers of other controller modules are empty, the data is collectively transmitted in units of a generation. Since the data is transmitted in spite of no data being stored in some dedicated buffers, storage areas of the dedicated buffers are used inefficiently. Japanese Laid-open Patent Publication No. 2005-275525 is an example of the related art. 
       SUMMARY 
       [0006]    According to an aspect of the invention, a storage apparatus capable of copying data to a destination apparatus includes a plurality of storages for storing data, the plurality of storages including a plurality of volumes, a plurality of memories for temporarily storing data to be copied to the destination apparatus, and a plurality of processors for controlling to copy the data, each of the plurality of the processors being configured to manage the plurality of the memories and parts of the plurality of volumes, respectively, wherein at least one of the plurality of the processors executes receiving write data, storing the write data to one of the plurality of the memories and the part managed by the at least one of the plurality of the processors, detecting each amounts of data stored in the plurality of the memories when an amount of the data stored in one of the plurality of the memories is greater than a predetermined amount, allocating management of a part of the part managed by the at least one of the plurality of the processors to other of the plurality of the processors when the amount of the data stored in the at least one of the plurality of the memories is greater than an amount calculated by using the data stored in the plurality of the memories, and transmitting the data stored in the plurality of the memories to the destination apparatus. 
         [0007]    The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
         [0008]    It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0009]      FIG. 1  illustrates a storage system according to an embodiment of the present invention; 
           [0010]      FIG. 2  is an explanatory diagram illustrating a REC buffer controlled by each controller module of a source storage apparatus  20  according to the embodiment; 
           [0011]      FIG. 3  illustrates a buffer index table according to the embodiment; 
           [0012]      FIG. 4  illustrates a session control table according to the embodiment; 
           [0013]      FIG. 5  illustrates a buffer set table according to the embodiment; 
           [0014]      FIG. 6  is a flowchart illustrating a process for storing data in a REC buffer according to the embodiment; 
           [0015]      FIG. 7  is a flowchart illustrating a process performed when a buffer set is switched; 
           [0016]      FIG. 8  illustrates an example of session allocation information transmitted to a controller module CM 01  according to the embodiment; 
           [0017]      FIG. 9  illustrates an example of session allocation information transmitted to a controller module CM 02  according to the embodiment; 
           [0018]      FIG. 10  illustrates an example of session allocation information transmitted to a controller module CM 03  according to the embodiment; 
           [0019]      FIG. 11  illustrates an example of session allocation information transmitted to a controller module CM 04  according to the embodiment; 
           [0020]      FIG. 12  is a flowchart illustrating a process for allocating the number of sessions according to the embodiment; 
           [0021]      FIG. 13  illustrates an example of the number of I/Os obtained from each controller module according to the embodiment; 
           [0022]      FIG. 14  illustrates an example of obtained I/O ratios according to the embodiment; 
           [0023]      FIG. 15  illustrates an example of the number of sessions for each controller module according to the embodiment; 
           [0024]      FIG. 16  illustrates an example of a maximum number of sessions to be allocated to each controller module according to the embodiment; 
           [0025]      FIG. 17  illustrates an example of the determined number of sessions to be distributed to each controller module according to the embodiment; and 
           [0026]      FIG. 18  is a flowchart illustrating a process for determining a REC buffer in which data is stored according to the embodiment. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
       [0027]    Hereinafter, embodiments will be described in detail with reference to drawings. 
         [0028]    Storage System 
         [0029]      FIG. 1  illustrates a storage system according to an embodiment. 
         [0030]    A storage system  1  includes a host apparatus  10 , a storage apparatus  20 , and a storage apparatus  30 . In this embodiment, the host apparatus  10  is connected to the storage apparatus  20  by a network such as a storage area network (SAN), and the storage apparatus  20  is connected to the storage apparatus  30  by a network  40  such as a wide area network (WAN). The storage apparatus  20  and the storage apparatus  30  are, for example, located at places distant from each other. In this embodiment, the host apparatus  10  writes data to the storage apparatus  20  and the data in the storage apparatus  20  is copied into the storage apparatus  30 . That is, the storage apparatus  20  is a source apparatus having data to be subjected to a copying process and the storage apparatus  30  is a destination apparatus in which the data is newly stored through the copying process. In the following description, data to be subjected to the copying process will also be called “copy data”. 
         [0031]    The host apparatus  10  sends an instruction regarding reading or writing of data to the storage apparatus  20  in order to perform reading of data from or writing of data to the storage apparatus  20 . 
         [0032]    Storage Apparatuses 
         [0033]    Next, the storage apparatus  20  and the storage apparatus  30  will be described. For convenience of description, the storage apparatus  20  and the storage apparatus  30  have the same volume configuration in this embodiment. The storage apparatus  20  will be described hereinafter and description of modules in the storage apparatus  20  having the same functions and portions of the storage apparatus  30  similar to those of the storage apparatus  20  is omitted. In addition, the storage apparatus  20 , which is a source apparatus, may be simply called a “source”, and the storage apparatus  30 , which is a destination apparatus, may be simply called a “destination” in the following description. 
         [0034]    The storage apparatus  20  and the storage apparatus  30  in this embodiment are of a distributed cache memory type in which each controller module (hereinafter may be referred to as CM) has a cache memory, respectively. Therefore, a volume, which is a control unit of a disk apparatus in which data is stored, is preset to belong to any one of CMs in a storage apparatus. A volume includes a logical volume that exists logically and that is defined by a technique such as combining two or more physically independent volumes, that is, for example, by organizing storage areas of each disk apparatus, by configuring a RAID from volumes, or by combining RAIDs. The copying process from the storage apparatus  20  to the storage apparatus  30  can be performed even when the RAID configurations of the source and destination storage apparatuses are different, so long as an area for storing the data to be copied can be reserved in the destination storage apparatus  30 . 
         [0035]    The storage apparatus  20  has a plurality of controller modules CM 01 , CM 02 , CM 03 , and CM 04 . Storages  250 ,  260 ,  270 , and  280  are, for example, magnetic disks that store data. A back-end router (BRT)  240  serves as a connection between disks ( 250 ,  260 ,  270 , and  280 ) and the controller modules (CM 01 , CM 02 , CM 03 , and CM 04 ). Each controller module can perform reading (READ) of data from and writing (WRITE) of data to each disk. 
         [0036]    The storage apparatus  20  makes any of the CMs therein function as a master CM. In the following description, a CM that controls each CM of the storage apparatus  20  is called a “master CM” and CMs other than the master CM are simply called “CMs”. The master CM has a function of an ordinary CM as well as a function of controlling the other CMs. In this embodiment, the controller module CM 01  is a master CM. Upon a generational change, the master CM obtains input/output (I/O) information of REC buffers from the controller modules CM 01 , CM 02 , CM 03 , and CM 04  of the storage apparatus  20  with which the REC buffers are each associated, so as to execute a process for determining which volumes are to be associated with each CM in the next generation. 
         [0037]    The controller module CM 01  (CM 02 , CM 03 , or CM 04 ) has a channel adapter (CA)  201  ( 211 ,  221 , or  231 ), a remote adapter (RA)  202  ( 212 ,  222 , or  232 ), a memory  204  ( 214 ,  224 , or  234 ), a central processing unit (CPU)  203  ( 213 ,  223 , or  233 ), and device adapters (DAs)  205  and  206  ( 215  and  216 ,  225  and  226 , or  235  and  236 ). The CA  201  is an interface for connecting the host apparatus  10  and the storage apparatus  20 . The RA  202  is an interface for connecting to an RA  302  of the storage apparatus  30 . Each RA is used for, for example, transferring data during a copying process between storage apparatuses. The DAs are interfaces for connecting the BRT  240  and the CMs. 
         [0038]    The memory  204  includes a REC buffer that is an area for storing buffer data. The memory  204  stores a buffer index table (BIT table), a buffer set control table, and a session control table. In addition, the memory includes an area for storing a control program to be operated by the CPU  203 . 
         [0039]    The CPU  203  controls the entire operation of the controller module CM 01 , such as disk control for controlling reading data from and writing data to a disk apparatus, CA control for controlling transmission and reception of information to/from the host apparatus  10 , cache memory control for controlling the memory  204 , copying process control for controlling the copying process, and a request for a copying process to another CM. 
         [0040]    Description of controller modules CM 11 , CM 12 , CM 13 , and CM 14 , a BRT  340 , disks  350 ,  360 ,  370 , and  380  in the storage apparatus  30 , which is the destination in this embodiment, is omitted. In addition, description of a CA  301  ( 311 ,  321 , or  331 ), an RA  302  ( 312 ,  322 , or  332 ), a memory  304  ( 314 ,  324 , or  334 ), a CPU  303  ( 313 ,  323 , or  333 ) and fibre channels (FCs)  305  and  306  ( 315  and  316 ,  325  and  326 , or  335  and  336 ) included in the controller module CM 11  (CM 12 , CM 13 , or CM 14 ) is also omitted. 
         [0041]    I/O data is data to be read or written that is received from the host apparatus  10 . The I/O data includes main data and copying control information for controlling copying of data. The copying control information includes addresses of a source and a destination. 
         [0042]    When the I/O data is received, the main data is stored in a REC buffer and the copying control information is stored in a BIT table. 
         [0043]    When securing the order of a process for reading or writing data, the storage system  1  uses a REC buffer to transfer data to be copied from a disk apparatus in the source storage apparatus  20  to a disk apparatus in the destination storage apparatus  30 . For example, data is copied from the source into the destination in the following steps. First, a CM of the source storage apparatus  20  that controls a volume for copying stores copy data from a disk apparatus in a REC buffer. When any of REC buffers of a generation to be processed in a buffer set is filled up, or a predetermined period of time has elapsed since the copy data was stored in the REC buffer, the source collectively transfers the copy data in the buffer set to a REC buffer in the destination storage apparatus  30 . The destination storage apparatus  30  stores the received copy data in the REC buffer. The destination storage apparatus  30  stores all the copy data in a dedicated buffer and then stores all the copy data in a recording medium  12  thereof. When the expansion of the copy data has been completed, the destination storage apparatus  30  notifies the source storage apparatus  20  of the completion of the expansion. After that, the source storage apparatus  20  releases the copy data in the REC buffer. 
         [0044]    Next, information included in a REC buffer, a BIT table, a buffer set control table, and the like that are stored in a memory will be described. 
         [0045]    REC Buffer 
         [0046]    First, a REC buffer (dedicated buffer) is described. A REC buffer is included in each CM and is used as a storage area for storing information necessary for a copying process during a copying process executed between storage apparatuses (Remote Equivalent Copy (REC)). In this embodiment, main data to be copied is stored in REC buffers. 
         [0047]      FIG. 2  is an explanatory diagram illustrating REC buffers controlled by the CMs of the source storage apparatus  20 . A REC buffer area CM# 01  of the controller module CM 01 , a REC buffer area CM# 02  of the controller module CM 02 , a REC buffer area CM# 03  of the controller module CM 03 , and a REC buffer area CM# 04  of the controller module CM 04  are illustrated. Each CM controls the corresponding REC buffer by dividing the REC buffer into a plurality of areas in order to guarantee the order of processes for reading and writing data. Each divided area is controlled as a generation. In  FIG. 2 , each REC buffer is divided into eight generations in total, namely a generation  1 , generation  2 , generation  3 , generation  4 , and four other unused generations. Each generation is controlled in a chronological manner. The REC buffer of each CM is divided into the same number of areas. Each generation is released after completion of the copying process between storage apparatuses. “Released” refers to a condition in which data can be stored in a generation. A generation is further divided into a plurality of fixed-length buffers. A CM stores data by, for example, appropriately dividing the data so that the divided pieces of data can be stored in the fixed-length buffers. 
         [0048]    CMs to which volumes  301 ,  302 ,  303 ,  304 , and  305  belong are defined in advance. During the copying process between storage apparatuses, each CM in the source reads data in an associated volume to the REC buffer thereof. 
         [0049]    For example, in  FIG. 2 , the volumes  301 ,  302 , and  303  belong to the controller module CM 01 . Therefore, upon receiving a write instruction (write I/O) for any of the volumes  301 ,  302 , and  303  from the host apparatus  10  between storage apparatuses, the controller module CM 01  stores data corresponding to the write instruction in a buffer area currently serving as a storage in a REC buffer corresponding to the write instruction. 
         [0050]    In addition, suppose that, for example, the host apparatus  10  does not issue a write instruction to any volume, which is not illustrated, other than the volumes  301 ,  302 ,  303 ,  304 , and  305  in  FIG. 2 . In this case, when a process that will be described later is not applied, the controller modules CM 02  and CM 04  do not store data in the REC buffers controlled thereby. Each CM controls the corresponding REC buffer. 
         [0051]    In order to collectively control the REC buffers of the respective CMs, copying between storage apparatuses is executed using a buffer set  306  as a unit. In  FIG. 2 , a set of buffers surrounded by a broken line is a buffer set. The buffer set  306  is a group of data that is organized by generations, which have been obtained by chronologically dividing the REC buffer of each CM, of a plurality of CMs. By executing the copying process using a buffer set as a unit, the order of data writing can be maintained in the REC buffers that are included in the buffer set and are associated with a plurality of CMs. 
         [0052]    In  FIG. 2 , a generation of the controller modules CM 02  and CM 04  that belongs to the buffer set  306  is used as a buffer set in spite of no data being stored therein. On the other hand, since the controller module CM 01  is associated with the volumes  301 ,  302 , and  303  that have received the write instruction, there is a lot of data to be stored in the REC buffer thereof. For this reason, an area of the generation of the REC buffer of the controller module CM 01  is liable to be full. When any of the REC buffers included in a buffer set becomes full, the storage apparatus  20  switches the generation of the REC buffers that serves as a storage so as to store data in the next generation. In  FIG. 2 , when a REC buffer of the generation  4  that is currently being used becomes full, the storage apparatus  20  creates a generation  5  from the unused generations of the REC buffers. The storage apparatus  20  also specifies the buffer set of the generation  4  as a target of transfer to the storage apparatus  30 . 
         [0053]    BIT Table 
         [0054]    Next, a BIT table is described.  FIG. 3  illustrates a BIT table. Each CM has a BIT table. The BIT tables store information for copying data stored in an individual buffer in a generation of each CM from a source to a destination. The information in a BIT table can be identified with a CM ID and a BIT ID stored in a buffer set table. 
         [0055]    During the copying between storage apparatuses, the source transmits copy data from a REC buffer and information from a BIT table to the destination. As a result, the destination can store the copy data in the REC buffer in an appropriate area. 
         [0056]    A BIT table includes a buffer ID, status, size, buffer address, and the number of I/Os stored. The buffer ID is identification information for identifying a buffer to be controlled. The status is information for identifying the condition of the buffer, such as storing, segmenting, segmentation complete, or transferring. The size is information for identifying the size of the buffer. The buffer address is information for identifying an actual address of data contained in the buffer, such as volume identifiers of the source and the destination and a logical block address in a volume. 
         [0057]    The number of I/Os stored is information indicating the number of pieces of write I/O data that have been copied to a generation that is currently being processed in the REC buffers. The initial value of the number of I/Os stored is 0. When the generation of the buffer set is switched, each CM transmits the number of I/Os stored corresponding to the switched buffer to the master CM. 
         [0058]    Session Control Table 
         [0059]    A session control table that controls REC sessions is described here. A session is information in which the source and the destination are related.  FIG. 4  illustrates a session control table. Each CM stores a session control table. For example, an administrator sets a combination between volumes in the source and the destination before starting copying. Each CM retains information of a combination in the memory thereof as a session control table. 
         [0060]    The session control table contains information regarding a copying session and therefore exists for each session. The information regarding a copying session includes a session ID, session type, source volume, destination volume, copying range, and storage CM. The session ID is unique identification information that exists for each session. The session type is information indicating a copying type such as remote copying or local copying. The source volume is information including a volume number of the source and an initial logical block address (LBA) of the source. The destination volume is information including a volume number of the destination and an initial logical block address of the destination. The copying range is information indicating the range of the copying process. For example, the range of the copying process is indicated by the number of blocks. 
         [0061]    The storage CM is information for identifying a CM that controls a REC buffer in which a session is to be stored. Since a REC buffer is divided into a plurality of generations and controlled, the storage CM has the number of members corresponding to the number into which the REC buffer is divided. In this embodiment, each REC buffer is divided into eight, and accordingly the storage CM in the session control table includes eight members. Upon a generational change, each CM initializes each member. When a generation receives a write I/O of copy data regarding a session whose storage REC buffer has not been determined, each CM determines the storage REC buffer and specifies a CM that controls the determined REC buffer as the storage CM. The storage CM includes “Storage CM Information [0]” that indicates a storage CM associated with a first area of the REC buffer that has been divided into eight and “Storage CM Information [1]” that indicates a storage CM associated with a second area of the REC buffer that has been divided into eight. The storage CM also includes “Storage CM Information [2]” to “Storage CM Information [7]” that indicate associated storage CMs in the same manner as above. 
         [0062]    Buffer Set Table 
         [0063]    Next, a buffer set table controlled by the master CM is described.  FIG. 5  illustrates a buffer set table. A buffer set table controls a buffer set and more particularly controls a combination of REC buffer between the source and the destination. A buffer set is a group of the REC buffers of CMs and includes all the REC buffers that form the buffer set. 
         [0064]    Since each REC buffer is divided into eight generations in this embodiment, each CM has eight buffer set tables corresponding to the number of generations. The generations of a buffer set are controlled in a chronological manner. 
         [0065]    A buffer set table contains a CM ID, a BIT ID, the number of sessions allocated, information of the number of I/Os, and next storage CM information for each CM included in a buffer set. 
         [0066]    The CM ID is information for identifying a CM that controls a buffer. The BIT ID is information for identifying a buffer. 
         [0067]    The number of sessions allocated is set for a CM when a buffer set is newly created. Upon the initial write I/O process in a generation, each CM refers to this member (also to those of the other CMs) and “Next Storage CM Information” to determine a REC buffer for storing data. The number of I/Os indicates the number of I/Os stored in a buffer of the previous generation about which notification is performed by each CM upon switching of a buffer set. The next storage CM information is information for determining a REC buffer that serves to store a session of a CM in the initial write I/O in a generation. At the time when the master CM creates a buffer set table related to a new generation, an initial value (invalid value) is set to the buffer set table as the number of sessions allocated. The number of sessions allocated is updated when the initial write I/O in a generation is processed in a session of a CM. 
         [0068]    Next, the copying process between storage apparatuses is described. 
         [0069]    Process for Storing Data in REC Buffer 
         [0070]    First, a process for storing data in a REC buffer is described in accordance with a flowchart of  FIG. 6  illustrating the process for storing data in the REC buffer. Any of the controller modules CM 01  to CM 04  performs the process for storing data in the REC buffer thereof. 
         [0071]    A CM receives a write instruction (write I/O) from the host apparatus  10  (S 01 ). The write I/O contains information such as address information of a volume to which data will be written, size information of the write data, and main data. 
         [0072]    The CM determines a REC buffer for storing the write I/O (S 02 ). A process for determining the REC buffer in S 02  will be described in detail later. 
         [0073]    After the REC buffer for storing the write I/O is determined, the main data is written to a disk apparatus corresponding to the address information of the volume identified with the write I/O received in S 01  (S 03 ). The main data may be written to a cache memory for temporarily storing data, instead of the disk apparatus. 
         [0074]    After the main data has been written in S 03 , the CM outputs a reply to the write instruction to the host apparatus  10  (S 04 ). The data written in the writing process is then stored in the REC buffer determined in S 02  (S 05 ). The processes in S 04  and S 05  do not need to be synchronized. 
         [0075]    After the process for storing data to the REC buffer in S 05 , the CM that controls the REC buffer to which the data has been written increments the number of I/Os in the BIT table by 1 (S 06 ). In addition, the CM that controls the REC buffer to which the data has been written writes the buffer ID, status, size, buffer address, and the number of I/Os stored to the BIT table. 
         [0076]    Commission of Storing Data to Another CM 
         [0077]    The data expansion in S 05  is described here. In this embodiment, each CM may store not only data in the REC buffer controlled thereby, but also data in a REC buffer controlled by another CM. 
         [0078]    An example of a process that is performed when a CM stores copy data in a REC buffer controlled by another CM is described here. First, a CM queries another CM (a CM that controls a REC buffer in which data will be stored) as to whether the another CM can reserve an area large enough to store copy data. The CM executes the query by, for example, transmitting the size of the copy data to the other CM. The other CM reserves an area having the size notified through the query. The other CM then replies by notifying the querying CM of an area corresponding to the size having been reserved in the REC buffer. The CM transmits the copy data to the other CM. The other CM stores the copy data received from the CM in the REC buffer thereof. 
         [0079]    When a CM stores data input to a volume associated therewith in a REC buffer thereof, the CM secures an area of the REC buffer and stores the data in the area. Thus, a process for storing data in its own REC buffer of a CM is simpler than a process for storing copy data input to a volume associated with a CM in a REC buffer of another CM. 
         [0080]    Buffer Set Switching Process 
         [0081]    Next, a process for switching a buffer set is described in accordance with a flowchart of  FIG. 7  illustrating the process for switching a buffer set. When a buffer set is switched, transmission and reception of information between the master CM (CM 01 ) and the other CMs (CM 02 , CM 03 , and CM 04 ) occur. The master CM determines the number of sessions associated with each CM in the next generation upon the switching of the buffer set. 
         [0082]    In this embodiment, sessions associated with each CM are changed upon a generational change. When individual pieces of copy data that belong to a single session are stored in a plurality of REC buffers while the same generation continues, each CM in the destination needs to perform a process for determining the order of I/Os between the REC buffers, which is inefficient. 
         [0083]    The master CM receives a request to switch the generation of the buffer set (S 11 ). The generation of a buffer set is switched when a predetermined period of time has elapsed since the process of the current generation began, or when the storage area of the current generation in the REC buffer of any of the controller modules CM 01  to CM 04  is filled up with copy data. In this embodiment, a request to switch the generation of the buffer set may be received upon detection of a predetermined period of time having elapsed since the current generation began, or upon notification that the storage area of the current generation in the REC buffer of any of the controller modules CM 01  to CM 04  is filled up with copy data. 
         [0084]    When receiving a request to switch the generation in S 11 , the master CM requests each CM (CM 01 , CM 02 , CM 03 , or CM 04 ) to stop the process for storing data in the corresponding REC buffer (S 12 ). At this time, the master CM transmits an instruction to stop the process for storing copy data in the REC buffers to all the CMs that form the buffer set. 
         [0085]    Operation of the other CMs is described here. Before the master CM executes a process for switching the buffer set, each CM (CM 01 , CM 02 , CM 03 , or CM 04 ) is in a condition in which the process for storing data in the corresponding REC buffer can be performed. The other CMs (CM 02 , CM 03 , and CM 04 ) execute data storage in the REC buffers in accordance with a write I/O from the host apparatus  10  (S 21 ). In S 12 , the master CM requests each CM (CM 01 , CM 02 , CM 03 , or CM 04 ) to stop the storing process. At this time, each CM obtains the request to stop the storage process from the master CM. 
         [0086]    Each CM determines whether or not a request to stop the storage has been detected (S 22 ). If a request to stop the storage has not been detected (S 22 : NO), each CM continues to execute the storage process. If a request to stop the storage has been detected (S 22 : YES), each CM stops further storage of data in the current generation (S 23 ). After stopping the storage process of S 23 , each CM transmits a reply to the request to stop the storage and the number of I/Os of data stored in the REC buffer of the current generation, which has served as a storage of each CM, to the master CM by reading the number of I/Os from the BIT table. 
         [0087]    On the other hand, the master CM executes processes of S 13  and S 14  as the controller module CM 01  in a similar manner as the other CMs (CM 02 , CM 03 , and CM 04 ). The master CM (CM 01 ) determines whether or not a request to stop the storage has been detected (S 13 ). If a request to stop the storage has not been detected (S 13 : NO), the master CM continues to wait for a request to stop the storage to be obtained. If a request to stop the storage has been detected (S 13 : YES), the master CM (CM 01 ) stops further storage of data in the current generation (S 14 ). 
         [0088]    The master CM records the number of I/Os of the previous generation served as a storage that is obtained from each CM on the buffer set table of the new generation as the number of I/Os of each CM (S 15 ). The master CM then determines whether or not a reply to the request to stop the storage has been obtained from each CM (S 16 ). If a reply to the request to stop the storage has not been obtained from each CM (S 16 : NO), the master CM waits for a reply from each CM. On the other hand, if a reply to the request to stop the storage has been obtained from each CM (S 16 : YES), the master CM calculates the number of sessions to be allocated to each CM for a buffer set of the new generation (new buffer set) (S 17 ). A process for allocating the number of sessions will be described later. 
         [0089]    The master CM transmits an instruction to switch to the new buffer set to each CM (S 18 ). Along with the switching instruction of S 18 , the master CM also transmits the buffer set table including updated session allocation information to each CM.  FIGS. 8 to 11  illustrate examples of the session allocation information transmitted to the controller modules CM 01  to CM 04 , respectively.  FIG. 9  illustrates that the number of sessions that the controller module CM 02  stores in the REC buffer of the controller module CM 01  is 6 and the number of sessions that the controller module CM 02  stores in the REC buffer of the controller module CM 02  itself is 14. 
         [0090]    Operation of the other CMs is described here. In S 18 , the master CM transmits the instruction to switch to the new buffer set to each CM. At this time, each CM executes a process for switching the generation in accordance with the received switching instruction (S 24 ). Each CM stores the session allocation information transmitted from the master CM. On the other hand, the master CM executes a process for switching the generation as the controller module CM 01  in a similar manner as the other CMs (CM 02 , CM 03 , and CM 04 ) (S 19 ). The master CM (CM 01 ) stores the session allocation information. Each CM transmits a reply to the request to switch the generation to the master CM. 
         [0091]    The master CM determines whether or not a reply to the request to switch the generation has been obtained from each CM (S 20 ). If a reply to the request to switch the generation has not been obtained from each CM (S 20 : NO), the master CM waits for a reply from each CM. If a reply to the request to switch the generation has been obtained from each CM (S 20 : YES), the master CM terminates the switching process. Each CM restarts a process for storing copy data in areas of a new generation. 
         [0092]    Process for Allocating Number of Sessions 
         [0093]    Next, a process of S 17  for allocating the number of sessions to the REC buffer of each CM in the new buffer set is described in accordance with a flowchart of  FIG. 12  illustrating the process for allocating the number of sessions. In this embodiment, allocation of the number of sessions to CMs is determined in units of sessions. How the number of sessions are to be allocated in the new buffer set is determined on the basis of the situation of data storage in the previous buffer set that has served as a storage (the buffer set of the previous generation). The master CM determines how the number of sessions are to be allocated on the basis of the number of I/Os obtained from each CM. 
         [0094]    The reason why the number of sessions to be allocated is determined on the basis of the number of I/Os is that it is possible to suppose that the number of I/Os and the number of sessions are in proportion to each other. In addition, since the allocation of the number of sessions is performed upon a generational change, it is possible to distribute volumes associated with each controller module in accordance with the situation of data storage in the previous REC buffers. The master CM calculates the ratio of I/Os of each CM on the basis of the buffer set of the previous generation (S 31 ).  FIG. 13  illustrates the number of I/Os obtained from each CM. In  FIG. 13 , the number of I/Os of the controller module CM 01  is 0, the number of I/Os of the controller module CM 02  is 500, the number of I/Os of the controller module CM 03  is 200, and the number of I/Os of the controller module CM 04  is 300. The master CM detects, for example, the smallest number of I/Os from among the numbers of I/Os more than 0. The controller module CM 03  having a number of I/Os of 200 is the one in the case of  FIG. 13 . The master CM calculates an I/O ratio by dividing the number of I/Os of each another CM by the detected number of I/Os. Decimals are rounded up here.  FIG. 14  illustrates the obtained I/O ratios. 
         [0095]    The master CM calculates the sum of the I/O ratio of each CM obtained in S 31  and divides the sum by the number of CMs included in the buffer set in order to obtain the average number of I/Os per CM (S 32 ). For example, the sum of the I/O ratio of each CM illustrated in  FIG. 14  is obtained by an expression 0+3+1+2=6. The master CM divides the obtained sum by the number of CMs included in the buffer set, which is 4. Decimals are rounded up here. As a result, an average number of the I/O ratios of 2 is obtained. 
         [0096]    Next, the master CM divides the total number of sessions by the sum of the I/O ratios in order to obtain the number of sessions per I/O ratio (S 33 ). The master CM obtains the number of sessions of each CM from the session control tables.  FIG. 15  illustrates an example of the number of sessions of each CM. The sum of the number of sessions of each CM illustrated in  FIG. 15  is obtained by an expression 0+20+8+10=38. The master CM divides the sum of the number of sessions of 38 by the sum of the I/O ratios of 6 in order to obtain the number of sessions per I/O ratio, which is 7. 
         [0097]    The master CM multiplies the number of sessions per I/O ratio obtained in S 33  and the average number of the I/O ratio of each CM obtained in S 32  in order to obtain the number of sessions to be allocated to each CM (S 34 ).  FIG. 16  illustrates a maximum number of sessions to be allocated to each CM. 
         [0098]    Next, the master CM determines whether or not the process for allocating the number of sessions is being performed for the first time (S 35 ). If the process for allocating the number of sessions is being performed for the first time (S 35 : YES), the master CM uses the number of sessions associated with each CM as the session allocation information (S 36 ). If the process for allocating the number of sessions is being performed for the second time or more (S 35 : NO), the master CM uses the session allocation information that was used in the previous generation (S 37 ). 
         [0099]    The master CM then determines the number of sessions to be allocated to each REC buffer of the new generation on the basis of the average number of the sessions to be allocated to each CM determined in S 34  and the session allocation information of the previous generation determined in S 35  to S 37  (S 38 ). An example is described hereinafter. For example, the average number of sessions to be allocated to each CM is 14. If the process for allocating the number of sessions is being performed for the first time, the master CM uses the number of sessions associated with each CM illustrated in  FIG. 15  as the session allocation information. The master CM distributes the number of sessions to each CM in such a manner that the maximum number of sessions to be allocated, which is 14, is not exceeded. According to the number of sessions associated with each CM illustrated in  FIG. 15 , the number of sessions of the controller module CM 02  is 20, which is larger than the average number of 14. Therefore, the master CM restricts the number of sessions associated with the controller module CM 02  so that the number of sessions does not exceed 14. As a result of the restriction, there are 6 sessions left. The master CM distributes these sessions to a controller module to which the smallest number of sessions has been distributed. The distribution of the number of sessions has thus been completed.  FIG. 17  illustrates the determined number of sessions to be distributed. The master CM determines the number of sessions to be allocated to each CM of the next generation in the above-described manner. After the determination, the master CM executes the process of S 18  and the subsequent processes. 
         [0100]    Process for Determining Storage REC Buffer Executed by Each CM 
         [0101]    Next, a process for determining a storage REC buffer that is executed by each CM is described in accordance with a flowchart of  FIG. 18  illustrating the process for determining the storage REC buffer. 
         [0102]    The process for determining the storage REC buffer is executed by each CM (CM 01 , CM 02 , CM 03 , and CM 04 ) of the storage apparatus. 
         [0103]    Each CM has a session control table. Each CM initializes storage CM information in the session control table before S 41  is performed after a generational change. 
         [0104]    Each CM also has a buffer set table. Each CM initializes next storage CM information in the buffer set table before S 43  is performed after a generational change. 
         [0105]    Any CM (CM 01 , CM 02 , CM 03 , or CM 04 ) in the source receives a write I/O transmitted from the host apparatus  10  to volumes associated therewith. A CM that has received a write I/O is called a “main CM” hereinafter. 
         [0106]    The main CM determines whether or not the received I/O is the initial I/O in a generation that is currently serving a storage process on the basis of the storage CM information in the session control table corresponding to the generation that is currently serving a storage process (S 41 ). If the storage CM information has been set, that is, if the received I/O is not the initial I/O (S 41 : NO), the main CM transmits a request to obtain a storage area to the REC buffer of a CM for which it has already been determined which REC buffer the sessions are to be stored in (S 42 ). If the storage CM information has not been set, that is, the received I/O is the initial I/O (S 41 : YES), the main CM determines whether or not the next storage CM information in the buffer set table is invalid (whether or not the next storage CM information has been set) (S 43 ). 
         [0107]    If the next storage CM information has been set, that is, if it is determined in S 43  that the next storage CM information is not invalid (S 43 : NO), the main CM decreases the number of sessions to be allocated to a storage CM whose buffer set table has been set (S 44 ). After that, the main CM executes a process of S 48 . On the other hand, if the next storage CM information in the buffer set table has not been set, that is, if it is determined in S 43  that the next storage CM information is invalid (S 43 : YES), since it is a session using a combination between a source volume and a destination volume that has not been previously used in the generation, it is determined whether or not the number of sessions to be allocated to the main CM is 1 or more by checking a buffer set table of the generation that is serving as a storage for the number of sessions to be allocated to each CM (S 45 ). If it is determined in S 45  that the number of sessions to be allocated to the main CM is 1 or more (S 45 : YES), the number of sessions to be allocated to the main CM is decreased by 1 (S 46 ). On the other hand, if it is determined in S 45  that the number of sessions to be allocated to the main CM is less than 1 (S 45 : NO), another CM to which the number of sessions of 1 or more is allocated is selected and the number of sessions to be allocated to the selected CM is decreased by 1 (S 47 ). In a process according to this embodiment, the number of sessions to be allocated to the main CM is preferentially used. It is also possible to alternately distribute the number of sessions to the main CM and the other CMs. 
         [0108]    After determining how to allocate the number of sessions, the main CM sets the storage CM information of the generation determined in S 44 , S 46 , or S 47  as the storage CM information of the corresponding sessions in the session control table (S 48 ). 
         [0109]    The main CM updates the next storage CM information in the buffer set table of the generation that is serving as a storage (S 49 ). For example, the main CM searches for a CM to which a number of sessions of 1 or more has been allocated from among CMs other than the one set in S 48 , and sets the found CM as the next storage CM information. In addition, when the number of sessions to be allocated to the main CM and those to other CMs are all 0, the main CM sets itself as the next storage CM information. The main CM then transmits a request to obtain a storage area to the REC buffer of the determined CM (S 50 ). 
         [0110]    Release of Generation 
         [0111]    The source prevents exhaustion of a buffer area by changing the generation of the buffer set that has completed copying to an unused buffer. When a process for storing data contained in a generation of a buffer set that is subjected to the processing in a disk apparatus has been completed, the master CM in the destination transmits information regarding the completion of the copying in the generation to the source. Upon receiving the information regarding the completion of the copying from the destination, the master CM in the source changes the generation corresponding to the notification of the completion to an unused generation. 
         [0112]    A storage apparatus according to this embodiment can be applied to a storage system that stores data. The storage apparatus changes volumes associated with each controller module in such a way as to reduce bias in the amount of use of buffers used for copying between storages apparatuses and, accordingly, makes it possible to use the buffer of each controller module with less loss. As a result, the storage apparatus disclosed herein can efficiently execute a copying process. 
         [0113]    All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present inventions have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.