Backup data management system and backup data management method

Provided are a backup data management system and a backup data management method capable of facilitating the management of backup data that is multiplexed between different storage apparatuses. The backup data management system includes a storage apparatus having a volume to be used by a host computer, at least one storage apparatus having volumes, and a management computer. The management computer creates a copy pair so that a snapshot of the volume included in a backup group is stored in all the volumes included in the backup group, and sets all copy pairs included in the designated backup group to a PAIR status when there is a creation request of the snapshot of the volume.

This application relates to and claims priority from Japanese Patent Application No. 2008-056907, filed on Mar. 6, 2008, the entire disclosure of which is incorporated herein by reference.

BACKGROUND

The present invention relates to a backup data management system and a backup data management method for managing backup data that is multiplexed between different storage apparatuses in a computer system comprising a plurality of storage apparatuses.

Conventionally, a highly reliable memory apparatus (hereinafter referred to as a “storage apparatus”) has a storage area (hereinafter referred to as a “volume”) in the storage apparatus, and retains a local copy program for creating a replication (hereinafter referred to as a “snapshot”) of the volume in another volume of the storage apparatus.

Nevertheless, there are cases where the storage apparatus becomes unavailable due to failure caused by water damage or the like, or maintenance work. Thus, demanded is a highly available storage system that multiplexes a snapshot of a volume used by a business application between different storage apparatuses so that, when the storage apparatus retaining that volume becomes unavailable, the business application is able to continue business operations with the snapshot of the other storage apparatus, and to enable the restoration (hereinafter referred to as “restore” or “restoration”) of data to the volume used by the business application.

As a method of multiplexing snapshots between different storage apparatuses (hereinafter simply referred to as a “snapshot multiplexing method”), disclosed is a method of remote-copying a volume used by a business application stored in a first storage apparatus to a second storage apparatus, the first storage apparatus local-copying the volume used by the business application in the first storage apparatus, the second storage apparatus local-copying the remote-copied volume in the second storage apparatus, and coinciding the data content and created time of the snapshots created in the respective storage apparatuses (refer to Japanese Patent Laid-Open Publication No. 2003-242011).

As another snapshot multiplexing method, disclosed is a method of local-copying a volume used by a business application stored in a first storage apparatus to the first storage apparatus, and replicating the local-copied volume in a volume of a second storage apparatus via remote copy (refer to Japanese Patent Laid-Open Publication No. 2006-48300).

SUMMARY

Nevertheless, the configuration between the storage apparatuses (hereinafter simply referred to as the “inter-storage apparatus configuration”) in which the snapshot is to be multiplexed; for instance, the network configuration between the storage apparatuses and the volume configuration in the storage apparatus to become the snapshot creation source and the volume configuration in the storage apparatus to become the snapshot creation destination, is diverse. Generally, since there are a plurality of inter-storage apparatus configurations in a storage system, the snapshot multiplexing method to be adopted will differ depending on the inter-storage apparatus configuration, and the method adopted in the snapshot creation routine or restoration routine will also differ for each snapshot multiplexing method.

Accordingly, upon creating a snapshot in a storage system having a plurality of inter-storage apparatus configurations, a storage administrator in charge of operating and managing snapshots needs to select the snapshot multiplexing method to be adopted in each inter-storage apparatus configuration, and configure settings based on the snapshot creation routine of the selected snapshot multiplexing method, and there is a problem in that the processing for managing snapshots is cumbersome.

The present invention was made in view of the foregoing problems. Thus, an object of the present invention is to provide a backup data management system and a backup data management method capable of facilitating the management of backup data that is multiplexed between different storage apparatuses.

In order to achieve the foregoing object, the present invention proposes a backup data management system comprising a first storage apparatus connected to a computer and having a first storage area to be used by the computer, at least one second storage apparatus connected to the first storage apparatus and having a second storage area, and a backup data management computer connected to the first storage apparatus and the second storage apparatus. The backup data management computer includes a backup group creation unit configured to create a backup group including the first storage area and at least one second storage area, a copy pair creation unit configured to create a copy pair of a copy source and a copy destination forming a pair so that backup data of the first storage area included in the backup group is stored in all second storage areas included in the backup group, and a copy command unit configured to set all copy pairs included in the backup group to a copy status when there is a creation request of backup data of the first storage area designating the backup group.

In order to achieve the foregoing object, the present invention additionally proposes a backup data management method to be used in a backup data management system comprising a first storage apparatus connected to a computer and having a first storage area to be used by the computer, at least one second storage apparatus connected to the first storage apparatus and having a second storage area, and a backup data management computer connected to the first storage apparatus and the second storage apparatus. The backup data management methods comprises a backup group creation step for creating a backup group including the first storage area and at least one second storage area, a copy pair creation step for creating a copy pair of a copy source and a copy destination forming a pair so that backup data of the first storage area included in the backup group is stored in all second storage areas included in the backup group, and a copy command step for setting all copy pairs included in the backup group to a copy status when there is a creation request of backup data of the first storage area designating the backup group.

According to the present invention, since a copy pair of a copy source and a copy destination forming a pair is created so that backup data of the first storage area included in a backup group is stored in all second storage areas included in that backup group, and all copy pairs included in the backup group is set to a copy status when there is a creation request of backup data of the first storage area designating that backup group, backup data of the first storage area included in the designated backup group can be stored in all second storage areas included in the designated backup group.

According to the present invention, since backup data of the first storage area included in the designated backup group can be stored in all second storage areas included in the designated backup group, the backup data to be duplicated between the first storage apparatus and the second storage apparatus can be consolidated with only the backup group, and the management of backup data to be multiplexed between different storage apparatuses can be facilitated thereby.

DETAILED DESCRIPTION

An embodiment of the present invention is now explained in detail with reference to the attached drawings.

The configuration of the present invention is foremost explained with reference toFIG. 1toFIG. 9. As shown inFIG. 1, the computer system comprises a host computer100, storage apparatuses200A,200B, and a management computer300to be used by a storage administrator A. The host computer100and the storage apparatuses200A,200B are connected via a storage network N1using an FC (Fibre Channel) protocol, and the host computer100and the storage apparatuses200A,200B and the management computer300are connected via a management network N2using an IP protocol.

As shown inFIG. 2, the host computer100comprises a processor110, a memory120, one or more network I/Fs130for connecting to the storage network N1, a management port140for connecting to the management network N2, an input unit150such as a keyboard or a mouse, and an output unit160such as a display apparatus for outputting the processing result, and the respective components are mutually connected via an internal bus170.

A business application121that uses a volume235described later via the storage network N1and for performing processing associated with the I/O issuance to the volume235, a configuration information notification program122for notifying the configuration information of the host computer100, and an OS (operating system; not shown) are loaded into the memory120from a storage medium (not shown) such as a hard disk, and are executed by the processor110.

As shown inFIG. 3, the storage apparatus200A comprises a controller210for controlling the respective components in the storage apparatus200A, a memory220for storing programs and management information retained in the storage apparatus200A, a disk apparatus230including disks231as storage mediums configured from a hard disk or the like, one or more network I/Fs240for connecting the storage network N1, a management port250for connecting to the management network N2, and a cache memory260for temporarily retaining a write command for writing data into the volume235described later, and the respective components are connected to the controller210via an internal path270.

A control program221for managing the configuration of the storage apparatus200A, a local copy program222for replicating data of the volume235described later and a write command for writing data into such volume235in another volume235of the storage apparatus200A, and a remote copy program223for replicating data of the volume235and a write command for writing data into such volume235in a volume235of another storage apparatus200B are loaded into the memory220from a storage medium (not shown) such as a hard disk when the storage apparatus200A is booted, and are executed by the controller210.

The control program221receives commands to be issued to the functions of the storage apparatus200A from the management computer300and executes the local copy program222and the remote copy program223and sets the volumes235and the copy pairs. The control program221notifies the configuration of the storage apparatus200A as well as the setting information and copy status of the various copy pairs to the management computer300via the management port250. The control program221simultaneously suspends the copy of a plurality of copy pairs in which the same volume235in the storage apparatus200A is the copy source volume. Here, the simultaneous suspension of a plurality of copy pairs refers to the suspension of the copy operation of the plurality of copy pairs during the period of receiving a write command from the host computer100, passing on the same write command to a copy line of the plurality of copy pairs and receiving all processing completion notices of that write command from the copy destination, and passing on the subsequent write command to a copy line of the plurality of copy pairs.

The disk apparatus230is configured from one or more disks231having an array group (not shown) of a RAID configuration, and the volume235is a logical storage area in the array group.

The storage apparatus200B is configured the same as the storage apparatus200A, and the explanation thereof is omitted.

As shown inFIG. 4, the management computer300comprises a processor310, a memory320, a management port330for connecting to the management network N2, an input unit340such as a keyboard or a mouse, and an output unit350such as a display apparatus for outputting the processing result, and the respective components are mutually connected via an internal bus360.

An apparatus management table321retaining information of the host computer100and the storage apparatuses200A,200B to be managed, a volume management table322for managing the configuration and allocation status of the volumes235in the storage apparatuses200A,200B to be managed, a copy pair management table323for managing the pair configuration and pair status of the volumes235based on the local copy program222and the remote copy program223retained in the storage apparatuses200A,200B to be managed, a snapshot management table324for managing the snapshots of the volumes235used by the business application121in the host computer100, a snapshot duplication method evaluation table325for managing the evaluation of the method for duplicating snapshots between two different storage apparatuses200A,200B, a snapshot management program326for providing an interface to be used by the storage administrator A for managing snapshots, a copy control command program327for commanding the storage apparatuses200A,200B to execute the local copy program222and the remote copy program223retained in the storage apparatuses200A,200B via the management port330and create the volumes235, a configuration information collection program328for collecting the configuration information of the host computer100and the configuration information of the storage apparatuses200A,200B via the management port330, a failure information collection program329for collecting the failure information of the host computer100and the failure information of the storage apparatuses200A,200B via the management port330, an input/output program (not shown) for acquiring a request of the storage administrator A from the input unit340and outputting the execution result of the respective programs to the output unit350, and an OS (not shown) are loaded into the memory320from a storage medium (not shown) such as a hard disk, and are executed by the processor310.

As shown inFIG. 5, the apparatus management table321is configured from an apparatus ID field3211for storing identifiers to be used in uniquely identifying the host computer100and the storage apparatuses200A,200B in the management computer300, an apparatus type field3212for storing the type of apparatus, an apparatus identifying information field3213for storing information to be used in uniquely identifying the apparatus to be managed by combining the vender name, model name, serial number, host name and so on of the apparatus, an IP address field3214for storing the IP address of the management port of the apparatus to be managed, and a data communicable apparatus ID field3215for storing an apparatus ID of the other party with which the apparatus to be managed is able to engage in data communication. The information to be stored in the respective fields3211to3215may be registered in advance by the storage administrator A via the input unit340of the management computer300, or automatically registered using a name service (not shown) on the storage network N1or the management network N2.

As shown inFIG. 6, the volume management table322is configured from a storage apparatus ID field3221for storing identifiers to be used in uniquely identifying the storage apparatuses200A,200B in the management computer300, a volume ID field3222for storing identifiers to be used in uniquely identifying the volumes235in the storage apparatuses200A,200B, a capacity field3223for storing the allocated logical storage capacity of the volume235, an AGID field3224for storing identifiers to be used in uniquely identifying the array group configuring the volumes235in the storage apparatuses200A,200B, and an allocation destination field3225for storing identifiers to be used in identifying the allocation destination of the volume235in the management computer300. For example, if a snapshot is stored in the volume235, the character string of “Snapshot” is stored in the allocation destination field3225. If the volume235is not allocated to the host computer100or if a snapshot is not stored in the volume235, “null” is stored in the allocation destination field3225to show that a volume can be allocated to the host computer100and that the host computer100can be used as the snapshot copy destination. The information to be stored in the respective fields3221to3225is added, changed or deleted by executing the configuration information collection program328described later or executing the copy control command program327described later when a new volume235is created.

As shown inFIG. 7, the copy pair management table323is configured from a copy pair ID field3231for storing identifiers to be used in uniquely identifying copy pairs in the management computer300, a storage apparatus ID field3232for storing identifiers to be used in uniquely identifying the copy source storage apparatuses200A,200B in the management computer300, a volume ID field3233for storing identifiers to be used in uniquely identifying the copy source volume235in the storage apparatuses200A,200B, a copy type field3234for storing the copy type (local copy or remote copy) of the copy pair, a copy status field3235for storing the copy status of the copy pair, a copy destination storage apparatus ID field3236for storing identifiers to be used in uniquely identifying the copy destination storage apparatuses200A,200B in the management computer300, and a copy destination volume ID field3237for storing identifiers to be used in uniquely identifying the copy destination volume235in the storage apparatuses200A,200B. The information to be stored in the respective fields3231to3237is added, changed or deleted by executing the configuration information collection program328described later, or performing the copy operation based on the copy control command program327described later.

As shown inFIG. 8, the snapshot management table324is configured from a snapshot group ID field3241for storing identifiers to be used in uniquely identifying a group compiled from the volumes235for storing the same snapshot; that is, the simultaneous replicated data of the volumes235used by the business application of the host computer100, a primary ID field3242storing identifiers of the volumes (hereinafter referred to as the “primary volumes”)235used by the business application121, a snapshot storage volume ID field3243storing identifiers of the volumes325for storing the same snapshot as the primary volumes235of the primary ID field3242, a snapshot duplication method field3244for storing the snapshot duplication method adopted for each volume235of the snapshot storage volume ID field3243, a target copy pair field3245for storing identifiers to be used in uniquely identifying the copy pair to be subject to the adopted snapshot duplication method in the management computer300by combining the model number of the copy line and the copy pair ID field3231of the copy pair management table323, and a snapshot creation date and time field3246for storing the date and time that the snapshot was created. For instance, if the volume235of the snapshot storage volume ID field3243is stored in the same storage apparatus200A as the primary volume235of the primary ID field3242, the character string of “LOCAL” is stored in the snapshot duplication method field3244. The information to be stored in the respective fields3241to3246is added, changed or deleted by executing the snapshot management program326described later.

As shown inFIG. 9, the snapshot duplication method evaluation table325is configured from an evaluated value field3251for storing the evaluated value representing the index for evaluating the snapshot duplication method described later in five stages, an RTO field3252for storing the snapshot duplication method corresponding to the respective evaluated values of the RTO (Recovery Time Objective), a business I/O processing speed field3253for storing the snapshot duplication method corresponding to the respective evaluated values of the processing speed (hereinafter referred to as the “business I/O processing speed”) of write commands issued by the business application121, an RC line dominance field3254for storing the snapshot duplication method corresponding to the respective evaluated values of the dominance of the remote copy line (hereinafter referred to as the “RC line dominance”), and an inter-storage apparatus distance field3255for storing the snapshot duplication method corresponding to the respective evaluated values of the distance between the storage apparatuses. For instance, if the evaluated value field3251is “5,” the snapshot duplication method with the shortest RTO is stored in the RTO field3252, the snapshot multiplexing method with the fastest business I/O processing speed is stored in the business I/O processing speed field3253, the snapshot multiplexing method with the lowest RC line dominance is stored in the RC line dominance field3254, and the snapshot duplication method in which the distance between the storage apparatuses is a “long distance” is stored in the inter-storage apparatus distance field3255, respectively. Meanwhile, if the evaluated value field3251is “1,” the snapshot duplication method with the longest RTO is stored in the RTO field3252, the snapshot duplication method with the slowest business I/O processing speed is stored in the business I/O processing speed field3253, a snapshot duplication method with a high RC line dominance is stored in the RC line dominance field3254, and a snapshot duplication method in which the distance between the storage apparatuses is a “short distance” is stored in the inter-storage apparatus distance field3255, respectively. The information to be stored in the respective fields3251to3255is set forth based on the performance of the storage apparatuses200A,200B and the operation policy of the computer system, and registered in advance by the storage administrator A via the input unit340of the management computer300.

Operations of the copy control command program327and the snapshot management program326retained in the management computer300are now explained with reference toFIG. 10toFIG. 18.

The copy control command program327retains a plurality of snapshot duplication methods for duplicating a snapshot between two different storage apparatuses200A,200B, and creates an environment for duplicating a snapshot based on a request from the snapshot management program326.

In the environment creation request for duplicating a snapshot to be issued to the copy control command program327, designated are the primary volume235as the snapshot creation source, a volume (hereinafter referred to as the “A/S volume”)235for storing the snapshot of the same storage apparatus200A as the primary volume235, a volume (hereinafter referred to as the “S/volume”)235for storing the same snapshot as the A/S volume of a storage apparatus200B that is different from the primary volume235, and the applicable snapshot duplication method. If there is no A/S volume in the storage apparatus200A, it is not necessary to designate the A/S volume.

Each of the plurality of snapshot duplication methods retained in the copy control command program327is now explained.

As shown inFIG. 10, in the first snapshot duplication method (hereinafter simply referred to as the “first method”), data of the primary volume2351is replicated in the A/S volume2352with the local copy program222retained in the storage apparatus200A, replicated in a volume2354(hereinafter referred to as the “Temp volume”) of a different storage apparatus200B with the remote copy program223retained in the storage apparatus200A, and data of the Temp volume2354is replicated in the S/S volume2355with the local copy program222retained in the storage apparatus200B. Here, the copy pair of the primary volume2351and the A/S volume2352is referred to as L1, the copy pair of the primary volume2351and the Temp volume2354is referred to as R1, and the copy pair of the Temp volume2354and the S/S volume2355is referred to as L2. In the first method, the copy line of all copy pairs is constantly connected, and the copy status field3235of the copy pair management table323becomes a “PAIR” status showing that the data is constantly replicated. Thus, after the writing of data into the primary volume2351, the A/S volume2352and the Temp volume2354in response to the write command issued by the business application121is complete, a write completion notice is issued to the business application121.

When the first method is designated as the applicable snapshot duplication method in the environment creation request, the copy control command program327searches for a copy pair corresponding to the copy pair R1that can be shared by a plurality of snapshot groups from the copy pair management table323. If a corresponding copy pair does not exist, the copy control command program327requests the control program221of the storage apparatuses200A,200B to newly set copy pairs L1, R1, L2, requests the control program221of the storage apparatus200B to newly set a volume235to become the Temp volume2354, registers the newly set copy pairs L1, R1, L2in the copy pair management table323, and registers the newly set Temp volume2354in the volume management table322. If a copy pair corresponding to the copy pair R1exists in the copy pair management table323, the copy control command program327requests the control program221of the storage apparatuses200A,200B to newly set copy pairs L1, L2, and registers the newly set copy pairs L1, L2in the copy pair management table323.

The snapshot creation routine of the first method is now explained. In the first method, upon creating a snapshot, the write command for writing data into the primary volume2351of the business application121is queued in the cache memory260to realize a status where data is not written into the primary volume2351. Subsequently, the copy pairs L1, R1are simultaneously suspended, and then the copy pair L2is suspended. The same snapshot is thereby created in the A/S volume2352and the S/S volume2355.

The write command queued in the cache memory260is applied to the primary volume2351at the time the copy pairs L1, R1are suspended. When restoring data of the primary volume2351using the snapshot created with the first method, if the A/S volume2352is available, the copy direction of the copy pair L1is reversed (hereinafter referred to as “resynching” or “resynchronization”). If the A/S volume2352is unavailable, the copy pair L2is resynched, and the copy pair R1is thereafter resynched. In a case where the backup of the primary volume2351is the Temp volume2354, and the primary volume2351becomes unavailable and the business application121is using the Temp volume2354, the volume235used by the business application121can be restored by resynching the copy pair L2.

When applying the first method to each of the plurality of snapshot groups having the same primary volume2351, the copy pair R1and the Temp volume2354are shared between the snapshot groups.

As shown inFIG. 11, the second snapshot duplication method (hereinafter simply referred to as the “second method”) replicates the data of the primary volume2351in the A/S volume2352with the local copy program222retained in the storage apparatus200A, and replicates the data of the A/S volume2352in the S/S volume2354with the remote copy program223retained in the same storage apparatus200A. Here, the copy pair of the primary volume2351and the A/S volume2352is referred to as L1, and the copy pair of the A/S volume2352and the S/S volume2354is referred to as R2. In the second method, the copy line of all copy pairs is constantly connected, and the copy status field3235of the copy pair management table323becomes a “PAIR” status representing that the data is constantly replicated. Thus, a write completion notice is issued to the business application121in response to the write command issued by the business application121after the writing of data into the primary volume2351and the A/S volume2352is complete.

When the second method is designated as the applicable method in the environment creation request, the copy control command program327requests the control program221of the storage apparatuses200A,200B to newly set the copy pairs L1, R2using the volume235of the primary volume2351, the A/S volume2352, and the S/S volume2354designated together with the second method, and registers the newly set copy pairs in the copy pair management table323.

The snapshot creation routine of the second method is now explained. With the second method, upon creating a snapshot, the copy pair L1is foremost suspended, and the copy pair R2is subsequently suspended. Thereby, the same snapshot is created in the A/S volume2352and the S/S volume2354.

When restoring data of the primary volume2351using the snapshot created with the second method, the copy pair L1is resynched if the A/S volume2352is available and, if the A/S volume2352is unavailable, data of the S/S volume2354is replicated in the primary volume2351with the remote copy program223retained in the storage apparatus200B storing the S/S volume2354.

When applying the second method to each of the plurality of snapshot groups having the same primary volume2351, there are no copy pairs to be shared between the snapshot groups.

The third snapshot duplication method (hereinafter simply referred to as the “third method”) is the same as the second method other than that the copy pair R2of the second method is asynchronous.

The snapshot creation routine of the third method foremost suspends the copy pair L1, and subsequently suspends the copy pair R2when data of the A/S volume2352and data of the S/S volume2354coincide; that is, when all data to be sent from the storage apparatus200A to the S/S volume2354are transferred, and the storage apparatus200B reflects the received data in the S/S volume2354. Thereby, the same snapshot will be created in the A/S volume and the S/S volume.

The method of restoring the primary data of the third method using the created snapshot is the same as the second method.

As shown inFIG. 12, the fourth snapshot duplication method (hereinafter simply referred to as the “fourth method”) replicates the data of the primary volume2351in the A/S volume2352with the local copy program222retained in the storage apparatus200A, and replicates the data of the A/S volume2352in the Temp volume2354retained in a different storage apparatus200B with the remote copy program223retained in the same storage apparatus200A. Subsequently, the storage apparatus200B retaining the Temp volume2354replicates the data of the Temp volume2354in the S/S volume2355with the local copy program222retained in the same storage apparatus200B. Here, the copy pair of the primary volume2351and the A/S volume2352is referred to as L1, the copy pair of the A/S volume2352and the Temp volume2354is referred to as R3, and the copy pair of the Temp volume2354and the S/S volume2355is referred to as L3. The copy line of the copy pair R3is in a suspended state during normal times, and becomes a connected state when a snapshot is to be created.

When the fourth method is designated as the applicable method in the environment creation request, the copy control command program327searches for a copy pair corresponding to the copy pair R3that is sharable among a plurality of snapshot groups, and, if a corresponding copy pair does not exist, the copy control command program327requests the control program221of the storage apparatuses200A,200B to newly set copy pairs L1, R3, L3, requests the control program221of the storage apparatus200B to newly set the volume235to become the Temp volume2354, registers the newly set copy pairs L1, R3, L3in the copy pair management table323, and registers the newly set Temp volume2354in the volume management table322. If a copy pair corresponding to the copy pair R3exists in the copy pair management table323, the copy control command program327requests the control program221of the storage apparatuses200A,200B to newly set copy pairs L1, L3, and registers the newly set copy pairs L1, L3in the copy pair management table323.

The snapshot creation routine of the fourth method is now explained. In the fourth method, the copy pair L1is foremost suspended, the copy of the copy pair R3is subsequently resumed, and, when the copy pair R3becomes a PAIR status, the copy pairs R3, L3are suspended. Thereby, the same snapshot will be created in the A/S volume2352and the S/S volume2355.

When applying the fourth method to each of the plurality of snapshot groups having the same primary volume2351, it is necessary to change the copy pair R3as needed upon creating a snapshot.

As shown inFIG. 13, in the case of a computer system with a plurality of inter-storage apparatus configurations, the same snapshot can be created between a plurality of different storage apparatuses by selecting the snapshot duplication method to be applied to different storage apparatuses among a plurality of snapshot duplication methods. In the ensuing explanation, as a specific example, let it be assumed that the first method is used as the snapshot duplication method between the storage apparatus200A and the storage apparatus200B, the second method is applied between the storage apparatus200A and the storage apparatus200C, and the fourth method is applied between the storage apparatus200A and the storage apparatus200D. Moreover, let it be assumed that the volumes2352,2354,2355,2357belong to the same snapshot group as volumes for storing the snapshots S1to S4having the same data contents as the primary volume2351.

The snapshot management program326includes snapshot group creation processing S100, and snapshot creation processing S200in snapshot group units.

The snapshot group creation processing S100in the computer system shown inFIG. 13is foremost explained with reference to the time chart ofFIG. 14.

When the storage administrator A designates the primary volume2351as the snapshot creation target via the input unit340of the management computer300, a snapshot group creation request is issued to the snapshot management program326(S101).

Here, the snapshot group management screen D100to be provided by the snapshot management program326to the storage administrator A when the storage administrator A is to issue the snapshot group creation command, as shown inFIG. 15, is configured from a snapshot creation source volume field D110for displaying the identifier to be used in uniquely identifying the snapshot creation source volume; that is, the primary volume2351, with the computer system, a selection field D121for selecting the snapshot group to be edited or deleted, a snapshot group ID field D122for displaying the identifier to be used in uniquely identifying the snapshot group belonging to the primary volume2351with the computer system, a snapshot storage volume field D123for displaying the identifier to be used in uniquely identifying the volume235storing the snapshot belonging to the snapshot group of the snapshot group ID field D122with the computer system, a snapshot creation date and time field D124for displaying the date and time the snapshot of the primary volume2351was created in the volume235displayed in the snapshot storage volume field D123belonging to the snapshot group of the snapshot group ID field D122, a group create button D125for starting the processing for creating a new snapshot group belonging to the primary volume2351, a group delete button D126for starting the processing of deleting the snapshot group selected in the selection field D121, and a group edit button D127for starting the processing of adding or deleting the volume235belonging to the snapshot group selected in the selection field D121.

Subsequently, the snapshot management program326executes the candidate volume selection processing for electing a candidate volume235capable of storing the snapshot of the designated primary volume2351from the volumes235of the storage apparatuses200A to200D (S102).

As shown inFIG. 16, the candidate volume selection processing selects a volume235that has not yet been subject to the candidate volume selection processing from the volume management table322(S301), and determines whether data of the volume235input to the snapshot creation source volume field D110can be copied to the selected volume235(S302). In this embodiment, copying is determined to be possible if the capacity is equivalent, and the storage apparatuses200A to200D retaining the selected volume235and the storage apparatuses200A to200D retaining the snapshot creation source volume235are able to engage in data communication. Data communication is considered possible if the storage apparatuses200A to200D are registered in the data communicable apparatus ID field3215of the apparatus management table321. The process proceeds to S305if it is determined that copying is not possible at S302, and the process proceeds to S303if it is determined that copying is possible at S302.

The determination of whether data can be copied may also include a condition of the RAID level of the array group being equivalent in addition to the capacity being equivalent. Thereby, since volumes in which the RAID levels of the array groups do not coincide and must be changed or volumes that cannot be changed to equivalent RAID levels will not be selected, candidate volumes for storing snapshots can be easily selected upon storing snapshots.

Subsequently, whether the volume235selected at S301is unused is determined (S303). In this embodiment, the volume235is determined to be unused if the allocation destination field3225of the volume management table322is “null.”

If it is determined that the volume235selected at S301is being used, the process proceeds to S305, and, if it is determined that the volume235is not being used, the volume235selected at S301is added to the display list (not shown) which is temporarily created in the candidate volume selection processing (S304).

Subsequently, the volume235selected at S301is treated as processed (S305), and whether all volumes235of the volume management table322have been processed is determined (S306). If there is an unprocessed volume235in the volume management table322, the process returns to S301, and, if all volumes235have been processed, the routine proceeds to S307.

Subsequently, the volume235added to the created display list is displayed on the volume selection screen D200described later to be provided to the storage administrator A (S307), and, after displaying all volumes235added to the display list, the snapshot management program326ends the candidate volume selection processing. Thereby, the candidate volume235that can be added to the new snapshot group is notified to the storage administrator A.

Subsequently, the snapshot management program326notifies the processing result of the candidate volume selection processing to the storage administrator A via the output unit250of the computer300(S103). When the storage administrator A inputs the selection of at least one volume235to be added to the snapshot group to be created from the notified candidate volume235and the system requirement of the volume235to become the S/S volume via the input unit340, these are notified to the snapshot management program326(S104).

Here, when the storage administrator A is to input the selection of the volume235and the system requirement of the S/S volume, the volume selection screen D200to be provided by the snapshot management program326to the storage administrator A, as shown inFIG. 17, is configured from a snapshot creation source volume field D210for displaying the identifier to be used in uniquely identifying the snapshot creation source volume; that is, the primary volume2351, with the computer system, a selection field D220for selecting the volume235to be added to the snapshot group to be created, a storage apparatus ID field D230for displaying the identifier to be used in uniquely identifying the storage apparatuses200A to200D retaining the candidate volume235selected in the candidate volume selection processing with the computer system, a volume ID field D240for displaying the identifier to be used in uniquely identifying the candidate volume235selected in the candidate volume selection processing in the storage apparatuses200A to200D, a system requirement field D250for inputting the system requirement of the volume235, an OK button D260for notifying the volume235to be added to the snapshot group selected in the selection field D220and its system requirement to the snapshot management program326, and a cancel button D270for ending the processing for newly creating the snapshot group belonging to the primary volume2351. When inputting the system requirement of the volume235, the screen for inputting the system requirement is launched by pressing the registration button D251, and the clear button D252is pressed for clearing the registered system requirement.

Subsequently, the snapshot management program326executes the snapshot duplication method selection processing for selecting the snapshot duplication method that satisfies the system requirement of the volume235input for each S/S volume (S105).

Here, the system requirement requested in the snapshot duplication method to be applied to the S/S volume; that is, the five-stage evaluated values of the RTO, the business I/O processing speed, the RC line dominance, and the inter-storage apparatus distance as indexes of the snapshot duplication method evaluation table325are input by the storage administrator A from the system requirement field D250of the volume selection screen D200.

The snapshot duplication method selection processing selects the snapshot duplication method for all volumes235to become the S/S volume using the primary volume2351and the A/S volume. Specifically, a volume235not yet set with the snapshot duplication method to be applied is foremost extracted from the S/S volume, an index with a high evaluated value is selected by referring to the five-stage evaluated values of the respective indexes input from the system requirement field D250to the extracted volume235, and the method corresponding to the evaluated value of the fields3252to3255of the indexes selected from the snapshot duplication method evaluation table325is selected as the snapshot duplication method to be applied to the extracted volume235. If a plurality of methods are selected, the method used in the inter-storage apparatus configuration is selected as the applicable snapshot duplication method when there is an existing inter-storage apparatus configuration that can be diverted, and, when there is no divertible inter-storage apparatus configuration, the storage administrator A is made to select one index that is to be evaluated the highest among a plurality of indexes so as to decide one applicable snapshot duplication method. Thereby, the optimal snapshot duplication method upon storing the snapshot from the volume235of the storage apparatus200A to the volume235of the storage apparatuses200B to200D is decided.

When the storage administrator A is to input an evaluated value from the system requirement field D250, in order to prevent all evaluated values from becoming “5,” the input of the system requirement field D250may be an exclusive input giving consideration to the correlation of the respective indexes so that, for example, if the evaluated value of the RTO is “5,” the evaluated value of the business I/O processing speed becomes “1.”

Subsequently, the snapshot management program326sends the snapshot environment creation request designating the primary volume2351and the A/S volume and the snapshot duplication method selected at S105to the copy control command program327for every S/S volume selected at S104(S106), and the copy control command program327sends the copy pair configuration request for creating a snapshot duplication environment that satisfies the snapshot environment creation request to the control program221of the storage apparatuses200A to200D (S107).

The control program221of the storage apparatuses200A to200D sets the copy pair so as to satisfy the copy pair configuration request (S108). In the specific example of this embodiment, the copy pairs L1, R1, L2, R2, R3, L3are set.

Upon receiving a copy pair configuration completion notice from the storage apparatuses200A to200D, the copy control command program327updates the volume management table322and the copy pair management table323, and sends an environment creation completion notice to the snapshot management program326.

When the snapshot management program326receives the environment creation completion notice, it creates the identifier that is able to uniquely identify the snapshot group with the management computer300, registers the identifier of the created snapshot group in the snapshot group ID field3241of the snapshot management table324, registers the identifier of the primary volume2351requested at S101in the primary ID field3242, registers the volume235selected at S104in the snapshot storage volume field3243, registers the identifier of the snapshot duplication method selected at S105in the snapshot duplication method field3244, registers the target copy pair contained in the environment creation completion notice of the copy control command program327and the copy pair ID field3231of the copy pair management table323in the target copy pair field3245, and registers the snapshot group in the snapshot management table324(S109).

The snapshot management program326notifies the identifier of the created snapshot group to the storage administrator A via the output unit350(S110), and then ends the snapshot group creation processing S100.

The snapshot creation processing S200in the computer system shown inFIG. 13is now explained with reference to the time chart shown in theFIG. 18.

When the storage administrator A designates a snapshot group, a snapshot creation command is issued to the snapshot management program326(S201). In the specific example of this embodiment, the snapshot group G001is designated.

The snapshot management program326extracts the identifiers of the primary volume, the snapshot storage volume, the snapshot duplication method, and the target copy pair belonging to the snapshot group designated from the respective fields3242to4245of the snapshot management table324(S202).

Subsequently, the snapshot management program326extracts all copy pairs in which the primary volume2351is the copy source from the extracted target copy pair, changes the status of all extracted copy pairs to a PAIR status, and requests the copy control command program327to suspend the copy of the copy pairs at the same timing (S203). In the specific example of this embodiment, the copy pairs L1, R1are extracted.

The copy control command program327updates the copy pair management table323to the latest status in order to change the status of all copy pairs requested at S203to a PAIR status, and sends a command for starting the copy of the copy pairs in a suspended status to the control program221of the storage apparatuses200A,200B relating to the copy pairs (S204). Thereby, the snapshot of the primary volume2351included in the designated snapshot group G001is stored in the volumes2352,2353of the copy destination of all copy pairs L1, R1included in the designated snapshot group G001and in which the primary volume2351is the copy source.

The control program221of the storage apparatuses200A,200B starts the copy of all designated copy pairs (S205), and the copy control command program327updates the copy pair management table323, and, when all copy pairs that started copying at S205becomes a PAIR status, a command for suspending all copy pairs requested at S203at the same timing is sent to the control program221of the storage apparatus200A retaining the primary volume2351(S206). Thereby, the same snapshot will be stored in the volumes2352,2353of the copy destination of all copy pairs L1, R1in which the primary volume2351is the copy source.

The control program221is able to suspend the copy of a plurality of copy pairs having the volumes235in the same storage apparatuses200A to200D as the copy source at the same timing, and the control program221simultaneously suspends the copy of the copy pairs designated at S206(S207).

When the copy control command program327receives the completion notice at S207, it updates the copy pair management table323and confirms that the copy of all copy pairs commanded at S206is suspended, and notifies the copy suspend request completion at S203to the snapshot management program326.

When the snapshot management program326receives the copy suspend request completion notice of S203, it requests the copy control command program327to suspend the copy of all copy pairs at the same timing after changing the copy pairs other than the copy pairs subject to copy suspend at S203among the copy pairs extracted at S202to a PAIR status (S208). In the specific example of this embodiment, the copy pairs L2, R2, R3, L3are the target copy pairs.

The copy control command program327sends a command for starting the copy for changing all copy pairs requested at S208to a PAIR status to the control program221of the storage apparatuses200A to200D relating to the copy pairs (S209). Thereby, the snapshot of the primary volume2351included in the designated snapshot group G001is stored in the volumes2354,2355,2356,2357of the copy destination of the copy pairs L2, R2, R3, L3other than all copy pairs included in the designated snapshot group G001and in which the primary volume2351is the copy source.

The control program221of the storage apparatuses200A to200D starts the copy of all designated copy pairs (S210), the copy control command program327updates the copy pair management table323, and, when all copy pairs designated at S209become a PAIR status, a command for suspending the copy of the copy pairs designated at S208is sent to the control program221of the storage apparatuses200A to200D relating to the copy pairs (S211). Thereby, the same snapshot will be stored in the volumes2354,2355,2356,2357of the copy destination of the copy pairs L2, R2, R3, L3other than all copy pairs in which the primary volume2351is the copy source.

The control program221that received the command simultaneously suspends the copy of the copy pairs designated at S211(S212).

When the copy control command program327receives the completion notice of S212, it updates the copy pair management table323, confirms that the copy of all copy pairs designated at S211has been suspended, and sends the copy suspend request completion notice of S208to the snapshot management program326.

When the snapshot management program326receives the copy suspend request completion notice of S208, it registers the current date and time in the snapshot creation date and time field3246of the snapshot management table324. Although this embodiment registers the date and time that the copy suspend request completion notice of S208was received in the snapshot creation date and time field3246, the present invention is not limited thereto, and the snapshot management program326may also register the date and time that the copy suspend request completion notice of S203was received in the snapshot creation date and time field3246.

Subsequently, the snapshot management program326extracts the copy pairs being used in the snapshot groups other than the snapshot groups requested at S201, and sends the copy start command of the extracted copy pairs to the copy control command program327(S213). Thereby, copy of the copy pairs to be shared with the snapshot groups other than the designated snapshot groups is started.

Here, the copy pair identifier can be uniquely identified in the management computer300, and the identifier of the copy pair to be used in the snapshot group is registered in the target copy pair field3245of the snapshot management table324. Thus, in order to extract the copy pairs being used in the other snapshot groups, the target copy pair field3245of other snapshot groups of the snapshot management table324is referred to and, if a copy pair coinciding with the identifier of the copy pair used by the snapshot group requested at S201is registered in the target copy pair field3245, this is extracted as the copy pair being used by other snapshot groups.

The copy control command program327sends a command for starting the copy of the copy pairs requested at S213to the control program221of the storage apparatuses200A to200D relating to the extracted copy pairs (S214), and the control program221starts the copy of the designated copy pairs (S215).

When the copy control command program327receives a copy start completion notice of the copy pairs from the control program221, it sends the request processing completion notice of S213to the snapshot management program326, and the snapshot management program326sends the snapshot creation completion notice to the storage administrator A (S216), and then ends the snapshot creation processing S200.

The operation of notifying the available snapshot when a failure occurs in the computer system shown inFIG. 13is now explained with reference toFIG. 19toFIG. 21.

The failure information collection program329periodically requests a failure status notification to the configuration information notification program122of the host computer100and the control program221of the storage apparatuses200A to200D via the management network N2. If the storage administrator A requests the snapshot storage location notification via the input unit340of the management computer300, a failure status notification is requested to the control program221of the storage apparatuses200A to200D. If failure information is included in the response from the host computer100and/or the storage apparatuses200A to200D, the failure information collection program329notifies the failure information to the snapshot management program326.

When the failure information is notified, the snapshot management program326extracts an available volume235from the snapshot group, and notifies the extracted volume235to the storage administrator A via the output unit350of the management computer300. If there is a snapshot group in which a snapshot is not created, the volume235that is not able to create a snapshot is extracted from the notified failure information, and the extracted volume235is notified to the storage administrator A via the output unit350of the management computer300.

More specifically, when the snapshot management program326is notified of the failure information from the failure information collection program329, it starts the processing upon detecting a failure as shown inFIG. 19.

The snapshot management program326refers to the volume management table322, the copy pair management table323and the notified failure information, and extracts the volumes235and the copy pairs that are no longer available due to a failure (S401).

Subsequently, the snapshot management program326refers to the snapshot management table324and selects the snapshot group that has not yet been subject to the processing of S402(S402), refers to the snapshot creation date and time field3246of the snapshot management table324, and determines whether a snapshot is created in the snapshot group selected at S402(S403). Here, it is determined that a snapshot has not been created if the snapshot creation date and time field3246is “null,” and determined that a snapshot has been created in all cases other than “null.”

When it is determined that a snapshot has been created at S403, the snapshot management program326selects a volume235belonging to the snapshot group selected at S402and which has not yet been subject to the processing of S404(S404), and determines whether the selected volume235is available (S405). Here, it is determined that the volume235is unavailable if the volume235selected at S404corresponds to the volume235extracted at S401, and determined that the volume235is available if the volume235selected at S404does not correspond to the volume235extracted at S401.

If the volume235is determined to be unavailable at S405, the process proceeds to S407, and if the volume235is determined to be available at S405, the volume235selected at S404is added to the available volume list (not shown), which is a list that is temporarily created in the processing to be performed upon detecting a failure (S406).

Subsequently, the volume235selected at S404is treated as having been subject to the processing of S404(S407), and the snapshot management program326determines whether the processing of S404has been performed to all volumes235belonging to the snapshot group selected at S402(S408). If there is an unprocessed volume235, the routine returns to S404, and if there is no unprocessed volume235; that is, if the processing of S404has been performed to all volumes235belonging to the snapshot group selected at S402, the process proceeds to S414.

If it is determined that a snapshot has not been created at S403, the snapshot management program326selects a volume belonging to the snapshot group selected at S402and which has not yet been subject to the processing of S409(S409), and determines whether the data of the primary volume2351cannot be copied (S410). Here, the snapshot management program326refers to the copy pair management table323to extract all copy pairs up to the primary volume2351from the volume235selected at S409, and determines that the data of the primary volume2351can be copied if none of the extracted copy pairs corresponds to the copy pair extracted at S401, and determines that the data of the primary volume2351cannot be copied if even one of the extracted copy pairs corresponds to the copy pair extracted at S401.

If it is determined that the data can be copied at S410, the process proceeds to S412, and if it is determined that the data cannot be copied at S410, the volume235selected at S409is added to a snapshot uncreatable volume list (not shown), which is a list to be temporarily created in this processing (S411).

Subsequently, the volume235selected at S409is treated as having been subject to the processing of S409(S412), and the snapshot management program326determines whether the processing of S409has been performed to all volumes235belonging to the snapshot group selected at S402(S413), returns to S409is there is an unprocessed volume235, and proceeds to S414if there is no unprocessed volume235; that is, if the processing of S409has been performed to all volumes235belonging to the snapshot group selected at S402.

Subsequently, the snapshot group selected at S402is treated as having been subject to the processing at S302(S414), and the snapshot management program326determines whether the processing of S302has been performed to all snapshot groups (S415), returns to S402if there is a snapshot group that has not yet been subject to the processing of S402, and proceeds to S416is there is no unprocessed snapshot group; that is, if the processing of S302has been performed to all snapshot groups.

Subsequently, the snapshot management program326refers to the available volume list created at S406, and, as shown inFIG. 20, notifies the available volume list screen D300including the volume235registered in the available volume list, the snapshot group to which the volume235belongs, and the snapshot creation date and time to the storage administrator A via the output unit350, refers to the snapshot uncreatable volume list created at S411, and, as shown inFIG. 21, notifies the snapshot creation availability screen D400including the volume235registered in the snapshot uncreatable volume list and the snapshot group to which the volume235belongs to the storage administrator A via the output unit350(S416). Thereby, the volumes235other than volumes235that are no longer available due to a failure among the volumes235storing the snapshot of the primary volume2351are notified to the storage administrator A. In addition, the volumes235that are not able to store the snapshot of the primary volume2351due to a failure are also notified to the storage administrator A.

After notifying the available volume list screen D300and/or the snapshot creation availability screen D400to the storage administrator A, the snapshot management program326ends the processing to be performed when a failure is detected.

As a more specific example, processing to be performed when a failure is detected is now explained in a case where the volumes2352,2354,2355,2357shown inFIG. 13belong to the same snapshot group G001, a snapshot is already created in the snapshot group G001, and the failure information collection program329detects that the storage apparatuses200B,200C are damaged and all of the volumes2353,2354,2355of the storage apparatuses200B,200C are unavailable.

When the snapshot management program326receives a failure information notification, it extracts the Temp volume2353and the volumes2354,2345as the volumes235that are unavailable in the processing of S401.

Subsequently, the snapshot management program326selects the snapshot group G001in the processing of S402, and, since a snapshot is already created in the snapshot group G001, performs the processing of S404. When the processing of S404to S407is performed, the Temp volume2356and the volumes2352,2357are added to the available volume list created at S406.

Subsequently, in the processing of S416, the snapshot management program326notifies the available volumes2352,235in the snapshot group G001as shown inFIG. 20to the storage administrator A.

The processing to be performed when a failure is detected is now explained in a case where the volumes2352,2354,2355,2357belong to the same snapshot group G002, a snapshot is not created in the snapshot group G002, and the failure information collection program329detects that the array group to which the volume2352belongs is unavailable from the storage apparatus200A.

When the snapshot management program326receives the failure information notification, it extracts information indicating that the volume2352and the copy pairs L1, R2, R3are unavailable in the processing of S401.

Subsequently, the snapshot management program326selects the snapshot group G002in the processing of S402, and performs the processing of S409since a snapshot has not been created in the snapshot group G002.

Subsequently, the snapshot management program326selects the volume2352that has not yet been subject to the processing of S409in the snapshot group G002in the processing of S409. The copy pair L1from the volume2352to the primary volume2351is extracted as an unavailable copy pair at S402. Thus, the volume2352is added to the snapshot uncreatable volume list in the processing of S411.

The snapshot management program326thereafter returns to S409and selects the volume2354. The copy pairs L2, R1from the volume2354to the primary volume2351do not correspond to the unavailable copy pair extracted at S401. Thus, the snapshot management program326returns to S409without performing the processing of S411, and selects the volume2355. The copy pairs R2, L1from the volume2355to the primary volume2351correspond to the unavailable copy pairs extracted at S401. Thus, the volume2355is added to the snapshot uncreatable volume list.

The snapshot management program326thereafter returns to S409and selects the volume2357, and the copy pair R3among the copy pairs L3, R3from the volume2357to the primary volume2351corresponds to the unavailable copy pair extracted at S401. Thus, the volume2357is added to the snapshot uncreatable volume list.

Subsequently, the snapshot management program326notifies the volumes2352,2355,2357as the snapshot uncreatable volumes shown inFIG. 21to the storage administrator A in the processing of S416.

Like this, according to the present invention, since a copy pair of a copy source and a copy destination forming a pair is created so that the snapshot of the primary volume2351included in the snapshot group is stored in all volumes235included in that snapshot group, and all copy pairs included in that snapshot group becomes a PAIR status when there is a creation request of a snapshot of the primary volume2351designating the snapshot group, the snapshot of the primary volume2351included in the designated snapshot group can be stored in all volumes235included in the designated snapshot group. Thus, the snapshots to be duplicated between the storage apparatus200A and the storage apparatuses200B to200D can be consolidated with only the snapshot group, and the management of snapshots to be multiplexed between different storage apparatuses can be facilitated thereby.

In addition, when there is a creation request of a snapshot of the primary volume2351designating a snapshot group, all copy pairs will simultaneously become a suspended status after the copying of all copy pairs in which the primary volume2351included in the snapshot group is the copy source is completed. Thus, since the same snapshot can be stored in the volume235of the copy destination of all copy pairs having the primary volume2351included in the designated snapshot group as the copy source, the creation date and time of the snapshot of all volumes235included in the snapshot group will coincide, and the generation management of snapshots to be multiplexed between different storage apparatuses can be facilitated thereby.

Moreover, since the copy pairs included in the other snapshot groups among the copy pairs in a suspended status will become a PAIR status, the copying of copy pairs to be shared with the snapshot groups other than the designated snapshot group can be started, and the management of snapshots to be multiplexed between different storage apparatuses can be further facilitated without affecting the snapshot creation of other snapshot groups.

Further, since the snapshot duplication method to be applied between the storage apparatus200A and the storage apparatuses200B to200D is decided based on the snapshot duplication evaluation table325which evaluates a plurality of snapshot duplication methods with a prescribed index, it is possible to decide the optimal snapshot duplication method upon storing the snapshot from the volume235of the storage apparatus200A to the volume235of the storage apparatuses200B to200D. Thus, it is possible to issue a snapshot creation command without having to give consideration to the inter-storage apparatus configuration between the storage apparatus200A and the storage apparatuses200B to200D, and the management of snapshots to be multiplexed between different storage apparatuses can be further facilitated.

In addition, upon creating a snapshot group, since a volume235capable of storing the snapshot of the primary volume2351and which is not included in an existing snapshot group is output, it is possible to notify the candidate volume235that may be added to the new snapshot group to the storage administrator A, and the storage administrator A will thereby be able to easily select the volume235to be added to the snapshot group to be newly created.

Moreover, since the available volume235is output based on the acquired failure information among the volumes235storing the snapshot of the primary volume2351, it is possible to notify the storage administrator A of the volumes235storing the snapshot of the primary volume2351and which are other than the volumes235that are no longer available due to a failure. Thus, the storage administrator A is able to easily comprehend the available volumes235storing the snapshot of the primary volume2351, and, for example, will be able to change the volume235to be used by the business application or restore the same to the primary volume2351.

Further, since the volumes that are not able to store the snapshot of the primary volume2351are output based on the acquired failure information, it is possible to notify the storage administrator A of the volumes235that are not storing the snapshot of the primary volume2351and which cannot store the snapshot of the primary volume2351due to a failure. Thus, the storage administrator A is able to easily comprehend the volumes that are not storing, and which cannot store, the snapshot of the primary volume2351.

The configuration of the present invention is not limited to the foregoing embodiments, and may be subject to various modifications within a range that does not deviate from the gist of this invention.