Abstract:
This invention intends to provide the computer system equalizing the storage capacity immediately and appropriately to multiple real logical areas dynamically providing storage capacity to virtual logical areas. This invention is the computer system which, in the process of performing the dynamic allocation of storage areas to virtual volumes in response to accesses from the higher-level device, when adding pool volumes to the pool, migrates storage areas among multiple logical areas, and maintains the balance of the storage capacity.

Description:
TECHNICAL FIELD 
       [0001]    This invention relates to a computer system, specifically to a computer system which dynamically allocates storage capacity to host devices. 
       BACKGROUND ART 
       [0002]    Conventionally, a computer system providing a large-scale data storage service to the host devices exists. This system is known to include host devices, a storage apparatus to which the host devices are connected, and a management device of the storage apparatus. 
         [0003]    The storage apparatus manages multiple hard disks by the RAID (Redundant Array of Independent/Inexpensive Disks) method. The storage apparatus logicalizes physical storage areas included in multiple hard disks and provides them as logical volumes to the host devices. The host devices access the logical volumes and request and reading and writing of data. 
         [0004]    One of the logicalization technologies of this type is called thin provisioning. The storage apparatus sets a virtual volume, which does not include any physical storage areas, for the host devices. As the host devices make write accesses to the virtual volume, the storage apparatus sequentially allocates storage areas to the virtual volume. 
         [0005]    Therefore, this technology is more effective, compared with the method of allocating a large capacity storage area to logical volumes from the beginning, in that storage resources can be efficiently utilized. This technology is disclosed in Japanese Patent Application No. 2003-015915A and Japanese Unexamined Patent Application Publication No. 2006-338341A. 
         [0006]    The means for providing storage areas to the virtual volume consists of a logical group, called a pool, of multiple logical volumes to each of which physical storage areas are allocated. Each of the multiple logical volumes is called a pool volume. If there is a write access from the host device to the virtual volume, the storage area in the corresponding area of the pool volume is allocated to the access destination of the virtual volume. 
         [0007]    The storage apparatus can store write data from the host device by making the area accessed by the host device for the virtual volume correspond with the area in the pool volume to which the storage capacity is allocated. 
         [0008]    Though the storage apparatus intends to allocate storage areas to the multiple pool volumes equally, with the passage of time, the problem arises that this equality is impaired. Therefore, Japanese Patent Application No. 2008-234158A discloses that, by migrating storage areas among multiple pool volumes, the storage area capacity becomes equal among the multiple pool volumes. 
       CITATION LIST 
     Patent Literature 1 
       [0000]    
       
         PTL 1 Japanese Patent Application No. 2003-015915A 
         PTL 2 Japanese Patent Application No. 2006-338341A 
         PTL 3 Japanese Patent Application No. 2008-234158A 
       
     
       SUMMARY OF INVENTION 
     Technical Problem 
       [0012]    Japanese Patent Publication No. 2008-234158, though disclosing that storage areas are migrated by the external operation, did not go far enough to propose the immediate and reliable equalization of storage capacity among multiple pool volumes. 
         [0013]    Therefore, an object of this invention is to provide a computer system and its control method capable of performing the immediate and reliable equalization of storage capacity to multiple real logical areas that dynamically provides storage capacity to virtual logical areas. 
       Solution to Problem 
       [0014]    In order to achieve the foregoing object, this invention provides a computer system and its control method which migrate a storage area among a plurality of pool volumes so as to maintain the balance of the storage capacity when a pool volume is added to a pool during the course of executing an operation of dynamically allocating a storage area to a virtual volume in response to an access from a host system. 
       Effects of Invention 
       [0015]    This invention can provide the computer system and control methods for the same performing the immediate and reliable equalization of storage capacity to multiple real logical areas that dynamically provides storage capacity to virtual logical areas. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0016]      FIG. 1  is a hardware block diagram showing an embodiment of the computer system related to this invention; 
           [0017]      FIG. 2  is a function block diagram showing the operation of dynamically allocating storage areas performed by the storage apparatus in the computer system shown in  FIG. 1 ; 
           [0018]      FIG. 3  is a memory block diagram of the storage apparatus; 
           [0019]      FIG. 4  is a block diagram showing the VDEV management information; 
           [0020]      FIG. 5  is a block diagram showing the LDEV management information; 
           [0021]      FIG. 6  is a block diagram showing the address management table; 
           [0022]      FIG. 7  is a block diagram showing the POOL management information; 
           [0023]      FIG. 8  is a block diagram showing the VVOL-DIR and the PSCB; 
           [0024]      FIG. 9  is a block diagram showing the mapping relationship among target devices, LDEVs and PDEVs; 
           [0025]      FIG. 10  is a block diagram showing the process in which pages are equally allocated among a virtual volume and pool volumes; 
           [0026]      FIG. 11  is a block diagram showing the configuration of free queues in the above-mentioned case; 
           [0027]      FIG. 12  is a block diagram showing the case the page allocation concentrates in the pool volume added to the pool; 
           [0028]      FIG. 13  is a block diagram showing the configuration of free queues in the above-mentioned case; 
           [0029]      FIG. 14  is a block diagram of a virtual volume and pool volumes before the page rebalance processing; 
           [0030]      FIG. 15  is a block diagram of a virtual volume and pool volumes after the page rebalance processing; 
           [0031]      FIG. 16  is a block diagram of a virtual volume and pool volumes showing the process of considering the capacity ratio of pool volumes and performing the rebalance among the multiple pool volumes; 
           [0032]      FIG. 17A  is a table for managing a pool,  FIG. 17B  is a table for managing a pool volume, and  FIG. 17C  is a table for managing a free queue unit; 
           [0033]      FIG. 18  is a block diagram showing the order of allocating pages from pool volumes belonging to the pool to the virtual volume in response to a write access from the host device to the virtual volume; 
           [0034]      FIG. 19  is a table for managing this allocation order; 
           [0035]      FIG. 20  is a flowchart showing that the storage apparatus performs the rebalance processing when adding pool volumes to the pool; 
           [0036]      FIG. 21  is a flowchart showing the rebalance processing; 
           [0037]      FIG. 22  is a flowchart showing the processing of calculating the number of pages migrated among multiple pool volumes; 
           [0038]      FIG. 23  is a block diagram of a virtual volume and pool volumes showing the calculated number of pages and the process of page migration; 
           [0039]      FIG. 24  is a management table for achieving the migration embodiment shown in  FIG. 23 ; 
           [0040]      FIG. 25  is a flowchart showing the details of the migration processing; 
           [0041]      FIG. 26  is a flowchart showing the further details of the migration processing. 
           [0042]      FIG. 27  is a block diagram of a cache memory performing the migration processing; 
           [0043]      FIG. 28  is a flowchart showing that the I/O processing from the host device is prioritized over page migration; 
           [0044]      FIG. 29  is a block diagram showing the process of queue structure reconfiguration; 
           [0045]      FIG. 30  is a block diagram of the structure of a work queue. 
           [0046]      FIG. 31  is the first flowchart showing queue structure reconfiguration; 
           [0047]      FIG. 32  is the second flowchart showing queue structure reconfiguration; 
           [0048]      FIG. 33  is a block diagram showing the process of page migration among multiple pool volumes and is the precondition for  FIG. 34 ; 
           [0049]      FIG. 34  is a block diagram showing the free queue processing for achieving the page migration; 
           [0050]      FIG. 35  is a block diagram as the precondition for  FIG. 36 ; 
           [0051]      FIG. 36  is a block diagram showing the free queue processing in the case the migration is suspended; 
           [0052]      FIG. 37A  to  FIG. 37F  are the migration progress management screens in which the page usage status is aggregated in units of pool volumes; and 
           [0053]      FIG. 38A  and  FIG. 38B  are the screens in which the number of pages is aggregated in units of virtual volumes. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
       [0054]    An embodiment of this invention is described below.  FIG. 1  is a hardware block diagram showing the first example of a computer system related to this invention. The computer system includes a host computer  10 , a management device  20 , and a storage apparatus  30  to which the above-mentioned devices are connected. The storage apparatus is also referred to as a storage system or a storage subsystem. 
         [0055]    The host computer  10  accesses the logical storage resources of the storage apparatus  30 . The management device  20  manages the configuration of the storage areas in the storage apparatus  30 . The storage apparatus  30  stores data in the storage areas set in the physical devices  34 . 
         [0056]    The host computer  10  includes an input means  110 , an output means  120 , a CPU  130 , a memory  140 , a disk adapter  150 , a network adapter  160 , and a disk drive  170 . The input means  110  is a means for accepting inputs from the administrator operating the host computer  10  and others. The input means  110  is configured of, for example, a keyboard. The output means  120  is a means for displaying the status and the setting items of the host computer  10 . The output means  120  is configured of, for example, a display device. 
         [0057]    The CPU  130  reads the programs stored in the disk drive  170  to the memory  140 , and performs the processing specified by the programs. The memory  140  is configured of, for example, a RAM and others, and stores programs, data, and others. 
         [0058]    The disk adapter  150  is connected to the storage apparatus  30  via a storage network  50 , and transmits and receives data to and from the storage apparatus  30 . The storage network  50  is configured of protocols suitable for data transfer (e.g. Fibre Channel). 
         [0059]    The network adapter  160  transmits and receives data to and from the storage system management device  20  or the storage apparatus  30  via the management network  40 . The management network  40  is configured of, for example, Ethernet. 
         [0060]    The disk drive  170  is configured of, for example, a hard disk device, and stores data and programs. 
         [0061]    The management device  20  includes an input means  210 , an output means  220 , a CPU  230 , a memory  240 , a network adapter  250 , and a disk drive  260 . 
         [0062]    The input means  210  is a means for accepting inputs from the administrator operating the management device  20  and others. The input means  210  is configured of, for example, a keyboard. The output means  220  is a means for displaying the status and the setting items of the management device  20 . The output means  220  is configured of, for example, a display device. 
         [0063]    The CPU  230  reads the management programs stored in the disk drive  260  to the memory  240 , and performs the management processing for the storage apparatus based on the programs. The memory  240  is configured of, for example, a RAM and others, and stores programs, data, and others. 
         [0064]    The network adapter  250  transmits and receives data to and from the host computer  10  or the storage apparatus  30  via the management network  40 . 
         [0065]    The disk drive  260  is configured of, for example, a hard disk device, and stores data and programs. 
         [0066]    The storage apparatus  30  includes a controller  31 , a storage cache memory  32 , a shared memory  33 , physical devices (PDEVs)  34 , a power supply switch  35 , and a power supply  36 . The controller  31  controls the storage of data in storage areas configured in the PDEVs  34 . The storage cache memory  32  temporarily stores the data read from and written into the PDEVs  34 . The shared memory  33  stores the configuration information of the controller  31  and the PDEVs  34 . The PDEVs  34  are configured of multiple disk devices. The power supply  36  supplies power to each unit in the storage apparatus  30 . The power supply switch  35  is the switch for turning on and off the power supply from the power supply  36 . The disk devices (storage apparatuses) are configured of, for example, hard disk drives, and mainly store user data. As the storage apparatuses, drives configured of a semiconductor memory such as a flash memory may also be used. 
         [0067]    The controller  31  is configured of a host adapter  310 , a network adapter  320 , a nonvolatile memory  330 , a power supply control unit  340 , a memory  350 , a processor  360 , a storage adapter  370 , and a shared memory adapter  380 . 
         [0068]    The host adapter  310  transmits and receives data to and from the host computer  10  via the storage network  50 . The network adapter  320  transmits and receives data to and from the host computer  10  or the management device  20  via the storage network  40 . 
         [0069]    The nonvolatile memory  330  is configured of hard disks or flash memory, and stores the programs operating in the controller  31  and the configuration information. The power supply control unit  340  controls power supplied by the power supply  36 . 
         [0070]    The memory  350  is configured of, for example, RAM and others, and stores programs, data and others. The processor  360  reads the programs stored in the nonvolatile memory  330  to the memory  350 , and performs the processing specified by the programs. 
         [0071]    The storage adapter  370  transmits and receives data to and from the PDEVs  34  and the storage cache memory  32 . The shared memory adapter  380  transmits and receives data to and from the shared memory  33 . 
         [0072]      FIG. 2  is a function block diagram showing the operation of dynamically allocating storage areas performed by the storage apparatus  30  in the computer system shown in  FIG. 1 . From the PDEVs  34 , a RAID group is configured by the RAID configuration. From this RAID group, a virtual device (VDEV)  400  is configured (S 101 ). The VDEV  400  is split into multiple logical devices (LDEVs)  500  which are storage areas. The VDEV configured of PDEVs  34  is called a “Type 1 VDEV.” The LDEVs included in this Type 1 VDEV are called “LDEVs Type 1.” 
         [0073]    The host computer  10  accesses the volumes (logical areas) in the storage apparatus  30 . The volumes seen from the host computer  10  are called “target devices.” The target devices  700  are set by defining a path in the volume including the Type 1 LDEVs to the host computer  10  (S 102 ). 
         [0074]    External physical devices  600  connected outside the storage apparatus  30  may also be treated the same as the PDEVs  34 . That is, by the RAID configuration, multiple Type 1 VDEVs  400  are configured from multiple external physical devices (EDEVs)  600  (S 103 ). The Type 1 VDEVs  400  are split into one or more storage areas; i.e. LDEVs Type 1  500 . The target devices  700  are set by setting a path in this LDEV Type 1  500  to the host computer  10  (S 104 ). 
         [0075]    Furthermore, in the storage apparatus  30 , a Type 2 VDEV  401  can be set. The Type 2 VDEV is, unlike the Type 1 VDEV configured of the PDEVs  34 , a virtual device including an address and an area but not including any areas corresponding with the PDEVs  34 . It is possible to set a cache memory area for a Type 2 VDEV. For this Type 2 VDEV  401 , one or more LDEVs are configured. This LDEV is called a Type 2 LDEV  501 . 
         [0076]    The target devices  701  are set by setting a path in this LDEV Type 2  501  to the host computer  10  (S 110 ). These target devices correspond with the virtual volumes. 
         [0077]    To Type 2 VDEVs and LDEVs Type 2, physical storage areas are not allocated from the PDEVs (they do not have any storage areas). For the host computer  10  to use them, the Type 2 LDEVs must be associated with storage pools (or simply called “pools”)  60 . 
         [0078]    The storage pools  60  are groups of one or more LDEVs Type 1  500  that are coordinated by the storage pool attribute. The Type 1 LDEV  500  is allocated to the storage pool (S 112 ). The Type 1 LDEV  500  is a pool volume. 
         [0079]    This LDEV Type 1  500  set as a storage pool is associated with the Type 2 LDEV  501  by using an address (S 111 ). This association is called “mapping” or “allocation.” 
         [0080]    By doing this, the host computer  10  can make write/read accesses to and from the virtual volumes. Furthermore, by changing the mapping between the Type 1 LDEV and the Type 2 LDEV, the storage areas of the virtual volumes can be changed. 
         [0081]      FIG. 3  is a block diagram of a memory  350  in the storage apparatus  30 . The memory  350  stores various types of programs read and performed by the processor  360 , the configuration information  351  related to the LDEV settings, and the pool information  352  related to the storage pool settings. 
         [0082]    The command control program  3501  interprets commands from the host computer  10  or the management device  20 , and performs the processing specified by the commands. The configuration control program  3503  controls the configuration of the storage apparatus  30 . The disk I/O program  3505  controls accesses to the PDEVs  34 . The pool control program  3507  sets storage pools. 
         [0083]    The configuration information  351  stores the settings related to the VDEVs and LDEVs in the storage apparatus  30 . The pool information  352  stores the settings related to the storage pools. 
         [0084]    The configuration information  351  stores the settings related to the VDEVs and LDEVs in the storage apparatus  30 . The pool information  352  stores the settings related to the storage pools. 
         [0085]    The configuration information  351  includes the address management table  3511 , the LDEV management information  3512 , and the VDEV management information  3514 . 
         [0086]    The address management table  3511  stores the address mapping information among target devices, LDEVs, VDEVs, and physical devices. The address management table  3511  includes the target device-LDEV mapping information  35111 , the LDEV-VDEV mapping information  35112 , and the VDEV-PDEV mapping information  35113 . The LDEV management information  3512  stores the information related to the LDEVs. The VDEV management information  3514  stores the information related to virtual logical volumes. 
         [0087]    The pool information  352  includes the POOL management information  3521 , the POOL-VOL management information  3522 , the VVOL-DIR  3523 , and the PSCB  3524 . The POOL management information  3521  stores the settings of the storage pools. The POOL-VOL management information  3522  stores the information of the volumes in the storage pools. The VVOL-DIR  3523  stores the address allocation to the LDEVs in the storage pools. The PSCB information stores the information of the addresses of the LDEVs in the storage pools. 
         [0088]    Furthermore, the memory  350  includes the allocation processing program  3508  for equally allocating storage areas from the PDEVs to the multiple LDEVs (pool volumes)  500  belonging to the storage pools and the rebalance processing program  3509  for equalizing the storage capacity of multiple pool volumes by migrating storage areas among multiple pool volumes. 
         [0089]    Furthermore, the memory  350  includes the flag information capable of setting whether to perform rebalance or not in units of pools or in units of virtual volumes. 
         [0090]      FIG. 4  is a block diagram showing the VDEV management information. The VDEV management information is configured of the VDEV unique information  35141 . The VDEV unique information  35141  is configured of the VDEV number (VDEV #)  35142 , the emulation type  35143 , the total size  35144 , the remaining size  35145 , the device attribute  35146 , the device status  35147 , the set number of LDEVs  35148 , the LDEV number  35149 , the head VDEV-SLOT #  35150 , and the last VDEV-SLOT #  35151 . 
         [0091]    The VDEV # (VDEV number)  35142  is the identifier of the VDEV. The emulation type  35143  is the identifier of the emulation type of the VDEV. The total size  35144  is the total size set for the VDEV. The remaining size  35145  is the size of the unused area in the VDEV. 
         [0092]    The device attribute  35146  is the identifier of the attribute defined for the VDEV. If the VDEV is a Type 1 VDEV, the identifier showing the Type 1 VDEV is stored and, if the VDEV is a Type 2 VDEV and set as a virtual volume, the identifier showing the Type 2 VDEV is stored. 
         [0093]    The device status  35147  is the identifier showing the VDEV status. The VDEV status includes normal, blockade, failure blockade, and others. Blockade shows the blockade due to a cause other than the occurrence of a failure such as puncture blockade. Failure blockade shows the blockade due to the occurrence of a failure in any one of the devices. 
         [0094]    The set number of LDEVs  35148  is the total number of LDEVs set for the VDEVs. The LDEV number  35149  stores the numbers of the LDEVs set in the VDEV. The head VDEV-SLOT #  35150  is the identifier of the physically head slot number of the set LDEV. 
         [0095]    The last VDEV-SLOT #  35151  is the physically last slot number of the set LDEV. The above-mentioned LDEV number  35149 , the head VDEV-SLOT #  35150 , and the last VDEV-SLOT #  35151  are set for each LDEV number; i.e. the same number as that of LDEVs. 
         [0096]      FIG. 5  is a block diagram showing the LDEV management information. The LDEV management information is configured of VDEV unique information  35121 . The VDEV unique information  35121  is configured of the LDEV number (LDEV #)  35122 , the emulation type  35123 , the size  35124 , the head slot number  35125 , the last slot number  35126 , the path definition information  35127 , the device attribute  35128 , the device status  35129 , the program use status  351300 , and the POOL-ID  351301 . 
         [0097]    The LDEV #  35122  is the identifier of the LDEV. The emulation type  35123  is the identifier of the emulation type of the LDEV. The size  35124  is the total size set for the LDEV. 
         [0098]    The head slot number  35125  is the identifier of the head slot number of the set LDEV. The last slot number  35126  is the last slot number of the set LDEV. The path definition information  35127  is the identifier of the path defined for the host computer  10 . 
         [0099]    The device attribute  35128  is the identifier of the LDEV attribute. If the LDEV is a Type 1 LDEV, the identifier showing the Type 1 LDEV is stored and, if the LDEV is a LDEV Type 2, the identifier showing the Type 2 LDEV is stored. Furthermore, if the LDEV is set in the storage pool, the identifier showing the pool attribute is stored. 
         [0100]    The device status  35129  is the identifier showing the status of the VDEV to which the LDEV belongs. The status of the VDEV includes normal, blockade, failure blockade, and others. Blockade shows the blockade due to a cause other than the occurrence of a failure such as puncture blockade. Failure blockade shows the blockade due to the occurrence of a failure in any one of the devices. 
         [0101]    The program use status  351300  stores, if the LDEV is being used by a program, the identifier of the program. The POOL-ID  351301 , if the LDEV is set in the storage pool, stores the identifier. 
         [0102]      FIG. 6  is a block diagram showing the address management table. The address management table  3511  stores the mapping information of the target device, the LDEV, and the VDEV. The address management table  3511  includes the target device-LDEV mapping information  35111 , the LDEV-VDEV mapping information  35112 , and the VDEV-PDEV mapping information  35113 . 
         [0103]    The target device-LDEV mapping information  35111  stores the correspondence of the target device address with the LDEV address. The LDEV-VDEV mapping information  35112  stores the LDEV address and the VDEV address. 
         [0104]    The VDEV-PDEV mapping information  35113  stores the VDEV address and the RAID group (or parity group) number for the same, and the PDEV address. 
         [0105]    The storage apparatus  30 , by referring to this address management table, can ascertain which address of which LDEV the target device address corresponds with. It can also be ascertained which address of which VDEV the LDEV address corresponds with. Furthermore, it can also be ascertained which RAID group the VDEV address belongs to and which address of which PDEV the VDEV address corresponds with. 
         [0106]      FIG. 7  is a block diagram showing the POOL management information. The POOL management information  3521  is configured of POOL unique information  35211 . The POOL unique information  35211  is configured of the POOL-ID  35212 , the attribute/purpose  35213 , the emulation type  35214 , the capacity  35215 , the unused capacity  35216 , the threshold  35217 , the status  35218 , the number of POOL-VOLs  35219 , the POOL-VOL device number list  35220 , the number of POOL-using devices  35221 , and the POOL-using device numbers  35222 . 
         [0107]    The POOL-ID  35212  is the identifier of the POOL. The attribute/purpose  35213  is the identifier showing the attribute and the purpose of the storage pool. The purpose is the purpose in the operation form such as snapshots and SYS areas. 
         [0108]    The emulation type  35214  is the identifier of the emulation type of the storage pool. The capacity  35215  is the total capacity of the storage pool. The unused capacity is the size of the unused area in the storage pool. 
         [0109]    The threshold  35217  is the maximum data storage capacity allowed by the storage pool. The status  35218  is the current status of the storage pool, for example, being defined, being extended, valid, etc. The POOL-VOL number  35219  is the total number of LDEVs set as the storage pool. 
         [0110]    The POOL-VOL device number list  35220  is a list of LDEV numbers set as the storage pool. The number of POOL-using devices  35221  is the number of the Type 2 LDEVs with which the LDEVs of the storage pool are associated. The POOL-using device number  35222  is a list of the numbers of the Type 2 LDEVs with which the LDEVs of the storage pool are associated. 
         [0111]      FIG. 8  is a block diagram showing the VVOL-DIR  3523  and the PSCB  3524 . The VVOL-DIR  3523  is the configuration information of the Type 2 LDEVs which are the storage areas of the virtual volume. The PSCB (POOL Slot Control Block)  3524  is the configuration information of the Type 1 LDEVs as a storage pool. 
         [0112]    As mentioned above, the storage apparatus  30  configures a Type 1 VDEV from the PDEVs  34  by the RAID configuration. This Type 1 VDEV is split into LDEVs Type 1 which are storage areas. The LDEVs Type 1 are set as the storage pool. The volume configured of these LDEVs Type 1 set as the storage pool is considered to be a POOL-VOL  900 . 
         [0113]    block diagram of Furthermore, the storage apparatus  30  sets a virtual volume (VVOL) and configures a Type 2 VDEV. This type of Type 2 VDEV is split into LDEVs Type 2 which are storage areas. 
         [0114]    The storage apparatus  30  allocates the Type 2 LDEVs of the VVOL  800  to the Type 1 LDEVs of the POOL-VOL  900 . By doing this, the storage areas of the virtual volumes accessed by the host computer  10  are set for the Type 1 LDEVs configured of the PDEVs  34  which are physical devices. 
         [0115]    The VVOL configuration is stored in the VVOL-DIR  3523 . The VVOL-DIR  3523  is configured of the LDEV number (LDEV #)  35231  and the entry  35232 . 
         [0116]    The LDEV number (LDEV #)  35231  is the identifier of the Type 2 LDEV. The entry  35232  is the configuration information of the Type 2 LDEV. This entry  35232  is configured of the Type 2 LDEV address  35233  and the PSCB pointer  35234 . 
         [0117]    The Type 2 LDEV address  35233  stores the address of the Type 2 LDEV of the VVOL. 
         [0118]    The PSCB pointer  35234 , if the Type 2 LDEV is allocated to the Type 1 LDEV of the POOL-VOL  900 , stores the pointer for the area of the Type 1 LDEV. Note that, as the Type 2 LDEV is not allocated to the Type 1 LDEV in the initial status, the PSCB pointer  35234  stores “NULL.” 
         [0119]    The PSCB (POOL Slot Control Block)  3524  is the information of the Type 1 LDEV set in the storage pool. This PSCB  3524  is set for each slot of the Type 1 LDEVs set in the storage pool. 
         [0120]    The PSCB  3524  is configured of the LDEV number (LDEV #)  35241 , the POOL-VOL address  35242 , the PSCB front pointer  35243  and the PSCB rear pointer  35244 . 
         [0121]    The LDEV number (LDEV #)  35241  is the identifier of the Type 1 LDEV in the POOL-VOL. The POOL-VOL address  35242  is the address of the Type 1 LDEV in the POOL-VOL  900 . 
         [0122]    The PSCB front pointer  35243  and the PSCB rear pointer  35244  are the identifiers of the front and rear slots of the Type 1 LDEV in the POOL-VOL  900 . 
         [0123]    Furthermore, among the areas in the POOL-VOL  900 , as for the unused area, its head is shown by the free PSCB queue  35240 . The free PSCB queue  35240  includes the pointer for the PSCB  3524  showing the next slot. 
         [0124]    The storage apparatus  30 , referring to the pointer shown by the free PSCB queue  35240 , obtains the next PSCB  3524 . Furthermore, referring to the PSCB rear pointer  35245  of the next PSCB  3524 , the storage apparatus  30  gradually traces PSCBs  3524 , and obtains the PSCB  3524  corresponding with the last slot of the unused area. 
         [0125]    The PSCB rear pointer  35244  of this last PSCB  3524  is the free PSCB queue  35240 . The storage apparatus  30  traces the free PSCB queues  35240  and, by the set linked by the pointers of the PSCBs  3524 , can ascertain the unused areas of the POOL-VOLs  900  in the storage pool. 
         [0126]    The storage apparatus  30  sets the PSCBs  3524  corresponding with the Type 1 LDEVs set as the storage pool. As more specifically described, the PSCB  3524  corresponding with each slot of the Type 1 LDEVs set as the storage pool is set, and the free PSCB queue  35240  is also set. As the entire storage pool is unused in the initial status, the set linked by the free PSCB queue  35240  corresponds with all the areas in the Type 1 LDEVs set as the storage pool. 
         [0127]    When the storage apparatus  30  uses the areas of this storage pool, by allocating the PSCBs  3524  for the required slots to the VVOL-DIR  3523  which are LDEVs Type 2, the relevant areas become usable. Multiple sets of slots correspond with pages. Pages are specified from multiple PSCBs. Accesses from the host device to VVOLs  800  and allocation of storage areas from the POOL-VOLs to the access areas of the VVOLs  800  is performed in units of pages. 
         [0128]    As more specifically described, the storage apparatus  30  refers to the free PSCB queue  35240 , and then obtains the PSCB  3524  for the required areas (pages) to be allocated to the Type 2 LDEVs. Each of these obtained PSCBs  3524  is allocated to the entry of the VVOL-DIR  3523 . That is, the PSCB pointer  35234  of each entry of the VVOL-DIR  3523  stores the pointer showing the corresponding PSCB  3524 . 
         [0129]    Note that the allocated PSCBs  3524  are excluded from the linkage by the free PSCB queue  35240 . 
         [0130]    By this processing, each page (slot) of the Type 2 LDEVs is allocated to the PSCB  3424  shown by the PSCB pointer  35234  of each entry of the VVOL-DIR  3523 . As the PSCBs  3524  correspond with the slots of the Type 1 LDEVs, as a result, the Type 2 LDEVs are allocated to the Type 1 LDEVs, and the virtual volumes as the access targets of the host computer  10  become usable as physical devices. 
         [0131]      FIG. 9  is a block diagram showing the relationship among target devices, LDEVs and PDEVs. The host computer  10  accesses the target device  700  set in the storage apparatus  30 . The storage area of the target device  700  is a Type 1 LDEV  500 . 
         [0132]    The Type 1 LDEV  500  is the area of the Type 1 VDEV  400  configured by the RAID configuration from the physical volume. The Type 1 VDEV  400  is split into one or more LDEVs Type 1  500 . 
         [0133]    Furthermore, the host computer  10  accesses the virtual volume set in the storage apparatus  30  as a target device  701 . The virtual storage area of this target device  701  is a Type 2 LDEV  501 . The Type 2 LDEV  501  is associated with the Type 1 LDEV  500  set as a storage pool. 
         [0134]    The host computer  10  accesses the target device  701  which is a virtual volume. The storage area of this virtual volume is a Type 2 LDEV  501 . The storage apparatus  30 , when receiving an access to the Type 2 LDEV  501 , converts the Type 1 LDEV  500  associated with the Type 2 LDEV  501  into the access destination. 
         [0135]      FIG. 10  shows the process in which pages as access targets and units of storage areas are equally allocated among a virtual volume (Type 2 LDEV) and multiple pool volumes (Type 1 LDEVs). 
         [0136]    In  FIG. 10 , for the accesses to the virtual volume (HDP-Vol) # 0  and the virtual volume # 1  from the host computer, the processor (MP) # 0  allocates pages from the pool volumes (Pool Vols) # 0  to # 2  to the virtual volumes # 0  and # 1 . 
         [0137]    For the accesses to the virtual volume # 2  and the virtual volume # 3  from the host computer, the processor (MP) # 1  allocates pages from the pool volumes (Pool Vols) # 0  to # 2  to the virtual volumes # 2  and # 3 . 
         [0138]    Accesses from the host computer are made to the virtual volumes # 0  to # 3  sequentially, such as WR 1  to WR 5 . MP # 0  allocates the page (WR 1 ) of the pool volume # 0  to the page (WR 1 ) of the virtual volume # 0  accessed by the host computer, and allocates the page (WR 2 ) of the pool volume # 1  to the page (WR 2 ) of the virtual volume # 1  accessed by the host computer. 
         [0139]    Next, the MP # 1  allocates the page (WR 3 ) of the pool volume # 0  to the page (WR 3 ) of the virtual volume # 2  accessed by the host computer, and allocates the page (WR 4 ) of the pool volume # 1  to the page (WR 4 ) of the virtual volume # 3  accessed by the host computer. 
         [0140]    Next, the MP # 0  allocates the page (WR 5 ) of the pool volume # 2  to the page (WR 5 ) of the virtual volume # 0  accessed by the host computer. 
         [0141]    This allocation is achieved by the above-mentioned queue structure in  FIG. 8 .  FIG. 11  is an example of the conventional free queue structure. The addresses of unallocated pages in each pool volume (freeM-N: M shows that the page belongs to the pool volume #M, and N shows that the page is the Nth unallocated page of the pool volume M) are sequentially connected to the free queues from ( 0 ) to ( 2 ). Each MP accesses the free queues from ( 0 ) to ( 2 ) sequentially, and allocates unallocated areas of the pool volumes to the virtual volumes. 
         [0142]    The MP # 0  accesses the free queue ( 0 ). As the free 0 - 1  is connected to the free queue ( 0 ) at first, the first unallocated page of the pool volume # 0  is allocated to the page WR 1  of the virtual volume # 0 . This unallocated page becomes the allocated (used) page (WR 1 ). 
         [0143]    Next, the MP # 0  accesses the free queue ( 1 ). As the free 1 - 2  is connected to the free queue ( 1 ) at first, the second unallocated page of the pool volume # 1  is allocated to the page WR 2  of the virtual volume # 1 . This unallocated page becomes the allocated (used) page (WR 2 ). 
         [0144]    Next, the MP # 1  accesses the free queue ( 0 ). As the free 0 - 1  of the free queue ( 0 ) was used for allocation, the next free 0 - 4  is selected, and is allocated to the page WR 3  of the virtual volume. The page used for the allocation is released from the free queue. 
         [0145]    Next, the MP # 1  accesses the free queue ( 1 ). As the free 1 - 2  of the free queue ( 1 ) was used for allocation, the next free 1 - 5  is selected, and is allocated to the page WR 4  of the virtual volume. 
         [0146]    Next, the MP # 0  accesses the free queue ( 2 ). As the free queue ( 2 ) is connected to the free 2 - 3  at first, the third unallocated page of the pool volume # 2  is allocated to the page WR 5  of the virtual volume # 0 . 
         [0147]    As shown in  FIG. 11 , for making the equal number of allocated pages exist for multiple pool volumes, the pool volumes to which the free areas connected to the free queues belong are supposed to shift next by each free queue. Furthermore, the page allocation order for the accesses from the host computer, that is, 
         [0000]    free queue # 0 →# 1 →# 2 →# 0 →# 1  . . . .
 
is stored in the memory  350  as the page allocation order management table. The processor always refers to this order table and performs page allocation. The processor stores write data from the host device in the pages allocated in the virtual volumes.
 
         [0148]      FIG. 12  shows the process in which the pool volume # 3  is added when the number of unallocated pages in the pool volumes # 0  to # 2  belonging to the pool decreased.  FIG. 13  shows the queue structure for managing the unused areas of the added pool volume. As the pool volume # 3  was newly added, a large number of unused areas of the pool volume # 3  are connected to each of the free queues. 
         [0149]    In the embodiment of  FIG. 10 , for distributing the load from access processing from the host computer by setting multiple processors, a different virtual volume is allocated to each processor. As the accesses from the host computer to the virtual volumes do not become equal among the multiple virtual volumes, with the passage of time, the balance of the rates of used pages among multiple pages is lost. 
         [0150]    Furthermore, in the embodiment of  FIG. 13 , as accesses from the processor concentrate in the pool volume # 3  including a number of free areas, the processing performance for I/O from the host computer might be deteriorated. 
         [0151]    Therefore, as shown in  FIG. 14  and  FIG. 15 , the processor causes the allocated pages to migrate from the existing pool volumes to the pool volume added to the pool and equalizes the rate of the allocated pages among the existing pool volumes and the added pool volume. 
         [0152]      FIG. 14  shows the status of the above-mentioned processing and  FIG. 15  shows the status after the above-mentioned processing. In  FIG. 14 ,  1400  is the used area for which the page allocation was performed, and  1402  is the unused area. As shown in  FIG. 15 , the data of a part of the used pages in the pool volume # 0  is copied to the unused pages of the added pool volume # 2  (S 1404 ), and further, the data of a part of the used pages in the pool volume # 1  is copied to the unused pages of the pool volume # 2  (S 1406 ). 
         [0153]    Then, the processor changes the allocation from the virtual volume # 0  to the pool volume # 1  (S 100 ) to the allocation to the pool volume # 2  (S 104 ), and changes the allocation from the virtual volume # 2  to the pool volume # 0  (S 102 ) to the allocation to the pool volume # 2  (S 106 ). This sequence of process is called page migration. 
         [0154]    The process from  FIG. 14  to  FIG. 15  shows that the multiple used pages are relocated and equalized among multiple pool volumes, that is, the page rebalance is achieved. Equal page allocation is performed among multiple volumes in the pool by the processor performing the equal allocation program  3508 . The processor performs the rebalance program  3509  and performs the rebalance processing. 
         [0155]    As a result of the above-mentioned processing, the page allocation for entire multiple pool volumes becomes equal, and therefore, the I/O rate for each of the multiple pool volumes becomes the same level, and the processing performance of the storage apparatus for the accesses from the host device is kept appropriate even after adding pool volumes. 
         [0156]    Note that, depending on the date and time of updating the data in the migration source pages, the newer the date and time of update, when migrating data to the migration destination pages, the closer to the outer edge of the hard disk the data of the migration source pages is copied. 
         [0157]    For equalizing the number of allocated pages among the multiple pool volumes, it is important to consider the capacity of the pool volumes. That is, it is important to equalize the rate of pages used for the allocation (page usage rate) of all the pages (equivalent to “the total number of pages that can be allocated: the pool volume capacity”) among the multiple pool volumes. The same consideration is important if the page usage rate is calculated by the storage capacity instead of the number of pages. 
         [0158]      FIG. 16  is a block diagram showing the above-mentioned consideration. The reference sign  1300  is the total capacity of the pages allocated to each volume, and  1302  is the capacity of the pages to be migrated among the volumes. For the capacity ratio of the pool volumes, the pool volume # 0 :# 1 :# 2  is 1:3:2. 
         [0159]    Therefore, for equalizing the page usage rate, pages are migrated to the pool volume # 2  from the pool volumes # 1  and # 3  respectively. 
         [0160]      FIG. 17A  is a table for managing a pool, and  FIG. 17B  is a table for managing a pool volume. These tables are stored in the memory  350  as part of the pool information  352  ( FIG. 3 ). These tables are created from the management information. The rebalance processing program  3509  of the storage system refers to these tables and establishes the rebalance processing system to be described in  FIG. 15 . 
         [0161]      FIG. 17C  is a table for managing a free queue unit. The allocation of unused pages to the access areas in the virtual volume is, as mentioned above, managed by free queues. The free queues are set in units of pool volumes. To the free queue, the multiple unused pages of the pool volumes are connected by the FIFO structure. When unused pages are allocated to the virtual volume, the unused pages are sequentially released from the free queue, and processed as the destination of mapping from the virtual volume and of data copy. 
         [0162]    This table is configured of the free queue identification number (#), the pool volume # as the target of the management for unused areas by the free queue, the number of free areas linked to the free queue, and the pool # to which the free queue belongs. This table is also recorded in the memory  350  as a part of the pool information. 
         [0163]      FIG. 18  is a block diagram showing the order of allocating pages from pool volumes belonging to the pool to the virtual volume in response to a write access from the host device to the virtual volume. The description here assumed the pool including multiple pool volumes whose storage capacity ratio is 2:3:2. 
         [0164]    For allocating pages of the virtual volume to the pool volumes equally, it is required to specify the order of page allocation for the volumes including the volumes added to the pool volumes. This allocation order is registered in the memory as the management information (refer to  FIG. 19 ). 
         [0165]    The allocation processing program  3508  refers to this management information, calculates the address of the targeted free queue from the multiple free queues, obtains the addresses of the unused pages connected to the free queues (pointer information: PSCB), and stores the write data transmitted to the pages of the virtual address from the host device in those addresses. 
         [0166]    For realizing this, the free queue address and the pool volume identification information correspond with each other by 1:1. For example, the free queue # 0  manages the free area of the pool volume # 0 , the free queue # 1  manages the free area of the pool volume # 1 , and the free queue # 2  manages the free area of the pool volume # 2 . 
         [0167]      FIG. 18  shows that the pages are allocated to the pool volume # 0  twice in a row, then to the pool volume # 1  three times in a row, and further to the pool volume # 2  twice in a row, and this cycle is repeated afterwards. 
         [0168]      FIG. 19  is a table for managing this allocation order and registers the allocating method shown in  FIG. 18 . This table is also recorded in the memory as the pool information. The allocation processing program  3508  refers to this management table and manages the order control processing of the page allocation for the pool volumes. 
         [0169]    This program includes a software counter and manages to which pool volumes pages have been allocated so far. As the counter is incremented each time a page is allocated, the pool volume as the target of the current page allocation can be ascertained by the position of the counter.  FIG. 19  shows that the counter is at the pool volume # 1  (free queue # 1 ). 
         [0170]    The rebalance processing program  3509  of the storage apparatus, as shown in  FIG. 20 , with page relocation being the triggers for performing, detects the addition of pool volumes to the pool. 
         [0171]    In  FIG. 20 , the rebalance processing program  3509  in the storage apparatus monitors the unused capacity of the pool for a certain period of time (S 33001 , S 3302 ), and determines whether the total usage amount of the pool (the capacity of all the pages used for allocation) has exceeded the threshold or not (S 33003 ). 
         [0172]    If the usage amount exceeds the threshold, the management device  20  or the maintenance computer performs the processing for newly adding pool volumes to the pool. The storage apparatus updates the management information shown in  FIGS. 5 to 7  in accordance with the addition of pool volumes (S 33004 ). After that, the processing shifts to the rebalance (S 33020 ). 
         [0173]    The details of the rebalance processing are described below with reference to  FIG. 21 . The rebalance processing program determines the possibility of rebalancing the pool which is the target of the rebalance (S 7010 A). This determination is performed by, for example, referring to the pool information  352  in  FIG. 3 . 
         [0174]    An example of the cases in which rebalance is determined to be impossible for the pool as the target of rebalance is described below. 
         [0000]    Pool status: Pool is blocked
       The number of volumes in the pool is equal to or larger than the specified value ( 8 )
 
Pool volume status: Being formatted
   In process of degenerate   Blocked
 
Virtual volume: Blocked
       
 
         [0178]    Rebalance is also determined to be impossible due to a failure in hardware resources in the storage apparatus (the processor, the shared memory, the cache memory, and the PDEVs), depending on the level of the failure. 
         [0179]    Furthermore, if the rebalance OFF flag is set for the pool as the rebalance target, the processor determines rebalance is impossible to perform. The rebalance ON/OFF setting information can be set in units of pools or in units of virtual volumes. 
         [0180]    If the rebalance is determined to be possible, the rebalance processing program performs the preprocessing (S 7010 ), for example, determines the pool volumes as the source and as the destination of page migration (changing the page allocation or migration), determines the order of selecting the pool volumes for which the page migration processing is performed, and simulates the capacity of the pool volumes after the page migration, etc. 
         [0181]    Next, the rebalance processing program calculates the number of pages to be migrated in the pool (S 7011 ), and then performs the page migration processing (S 7012 ). At this point of migration, as described later, the fixed pattern existing in the multiple pool volumes (e.g. the page whose data is all “0”) is discarded, and the processing for reducing the pool volume capacity is performed at the same time. 
         [0182]    In page migration, the rebalance processing program determines whether there are any conditions for suspending the rebalance processing or not. For example, if there is any I/O to the page as a rebalance target from the host computer, the rebalance is suspended. The rebalance processing may be resumed after the cause of suspending the rebalance is removed. 
         [0183]    Next, the rebalance processing program manages the progress of the page migration (S 7013 ). By this management, the rebalance processing program can disclose the progress of the rebalance to the user. 
         [0184]      FIG. 22  is a flowchart showing the details of calculating the number of pages to be migrated in  FIG. 21  (S 7011 ). The method of calculating the number of pages at this point is specifying the number of pages balanced among the multiple pool volumes as the ideal number of pages and making up the difference between this number and the number of the currently used pages. The rebalance processing program searches the usage amount of pages in each pool volume which is allocated to an HDP VOL (virtual volume). 
         [0185]    The description of  FIG. 22  assumes the following items. The total usage amount of the pool by the HDP VOL #i (the total number of pages allocated to (used by) all the volumes belonging to the pool): A [i] The total usage amount of the pool volume #j by the HDP VOL #i: B [i] [j] The capacity of the pool volume #j (total number of pages): C [j] The capacity of the entire pool (total number of pages): D The capacity of all the added pool volumes (total number of pages): E 
         [0186]    Firstly, the rebalance processing program calculates the number of pages by the pool volume [j] allocated to the HDP VOL [i] before adding pool volumes, and stores it in B [i] [j] (S 1900 ). B [i] [j] is set as the work area of the memory  350 . 
         [0187]    Next, the rebalance processing program calculates the number of pages in the pool volume [j] to be used by the HDP VOL [i] after adding pool volumes to the pool by the method described below (S 1902 ). 
         [0188]    Total usage amount of pool by HDP VOL #i× Ideal usage rate of pool volume 
         [0000]      [ j]=A[i]×C[j ]/( D+E ) 
         [0189]    The ideal usage rate of the pool volume is the rate of the pages allocated to the pool volumes to the total number of pages that can be stored in the pool volume, and is the value assuming that the pages are equally allocated among multiple pool volumes. 
         [0190]    The number of pages involving migration for the pool volume [j] is the difference between before and after adding pool volumes as described below (S 1904 ). 
         [0000]      Number of pages= B[i][j]−A[i]×C[j ]/( D+E ) 
         [0191]    If the above-mentioned calculated value is a positive value (+), the pool volume [j] is a migration source volume and, if the calculated value is a negative value (−), the pool volume [j] is a migration destination volume. 
         [0192]    After calculating the number of migrated pages, the rebalance processing program, for each pool number, registers the number of migrated pages and whether the pool volume is the migration source or the migration destination in the exclusive table. 
         [0193]      FIG. 23  is a block diagram showing the number of pages calculated by the processing of  FIG. 22  and the process of page migration. The page migration is based on the result of calculating page migration of each virtual volume, in the multiple pool volumes belonging to a storage pool group and in the multiple virtual volumes corresponding with them. The numeric values in the symbols of pool volumes show the number of pages to be used for allocation in the pool volumes. 
         [0194]    As shown in  FIG. 23 , as for the virtual volume # 0 , no page migration is performed among multiple pool volumes. As for the virtual volumes # 1  and # 2 , in the direction of the arrows shown in  FIG. 23 , the pages of the number shown above the arrows are migrated. The rebalance processing program performs page migration in units of pools. That is, all the page migration shown in  FIG. 23  is performed. Note that the page migration in units of virtual volumes may also be permitted. For example, it is possible to perform page migration among multiple pool volumes for the virtual volume # 1  and not to perform page migration for the virtual volume # 2 . 
         [0195]      FIG. 24  is a management table for achieving the page migration embodiment shown in  FIG. 23 . This management table is stored in the specified area of the memory  350 . The management table exists in each virtual volume and is created with reference to the processing of the step S 1906  in  FIG. 22 . Whether each pool volume is the migration source or the migration destination is registered for each virtual volume, and the number of pages migrated from the pool volume as the migration source to the pool volume as the migration destination is registered in the pool volumes as the migration source and the migration destination respectively. 
         [0196]    The rebalance processing program creates and registers this table by the processing of S 7011  ( FIG. 21 ), refers to this table at S 7012 , and performs migration. 
         [0197]      FIG. 25  is a flowchart showing the details of the migration processing in  FIG. 21  (S 7012 ). This processing is, if there are multiple virtual volumes, performed for each of the virtual volumes. 
         [0198]    The rebalance processing program detects the related pages from the virtual volume. Though the migration target is the pages of the pool volume, by referring to the address management table  3511 , the related pages corresponding with the pages as the migration target can be identified from all the pages in the virtual volume. 
         [0199]    The rebalance processing program determines whether each of all the pages from the head page in the virtual volume is a related page or not. The software counter is used for managing to which pages the check has been done so far. As the page check progresses, the counter is incremented. Therefore, the rebalance processing program performs the counter initialization (S 2300 ). 
         [0200]    Next, the rebalance processing program selects the pages in the virtual volumes from the head pages (S 2302 ). Next, whether the pages as the migration target of the pool volume are allocated to the selected pages or not is determined (S 2304 ). 
         [0201]    The rebalance processing program, if affirming the determination, detects the identification information of the pool volume to which the pages as the migration target belong with reference to the address management table  3511  (S 2308 ). 
         [0202]    Next, with reference to the tables in  FIG. 24 , whether the pool volumes are the migration source or the migration destination is determined (S 2306 ). The rebalance processing program, if the pool volumes are the migration destination, saves unused areas for migrating pages from the migration source pool volume in the migration destination pool volume (S 2310 ). 
         [0203]    For the pool volume to which the pages are migrated, the one with the low allocation balance is selected but, if multiple pool volumes compete as the migration destinations, the added pool volumes are selected first, and then the pool volumes with the higher priority to be the migration destinations are sequentially selected. In the migration destination priority table, the priority of each pool volume to be the migration destination is registered. 
         [0204]    The rebalance processing program, after determining the migration destination pool volume, the migration source pool volume, and the number of migration target pages, determines the page migration schedule, and registers it in the migration execution job of the memory  350  (S 2312 ). 
         [0205]    This schedule is, for involving the migration of multiple pages for example, for determining the order of performing the page migration. This schedule is updated and registered each time pages are selected sequentially at S 2302  and each time a new migration destination volume or others are determined. 
         [0206]    Next, the rebalance processing program increments the count value of the counter (S 2316 ), and determines whether any pages not selected at S 2302  still exist or not (S 2318 ). If any pages not considered yet still exist, the processing returns to the step S 2302 , and further, if there is no page, the migration is performed in accordance with the schedule sequentially updated at S 2312  (S 2320 ). Note that the migration processing (S 2320 ) may also be performed after S 2312 . 
         [0207]      FIG. 26  shows the migration processing in  FIG. 25  (S 2320 ). The storage adapter  370 , in accordance with the rebalance processing program, stages all the data in the migration source page from the migration source pool volume to the cache memory  32  in units of slots (S 2400 ). 
         [0208]      FIG. 27  shows the above-mentioned process. The pool volume # 1  is the migration source, and the pool volume # 2  is the migration destination. S 2600  shows the process of the migration source page being staged to the first area  32 A of the cache memory. The reference sign  2601  is a slot. Staging is performed in units of slots. The reference sign  2601 A is a migration source page, and the reference sign  2601 B is a migration destination page. 
         [0209]    The DRR circuit in the storage adapter  370  (Data Recover and Reconstruct circuit) checks the data in the migration source page  2601 A in units of slots and checks whether the data is “0” or not ( FIG. 26 : S 2402 -S 2404 ,  FIG. 27 : S 2602 ). 
         [0210]    This determination is performed for determining whether the data of the migration source page is one of the fixed patterns (all data is “0”) or not. If data is a fixed pattern, page migration is not performed, and mapping for the migration target pages in the virtual volumes is changed to the area exclusive for storing fixed pattern data. 
         [0211]    If the DRR detects the existence of “1,” the processing shifts to the step  2414 , and the memory controller in the cache memory performs the copy between cache memories of the data in the migration source pages from the first area  32 A to the second area  32 B ( FIG. 26 : S 2414 ,  FIG. 27 : S 2604 ). Then, the storage adapter  370  clears the migration source cache memory  32 A ( FIG. 27 : S 2416 ). 
         [0212]    Next, the data in the second area  32 B is destaged to the migration destination page  2601 B of the migration destination pool volume # 2  ( FIG. 27 : S 2606 ), the migration source page  2601 A is changed to an unused page, and further, the migration processing such as changing the allocation for the pages in the virtual volume from the migration source page to the migration destination page ( FIG. 26 : S 2418 ). 
         [0213]    When the data of the slot determined by the DRR is “0,” the storage adapter  370  determines whether the check for all the data in units of slots is completed or not (S 2406 ), and if it is determined to be negative, the processing returns to the step S 2402 , and whether checks the other data in the same slot is “0” or not. When the check for all the data in the slots is completed, the processing shifts to the step S 2410 , checks whether there are slots remaining in the migration source page or not and, if it is determined to be positive, performs the above-mentioned check for the next slot. 
         [0214]    When the determination for all the slots affiliated to the page is completed, the processing shifts to the step S 2412  and, as for the migration source page whose data is all “0,” the migration is determined to be unnecessary and is changed to an unused page. 
         [0215]    If the storage apparatus performs the processing shown in  FIG. 26 , when relocating pages, the pages with the fixed pattern data are released from the already used management, which prevents the duplicated allocation of the fixed pattern data to the virtual volume, and consequently, enables the host device to efficiently utilize the storage areas in the pool volume. 
         [0216]    In the process of migration described in  FIG. 26 , if I/O from the host computer or copy I/O in the storage apparatus is issued for the migration source page, the storage apparatus gives priority to the I/O processing and suspends the page migration. 
         [0217]    That is, as shown in  FIG. 28 , if read or write I/O is provided to the virtual volume, the processor checks whether the logical address to the migration source page is included or not (S 2500 ). 
         [0218]    As the fact that the logical address to the migration source page is included means that the path between the virtual volume and the migration source page is not disconnected and that the page migration processing is not completed, the processor clears the data in the second area of the cache memory ( 32 B in  FIG. 27 ) where the data of the migration source page is stored, and further, performs the clearing processing for the migration destination page (S 2504 ). 
         [0219]    The clearing processing described here means, if the mapping is for the virtual volume, the deletion of the mapping management information and, if the migration destination page is managed as the used page, means the deletion of the management information. 
         [0220]    Next, the processor, at S 2506 , skips (completes) the migration processing in  FIG. 26 . Note that, as for read I/O, it may also be permitted to suspend the migration processing until this processing is completed and then to resume it. 
         [0221]    As described in  FIG. 27 , the cache memory  32  includes the first area (cache area for the migration source)  32 A and the second area (cache area for the migration destination)  32 B and, as the data in the migration source page of the second area is not copied to the migration destination page  2601 B until there is no difference between them, the page migration processing can be suspended when I/O reaches the migration source page. 
         [0222]    In the free queue structure shown in  FIG. 11 , the unused page information (PSCB) of multiple pool volumes was connected to one free queue. By this method, when allocating pages to a virtual volume, the processor could not recognize of whose pool volume the unused pages were to be allocated to the virtual volume. 
         [0223]    In the rebalance processing, as allocated pages are migrated from the pool volumes with the higher rate of allocated pages to the pool volumes with the lower rate, it is necessary to recognize to which pool volume the migration destination pages belong. Therefore, the processor must change the free queue structure to the structure in which one pool volume is allocated to one free queue. 
         [0224]      FIG. 29  is a block diagram showing the process in which the queue structure is changed. The processor releases the information of the unused areas connected to the free queue # 0  from the free queue # 0  and connects it to the work queue  2700  # 1  (( 1 )→( 2 )). The processor configures the free queues for making the correspondence of the free queue # 0  with the pool volume # 1  (( 3 )), releases the information of the free areas ( 1 - 1 ,  1 - 2 ) belonging to the pool volume # 1  from the work queue # 1 , and connects it to the free queue # 0  (( 3 )). 
         [0225]      FIG. 30  shows the structure of a work queue. By releasing the link of the information of the unused pages connected to the free queue from the free queue and connecting it to the work queue, no I/O from the host computer is allowed to be provided to the free queue while reconfiguring the queue structure. Next, the processor releases the information of the unused pages from the free queue # 1 , and connects it to the work queue # 2  (( 4 )). The processor makes the correspondence of the free queue # 1  with the pool volume # 2 , then releases the unused areas ( 1 - 3 ,  1 - 4 ) affiliated to the pool volume # 1  from the work queue # 2 , and connects them to the free queue # 0  (( 4 )-( 5 )). 
         [0226]    Next, the processor releases the information of the unused pages from the free queue # 2 , and connects it to the work queue # 3 . The processor makes the correspondence of the free queue # 2  with the pool volume # 3 , then releases the unused areas ( 1 - 5 ,  1 - 6 ) affiliated to the pool volume # 1  from the work queue # 3 , and connects them to the free queue # 0 . 
         [0227]    As a result of the above-mentioned processing, the free queue # 0  is allocated to the pool volume # 1 , and to the pool volume # 1 , the information of the unused pages all from the pool volume  1  ( 1 - 1  to  1 - 6 ) are connected. 
         [0228]    Furthermore, the processor releases the information of the unused pages in the pool volume # 2  connected to the head of the work queue # 1  from the work queue # 1 , and connects it to the pool volume # 2 . The same processing is performed for the work queue # 2  and then for the work queue # 3 . As a result of this processing, the information of the unused pages in the pool volume # 2  is connected to the free queue # 1  allocated to the pool volume # 2 . 
         [0229]    Furthermore, the processor releases the information of the unused pages in the pool volume # 3  connected to the head of the work queue # 1  from the work queue # 1 , and connects it to the pool volume # 3 . The same processing is performed for the work queue # 2  and then for the work queue # 3 . As a result of this processing, the information of the unused pages in the pool volume # 3  is connected to the free queue # 2  allocated to the pool volume # 3 . 
         [0230]      FIG. 31  is a flowchart for converting the free queue structure. For converting the queue structure, when a host I/O is requested for a free queue, the processor locks the free queue and makes the queue structure conversion wait until the I/O processing is completed. 
         [0231]    The processor checks whether the free queue as the I/O target is locked or not ( 2900 ). If it is determined to be locked, the processor determines whether there is a next free queue to be the I/O target ( 2902 ) and, if this is affirmed, refers to the next free queue ( 2904 ), and returns to the step  2900 . 
         [0232]    If the selected free queue is not locked, the processor locks it and saves the free queue ( 2906 ,  2908 ), allocates unused pages from the free queue to the I/O, and then releases the lock of the free queue ( 2910 ). 
         [0233]    If there is no next free queue at the step  2902 , the processor reports to the host I/O that the target free queue is locked and busy ( 2912 ). 
         [0234]      FIG. 32  is a flowchart for performing the conversion of the queue structure. The processor checks whether the free queue as the target of migration to the work queue is locked or not and, if it is locked, repeats the processing at the step  3000  until the I/O processing for the free queue is completed ( 3002 ). 
         [0235]    If the target free queue is not locked, the processor locks the target free queue ( 3004 ), and connects it to the work queue ( 3006 ). Next, for reconnecting the unused page information from the work queue to another free queue, the processor checks whether the connection destination free queue is locked or not ( 3008 ) and, if it is locked, waits until the I/O processing for the connection destination free queue is completed ( 3010 ). 
         [0236]    If the target free queue is not locked, the processor locks the connection destination free queue ( 3012 ) to this, connects the unused page information (PSCB) released from the work queue ( 3014 ), and releases the lock of the target queue and the connection destination queue ( 3016 ). 
         [0237]    Next, the processor determines whether there is a next connection destination queue to be reconnected from the work queue to another free queue ( 3018 ) and, if this is affirmed, returns to the step  3008  or, if this is not affirmed, completes the queue structure conversion processing. 
         [0238]    Next, the page migration processing of S 2418  in  FIG. 26  is described related to the free queues. This description assumes the migration shown in  FIG. 33 . That is, the process in which the data of the page # 1000  in the pool volume # 0  is copied to the page # 2000  in the pool volume # 3  and the mapping to the former page in the virtual volume (HDP Vol # 0 ) is changed to the mapping to the latter page is described with reference to  FIG. 34 . 
         [0239]    The rebalance processing program  3509  saves an in-migration queue, connects it to the free queue # 3 , then, from the free areas connected to the free queue # 3  allocated to the pool volume # 3 , selects the target area  3302  to be the migration destination page, and connects this pointer information (Pool-Vol # 3 , # 2000 )  3303  to the end of the in-migration queue (queue # 0 )  3300  (S 3300 ). The area to be the migration destination page is managed by a different queue from the free queue for preventing the area to be the migration destination page from being allocated to the virtual volume. 
         [0240]    When the rebalance processing program receives the report that the page data migration was normally completed from the copy job, the pointer information  3304  which is the information of the mapping  3306  for the page in the VVOL DIR (pool volume # 0 , page # 1000 ) is cleared, changed to the pointer information for the address of the migration destination page (Pool-Vol # 3 , # 2000 ) (S 3302 ), and inserts the address of the migration source page in the appropriate position in the queue structure of the free queue #. Furthermore, the migration destination page is released from the in-migration queue. 
         [0241]    As shown in  FIG. 35 , if the process of the data migration from the migration source page to the migration destination page is suspended ( 3400 ) due to I/O from the host device or other causes, as shown in  FIG. 36 , the rebalance processing program saves an in-migration queue, clears the pointer information  3304  set for the migration destination queue, and returns it to the information before the change. 
         [0242]    Furthermore, the program saves the free queue # 3  of the pool volume where the migration destination page  3303  originally existed and, for this free queue, searches for the position for inserting the migration destination page  3303  as a free area. 
         [0243]    The rebalance processing program connects the in-migration queue  3300  to the free queue (Queue # 3 ), releases the migration destination page  3303  from the in-migration queue  3300 , and further, connects the migration destination page to the free queue (Queue # 3 ). 
         [0244]    As mentioned above, the storage apparatus, by utilizing an in-migration queue which is a different queue from a free queue, can achieve the page relocation as well as, even if the factors such as host I/O competing with the page relocation are added to the storage apparatus, give priority to the I/O processing. 
         [0245]      FIGS. 37A to 37F  are the screens for managing the progress of migration and provided to the management user. The page migration progress status is displayed with reference to the result of the above-mentioned calculation of migrated pages.  FIGS. 37A to 37F  are the screens in which the page usage status before migration is aggregated in units of pool volumes.  FIG. 37A  shows the status before the page migration and  FIG. 37F  shows the status after the page migration. The numerals in the figures show the number of pages. 
         [0246]      FIG. 38A  is the screen in which the number of pages before the migration is aggregated in units of virtual volumes.  FIG. 38B  is the screen in which the number of pages after the migration is aggregated in units of virtual volumes. 
         [0247]    Therefore, for the administrative user, the migration progress status is displayed via the management screen. 
         [0248]    In the above-mentioned embodiment, the determination on the page rebalance trigger and the possibility of performing rebalance is performed by the storage apparatus, but this can be performed by the management device. In the former case, the storage apparatus is the computer system of this invention and, in the latter case, the combination of the management device and the storage apparatus is the computer system. 
         [0249]    Furthermore, though the above-mentioned embodiment describes that the number of migrated pages are calculated at the time of migration, it may also be permitted for the administrative user to input the number of migrated pages.