Storage apparatus, storage system, and control method of storage system for dynamically securing free space when a storage apparatus is disused

A storage system capable of dynamically securing free space when a certain storage apparatus is disused and reducing the influence on the system upon dynamically securing the free space without having to continuously secure, in a memory of another storage apparatus, free space capable of storing management information of the corresponding storage apparatus in preparation of a case where such storage apparatus is disused. Each storage apparatus comprises a memory including a management information storage area for storing management information and a cache area for storing cache information, and a processor which manages a status of the cache area. At least certain processors determine a storage apparatus to become a copy destination of the copy target management information. The storage apparatus of the copy destination releases at least a part of the cache area and stores the management information in the released cache area.

TECHNICAL FIELD

The present invention relates to a storage apparatus, a storage system, and a control method of a storage system.

BACKGROUND ART

As the background art of this technical filed, there is the technology described in Japanese Patent Application Publication No. 2006-221526 (PTL 1). This publication describes a technology related to a storage apparatus comprising a plurality of memories, wherein management information is made redundant between different memories and, when a failure occurs in a memory storing the management information, the management information is copied to another memory to ensure redundancy.

CITATION LIST

Patent Literature

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

In connection with a storage system which stores business data required for companies and the like to carry out their business, there are demands for building a system by coordinating a plurality of storage apparatuses in order to flexibly change the system size in accordance with changes in the business circumstances. For example, by building a system with storage apparatuses comprising the storage resources initially required upon introducing the system, and subsequently adding new storage apparatuses while utilizing the existing storage apparatuses when the business grows and the usage of storage resources increases, the storage resources of the overall system can be expanded.

When building a storage system with a plurality of storage apparatuses as described above, the availability of the storage system can be improved by making redundant the management information required for operating the storage system between memories of different storage apparatuses. Moreover, if a failure occurs in a certain storage apparatus configuring the storage system or if a certain storage apparatus configuring the storage system is to be removed and that storage apparatus is disused, the redundancy can be maintained by copying the management information stored in that storage apparatus to another storage apparatus.

In PTL 1, it is necessary to secure free space capable of storing management information in the memory of the copy destination in order to copy the management information upon the occurrence of a failure. Nevertheless, when free space capable of storing management information is continuously secured in the storage apparatus of the copy destination, the usage rate of the memory in normal times will deteriorate, and costs will increase. Meanwhile, when free space capable of storing management information is dynamically secured in the storage apparatus of the copy destination, while the usage rate of the memory in normal times will improve, it is necessary to reduce the influence on the system upon dynamically securing the free space. Thus, it is important to appropriately decide the storage apparatus of the copy destination in light of the usage status of the memory that is managed individually in each storage apparatus.

Thus, an object of this invention is to provide a storage system capable of dynamically securing free space when a certain storage apparatus is disused and reducing the influence on the system upon dynamically securing the free space without having to continuously secure, in a memory of another storage apparatus, free space capable of storing management information of the corresponding storage apparatus in preparation of a case where such storage apparatus is disused.

Means to Solve the Problems

In order to achieve the foregoing object, the storage system of the present invention comprises a plurality of storage apparatuses including a first storage apparatus. Each of the plurality of storage apparatuses comprises a memory which includes a management information storage area for storing management information and a cache area for storing cache information, and a processor which manages a status of the cache area. When a memory of the first storage apparatus is to be disused, at least certain processors of the plurality of storage apparatuses determine a copy destination storage apparatus to become a copy destination of the copy target management information, which is the management information stored in the memory to be disused, based on the status of the cache area that is managed by each of the storage apparatuses other than the first storage apparatus. The processor of the copy destination storage apparatus releases at least a part of the cache area of the memory of the copy destination storage apparatus, and stores the copy target management information in the released cache area.

Advantageous Effects of the Invention

The present invention is able to improve the usage efficiency of the memory by dynamically securing free space when a certain storage apparatus is disused upon dynamically securing the free space without having to continuously secure, in a memory of another storage apparatus, free space capable of storing management information of the corresponding storage apparatus in preparation of a case where such storage apparatus is disused. The present invention is also able to reduce the influence on the storage system upon dynamically securing the free space. Other objects, configurations and effects of the present invention will become more apparent based on the detailed description of the embodiments.

DESCRIPTION OF EMBODIMENTS

An embodiment of the present invention is now explained with reference to the appended drawings.

FIG. 1is a diagram showing a concept of the save processing of the management information in this embodiment.

With the storage system of this embodiment, a plurality of storage apparatuses2are connected via a network41. A memory9of each storage apparatus2includes a management information storage area11for storing management information, and a cache area12for storing cache information. Moreover, a processor8of each storage apparatus2manages the access frequency of the cache area12of the corresponding storage apparatus2.

The management information is information which is stored in at least one of the memories9of the plurality of storage apparatuses2and which can be referenced by at least one of the processors8of the plurality of storage apparatuses2from the memory9when the plurality of storage apparatuses2are operating. In particular, when the respective storage apparatuses2are operating, the processor8of the corresponding storage apparatus2can refer to the management information stored in the memory9of the corresponding storage apparatus2.

The cache information is information which can be deleted from any one of the memories9of the plurality of storage apparatuses2when the same data is stored in a storage device (drive4) equipped in at least one of the plurality of storage apparatuses2. In particular, when the respective storage apparatuses2are operating, the cache information, which is the same data as the data stored in the storage device (drive4) of the corresponding storage apparatus2, may be deleted from the cache area12of the memory9. Here, the term “delete (deletion)” may mean any one among overwriting the corresponding cache information with other cache information, erasing the corresponding cache information, or disabling access to the corresponding cache information.

InFIG. 1, storage apparatuses A, B, C, D are illustrated as examples of the plurality of storage apparatuses2. However, the number of storage apparatuses2to configure the storage system is not limited to this example, and any number of storage apparatuses2may be used so as long as there are at least two or more storage apparatuses2.

The management information storage area11of the memory9of the storage apparatus A and the management information storage area11of the memory9of the storage apparatus B respectively store the same shared management information X, thereby making redundant the shared management information X. By making the management information redundant between different storage apparatuses2as described above, even if one storage apparatus2is subject to a failure and the memory9becomes unusable, the management information stored in the memory9of the other storage apparatus2can be used to continue operations, and the availability of the storage system can thereby be improved.

An example of the management information save processing to be executed in the storage system in the case of disusing the memory9of the storage apparatus B is now explained. When the memory9of the storage apparatus B is disused, the redundancy of the shared management information X will be lost, and the reliability will deteriorate. Thus, based on this processing, the shared management information X is copied to at least either the storage apparatus C or the storage apparatus D, other than the storage apparatuses A, B, to make redundant the shared management information X, and restore the reliability. Here, a case of disusing the memory9of the storage apparatus2is typically a case of disusing the storage apparatus2, and, for instance, this would correspond to cases where the storage apparatus2is subject to a failure or the storage apparatus2is to be removed. For example, the operation/management cost can be reduced by removing certain storage apparatuses2from the plurality of storage apparatuses2configuring the storage system. The shared management information X stored in the memory9to be disused is also referred to as the “copy target management information”.

When the memory9of the storage apparatus B is to be disused, for example, the storage apparatus A that is making redundant the shared management information X with the storage apparatus B detects the disuse of the memory9of the storage apparatus B. The foregoing disuse may be detected by the storage apparatus A periodically or randomly accessing the storage apparatus B, or detected based on an instruction from a management terminal connected to the storage system.

The storage apparatuses C, D other than the storage apparatuses A, B are the copy destination candidate storage apparatuses to become the copy destination candidates of the shared management information X. In the ensuing explanation, let it be assumed that all memories9of the copy destination candidate storage apparatuses are storing the management information (shared management information Y and the like) and the cache information (cache information C1, C2, D1, D2and the like), and do not have any free space for storing the shared management information X. By refraining from continuously securing free space in the copy destination candidate storage apparatuses for storing the management information (shared management information X) of another storage apparatus2, and rather securing a larger cache area12, the usage efficiency of the memory9can be improved.

In this embodiment, information of one of the memories9of the copy destination candidate storage apparatuses is deleted to store the shared management information X. Because the management information is information required for the operation of the storage system and cannot be deleted, the cache information is deleted. The cache area12of the memory9of the storage apparatus2stores cache information (cache data) for accelerating the response performance when a data access request is sent from the host computer1, and often does not have any capacity available for storing the management information that needs to be copied. The cache area12has both a cache area with a high access frequency and a cache area with a low access frequency. Thus, if cache information is randomly deleted to release the cache area, the cache hit ratio will decrease and the access performance will deteriorate considerably. In this embodiment, the access performance is deemed the cache hit ratio, and the deterioration in the access performance is explained as the deterioration in the cache hit ratio. However, the access performance and the deterioration in the access performance are not limited to this example.

Thus, at least a part of the cache area12is released based on the access frequency of the cache area12so that the deterioration in the cache hit ratio will decrease. Consequently, without having to secure unused memory capacity in advance, in cases where the storage apparatus2is subject to a failure or removed, the management information of that storage apparatus2can be stored in the memory9of another storage apparatus2.

In the ensuing explanation, illustrated is a case of storing the shared management information X, which is one type of copy target management information, in one of the copy destination candidate storage apparatuses. Nevertheless, when there are a plurality of pieces of copy target management information and such copy target management information can be stored separately in different storage apparatuses2, the following processing may also be executed for each piece of copy target management information. In the foregoing case, the copy destination storage apparatus may be decided preferentially from the copy target management information with the largest capacity.

Foremost, the storage apparatus A notifies the capacity of the shared management information X to each of the copy destination candidate storage apparatuses C, D. The copy destination candidate storage apparatuses C, D predict the deterioration in the access performance when the shared management information X is copied based on the access frequency of the cache area12managed individually by each of the copy destination candidate storage apparatuses C, D, and send the predicted deterioration to the storage apparatus A. The storage apparatus A determines, as the copy destination storage apparatus, the copy destination candidate storage apparatus in which the predicted deterioration in the access performance is low based on the deterioration predicted by each of the copy destination candidate storage apparatuses C, D (S100). It is thereby possible to reduce the influence on the access frequency of the overall system when the cache area is released. In the example shown inFIG. 1, the copy destination storage apparatus is the storage apparatus C. Details of the access performance deterioration prediction processing performed by each of the copy destination candidate storage apparatuses C, D will be described later with reference toFIG. 10.

Here, while the copy destination storage apparatus was explained as the copy destination candidate storage apparatus in which the deterioration in the access performance is low, the determination does not necessary need to be made only based on the deterioration in the access performance. For example, when there are storage apparatuses2in which the copy target management information is preferentially stored among the plurality of copy destination candidate storage apparatuses, the copy destination storage apparatus may also be determined based on such priority. In the foregoing case, necessary information may also be acquired from each of the copy destination candidate storage apparatuses. For example, because the access performance may deteriorate when the storage apparatus2, in which its processor8is of a high load, performs the copy processing, the load of the processor8of each storage apparatus2may be acquired, and the storage apparatus2in which the load of the processor8is low may be determined to be the copy destination storage apparatus.

The copy destination storage apparatus may be determined based on the status of the cache area12managed by each of the copy destination candidate storage apparatuses. The status of the cache area12may include information of the access frequency of the cache area12. The status of the cache area12may also include a last access time185of the cache area12, information on whether or not the cache area12can be used, and information (busy flag183) regarding whether the cache area12is being used.

The storage apparatus A instructs the copy destination storage apparatus C to secure a capacity for copying the shared management information X. The processor of the copy destination storage apparatus C rearranges the cache information (C1, C2and the like) of the copy destination storage apparatus C based on the access frequency so that the cache area storing the shared management information X becomes a continuous area (S101). Details of this cache information rearrangement processing (S101) will be described later with reference toFIG. 11.

The copy destination storage apparatus C secures free space for storing the shared management information X by releasing at least a part of the cache area12of the memory9of the copy destination storage apparatus C (S102). The copy destination storage apparatus C copies the shared management information X from the storage apparatus A and stores the copied shared management information X in the released cache area12(S103). Accordingly, the shared management information X is made redundant between the memory9of the storage apparatus A and the memory9of the copy destination storage apparatus C.

The copy destination storage apparatus C notifies the storage destination of the shared management information X to at least some or all of the plurality of storage apparatuses2. For example, the copy destination storage apparatus C notifies the storage destination of the shared management information X to the storage apparatus A and the storage apparatus D. Because the storage destination of the shared management information X is notified to the other storage apparatuses2as described above, the notified storage apparatus2can access the shared management information X stored in the copy destination storage apparatus C.

It is possible to dynamically secure free space upon disusing a storage apparatus2and improve the usage efficiency of the memory9without having to continuously secure, in the memory9of another storage apparatus2, free space capable of storing copy target management information of the corresponding storage apparatus2in preparation of a case where such storage apparatus2is disused. It is also possible to reduce the influence on the storage system upon dynamically securing the free space.

FIG. 2is a diagram showing a configuration of the storage apparatus2.

The storage apparatus2includes a drive4as a data storage device, and a storage controller5which executes processing according to a command. The drive4is a non-volatile storage device. The drive4may be, for example, a semiconductor memory device such as a magnetic disk or an SSD (Solid State Drive). One or more drives4and storage controllers5may be provided.

The storage controller5includes a server I/F (interface)6, a drive I/F7, an inter-device coupling I/F40, a processor8, and a memory9, and these components are mutually connected via an internal network. The server I/F6is connected to a host computer1via a network3, and executes transmission/reception processing of commands and data to and from the host computer1. The drive I/F7executes transmission/reception processing of commands and data to and from the drive4. The inter-device coupling I/F40is connected to other storage apparatuses2via a network41, and executes transmission/reception processing of commands and data to and from the other storage apparatuses2. One or more memories9may be provided in one storage controller5. The network3and the network41are communication channels for exchanging commands and data with the devices (host computer1and storage apparatus2) connected thereto, and are, for example, a SAN (Storage Area Network).

The network3and the network41may be the same network, or mutually independent networks. Note that, when the network3and the network41are mutually independent networks, there is an advantage in that the communication performed between the plurality of storage apparatuses2via the network41will not affect the communication performed between the host computer1and the storage apparatuses2via the network3.

The processor8executes the programs in the memory9and performs various types of processing according to the command. The processor8may be an arithmetic unit or a control unit which executes programs. In the ensuing explanation, the processing that is performed by the processor8executing the programs in the memory9may be described as being performed by the storage apparatus2or the storage controller5.

The memory9is a volatile storage device. The memory9may be a storage unit which stores data. The plurality of storage apparatuses2are linked, and mutually share the memories9. The memories9shared by the plurality of storage apparatuses2are assigned a logical memory ID as a unique identifier among the plurality of storage apparatuses2, and a physical memory ID as a unique identifier within the storage apparatus2which includes the corresponding memory9.

The logical volume10is configured from physical storage areas of a plurality of drives5by the storage controller5. The configuration example is, for example, RAID (Redundant Arrays of Inexpensive Disks), and, by using a plurality of drives4, improved reliability based on data redundancy and improved performance based on the parallel operation of the drives4can be expected. The storage controller5provides the logical volume10to the host computer1. The storage apparatus2includes one or more logical volumes10.

The host computer1may be a physical computer including a processor and a memory, or a virtual computer that runs on a physical computer. The host computer1is also referred to as a server. The host computer1is, for example, a workstation that provides an online mail order service. The host computer1stores, in the storage apparatus2, at least a part of the data required for the service provided by the host computer1. The host computer1reads and writes data stored in the storage apparatus2by sending a data access request to the storage apparatus2. As the data access request, for example, used is a read command or a write command that is compliant with the SCSI (Small Computer System Interface) standard. The data access request is also referred to as an I/O (Input/Output) request.

FIG. 3is a diagram showing a configuration of the memory9. The processor8manages the storage area of the memory9by dividing it into a management information storage area11and a cache area12by using a management information arrangement table13and a cache management table18described later.

Management information is stored in the management information storage area11. As the types of management information, there are integrated management information, shared management information, and individual management information. Management information may also be referred to as configuration information or control information depending on the contents of the management information.

The integrated management information is information which is provided in each of the plurality of storage apparatuses2configuring the storage system. As examples of the integrated management information, there are a management information arrangement table13, and a logical/physical conversion table35.

The shared management information is management information that can be referenced not only from the host storage apparatus2, but also from other storage apparatuses2. In the ensuing explanation, the shared management information will be mainly explained as the management information which is made redundant between certain different storage apparatuses2among the plurality of storage apparatuses2configuring the storage system. However, depending on the shared management information, the shared management information may also be made redundant between different memories9within the same storage apparatus2.

The location (address) of the shared management information is managed with the management information arrangement table13, which is integrated management information. Thus, the shared management information stored in the memory9of a certain storage apparatus2can be shared with the other storage apparatuses2. The shared management information includes at least one of either a program to be executed by one of the processors of the plurality of storage apparatuses2, or management information to be used in executing one of the programs. As an example of a program to be executed by the processor8, there is a backup function program which realizes a data backup function. As examples of management information to be used in executing a program, there are backup function control information to be used in executing the backup function program, a logical volume management table26, security setting information, billing information and so on.

The individual management information is management information to be individually managed by each storage apparatus2. As an example of the individual management information, there is a cache management table18including information of the access frequency of the cache area12of the corresponding storage apparatus2. Contents of the individual management information may differ between different storage apparatuses2. For example, because the cache management table18of each storage apparatus2is the management information managing the access frequency of the cache area12of the corresponding storage apparatus2, contents of the cache management table18will differ between different storage apparatuses2. Moreover, the cache management table18is information in which the contents thereof are changed suitably depending on the data access request of the host computer1. When the cache management table18is made redundant between different storage apparatuses2, because synchronization is required each time the contents thereof are changed, the load of the storage system will increase, and the access response performance may deteriorate. Because the cache management table18is normally information that is used within the corresponding storage apparatus2, the cache management table18does not need to be made redundant between different storage apparatuses2.

Accordingly, the storage apparatus2to access the management information will differ depending on the usage and contents of the management information. Management information that is accessed by a plurality of storage apparatuses2needs to be made redundant between storage apparatuses2, and the amount of management information to be copied may be reduced by limiting the copy target management information to the foregoing management information. For example, the copy target management information may be limited to be the shared management information.

Moreover, input/output data based on a data access request from the host computer1is temporarily stored in the cache area12. Data stored in the cache area12is referred to as “cache information”. As the cache information, there are write data and read data. The cache information is information for improving the performance (speeding up) of access to data in a drive4that has a lower performance (that is slower) than the memory9. In other words, the cache information of each storage apparatus2includes at least one of either write data based on a write request from the host computer1connected to the storage apparatus2or read data for use in response to a read request from the host computer1.

For example, when a storage apparatus2receives a write request from the host computer1, the storage apparatus2once stores the write data pertaining to the write request in the memory9, and returns a write completion response to the host computer1. Because the storage apparatus2is able to asynchronously perform the write processing of the write data from the memory9to a low performance drive4(this processing is hereinafter also referred to as the “destage processing”), the storage apparatus2can return a write completion response in a short time in response to the write request from the host computer1.

Moreover, for example, when data pertaining to a read request is stored in the memory9when the storage apparatus2received the read request from the host computer1, the storage apparatus2acquires the corresponding data from the memory9, and sends a reply to the host computer1. When the data pertaining to the read request is not stored in the memory9, the storage apparatus2acquires the corresponding data from the drive4and stores the acquired data in the memory9, and then sends a reply to the host computer1. The data pertaining to the read request is referred to as “read data”. When the read data is stored in the cache area12of the memory9, the storage apparatus2is not required to read the read data from the drive4, which is of a lower performance than the memory9, and can send a reply in a short time in response to the read request from the host computer1.

In order to prevent the loss of individual management information and cache information in the event the memory9is subject to a failure, each storage apparatus2copies and duplicates the individual management information and the cache information between different memories9of the same storage apparatus2to achieve redundancy. However, with regard to the read data, even if the read data stored in the cache of the memory9is lost, because the read data can be read from the drive4and a reply can then be sent to the host computer1, the read data does not need to be duplicated between a plurality of memories9. With regard to the write data, the redundancy may be continued or restored as described later, or the redundancy may be avoided by executing the destage processing. Moreover, when the storage apparatus2comprises a plurality of storage controllers5including a memory9, the individual management information and the cache information may also be made redundant by being stored in the memories9of different storage controllers5within the corresponding storage apparatus2.

Moreover, when the memory9is disused, such as when the memory9is subject to a failure, each storage apparatus2may copy the management information stored in the memory9to be disused to a different memory9within the same storage apparatus2to restore the redundancy. The management information (copy target management information) in the foregoing case may include at least one among individual management information, shared management information, and integrated management information.

Here, the corresponding storage apparatus2determines the cache information to be deleted based on the access performance deterioration prediction processing (FIG. 10) performed with regard to the plurality of memories9in the host storage apparatus2. It is thereby possible to reduce the influence on the access frequency of the corresponding storage apparatus2upon releasing the cache area12. Furthermore, in order to store the copy target management information, the corresponding storage apparatus2executes the cache information rearrangement processing (S101,FIG. 11) based on the access frequency so that the released cache area12will be a continuous area. Here, upon selecting the cache area12to be released, the selection may be made based on a predetermined policy or priority. For example, the memory9including the cache area12to be released may be properly selected based on a policy of making redundant the management information by using the memories9between different storage controllers5within the corresponding storage apparatus2, or a policy of making redundant the management information between different memories9within the corresponding storage apparatus2. The corresponding storage apparatus2secures free space for storing the copy target management information by releasing at least a part of the cache area12of the memory9of the corresponding storage apparatus2. The corresponding storage apparatus2copies the copy target management information from the memory9storing the copy target management information, and stores the copied copy target management information in the released cache area12to restore the redundancy.

FIG. 4is a diagram showing a configuration of the management information arrangement table13. The storage apparatus2uses the management information arrangement table13and manages the management information storage area11of the memory9storing the shared management information. The management information arrangement table13is management information which associates a management information name130, a logical memory ID(A)131, a logical address (A)132, a logical memory ID(B)133, a logical address (B)134, and a size135.FIG. 4shows that, as the storage area of the memory9storing the management information that is identified by the management information name130, an area of the size135has been secured from the logical address (A)132of the memory9that is identified by the logical memory ID(A)131, and an area of the size135has been secured from the logical address (A)134of the memory9that is identified by the logical memory ID(B)133.

For example, according to the example shown inFIG. 4, in the entry in which the management information name130is “security setting information”, as the storage area of the storage destination of the management information referred to as “security setting information”, a storage area having a size of 0x1000 has been secured respectively from the logical address 0x1000 of the logical memory ID1and from the logical address 0x1000 of the logical memory ID2.

With regard to each piece of management information, so that the management information will not be lost even when one of the memories9of the storage destination becomes unusable, different IDs are set to the logical memory ID(A)131and the logical memory ID(B)133that the physical memories9will be different.

Note that, in this embodiment, because the logical volume management table26is shared management information as described above, the logical volume management table26is managed by the management information arrangement table13, but it is not illustrated inFIG. 4. Each storage apparatus2creates a logical volume management table26with regard to the logical volume10of the host storage apparatus2. Accordingly, the logical volume management table26of each storage apparatus2may also be managed by the management information arrangement table13.

FIG. 5is a diagram showing a configuration of the cache management table18. The processor8of the storage apparatus2uses the cache management table14and manages the status of the cache area12of the memory9storing the cache information. In the cache area12, the stored cache information changes dynamically according to the data access of the host computer1. Thus, the small areas of the cache area12indicated by the physical memory ID181and the physical memory address182are managed for each predetermined size. In the example shown inFIG. 5, the predetermined size is represented as “0x1000”.

The cache management table18is management information which associates a cache ID180for identifying the entry, a physical memory ID181, a physical address182, a busy flag183, a use start time184, a last access time185, a dirty data flag186, and a cache hit count187. The cache ID180is the identifier of each entry. The physical memory ID181and the physical address182indicate that the cache area12of the predetermined size is the target storage area of that entry from the physical address182in the memory9that is identified by the physical memory ID181.

The busy flag183indicates the existence of cache information stored in the target cache area12of that entry. The use start time184indicates the time that the current cache information was stored in the cache area12. The last access time185indicates the time that the host computer1last accessed the cache area12of that entry. While the use start time184and the last access time185are indicated in the form of hours, minutes and seconds, they may also be indicated in the form of year, month and day.

The dirty data flag186indicates whether or not the cache information of that entry is dirty data. When the dirty data flag186is ON; that is, when the cache information of that entry is dirty data, this indicates that the cache information is the latest write data, and old data before the update is stored in the drive4. When the dirty data flag is OFF; that is, when the cache information of that entry is not dirty data, this indicates that the cache information is the latest data, and the data stored in the drive4is also the latest data.

The cache hit count187is the frequency that the data pertaining to the read requests from the host computer1existed in the cache area12within a predetermined unit time. The cache hit count187may also be measured with a counter equipped in each storage apparatus2. The cache hit count187within a predetermined unit time may also be calculated from the counter's frequency and the counter's measurement time.

In this embodiment, the access frequency is explained as the cache hit count187within a predetermined unit time. Moreover, in this embodiment, the access frequency being low is explained as the cache hit count187within a predetermined unit time being small. The comparative target is another storage apparatus2or another memory9of the host storage apparatus2. However, the access frequency is not limited to this example. For instance, the access frequency may also be the last access time185. In the foregoing case, the access frequency being low means that the last access time185is old.

FIG. 6is a diagram showing a configuration of the logical volume management table26. The logical volume management table26is management information which associates a logical volume ID260for identifying each logical volume10, a cache guarantee261, a logical volume address262, a cache ID(A)263, a cache ID(B)264, a drive ID265, and a drive address266.

The storage apparatus2manages the storage areas of the logical volume indicated by the logical volume ID260and the logical volume address262for each predetermined size. In the example shown inFIG. 6, the predetermined size is represented as “0x1000”. The logical volume ID260is the identifier of the logical volume10. The cache guarantee261indicates the cache guarantee that is set in the logical volume10. The logical volume address262indicates the address in the logical volume10.

The cache ID(A)263and the cache ID(B)264are the cache areas12corresponding to the logical volume address262. When both the cache ID(A)263and the cache ID(B)264are set, this indicates that write data is stored in these cache areas12. The cache ID(B)263indicates the cache area12of the duplication destination of the write data, and is not set in the case of a read data. Moreover, when neither the cache ID(A)263nor the cache ID(B)264are set, this indicates that no data is stored in the cache area12.

The drive ID265is the identifier for identifying the drive4. The drive address266is a physical address in the drive4, and indicates the physical storage area in the drive4storing the data designated by the logical volume ID260and the address262.

The data access processing to be executed by the storage apparatus2when the host computer1performs data access to the storage apparatus2is now explained. The storage apparatus2receives, from the host computer1, a data access request designating the ID and address of the logical volume10. The storage apparatus2identifies the address of the cache and the address of the drive4associated with the designated address of the logical volume10. Specifically, the storage apparatus2refers to the logical volume management table26, and acquires the cache ID(A)263and the cache ID(B)264, and the drive ID265and the drive address266, which are associated with the logical volume ID260and the logical volume address262designated in the data access request. The acquired cache ID(A)263and cache ID(B)264respectively correspond to the cache ID180in the cache management table18.

Here, as the probability that the data pertaining to the read requests from the host computer1existing in the cache area12(this is hereinafter referred to as the “cache hit ratio”) is higher, the response to the read request will be of higher performance. However, because the cache area12is a shared resource among a plurality of logical volumes10, when the busy flag183is ON (busy) regarding the entries of all cache IDs180, the cache information of an arbitrary entry is deleted, and new cache information is stored therein. Consequently, due to a data access to a certain logical volume10, the cache hit ratio of another logical volume10may deteriorate. Because the user may use a different logical volume10for each service, there are demands for suppressing the interference of performance between the services.

Thus, the storage apparatus2provides a function of guaranteeing the minimum capacity of the cache area12that is usable for each logical volume10. For example, as shown inFIG. 6, the cache guarantee261set by the user is stored for each logical volume10, and, upon selecting the cache information to be deleted, whether or not the cache guarantee261can be satisfied even after the deletion is evaluated. The cache information is deleted when the cache guarantee261can be satisfied, and other cache information is deleted when the cache guarantee261cannot be satisfied.

FIG. 7is a diagram showing a configuration of the logical/physical conversion table35. The logical/physical conversion table35is management information which associates a logical memory ID350and a logical address351as the storage destination of the data recognized by the various programs, a storage ID352for identifying the storage apparatus2as the storage destination that is actually storing the indicated data, and a physical memory ID353and a physical address354in the storage apparatus2. The processor8of each storage apparatus2performs the conversion of the logical memory ID350and the logical address351into the storage ID352, the physical memory ID353and the physical address354, or the reverse conversion thereof, by using the logical/physical conversion table35.

In S102and S103described above, explained was the processing of the copy destination storage apparatus2releasing at least a part of the cache area12of the memory9of the copy destination storage apparatus2and storing the copy target management information in the released cache area12. This processing is now explained in further detail. The copy destination storage apparatus2sets “None” in the cache ID(A)263and the cache ID(B)264of the cache area12storing the cache information to be deleted in the logical volume management table26, and then disuses the cache area12. Furthermore, the copy destination storage apparatus2refers to the cache management table18, and stores the area indicated by the physical memory ID181and the physical address182as the copy destination of the management information regarding the cache ID180to be deleted, and then deletes that entry.

Subsequently, the copy destination storage apparatus2copies the management information to the copy destination of the management information stored in the step of S11. Specifically, the storage apparatus2refers to the management information arrangement table13, and searches for an entry of a memory ID in which the value of the logical memory ID(A)131or the logical memory ID(B)133will become unusable. The storage apparatus2uses the accessible memory address of that entry to read the management information, and write the management in the copy destination of the management information recorded in the step of S11. The storage apparatus2thereafter overwrites the memory ID and memory address of the copy destination on the logical memory ID and the logical memory address of that entry that will become unusable.

The removal processing (FIG. 8) and the failure processing (FIG. 9) of the storage apparatus2are now explained. The foregoing processing is performed by the storage apparatus2, and any storage apparatus2of the storage system may perform these processing. However, desirably, a storage apparatus2other than the storage apparatus2that was subject to a failure performs the foregoing processing. For example, a storage apparatus2storing a copy of the management information stored in the storage apparatus2to be removed or the storage apparatus2subject to a failure may perform these processing. Moreover, when there is a management which is connected to each storage apparatus2and which manages the storage system, the management computer may execute these processing in coordination with the respective storage apparatuses2.

Moreover, the trigger for commencing the removal processing or the failure processing may be the time that the storage apparatus2to execute the processing receives a removal notification or a failure notification from the storage apparatus2to be removed or the storage apparatus2subject to a failure, or the time that the user instructs the execution of the removal processing or the failure processing from the management terminal. Moreover, the failure processing may also be executed when a storage apparatus2that is communicating with the respective storage apparatuses2detects a failure of another storage apparatus2.

FIG. 8is a flowchart showing the removal processing of the storage apparatus2. The processing of removing one or more storage apparatuses2among the plurality of storage apparatuses2is now explained.

The storage apparatus2calculates the total of the capacity of the management information (this is hereinafter referred to as the “management information amount”) and the cache guarantee in each storage apparatus2, and the total memory amount after the removal of the storage apparatuses2(this is hereinafter referred to as the “memory amount after removal”). The storage apparatus2evaluates whether or not the memory amount after removal is greater than the total of the management information amount and the cache guarantee (this is hereinafter referred to as the “required memory amount”) (S1). Here, the storage apparatus2may also calculate the required memory amount and the memory amount after removal based on the management information arrangement table13, the cache management table18and the logical volume management table26of each storage apparatus2. Otherwise, the storage apparatus2may request each storage apparatus2to return the memory amount, the management information amount and the cache guarantee, and calculate the required memory amount and the memory amount after removal based on the memory amount, the management information amount and the cache guarantee returned from each storage apparatus2.

When the memory amount after removal is greater than the required memory amount, the save processing of saving the management information stored in the memory9of the storage apparatus2to be removed to the memory9of another storage apparatus2is executed (S2). This management information save processing (S2) has been previously explained with reference toFIG. 1. Subsequently, information indicating that the storage apparatus2may be removed may also be displayed on the user's management terminal (S3). The user can remove the storage apparatus2upon confirming the displayed message.

In the evaluation of S1, when the memory amount after removal is not greater than the required memory amount, a message to the effect that the storage apparatus2may not be removed because, if removed, necessary information cannot be stored in the memory of the remaining storage apparatuses2, may also be displayed on the user's management terminal (S4). Here, the storage apparatus2may also display to the user, for example, information indicating the level of insufficiency of the memory amount after the removal. By referring to the displayed message, the user may review the cache guarantee28and reduce the required memory amount, and then cause the storage apparatus2to once again execute the removal processing.

FIG. 9is a flowchart showing the failure processing of the storage apparatus2. The processing to be performed when one or more storage apparatuses2among the plurality of storage apparatuses2is subject to a failure is now explained.

When the storage apparatus2detects a failure in another storage apparatus2, the storage apparatus2executes the same processing as the step of S1inFIG. 8. When the memory amount after removal is greater than the required memory amount in the evaluation of S1, the management information save processing of S2is executed.

When the memory amount after removal is not greater than the required memory amount in the evaluation of S1, the storage apparatus2refers to the failure policy (S5). A “failure policy” is information which prescribes whether to give preference to reliability or performance when it is not possible to support both the duplication of management information and the cache guarantee, and is set by the user.

When the policy gives preference to performance (S5: No), the single status of the management information is maintained, and the reliability is not restored (S8). In step S8, for example, the storage apparatus2may also display a message on the user's management terminal to the effect that the reliability cannot be restored due to an insufficient memory amount. Moreover, the storage apparatus2may also display information on the user's management terminal for urging the user to review the cache guarantee or promptly replace the defective parts.

When the policy gives preference to reliability in S5(S5: Yes), the required memory amount is recalculated by deeming only the cache amount that is actually being used among the cache guarantee as the required memory amount (S6). Specifically, the existence of the cache ID(A)263and the cache ID(B)264is tabulated for each logical volume ID260of the logical volume management table26, and the busy cache amount is thereby calculated. The calculated busy cache amount and the cache guarantee261are compared, and the smaller value is set as the memory amount to be guaranteed by the logical volume10. The total of double the size135of the management information arrangement table13and the memory amount to be guaranteed by each logical volume10is set as the required memory amount, and then compared with the memory amount after removal.

When the memory amount after removal is greater than the recalculated required memory amount (S6: Yes), the management information save processing of S2is executed. When the memory amount after removal is not greater than the recalculated required memory amount (S6: No), the step of S8is executed.

It is thereby possible to satisfy the user's request of giving preference to the reliability of the service than the performance of the service. Moreover, because the cache amount to be used will not increase unless the data access pattern of the host computer1is changed, it is possible to maintain the reliability without deteriorating the performance.

After executing the step of S2, the restoration of the reliability of the management information is completed (S7). In the step of S7, for example, the storage apparatus2may also display information on the user's management terminal indicating the occurrence of a failure and the restoration of the reliability. Moreover, the storage apparatus2may also display information on the user's management terminal to the effect that the storage apparatuses2that were not subject to a failure may continued to be operated as is. Moreover, the storage apparatus2may also display information on the user's management terminal for urging the user to replace the defective parts.

FIG. 10is a flowchart of the access performance deterioration prediction processing. In the ensuing explanation, an example of a case where the deterioration in the access performance is the deterioration in the cache hit ratio is described. This processing is executed by each of the copy destination candidate storage apparatuses. Thus, the copy destination storage apparatus and the cache area12to be released can be determined based on the usage status of the cache area12of the memory9of each of the copy destination candidate storage apparatuses that the deterioration in the cache hit ratio can be suppressed.

Foremost, the copy destination candidate storage apparatus2acquires the capacity of the copy target management information (S14). As explained in the example ofFIG. 1, the processing may also be executed by the storage apparatus A notifying the capacity of the copy target management information (shared management information X) to each of the copy destination candidate storage apparatuses C, D. Otherwise, the copy destination candidate storage apparatus refers to the management information arrangement table13, and searches for an entry with a memory ID in which the value of the logical memory1D(A)131or the logical memory ID(B)133will become unusable. The total value of the size135of the entry may also be the capacity of the copy target management information. The capacity of the copy target management information is set to be the “cache information amount to be deleted” at such point in time.

Subsequently, the copy destination candidate storage apparatus2uses the cache management table18, which is individual management information, and rearranges the cache areas12of all of its memories9in order from the cache area12with the lowest usage frequency (S15). The objective of this process is to rearrange the cache information in the order that is unlikely to deteriorate the cache hit ratio, and, for example, the cache information is rearranged in the order of last access time185of the cache management table18. Otherwise, when the last access time185is the same, a method of rearranging the cache information by giving preference to the oldest use start time184or a method of rearranging the cache information by giving preference to the lowest cache hit count187may also be adopted, and this method is not limited to the method described in the step of S15so as long as the past usage status of the cache is used.

The copy destination candidate storage apparatus2identifies the cache areas12in order from the cache area12with the lowest usage frequency as rearranged in the step of S15(S16), and evaluates whether or not it is necessary to secure the corresponding cache area12(S17). Specifically, the copy destination candidate storage apparatus2refers to the cache management table18, identifies the entry that matches the extracted cache area12, and identifies the entry of the logical volume management table26having the corresponding entry ID180. The copy destination candidate storage apparatus2compares the amount of assignment of the cache area12, which is indicated by the cache ID(A)263and the cache ID(B)264, and the cache guarantee261, in the identified entry. When the amount of assignment of the cache area12of the logical volume10will not fall below the cache guarantee even when the cache area12is released (S17: No), the copy destination candidate storage apparatus2evaluates that no guarantee is required, and executes the step of S18. Meanwhile, when the amount of assignment of the cache area12of the logical volume10will fall below the cache guarantee even when the cache area12when the cache area12is released (S17: Yes), the copy destination candidate storage apparatus2determines that guarantee is required, and executes the step of S16.

After evaluating that guarantee is required in the step of S17, the copy destination candidate storage apparatus2sets the corresponding cache area12as the cache area12to be deleted (S18), subtracts the size of the corresponding cache area12from the “cache information amount to be deleted”, and updates the “cache information amount to be deleted” (S19). When the updated “cache information amount to be deleted” is greater than 0, the copy destination candidate storage apparatus2evaluates that the deletion of further cache information is required, and executes the step of S16(S20: Yes).

When the updated “cache information amount to be deleted” is 0 or less (S20: No), the copy destination candidate storage apparatus2refers to the cache management table18, and calculates the predictive value of the deterioration in the access performance by diving the total cache hit count187per predetermined unit time in the cache area12to be deleted by the read frequency, or number of read requests, that the storage apparatus received per predetermined unit time (S200). However, the predictive value of the deterioration in the access performance may also be calculated based on other methods, and it is not limited to the method described in S200.

FIG. 11is a flowchart of the cache information rearrangement processing. This processing is executed by the copy destination storage apparatus. Based on this processing, it is possible to secure a continuous memory area required for storing the copy target management information, and suppress the deterioration in the access performance after the release of the cache area12.

While the ensuing explanation describes the processing of rearranging the cache areas12to be deleted in one location for the sake of simplification, rearrangement is not limited to one location. Moreover, if it is possible to manage the copy target management information by dividing it into a plurality of memory areas, the cache areas12may be rearranged in a plurality of memory areas, or, if it is possible to dispose the copy target management information in a continuous memory area of the cache area12, then the rearrangement itself may be omitted. However, as the copy target management information is managed by being divided into a plurality of smaller memory areas, the memory consumption required for storing the segmentalized management information in the storage destination memory will increase. For example, the entries of the management information arrangement table13will increase. Thus, the usage efficiency of the memory9can be improved by storing the management information in a relatively large continuous area.

Foremost, the copy destination storage apparatus2selects an available memory9(S21), and lists the cache areas12to be deleted (S22).

Subsequently, the copy destination storage apparatus2sets a highly dense area of the cache areas12to be deleted in the cache area12of the rearrangement destination (S23). The objective of this process is to reduce the number of pieces of cache information to be rearranged.

The copy destination storage apparatus2extracts the top cache area12among the listed cache areas12(S24), and evaluates whether that cache area12exists outside the memory area of the rearrangement destination (S25). When that cache area12exists outside the memory area of the rearrangement destination (S25: Yes), the copy destination storage apparatus2exchanges the data of the cache information, which is not a deletion target and which exists within the area, and the data of the extracted cache information (S26). Specifically, the copy destination storage apparatus2mutually moves the cache information of the areas indicated by the physical address182between the entries of the cache management table18, and thereafter also mutually moves the physical address182, the use start time184, the last access time185, the dirty data flag186, and the access hit count187between the entries. Note that, because this processing is the rearrangement of the cache information and not a data access to the cache area12, the use start time184and the last access time185are not changed.

When it is evaluated that the cache information is within the memory area of the rearrangement destination in the step of S25(S25: No), or upon executing the step of S26, the copy destination storage apparatus2refers to the cache management table18, and refers to the dirty data flag186of the entry corresponding to the cache area12extracted in the step of S24(S27).

When the dirty data flag186is ON (S27: Yes), because this means that the data of the drive4is old data and the cache information of the cache area12is the latest data, the copy destination storage apparatus2stores the cache information in the drive4before deleting the cache information (S28). Specifically, the copy destination storage apparatus2refers to the logical volume management table26, and identifies the entry in which the value of the cache ID(A)263and (B)264coincides with the cache ID180. The cache information is stored in the area indicated by the drive ID265and the drive address266of the identified entry. The dirty data flag186is thereafter set to OFF. It is thereby possible to delete the cache information of the cache area12.

When the dirty data flag186is OFF (S27: No), because this means that the data of the drive4is the latest data, the copy destination storage apparatus2can delete the cache information of the cache area12.

Subsequently, the copy destination storage apparatus2confirms whether or not there is any remaining cache area12among the listed cache areas12(S29). If there is a remaining cache area12(S29: Yes), the copy destination storage apparatus2executes the step of S24. If there is no remaining cache area12(S29: No), the copy destination storage apparatus2ends the processing.

Note that, in this embodiment, in the cache information rearrangement processing, the processing was explained in the order of replacing the cache information inside/outside the memory area of the rearrangement destination (S26), storing the cache information to become the latest data in the drive (S28), and deleting the cache information (S102). Nevertheless, when the step of S28and the step of S102are executed first, because the cache information, which is within the memory area of the rearrangement destination and which is not to be deleted, only needs to be moved to an unused area outside the area in the step of S26, the data copy frequency can be reduced in comparison to the case of replacing the cache information. The order of processing is not limited to the order described above so as long as the cache area12to be deleted is selected so that the cache hit ratio will not deteriorate, and the copy target management information is duplicated in the deleted area.

By executing the access performance deterioration prediction processing explained with reference toFIG. 10and the cache information rearrangement processing explained with reference toFIG. 11, the copy destination storage apparatus2executes the following processing. The processor8of the copy destination storage apparatus2uses the cache management table18and divides the cache area12of the memory9of the copy destination storage apparatus into a plurality of small areas, and manages the access frequency of each of the small areas. Here, the term “small area” refers to the cache area12that is managed in each entry of the cache ID180of the cache management table. The copy destination storage apparatus2releases at least a part of the cache area12by disusing the cache information stored in a small area with a low access frequency based on the access frequency of each small area. Here, the processor8of the copy destination storage apparatus2uses the logical volume management table26to manage the cache guarantee261set in the logical volume10, and disuses the cache information stored in a small area o the cache area12with a low access frequency when the cache guarantee is satisfied even when such small area is released. It is thereby possible to suppress the deterioration in the access performance of the copy destination storage apparatus after the cache area12is released.

Moreover, in the processing of determining the copy destination storage apparatus (S100), by refraining from using a storage apparatus2which was subject to a memory failure in the past as the copy destination storage apparatus, it may be possible to omit the management information save processing in a future failure. For example, when the number of storage apparatuses2is gradually increased in accordance with the growth of the service, the storage apparatus will comprise both new and old memories9. Here, if a failure occurs in a certain memory9of a relatively old storage apparatus2which was purchased initially, even if the cache area12of another memory9of that storage apparatus2is released and the management information is copied therein, because this memory9is also relatively old as with the malfunctioned memory9, it is likely that a failure will occur. Thus, it is also possible to preferentially release the cache area12of the memory9of another relatively new storage apparatus2and copy the management information therein.

In the management information copy processing of S103, as described above, the copy destination storage apparatus stores the copy target management information in the released cache area12. Here, the logical memory ID131and the logical address132of the management information arrangement table13may be changed. Nevertheless, when the various programs to be executed by the storage apparatus2are statically storing the logical memory ID131and the logical address132as the storage destination of the shared management information, or the values previously read from the management information arrangement table13during the initial startup of the programs are being reused, the copy target management information cannot be accessed if the storage destination of the copy target management information is changed in the step of S103.

Thus, in this embodiment, if the storage destination of the copy target management information is changed in the step of S103, the logical/physical conversion table35is updated without changing the logical memory ID131and the logical address132of the management information arrangement table13. Specifically, the copy destination storage apparatus2uses the configuration information arrangement table13and the logical/physical conversion table35and identifies the logical memory ID131and the logical address132of the memory9storing the copy target management information in the storage apparatus2subject to a failure or the storage apparatus2to be removed. The copy destination storage apparatus2updates the entries of the identified logical memory ID131and logical address132in the logical/physical conversion table35by using the physical memory ID353and the physical address354of its own memory9which is its own storage ID352and storage destination of the copy target management information.

Moreover, because the logical/physical conversion table35is integrated management information respectively provided in all of the storage apparatuses2, the logical/physical conversion table35of each storage apparatus2is updated. For example, the copy destination storage apparatus2updates the logical/physical conversion table35of each storage apparatus2by notifying the storage destination of the copy target management information (storage ID352, physical memory ID353, physical address354) to at least some or all of the plurality of storage apparatuses2. For example, the notified storage apparatus2notifies the storage destination of the copy target management information to the storage apparatus2to be disused and the storage apparatuses2other than the notified storage apparatus2. Each of the notified storage apparatuses2updates the logical/physical conversion table35.

FIG. 12is a flowchart showing the access processing of accessing the shared management information using the logical/physical conversion table35. This processing is executed by each storage apparatus2. For example, when the program executed by the processor8of the storage apparatus2designates a logical memory ID and a logical address and attempts to access the shared management information, the following access processing for accessing the shared management information is executed.

Foremost, the storage apparatus2refers to the logical/physical conversion table35, and searches for an entry which matches the designated logical memory ID and logical address from the management information arrangement table13(S30). When there is a matching entry (S30: Yes), the storage apparatus2evaluates that the designated logical memory ID131and logical address132indicate the management information storage area, and executes the step of S31. Meanwhile, when there is no matching entry (S30: No), the storage apparatus2determines that it is not the management information storage area, and determines that the data access was unsuccessful (S34).

In the step of S31, the storage apparatus2refers to the logical/physical conversion table35, and searches for an entry having the logical memory ID350and the logical address351that match the designated logical memory ID and logical address.

Subsequently, the storage apparatus2accesses the data in the memory area indicated by the storage ID352, the physical memory ID353and the physical address354in the entry searched in the step of S31(S32), and determines that the data access was successful (S33).

As described above, as a result of providing the logical/physical conversion table35, the program executed by the processor8of the storage apparatus2can access the management information by using the same logical memory ID131and logical address132even when the storage location of the shared management information is changed in S103.

In the foregoing explanation, a method of continuing the duplication of management information in cases where the number of available memories9will decrease due to the removal or failure of a storage apparatus2was described. Here, when a storage apparatus2is added or the number of available memories9increases based on failure recovery, the storage apparatus2seeks to improve performance by storing the cache information in the added memory9. However, when the step of S8, which is processing to be performed during a memory failure, is implemented and the status is such that the management information has not been duplicated, the added memory9may be selected as the save destination of the management information, and the management information may be duplicated based on the step of S103to restore the reliability.

Moreover, when the status during the normal times of the management information arrangement table13and the logical/physical conversion table35of the storage system (status before failure or removal of the storage apparatus) is stored, and one or more storage apparatuses2are to be added to the storage system after the failure or removal of the storage apparatus, the status may be restored to the status during normal times. For example, when adding one or more storage apparatuses2to the storage system, it is also possible to store the integrated management information and the copy target management information in the memory9of the added storage apparatus2, release the area storing the copy target management information among the memories9of the copy destination storage apparatus2, and store the cache information in the released area.

While the foregoing explanation described the various types of information of the storage system by using expressions such as “table” and the like, the data structure of such information is not limited thereto, and other data structures may also be used. Because each piece of information is not dependent on the data structure, for instance, “kkk table” may also be referred to as “kkk information”.

Moreover, in the foregoing explanation, the processor8executes programs and performs processing while using storage resources (for instance, memory9) and/or communication interface devices (for instance, communication port). The processing to be performed by the processor8may also be interpreted as being performed by executing one or more programs. Moreover, the processing to be performed by the storage apparatus2may also be interpreted as being executed by the processor8of the storage apparatus2. While the processor8is typically a microprocessor such as a CPU (Central Processing Unit), it may also include a hardware circuit which executes a part of the processing (for instance, encryption/decryption, compression/decompression).

REFERENCE SIGNS LIST