Patent ID: 12204776

DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment will be described with reference to the drawings. The embodiments described below do not limit the invention according to the claims, and all elements and combinations thereof described in the embodiments are not necessarily essential to the solution of the invention.

In the following description, information may be described by an expression of “AAA table”, and the information may be expressed by any data structure. That is, in order to indicate that the information does not depend on the data structure, the “AAA table” may be referred to as “AAA information”.

In the following description, processing may be described with a program as a subject of an operation. The program is executed by a processor (for example, a CPU) to execute predetermined processing while appropriately using a storage resource (for example, a memory) and/or a communication interface device (for example, a network interface card (NIC)). Therefore, the processor may serve as a subject of the processing. The processing described with the program as the subject of the operation may be processing executed by a processor or a computer (system) including the processor.

Two or more programs may be implemented as one program, or one program may be implemented as two or more programs in the following description.

In the following description, an “instance” refers to a virtual computer constituted by software using resources on one or more physical computers, and may be configured on a public cloud or on a private cloud.

In the following description, an “instance type” is determined by a combination of specification values of resources such as a CPU frequency, the number of cores, a memory speed, a memory capacity, and a network interface (I/F) bandwidth, and refers to a type of configuration of the instance. The specification value may be a CPU frequency, the number of cores, a memory speed, a memory capacity, a network I/F bandwidth, or any other value.

FIG.1is an overall configuration diagram of a storage management system according to an embodiment.

A storage management system1includes a storage system1000, a management device200, one or more host computers300, and an instance management node600. The storage system1000includes one or more storage nodes100. In the present embodiment, the instance management node600and the storage system1000are provided in a cloud10, and are constituted by one or more physical computers of the cloud10.

The storage node100, the management device200, the host computer300, and the instance management node600are connected via a network400. The network400may be, for example, a local area network (LAN) or a wide area network (WAN).

The host computer300includes, for example, a computer or a file server serving as a core of a business system, and requests the storage system1000to read/write. The host computer300may be a physical computer or a virtual computer. In the host computer300, for example, when a plurality of storage nodes100constitute a cluster, a multipath is set between the host computer300and the storage nodes constituting the cluster. For example, when Linux (registered trademark) is used, multipath-tools can be used for setting the multipath, and when the host computer300is Windows (registered trademark) server, MPIO service can be used.

The management device200is, for example, a computer including hardware resources such as a CPU, a memory, and a network I/F, and software resources such as a management program. The management device200may be a physical computer or a virtual computer. The management device200acquires information from the storage system1000by the management program, and displays the information via a user interface (graphic user interface (GUI), command line interface (CLI)). The management device200has a function of transmitting an instruction, which is received from a system administrator, to the storage system1000via the user interface. The management device200may be an on-premises device or an on-cloud device.

The instance management node600is a node that manages instances in the cloud10.

The storage node100manages user data used by the host computer300.

Next, the storage node100will be described in detail.FIG.2is a configuration diagram of the storage node according to the embodiment.

The storage node100includes one or more instances110and one or more storage devices120.

The instance110is a virtual computer constituted by software using resources on a physical computer of the cloud10. The instance110may be a virtual machine.

The instance110includes a CPU111, a memory112, and a network I/F113. A resource amount of the CPU111, the memory112, or the network I/F113in the instance110is a resource amount corresponding to a predetermined instance type. The CPU111is a virtual CPU to which a physical CPU of the physical computer of the cloud10is virtually allocated. The CPU111executes processing such as access control to the storage device120based on a program and management information stored in the memory112. The memory112is a virtual memory to which a physical memory of the physical computer of the cloud10is virtually allocated. The memory112stores the program executed by the CPU111and the management information referred to or updated by the CPU111. The network I/F113is an I/F for communicating with the storage device120, other storage nodes100, the management device200, the host computer300, and the instance management node600via the network400.

The storage device120is a physical or virtual storage device, and typically may be a nonvolatile storage device. The storage device120may be, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage device120stores user data used by the host computer300.

Next, the management device200will be described in detail.

FIG.3is a configuration diagram of the management device according to the embodiment.

The management device200includes a CPU210, a memory220, and a network I/F230. The CPU210executes processing of controlling all the storage nodes100and the entire storage system1000based on a program and management information stored in the memory220. The memory220stores the program executed by the CPU210and the management information referred to or updated by the CPU210. The network I/F230is an I/F for communicating with the storage node100, the host computer300, and the instance management node600via the network400.

Next, an outline of processing of the storage management system1will be described.

FIG.4is a diagram showing an outline of processing of the storage management system according to the embodiment.

The management device200of the storage management system1executes processing of collecting operation information500from the storage nodes100(information collection processing: S4100) ((1) inFIG.4). Here, the operation information500is information necessary for grasping performance of the storage system1000and a usage status of the storage system, such as I/O information (I/O response time, I/O throughput) from the host computer300to the storage system1000and a hardware operation rate (a CPU operation rate, a memory usage amount, a network usage bandwidth) of each storage node100. The operation information500is periodically recorded in the storage node100, and is stored in the memory112of the instance110of the storage node100or in the storage device120with which the instance110can communicate.

In (1) inFIG.4, the management device200may collect the operation information500from the storage node100by issuing an information collection request to the storage node100, or the management device200may collect the operation information500by the storage node100periodically and spontaneously transmitting the operation information500to the management device200.

The management device200stores the collected operation information500in the memory220. The management device200may store the collected operation information500in a connected nonvolatile storage device.

Next, the management device200monitors and analyzes the operation information500to detect a performance problem in the storage system1000. When the performance problem is detected, the management device200analyzes the operation information500and management information510to execute processing (performance bottleneck analysis processing: step S4200) of detecting a portion that is a performance bottleneck (performance bottleneck portion) in the storage system1000((2) inFIG.4). The management information510is setting information for the storage system1000and the storage node100, and static information necessary for performance analysis. The management information510is also information on a current instance type set for the instance110, logical configuration information on a volume configuration of the storage system1000, whether a compression/deduplication function is applied, and the like, and target I/O performance information for current specifications of the storage system1000. In the performance bottleneck analysis processing in (2) inFIG.4, the management device200detects which resource (CPU, memory, network I/F) in the instance110of the storage node100is insufficient, that is, where the performance bottleneck portion is located.

Next, the management device200executes processing (bottleneck elimination method determination processing: S4300) of determining a method for eliminating the performance bottleneck for the detected performance bottleneck portion ((3) inFIG.4). For example, when the management device200determines that target I/O performance (reference value) of the storage system1000has not been achieved because a specification of the CPU111of the instance110of a certain storage node100is insufficient, the management device200determines an instance type capable of improving the specification of the CPU111as the changed instance type. The management device200determines a trigger (type change trigger) for changing the instance type for eliminating the performance bottleneck. For example, in order to reduce an influence on I/O of the host computer300, the management device200determines a midnight time zone when the number of I/Os is small as the type change trigger.

Next, the management device200transmits a type change instruction to the storage node100whose instance type is to be changed ((4) inFIG.4), and executes processing (countermeasure execution processing: step S4400) for transmitting the type change instruction to the instance management node600((5) inFIG.4). The type change instruction may instruct the storage node100to transition to a state in which the instance type is changeable and then instructs the storage node100to transition to a steady operation state after a change of the instance type. Here, the state in which the instance type is changeable may be a state in which the storage node100is closed for maintenance or a state in which the storage node100is stopped. The instruction for allowing the storage node100to transition to the steady operation state may be a maintenance recovery instruction or a node start-up instruction.

When there are two or more storage nodes100each including an instance whose type is to be changed, the management device200may execute type change processing for each storage node in order to maintain redundancy of the storage system1000. In order to execute the change processing for each storage node, an instance type change instruction for another storage node100is waited until execution of processing of changing an instance type for a certain storage node100is normally completed. Then, when the instance type change processing for all the change target storage nodes100is completed, the countermeasure execution processing is ended.

Next, a configuration of the memory220of the management device200will be described.

FIG.5is a configuration diagram of the memory of the management device according to the embodiment.

The memory220of the management device200stores a program2000and a management table3000.

The program2000includes an operation information collection program2100, a performance bottleneck analysis program2200, a performance bottleneck elimination method determination program2300, and a countermeasure execution program2400.

The management table3000includes an instance management table3100, an operation information management table3200, a host I/O management table3300, a performance bottleneck management table3400, an instance type management table3500, and a countermeasure execution management table3600.

The operation information collection program2100collects the operation information500from the storage node100, and records data into the operation information management table3200and the host I/O management table3300.

The performance bottleneck analysis program2200analyzes the host I/O management table3300to detect occurrence of a performance problem, and analyzes the operation information management table3200to detect a performance bottleneck portion. The performance bottleneck analysis program2200records information on the performance bottleneck portion obtained by the analysis into the performance bottleneck management table3400.

Based on the information of the performance bottleneck management table3400, the performance bottleneck elimination method determination program2300determines which storage node100is to be changed in order to eliminate a performance bottleneck, which instance type the instance110of the storage node100is to be changed to, and which timing the change is to be made in order to minimize an influence on host I/O, and records the determined information into the countermeasure execution management table3600. When determining which instance type the storage node100is to be changed to, the performance bottleneck elimination method determination program2300selects an instance type having sufficient specifications for a resource that is a performance bottleneck in order to eliminate the performance bottleneck with reference to the instance type management table3500.

Based on the information of the countermeasure execution management table3600, the countermeasure execution program2400transmits an instance type change instruction to the storage node100and transmits the instance type change instruction to the instance management node600at a predetermined change trigger.

Next, the instance management table3100will be described.

FIG.6is a configuration diagram of the instance management table according to the embodiment.

The instance management table3100is a table for managing information on the instance110in the storage node100of the storage system1000. The instance management table3100stores an entry for each instance110of the storage system1000. The entry of the instance management table3100includes fields of a node ID3101, a CPU frequency3102, the number of CPU cores3103, a memory capacity3104, and a network bandwidth3105.

The node ID3101stores an identification number (node ID) of the storage node100having the instance110corresponding to the entry. The CPU frequency3102stores a frequency of the CPU111allocated to the instance110corresponding to the entry. The number of CPU cores3103stores the number of cores of the CPU111allocated to the instance110corresponding to the entry. The memory capacity3104stores a capacity of the memory allocated to the instance110corresponding to the entry. The network bandwidth3105stores a bandwidth of the network I/F113allocated to the instance110corresponding to the entry.

Next, the operation information management table3200will be described.

FIG.7is a configuration diagram of the operation information management table according to the embodiment.

The operation information management table3200is a table for managing information on an operation state of the storage node100, and stores an entry for each storage node100. The entry of the operation information management table3200includes fields of a node ID3201, an information collection time point3202, a CPU operation rate3203, a memory usage amount3204, and a network usage bandwidth3205.

The node ID3201stores an identification number (node ID) of the storage node100corresponding to the entry. The information collection time point3202stores a time point (date and time) when information corresponding to the entry is collected from the storage node100and stored. The CPU operation rate3203stores an operation rate indicating a magnitude of a load of the CPU111in the storage node100corresponding to the entry. The memory usage amount3204stores a usage amount (memory usage amount) of the memory112in the storage node100corresponding to the entry. The network usage bandwidth3205stores a network bandwidth used by the network I/F113of the storage node100corresponding to the entry (network usage bandwidth).

Next, the host I/O management table3300will be described.

FIG.8is a configuration diagram of the host I/O management table according to the embodiment.

The host I/O management table3300is a table for managing I/O information by the host computer300with respect to the storage system1000, and stores an entry for each time point when the I/O information is acquired. The entry of the host I/O management table3300includes fields of an information collection time point3301, an IOPS3302, and a transfer rate3303. The information collection time point3301stores a time point when information corresponding to the entry is acquired and stored (information acquisition time point). The IOPS3302stores the number of I/Os per time unit (IOPS: input/output per second) with respect to the storage system1000. The transfer rate3303stores an amount of data transferred per time unit (transfer rate) to the storage system1000.

Next, the performance bottleneck management table3400will be described.

FIG.9is a configuration diagram of the performance bottleneck management table according to the embodiment.

The performance bottleneck management table3400is a table for managing information on performance bottlenecks, and stores an entry for each detected performance bottleneck. The entry of the performance bottleneck management table3400includes fields of a bottleneck ID3401, an information recording time point3402, a node ID3403, a performance bottleneck portion3404, a required resource enhancement ratio3405, and bottleneck elimination method determination completion3406.

The bottleneck ID3401stores an identification number (bottleneck ID) for uniquely identifying a detected performance bottleneck. The information recording time point3402stores a time point when information on the detected performance bottleneck is registered. The node ID3403stores a node ID of the storage node100having a portion (performance bottleneck portion) that is a performance bottleneck in the performance bottleneck corresponding to the entry. The performance bottleneck portion3404stores information indicating a performance bottleneck portion of the storage node100in the performance bottleneck corresponding to the entry. The performance bottleneck portion3404stores, for example, a CPU, a memory, and a network I/F, which are components (resources) of the instance110of the storage node100. The required resource enhancement ratio3405stores a resource ratio required to eliminate the performance bottleneck corresponding to the entry. For example, when the number of CPU cores of the current instance110is 16, the performance bottleneck is the CPU, and 32 or more CPU cores are required to eliminate the performance bottleneck, 2 times is stored in the required resource enhancement ratio3405. The bottleneck elimination method determination completion3406stores information indicating whether a method for eliminating the performance bottleneck (performance bottleneck elimination method) corresponding to the entry is determined. For example, when the performance bottleneck elimination method is determined, True is stored in the bottleneck elimination method determination completion3406.

Next, the instance type management table3500will be described.

FIG.10is a configuration diagram of the instance type management table according to the embodiment.

The instance type management table3500is a table for managing types (instance types) that can be used for instances, and stores an entry for each instance type. The entry of the instance type management table3500includes fields of an instance type ID3501, a CPU frequency3502, the number of CPU cores3503, a memory capacity3504, and a network bandwidth3505.

The instance type ID3501stores an identification number (instance type ID) for uniquely identifying an instance type corresponding to the entry. The CPU frequency3502stores a frequency of a CPU allocated by the instance type corresponding to the entry. The number of CPU cores3503stores the number of cores of the CPU allocated by the instance type corresponding to the entry. The memory capacity3504stores a capacity of a memory (memory capacity) allocated by the instance type corresponding to the entry. The network bandwidth3505stores a bandwidth of a network I/F (network bandwidth) allocated by the instance type corresponding to the entry. The instance type of the instance type management table3500may include a plurality of instance types in which only a value of one resource (for example, the number of CPU cores) is different.

Next, the countermeasure execution management table3600will be described.

FIG.11is a configuration diagram of the countermeasure execution management table according to the embodiment.

The countermeasure execution management table3600is a table for managing information on countermeasures for performance bottlenecks, and stores an entry for each countermeasure. The entry of the countermeasure execution management table3600includes fields of a countermeasure execution ID3601, a node ID3602, a changed instance type ID3603, a change trigger3604, and countermeasure execution completion3605.

The countermeasure execution ID3601stores an identification number uniquely assigned to a countermeasure corresponding to the entry. The node ID3602stores a node ID of the target storage node100for which the countermeasure corresponding to the entry is executed. The changed instance type ID3603stores an identification number of an instance type (instance type ID) of an instance changed by the countermeasure corresponding to the entry. The change trigger3604stores a trigger for changing the instance type of the instance (change trigger) by executing the countermeasure corresponding to the entry. The change trigger may be a specific date and time, may be a specific day of the week, or may be a specific time zone. The countermeasure execution completion3605stores whether the execution of the countermeasure corresponding to the entry is completed. In the present embodiment, True is stored in the countermeasure execution completion3605when the execution of the countermeasure is completed, and False is stored in the countermeasure execution completion3605when the execution of the countermeasure is not completed.

Next, the information collection processing (step S4100) executed by the management device200will be described.

FIG.12is a flowchart of the information collection processing according to the embodiment.

The information collection processing is executed, for example, periodically by the management device200. An execution cycle of the information collection processing may be, for example, a cycle that is determined based on characteristics of the storage system1000and is necessary for detection of a performance problem of the storage system1000and analysis of a bottleneck portion.

The operation information collection program2100of the management device200(strictly speaking, the CPU210that executes the operation information collection program2100) transmits, to the storage system1000, an information acquisition request for acquiring operation information on the storage node100and I/O information from the host computer300(step S4101). Here, the information acquisition request may be transmitted to each storage node100of the storage system1000, and the operation information and I/O information may be transmitted from each storage node100in response thereto. Alternatively, the information acquisition request may be transmitted to the representative storage node100of the storage system1000, the operation information and the I/O information on each storage node100may be acquired by the representative storage node100, and the summarized information may be transmitted.

Next, the operation information collection program2100receives the operation information and I/O information transmitted from the storage system1000(step S4102). The operation information and I/O information may be received by communication of one time or may be received by communication of a plurality of times.

Next, the operation information collection program2100stores the received operation information and I/O information into the management table3000(step S4103). Specifically, the operation information collection program2100stores the operation information (for example, a CPU operation rate, a memory usage amount, and a network usage bandwidth) into the operation information management table3200, and stores the I/O information into the host I/O management table3300.

Next, the performance bottleneck analysis processing (step S4200) executed by the management device200will be described.

FIG.13is a flowchart of the performance bottleneck analysis processing according to the embodiment.

The performance bottleneck analysis processing is, for example, periodically executed by the management device200. An execution cycle of the performance bottleneck analysis processing may be, for example, a cycle that is determined based on the characteristics of the storage system1000and is necessary for the detection of the performance problem of the storage system1000and the analysis of the bottleneck portion.

The performance bottleneck analysis program2200of the management device200(strictly speaking, the CPU210that executes the performance bottleneck analysis program2200) acquires the operation information on each storage node100of the storage system1000from the operation information management table3200(step S4201). Here, a plurality of pieces of acquired operation information may be aggregated for each storage node100, such as averaging values in a certain period of time.

Next, the performance bottleneck analysis program2200determines whether an operation rate (usage rate: operation performance) of a resource of the instance110of the storage node100exceeds an upper limit of a recommended operation rate (reference value) based on the acquired operation information (step S4202). The processing of steps S4202to S4208may be executed by setting each of the operation rate of each resource and the usage rate of the resource included in the operation information as a determination target. The upper limit of the recommended operation rate of each resource may be managed by a table, or may be managed by another data structure. The upper limits of these recommended operation rates may be stored in advance when the storage system1000is started up, and may be referred to in the performance bottleneck analysis processing.

As a result, when it is determined that the operation rate (usage rate) of the resource does not exceed the upper limit of the recommended operation rate (usage rate) (step S4202: No), it means that there is no portion that is a performance bottleneck in the storage system1000, and thus the performance bottleneck analysis program2200ends the performance bottleneck analysis processing.

On the other hand, when it is determined that the operation rate (usage rate) of the resource exceeds the upper limit of the recommended predetermined operation rate (usage rate) (step S4202: Yes), it means that a portion (resource) to be determined in the storage system1000is a performance bottleneck, and thus the performance bottleneck analysis program2200determines whether a performance problem is constant, that is, whether it is constant that the operation rate (usage rate) of the resource exceeds the upper limit of the recommended predetermined operation rate (usage rate) (step S4203). Here, whether the performance problem is constant can be determined with reference to the past operation information in the operation information management table3200. Whether the performance problem is constant may be determined based on, for example, information indicating how long the performance problem has continued, and a threshold of a period for determining that the performance problem is constant may be determined based on the characteristics of the storage system1000.

As a result, when it is determined that the performance problem is not constant (step S4203: No), it is considered that the performance problem is a temporary performance problem and does not need to be handled, and thus the performance bottleneck analysis program2200ends the performance bottleneck analysis processing.

On the other hand, when it is determined that the performance problem is constant (step S4203: Yes), the performance bottleneck analysis program2200determines a resource of the storage node100that constantly exceeds the upper limit of the recommended operation rate as a performance bottleneck portion (step S4204). Here, the resource of the storage node100determined as the performance bottleneck portion includes, for example, the CPU111, the memory112, or the network I/F113of the instance110.

Next, the performance bottleneck analysis program2200determines a resource enhancement ratio (enhanced resource amount) required to eliminate a performance bottleneck in the determined performance bottleneck portion (step S4205). For example, the resource enhancement ratio required to eliminate the performance bottleneck may be determined based on the operation information management table3200and the recommended operation rate. For example, when a current CPU operation rate is 70% and a recommended CPU operation rate is 50%, if a resource, namely the CPU is multiplied by 1.4 times, the CPU operation rate becomes 50% and the performance bottleneck can be eliminated. Therefore, an enhanced resource amount may be 1.4 times, or may exceed 1.4 times and be the lowest possible magnification (for example, 2 times) in consideration of a possible magnification as an instance type. For example, when the number of CPU cores of the current instance110is 16, the performance bottleneck is the CPU, and 32 or more CPU cores are required to eliminate the performance bottleneck, the resource enhancement ratio required to eliminate the bottleneck is 2 times.

Next, the performance bottleneck analysis program2200performs resource enhancement on the performance bottleneck portion at the resource enhancement ratio determined in step S4205, and when the performance bottleneck is eliminated, determines whether another resource is a performance bottleneck (step S4206). Here, another resource may be another resource in the instance110the same as the instance110in which the performance bottleneck portion determined in step S4204is present, or may be a resource of the storage node100different from the storage node100in which the performance bottleneck portion is present. When detecting another performance bottleneck portion, it may be detected based on information such as a volume configuration and function setting of the storage system1000, and an amount of communication between the storage nodes100.

For example, when the performance bottleneck in the CPU of a certain storage node100is eliminated, an amount of processing executed by the CPU of the storage node100increases. In this case, since the processing executed by the CPU also includes communication with other storage nodes, a usage amount of a network bandwidth increases. Since a cluster is constituted by a plurality of storage nodes and a cooperative operation is performed by the plurality of storage nodes, operation rates of the CPUs of other storage nodes also increase. In consideration of such a situation, for example, when the performance bottleneck in the CPU of the certain storage node is eliminated, if a network bandwidth of the storage node is equal to or greater than a predetermined threshold, it may be determined that the network bandwidth is a performance bottleneck, and if a CPU operation rate of another storage node is equal to or greater than a predetermined threshold, it may be determined that the CPU of another storage node is a performance bottleneck.

As a result, when another resource is a performance bottleneck (step S4206: Yes), the performance bottleneck analysis program2200determines a resource enhancement ratio required to eliminate the performance bottleneck for another resource that is the performance bottleneck (step S4207), and advances the processing to step S4206. A method for determining the resource enhancement ratio required to eliminate the performance bottleneck may be the same method as in step S4205.

On the other hand, when another resource is not a performance bottleneck (step S4206: No), the performance bottleneck analysis program2200stores information on a performance bottleneck portion into the performance bottleneck management table3400(step S4208), and ends the processing. Specifically, the performance bottleneck analysis program2200stores, into the performance bottleneck management table3400, information on the performance bottleneck portion such as the performance bottleneck portions detected in steps S4204and S4206and the required resource enhancement ratios determined in steps S4205and S4207.

According to the performance bottleneck analysis processing, it is possible to detect the performance bottleneck portion in the instance110in the storage system1000, and it is possible to appropriately detect the resource required to eliminate the performance bottleneck in the performance bottleneck portion.

Next, the performance bottleneck elimination method determination processing (step S4300) executed by the management device200will be described.

FIG.14is a flowchart of the performance bottleneck elimination method determination processing according to the embodiment.

The performance bottleneck elimination method determination processing is executed, for example, periodically or immediately after the performance bottleneck analysis processing (step S4200) is ended.

The performance bottleneck elimination method determination program2300of the management device200(strictly speaking, the CPU210that executes the performance bottleneck elimination method determination program2300) determines whether the performance bottleneck management table3400includes an entry of a performance bottleneck for which a bottleneck elimination method is not determined, that is, for which the bottleneck elimination method determination completion3406is False (step S4301).

As a result, when no corresponding entry is present (step S4301: No), the performance bottleneck elimination method determination program2300ends the performance bottleneck elimination method determination processing, whereas when a corresponding entry is present (step S4301: Yes), the performance bottleneck elimination method determination program2300acquires the corresponding entry from the performance bottleneck management table3400(step S4302).

Next, the performance bottleneck elimination method determination program2300determines an instance type including a resource sufficient to eliminate a performance bottleneck of the acquired entry based on information of the acquired entry, information of the instance management table3100, and information of the instance type management table3500(step S4303). In the present embodiment, the performance bottleneck elimination method determination program2300determines an instance type in which a resource that is a performance bottleneck is set to a resource amount at which the performance bottleneck is to be eliminated, and other resources are not changed from before the change or increased by a smallest resource amount. Accordingly, it is possible to appropriately prevent the resource not related to the performance bottleneck from being allocated to the instance uselessly.

Next, the performance bottleneck elimination method determination program2300determines a trigger (change trigger) for changing the instance type to the determined instance type (step S4304). In order to minimize an influence on the host I/O, the change trigger may be a time zone or a day of the week in which the number of host I/Os is small, or may be a timing of a maintenance operation planned in advance. When the time zone or the day of the week in which the number of the host I/Os is small is selected as the change trigger, for example, the performance bottleneck elimination method determination program2300analyzes information of the host I/O management table3300and finds periodicity of the host I/O, thereby selecting the time zone or the day of the week in which the number of host I/Os is small as the trigger.

The performance bottleneck elimination method determination program2300stores, into the countermeasure execution management table3600, an entry including a node ID of a node in which a performance bottleneck was present, the determined instance type changed for eliminating the performance bottleneck, and the determined change trigger (step S4305).

Next, the performance bottleneck elimination method determination program2300records True, indicating that the performance bottleneck elimination method is determined, into the bottleneck elimination method determination completion3406of the corresponding entry in the performance bottleneck management table3400(step S4306), and ends the processing.

According to the performance bottleneck elimination method processing described above, it is possible to determine the instance type including the resource required to eliminate the performance bottleneck, and to determine an appropriate change trigger.

Next, the countermeasure execution processing (step S4400) executed by the management device200will be described.

FIG.15is a flowchart of the countermeasure execution processing according to the embodiment.

The countermeasure execution processing is executed, for example, periodically. The countermeasure execution program2400(strictly speaking, the CPU210that executes the countermeasure execution program2400) acquires an entry of a countermeasure that is not executed, that is, an entry in which the countermeasure execution completion3605is False, from the countermeasure execution management table3600(step S4401).

The countermeasure execution program2400determines whether a change trigger of the acquired entry (target entry) is satisfied (step S4402). As a result, when the change trigger of the target entry is not satisfied (step S4402: No), the countermeasure execution program2400advances the processing to step S4408.

On the other hand, when the change trigger of the target entry is satisfied (step S4402: Yes), the countermeasure execution program2400transmits a closing instruction for maintenance (maintenance closing instruction) of the change target storage node100corresponding to the target entry, that is, the storage node100having the node ID, namely the node ID3602in the target entry, to the predetermined storage node100(for example, the storage node100serving as a representative of the cluster) (step S4403). As a result, the storage node100that has received the maintenance closing instruction executes maintenance closing processing (seeFIG.17). After the maintenance closing instruction is issued, the countermeasure execution program2400waits until the maintenance closing processing of the storage node100is completed.

Next, after the maintenance closing processing of the storage node100is completed, the countermeasure execution program2400transmits, to the instance management node600, an instruction (change instruction) to change an instance of a change target storage node into an instance type corresponding to the changed instance type ID3603in the acquired entry (step S4404). Accordingly, the instance management node600can change the instance of the change target storage node into the changed instance type, and can eliminate the performance bottleneck. After the change instruction is issued, the countermeasure execution program2400waits until the instance type of the instance110of the storage node100is changed.

Next, after the instance type of the instance110of the storage node100is changed, the countermeasure execution program2400transmits an instruction (maintenance recovery instruction) to cause the change target storage node to recover from maintenance, to the predetermined storage node100(for example, the storage node100serving as a representative of the cluster) (step S4405). As a result, the storage node100that has received the maintenance recovery instruction executes maintenance recovery processing (seeFIG.18). After the maintenance recovery instruction is issued, the countermeasure execution program2400waits until the maintenance recovery processing of the storage node100is completed.

Next, after the maintenance recovery processing of the storage node100is completed, the countermeasure execution program2400updates information of the entry corresponding to the change target storage node in the instance management table3100into specification information on a resource having the changed instance type (step S4406). Here, the specification information on the changed instance type can be acquired from an entry acquired from the instance type management table3500using the changed instance type ID3603of the target entry as a key.

Next, the countermeasure execution program2400records True into the countermeasure execution completion3605of the target entry in the countermeasure execution management table3600(step S4407), and advances the processing to step S4408.

In step S4408, the countermeasure execution program2400determines whether the countermeasure execution management table3600includes an entry in which the countermeasure execution completion3605is False and that is not a target of change trigger determination (step S4402).

As a result, when the countermeasure execution completion3605is False and the entry that is not the target of the change trigger determination (step S4402) is present (step S4408: Yes), the countermeasure execution program2400advances the processing to step S4401, and executes the subsequent processing on the entry that is present.

On the other hand, when the countermeasure execution completion3605is False and no entry that is not the target of the change trigger determination (step S4402) is present (step S4408: No), the countermeasure execution program2400ends the countermeasure execution processing.

Next, a configuration of the storage node of the storage system1000will be described.

FIG.16is a configuration example of the storage nodes of the storage system according to the embodiment.

The storage system1000includes one or more control unit clusters160. The control unit cluster160includes a storage control unit150A in an active state (also referred to as an active storage control unit) and a storage control unit150S in a standby state (also referred to as a standby storage control unit). The storage control unit150A provides a volume capable of I/O by the host computer300. When an I/O request designating a volume to be provided is received from the host computer300, the storage control unit150A executes I/O processing on the designated volume. When the storage control unit150A constituting the control unit cluster160is stopped, the storage control unit150S is brought into an active state by performing failover, and executes I/O processing on the volume in place of the storage control unit150A. The storage control unit150A and the storage control unit150S of the same control unit cluster160are configured in different storage nodes100. In the storage system1000, the representative storage node100of the clusters160includes a cluster controller170. Each storage node100includes a node controller180. The cluster controller170grasps a state of all the clusters160and controls a configuration of each storage node100. The node controller180notifies the cluster controller170of information on the storage node100, and controls the configuration of the storage node100according to an instruction from the cluster controller170. In the present embodiment, the storage node100implements the storage control unit150A, the storage control unit150S of another control unit cluster160, the cluster controller170, and the node controller180using the instance110.

Next, the maintenance closing processing (step S4500) will be described.

FIG.17is a flowchart of the maintenance closing processing according to the embodiment.

The maintenance closing processing is executed when the cluster controller170of the storage node100receives a maintenance closing instruction from the management device200.

The cluster controller170of the storage node100checks in advance whether the target storage node100can be closed for maintenance (step S4501). For example, the cluster controller170checks whether a failure, in which the processing cannot be handed over, occurs in a storage node to which the processing is to be handed over, that is, a storage node including the storage control unit150S corresponding to the storage control unit150A to be closed (also referred to as a hand-over destination node). If the failure has occurred, the maintenance closing processing is ended.

Next, the cluster controller170instructs the node controller180of the storage node100that is a maintenance closing target (also referred to as a maintenance closing node in description of this processing) to close a port for communicating with the host computer300, and upon receiving the instruction, the node controller180closes the port according to the instruction (step S4502). When the port is closed, in the host computer300, a transmission destination of the I/O request is switched to the hand-over destination node according to setting of the multipath.

Next, the cluster controller170instructs the node controller180of the hand-over destination node to temporarily stop reception of the host I/O, and the node controller180temporarily stops the reception of the host I/O in response to the instruction (step S4503).

Next, the cluster controller170instructs the node controller180of the maintenance closing node100to stop the active storage control unit150A, and the node controller180stops the storage control unit150A in response to the instruction (step S4504).

Next, the cluster controller170instructs the node controller180of the hand-over destination node100to switch (operate) the standby storage control unit150S to an active state, and the node controller180switches the storage control unit150S to the active state in response to the instruction (step S4505).

Next, the cluster controller170instructs the node controller180of the hand-over destination node to cancel stop of the host I/O, and the node controller180cancels the stop of the host I/O in response to the instruction (step S4506). Accordingly, the storage control unit150S of the hand-over destination node can process the host I/O.

Next, the cluster controller170instructs the node controller180of the maintenance closing node100to stop the maintenance closing node100, and when receiving the instruction, the node controller180stops the maintenance closing node100according to the instruction (step S4507), and ends the maintenance closing processing.

Next, the maintenance recovery processing (step S4600) will be described.

FIG.18is a flowchart of the maintenance recovery processing according to the embodiment.

The maintenance recovery processing is executed when the cluster controller170of the storage node100receives a maintenance recovery instruction from the management device200.

The cluster controller170of the storage node100turns on power of the storage node100(maintenance recovery node) that is a maintenance recovery target, and starts up the node controller180(step S4601). At this time, the cluster controller170monitors a start-up status of the maintenance recovery node100.

Next, the cluster controller170instructs the node controller180of the maintenance recovery node100to start up the storage control units150A and150S as standby, and the node controller180starts up the storage control units150A and150S as standby in response to the instruction (step S4602).

Next, the cluster controller170instructs the node controller180of the hand-over destination node to temporarily stop reception of the host I/O, and the node controller180temporarily stops the reception of the host I/O in response to the instruction (step S4603).

Next, the cluster controller170instructs the node controller180of the maintenance recovery node100to switch (operate) the standby storage control unit150A to an active state, and the node controller180switches the storage control unit150A to the active state in response to the instruction (step S4604).

Next, the cluster controller170instructs the node controller180of the hand-over destination node to cancel stop of the host I/O, and the node controller180cancels the stop of the host I/O in response to the instruction (step S4605).

Next, the cluster controller170instructs the node controller180of the hand-over destination node100to switch (operate) the active storage control unit150S to a standby state, and the node controller180switches the storage control unit150S to the standby state in response to the instruction (step S4506).

Next, the cluster controller170instructs the node controller180of the maintenance recovery node100to restore the port for communicating with the host computer300, and upon receiving the instruction, the node controller180restores the port according to the instruction (step S4607). When the port is restored, in the host computer300, the transmission destination of the I/O request is switched to the maintenance recovery node according to the setting of the multipath.

Next, the cluster controller170instructs the node controllers180of all the storage nodes100in a group for creating redundancy with the maintenance recovery node100to restore the redundancy, and upon receiving the instruction, each node controller180executes processing of restoring the redundancy according to the instruction (step S4608), and ends the maintenance recovery processing. Accordingly, the storage control unit150A is constituted by the instance of the changed instance type, and the performance bottleneck does not occur.

The invention is not limited to the embodiment described above, and may be appropriately modified and implemented without departing from the gist of the invention.

For example, in the embodiment described above, the performance bottleneck is analyzed, and the configuration of the instance is changed so as to increase the resource amount of the resource that is the performance bottleneck. However, for example, it may be configured such that designation of an instance whose resource amount is to be increased is received from the user, and it is determined that a resource configuration needs to be changed according to the designation, and a configuration of the instance is changed so as to increase the resource amount.

In the embodiment described above, a part or all of the processing executed by the CPU may be executed by a hardware circuit. The programs in the embodiment described above may be acquired from a program source and installed. The program source may be a program distribution server or a storage medium (for example, a portable storage medium).