Patent Publication Number: US-9430292-B2

Title: Information processing system, method of controlling information processing system and storage medium

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2014-090271, filed on Apr. 24, 2014, the entire contents of which are incorporated herein by reference. 
     FIELD 
     The embodiments discussed herein are related to an information processing system, a method of controlling the information processing system, and a storage medium. 
     BACKGROUND 
     When a virtual computation machine defined on a computation machine is moved to another computation machine, information used in a process executed by the virtual computation machine is sent to an external storage device and then is moved to the other computation machine. 
     In a system including a plurality of server devices, each executing a process based on a request, session information included in a server device having a load larger than that of another server device is moved to another server device so as to decentralize the load. 
     When a process of a virtual computation machine operated on a computation machine is executed by a virtual computation machine on another computation machine, configuration information on a storage interface and the like of the original virtual computation machine is set in the virtual computation machine defined on the other computation machine, and then the virtual computation machine defined on the other computation machine is operated. For example, the original virtual computation machine is stopped and a new virtual computation machine is suppressed from starting an operation during a time period when configuration information is exchanged between the original virtual computation machine and the new virtual computation machine. 
     Examples of the related art include Japanese Laid-open Patent Publication No. 2009-145931, Japanese Laid-open Patent Publication No. 2007-219964, and Japanese Laid-open Patent Publication No. 2010-33404. 
     SUMMARY 
     According to an aspect of the invention, an information processing system includes: a plurality of processors configured to cause a virtual machine on each processor to be operated or terminated; a plurality of storage devices; and a control device configured to: collect processing loads of the respective processors, cause a first virtual machine which operates on a first processor in which a frequency of a processing load exceeding a first threshold exceeds a first proportion, to be terminated, cause a second processor which is stopped among the plurality of processors, to be operated, and cause a first storage device which is coupled with the terminated first virtual machine among the plurality of storage devices, to be coupled with the operated second processor. 
     The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  illustrates an embodiment of an information processing system, a control method of the information processing system, and a control program of the information processing system; 
         FIG. 2  illustrates an example of a state of the information processing apparatus different from a state illustrated in  FIG. 1 ; 
         FIG. 3  illustrates an example in which a control device collects load information; 
         FIG. 4  illustrates an example of frequency of load information in each physical server, collected by the control device; 
         FIG. 5  illustrates an example of an operation of the information processing system; 
         FIG. 6  illustrates an example of change of a load in an operation of the information processing system; 
         FIG. 7  illustrates another example of change of a load in an operation of the information processing system; 
         FIG. 8  illustrates an example of an operating flow in the control device; 
         FIG. 9  illustrates an example of an ST 0  control routine; 
         FIG. 10  illustrates an example of an ST 1  control routine; 
         FIG. 11  illustrates an example of an ST 2  control routine; 
         FIG. 12  illustrates an example of an ST 3  control routine; 
         FIG. 13  illustrates an example of an ST 4  control routine; 
         FIG. 14  illustrates an example of an ST 5  control routine; 
         FIG. 15  illustrates an example of a program of executing the ST 0  control routine; 
         FIG. 16  illustrates an example of a program of executing the ST 1  control routine; 
         FIG. 17  illustrates an example of a program of executing the ST 2  control routine; 
         FIG. 18  illustrates an example of a program of executing the ST 3  control routine; 
         FIG. 19  illustrates an example of a program of executing the ST 4  control routine; 
         FIG. 20  illustrates an example of a program of executing the ST 5  control routine; 
         FIG. 21  illustrates an example in which a control device changes the state of an information processing apparatus in another embodiment of the information processing system, the control method of the information processing system, and the control program of the information processing system; and 
         FIG. 22  illustrates an example of a relationship between each process, a load, and frequency in another embodiment. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     For example, when a server device executing information processing is switched to another server device or when a server device is added, a process of switching or adding causes a load of the server device to increase. For this reason, it is preferable that switching and adding of a server device are executed before a load of a server device exceeds a predetermined threshold, in order to suppress degradation of a service provided by the server device. It is preferable that a type of a switched server device or an added server device is selected in accordance with the extent that a load of the server device changes. 
     The object of an information processing system, a control method of the information processing system, and a recording medium in which a control program of the information processing system is recorded, according to this disclosure is to change processing performance of the information processing system depending on a change of a load without an influence on a service provided by the information processing system. 
     Hereinafter, embodiments will be described using the drawings. 
       FIG. 1  illustrates an embodiment of an information processing system, a control method of the information processing system, and a control program of the information processing system. An information processing system SYS 1  according to this embodiment includes an information processing apparatus IPE 1  and a control device CNTL configured to control the information processing apparatus IPE 1 . The information processing apparatus IPE 1  includes a plurality of servers P (P 0 , P 1 , P 2 , P 3 , P 4 , and P 5 ), a switch device SW, and a disk pool DPL. For example, each server P and the control device CNTL are coupled with each other through a network switch NSW such as Ethernet (registered trademark). For example, each server P and the control device CNTL are coupled with a user terminal TM which uses the information processing system SYS 1  as a network storage, through the network switch NSW. For example, the user terminal TM is coupled with the information processing system SYS 1  through a network NW such as the Internet or an intranet. For example, the information processing system SYS 1  functions as a storage server or a data center for holding information such as data, transmitted from the user terminal TM. For example, information transmitted from the user terminal TM may be stored in duplicate in each of a plurality of storage devices D (for example, three storage devices D). 
     Each server P is operated based on control of the control device CNTL and is stopped based on control of the control device CNTL. The servers P 0 , P 1 , and P 2  indicated by solid lines represent the operating servers and the servers P 3 , P 4 , and P 5  indicated by dot lines represent the stopped servers. The currently stopped servers P 3 , P 4 , and P 5  are to be operated based on an instruction from the control device CNTL. For example, the currently stopped servers P 3 , P 4 , and P 5  are in a state where power is cut off. The number of servers P is not limited to six. The server P is an example of an information processing section. 
     Each server P includes a processor such as a central processing unit (CPU) and a storage device such as a memory MEM for storing an operating system (OS) executed by the processor and a program which is for implementing a function of the server. Each server P may include a storage device such as a hard disk drive (HDD), for storing the program. The OS or the program executed by each server P may be stored in the storage device D disposed in the disk pool DPL and may be transmitted to the memory MEM of the server P based on an operating instruction received from the control device CNTL. 
     Each server P includes a monitoring section LDM for monitoring its own load. The monitoring section LDM is, for example, a load monitoring agent. The monitoring section LDM monitors a load of the CPU at a predetermined interval and notifies the control device CNTL of the monitored load as load information indicating a load of each server P. For example, a “load of a CPU” refers to CPU usage (for example, an average value) monitored by the monitoring section LDM for a predetermined period (for example, one second). The monitoring section LDM may monitor input/output (I/O) usage (for example, an access rate of the storage device D) instead of the CPU usage. The monitoring section LDM may be implemented through a program which is executed by the CPU and may be implemented by hardware. Additionally, the monitoring section LDM may be implemented by combining a program and hardware. 
     A plurality of virtual machines V (V 0  and V 1 ) for implementing a function of the server using software may be mounted in each server P.  FIG. 1  illustrates an example in which the virtual machines V 0  and V 1  are operated on the server P 0 . The number of virtual machines V allowed to be operated on each server P is not limited to two. In the following description, the server P physically mounted in the information processing system SYS 1  is referred to as a physical server P and the virtual machine V installed on the physical server P is referred to as a virtual server V. 
     The disk pool DPL includes a plurality of storage devices D (D 0  to D 7 ). For example, the storage device D refers to an HDD or a solid state drive (SSD), but may refer to other storage devices capable of holding information in a non-volatile manner. For example, the storage devices D 0  to D 4  indicated by solid lines represent a state where power is supplied and information such as data is written and read. The storage devices D 5  to D 7  indicated by dot lines represent a state where the power is cut off and information is not written and read. For example, information such as data is deleted in the storage devices D 5  to D 7  indicated by dot lines. The power may be supplied to the storage devices D 5  to D 7  in which information is not written and read. For example, supply and cutting of the power in the storage devices D 0  to D 7  is controlled by the control device CNTL. The number of the storage devices D 0  to D 7  is not limited to eight. The storage device D is an example of a storage section. 
     The switch device SW couples the storage device D with the physical server P or the virtual server V based on control of the control device CNTL. A dot line seen inside the switch device SW indicates a state in which the storage device D is coupled with the physical server P or the virtual server V. For example, the switch device SW may couple an arbitrary storage device D with the physical server P or the virtual server V based on control from the control device CNTL. In  FIG. 1 , for easily understandable description, one storage device D is coupled with one physical server P or one virtual server V. However, a plurality of storage devices D may be coupled with one physical server P or one virtual server V. The switch device SW is an example of a coupling section. A disk area network or a storage area network (SAN) for coupling the storage device D with the physical server P or the virtual server V is constructed in the information processing apparatus IPE 1  by using the switch device SW. 
     The switch device SW has performance sufficient for coupling the storage device D with the physical server P or the virtual server V without a band (data transmission speed) of the storage device D being reduced. The storage device D is operated as a direct attached storage (DAS). That is, performance of the storage device D which is coupled with the physical server P or the virtual server V through the switch device SW is equal to performance of a storage device such as an HDD, mounted in the physical server P. Since the virtual server V is implemented through a program which is executed by the physical server P, access efficiency of the storage device D by the virtual server V is lower than access efficiency of the storage device D by the physical server P. 
     The control device CNTL includes a processor such as a CPU and a storage device such as a memory MEM, for storing an OS executed by the processor and a program which is for implementing a function of the control device CNTL. The program which is stored in the memory MEM and executed by the CPU of the control device CNTL refers to a control program for controlling an operation of the information processing system SYS 1 . The control method of the information processing system SYS 1  is implemented through the program which is stored in the memory MEM and executed by the CPU of the control device CNTL. 
     The control device CNTL includes a holding section HLD for holding the load information of each server P transferred from the monitoring section LDM of the server P. For example, the holding section HLD may be implemented by using registers mounted in the CPU of the control device CNTL and may be held in the memory MEM of the control device CNTL. The control device CNTL may include a storage device such as an HDD, for storing a program. 
     In the example illustrated in  FIG. 1 , the physical servers P 0 , P 1 , P 2  are operated. The storage device D 0  is coupled with the physical server P 0 , the storage device D 3  is coupled with the physical server P 1 , and the storage device D 4  is coupled with the physical server P 2 . The virtual servers V 0  and V 1  are operated on the physical server P 0 , the storage device D 1  is coupled with the virtual server V 0 , and the storage device D 2  is coupled with the virtual server V 1 . In  FIG. 1 , the physical servers P 0 , P 1 , and P 2  and the virtual servers V 0  and V 1  operate five servers and the information processing system SYS 1  is operated as a storage server or a data center, for example. 
     The monitoring section LDM of each of the physical servers P 0 , P 1 , and P 2  in operation monitors a load of the CPU periodically and transmits the load information indicating the monitored load to the control device CNTL. The control device CNTL holds the received load information in the holding section HLD. The control device CNTL controls to switch the virtual server V to the physical server P or to switch the physical server P to the virtual server V, based on the load information of each of the physical servers P 0 , P 1 , and P 2  in operation, which is held in the holding section HLD. The control device CNTL controls to add a virtual server V, to add a physical server P, or to delete a virtual server V. The control method of the information processing system SYS 1  executed by the control device CNTL will be described using  FIGS. 3 to 20 . 
     For example, the control device CNTL controls to equally write information such as file data and image data, which is transmitted from the user terminal TM in the storage devices D respectively coupled with servers which include the physical server P and the virtual server V. For example, the storage devices D respectively coupled with servers which include the physical server P and the virtual server V are disposed in a hash space evenly. In other words, the control device CNTL assigns information to the servers such as data such that loads of the physical server P in operation and the virtual server V in operation are equal to each other. A parent server which is any one of the physical servers P in operation may assign information such as data (that is, designation of a server which transmits information such as data). Information including data and the like is simply referred to as data below. 
     Access performance of the virtual server V accessing the storage device D is lower than access performance of the physical server P accessing the storage device D. For this reason, a load of the virtual server V may be smaller than a load of the physical server P by the number of storage devices D coupled with the virtual server V being less than the number of the storage device D coupled with the physical server P. A load of the virtual server V may be smaller than a load of the physical server P by a hash space allocated to the storage device D coupled with the virtual server V being smaller than a hash space allocated to the storage device D coupled with the physical server P. 
       FIG. 2  illustrates an example of a state of the information processing apparatus IPE 1  different from a state of the information processing apparatus IPE 1  illustrated in  FIG. 1 . For example, in  FIG. 2 , the virtual server V 0  ( FIG. 1 ) which has operated on the physical server P 0  is removed, the stopped physical server P 3  starts to be operated, and the storage device D 1  which has been coupled with the virtual server V 0  is coupled with the physical server P 3 . That is,  FIG. 2  illustrates an example in which the virtual server V 0  ( FIG. 1 ) is switched to the physical server P 3  based on a change of a load in any one of the physical servers P 0 , P 1 , and P 2 . A rectangle indicated by a dot line in the physical server P 0  means that a program for implementing the virtual server V 0  illustrated in  FIG. 1  is terminated and an operation of the virtual server V 0  on the physical server P 0  is terminated. 
     Switching the virtual server V 0  to the physical server P 3  is performed by exchanging configuration information (for example, a host name which is a unique name of the server and an internet protocol (IP) address) of the physical server P 3  and configuration information of the virtual server V 0 , for example. At this time, the control device CNTL controls the switch device SW to separate the storage device D 1  from the virtual server V 0  and to couple the separated storage device D 1  with the physical server P 3 . 
     For example, the control device CNTL changes a state of the information processing apparatus IPE 1  to a state illustrated in  FIG. 2  from a state illustrated in  FIG. 1  when a frequency of a load in any one of the physical servers P 0  to P 2  to the maximum load applied to each physical server P being more than 30% and equal to or less than 80% exceeds a predetermined proportion. For example, the frequency refers to a proportion of the number of pieces of load information which indicates a load of more than 30% and equal to or less than 80% to the number of pieces of load information which is held in the holding section HLD and is transmitted to each of the physical servers P 0 , P 1 , and P 2 . 
       FIG. 3  illustrates an example in which the control device CNTL illustrated in  FIG. 1  collects load information. For example, each bar of a bar graph illustrated on an upper side of  FIG. 3  indicates load information of which the control device CNTL is notified by one monitoring section LDM of the physical server P in operation. In the example illustrated in  FIG. 3 , the control device CNTL holds 100 pieces of load information received by the physical server P in the holding section HLD. The load information indicates a load (any value from 0% to 100%) of the CPU mounted in the physical server P performing notification of the load information. In  FIG. 3 , for easily understandable description, the load information is indicated by the bar graph. However, the server P notifies the control device CNTL of the load information as a load value in practice. For example, the monitoring section LDM notifies the control device CNTL of the load information once a second. For this reason, a time count TC which is indicated on a horizontal axis of  FIG. 3  and refers to the number of notifications of the load information also indicates the number of seconds. 
     The control device CNTL compares load values indicated by the load information held in the holding section HLD with threshold values VT 1 , VT 2 , and VT 3  each time load information is received 100 times from each operating physical server P. For example, the threshold value VT 1  refers to 80%, the threshold value VT 2  refers to 30%, and the threshold value VT 3  refers to 2%. The threshold values VT 1 , VT 2 , and VT 3  may refer to other values as long as the threshold values VT 1 , VT 2 , and VT 3  satisfy a relationship of VT 1 &gt;VT 2 &gt;VT 3  and the number of pieces of load information to be compared is not limited to 100. For example, in an example of a processing flow illustrated in  FIGS. 9 to 14 , the control device CNTL compares each load value indicated by each piece of load information held in the holding section HLD with the threshold values VT 1 , VT 2 , and VT 3  each time the load information is received 600 times (that is, for each 600 seconds). 
     The control device CNTL increases a variable LUTL by “one” when a load indicated by the load information is less than the threshold value VT 3 . The control device CNTL increases a variable MUTL by “one” when a load indicated by the load information is more than the threshold value VT 2  and equal to or less than the threshold value VT 1 . The control device CNTL increases a variable HUTL by “one” when a load indicated by the load information exceeds the threshold value VT 1 . The variables LUTL, MUTL, and HUTL are assigned to each physical server P. 
     The values of variables LUTL, MUTL, and HUTL are initialized to “0” before the load information is collected. For example, when the control device CNTL collects the load information in a unit of 100 seconds (TC=100), the variables LUTL, MUTL, and HUTL are initialized for each 100 seconds. In the following description, a region of exceeding the threshold value VT 1  (80%) is also referred to as a high load region, a region of more than the threshold value VT 2  (30%) and equal to or less than the threshold value VT 1  (80%) is also referred to as a middle load region, and a region of less than the threshold value VT 3  (2%) is also referred to as a low load region. 
     In  FIG. 3 , the variable LUTL has a value of “5”, the variable MUTL has a value of “82”, and the variable HUTL has a value of “9” in a state where the control device CNTL collects the load information 100 times. The control device CNTL divides the values of the variables LUTL, MUTL, and HUTL by a value (=100) of the time count TC to obtain the frequency of the load (ratio of time) at each of the high load region, the middle load region, and the low load region for a predetermined period (for example, 100 seconds). For example, in the example illustrated in  FIG. 3 , the frequency of the load at the high load region is “0.09”, the frequency of the load at the middle load region is “0.82”, and the frequency of the load at the low load region is “0.05”. The frequency of the load refers to a proportion of loads distributed at a load region having a predetermined range. A relationship of the frequency of the load indicated by the variables LUTL, MUTL, and HUTL which are assigned to each of the plurality of physical servers P which are operating will be described with reference to  FIG. 4 . 
     As illustrated in  FIGS. 9 to 14 , the control device CNTL determines whether or not a result of collecting the load information satisfies the following five conditions of Condition A to Condition E.
         (Condition A) the frequency of the load at the middle load region exceeds “0.8”.   (Condition B) the frequency of the load at the middle load region exceeds “0.7”   (Condition C) the frequency of the load at the high load region exceeds “0.9”.   (Condition D) the frequency of the load at the low load region exceeds “0.8”.   (Condition E) the frequency of the load at the low load region exceeds “0.9”.       

     The control device CNTL controls the physical server P, the virtual server V, and the switch device SW based on a state of the information processing apparatus IPE 1  including information such as the number of the operating physical servers P and the number of the operating virtual servers V.  FIG. 5  illustrates an example in which the control device CNTL controls the physical server P, the virtual server V, and the switch device SW. As illustrated in  FIG. 5 , the condition used for determination is different in accordance with states ST 0  to ST 5  of the information processing apparatus IPE 1 . 
       FIG. 4  illustrates an example of the frequency of the load information of each of the physical servers P 0 , P 1 , and P 2 , collected by the control device CNTL illustrated in  FIG. 1 .  FIG. 4  illustrates an example of the frequency of the load information when the control device CNTL detects that (Condition A) illustrated in  FIG. 3  is satisfied and changes a state of the information processing apparatus IPE 1  to the state illustrated in  FIG. 2  from the state illustrated in  FIG. 1 . 
     For example, when the storage devices D are disposed evenly using a hash space, it is highly likely that changes of the load in the respective physical servers P in operation have a tendency to be similar to each other. For this reason, the frequency of the load of one operating physical server P at the low load region, the middle load region, and the high load region has a similar tendency as the frequency of the load of another operating physical server P at the low load region, the middle load region, and the high load region. For example, when the frequency of the load of one operating physical server P at the middle load region is high, the frequency of the load of another operating physical server P at the middle load region also is high. 
     The load of the physical server P including the operating virtual server V is higher than the load of the physical server P not including the virtual server V. For example, the frequency of the load at each of the high load region, the middle load region, and the low load region in the physical server P 0  causing the two virtual servers V 0  and V 1  to be operated in the state illustrated in  FIG. 1  is higher than the frequency of the load at each of the high load region, the middle load region, and the low load region in the others physical servers P 1  and P 2 . For this reason, for example, it is much likely that each of (Condition A), (Condition B), and (Condition C) illustrated in  FIG. 3  is satisfied based on the frequency of the load of the physical server P causing the virtual server V to be operated. 
       FIG. 5  illustrates an example of an operation of the information processing system SYS 1  illustrated in  FIG. 1 . That is,  FIG. 5  illustrates an example of the control method of the information processing system SYS 1  performed by the control device CNTL. 
     In the example illustrated in  FIG. 5 , the state of the information processing apparatus IPE 1  is changed between the states ST 0 , ST 1 , ST 2 , ST 3 , ST 4 , and ST 5 . The state ST 0  refers to a state illustrated in  FIG. 1  and the state ST 1  refers to a state illustrated in  FIG. 2 . When the load of the physical server P in operation does not satisfy a condition for changing the state in each of the states ST 0  to ST 5 , the state is held (marks of (a), (b), (c), (d), (e), and (f) in  FIG. 5 ). The operation of the information processing system SYS 1  is not limited to the example illustrated in  FIG. 5 . That is, the number of the states ST 0  to ST 5  is not limited to six and specifications of transitions between the states are not limited to the example illustrated in  FIG. 5 . 
     For example, when the frequency of the load included at the middle load region exceeds “0.8” in the state ST 0  and in any one of the physical servers P 0  to P 2  (Condition A), the state of the information processing apparatus IPE 1  is changed to the state ST 1  (the mark (g) in  FIG. 5 ). In this case, the virtual server V 0  which has operated on the physical server P 0  is terminated, the physical server P 3  which has been stopped starts to be operated, and the storage device D 1  which has been coupled with the virtual server V 0  is coupled with the physical server P 3 . That is, the virtual server V 0  is replaced with the physical server P 3 . The number of servers operated in the information processing apparatus IPE 1  and in the state ST 1  is five and is the same as that in the state ST 0 . However, performance of the information processing apparatus IPE 1  is improved by replacing the virtual server V 0  with the physical server P 3 . That the performance of the information processing apparatus IPE 1  is improved without changing the number of servers is also referred to as scale-out for performance or is also simply referred to as scale-out in the following description. 
     The virtual server V 1  may be replaced with the physical server P 3  for changing the state ST 0  to the state ST 1 . When the virtual server V is operated on the physical server P 1  or P 2 , the virtual server V operated on the physical server P 1  or P 2  may be replaced with the physical server P 3 . 
     When the load of the physical server P tends to increase, the control device CNTL performs scale-out of replacing the virtual server V 0  to the physical server P 3  and thus it is possible to increase a processing capacity of the information processing apparatus IPE 1 . Accordingly, it is possible to stably operate the information processing apparatus IPE 1  even when the load of the information processing apparatus IPE 1  further increases after that. Since the storage device D 1  which has been coupled with the virtual server V 0  is coupled with the physical server P 3 , transition of data between the storage devices D does not occur and thus it is possible to hold allocation of the storage device D 1  to the hash space. 
     The configuration information such as an IP address is changed between the virtual server V 0  and the physical server P 3  with switching from the virtual server V 0  to the physical server P 3 . For this reason, an increment of the load of the information processing apparatus IPE 1  by switching from the virtual server V 0  to the physical server P 3  is smaller than an increment of the load when data is transitioned. The scale-out performed by switching the servers is performed in a state where the load of the physical server P tends to increase. Thus, an increase of the load by switching the servers does not have an influence on a service provided by the information processing system SYS 1 . Accordingly, an increase of the load of the information processing apparatus IPE 1  due to switching the servers is minimized and thus it is possible to improve processing performance of the information processing system SYS 1  without an influence on a service provided by the information processing system SYS 1 . 
     When the frequency of the load included at the middle load region exceeds “0.8” in the state ST 1  and in any one of the physical servers P 0  to P 3  (Condition A), the state of the information processing apparatus IPE 1  is changed to the state ST 2  (the mark (h) in  FIG. 5 ). In this case, the virtual server V 1  which has operated on the physical server P 0  is terminated, the physical server P 4  which has been stopped starts to be operated, and the storage device D 2  which has been coupled with the virtual server V 1  is coupled with the physical server P 4 . Switching the virtual server V 1  to the physical server P 4  is performed by exchanging configuration information (for example, a host name and an IP address) of the physical server P 4  and configuration information of the virtual server V 1  and switching the coupling of the storage device D 1  from the virtual server V 0  to the physical server P 0 , for example. 
     Improvement of performance of the information processing apparatus IPE 1  caused by performing scale-out from the state ST 1  to the state ST 2  is similar to improvement of performance of the information processing apparatus IPE 1  caused by changing the state ST 0  to the state ST 1 . It is possible to obtain the same effect as through scale-out from the state ST 0  to the state ST 1  for performance by performing scale-out from the state ST 1  to the state ST 2  for performance. The virtual server V replaced with the physical server P 4  is not limited to the virtual server V 1 , similarly to the change of the state ST 0  to the state ST 1 . 
     When the frequency of the load included at the middle load region exceeds “0.7” in the state ST 2  and in any one of the physical servers P 0  to P 4  (Condition B), the state of the information processing apparatus IPE 1  is changed to the state ST 3  (the mark (i) in  FIG. 5 ). In this case, a new virtual server V 2  is operated on the physical server P 0  and a new storage device D 5  is coupled with the virtual server V 2 . That is, scale-out of adding the virtual server V 2  to the information processing apparatus IPE 1  is performed. Since the new storage device D 5  is added, reallocation of the hash space is performed for the storage devices D 0  to D 5  respectively coupled with the physical servers P 0  to P 4  in operation and the virtual server V 2  in operation, for example. The reallocation of the hash space results in transition of data between the storage devices D 0  to D 5 . The virtual server V operated on the physical servers P 1  to P 4  may be added to the information processing apparatus IPE 1  instead of the virtual server V 2 , for changing the state ST 2  to the state ST 3 . 
     The control device CNTL adds the new virtual server V 2  in the load being low when a tendency of increasing in the load of the physical server P is lower than a tendency of increasing in the load when the state ST 0  is changed to the state ST 1 . Accordingly, it is possible to include the new virtual server V 2  when the load of the information processing apparatus IPE 1  is further increased. Since data is transitioned between the storage devices D in the load being low, it is possible to have a reduced influence on the information processing apparatus IPE 1  due to transition of data, compared to when the load is high and thus data is transitioned. 
     When the frequency of the load of any one of the physical servers P 0  to P 4  included at the high load region exceeds “0.9” in the state ST 2  (Condition C), the state of the information processing apparatus IPE 1  is changed to the state ST 4  (the mark (j) in  FIG. 5 ). In this case, a new physical server P 5  is operated and the new storage device D 5  is coupled with the physical server P 5 . That is, scale-out of adding the physical server P 5  to the information processing apparatus IPE 1  is performed. Since the new storage device D 5  is added, reallocation of the hash space is performed for the storage devices D 0  to D 5  respectively coupled with the physical servers P 0  to P 5  in operation, for example. The reallocation of the hash space results in transition of data between the storage devices D 0  to D 5 . Improvement of performance of the information processing apparatus IPE 1  caused by changing the state ST 2  to the state ST 4  is higher than improvement of performance of the information processing apparatus IPE 1  caused by changing the state ST 2  to the state ST 3 . The other stopped physical server P may be added to the information processing apparatus IPE 1  for changing the state ST 2  to the state ST 4 . 
     The control device CNTL adds a new physical server P having a processing capacity higher than that of the virtual server V when the load of the physical server P tends to increase rapidly. With this, it is possible to transition data between the storage devices D while an influence of transition of data on the information processing apparatus IPE 1  is minimized. The frequency of the load may satisfy Condition A or Condition B ahead of satisfying Condition C, in a normal operation of the information processing system SYS 1  in many cases. For this reason, for example, the frequency of changing the state ST 2  to the state ST 4  is lower than frequency of changing the state ST 2  to the state ST 3 . A state of the information processing apparatus IPE 1  in the state ST 5  is the same as a state of the information processing apparatus IPE 1  in the state ST 4 . 
     When the frequency of the load of any one of the physical servers P 0  to P 4  included at the middle load region exceeds “0.8” in the state ST 3  (Condition A), the state of the information processing apparatus IPE 1  is changed to the state ST 5  (the mark (k) in  FIG. 5 ). In this case, the virtual server V 2  which has operated on the physical server P 0  is removed, the stopped physical server P 5  starts to be operated, and the storage device D 5  which has been coupled with the virtual server V 2  is coupled with the physical server P 5 , similarly to the change of the state ST 0  to the state ST 1 . That is, scale-out for performance is performed. 
     Switching the virtual server V 2  to the physical server P 5  is performed by exchanging configuration information (for example, a host name and an IP address) of the physical server P 5  and configuration information of the virtual server V 2  and switching the coupling of the storage device D 5  from the virtual server V 2  to the physical server P 5 , for example. 
     When the frequency of the load of any one of the physical servers P 0  to P 5  included at the low load region exceeds “0.8” in the state ST 5  (Condition D), the state of the information processing apparatus IPE 1  is changed to the state ST 3  (the mark (l) in  FIG. 5 ). In this case, the physical server P 5  is stopped, the virtual server V 2  is operated on the physical server P 0 , and the storage device D 5  which has been coupled with the physical server P 5  is coupled with the virtual server V 2 . That is, the physical server P 5  is replaced with the virtual server V 2  and a state of the information processing apparatus IPE 1  is changed in reverse to being changed from the state ST 3  to the state ST 5 . The number of servers operated in the information processing apparatus IPE 1  in the state ST 4  is six and is the same as that in state ST 5 . However, performance of the information processing apparatus IPE 1  is degraded by replacing the physical server P 5  with the virtual server V 2 . That the performance of the information processing apparatus IPE 1  is degraded without changing the number of servers is also referred to as scale-in for performance or is also simply referred to as scale-in in the following description. The physical server P 5  may be replaced with the virtual server V on the physical servers P 1  to P 4  in operation for changing the state ST 5  to the state ST 3 . 
     Switching the physical server P 5  to the virtual server V 2  is performed by exchanging configuration information (for example, a host name and an IP address) of the physical server P 5  and configuration information of the virtual server V 2  and switching the coupling of the storage device D 5  from the physical server P 5  to the virtual server V 2 , for example. 
     Since the storage device D 5  which has been coupled with the physical server P 5  is coupled with the virtual server V 2 , transition of data between the storage devices D does not occur and thus it is possible to hold allocation of the storage device D 1  to the hash space. For this reason, transition of data held in the storage device D due to reallocation to the hash space does not occur in the entirety of the information processing apparatus IPE 1 . The load of all of the operating physical servers P 0  to P 5  is reduced in a state where the load of any one of the physical servers P 0  to P 5  is included at the low load region. For this reason, an increase in load due to replacing the physical server P 5  with the virtual server V 2  does not influence a service provided by the information processing system SYS 1 . Accordingly, an increase of the load of the information processing apparatus IPE 1  by switching the servers is minimized and thus it is possible to degrade processing performance of the information processing system SYS 1  without an influence on a service provided by the information processing system SYS 1 . 
     The control device CNTL may stop one of the physical servers P by performing scale-in for performance in which the physical server P is replaced with the virtual server V when the load of the physical server P is small. With this, it is possible to decrease a processing capacity of the information processing apparatus IPE 1  and to largely reduce power consumed by the information processing apparatus IPE 1 , compared to when one of the physical servers P continues to be operated. 
     When the frequency of the load of any one of the physical servers P 0  to P 5  included at the low load region exceeds “0.8” in the state ST 4  (Condition D), the state of the information processing apparatus IPE 1  is changed to the state ST 3  (the mark (m) in  FIG. 5 ). An operation in this case is performed similarly to the change of the state ST 5  to the state ST 3 . For example, configuration information (for example, a host name and an IP address) of the physical server P 5  and configuration information of the virtual server V 2  are exchanged and the coupling of the storage device D 5  is switched from the physical server P 5  to the virtual server V 2 . It is possible to obtain the same effect as the effect obtained by changing the state ST 5  to the state ST 3 , through scale-in for performance in which the state ST 4  is changed to the state ST 3 . 
     When the frequency of the load of any one of the physical servers P 0  to P 4  included at the low load region exceeds “0.8” in the state ST 3  (Condition E), the state of the information processing apparatus IPE 1  is changed to the state ST 2  (the mark (n) in  FIG. 5 ). In this case, data which is held in the storage device D 5  coupled with the virtual server V 2  is transitioned to other storage devices D 0  to D 4  and the virtual server V 2  on the physical server P 0  is terminated. That is, scale-in is performed. When data transmitted from the user terminal TM is held in duplicate in the plurality of storage devices D as replica data, the virtual server V 2  may be terminated ahead of transitioning data. In this case, the virtual server V 2  is terminated and then data may be transitioned by using the replica data read from the storage device D which is coupled with the physical servers P 0  to P 4  in operation or the other virtual server V in operation. 
     The load (Condition E) of the information processing apparatus IPE 1  when the state ST 3  is changed to the state ST 2  is smaller than the load (Condition D) of the information processing apparatus IPE 1  when the state ST 5  is changed to the state ST 3 . For this reason, when data is transitioned between the storage devices D depending on the change of the states of the information processing apparatus IPE 1 , it is also possible to reduce an influence on a service provided by the information processing system SYS 1  to the minimum. When Condition E is satisfied, one of the physical servers P may be stopped instead of terminating one of the virtual servers V. Accordingly, it is possible to largely reduce the power consumed by the information processing apparatus IPE 1 , compared to when one of the virtual servers V is terminated. 
     When the frequency of the load of any one of the physical servers P 0  to P 4  included at the low load region exceeds “0.8” in the state ST 2  (Condition D), the state of the information processing apparatus IPE 1  is changed to the state ST 1  (the mark (o) in  FIG. 5 ). An operation in this case is performed similarly to the change of the state ST 5  to the state ST 3 . For example, configuration information (for example, a host name and an IP address) of the physical server P 4  and configuration information of the virtual server V 1  are exchanged and the coupling of the storage device D 2  is switched from the physical server P 4  to the virtual server V 1 . That is, scale-in for performance is performed. 
     When the frequency of the load of any one of the physical servers P 0  to P 3  included at the low load region exceeds “0.8” in the state ST 1  (Condition D), the state of the information processing apparatus IPE 1  is changed to the state ST 0  (the mark (p) in  FIG. 5 ). Scale-in for performance is performed as an operation performed in this case, similarly to the change of the state ST 5  to the state ST 3 . 
     In  FIG. 5 , the control device CNTL replaces the virtual server V with the physical server P when the frequency of the load of the physical server P included at the middle load region increases to exceed a predetermined value. That is, when the load of the information processing apparatus IPE 1  is small, the virtual server V is operated. When the load of the information processing apparatus IPE 1  is large, the physical server P is operated. When the load of the information processing apparatus IPE 1  is small, the virtual server V having access efficiency (I/O performance) lower than that of the physical server P is operated and thus it is possible to suppress an influence of the I/O performance of the virtual server V on a service provided by the information processing system SYS 1 . When data transmitted from the user terminal TM is stored in duplicate in the plurality of storage devices D, the physical server P is coupled with one of the plurality of storage devices D storing the data and thus access efficiency of data is determined based on the I/O performance of the physical server P. In this case, it is possible to cover the I/O performance of the virtual server V with the I/O performance of the physical server P. 
       FIG. 5  illustrates an example in which the control device CNTL changes the state of the information processing apparatus IPE 1  in accordance with the frequency of the load of any one of the physical servers P. However, the control device CNTL may change the state of the information processing apparatus IPE 1  in accordance with a frequency of an average value of the load of the operating physical server P. This is because the frequencies of the load of the respective operating physical servers P have a tendency to be similar to each other, as illustrated in  FIG. 4 . 
       FIG. 6  illustrates an example of the change of the load in an operation of the information processing system SYS 1  illustrated in  FIG. 5 . Marks of (a) to (l) and (n) in  FIG. 6  are the same as the marks indicating holding or switching of the state of the information processing apparatus IPE 1  illustrated in  FIG. 5 .  FIG. 6  illustrates the frequency of the load of any one of the physical servers P in operation. Marks of CYC 1  to CYC 15  in  FIG. 6  indicate a cycle of the control device CNTL collecting the load information. In  FIG. 6 , for easily understandable description, the frequency of performing switching between the states ST is higher than that in an actual operation of the information processing system SYS 1 . 
     First, in the state ST 0 , the load of the physical server P gradually becomes higher and the frequency at the middle load region exceeds “0.8” in a cycle CYC 3 . Thus, the control device CNTL causes a state of the information processing apparatus IPE 1  to be switched from the state ST 0  to the state ST 1  ((g) in  FIG. 6 ). Since switching of the state ST 0  to the state ST 1  causes the virtual server V 0  to be replaced with the physical server P 3 , the processing capacity of the information processing apparatus IPE 1  is improved and the frequency at the middle load region decreases in a cycle CYC 4 . 
     Then, in the state ST 1 , the load of the physical server P gradually becomes higher and the frequency at the middle load region exceeds “0.8” in a cycle CYC 5 . Thus, the control device CNTL causes a state of the information processing apparatus IPE 1  to be switched from the state ST 1  to the state ST 2  ((h) in  FIG. 6 ). Since switching of the state ST 1  to the state ST 2  causes the virtual server V 1  to be replaced with the physical server P 4 , the processing capacity of the information processing apparatus IPE 1  is improved and the frequency at the middle load region decreases in a cycle CYC 6 . 
     In the state ST 2 , the load of the physical server P becomes higher and the frequency at the high load region exceeds “0.9” in a cycle CYC 7  ( 1 ). Thus, the control device CNTL causes a state of the information processing apparatus IPE 1  to be switched from the state ST 2  to the state ST 4  ((j) in  FIG. 6 ). Since switching of the state ST 2  to the state ST 4  causes the new physical server P 5  to be operated, the processing capacity of the information processing apparatus IPE 1  is improved and the frequency at the high load region decreases in a cycle CYC 8 . 
     In the state ST 2 , the load of the physical server P gradually becomes higher and the frequency at the middle load region exceeds “0.7” in the cycle CYC 5 . Thus, the control device CNTL causes a state of the information processing apparatus IPE 1  to be switched from the state ST 2  to the state ST 3  ((i) in  FIG. 6 ). Since switching of the state ST 2  to the state ST 3  causes the new virtual server V 2  to be operated, the processing capacity of the information processing apparatus IPE 1  is improved and the frequency at the middle load region decreases in a cycle CYC 9 . 
     In the state ST 3 , the load of the physical server P gradually becomes higher and the frequency at the middle load region exceeds “0.8” in a cycle CYC 10 . Thus, the control device CNTL causes a state of the information processing apparatus IPE 1  to be switched from the state ST 3  to the state ST 5  ((k) in  FIG. 6 ). Since switching of the state ST 3  to the state ST 5  causes the virtual server V 2  to be replaced with the physical server P 5 , the processing capacity of the information processing apparatus IPE 1  is improved and the frequency at the middle load region decreases in a cycle CYC 11 . 
     In the state ST 5 , the load of the physical server P becomes lower and the frequency at the low load region exceeds “0.8” in a cycle CYC 12 . Thus, the control device CNTL causes a state of the information processing apparatus IPE 1  to be switched from the state ST 5  to the state ST 3  ((l) in  FIG. 6 ). Since switching of the state ST 5  to the state ST 3  causes the physical server P 5  to be replaced with the virtual server V 2 , the processing capacity of the information processing apparatus IPE 1  decreases and the frequency at the middle load region increases in a cycle CYC 13 . 
     In the state ST 3 , the load of the physical server P becomes further lower and the frequency at the low load region exceeds “0.9” in a cycle CYC 14 . Thus, the control device CNTL causes a state of the information processing apparatus IPE 1  to be switched from the state ST 3  to the state ST 2  ((n) in  FIG. 6 ). Since switching of the state ST 3  to the state ST 2  causes the virtual server V 2  to be terminated, the processing capacity of the information processing apparatus IPE 1  decreases and the frequency at the middle load region increases in a cycle CYC 15 . 
       FIG. 7  illustrates another example of a change of the load in an operation of the information processing system SYS 1  illustrated in  FIG. 5 . Detailed description of operations the same as or similar to those in  FIG. 6  will be omitted. Marks of (c) to (e) and (m) to (o) in  FIG. 7  are the same as the marks indicating holding or switching of the state of the information processing apparatus IPE 1  illustrated in  FIG. 5 . For example,  FIG. 7  illustrates operations subsequent to the cycle CYC 8  in  FIG. 6 . The change of the state ST 4  to the state ST 3  and the change of the state ST 2  to the state ST 1  are similar to the change of the state ST 5  to the state ST 3 . 
       FIG. 8  illustrates an example of an operating flow in the control device CNTL illustrated in  FIG. 1 . For example, the operating flow illustrated in  FIG. 8  is implemented by the control device CNTL executing the control program of the information processing system SYS 1 . The control device CNTL is started when the information processing system SYS 1  starts a function as a storage server or a data center. 
     First, in Step S 10 , the control device CNTL initializes the variables HUTL, MUTL, and LUTL to be “0”. In Step S 12 , the control device CNTL determines whether or not the information processing apparatus IPE 1  is in the state ST 0 . When the information processing apparatus IPE 1  is in the state ST 0 , the process proceeds to an ST 0  control routine of Step S 100 . When the information processing apparatus IPE 1  is not in the state ST 0 , the process proceeds to Step S 14 . In the ST 0  control routine, it is determined whether a state of the information processing apparatus IPE 1  is transitioned from the state ST 0  to the state ST 1  or is held to be in the state ST 0 .  FIG. 9  illustrates an example of the ST 0  control routine and  FIG. 15  illustrates an example of a program of the ST 0  control routine. 
     In Step S 14 , the control device CNTL determines whether or not the information processing apparatus IPE 1  is in the state ST 1 . When the information processing apparatus IPE 1  is in the state ST 1 , the process proceeds to an ST 1  control routine of Step S 200 . When the information processing apparatus IPE 1  is not in the state ST 1 , the process proceeds to Step S 16 . In the ST 1  control routine, it is determined whether a state of the information processing apparatus IPE 1  is transitioned from the state ST 1  to the state ST 2 , is transitioned from the state ST 1  to the state ST 0 , or is held to be in the state ST 1 .  FIG. 10  illustrates an example of a flow of the ST 1  control routine and  FIG. 16  illustrates an example of a program of the ST 1  control routine. 
     In Step S 16 , the control device CNTL determines whether or not the information processing apparatus IPE 1  is in the state ST 2 . When the information processing apparatus IPE 1  is in the state ST 2 , the process proceeds to an ST 2  control routine of Step S 300 . When the information processing apparatus IPE 1  is not in the state ST 2 , the process proceeds to Step S 18 . In the ST 2  control routine, it is determined whether a state of the information processing apparatus IPE 1  is transitioned from the state ST 2  to the state ST 4 , is transitioned from the state ST 2  to the state ST 3 , is transitioned from the state ST 2  to the state ST 1 , or is held to be in the state ST 2 .  FIG. 11  illustrates an example of a flow of the ST 2  control routine and  FIG. 17  illustrates an example of a program of the ST 2  control routine. 
     In Step S 18 , the control device CNTL determines whether or not the information processing apparatus IPE 1  is in the state ST 3 . When the information processing apparatus IPE 1  is in the state ST 3 , the process proceeds to an ST 3  control routine of Step S 400 . When the information processing apparatus IPE 1  is not in the state ST 3 , the process proceeds to Step S 20 . In the ST 3  control routine, it is determined whether a state of the information processing apparatus IPE 1  is transitioned from the state ST 3  to the state ST 5 , is transitioned from the state ST 3  to the state ST 2 , or is held to be in the state ST 3 .  FIG. 12  illustrates an example of a flow of the ST 3  control routine and  FIG. 18  illustrates an example of a program of the ST 3  control routine. 
     In Step S 20 , the control device CNTL determines whether or not the information processing apparatus IPE 1  is in the state ST 4 . When the information processing apparatus IPE 1  is in the state ST 4 , the process proceeds to an ST 4  control routine of Step S 500 . When the information processing apparatus IPE 1  is not in the state ST 4 , the process proceeds to Step S 22 . In the ST 4  control routine, it is determined whether a state of the information processing apparatus IPE 1  is transitioned from the state ST 4  to the state ST 3  or is held to be in the state ST 4 .  FIG. 13  illustrates an example of a flow of the ST 4  control routine and  FIG. 19  illustrates an example of a program of the ST 4  control routine. 
     In Step S 22 , the control device CNTL determines whether or not the information processing apparatus IPE 1  is in the state ST 5 . When the information processing apparatus IPE 1  is in the state ST 5 , the process proceeds to an ST 5  control routine of Step S 600 . When the information processing apparatus IPE 1  is not in the state ST 5 , the process returns to Step S 10 . In the ST 5  control routine, it is determined whether a state of the information processing apparatus IPE 1  is transitioned from the state ST 5  to the state ST 3  or is held to be in the state ST 5 .  FIG. 14  illustrates an example of a flow of the ST 5  control routine and  FIG. 20  illustrates an example of a program of the ST 5  control routine. The control device CNTL executes Steps S 100 , S 200 , S 300 , S 400 , S 500 , and S 600  and then causes the process to return to Step S 10 . 
       FIG. 9  illustrates an example of the ST 0  control routine illustrated in  FIG. 8 . First, in Step S 102 , the control device CNTL resets the time count TC to “0”. In Step S 104 , the control device CNTL determines whether or not the load of any one of the physical servers P exceeds 80%. When the load exceeds 80%, the process proceeds to Step S 106 . When the load is equal to or less than 80%, the process proceeds to Step S 108 . In Step S 106 , the control device CNTL increases the variable HUTL by “one” and causes the process to proceed to Step S 116 . 
     In Step S 108 , the control device CNTL determines whether or not the load of any one of the physical servers P exceeds 30%. When the load exceeds 30%, that is, when the load exceeds 30% and is equal to or less than 80%, the process proceeds to Step S 110 . When the load is equal to or less than 30%, the process proceeds to Step S 112 . In Step S 110 , the control device CNTL increases the variable MUTL by “one” and causes the process to proceed to Step S 116 . 
     In Step S 112 , the control device CNTL determines whether or not the load of any one of the physical servers P is less than 2%. When the load is less than 2%, the process proceeds to Step S 114 . When the load is equal to or more than 2%, that is, when the load is 2% or more and 30% or less, the process proceeds to Step S 116 . In Step S 114 , the control device CNTL increases the variable LUTL by “one” and causes the process to proceed to Step S 116 . 
     In Step S 116 , the control device CNTL waits for, for example, one second and then causes the process to proceed to Step S 118 . In Step S 118 , the control device CNTL increases the time count TC by “one”. In Step S 120 , the control device CNTL determines whether or not the time count TC exceeds “600”. When the time count TC exceeds “600”, that is, when an aggregate time of the variables HUTL, MUTL, and LUTL exceeds 600 seconds, the process proceeds to Step S 122 . When the time count TC is equal to or less than “600”, the process returns to Step S 104 . 
     In Step S 122 , the control device CNTL determines whether or not the frequency of the load at the middle load region, which is a value obtained by dividing the value of the variable MUTL by the time count TC exceeds “0.8”. When the frequency of the load at the middle load region exceeds “0.8”, the process proceeds to Step S 124 . When the frequency of the load at the middle load region is equal to or less than “0.8”, the process proceeds to Step S 126 . 
     In Step S 124 , the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 0  to the state ST 1 . In Step S 126 , the control device CNTL causes the information processing apparatus IPE 1  to be held in the state ST 0 . After the processes of Steps S 124  and S 126 , the ST 0  control routine is ended. The ST 0  control routine is executed for each physical server P in operation. When any one of the physical servers P satisfies a determination condition in Step S 122  (Yes in S 122 ), Step S 124  is processed. 
       FIG. 10  illustrates an example of the ST 1  control routine illustrated in  FIG. 8 . Detailed description of processes the same as or similar to those in  FIG. 9  will be omitted. Processes from Step S 102  to Step S 120  are the same as those in  FIG. 9 . 
     In Step S 222 , the control device CNTL performs the same determination as that in Step S 122  illustrated in  FIG. 9 . When the frequency of the load at the middle load region exceeds “0.8”, the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 1  to the state ST 2  in Step S 224 . When the frequency of the load at the middle load region is equal to or less than “0.8”, the process proceeds to Step S 226 . 
     In Step S 226 , the control device CNTL determines whether or not the frequency of the load at the low load region, which is a value obtained by dividing the value of the variable LUTL by the time count TC exceeds “0.8”. When the frequency of the load at the low load region exceeds “0.8”, the process proceeds to Step S 228 . When the frequency of the load at the low load region is equal to or less than “0.8”, the process proceeds to Step S 230 . In Step S 228 , the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 1  to the state ST 0 . In Step S 230 , the control device CNTL causes the information processing apparatus IPE 1  to be held in the state ST 1 . After processes of Steps S 224 , S 228 , and S 230 , the ST 1  control routine is ended. 
     The ST 1  control routine is executed for each physical server P in operation. When any one of the physical servers P satisfies a determination condition in Step S 222  (Yes in S 222 ), Step S 224  is processed. When any one of the physical servers P satisfies a determination condition in Step S 226  (Yes in S 226 ), Step S 228  is processed. The summation of frequency MUTL/TC and frequency LUTL/TC does not exceed 1.0. Thus, the determination conditions of Steps S 222  and S 226  are not simultaneously satisfied. As illustrated in  FIG. 4 , the load of the plurality of physical servers P which are operating is changed with a similar tendency. For this reason, it does not occur that one of the plurality of physical servers P satisfies the determination condition of Step S 222  and another of the plurality of physical servers P satisfies the determination condition of Step S 226 . 
       FIG. 11  illustrates an example of the ST 2  control routine illustrated in  FIG. 8 . Detailed description of processes the same as or similar to those in  FIGS. 9 and 10  will be omitted. The processes from Step S 102  to Step S 120  are the same as those in  FIG. 9 . 
     In Step S 322 , the control device CNTL determines whether or not the frequency of the load at the high load region, which is a value obtained by dividing the value of the variable HUTL by the time count TC exceeds “0.9”. When the frequency of the load at the high load region exceeds “0.9”, the process proceeds to Step S 324 . When the frequency of the load at the high load region is equal to or less than “0.9”, the process proceeds to Step S 326 . In Step S 324 , the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 2  to the state ST 4 . 
     In Step S 326 , the control device CNTL determines whether or not the frequency of the load at the middle load region, which is a value obtained by dividing the value of the variable MUTL by the time count TC exceeds “0.7”. When the frequency of the load at the middle load region exceeds “0.7”, the process proceeds to Step S 328 . When the frequency of the load at the middle load region is equal to or less than “0.7”, the process proceeds to Step S 330 . In Step S 328 , the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 2  to the state ST 3 . 
     In Step S 330 , the control device CNTL performs determination similar to that in Step S 226  illustrated in  FIG. 10 . When the frequency of the load at the low load region exceeds “0.8”, the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 2  to the state ST 1  in Step S 332 . When the frequency of the load at the low load region is equal to or less than “0.8”, the process proceeds to Step S 334 . In Step S 334 , the control device CNTL causes the information processing apparatus IPE 1  to be held in the state ST 2 . After the processes of Steps S 324 , S 328 , S 332 , and S 334 , the ST 2  control routine is ended. 
     The ST 2  control routine is executed for each physical server P in operation. When any one of the physical servers P satisfies a determination condition in Step S 322  (Yes in S 322 ), Step S 324  is processed. When any one of the physical servers P satisfies a determination condition in Step S 326  (Yes in S 326 ), Step S 328  is processed. When any one of the physical servers P satisfies a determination condition in Step S 330  (Yes in S 330 ), Step S 334  is processed. Similarly to  FIG. 10 , the summation of frequency HUTL/TC, frequency MUTL/TC, and frequency LUTL/TC does not exceed 1.0. Thus, the determination conditions of Steps S 322 , S 326 , and S 330  are not simultaneously satisfied. Similarly to  FIG. 10 , it does not occur that the plurality of physical servers P which are operating satisfy the determination conditions (S 322 , S 326 , and S 330 ) different from each other. 
       FIG. 12  illustrates an example of the ST 3  control routine illustrated in  FIG. 8 . Detailed description of processes the same as or similar to those in  FIGS. 9 and 10  will be omitted. The processes from Step S 102  to Step S 120  are the same as those in  FIG. 9 . 
     In Step S 422 , the control device CNTL performs the same determination as that in Step S 122  illustrated in  FIG. 9 . When the frequency of the load at the middle load region exceeds “0.8”, the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 3  to the state ST 5  in Step S 424 . When the frequency of the load at the middle load region is equal to or less than “0.8”, the process proceeds to Step S 426 . 
     In Step S 426 , the control device CNTL performs the same determination as that in Step S 226  illustrated in  FIG. 10 . When the frequency of the load at the low load region exceeds “0.9”, the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 3  to the state ST 2  in Step S 428 . When the frequency of the load at the low load region is equal to or less than “0.9”, the process proceeds to Step S 430 . In Step S 430 , the control device CNTL causes the information processing apparatus IPE 1  to be held in the state ST 3 . After processes of Steps S 424 , S 428 , and S 430 , the ST 3  control routine is ended. 
     The ST 3  control routine is executed for each physical server P in operation. When any one of the physical servers P satisfies a determination condition in Step S 422  (Yes in S 422 ), Step S 424  is processed. When any one of the physical servers P satisfies a determination condition in Step S 426  (Yes in S 426 ), Step S 428  is processed. The summation of frequency MUTL/TC and frequency LUTL/TC does not exceed 1.0. Thus, the determination conditions of Steps S 422  and S 426  are not simultaneously satisfied. Similarly to  FIG. 10 , it does not occur that the plurality of physical servers P which are operating satisfy the determination conditions (S 422  and S 426 ) different from each other. 
       FIG. 13  illustrates an example of the ST 4  control routine illustrated in  FIG. 8 . Detailed description of processes the same as or similar to those in  FIGS. 9 and 10  will be omitted. The processes from Step S 102  to Step S 120  are the same as those in  FIG. 9 . 
     In Step S 522 , the control device CNTL performs the same determination as that in Step S 226  illustrated in  FIG. 10 . When the frequency of the load at the low load region exceeds “0.8”, the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 4  to the state ST 3  in Step S 524 . When the frequency of the load at the low load region is equal to or less than “0.8”, the process proceeds to Step S 526 . In Step S 526 , the control device CNTL causes the information processing apparatus IPE 1  to be held in the state ST 4 . After processes of Steps S 524  and S 526 , the ST 4  control routine is ended. The ST 4  control routine is executed for each physical server P in operation. When any one of the physical servers P satisfies a determination condition in Step S 522  (Yes in S 522 ), Step S 524  is processed. 
       FIG. 14  illustrates an example of the ST 5  control routine illustrated in  FIG. 8 . Detailed description of processes the same as or similar to those in  FIGS. 9 and 10  will be omitted. The processes from Step S 102  to Step S 120  are the same as those in  FIG. 9 . 
     In Step S 622 , the control device CNTL performs the same determination as that in Step S 226  illustrated in  FIG. 10 . When the frequency of the load at the low load region exceeds “0.8”, the control device CNTL causes a state of the information processing apparatus IPE 1  to be transitioned from the state ST 5  to the state ST 3  in Step S 624 . When the frequency of the load at the low load region is equal to or less than “0.8”, the process proceeds to Step S 626 . In Step S 626 , the control device CNTL causes the information processing apparatus IPE 1  to be held in the state ST 5 . After processes of Steps S 624  and S 626 , the ST 5  control routine is ended. The ST 5  control routine is executed for each physical server P in operation. When any one of the physical servers P satisfies a determination condition in Step S 622  (Yes in S 622 ), Step S 624  is processed. 
       FIG. 15  illustrates an example of a program of executing the ST 0  control routine illustrated in  FIG. 9 . For example,  FIG. 15  illustrates a source list of a program executed by the CPU of the control device CNTL and a row number is added to the forefront in each row, for convenient description, in  FIG. 15 . Characters subsequent to a mark of “//” indicate a comment. 
     Variables are defined by using “DEFINDE” from a first row to a ninth row. A variable HTH indicates the threshold value VT 1  (80%) illustrated in  FIG. 3 . A variable MTH indicates the threshold value VT 2  (30%) illustrated in  FIG. 3 . A variable LTH indicates the threshold value VT 3  (2%) illustrated in  FIG. 3 . A variable TIME indicates the time count TC illustrated in  FIG. 3  and a collecting period (for example, 600 seconds) for collecting the load information of the physical server P. 
     A variable RPEXP indicates the frequency of the load (0.9; threshold value) at the high load region, for determining addition of the physical server P. A variable RVEXP indicates the frequency of the load (0.8; threshold value) at the middle load region, for determining replacement of the virtual server V with the physical server P. A variable RVNEW indicates the frequency of the load (0.7; threshold value) at the middle load region, for determining addition of the virtual server V. A variable RSH indicates the frequency of the load (0.8; threshold value) at the low load region, for determining a termination of the virtual server V. A variable RVRM indicates the frequency of the load (0.9; threshold value) at the low load region, for determining replacement of the physical server P with the virtual server V. 
     When the state (State) of the information processing apparatus IPE 1  is ST 0  (=0) in an 11th row, rows from a 12th row to a 23rd row are executed. A variable measureCpuUtil in a 13th row, a 15th row, and a 17th row indicates the load of the physical server P for each time count TC, which is held in the holding section HLD. “do transState” in a 21st row and a 23rd row refers to a command for changing the state of the information processing apparatus IPE 1 . The comments added to the respective rows indicate corresponding of processes executed in the rows from the 12th row to the 23rd row to Steps illustrated in  FIG. 9 . 
       FIG. 16  illustrates an example of a program of executing the ST 1  control routine illustrated in  FIG. 10 . Detailed description regarding statements the same as or similar to those in  FIG. 15  will be omitted. Definitions of variables by “DEFINDE” in rows from a first row to a ninth row are the same as those in  FIG. 15 . When the state (State) of the information processing apparatus IPE 1  is ST 1  (=1) in an 11th row, rows from a 12th row to a 25th row are executed. Comments added to the respective rows indicate corresponding of processes executed in the rows from the 12th row to the 25th row to Steps illustrated in  FIG. 10 . 
       FIG. 17  illustrates an example of a program of executing the ST 2  control routine illustrated in  FIG. 11 . Detailed description regarding statements the same as or similar to those in  FIG. 15  will be omitted. Definitions of variables by “DEFINDE” in rows from a first row to a ninth row are the same as those in  FIG. 15 . When the state (State) of the information processing apparatus IPE 1  is ST 2  (=2) in an 11th row, rows from a 12th row to a 27th row are executed. Comments added to the respective rows indicate corresponding of processes executed in the rows from the 12th row to the 27th row to Steps illustrated in  FIG. 11 . 
       FIG. 18  illustrates an example of a program of executing the ST 3  control routine illustrated in  FIG. 12 . Detailed description regarding statements the same as or similar to those in  FIG. 15  will be omitted. Definitions of variables by “DEFINDE” in rows from a first row to a ninth row are the same as those in  FIG. 15 . When the state (State) of the information processing apparatus IPE 1  is ST 3  (=3) in an 11th row, rows from a 12th row to a 25th row are executed. Comments added to the respective rows indicate corresponding of processes executed in the rows from the 12th row to the 25th row to Steps illustrated in  FIG. 12 . 
       FIG. 19  illustrates an example of a program of executing the ST 4  control routine illustrated in  FIG. 13 . Detailed description regarding statements the same as or similar to those in  FIG. 15  will be omitted. Definitions of variables by “DEFINDE” in rows from a first row to a ninth row are the same as those in  FIG. 15 . When the state (State) of the information processing apparatus IPE 1  is ST 4  (=4) in an 11th row, rows from a 12th row to a 23rd row are executed. Comments added to the respective rows indicate corresponding of processes executed in the rows from the 12th row to the 23rd row to Steps illustrated in  FIG. 13 . 
       FIG. 20  illustrates an example of a program of executing the ST 5  control routine illustrated in  FIG. 14 . Detailed description regarding statements the same as or similar to those in  FIG. 15  will be omitted. Definitions of variables by “DEFINDE” in rows from a first row to a ninth row are the same as those in  FIG. 15 . When the state (State) of the information processing apparatus IPE 1  is ST 5  (=5) in an 11th row, rows from a 12th row to a 23rd row are executed. Comments added to the respective rows indicate corresponding of processes executed in the rows from the 12th row to the 23rd row to Steps illustrated in  FIG. 14 . 
     In the above description, in the embodiment illustrated in  FIGS. 1 to 20 , it is possible to change the processing performance of the information processing system SYS 1  depending on a change of the load of the physical server P without an influence of the information processing system SYS 1  on efficiency in information processing. For example, the control device CNTL detects that the load of the physical server P tends to increase and replaces the virtual server V with the physical server P and thus it is possible to increase the processing capacity of the information processing apparatus IPE 1 . At this time, switching the coupling of the storage device D from the virtual server V to the physical server P causes data not to be transitioned between the storage devices D. Accordingly, it is possible to improve the processing performance of the information processing system SYS 1  without an influence on a service provided by the information processing system SYS 1 . 
     Addition of the virtual server V resulting in transition of data is performed during a time period when a tendency of increasing the load of the physical server P is lower than a tendency of increasing the load when the virtual server V is replaced with the physical server P. Thus, it is possible to reduce an influence of data transition processing which occurs due to addition of the virtual server V, on a service of the storage server and the like, compared to when the load becomes large and thus data is transitioned. 
     When the load of the physical server P becomes small, it is possible to stop one of the physical servers P by replacing the physical server P with the virtual server V. Accordingly, it is possible to decrease the processing capacity of the information processing apparatus IPE 1  and to largely reduce the power consumed by the information processing apparatus IPE 1 , compared to when one of the physical servers P continues to be in operation. At this time, switching the coupling of the storage device D from the physical server P to the virtual server V causes data not to be transitioned between the storage devices D. Accordingly, it is possible to degrade the processing performance of the information processing system SYS 1  without an influence on a service provided by the information processing system SYS 1 . 
     Removal of the virtual server V resulting in transition of data is performed when the load of the physical server P is smaller than a load when the physical server P is replaced with the virtual server V. Accordingly, it is also possible to reduce an influence on a service provided by the information processing system SYS 1  to the minimum when removing the virtual server V causes data to be transitioned between the storage devices D. 
       FIG. 21  illustrates an example in which a control device changes the state of an information processing apparatus IPE 1  in another embodiment of the information processing system, the control method of the information processing system, and the control program of the information processing system. A control device CNTL which executes processes illustrated in  FIG. 21  and an information processing apparatus IPE 1  which is controlled by the control device CNTL are similar to those in  FIG. 1  except that a program executed by the control device CNTL is different.  FIG. 21  illustrates a control method executed by the control device CNTL mounted in the information processing system SYS 1  illustrated in  FIG. 1  and processes by the control program executed by the control device CNTL. Detailed descriptions of components the same as or similar to those in the embodiment illustrated in  FIGS. 1 to 20  will be omitted. 
     “NOW” illustrated in  FIG. 21  indicates the current state of the information processing apparatus IPE 1 . Processes A to E illustrated around the state “NOW” in  FIG. 21  represent operations of the information processing apparatus IPE 1  executed by control of the control device CNTL. Dot-lined arrows indicate that the state of the information processing apparatus IPE 1  which executes processes by control of the control device CNTL becomes the state “NOW” anew. In  FIG. 21 , upward arrows which are positioned behind numerical values of the load and the frequency indicate that the load and the frequency are larger than the numerical values. Each downward arrow which is positioned behind a numerical value of the load indicates that the load is smaller than each of the numerical values. In  FIG. 21 , “load” and “frequency” are the same as “load” and “frequency” which are described with reference to  FIGS. 1 to 20 . 
     For example, when the frequency of that the load of any one of the physical servers P in operation exceeds 30% and is equal to or less than 80% exceeds “0.8” ((Condition A) described with reference to  FIG. 3 ), the control device CNTL executes Process A of changing the state of the information processing apparatus IPE 1 . The control device CNTL replaces one of the virtual servers V with one of the physical servers P by terminating any one of the virtual servers V and operating any one of the physical servers P, in Process A. The storage device D which has been coupled with the virtual server V to be terminated is coupled with the physical server P to be operated. 
     For example, the change of the state of the information processing apparatus IPE 1  through Process A is the same as the change of the state ST 0  to the state ST 1 , the change of the state ST 1  to the state ST 2 , and the change of the state ST 3  to the state ST 5  which are illustrated in  FIG. 5 . When the load exceeds 80%, Process C may be executed prior to Process A in accordance with “frequency”. 
     When the frequency of the load of any one of the physical servers P in operation exceeds 30% and is equal to or less than 80% exceeds “0.7” ((Condition B) described with reference to  FIG. 3 ), the control device CNTL executes Process B of changing the state of the information processing apparatus IPE 1 . The control device CNTL adds one of the virtual servers V to the information processing apparatus IPE 1  by causing any one of the terminated virtual servers V to be operated on any one of the physical servers P in operation, in Process B. For example, the change of the state of the information processing apparatus IPE 1  through Process B is the same as the change of the state ST 2  to the state ST 3  illustrated in  FIG. 5 . When the frequency exceeds “0.8”, Process A is executed prior to Process B. When the load exceeds 80%, Process C may be executed prior to Process B in accordance with “frequency”. 
     When the frequency of the load of any one of the physical servers P in operation exceeding 80% exceeds “0.9” ((Condition C) described with reference to  FIG. 3 ), the control device CNTL executes Process C of changing the state of the information processing apparatus IPE 1 . The control device CNTL adds one of the physical servers P to the information processing apparatus IPE 1  by causing any one of the currently stopped physical servers P to be operated, in Process C. For example, change of the state of the information processing apparatus IPE 1  through Process C is the same as change of the state ST 2  to the state ST 4  illustrated in  FIG. 5 . 
     When the frequency of the load of any one of the physical servers P in operation at less than 2% exceeds “0.8” ((Condition D) described with reference to  FIG. 3 ), the control device CNTL executes Process D of changing the state of the information processing apparatus IPE 1 . The control device CNTL replaces one of the physical servers P with one of the virtual servers V by causing any one of the physical servers P to be stopped and causing any one of the virtual servers V to be operated, in Process D. The storage device D which has been coupled with the physical server P to be stopped is coupled with the virtual server V to be operated anew. 
     For example, the change of the state of the information processing apparatus IPE 1  through Process D is the same as the change of the state ST 1  to the state ST 0 , the change of the state ST 2  to the state ST 1 , the change of the state ST 4  to the state ST 3 , and the change of the state ST 5  to the state ST 3  which are illustrated in  FIG. 5 . When the frequency exceeds “0.9”, Process E is executed prior to Process D. 
     When the frequency of the load of any one of the physical servers P in operation at less than 2% exceeds “0.9” ((Condition E) described with reference to  FIG. 3 ), the control device CNTL executes Process E of changing the state of the information processing apparatus IPE 1 . The control device CNTL removes one of the virtual servers V from the information processing apparatus IPE 1  by terminating any one of the operating virtual servers V, in Process E. For example, the change of the state of the information processing apparatus IPE 1  through Process E is the same as the change of the state ST 3  to the state ST 2  illustrated in  FIG. 5 . 
     When the load of any one of the physical servers P in operation satisfies no condition for executing Process A, Process B, Process C, Process D, and Process E, the control device CNTL holds the state of the information processing apparatus IPE 1  without changing (“F” in  FIG. 21 ). 
       FIG. 22  illustrates an example of a relationship between each of Process A, Process B, Process C, Process D, and Process E, the load, and the frequency illustrated in  FIG. 21 . As described with reference to  FIG. 21 , Process C has the primary priority and Process A has the secondary priority in a condition obtained by overlapping the conditions for executing Process A, Process B, and Process C with each other. Process E has a priority in a condition obtained by overlapping the conditions for executing Process D and Process E with each other. 
     In the above description, in the embodiment illustrated in  FIGS. 21 and 22 , it is also possible to obtain an effect similar to the effect obtained in the embodiment illustrated in  FIGS. 1 to 20 . That is, it is possible to improve the processing performance of the information processing system SYS 1  depending on the change of the load of the physical server P without an influence of the information processing system SYS 1  on efficiency in information processing. Additionally, it is possible to degrade the processing performance of the information processing system SYS 1  depending on the change of the load of the physical server P without an influence of the information processing system SYS 1  on efficiency in information processing. 
     Features and advantages according to the embodiments will be apparent from the above-described detailed description. This intends to have the features and the advantages according to the embodiments without departing from the spirit and scope of the claims. A person with ordinary skill in the art will easily consider various enhancements and changes. Accordingly, appropriate improvements and substitutions which could be included in the scope disclosed in the embodiments may be made without the intention to limit the range of the embodiments having the inventive characteristics according to the above description. 
     All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.