Abstract:
The present invention provides for redistributing workloads among computers to optimize resource utilization. Utilization by software workloads of computer resources is monitored to yield utilization data. A utilization chronology is updated using the utilization data. The chronology is analyzed to yield resource utilization predictions. The workloads are redistributed among the resources at least in part as function of said predictions.

Description:
BACKGROUND OF THE INVENTION  
       [0001]     Herein, related art is discussed to aid in understanding the invention. Related art labeled “prior art” is admitted prior art; related art not labeled “prior art” is not admitted prior art.  
         [0002]     One of the classic challenges for information technology (IT) managers is to insure that they have capacity for their peak computational loads. As a result, they may typically have low utilization of their computers (for example 25% of capacity) except for the rare occurrence of 100% peak. For example, an Internet service provider (ISP), such as AOL, may have peak usage of their web servers once a day during the late evening hours since most of their clients surf the web after dinner. The ISP has their servers on all the time consuming power even though utilization during most of the day may be less than 25% of the capacity; as a result, at least 75% of the power is wasted. In addition, servers are usually not turned off or put in sleep mode (as is done with laptops) when not in use so that servers can be activated instantaneously on demand.  
         [0003]     To deal with this, IT managers have done the following: 1) they buy enough computers to deal with peak demands; the disadvantage of this is that full power is being used by computers even though they are just partially utilized; 2) they use “Instant Capacity” (ICAP, available from Hewlett-Packard Company) and comparable solutions that allow them to add processors based on demand; the disadvantage is that while the data center owner is not paying for the unused hardware, it is still paying for the power consumed by it; and 3) they buy computers to deal with the average loads; the disadvantage of this is that the IT manager&#39;s company then cannot support peak demand and thus loses revenue.  
         [0004]     For economic and energy-conservation purposes, it would be beneficial to improve the match between workload and resources. This can mean getting more done with the resources available, more accurately matching resource requirements to energy consumption, or freeing unneeded resources for inactivation so that energy consumption can be reduced. One way to do this is to select optimal power versus performance modes based on feedback from utilization monitors. For example, if utilization of a processor is 50% when it is in high-performance mode, its clock speed can be lowered to achieve a higher utilization ratio and a more-than-proportional reduction in electrical energy usage. Concomitantly, power can be reallocated among components based on utilization. However, such methods are not optimal. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0005]     The following drawings are of embodiments/implementations of the invention and not of the invention itself.  
         [0006]      FIG. 1  is a hybrid block diagram of a computer system and a flow chart of a method practiced in the context of the system in accordance with an embodiment of the present invention.  
         [0007]      FIG. 2  is a block diagram of another computer system in accordance with an embodiment of the invention.  
         [0008]      FIG. 3  is a flow chart of another method in accordance with an embodiment of the invention.  
     
    
     DETAILED DESCRIPTION  
       [0009]     In the course of the present invention, it was realized that the foregoing methods based on current measurements of utilization, while beneficial, were limited in what they could accomplish. They are limited because they control parameters that can be changed relatively quickly in response to changes in power demands. It was further realized that, if future use could be predicted, applications could be assigned to servers in a way that would minimize the number of servers without impacting performance. This would reduce energy consumption and maintenance costs. Furthermore, there would be less need to overbuy to avoid performance bottlenecks.  
         [0010]     A server environment AP 1  in accordance with the present invention is shown in  FIG. 1  at times T 1  and T 2 . Server environment AP 1  includes three servers  11 A,  11 B, and  11 C, and a management workstation  13 . Servers  11 A,  11 B, and  11 C have reduced power modes. In the illustrated embodiment, servers  11 A,  11 B, and  11 C, and management workstation  13  are unpartitioned stand-alone computers. However, in alternative embodiments, the servers can be blades in a rack, or hard partitions of host servers; also, the function of management workstation  13  may be assumed by one of the servers, partitions, or blades. More generally, the invention applies to server environments and computer networks with any number of computers of any configuration and with any combination of software installed.  
         [0011]     Server  11 A includes a power supply  15 A and runs software including a host operating system  17 A, and a utilization monitor  19 A. In addition, there can be plural virtual partitions and machines. At time T 1 , server  11 A runs a migrateable workload WL 1 , which can be an application program or a virtual machine running an application program. Server  11 B includes a power supply  15 B and runs a host operating system  17 B, and a utilization monitor program  19 B. At time T 1 , server  11 B runs migrateable workloads WL 2  and WL 3 . Server  11 C includes a power supply  15 C, and runs a host operating system  17 C and a utilization monitor program  19 C. At time T 1 , server  11 C also runs a workload WL 4 .  
         [0012]     Management workstation  13  runs a workload manager program  21  on computer readable media. Workload manager  21  predicts future application use based on real-time statistical evaluation of historic utilization data stored in workload chronology database  23 . This analysis is used to dynamically match workloads to servers to implement management policies  25 , e.g., minimizing the number of resources needed so that unused resources can be inactivated.  
         [0013]     Servers  11 A,  11 B, and  11 C, and management workstation  13  cooperate to implement a method ME 1  in accordance with the invention. Method segment M 1  involves collecting local utilization data. This collecting is performed concurrently at method segments M 1 A, M 1 B, and M 1 C, respectively, for servers  11 A,  11 B, and  11 C by the respective utilization monitoring programs  19 A,  19 B, and  19 C, each of which provides a measure of power consumption within particular computing resources. This power consumption can be a direct measurement of hardware power usage or it may be inferred from the activity level of elements within the computing resource. This monitoring can continue for days, weeks, months, and even years to capture usage patterns over like periods of time.  
         [0014]     Monitoring programs  19 A,  19 B, and  19 C are functionally similar, all performing method sub-segments SS 1 , SS 2 , and SS 3  as described below with respect to server  11 C. At method subsegment SS 1 , utilization monitor program  19 C monitors resource utilization by server  11 C in general and workload WL 4  in particular, generating resource utilization data. This data is time-stamped at method subsegment SS 2 ; the time-stamped utilization data is stored locally at method subsegment SS 3 .  
         [0015]     At method segment M 2 , workload manager  21  gathers the stored time-stamped data generated at method segment M 1 . Workload manager  21  monitors and records power consumption on an aggregate level via system software. This data aggregation can be across computing systems in a data center or across blade servers in a rack-mounted system.  
         [0016]     At method segment M 3 , workload manager  21  updates (or creates, if one is not already created) workload chronology  23  using the recently gathered utilization data. Over time, chronology  23  expands to include hourly, daily, weekly, monthly, and yearly utilization and configuration data. The configuration data includes: 1) server configuration data that can aid in determining the amount of resources available for workloads; and 2) workload configuration data from which can aid in determining the amount of resources a workload is likely to consume. For example, a chronology entry might indicate that a given application running on a given virtual machine with its respective specifications running on given hardware with its respective specifications is utilizing 30% of processor capacity, 20% of memory capacity, 10% of input/output device capacity, and 5% of disk storage capacity at a particular date and time.  
         [0017]     At method segment M 4 , workload manager  21  analyzes chronology  23 , regarding resource utilization by workloads WL 1 -WL 4 . For example, analysis of chronology  23  may indicate daily, weekly, monthly, and/or yearly patterns in the utilization associated with a workload. Some cases in point: 1) a web portal has peak usage in the evening because users tend to access the portal after dinner; 2) an accounting application might be active at the end of months and quarters; and 3) an on-line retail application might be impacted by surging sales during certain times of the year.  
         [0018]     The analysis of method segment M 4  can discover time-based patterns that can be extrapolated to predict future usage at method segment MS. The prediction can be of the form that the future will replicate the past, e.g., since a portal has been busy every weekday at 8 pm for the past two months, it will continue to be busy at 8 pm everyday in the future. However, predictions can also incorporate trending, e.g., the prediction can be to the following effect “the application continues to be busy at 8 pm and the magnitude of the busy peaks will grow 10% over the next six months”. Furthermore, the chronology can be associated with external events (e.g., economic indicators, weather) so that the predictions can be event based and not merely time based. Note that this data can be used for determining a demand growth rate and thus assist information technology buying decisions.  
         [0019]     At method segment M 6 , workloads WL 1 -WL 4  are redistributed at least in part as a function of the predictions. The function is derived from management policies  25 . For example, based on the analysis and management policies, workload management software migrates applications to the most effective computing system footprint that provides optimal power consumption in at least three variations: 1) explicit power level controls across a complex; 2) bin-packing applications into the smallest high-utilization footprint; and 3) a combination of 1 and 2. Also, the policy can stipulate how variations in available power are to be addressed, e.g., during a brownout.  
         [0020]     Once the redistribution is implemented, any “freed” resource (e.g., processor, input/output device, memory, storage, partition, or whole server) from which all workloads have been removed, can be shut down or placed into a lower-power state at method segment M 7 . For example a system may enter a lower-power state such as a C1 processor state, a D1 device state, a G1 global state, etc, as defined by the ACPI specification. For example, a Montecito Itanium processor (available from Intel Corporation) can be set to a “halt-lite” state. In this HALT_LITE state, processing is halted but state data is preserved so that full operation can be resumed with minimum latency. Also, memory modules can be turned off or put in lower power modes. In addition, storage media can enter low power modes or hard disk spindles can be turned off. Also, parts of a network fabric can be powered down.  
         [0021]     “Advanced Configuration and Power Interface Specification” (ACPI) promulgated by Hewlett-Packard Corporation, Intel Corporation, Microsoft Corporation, Phoenix Technologies Ltd., and Toshiba Corporation, Revision 3.0, Sep. 2, 2004, particularly pages 13-23. The ACPI specification defines global power states G0-G3, device states D0-D3, and processor states C0-C3. In addition, there are “sleeping” states with state G1 and performance level states P0, and P1-Pn within device state D0 and processor state C0. Not all systems, devices, and processors have all states. Systems, devices, and processors that do not conform nominally to the ACPI standard often have analogous states.  
         [0022]     Depending on the how much risk is acceptable according to management policies  25 , workstation manager  21  can use average, sigma, 2-sigma or 3-sigma statistical data. The analysis can be implemented with fine granularity (such as turning one server off at a time or in large blocks of servers depending on what makes more sense from a server management standpoint). Also, computers are powered on just prior to when the need for their services is expected to arise. What constitutes “just prior” depends on the precision of a prediction and confidence level of a prediction. A resource should be ready earlier when the confidence level and/or precision of a prediction is low.  
         [0023]     In  FIG. 1 , the arrow from “predict utilization” method segment M 5  to “collect data” method segment M 1  indicates that method ME 1  is iterated so that monitoring, chronology updates, analyses, and workload redistributions are ongoing. Continued power measurement and monitoring provides a feedback loop to workload manager  21  to fine tune—as needed—the workload/power analysis and continue to take appropriate policy-dictated action in the form of application migration. This allows inactivated resources to be brought online “just in time”, and provides savings analogous to those achieved by “just-in-time” manufacturing.  
         [0024]     The following description of purely hypothetical outcomes is presented for expository purposes. One example of a management policy would be to use as few servers as possible provided no server utilization exceeds 90% more than 10% of the time, and, if that part of the criterion is met, to distribute workloads to spread resource utilization evenly among the servers that are used. Such a policy might result in the workload distribution shown for server environment AP 1  at time T 2  in  FIG. 1  and described below.  
         [0025]     The chronology analysis performed at method segment M 4  may have determined that workload WL 1  can require 60% of the processor capabilities of a server during the day, but about 20% at night. On the other hand, the analysis might indicate that workload WL 2  uses about 20% during the day and 60% at night. Accordingly, even though the sum of the peak utilizations for workloads WL 1  and WL 2  exceeds 100%, the peak of their combined loads would be 80%. Extrapolating these percentages into the future, yields an expected peak combined resource utilization of 80%. In accordance with this prediction, they have been combined at T 2  on one server. This is a special case of having workloads having periodic peaks of the same periodicity and complementary phases sharing resources.  
         [0026]     Similarly, the analysis might predict that high-priority workload WL 2  is used intermittently. In that case, workload WL 2  might be assigned a high priority, for example, because it involves real-time video transfer. Since its requirements are intermittent it can be complemented by workload WL 4 , assuming the analysis indicates it is heavily used but is assigned a low priority, e.g., because its results can be other than real time.  
         [0027]     Pursuant to the hypothetical analysis, workload WL 3  is reallocated (migrated) from server  11 B to server  11 A as indicated by migration arrow  41 , while workload WL 2  is reallocated from server  11 B to server  11 C, as indicated by migration arrow  43 . Since server  11 B has no workloads after redistribution, it can be shutdown or put in a sleep or other low-power mode, thus saving energy and reducing operating costs. The predictions can also be used to combine workloads with other types of complementary relationships, e.g., a workload having a growing demand can be paired with workload for which demand is projected to decrease.  
         [0028]     Since the predictions allow for more effective workload distributions, they can be used to extend the usefulness of a current level of hardware. In other words, expansion of existing hardware can be delayed. In the case where space, cooling capacity, and power infrastructure are stressed, this delay can mean the lifetime of an existing plant can be extended and investments in new facilities can be forestalled.  
         [0029]     Generally, management policies, including management policies  25 , can be used to define a (possibly very complex) criterion for evaluating potential redistributions. Management policies  25  take power conservation into consideration for the reasons indicated in the background section above; however, other management policies include very different factors and not all specify power conservation as a factor. For example, policies can be designed to optimize individually or in combination factors such as hardware cost, highest utilization, and lowest cost of ownership (including energy use and depreciation). Further examples are considered below.  
         [0030]     Some interesting management policy factors apply to server environments that are geographically distributed over two or more continents. The policies may include restrictions based on export laws and software licenses; for example, it may not be possible to migrate certain encryption programs to an overseas site. The policies might require that mirror instances of a workload not be located at the same site where they both might fail in the event of a local power outage. The management policies might favor servers that are completely owned by a customer over those in which the vendor retains usage rights, e.g., reserve processors.  
         [0031]     Management policies can also take into account the relative ease or difficulty of migrating workloads. More headroom can be reserved when migration is time consuming and/or expensive. On the other hand, less headroom can be reserved where migration is fast and inexpensive.  
         [0032]     From the power perspective: 1) System software provides a measure of power consumption within a particular computing resource; this power consumption may be a direct measurement of hardware power usage or it may be inferred from the activity level of elements within the computing resource. 2) Workload management software monitors and records power consumption on an aggregate level via system software; this data aggregation can be across computing systems in a data center or across blade servers in a rack-mounted system. 3) The analysis takes current power consumption and historical data and, for a given policy, determines the best placement of workload on given computing systems to optimize power consumption in the context of workload priorities. 4) Based on the analysis and management policies workload management software migrates applications to the most effective computing system footprint that provides optimal power consumption. 5) Continued power measurement and monitoring provides a feedback loop to workload management software to fine tune, as needed, the workload/power analysis and continue to take appropriate policy-dictated action in the form of application migration.  
         [0033]     From the utilization/footprint perspective: 1) System software provides a measure of utilization within a particular computing resource. 2) Workload management software monitors and records data on utilization via system software. The aggregation of this data can be, for a pair of examples, across computing systems in a data center or across blade servers in a rack-mounted system. 3) The analysis can take current utilization and historical data and, for a given policy, determine the best placement of a workload on given computing systems to optimize utilization/footprint in the context of workload priorities. 4) Based on the analysis and management policies workload management software migrates applications to the most effective computing that provides optimal utilization/footprint within the server and/or across the data center. 5) Continued utilization measurement and monitoring provides a feedback loop to workload management software to fine tune—as needed—the workload/utilization/footprint analysis and continue to take appropriate policy-dictated action in the form of application migration.  
         [0034]     The invention provides for dynamic workload management via migration of applications into the optimal power footprint based on power consumption data analysis. The strong customer need for (both large and small scale) dynamic power management has been shown anecdotally. Explicitly stated, the benefits could be: 1) Lower cost to users since consolidation is maximized and, thus, the amount of hardware is minimized. 2) The power allotted to applications is minimized without impairing application performance; this lowers overall operating costs.  
         [0035]     An iteration of method ME 1  can be initiated periodically by workload manager  21  or it can be asynchronously triggered in response to an “alarm” from a workload monitor, e.g., monitor  19 A, when it detects that locally available resources are being stressed or some utilization threshold has been met. data. Thus a plan can be in place before an alarm is received so that the redistribution can be implemented more quickly. The orchestrator can also be configured to initiate redistribution proactively without involving the workload manager. In this case, the orchestrator can get the plan from the capacity advisor each morning and decide if and when something needs to move. The orchestrator then schedules this move based on downtime windows, etc. for each of the workloads.  
         [0036]     In general, a hardware, computer, or system resource is any part of a system that is capable of performing a task. Examples of resources typically found in computer systems include processors, input/output (I/O) devices, data storage devices, communication ports, displays, peripheral devices (e.g., printers, scanners, etc.), and so on. Of course, a “system” may be considered at any level of granularity. For instance, a plurality of computers communicatively coupled together via a communication network (e.g., a local area network (LAN), the Internet, or other wide area network (WAN), etc.) may collectively be considered a system, wherein each computer may be considered a system resource.  
         [0037]     As another example, a large data center may be considered a system, and a plurality of computers (and other power-consuming resources) in the data center may each be considered a system resource. As another example, a printed circuit board within a personal computer (PC) may be considered a system, wherein each device (e.g., chip, etc.) implemented on the printed circuit board may be considered a system resource. The term “system” encompasses various types of computing systems, such as data centers, servers, personal computers (PCs), digital cameras, personal digital assistants (PDAs), laptops, workstations, mobile telephones, etc. Further, the term “system,” unless otherwise qualified, is not limited to traditional computer systems, but may include any type of system having power-consuming resources, such as automobiles, airplanes, factories, etc.  
         [0038]     In the illustrated embodiment, time-stamping and storage are performed on each server. In other embodiments, these method sub-segments can be omitted or performed after management workstation gathers the data from the servers. These and other variations upon and modification to the illustrated embodiment are provided for by the present invention, the scope of which is defined by the following claims.