Source: https://patents.google.com/patent/US9207993B2/en
Timestamp: 2019-03-20 08:18:40
Document Index: 774801931

Matched Legal Cases: ['art 2', 'art-2', 'Application No. 201080009556', 'Application No. 12803686', 'Application No. 10744134', 'Application No. 201080009556', 'Application No. 201180024779', 'Application No. 2011', 'Application No. 201080009556', 'Application No. 12803686', 'Application No. 2010800095561', 'Application No. 10744134', 'Application No. 201080009556', 'Application No. 2011551110', 'Application No. 201080009556', 'Application No. 201080009556', 'Application No. 2011551110', 'Application No. 2011', 'Application No. 2011', 'Application No. 201080009556', 'Application No. 10744134', 'Application No. 10744134', 'Application No. 201080009556', 'Application No. 10744134', 'Application No. 10', 'Application No. 2010800095561', 'Application No. 10744134']

US9207993B2 - Dynamic application placement based on cost and availability of energy in datacenters - Google Patents
Dynamic application placement based on cost and availability of energy in datacenters Download PDF
US9207993B2
US9207993B2 US12/779,059 US77905910A US9207993B2 US 9207993 B2 US9207993 B2 US 9207993B2 US 77905910 A US77905910 A US 77905910A US 9207993 B2 US9207993 B2 US 9207993B2
US12/779,059
US20110282982A1 (en
2010-05-13 Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
2010-05-13 Priority to US12/779,059 priority Critical patent/US9207993B2/en
2010-05-13 Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: JAIN, NAVENDU
2011-11-17 Publication of US20110282982A1 publication Critical patent/US20110282982A1/en
2015-12-08 Publication of US9207993B2 publication Critical patent/US9207993B2/en
An optimization framework for hosting sites that dynamically places application instances across multiple hosting sites based on the energy cost and availability of energy at these sites, application SLAs (service level agreements), and cost of network bandwidth between sites, just to name a few. The framework leverages a global network of hosting sites, possibly co-located with renewable and non-renewable energy sources, to dynamically determine the best datacenter (site) suited to place application instances to handle incoming workload at a given point in time. Application instances can be moved between datacenters subject to energy availability and dynamic power pricing, for example, which can vary hourly in day-ahead markets and in a time span of minutes in realtime markets.
Energy costs are rising as systems scale up and becoming an increasingly larger fraction of running datacenters, accounting for up to a substantial portion of the total expenditure and amounting to tens of millions of dollars per annum. To reduce these costs, efforts are being made in two different directions to combine traditional grid power (e.g., electric grid) and backup power (e.g., UPS and diesel generators) with renewable energy sources (e.g., solar, wave, tidal, and wind), and to exploit the variation of electricity prices both temporally and geographically (e.g., utility prices can change across different regions and even across the same region, based on diurnal patterns—peak and off-peak periods—or even significantly over a window of 5-15 minutes across the country).
The disclosed architecture is an optimization framework that dynamically places application instances across multiple hosting sites (e.g., datacenters) based on the energy cost and availability of energy at these sites, application SLAs (service level agreements), availability of compute elements at hosting sites, and cost of network bandwidth between sites, just to name a few considerations. Existing cloud computing infrastructures can now employ the optimization framework and realize significant economic gains.
The architecture leverages a global network of hosting sites, possibly co-located with renewable and non-renewable energy sources, to dynamically determine the best datacenter (site) suited to place application instances to handle incoming workload at a given point in time, in order to minimize the total cost of running the requisite set of application instances across the hosting sites. Application instances can be moved between datacenters subject to energy availability and dynamic power pricing in a multi-electricity market environment, for example, which can vary hourly in day-ahead markets and in a time span of minutes in realtime markets.
The architecture is a closed-loop control system for dynamic application placement that is integrated with a reliable language runtime mechanism to transparently migrate application instances between datacenters, resulting in cutting the energy costs for operating datacenters. A decision component reasons over the hosting sites data and outputs power optimization decisions associated with the hosting sites. An application placement component automatically relocates application instances between hosting sites based on the power optimization decisions.
FIG. 1 illustrates an optimization system in accordance with the disclosed architecture.
FIG. 2 illustrates an alternative embodiment of a system that includes a learning component that learns about hosting site information, a simulator, and a decision component.
FIG. 3 illustrates a hosting sites model that can be employed for the optimization framework.
FIG. 4 illustrates an application instances model that can be employed for the optimization framework.
FIG. 5 illustrates a sorted sequence of power sources based on end time of source power availability.
FIG. 6 illustrates an algorithm that computes optimum partition intervals.
FIG. 7 illustrates that each power source may have different start and end times of availability for power pricing.
FIG. 8 illustrates an intelligent power control system for the hosting site.
FIG. 9 illustrates a computer-implemented optimization method in accordance with the disclosed architecture.
FIG. 11 illustrates further aspects of the method of FIG. 9.
FIG. 12 illustrates a block diagram of a computing system that executes the optimization framework in accordance with the disclosed architecture.
FIG. 13 illustrates a schematic block diagram of a computing environment that performs optimization in accordance with the disclosed architecture.
The disclosed optimization architecture leverages a global network of datacenters (also referred to as hosting sites), possibly co-located with renewable (and non-renewable) energy sources, to dynamically determine the best hosting location (e.g., a computer, a rack, a “colo” (co-location—multiple customer usage of single datacenter), a container, a datacenter, etc.) suited to place (or schedule) application instances at a given point in time. For example, applications may be interactive to handle user requests, or batch applications that process data or perform computations. The optimization framework provides the capability to move application instances between datacenters subject to energy availability and dynamic power pricing, which can vary (a) temporally: hourly in day-ahead markets and in a time span of five minutes in realtime markets, and (b) geographically: pricing variation in different regional markets, even when nearby, and even across locations in the same region. The pricing variation arises due to several factors such as regional demand differences, transmission inefficiencies, and generation diversity. To make decisions on application migration, the framework also takes into account dynamic availability of datacenter equipment due to factors such as power failures or server failures.
The disclosed architecture includes control system for dynamic application placement integrated with a reliable language runtime mechanism (e.g., a programming model for applications implemented as “grains/silos” (stateless, light-weight processes) that enables transparent code mobility) and/or run as an infrastructure service co-located with hosting virtual machines (VMs) to transparently migrate application instances hosted in VMs between datacenters (without impacting the end user), thereby cutting the electric costs for operating datacenters. Measurement tools can be employed to measure the dynamic power footprint of hosted application instances (or hosting VMs).
A general algorithm for dynamic application placement is described as follows: formulate the problem according to the system model described herein, and obtain the current values of the different parameters in the framework (either offline or in an online manner as needed), where realtime monitoring of application QoS (quality of service) and performance counters may also be performed. These values are fed into the optimization framework (which can run on a central controller machine in the datacenter), and solved to compute the optimal placement of application instances across different hosting locations (sites) based on cost and availability of energy as well as the availability of hosting compute elements, among other factors such as bandwidth cost and latency for migrating application instances between sites. These decisions are then actuated to migrate application instances from the current placement to the optimal placement. The optimization problem is solved with a fixed periodicity or on-demand.
The actuation of migrating application instances may also be integrated with request redirection of incoming user requests that can be performed transparently (e.g., via domain name service (DNS) redirection), and replication of application-state and user-data at different hosting sites.
Regarding where the optimization framework is executed, a central facility or a distributed set of machines may monitor execution of applications and hosting site and application instance models, perform software probes, and disseminate and check execution of migration decisions, among other functions.
Note that the algorithm can complement other closed-loop control frameworks for managing pools of resources (e.g., servers) in a datacenter. For example, in one approach, given a large number of servers hosting cloud applications in a datacenter, this approach aims to maximize the energy savings of operating these computers by only running the optimal number of servers needed to process the input workload under given service-level agreement (SLAs), while keeping remaining servers in low-power states (e.g., ACPI (advanced configuration and power interface) performance and sleep states such as sleep, hibernate) or in a shutdown mode to save energy.
FIG. 1 illustrates an optimization system 100 in accordance with the disclosed architecture. The system 100 includes multiple hosting sites 102 that run application instances to handle workload. For example, a first hosting site 104 includes first application instances 106, a second hosting site 108 includes second application instances 110, and a third hosting site 112 includes third application instances 114. Each of the hosting sites 102 operate according to realtime energy parameters 116, availability and pricing of energy sources 118, and available computing elements 120. An application placement component 122 automatically relocates application instances between hosting sites based on changes in the energy parameters, the availability of the energy sources, and the availability of the computing elements, among other factors such as cost of application migration.
As described in greater detail herein, the hosting sites 102 are modeled according to one or more of hosting capacity, energy available per source per site at a specific time in terms of watt power intensity and duration, energy cost per source per site at the specific time for each of different available sources of energy, network latency between sites, bandwidth capacity between sites, bandwidth cost per byte between sites, available renewable and non-renewable energy sources, temperature per site, and temperature threshold for each site. One or more of the above data points can be output every configured time interval for consumption by a decision engine (described below).
The application instances (106, 110, and 114) are modeled according to one or more of CPU usage per instance, memory footprint size per instance, number of network connections maintained by an instance, power footprint per instance, SLAs on hosting of application instances, and latency threshold for migrating an instance between two sites.
The availability of an energy source is computed based on at least one of predicted power per site for a specific time, predicted time until which energy is available per site for a specific time, energy per watt-hour cost per site for a specific time, or energy (power) used per site during intervals in a time window, among other factors. The application placement component may relocate application instances based on results of a simulator model. The application placement component 122 makes decisions based on minimizing energy cost and bandwidth cost per migration, meeting service level agreement requirements during the migration or including latency information in computation of cost.
FIG. 2 illustrates an alternative embodiment of a system 200 that includes a learning component 202 that learns about hosting site information, a simulator 204, and a decision component 206. The learning component 202 learns information of the availability of an energy source based on historical energy availability data and future event notifications, for example. The learned information is applied to enhance the prediction power cost and variation in pricing, availability, time until which power is available per site, and at a specific time.
The system 200 includes the multiple hosting sites 102 that run application instances to handle workload. The hosting sites 102 are managed according to data that includes realtime energy parameters, availability of renewable and non-renewable energy sources, and available computing elements. The decision component 206 reasons over the data and outputs power optimization decisions associated with the hosting sites 102. The decision component 206 can receive as inputs the parameters associated with a hosting model 208 (described in FIG. 3) and the parameters associated with an application instance model 210 (described in FIG. 4). The decision component 206 can also receive as inputs a location energy model 212, client geographic location (geolocation) information 214, and topology information of the hosting sites 216. The location energy model 212 estimates power consumption at a location as a function of number of active servers, server utilization, type and number of hosted applications, and infrastructure parameters, among other factors. The application placement component 122 automatically relocates application instances between hosting sites based on the power optimization decisions.
A prediction model can be employed based on electricity grid data as well as weather differentials and climate historical data available from weather services such as NASA (North American Space Administration) and NCAR (National Center for Atmospheric Research) to predict availability of different energy sources such as solar (e.g., intensity, time of day) and wind (e.g., intensity, direction) as used in the framework. The learning component 202 can utilize the data from the prediction model as well as the national agencies to provide additional learning information.
The simulator 204 can be built to use historical data and make predictions on availability of different energy sources. Further, the simulator model itself can be used to install/place generators of renewable energy at different geographic locations that will yield the maximum energy output per monetary cost and take into consideration other factors such as land availability and pricing, taxes, network connectivity, etc.
Put another way, an optimization system 200 is provided that comprises multiple hosting sites that run application instances to handle workload, the hosting sites are managed according to data that includes realtime energy parameters, availability of renewable and non-renewable energy sources, and available computing elements, the decision component that reasons over the data and outputs power optimization decisions associated with the hosting sites, and the application placement component that automatically relocates application instances between hosting sites based on the power optimization decisions.
The decision component reasons over data that includes at least one of hosting capacity, energy available in terms of power intensity and duration and price per site at a specific time, total energy cost per site at the specific time, network latency between sites, bandwidth capacity between sites, bandwidth cost per byte between sites, available renewable and non-renewable energy sources, length of partition intervals at the specific time, availability of hosting compute elements at different sites, temperature per site, or temperature threshold for each site.
The optimization decisions are based on at least one of minimizing monetary costs, minimizing impact on user interactiveness, maximizing longterm availability, maximizing use of renewable resources, minimizing carbon footprint and pollutant emission due to electricity generation and environmental impact, or balancing bandwidth cost and energy cost.
The decision component reasons over the data that includes application instance information related to at least one of CPU usage and memory footprint size per instance, number of network connections maintained by an instance, power footprint per instance, SLAs on hosting of application instances, or latency threshold for migrating an instance between two sites.
The system 200 can further comprise the learning component that learns information of the availability and price of an energy source based on historical energy availability and pricing data and future event notifications, the learned information applied to enhance a prediction time, power intensity, and energy cost until which power is available and its cost per site at a specific time.
Regarding learning component, it may also provide functionality of learning failure patterns of hosting compute elements from historical data and future event notifications so that the learned information can be used in the decision component 206.
FIG. 3 illustrates the hosting sites model 208 that can be employed for the optimization framework. The optimization is run at a time instant t and at a periodicity τ (e.g., seconds, minutes, or hours). For ease of explanation, the following assumptions on energy availability are made: the fixed power availability (intensity and time duration) and fixed energy cost for each source is known from time instant t to t+τ; and, each energy source is available at time instant t and the energy source availability may expire before time interval τ, but is available at most once from time t to t+τ.
The hosting sites model 208 can include the following:
a model item 302 that denotes the set of hosting locations: 1, . . . , N;
a model item 304 that denotes hosting capacity per site: Hi(# instances), iεN;
(Note that the hosting capacity of each site can depend on the number of servers available at that site, network constraints such as topology, bisection bandwidth and over-subscription, application constraints such as CPU, memory, I/O footprint of instances (virtual machines (VMs) or grains/silos), business/legal constraints such as a user-centric data can only be hosted in a specific geographic location (e.g., European Union data cannot be hosted in the United States and vice versa), and, failure characteristics of hardware and software, among other factors.)
a model item 306 that denotes power available per energy source, from which derives (computed from variables) energy available per location at time t: Ei(t) (Watt-hour or Watt-second), iεN;
a model item 308 that denotes cost per energy source, from which derives (computed from variables) total energy cost per location at time t: ECi(t) ($), iεN;
a model item 310 that denotes network latency between locations: Li,k (seconds), i, kεN;
a model item 312 that denotes bandwidth capacity between locations: Bi,k (Bps), i, kεN;
a model item 314 that denotes bandwidth cost per byte between locations: BCi,k ($/Byte), i, kεN;
a model item 316 that denotes a set of available energy (renewable and non-renewable) sources: 1, . . . , |Q|
(Note that the number of intervals in the τ window is at most |Q|, but equality can be used by making some interval lengths to zero. The intuition is that an interval boundary is set when an energy source becomes unavailable (end time Ti,q) in the τ window. In each such interval, all parameters: power Pi,q, availability Ti,q and power pricing ECi,q are constant. Hence, the number of boundaries would be at most |Q|. Further, this setting helps in computing the length of these intervals a priori. The assumption here is that each power source is available from start time of τ window and will appear only once in that window.);
For source qεQ, predicted power available per location at time t: Pi,q(t) (Watt), iεN;
For source qεQ, predicted time until which power is available per location at time t: Ti,q(t), iαN (Note that the assumptions here are that the energy source q is available with constant power Pi,q(t) at cost ECi,q(t) from t until Ti,q+t. Assumptions such as fixed power, fixed price, and availability starting time t are removed later.);
For source qεQ, energy cost per Watt-hour per location at time t:
ECi,q(t) ($/Watt-hour), iεN; and
(Variable) For source qεQ, power used per location during each of m intervals in the time window τ: ui,q,m(t) (Watt), iεN, mε1, . . . |Q|;
a model item 318 (variable) that denotes the length of each of the m partitioning intervals of the time window τ: di,m(t) (seconds), mε1, . . . , |Q| (the interval τ is partitioned into |Q| (i.e., number of available energy sources) intervals and in each interval, the decision component 206 uses ui,q,m(t) power from source q at site i; Σm=1 |Q|di,m=τ;
a model item 320 that denotes temperature per site at time t: θi(t) (degrees F or C), iεN;
a model item 322 that denotes a temperature threshold of running each site: Θi(degrees F or C), iεN; and,
a model item 324 that denotes availability of hosting compute elements.
Note that the prediction times, Ti,q, power Pi,q, and power pricing ECi,q for qεQ, for different energy sources such as solar, tidal, wind, grid, backup, etc., can be learned (in offline and/or online manner) from past history of power availability and future event notifications using the learning component 202. Further, each of these terms are set to be the minimum of corresponding predicted time of itself (Ti,q) and time interval τ, since the framework looks ahead for the next interval τ time to minimize the cost function.
The look-ahead may not be done beyond τ (e.g., to support instance migrations with large data footprint) as most migrations are expected to be for interactive, light-weight instances that can be quickly transferred, whereas instances using large amounts of persistent state can be transferred offline, moved to one of the data replica locations (obviating the need to transfer hard state), be pinned to the original data location, or use an external highly-available, replicated storage service which does not require the transfer of persistent data.
Note that Ei(t) may be fixed over the time interval τ or it may vary due to dynamic pricing at fine time granularities (e.g., 5-min fluctuations in realtime or spot markets); in the latter case, compute Ei(t) based on weighing energy prices by the time duration the prices remain constant and averaged over the total time interval τ.
The terms for available energy Ei(t) and energy cost ECi(t) at time t are computed as follows (Note that the product of two variables, ui,q,m(t)*di,m(t), will make the optimization problem non-linear; one approach is to use a Lagrange multipliers method; however, the problem can be converted back to linear by determining the values of di,m(t) a priori before solving the optimization problem.):
E i(t)=Σm=1 |Q|(ΣqεQ u i,q,m(t))*d i,m(t) (1)
EC i(t)=Σm=1 |Q|(ΣqεQ u i,q,m(t)*EC i,q(t))*d i,m(t) (2)
The equations (1) and (2) denote the total energy available and the total energy cost at time t at hosting location i powered by different sources of energy. Note that the availability and pricing of a given energy source may change dramatically over a short period of time.
FIG. 4 illustrates the application instances model 210 that can be employed for the optimization framework. The model 210 can be hosted to include the following:
a model item 402 that denotes the set of application instances: 1, . . . , J;
a model item 404 that denotes the memory footprint size per application instance: sj (Bytes) ∀jεJ;
a model item 406 that denotes the number of network connections maintained by application instance: cj (number) ∀jεJ;
a model item 408 that denotes the power footprint (measured using tools such as Joulemeter) per application instance: pj (Watt) ∀jεJ;
a model item 410 that denotes the latency SLA (service level agreement) threshold for migrating an application instance between two locations: LatSLA (secs);
a model item 412 that denotes the CPU usage per application instance; and
a model item 414 that denotes application-level SLAs.
The sj term primarily includes soft state of an instance loaded in memory. For grains using hard state, either, (1) migrated grains (e.g., for interactive applications) will use an external replicated persistent store, (2) a replicated instance is brought on-line (by recreating state if needed) on the new location and then the original instance is shutdown, or (3) grains (e.g., MapReduce jobs) can be pinned to host locations storing hard state in order to leverage data locality. The sj and pj costs can be measured offline using profiling or learned in an online manner during the application execution; profiling may include both peak and average loads.
The objective function is to minimize the total energy bill for running application instances as well as the bandwidth and latency costs incurred in migrating the instances. The “knobs” available for this optimization are placement of application instances on different hosting locations. Note that some instances may be pinned to only execute at a particular location, for example, user data and application placement based on legal regulations in Europe or Asia, compatibility of hosting platform (e.g., operating system, VM environment) with applications, application priority, etc. This constraint can be modeled as setting constraints λi,j=0 (described below) for all i locations, where instance j cannot run.
Application placement (the pinning of application instances or operators to locations) can be encoded using a set of indicator variables λi,jε{0,1} ∀iεN, ∀jεJ, i.e.,
λ i , j = { 1 , if ⁢ ⁢ j ⁢ ⁢ is ⁢ ⁢ assigned ⁢ / ⁢ pinned ⁢ ⁢ to ⁢ ⁢ location ⁢ ⁢ i 0 , otherwise ( 3 )
Similarly, define ui,j to denote initial placement of application instances.
u i , j = { 1 , if ⁢ ⁢ j ⁢ ⁢ is ⁢ ⁢ currently ⁢ ⁢ assigned ⁢ / ⁢ pinned ⁢ ⁢ to ⁢ ⁢ location ⁢ ⁢ i 0 , otherwise
The solution of the problem computes the values of λi,j and ui,q,m(t), which in turn determine the migration (i.e., current placement ui,j≠optimal placement λi,j) and decisions to shutdown servers (e.g., when Ei(t) is below a threshold, or site i has zero instances running).
Application Instance Constraint. An application instance (running replicated instances is equivalent to naming the instances uniquely as individual instances) is constrained to only run at a single location across all hosting sites,
∀jεJ:Σ iεNλi,j=1 (4)
Hosting Capacity Constraint. Similarly, the number of instances that can be hosted at a single site is constrained to the site capacity,
∀iεN:Σ jεJλi,j ≦H i (5)
Note that the hosting term can include multiple factors such as load balancing, power provisioning, physical location, proximity to users, etc. (In the general case, CPU, memory, I/O (e.g., disk, network) footprint of each instance can be included in the hosting capacity constraint,
∀iεN:Σ jεJλi,j*footprint(cpuj,memj,I/Oj)≦H i
where Hi can be a resource vector comprising CPU, memory, and I/O dimensions considered individually to satisfy the capacity constraint in each dimension or as an aggregate.
Business/Legal Data Constraint. Business/legal constraints can dictate that data and applications accessing the data corresponding to users in one location cannot be placed in other geographic locations. This constraint can be modeled by setting the indicator variables for all instances j that cannot be located at locations i, to zero,
∀iεN,∀jεJ that cannot be hosted at loc i:λ i,j=0
Energy Capacity Constraint. Next, model the energy constraint of hosting application instances at a site (demand equals supply),
∀iεN:Σ jεJλi,j *p j *τ=E i(t) (7)
Migration Latency Constraint. Similarly, the latency constraint on migrating (migration is feasible if below the SLA latency threshold) an application instance can be modeled as:
∀jεJ,∀iεN,∀kεN,i≠k: μ i,j*λk,j*ƒ(s j ,c j ,i,k)≦Lat SLA (8)
The term (ui,j*λk,j) evaluates to one when instance j is moved from location i to k incurring the bandwidth and latency migration cost, and to zero, otherwise. The function ƒ to compute migration latency is described in terms of the amount of state to be migrated (memory footprint size of the instance), the number of network connections that need to be re-established on migration, and a fixed constant overhead. A simple linear function suffices,
f ⁡ ( s j , c j , i , k ) = η 1 * c j + η 2 * s j B i , k + η 3 ( 9 )
where η1 is a constant to denote time latency per connection re-establishment, η2 is a multiplicative constant to denote overhead time for network communication, and η3 is a fixed constant overhead of network migration of an application instance.
Note that the term sj/Bi,k is a first-order approximates latency which may increase if the network becomes congested due to many migrations or if the bandwidth capacity gets reduced. However, this term is a good approximation for two reasons. First, the inter-DC (datacenter) bandwidths are roughly of the order of 400+ Mbps, whereas live objects/applications are expected to be tens to hundreds of KBs. Thus, several thousands of objects can be migrated at any time. Further, the inter-DC bandwidth capacities are massively over-provisioned compared to the current usage. Second, the objective function includes a bandwidth term as a weighted cost to be minimized where the weight is based on amortized costs/cost-benefit analysis (e.g., pay bandwidth cost of migration if the new location can yield 10× power savings). Hence, it is expected that there will not be too many migrations unless power prices fluctuate significantly.
Note also that setting up a new connection may be significantly more expensive (e.g., about 10× CPU/memory resulting in high latency due to initialization overhead and queuing delay) than keeping an existing connection alive.
Inter-component Latency Constraint. Many cloud applications comprise multiple components that require communication with each other, for example, in a multi-tier web service, a user request is first routed to a front-end load balancer, then to a mid-tier application server and finally to a back-end database server. Therefore, component migration may increase inter-component delay resulting in increased user-perceived latencies. To reduce the impact, component migration is allowed, as feasible, if the inter-component delay is within a latency threshold ξ:
∀jεJ,∀iεN,∀kεN,i≠k: μ i,j*λk,j *L i,k≦ξ (10)
The latency threshold ξ may be modeled more precisely, for example, for application instances that require low-latency communications, the constraint can be modeled as follows (however, this constraint makes the problem non-linear):
∀j,j′ε{app−communication−pairs}:λi,jλk,j′ L i,k≦ξ
Business Competitiveness Constraint. In some cases, two companies C1, C2 competing with each other to provide similar set of cloud services may have a placement constraint of being hosted in a different physical location than any of their competitors. The intuition is that even if a datacenter location is unavailable, then the competitor service hosted in a different location will still be able to serve customers and generate revenue. This constraint can be modeled as:
∀iεN,∀jεC1,∀j′εC2:λi,j≠λi,j′ (11)
Hosting Site Temperature Constraint. The temperature constraint of running each datacenter location checks that if a hosting location is running at least one application instance, then its current temperature should be below its temperature threshold. This constraint can be modeled as follows:
∀iεN,∀jεJ: θ i(t)*λi,j≦Θi (12)
Bandwidth Cost. The bandwidth cost (in $) of migrating application instances (e.g., from location i to k) is modeled as:
BwCost = ∑ i ∈ N ⁢ ∑ k ∈ N , i ≠ k ⁢ BC i , k * ( ∑ j ∈ J ⁢ μ i , j * λ k , j * s j ) ( 13 )
Note that in order to remove the constraint (i≠k) in the above equation, the intra-datacenter costs BCi,i can be modeled as zero.
Latency Cost. Similarly, the latency cost (in $) of migrating application instances (e.g., from location i to k) can be modeled as the following:
LatCost = ∑ i ∈ N ⁢ ∑ k ∈ N , i ≠ k ⁢ ∑ j ∈ J ⁢ μ i , j ⁢ λ k , j ⁢ Z ⁡ ( s j , c j , i , k ) ( 14 )
Note that the latency can be modeled as a cost function to be minimized as below (e.g., datacenter and application operators may be willing to pay SLA penalties at times to save on operational costs) or as a constraint to be satisfied as in the alternative problem formulation described below.
The cost function Z to convert latency to a $ cost can be described in terms of SLA violation during application migration. A simple cost function can be defined as follows:
Z ⁡ ( s j , c j , i , k ) = { 0 if ⁢ ⁢ f ⁡ ( s j , c j , i , k ) ≤ Lat SLA ρ if ⁢ ⁢ ( Lat SLA < f ⁡ ( s j , c j , i , k ) ⩓ f ⁡ ( s j , c j , i , k ) ≤ 3 * Lat SLA ) ∞ otherwise ( 15 )
where ρ is a fixed value or a dynamic value (e.g., proportional to overshoot of LatSLA) denoting the SLA violation penalty.
Energy Cost. The energy cost of running all application instances across the hosting locations can be modeled as:
EnergyCost = ∑ i ∈ N ⁢ EC i ⁡ ( t ) = ∑ i ∈ N ⁢ ( ∑ m = 1  Q  ⁢ ( ∑ q ∈ Q ⁢ u i , q , m ⁡ ( t ) * EC i , q ⁡ ( t ) ) * d i , m ⁡ ( t ) ) ( 16 )
Finally, the energy, bandwidth, and latency costs can be combined into a single cost metric.
Cost($)=αEnergyCost+βBwCost+δLatCost (17)
Note that the weights α, β, γε[0, 1] are assigned to give relative importance to each cost component; for normalization α+β+γ=1.
Constraint on Variable Values. The constraints on power used per source q (ui,q,m(t)), and length of partitioning intervals (di,m) variables can be modeled as follows:
∀ m ∈ [ 1 ,  Q  ] : d i , m ⁡ ( t ) ≤ τ ⁢ ⁢ ∑ m = 1  Q  ⁢ d i , m = τ ⁢ ⁢ ∀ i ∈ N , ∀ q ∈ [ 1 ,  Q  ] , ∀ m ∈ [ 1 ,  Q  ] : u i , q , m ⁡ ( t ) ≤ P i , q ⁡ ( t ) ⁢ ⁢ ∀ i ∈ N , ∀ m ∈ [ 1 ,  Q  ] : ∑ q ∈ Q ⁢ u i , q , m ⁡ ( t ) = ∑ j ∈ J ⁢ λ i , j * p j ⁢ ⁢ ∀ i ∈ N : ∑ q ∈ Q ⁢ ∑ m = 1  Q  ⁢ u i , q , m ⁡ ( t ) * d i , m ⁡ ( t ) = ∑ j ∈ J ⁢ λ i , j * p j * τ ( 18 )
Note that supply≧demand constraints are satisfied on both energy and power at each time instant. In particular, for power, since the power availability does not change between any partition interval, it suffices to check at the boundaries,
∀iεN,∀mε[1,|Q|]:Σ qεQ u i,q,m(t)=ΣjεJλi,j *p j
Using these formulations, the objective function can be defined as a constrained Mixed Integer Linear Programming (MILP) of minimizing the total cost of running application instances across hosting datacenters.
Objective: MIN Cost($) (19)
subject to the constraints (3), (4), (5), (6), (7), (10), (11), (12), (18) which is solved using a standard ILP (integer linear programming) solver on the above formulation.
Computing di,m(t): To make the optimization problem linear, the optimal setting of the length of partitioning intervals can be pre-computed: di,m(t); t is omitted from representation unless otherwise specified.
The intuition is at each hosting site i, to first sort into a list the end times Ti,q, until which power from source q is available,
T i,1 ′≦T i,2 ′≦ . . . ≦T i,Q′(mε[1,|Q|])
Next, split the time window (interval) τ into |Q| partition intervals with the boundaries set by the T′i,q. FIG. 5 illustrates a sorted sequence of power sources based on end time of source power availability. Note that all fluctuations of power availability only occur at the boundaries of these intervals. Then, set the lengths of these intervals based on the boundaries. FIG. 6 illustrates an algorithm that computes optimum partition intervals.
Alternative Problem Formulation. A related variant of the problem is to minimize the total cost of running all application instances across hosting datacenters subject to service level agreement (SLA) as the upper bound on migration latency of an instance between two hosting locations (e.g., the upper bound on migration latency LatSLA is at most 500 ms),
Objective: MIN Cost($)=αEnergyCost+βBwCost (20)
subject to the constraints (3), (4), (5), (6), (7), (10), (11), (12), (18) and the latency constraint (8) that migrating an application instance is feasible if it is below an SLA threshold,
∀jεJ,iεN,kεN,i≠k: μ i,j*λk,j*ƒ(s j ,c j ,i,k)≦Lat SLA (21)
which is solved using a standard ILP solver on the above formulation.
Following is an example implementation.
Input/Output: The inputs employed are defined in the system model for both hosting locations and application instances. The output is values of λi,jλiεN, jεJ denoting optimal placement of application instances across locations, and ui,q,m∀iεN, jεJ, qεQ, mε[1, |Q|] denoting the optimal power usage from each energy source during each of the |Q| partition intervals. These decisions are translated into migration and shutdown servers as previously described.
Initial Placement: The current placement of instances can be initialized to locations before running the optimization framework, that is, set ui,j, ∀iεN, jεJ.
The periodic time interval τ for running the controller (application management component 122) can be set such that (1) the system has sufficient time (not too short) to transition from one stable state to another (all migration of instances and state transitions are completed before again optimizing the placement) and (2) the cost function has not significantly changed (not too long) that most placement decisions soon become suboptimal. Ideally, interval τ can be set so that dynamic power pricing only incurs a modest cost increase over the optimal cost, before running the controller again.
Dynamic Power Fluctuation. Since the power availability of any energy source, such as grid and solar, may change dramatically over a short period of time, the algorithm recalculates the energy terms for the affected locations and if the energy available is insufficient (violates equation (7) to run instances hosted at that location until the next time to run the optimization controller), either the global optimization can be run at that time for all the datacenter locations, or simply the requisite number of instances moved from the affected locations to under-utilized locations while meeting the constraints on hosting capacity, temperature, and available energy at the new locations as well as the affected locations (until energy needed falls below available capacity).
Dynamic Temperature Fluctuation. Similarly, if the temperature at a hosting site risks (or as predicted) overshooting its threshold, the algorithm can follow the above procedure by either proactively re-running the global optimization, or migrating instances from affected locations to other locations while meeting the constraints on hosting capacity, temperature, and available energy at the new locations.
Dynamic Power Pricing Fluctuation. In case the energy prices vary significantly over a short period of time, the global optimization can be re-run or the affected locations ranked by decreasing cost of available energy and migrate instances in the following manner: from locations with highest energy cost to locations with least energy cost, while meeting the constraints on hosting capacity, temperature, and available energy at the new locations.
Bandwidth cost versus energy cost savings. To balance the trade-off between bandwidth cost and energy savings, a cost-benefit approach can be employed where an instance is only migrated when its bandwidth cost can be amortized over n execution periods of savings in the energy cost. The weighing constants α, β can be set accordingly.
Integrating caching with application execution. To maximize availability under limited capacity, application architects and cloud operators may integrate caching with application execution by running a small number of cache servers at places with expensive power, and (relatively) large number of application servers at locations with cheap power.
Speculative prefetching of bulk data or background replication. To support dynamic application migration, persistent user data and application state may be speculatively transferred to potential hosting locations as a background service.
Minimizing impact on user interactiveness. To reduce the impact of user-perceived latency due to multiple migrations of the interacting application, a count of the number of times an instance has been migrated may be maintained and input as a constraint to limit migrations of instances to below that threshold in a given time period.
Following is a description of extensions to the disclosed optimization architecture that can be employed.
Relaxing the assumptions. The assumptions, fixed power and fixed price during time window τ, and power availability from each source starting time t, can be relaxed. FIG. 7 illustrates that each power source may have different start and end times of availability for power pricing. Partitioning of the time window τ can be based on start time and end time of power availability. It is between these times during which the associated power pricing remains fixed and power from a source may be available (non-continuous) over different time intervals during time window τ.
In this model, a power source is represented as a four tuple: (start time, end time, watt power intensity, and price). Availability of a single power source during different time intervals can be treated as different (virtual) power sources. Then, partition the time period τ by marking boundaries at start and end times for each virtual power source, based on time instance when the value of watt power intensity of an energy source gets changed, based on time instance when the cost of watt power intensity of an energy source gets changed, and then compute the length of partitioning intervals as done previously.
Look ahead to multiple time steps. The look ahead for the optimization can be extended from a single step to multiple (say l) look-ahead steps by increasing the interval τ. Note that the dynamic pricing and availability of energy is already accounted in the formulation of a single look-ahead step.
Reducing impact on user latency. Dynamic application placement to distant geographic locations may result in increased user-perceived latency or turn-around time of batch jobs. One approach is to input a constraint that application migration may only be allowed to a subset of locations (e.g., based on geographic zone or availability zone) so that the impact on user latency is small. Another solution is to incorporate latency constraints between user and applications, and between application components (e.g., grains) in the optimization framework.
Dynamic bandwidth pricing. Dynamic pricing of wide-area network bandwidth cost can also be included in the optimization framework.
Delay-tolerant job execution. For background services such as web crawl jobs, or applications that can tolerate delayed execution (e.g., temporarily suspended and resumed at a later time) the jobs can be scheduled (start time, end time, resume time) to run when operating costs are minimum (e.g., low demand time periods such as evenings across different time zones) or executing at a limited/reduced scale at different times to reduce operating costs.
Regarding delay-tolerant execution, an alternative embodiment is to include dynamic pricing of running application instances when customers have flexibility in when the associated applications can run, for example, in cloud hosting services. A benefit of dynamic pricing is that large pools of resources can be made available to applications when needed.
Fine-grained allocation model. To support fine-grained resource allocation in utility computing infrastructures (e.g., allow users to allocate standard, high memory, or high CPU instances, or small, medium (2× CPU), large (4× CPU), and extra-large (8× CPU)), the hosting capacity Hi of location i can be specified in these units and constraints added on instances hosted on a given resource granularity can only be migrated to the same resource granularity (or higher) at a different location with spare capacity.
FIG. 8 illustrates an intelligent power control system 800 for hosting sites. The hosting site 104 includes servers 802 (as computing elements), a data collector 804 that collects data related to the servers 802, and a sensor collector 806 that reads hosting site parameters such as temperature, power consumption, and so on. The output of the data collector 804 and sensor collector 806 is passed as server instrumentation data to a data processing engine 808.
The functionality described herein and associated with the hosting site 104 also applies to a second hosting site 818, that includes servers 820 (as computing elements), a data collector 822 that collects data related to the servers 820, and a sensor collector 824 that reads hosting site parameters such as temperature, power consumption, and so on. The output of the data collector 822 and sensor collector 824 is passed as server instrumentation data to the data processing engine 808. The second hosting site 818 has an associated simulator 826 that performs similarly to the simulator 204.
The data processing engine 808 sends raw data along {circle around (1)} to a datacenter UI 810 for use viewing and interaction. The data processing engine 808 also sends data event notifications along {circle around (2)} to the decision component 206 (a decision engine).
The decision component 206 sends decision output along {circle around (3)} to the datacenter UI 810 for presentation. The decision component 206 also sends power state change decisions and responses along {circle around (4)} to a power state actuator 814. The power state actuator 814 sends power state change execution along {circle around (5)} to the hosting site 104 and along {circle around (5)} to the second hosting site 818. The decision component 206 also sends VM or application grain (instance) migration decisions and responses along {circle around (6)} to a management API 816. The management API 816 sends grain migration execution along {circle around (7)} to the hosting site 104 and along {circle around (7)}* to the second hosting site 818. The datacenter UI 810 interfaces to the simulators (204 and 826) to make adjustments to power variables at the simulators (204 and 826).
The data processing engine 808 receives all the data in the system 800, and then aggregates and correlates the data before passing it to the decision component 206. The data collector 804, the datacenter sensor collector 806 and the simulator (e.g., power and cost) 204 all pass data points to the data processing engine 808. Similarly, the data collector 822, the datacenter sensor collector 824 and the simulator (e.g., power and cost) 826 all pass data points to the data processing engine 808.
The following description details the how the data is passed to the data processing engine 808, what data will be passed and the queries, which will be to aggregate and correlate the data and how the data will be passed to the decision component 206 (an intelligent power controller). The sensor collectors (806 and 824) collect server power data (for each of the servers being used) and total power (for each unit being used).
The decision component 206 is the decision-making entity that takes the data coming from the data processing engine 808, reasons over the data and makes a decision on what each server's power state should be. The decision component 206 contains the following configuration data: base memory usage of the management API runtime, number of active servers, decision interval (in milliseconds), UI update interval (in milliseconds), file location where to hold data shared with UI application, base power consumption of the network of datacenters, the datacenters being controlled, the servers in each datacenter being controlled, communications channel for a directory service, TCP channel information for the power state actuator 814, and the TCP channel information for the datacenter UI 810.
The simulator 204 (and simulator 826) can collect historical parameters and realtime energy parameters (data points), not limited to: power, the available solar power (watt power intensity and duration), used solar power for last time interval (watt power intensity and duration), available grid power (watt power intensity and duration), used grid power for last time interval (watt power intensity and duration), available backup power (watt power intensity and duration), used backup for last time interval (watt power intensity and duration), unit cost, power cost ($/kWh) for different sources, and network bandwidth cost ($/kB) between it and one or more other datacenters, LatSLA, for example. The basic functionality of the simulator 204 (and simulator 826) is to calculate each of the above data points at the configured time intervals and send the data to the data processing engine 808. The simulator 204 (and simulator 826) then sends outputs for every configured time interval for each of the above data points.
The following are respective inputs into the simulators (204 and 826): total power drawn by the datacenter (arrives in regular intervals); weather (a discrete set of weather states such as cloudy, partially cloudy, and not cloudy; time of day (a discrete set of times of day such as dawn, morning, noon, afternoon, dusk, night; grid power availability (the percent of the max grid power that is available at any time interval); pricing historical data and models of different power sources, and input location for weather, time of day and grid power availability. These inputs can come either from a file or over the TCP channel. This method allows for changing between receiving these inputs through the TCP channel versus reading from a simulation file. If there is a change from the file to the TCP channel and then back to the file then the file will pickup exactly as it left off. If a simulation file is not passed in the command line arguments then this command is ignored.
The power state actuator 814 provides the execution engine for the power state decisions on datacenter servers (802 and 820) by the decision component 206. The power state actuator 814 provides the following functionality: takes power state decisions and executes power state changes associated with these decisions; manages any retries necessary to move a node to the correct power state; and, exposes the current power state for servers managed by the power state actuator.
In addition to the power state actuator 814, a lightweight power state agent runs on each managed server that allows the power state actuator 814 to remotely turn off, put to sleep, and to hibernate, a server. The power state actuator 814 also performs verification to ensure the power state of the server(s) being controlled.
FIG. 9 illustrates a computer-implemented optimization method in accordance with the disclosed architecture. At 900, multiple hosting sites are received that run application instances to process workload. At 902, data related to realtime energy parameters associated with each hosting site and energy sources for the hosting sites is tracked. At 904, data related to realtime computing resources associated with available computing elements of the hosting sites is tracked. At 906, the data is reasoned over to output optimization decisions. At 908, application instances are automatically relocating between the hosting sites based on the optimization decisions. Although illustrated as a single iteration, in practice, the flow can return to 900 from 908 to represent continuous processing.
FIG. 10 illustrates further aspects of the method of FIG. 9. Each of the blocks represents a single step that can be added to the method represented by the flow chart of FIG. 9. At 1000, the data further includes service level agreements that are reasoned over and application instances are relocated between the hosting sites based on the service level agreements. At 1002, the data further includes cost associated with network bandwidth and network latencies between the hosting sites that are reasoned over. At 1004, the optimization decisions output are based on renewable and non-renewable energy sources. At 1006, the optimization decisions are output based on dynamic variations in energy prices temporarily and geographically. At 1008, the availability and cost of energy sources at a specific time is predicted based on learned information that includes historical energy availability data and future event notifications. At 1010, the availability of hosting compute elements at different sites is predicted.
FIG. 11 illustrates further aspects of the method of FIG. 9. At 1100, application instances are migrated between datacenters based on integration of optimization decisions with a language runtime mechanism or virtual machine migration. At 1102, application-state and user-data are replicated at different hosting sites based on request redirection of incoming user requests. At 1104, execution of application instances is managed according to different times or at a reduced scale at different times based on at least one of delayed execution or temporary suspension and resumption of execution.
Referring now to FIG. 12, there is illustrated a block diagram of a computing system 1200 that executes the optimization framework in accordance with the disclosed architecture. In order to provide additional context for various aspects thereof, FIG. 12 and the following description are intended to provide a brief, general description of the suitable computing system 1200 in which the various aspects can be implemented. While the description above is in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that a novel embodiment also can be implemented in combination with other program modules and/or as a combination of hardware and software.
The one or more application programs 1222, other program modules 1224, and program data 1226 can include the entities and components of the system 100 of FIG. 1, the entities and components of the system 200 of FIG. 2, the entities of the hosting site model 208 of FIG. 3, the entities of the application instances model 210 of FIG. 4, the listing and interval computing represented by the charts of FIGS. 5 and 7, the algorithm 600 of FIG. 6, the system 800 of FIG. 8, and the methods represented by the flowcharts of FIGS. 9-11, for example.
Referring now to FIG. 13, there is illustrated a schematic block diagram of a computing environment 1300 that performs optimization in accordance with the disclosed architecture. The environment 1300 includes one or more client(s) 1302. The client(s) 1302 can be hardware and/or software (e.g., threads, processes, computing devices). The client(s) 1302 can house cookie(s) and/or associated contextual information, for example.
for hosting sites that run application instances to process workload:
tracking realtime energy parameters associated with the hosting sites and energy resources for the hosting sites, and
tracking realtime availability of computing resources associated with the hosting sites;
constructing a framework including:
a hosting sites model that associates the realtime energy parameters and the realtime availability of the computing resources with operation of the hosting sites;
an application instances model that associates the application instances with the computing resources and the energy resources, wherein the application instances model includes power footprints and memory footprints for individual application instances;
solving the framework to output decisions based on the hosting sites model and the application instances model, wherein the solving considers the realtime energy parameters, the realtime availability of the computing resources, and the power footprints and the memory footprints for the individual application instances; and
automatically migrating at least some of the application instances between the hosting sites based on the decisions from the solving the framework.
2. The method of claim 1, wherein the solving the framework further considers service level agreements associated with the application instances.
3. The method of claim 1, wherein the solving the framework further considers costs of the automatically migrating, other costs associated with network bandwidth, and network latencies between the hosting sites.
4. The method of claim 1, wherein the realtime energy parameters include costs and availability of the energy resources, and the energy resources include both renewable and non-renewable energy sources.
5. The method of claim 1, wherein the realtime energy parameters include both temporal and geographical dynamic variations in energy prices of the energy resources.
simulating future availability and cost of the energy resources based on historical energy pricing and availability data,
wherein the solving the framework further considers the simulated future availability and cost of the energy resources.
7. The method of claim 6, wherein the energy resources include both renewable and non-renewable energy resources.
redirecting incoming user requests to selected hosting sites based on the decisions from the solving the framework.
managing power states of the hosting sites based on the decisions.
a processor and a storage device;
a data collector stored on the storage device and executable by the processor, the data collector configured to collect computing parameters related to a plurality of datacenter servers, the computing parameters reflecting availability of computing elements;
a simulator stored on the storage device and executable by the processor, the simulator configured to:
collect historical parameters and realtime energy parameters of the datacenter servers reflecting availability of renewable and non-renewable energy sources and dynamic variations in energy prices, and
calculate data points relating to the historical parameters and the realtime energy parameters for designated time intervals;
a data processing engine stored on the storage device and executable by the processor, the data processing engine configured to correlate the computing parameters and the data points for the designated time intervals from the data collector and the simulator to produce correlated data related to the designated time intervals;
a decision component stored on the storage device and executable by the processor, the decision component configured to receive the correlated data from the data processing engine and to make decisions based on a framework including a hosting site model, an application instance model, and a location energy model, the decisions related to power state changes of the hosting sites associated with the designated time intervals; and
a power state actuator stored on the storage device and executable by the processor, the power state actuator configured to receive the decisions from the decision component and to manage power state changes associated with the received decisions within the designated time intervals.
11. The power control system of claim 10, wherein the simulator is further configured to collect the realtime energy parameters by computing the availability of an individual energy source based on predicted power available from the individual energy source for a specific datacenter server for a specific time.
12. The power control system of claim 10, wherein the decisions made by the decision component are based on reducing monetary costs of operating the plurality of datacenter servers.
13. The power control system of claim 10, wherein the decisions made by the decision component are based on reducing impact on user interactiveness.
14. The power control system of claim 10, wherein the decisions made by the decision component are based on increasing long term availability of the computing elements.
15. The power control system of claim 10, wherein the decisions made by the decision component are based on increasing use of the renewable energy sources.
16. The power control system of claim 10, wherein the decisions made by the decision component are based on reducing carbon footprint and pollutant emission due to electricity used in operation of the plurality of datacenter servers.
17. The power control system of claim 10, wherein the decisions made by the decision component are based on balancing bandwidth cost and energy cost.
18. The power control system of claim 10, wherein the application instance model is based on information related to at least one of processor usage and memory footprint size of an individual application instance, number of network connections maintained by the individual application instance, power footprint of the individual application instance, service level agreements on hosting of the individual application instance, or latency threshold for migrating the individual application instance between two sites.
receiving data related to servers, the data reflecting historical parameters of computing elements and energy sources, realtime energy parameters for the energy sources, and availability of the computing elements, wherein the energy sources comprise both renewable and non-renewable energy sources;
accessing a framework that includes:
a hosting site model associated with operations of the servers,
an application instances model of application instances that are run on the servers, and
a location energy model that estimates power consumption at the servers;
entering the received data into the framework to make server power state decisions based on the received data; and
automatically relocating individual application instances between the servers based on the server power state decisions.
20. The method of claim 19, the data further reflecting availability of an individual energy source, the availability being based on predicted power available from the individual energy source to a specific server for a specific time.
21. The method of claim 19, wherein the server power state decisions are based on at least one of reducing monetary costs for running the application instances, reducing impact on user interactiveness, increasing long term availability of the computing elements, increasing use of the renewable energy sources, reducing carbon footprint and pollutant emission due to the power consumption at the servers, or balancing bandwidth cost and energy cost.
US12/779,059 2010-05-13 2010-05-13 Dynamic application placement based on cost and availability of energy in datacenters Active 2032-07-04 US9207993B2 (en)
US12/779,059 US9207993B2 (en) 2010-05-13 2010-05-13 Dynamic application placement based on cost and availability of energy in datacenters
US20110282982A1 US20110282982A1 (en) 2011-11-17
US9207993B2 true US9207993B2 (en) 2015-12-08
ID=44912714
US12/779,059 Active 2032-07-04 US9207993B2 (en) 2010-05-13 2010-05-13 Dynamic application placement based on cost and availability of energy in datacenters
US (1) US9207993B2 (en)
US20170026305A1 (en) * 2015-07-23 2017-01-26 Schneider Electric It Corporation System to place virtual machines onto servers based upon backup runtime constraints
WO2004049157A2 (en) 2002-11-27 2004-06-10 International Business Machines Corporation Automated power control policies for distributed server pools based on application-specific computational requirements
US20060080213A1 (en) 2004-09-02 2006-04-13 Toyoake Flower Co., Ltd. Auction methods and system for perishable products
US20060277109A1 (en) 2005-06-06 2006-12-07 International Business Machines Corporation Dynamic pricing based on shelf life of perishable goods
CN101096942A (en) 2006-06-30 2008-01-02 通用电气公司 Wind energy system and method of operation thereof
US20080072090A1 (en) 2001-09-19 2008-03-20 Dell Products L.P. System and Method for Strategic Power Supply Sequencing in a Computer System with Multiple Processing Resources and Multiple Power Supplies
US20080313101A1 (en) 2007-06-15 2008-12-18 Helander Mary E Method and apparatus for capacity- and value-based pricing model for professional services
US20080316938A1 (en) 2006-03-06 2008-12-25 Huawei Technologies Co., Ltd. Method, system and device for allocating network resources in communication network
CN101430596A (en) 2007-11-07 2009-05-13 国际商业机器公司 Method and system for power management
US20090279444A1 (en) 2008-05-12 2009-11-12 Nortel Networks Limited Method and apparatus for discovering, negotiating, and provisioning end-to-end SLAs between multiple service provider domains
KR20090120704A (en) 2008-05-20 2009-11-25 엘에스전선 주식회사 Repeater power supplying device using wind force and solar heat
US20090295167A1 (en) 2007-02-26 2009-12-03 Jimmy Clidaras Water-based data center
US20100333105A1 (en) 2009-06-26 2010-12-30 Microsoft Corporation Precomputation for data center load balancing
US20110138708A1 (en) 2009-12-11 2011-06-16 Enia Architectes Superimposed Computer Room Building and Process for Cooling this Building
US20110205949A1 (en) 2008-08-27 2011-08-25 Telefonaktiebolaget L M Ericsson (Publ) Routing Mechanism for Distributed Hash Table Based Overlay Networks
US20110240265A1 (en) 2010-04-06 2011-10-06 American Power Conversion Corporation Container based data center solutions
US20120130554A1 (en) 2010-11-22 2012-05-24 Microsoft Corporation Dynamically placing computing jobs
US20120330711A1 (en) 2011-06-27 2012-12-27 Microsoft Corporation Resource management for cloud computing platforms
2010-05-13 US US12/779,059 patent/US9207993B2/en active Active
JP2006508445A (en) 2002-11-27 2006-03-09 インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｓｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ Automatic power control policy based on the application inherent redundancy characteristics
US20110161526A1 (en) 2008-05-12 2011-06-30 Ravishankar Ravindran Method and Apparatus for Discovering, Negotiating, and Provisioning End-to-End SLAS Between Multiple Service Provider Domains
US20140365402A1 (en) 2010-10-28 2014-12-11 Microsoft Corporation Data center system that accommodates episodic computation
"About Vibes Interactive"; Vibes Interactive LLC; http://www.vibesinteractive. com/about-us.html; 2010; 2 pages.
"Amendment after Notice of Allowance (Rule 312)," Filed on Jan. 22, 2015, from U.S. Appl. No. 12/951,977, 10 pages.
"Amendment," From U.S. Appl. No. 14/464,650, Filed Date: Aug. 20, 2014, 8 pages.
"ASG-Workload Optimization Suite for Comprehensive Workload Analysis and Forecasting"; Allen Systems Group, Inc.; 2009; 1 page.
"Carbon Dioxide Emissions from the Generation of Electric Power in the United States"; http://www.eia.gov/cneaf/electricity/p./co2-report/co2report.html; Jul. 2000.
"Consultants: How and Why Pay Per Performance Works (Part 2-Continuation from Last Post)"; Weblog posting published Apr. 4, 2010; http://bizrelationships.wordpress.com/2010/04/19/consultants-how-and-why-pay-perperformance-works-part-2-continuation-from-last-post ; 5 pages.
"Data Center Efficiency in the Scalable Enterprise"; reprinted from Dell Power Solutions; Dell Inc.; Feb. 2007; 7 pages.
"Decision on Rexamination," Mailed on Dec. 16, 2014, from Chinese Patent Application No. 201080009556.7, 3 pages.
"European Search Report," From European Patent Application No. 12803686.0, Mailed Date: Oct. 28, 2014, 3 pages.
"Examination Report," Mailed on Oct. 17, 2014, from European Patent Application No. 10744134.7, 7 pages.
"Final Office Action," From Chinese Patent Application No. 201080009556.7, Mailed Date: Jul. 31, 2014, 7 pages.
"First Office Action," Mailed on Sep. 16, 2014, from Chinese Patent Application No. 201180024779.5, pages.
"International Search Report and Written Opinion" for PCT Patent Application No. PCT/US2010/023253, Mailed Sep. 30, 2010, 9 pages.
"International Search Report," From PCT Patent Application No. PCT/US2012/042110; Mailed Jan. 31, 2013, 9 Pages.
"International Search Report," PCT Patent Application No. PCT/US2011/036364; Mailed Dec. 23, 2011; 8 pages.
"Leading the Evolution of the Data Center," Brocade Communications Systems, Inc., San Jose, CA; 2008; 8 pages.
"Levin, Y. et al.; "Dynamic Pricing in the Presence of Strategic Consumers and Oligopolistic Competition"; Management Science; vol. 55, No. 1; Jan. 2009; pp. 32-46."
"Managing Energy Efficiency in the Data Center," EMC Corporation, Hopkinton, MA; 2008; 6 pages.
"Non-final Office Action," From U.S. Appl. No. 12/781,445, Mailed Date: Sep. 23, 2014, 12 pages.
"Non-final Office Action," From U.S. Appl. No. 12/973,399, Mailed Date: Jul. 3, 2014, 26 pages.
"Notice of Allowance," Mailed on Dec. 24, 2014, from U.S. Appl. No. 12/951,977, 7 pages.
"Notice of Allowance," Mailed on Nov. 11, 2014, from Japanese Patent Application No. 2011-551110, pages.
"Notice on the Third Office Action," Mailed on Dec. 31, 2014, from Chinese Patent Application No. 201080009556.7, 7 pages.
"Office Action," From European Patent Application No. 12803686.0, Mailed Date: Dec. 1, 2104, 4 pages.
"PlateSpin Recon"; retrieved on May 11, 2009 at http://www.novell.com/products/recon/ consolidation-ratios.html, Novell, Inc.; 2009; 2 pages.
"Response to Final Office Action," Mailed on Nov. 14, 2014, from Chinese Patent Application No. 2010800095561, pages.
"Response to Non-final Office Action," From U.S. Appl. No. 13/169,890, Filed Feb. 17, 2015, 19 pages.
"Response to Non-final Office Action," From U.S. Appl. No. 13/169,923, Filed Feb. 9, 2015, 19 pages.
"Response to Office Action," From European Patent Application No. 10744134.7, Filed on Feb. 16, 2015, 11 pages.
"Response to Second Office Action," From Chinese Patent Application No. 201080009556.7, Filed Date: Apr. 30, 2014, 16 pages.
"Response to Second Office Action," From Japanese Patent Application No. 2011551110, Filed Date: Jun. 10, 2014, 16 pages.
"Response to Third Office Action," From Chinese Patent Application No. 201080009556.7, Filed Date: Mar. 4, 2015, 15 pages.
"Second Office Action," From Chinese Patent Application No. 201080009556.7, Mailed Date: Feb. 21, 2014, 10 pages.
"Second Office Action," From Japanese Patent Application No. 2011551110, Mailed Date: Mar. 11, 2014, 11 pages.
"Sensible Cloud: What is an SLA?"; http://www.sensiblecloud.com/2010/11/what-is-slai/; Retrieved Mar. 24, 2011; 8 pages.
"Solar Chimney", Wikipedia, Published on or before Feb. 27, 2005, 5 pages.
"Solar updraft tower", Wikipedia, Published on or before Aug. 30, 2006, 6 pages.
"Tout+Interactive"; Latest Publication Mar. 22, 2011; Earlier weblog entries published on Mar. 19, 2011; Mar. 18, 2011; Mar. 7, 2011; Mar. 6, 2011; and Nov. 29, 2010; http://toutinteractive.com/; 8 pages.
"Zeus Launches Data Center Load Balancing Solution"; retrieved at http://www. continuitycentral.com/news03343.htm, Continuity Central; Jun. 28, 2007; 1 page.
"Zhang, Q.; "Cloud Computing: State-of-the-Art and Research Challenges"; Journal of Internet Services and Applications; vol. 1 (1); May 2010; Published online Apr. 20, 2010;pp. 7-18."
After Final Pilot 2.0 Request and Response Filed Oct. 30, 2015 to the Final Office Action Mailed Jun. 5, 2015 From U.S. Appl. No. 13/169,890, 17 pages.
Amendment filed Jan. 30, 2013 from Chinese Patent Application No. 2011-551110, 7 pages.
Applicant Initiated Interview Summary mailed Aug. 12, 2015, from U.S. Appl. No. 13/169,923, 3 pages.
Applicant Initiated Interview Summary mailed Sep. 2, 2015 from U.S. Appl. No. 13/169,890, 3 pages.
Ardagna, et al., "SLA Based Profit Optimization in Multi-tier Systems", Retrieved at << http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1565967 >>, NCA, Proceedings of the Fourth IEEE International Symposium on Network Computing and Applications, Jul. 27-29, 2005.
Ardagna, et al., "SLA Based Profit Optimization in Multi-tier Systems", Retrieved at >, NCA, Proceedings of the Fourth IEEE International Symposium on Network Computing and Applications, Jul. 27-29, 2005.
Argument/Amendment filed Oct. 2, 2013 from Chinese Patent Application No. 2011-551110, 11 pages.
Bansal et al., "A Primal-Dual Randomized Algorithm for Weighted Paging", Retrieved at <<http://www.cs.technion.ac.il~nivb/papers/weighted-paging.pdf, 2007, 11 pages.
Bansal et al., "A Primal-Dual Randomized Algorithm for Weighted Paging", Retrieved at <<http://www.cs.technion.ac.il˜nivb/papers/weighted-paging.pdf, 2007, 11 pages.
Bansal et al., "Metrical Task Systems and the k-Server Problem on HSTs", Proceedings of the 37th International Colloquium on Automata, Languages and Programming, Jul. 6-10, 2010, 12 pages.
Bansal et al., "Randomized Competitive Algorithms for Generalized Caching", Retrieved at <<http://research.microsoft.com/en-us/um/people/nivbuchb/papers/web-caching/web-caching.pdf>>, May 17-20, 2008, 10 pages.
Bansal et al., "Randomized Competitive Algorithms for Generalized Caching", Retrieved at >, May 17-20, 2008, 10 pages.
Bansal et al.,"Towards the Randomized K-Server Conjecture: A Primal-Dual Approach", Retrieved at <<http://domino.research.ibm.com/comm/research-people.nsf/pages/nikhil.pubs.html/$File/extended-primal-dual.pdf>>, Jan. 17, 2010, 24 pages.
Bartal, et al., "A Polylog (N)-Competitive Algorithm for Metrical Task Systems", Retrieved at <<http://www.google.co.in/url?sa=t&source=web&cd=5&ved=0CC4QFjAE&url=http%3A%2F%2Fcitseerxist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.39.3857%26rep%3Drep1%26type%3Dpdf&rct=j&q=A%20polygon(n)-competitive%20algorithm%20for%20metrical%20task%20systems&ei=BIJaTPSsOYH-4AbO--2EAg&usg=AFQCNEdMbYyyaLhQ01jYc-1VAhdalpAeA>>, 1997, 9 pages.
Battles et al., "Reducing Data Center Power Consumption through Efficient Storage", Network Applianc, Inc., Sunnyvale, CA, WP-7010-0207, Feb. 2007, 9 pages.
Belady, Chrstian L., "In the data center, power and cooling costs more than the it equipment it supports", Retreived at <<http://www.electronics-cooling.com/2007/02/in-the-data-center-power-and-cooling-costs-more-than-the- it-equipment-it-supports/>>, Feb. 1, 2007, 6 pages.
Beloglazov et al., "SLA Based Profit Optimization in Multi-Tier Systems", in Proceedings of the 4th IEEE International Symposium on Network Computing and Applications, Jul. 27-29, 2005, 4 pages.
Bianchini, "Power and Energy Management for Server Systems", IEEE Computer, vol. 37, No. 11, Nov. 2004, 11 pages.
Bodik et al., "Response-Time Modeling for Resource Allocation and Energy-Informed SLA's," Proc. Workshop on Statistical Learning Techniques for Solving Systems Problems (MS Sys'07), Dec. 8, 2007, identified at <<http://homepages.inf.ed.ac.uk/csutton/publications/sysml.pdf>> on Jul. 27, 2015, 3 pages.
Bonvin, N. et al., "Autonomic SLA-driven Provisioning for Cloud Applications", 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May 23-26, 2011, Published online Feb. 2011, 10 pages.
Borenstein et al., "Dynamic Pricing, Advanced Metering, and Demand Response in Electricity Markets", University of California Energy Institute, 2002, 102 pages.
Borgs et al., "Optimal Mechanisms for Perishable Goods With Variable Supply and Demand", Aug. 11, 2010, 8 pages.
Borgs, "Pricing Mechanisms for Perishable Time-Varying Resources", U.S. Appl. No. 12/903,227, Filed: Oct. 13, 2010.
Borodin et al., "An Optimal On-Line Algorithm for Metrical Task System", Retrieved at <<http://www.google.co.in/url?sa=t&source=web&cd=3&ved=0CCMQFjAC&url=http%3A%2A%2Fcitseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.74.7160%26rep%3Drep1%26type%Dpdf&rct=j&q=an%20optimal%20on-line%20algorithm%20for%20metrical%20task%20system&ei=4NaTMuECMrS4galtq3-Aq&usg=AFQjCNFYrRVoLBy9GJ5Cfpc8vF342tpNXg>>, Sep. 1992, 19 pages.
Borodin et al., "Online Computation and Comptetive Analysis", Retrieved at <<http://www.google.co.in/url?sa=t&source=web&cd=5&ved=0CC4QFjAE&url=http%3A%2F%2Fwww.gbv.de%2Fdrns%2Fgoettingen%2F236832409.pdf&rct=j&q=Online%20computation%20and%20competitive%20analysis&ei=r4NaTIzROsij4Abe81TxAQ&usg=AFQjCEFoECsmmMPoASVdvcymLwjcVO4qg>>, 1998, 6 pages.
Buchbinder et al., "Online Job-Migration for Reducing the Electricity Bill in the Cloud", Lecture Notes in Computer Science, Networking 2011 Proceedings of the 10th International IFIP TC 6 Conference on Networking, vol. 6640, Part I, 2011, pp. 172-185.
Buchbinder et al., "The Design of Competitive Online Algorithms Via a Primaldual Approach", Retrieved at <<http://research.microsoft.com/en-us/um/people/nivbuchbpapers/pd-survey.pdf.>> 2007, 174 pages.
Buchbinder et al., "The Design of Competitive Online Algorithms Via a Primaldual Approach", Retrieved at > 2007, 174 pages.
Burger, "Wind-Powered Data Center", U.S. Appl. No. 12/781,445, Filed May 17, 2010, 26 pages.
Butler, Declan, "France Unvels Carbon Tax", Retrieved at http://www.nature.com/news/2009/090911/full/news2009.905.html, Jan. 29, 2010, 2 pages.
Buyya, R. et al.; "Cloudbus Toolkit for Market-Oriented Cloud Computing"; Proceedings of the 1st International Conference on Cloud Computing; Oct. 11, 2009; pp. 24-44.
Carbon Tax Center, Retrieved at <<www.carbontax.org>>, Aug. 5, 2010, pp. 7.
Chase, et al., "Managing Energy and Server Resources in Hosting Centers", Retrieved at << http://cseweb.ucsd.edu/˜vandat/papers/muse-sosp01.pdf >>, ACM SIGOPS Operating Systems Review, vol. 35, no. 5, Dec. 2001.
Chase, et al., "Managing Energy and Server Resources in Hosting Centers", Retrieved at >, ACM SIGOPS Operating Systems Review, vol. 35, no. 5, Dec. 2001.
Chatwin, R.E.; "Optimal Dynamic Pricing of Perishable Products with Stochastic Demand and a Finite Set of Prices"; European Journal of Operational Research 125 (2000); Aug. 1, 1997; pp. 149-174.
Chaver et al., "Energy-Aware Fetch Mechanism: Trace Cache and BTB Customization", in Proceedings of the 2005 International Symposium on Lower Power Electronics and Design, Aug. 8-10, 2005, 6 pages.
Chen, et al., "Energy-Aware Server Provisioning and Load Dispatching for Connection-Intensive Internet Services", Retrieved at << http://research.microsoft.com/en-us/um/people/zhao/pubs/nsdi08-xiao.pdf >>, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, Apr. 16-18, 2008.
Chen, et al., "Energy-Aware Server Provisioning and Load Dispatching for Connection-Intensive Internet Services", Retrieved at >, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, Apr. 16-18, 2008.
Cho, et al.; "Strategic Consumer Response to Dynamic Pricing of Perishable Products"; International Series in Operations Research & Management Science; vol. 131; Aug. 2008; pp. 435-458.
Clark et al., "Live Migration of Virtual Machines", in Proceedings of the 2nd ACM/USENIX Symposium on Networked Systems Design & Implementation, May 2-4, 2005, 14 pages.
CN Notice on the First Office Action mailed Jul. 29, 2013 for Application No. 201080009556.7, 11 pages.
Dasu, S. et al.; "Dynamic pricing when consumers are strategic: Analysis of a posted pricing scheme"; European Journal of Operational Research; vol. 204, Issue 3; 2010; pp. 1-23.
Dunn, D.; "Data Center Energy Efficiency Through BCM"; Industry Perspectives; Jan. 2010; 7 pages.
El Sweeting, A.; "Dynamic Pricing Behavior in Perishable Goods Markets: Evidence from Secondary Markets for Major League Baseball Tickets"; Duke University; Feb. 2010; pp. 1-65.
EP Communication for Application No. 10744134.7, Dec. 3, 2012, 1 page.
EP Communication for Application No. 10744134.7, Nov. 13, 2012, 6 pages.
Fan, X. et al.; "Power Provisioning for a Warehouse-sized Computer"; Proceedings of the ACM International Symposium on Computer Architecture; Jun. 2007; 11 pages.
Felter, "A Performance-Conserving Approach for Reducing Peak Power Consumption in Server Systems", Proceedings of the 19th Annual International Conference on Supercomputing, Jun. 20-22, 2005, 10 pages.
Fiat, et al., "Better Algorithms for Unfair Metrical Task Systems and Applications", Retrieved at<<http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.21.7258&rep=rep1&type=pdf>>, 2000, pp. 20.
Fiat, et al., "Better Algorithms for Unfair Metrical Task Systems and Applications", Retrieved at>, 2000, pp. 20.
Final Office Action mailed Dec. 17, 2012 from U.S. Appl. No. 12/951,977, 19 pages.
Final Office Action mailed Dec. 19, 2011 from U.S. Appl. No. 12/391,188, 17 pages.
Final Office Action mailed Feb. 5, 2014, from U.S. Appl. No. 12/903,227, 18 pages.
Final Office Action mailed Jun. 5, 2015 from U.S. Appl. No. 13/169,890, 16 pages.
Final Office Action mailed Mar. 15, 2012 from U.S. Appl. No. 12/492,385, 16 pages.
Final Office Action mailed May 14, 2015, from U.S. Appl. No. 13/169,923, 17 pages.
Final Office Action mailed May 5, 2014, from U.S. Appl. No. 12/781,445, 12 pages.
Final Office Action mailed Oct. 3, 2012 from U.S. Appl. No. 12/973,399, 25 pages.
Final Office Action mailed Oct. 9, 2013 from U.S. Appl. No. 13/169,890, 11 pages.
Final Office Action, mailed Dec. 18, 2013. from U.S. Appl. No. 13/169,923, 11 pages.
Giannakoudis et al., "Optimum design and operation under uncertainty of power systems using renewable energy sources and hydrogen storage", International Journal of Hydrogen Energy, ScienceDirect, Available online Nov. 27, 2009, 20 pages.
Gmach et al., "Capacity Planning and Power Management to Exploit Sustainable Energy", in Proceedings of the 6th International Conference on Network and Service Management, Oct. 25-29, 2010, 8 pages.
Grossman, R.L.; "The Case for Cloud Computing"; IT Professional; vol. 11, No. 2; Mar. 21, 2009; pp. 23-27.
Harrison, A., "Symantec Energy Efficient Data Center Solutions," Symantec Corporation, Cupertino, CA; Jan. 12, 2008; 15 pages.
Horvitz, "Precomputation for Data Center Load Balancing", U.S. Appl. No. 12/492,385, Filed: Jun. 26, 2009.
Iranli, "System-Level Power Management: An Overview", In The VLSI Handbook Second Edition, CRC Press, Dec. 26, 2006, 22 pages.
Jain, "An Optimization Framework for Dynamic Client-Server Partitioning in the IMA Application", Apr. 16, 2010, 3 pages.
Jain, "Dynamically Placing Computing Jobs", U.S. Appl. No. 12/951,977, Filed: Nov. 22, 2010.
Jain, "Energy-Aware Server Management", U.S. Appl. No. 12/391,188, Filed: Feb. 23, 2009.
Jain, "Pricing Batch Computing Jobs at Data Centers", U.S. Appl. No. 12/973,399, Filed: Dec. 20, 2010.
Jain, "SLA-Integrated Pricing Models for Cloud Computing", Microsoft Research, Oct. 18, 2010, 5 pages.
Jiang et al., "Enabling Autonomic Power-Aware Management of Instrumented Data Centers", Retrieved at<<http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=05160976>>, 2009, 8 Pages.
Jiang et al., "Enabling Autonomic Power-Aware Management of Instrumented Data Centers", Retrieved at>, 2009, 8 Pages.
Kahn, C.; "As power demand soars, grid holds up . . . so far"; Jul. 7, 2010; Associated Press; 1 page.
Kant, Krishna, "Data center evolution: A tutorial on state of the art issues, and challenges", Computer Networks, Nov. 12, 2009, 27 pages.
Kant, Krishna, "Distributed Energy Adaptive Computing", IEEE International Conference on Communications, May 23-27, 2010, 5 pages.
Kant, Krishna, "Power Control of High Speed Network Interconnects in Data Centers", in Proceedings of the 28th IEEE International Conference on Computer Communications, Apr. 19-25, 2009, 6 pages.
Karve, et al., "Dynamic Placement for Clustered Web Applications", Retrieved at << http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.80.7874&rep=rep1&type=pdf >>, International World Wide Web Conference, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006.
Karve, et al., "Dynamic Placement for Clustered Web Applications", Retrieved at >, International World Wide Web Conference, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006.
L. Liu, H. Wang, X. Liu, X. Jin, W.B. He, Q.B. Wang, and Y. Chen, GreenCloud: a new architecture for green data center, Proceedings of the 6th international conference industry session on Autonomic computing and communications industry session, Barcelona, Spain: ACM, 2009, pp. 29-38. *
Le et al., "Managing the Cost, Energy Consumption, and Carbon Footprint of Internet Services", SIGMETRICS'10, Jun. 14-18, 2010, 2 pages.
Levin, Y. et al.; "Optimal Dynamic Pricing of Perishable Items by a Monopolist Facing Strategic Consumers"; Production and Operations Management; 2009; 53 pages.
Li et al.; "Load Balancing for Multi-tiered Database Systems through Autonomic Placement of Materialized Views"; Proceedings of the 22nd International Conference on Data Engineering (ICDE 2006); 2006; 12 pages.
McGeehan, Patrick, "Heat Wave Report: 102 Degrees in Central Park", Retrieved at<<http://cityroom.blogs.nytimes.com/2010/07/06/ heatwave-report-the-days-first-power-loss/?hp>>, Jul. 6, 2010, 17 Pages.
McGeehan, Patrick, "Heat Wave Report: 102 Degrees in Central Park", Retrieved at>, Jul. 6, 2010, 17 Pages.
Non-Final Office Action mailed Apr. 15, 2013 from U.S. Appl. No. 13/169,890, 10 pages.
Non-Final Office Action mailed Apr. 3, 2013 from U.S. Appl. No. 12/781,445, 11 pages.
Non-Final Office Action mailed Aug. 1, 2011 from U.S. Appl. No. 12/391,188, 12 pages.
Non-Final Office Action mailed Dec. 3, 2013, from U.S. Appl. No. 12/781,445, 16 pages.
Non-Final Office Action mailed Dec. 31, 2012 from U.S. Appl. No. 12/913,763, 15 pages.
Non-Final Office Action mailed Jul. 3, 2014 from U.S. Appl. No. 12/973,399, 26 pages.
Non-Final Office Action mailed Jun. 21, 2012 from U.S. Appl. No. 12/973,399, 20 pages.
Non-Final Office Action mailed Mar. 27, 2014 from U.S. Appl. No. 13/169,890, 13 pages.
Non-Final Office Action mailed Mar. 29, 2013 from U.S. Appl. No. 12/903,227, 16 pages.
Non-Final Office Action mailed May 17, 2013 from U.S. Appl. No. 13/169,923, 11 pages.
Non-Final Office Action mailed Nov. 14, 2014 from U.S. Appl. No. 13/169,890, 13 pages.
Non-Final Office Action mailed Oct. 7, 2011 from U.S. Appl. No. 12/492,385, 14 pages.
Non-Final Office Action mailed Sep. 4, 2012 from U.S. Appl. No. 12/951,977, 10 pages.
Non-Final Office Action, mailed Oct. 6, 2014, from U.S. Appl. No. 13/169,923, 17 pages.
Notice of Allowance mailed Apr. 15, 2015 From U.S. Appl. No. 12/951,977. 8 Pages.
Notice of Allowance mailed Apr. 2, 2012 from U.S. Appl. No. 12/391,188, 5 pages.
Notice of Allowance mailed Apr. 9, 2014 from U.S. Appl. No. 12/951,977, 12 pages.
Notice of Allowance mailed Aug. 15, 2014, from U.S. Appl. No. 12/913,763, 6 pages.
Notice of Allowance mailed Aug. 23, 2013 from U.S. Appl. No. 12/913,763, 9 pages.
Notice of Allowance mailed Feb. 19, 2013 from U.S. Appl. No. 12/492,385, 6 pages.
Notice of Allowance mailed Jul. 18, 2013 from U.S. Appl. No. 12/492,385, 6 pages.
Notice of Allowance mailed Jul. 30, 2014, from U.S. Appl. No. 12/492,385, 7 pages.
Notice of Allowance mailed Jul. 7, 2014, from U.S. Appl. No. 12/913,763, 6 pages.
Notice of Allowance mailed Jun. 24, 2014, from U.S. Appl. No. 12/492,385, 8 pages.
Notice of Allowance mailed Mar. 31, 2014, from U.S. Appl. No. 12/492,385, 5 pages.
Notice of Allowance mailed May 21, 2014 from U.S. Appl. No. 12/951,977, 9 pages.
Notice of Allowance mailed Sep. 5, 2014 from U.S. Appl. No. 12/951,977, 7 pages.
Notice on Grant mailed Apr. 13, 2015 from Chinese Patent Application No. 201080009556.7, 7 pages.
Office Action mailed Aug. 10, 2015 from European Patent Application No. 10744134.7-1954, 6 pages.
Pauley, W.A.; "Cloud Provider Transparency: An Empirical Evaluation"; IEEE Security and Privacy; vol. 8, No. 6; Aug. 19, 2010; pp. 32-39.
Perry, Michael, "Australian Greens Propose a$23/ton Carbon Tax", Retrieved at <<http://uk.reuters.com/article/iduktre65s2u920100629>>, Jan 29, 2010, 2 Pages.
Perry, Michael, "Australian Greens Propose a$23/ton Carbon Tax", Retrieved at >, Jan 29, 2010, 2 Pages.
Pflueger, et al.; "Data Center Efficiency in the Scalable Enterprise"; Dell Power Solutions; Feb. 2007; pp. 8-14.
Popa, R.A. et al.; "Enabling Security in Cloud Storage SLAs with CloudProof"; Microsoft Research Technical Report. MSR-TR-2010-46; May 2010; pp. 1-12.
Qureshi, et al., "Cutting the electric bill for Internet-scale systems", Retrieved at << http://nms.csail.mit.edu/papers/sigcomm372-aqureshi.pdf >>, Applications, Technologies, Architectures, and Protocols for Computer Communication, Proceedings of the ACM SIGCOMM 2009 conference on Data communication, Aug. 16-21, 2009.
Qureshi, et al., "Cutting the electric bill for Internet-scale systems", Retrieved at >, Applications, Technologies, Architectures, and Protocols for Computer Communication, Proceedings of the ACM SIGCOMM 2009 conference on Data communication, Aug. 16-21, 2009.
Rao, et al., "Minimizing Electricity Cost: Optimization of Distributed Internet Data Centers in a Multi-Electricity-Market Environment", Retrieved at<< http://mc.eistar.net/Paper/INFOCOM-RAO.pdf >>, in Proceedings of IEEE Conference on Computer Communications, (INFOCOM), 2010.
Rao, et al., "Minimizing Electricity Cost: Optimization of Distributed Internet Data Centers in a Multi-Electricity-Market Environment", Retrieved at>, in Proceedings of IEEE Conference on Computer Communications, (INFOCOM), 2010.
Rasmussen, N.; "Implementing Energy Efficient Data Centers"; White Paper #114; American Power Conversion; Retrieved at http://www.apcmedia.com/ salestools/NRAN-6LXSHX-R0-EN.pdf; 2006; pp. 1-15.
Request for Examination and Voluntary Amendment filed Dec. 30, 2014 from Korean Patent Application No. 10-2011-7019493, 17 pages.
Requirement for Restriction/Election mailed Aug. 2, 2012 from U.S. Appl. No. 12/951,977, 7 pages.
Requirement for Restriction/Election mailed Feb. 1, 2013 from U.S. Appl. No. 12/903,227, 6 pages.
Requirement for Restriction/Election mailed Jun. 22, 2012 from U.S. Appl. No. 12/951,977, 6 pages.
Requirement for Restriction/Election mailed Jun. 7, 2012 from U.S. Appl. No. 12/913,763, 10 pages.
Response filed Apr. 1, 2014 to Non-Final Office Action mailed Dec. 3, 2013, from U.S. Appl. No. 12/781,445, 12 pages.
Response filed Apr. 17, 2013 to Final Office Action mailed Dec. 17, 2012 from U.S. Appl. No. 12/951,977, 15 pages.
Response filed Apr. 30, 2013 to Non-Final Office Action mailed Dec. 31, 2012 from U.S. Appl. No. 12/913,763, 16 pages.
Response filed Aug. 13, 2012 to Requirement for Restriction/Election mailed Aug. 2, 2012 from U.S. Appl. No. 12/951,977, 6 pages.
Response filed Aug. 15, 2013 to Non-Final Office Action mailed Apr. 15, 2013 from U.S. Appl. No. 13/169,890, 12 pages.
Response filed Aug. 2, 2013 to Non-Final Office Action mailed Apr. 3, 2013 from U.S. Appl. No. 12/781,445, 14 pages.
Response filed Aug. 22, 2013 to Non-Final Office Action mailed Mar. 29, 2013 from U.S. Appl. No. 12/903,227, 13 pages.
Response filed Aug. 24, 2015 to Final Office Action mailed May 14, 2015 from U.S. Appl. No. 13/169,923, 16 pages.
Response filed Aug. 27, 2012 to Non-Final Office Action mailed Jun. 21, 2012 from U.S. Appl. No. 12/973,399, 18 pages.
Response filed Dec. 9, 2013 to First Office Action mailed Jul. 29,2013 for Chinese Application No. 2010800095561, 15 pages.
Response filed Feb. 10, 2014 to Final Office Action mailed Oct. 9, 2013 from U.S. Appl. No. 13/169,890, 17 pages.
Response filed Feb. 24, 2014 to Final Office Action, mailed Dec. 18, 2013. from U.S. Appl. No. 13/169,923, 18 pages.
Response filed Feb. 27, 2013 to Requirement for Restriction/Election mailed Feb. 1, 2013 from U.S. Appl. No. 12/903,227, 7 pages.
Response filed Jan. 23, 2013 to Final Office Action mailed Oct. 3, 2012 from U.S. Appl. No. 12/973,399, 16 pages.
Response filed Jan. 30, 2013 to Official Communication for EP Application No. 10744134.7.
Response filed Jan. 6, 2012 to Non-Final Office Action mailed Oct. 7, 2011 from U.S. Appl. No. 12/492,385, 15 pages.
Response filed Jul. 16, 2012 to Final Office Action mailed Mar. 15, 2012 from U.S. Appl. No. 12/492,385, 17 pages.
Response filed Jul. 23, 2012 to Requirement for Restriction/Election mailed Jun. 22, 2012 from U.S. Appl. No. 12/951,977, 7 pages.
Response filed Jul. 24, 2014 to Non-Final Office Action mailed Mar. 27, 2014 from U.S. Appl. No. 13/169,890, 18 pages.
Response filed Jun. 27, 2012 to Requirement for Restriction/Election mailed Jun. 7, 2012 from U.S. Appl. No. 12/913,763, 10 pages.
Response filed Mar. 18, 2013 to Restriction/Election mailed Feb. 28, 2013 from U.S. Appl. No. 13/169,890, 11 pages.
Response filed Mar. 19, 2012 to Final Office Action mailed Dec. 19, 2011 from U.S. Appl. No. 12/391,188, 11 pages.
Response filed Nov. 16, 2011 to Non-Final Office Action mailed Aug. 1, 2011 from U.S. Appl. No. 12/391,188, 11 pages.
Response filed Nov. 29, 2012 to Non-Final Office Action mailed Sep. 4, 2012 from U.S. Appl. No. 12/951,977, 12 pages.
Response filed Sep. 15, 2014 to Final Office Action mailed May 5, 2014, from U.S. Appl. No. 12/781,445, 10 pages.
Response filed Sep. 6, 2013 to Non-Final Office Action mailed May 17, 2013 from U.S. Appl. No. 13/169,923, 14 pages.
Restriction/Election mailed Feb. 28, 2013 from U.S. Appl. No. 13/169,890, 6 pages.
Sleator, et al., "Amortized Efficiency of List Update and Paging Rules", Retrieved at<<http://www.google.co.in/url?sa=t&source=web&cd=1&ved=0CBUQFJAA&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.90.3388%26rep%3Drep1%26type%3Dpdf&rct=j&q=Amortized%20efficiency%20of%20list%20update%20and%20paging%20rules&ei=IIIhaTNvYINTP4gad6MWeAg&usg=AFQjCN FIS-H9stkUtfKpRwHtZ6HZLQX-oNg>>, vol. 28, No. 2, Feb. 1985, pp. 202-208.
Supplemental Amendment filed Jun. 6, 2014, from U.S. Appl. No. 12/492,385, 8 pages.
Timmer, J.; "Renewable Energy and the Future of the Datacenter"; Arstechnica.com business news; May 2008; 2 pages.
U.S. Appl. No. 12/391,188, filed Feb. 23, 2009.
U.S. Appl. No. 12/492,385, filed Jun. 26, 2009.
U.S. Appl. No. 12/781,445, filed May 17, 2010.
U.S. Appl. No. 12/903,227, filed Oct. 13, 2010.
U.S. Appl. No. 12/913,763, filed Oct. 28, 2010.
U.S. Appl. No. 12/951,977, filed Nov. 22, 2010.
U.S. Appl. No. 12/973,399, filed Dec. 20, 2010.
U.S. Appl. No. 13/169,890, filed Jun. 27, 2011.
U.S. Appl. No. 13/169,923, filed Jun. 27, 2011.
Urgaonkar, "Dynamic Provisioning of Multi-Tier Internet Applications", In Proceedings of the Second International Conference on Autonomic Computing, Jun. 13-16, 2005, 12 pages.
USENIX, "Technical Sessions", 5th USENIX Symposium on Networked Systems Design and Implementation, Apr. 16-18, 2008, 4 pages.
Vazirani, V.; "Approximation Algorithms"; Springer; 2001; 396 pages.
Verma et al., "Brownmap: Enforcing Power Budget in Shared Data Centers", IBM Research Report, Dec. 17, 2009, 15 pages.
Wang, "Feedback Control Algorithms for Power Management of Servers", Third International Workshop on Feedback Control Implementation and Design in Computing Systems and Networks, Jun. 6, 2008.
Wang, C. et al.; "Markdown Money Contracts for Perishable Goods with Clearance Pricing"; European Journal of Operational Research; vol. 196; 2009; pp. 1113-1122.
Windows Azure Platform, Retrieved at<<http://www.azure.com>>, 2 Pages.
Windows Azure Platform, Retrieved at>, 2 Pages.
US20110282982A1 (en) 2011-11-17
US8849469B2 (en) 2014-09-30 Data center system that accommodates episodic computation
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:JAIN, NAVENDU;REEL/FRAME:024376/0800