Source: https://patents.google.com/patent/US10289183B2/en
Timestamp: 2019-06-27 04:58:31
Document Index: 637375043

Matched Legal Cases: ['Application No. 62', 'Application No. 15833856', 'Application No. 15833962', 'Application No. 15834071', 'Application No. 200410101227', 'Application No. 0425264', 'Application No. 0609876']

US10289183B2 - Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system - Google Patents
Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system Download PDF
US10289183B2
US10289183B2 US14/582,772 US201414582772A US10289183B2 US 10289183 B2 US10289183 B2 US 10289183B2 US 201414582772 A US201414582772 A US 201414582772A US 10289183 B2 US10289183 B2 US 10289183B2
US14/582,772
US20160054781A1 (en
2014-08-22 Priority to US201462040576P priority Critical
2014-12-24 Priority to US14/582,772 priority patent/US10289183B2/en
2015-11-24 Assigned to INTEL CORPORATION reassignment INTEL CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HOFFMAN, ANDY, SONG, JUSTIN, ALEXANDER, JAMES, BODAS, DEVADATTA, RAJAPPA, MURALIDHAR
2016-02-25 Publication of US20160054781A1 publication Critical patent/US20160054781A1/en
2019-05-14 Publication of US10289183B2 publication Critical patent/US10289183B2/en
A non-transitory computer readable storage medium storing instructions executable by one or more processors of a distributed computer system to perform operations including determining whether a power consumed by the distributed computer system is greater than a power allocated to the distributed computer system, responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system and responsive to determining all jobs being processed by the distributed computer system are processing at a lowest power state for each job, suspending a job having a lowest priority among all jobs being processed by the distributed computer system is shown.
The present application claims the benefit of prior U.S. Provisional Patent Application No. 62/040,576, entitled “SIMPLE POWER-AWARE SCHEDULER TO LIMIT POWER CONSUMPTION BY HPC SYSTEM WITHIN A BUDGET” filed on Aug. 22, 2014, which is hereby incorporated by reference in its entirety.
The present application is related to the U.S. patent application Ser. No. 14/582,795 entitled METHODS AND APPARATUS TO ESTIMATE POWER PERFORMANCE OF A JOB THAT RUNS ON MULTIPLE NODES OF A DISTRIBUTED COMPUTER SYSTEM, filed Dec. 24, 2014; the U.S. patent application Ser. No. 14/582,783 entitled METHOD AND APPARATUS TO GENERATE AND USE POWER, THERMAL AND PERFORMANCE CHARACTERISTICS OF NODES TO IMPROVE ENERGY EFFICIENCY AND REDUCING WAIT TIME FOR JOBS IN THE QUEUE, filed Dec. 24, 2014; the U.S. patent application Ser. No. 14/582,979 entitled ADJUSTMENT OF EXECUTION OF TASKS, filed Dec. 24, 2014; the U.S. patent application Ser. No. 14/582,985 entitled CONTROL OF POWER CONSUMPTION, filed Dec. 24, 2014; the U.S. patent application Ser. No. 14/582,988 entitled FORECAST FOR DEMAND OF ENERGY, filed Dec. 24, 2014; the U.S. patent application Ser. No. 14/582,743 entitled MANAGING POWER PERFORMANCE OF DISTRIBUTED COMPUTING SYSTEMS, filed Dec. 24, 2014; the U.S. patent application Ser. No. 14/582,756 entitled “PROFILING A JOB POWER AND ENERGY CONSUMPTION FOR A DATA PROCESSING SYSTEM,” filed Dec. 24, 2014; and the U.S. patent application Ser. No. 14/582,764 entitled “A POWER AWARE JOB SCHEDULER AND MANAGER FOR A DATA PROCESSING SYSTEM”, filed Dec. 24, 2014.
Embodiments of the disclosure generally relate to the field of power conservation in computer systems. More specifically, one embodiment of the disclosure relates to managing jobs of a distributed computer system when there is a change to the power allocation to the distributed computer system.
A distributed computer system may perform parallel computing by the simultaneous use of multiple nodes to execute a computational assignment referred to as a job. Each node may include one or more processors, memory, an operating system, and one or more input/output (I/O) components. The nodes may communicate with each other through a high speed network fabric and may use shared file systems or storage. The job may be divided into thousands of parallel tasks distributed over thousands of nodes. These nodes may synchronize with each other hundreds of times a second.
Future distributed computer systems are projected to require tens of megawatts of power, making their power management a foremost concern in the industry. These distributed computer systems will be expected to deliver exascale performance with limited power and energy budgets. Current distributed computer systems may apply power capping to adhere to the limited power and energy budgets. However, regardless of power capping, the power allocation to a distributed computer system (“the system”) may be decreased such that the power being consumed by the system exceeds the power allocated to the system.
The management of currently running jobs, suspended jobs and newly requested jobs in a queue of the system is critical in maintaining expected performance of the system and ensuring power consumed by the system remains less than the power allocated to the system. As the power allocated to the system fluctuates, there may be a need to suspend and/or terminate one or more currently running jobs, resume one or more suspended jobs and/or start one or more new jobs already in the queue. In addition, the system may be required to adhere to priorities regarding power allocation to certain types of jobs.
FIG. 1 is an exemplary block diagram of a HPC system 100 receiving a plurality of inputs.
FIG. 2 is an exemplary block diagram of logic comprising the HPC system 100.
FIG. 3 is a flowchart illustrating a sample method of managing jobs within the HPC system 100 when the power consumed by the system exceeds the power allocated to the system.
FIGS. 4A-4C comprise a flowchart illustrating one embodiment of a method for managing one or more jobs when the power consumed by the HPC system 100 is less than or approximately equal to the power allocated to the HPC system 100.
Various embodiments of the disclosure relate to managing jobs being processed by a distributed computer system (“currently running jobs”), jobs suspended by the distributed computer system (“suspended jobs”) and newly requested jobs placed in the queue of the distributed computer system (“new jobs”). In one embodiment, the management of the jobs may require that the distributed computer system attempt to reduce the power consumed by one or more jobs prior to suspending and/or terminating one or more jobs. In a second embodiment, the management of the jobs may require that the distributed computer system suspend and/or terminate one or more jobs when the power allocated to the distributed computer system is decreased. Additionally, the management of the jobs may allow the distributed computer system to resume one or more suspended jobs and/or start one or more new jobs. The management of the jobs may rely on estimates generated by the distributed computer system as discussed in the U.S. patent application Ser. No. 14/582,795 entitled “Methods and apparatus to estimate power performance of a job that runs on multiple nodes of a distributed computer system.” The distributed computer system may be, for example, a High Performance Computing (HPC) system. In some embodiments of the disclosure, a job may not be subjected to a power policy that limits the power supplied to a job as set forth by the owner of the job; however, a HPC system may, and likely will, have an overall limited power budget that cannot be exceeded by the combination of jobs processed by the HPC system.
Embodiments of the disclosure relate to determining whether a decrease in power allocated to the system requires that one or more current jobs be suspended and/or terminated. Other embodiments of the disclosure relate to determining whether an increase in power allocated to the system, or an increase in available power (e.g., a job finished processing) allows the HPC system to resume one or more suspended jobs and/or start a new job waiting in the queue of the HPC system. Certain embodiments relate to determining priorities of particular jobs and/or job types within the HPC system when determining which one or more jobs to suspended, terminate, resume and/or start.
Some embodiments of the disclosure relate to determining whether power consumption of one or more jobs may be reduced when the power consumed by the system is within a predefined threshold (e.g., a system guardband) from the power allocated to the system. The power consumed by one or more jobs may be reduced in order to utilize the greatest number of compute nodes as possible. Therefore, in one embodiment, when the power consumed by one or more jobs is reduced, the retrieved power may be combined with unallocated power and used to start a new job or resume a suspended job.
Referring to FIG. 1, an exemplary block diagram of a HPC system receiving various inputs is shown. The HPC system 100 includes one or more operating system (OS) nodes 101, one or more compute nodes 102, one or more input/output (I/O) nodes 103 and a storage 104. A high-speed fabric communicatively connects the OS node 101, the compute nodes 102 and the I/O nodes 103. The high-speed fabric may be a network topology of nodes interconnected via one or more switches. In one embodiment, as illustrated in FIG. 1, the I/O nodes 103 are communicatively connected to the storage 104. The storage 104 may be non-persistent storage such as volatile memory (e.g., any type of random access memory “RAM”); persistent storage such as non-volatile memory (e.g., read-only memory “ROM”, power-backed RAM, flash memory, phase-change memory, etc.), a solid-state drive, hard disk drive, an optical disc drive, or a portable memory device.
The OS node 101 may provide a gateway to accessing the compute nodes 102. For example, prior to submitting a job for processing on the compute nodes 102, a user may be required to log-in to the HPC system 100 which may be through the OS node 101. In embodiments of the disclosure, the OS node 101 may accept jobs submitted by users and assist in the launching and managing of jobs being processed by the compute nodes 102. In one embodiment, the OS node 101 comprises a power monitor (not shown), as described in the U.S. patent application Ser. No. 14/582,756 entitled “Methods and apparatus to profile power and energy consumption by a job running in multiple nodes and uses shared resources of a distributed computer system (HPC)”; a power estimator (not shown); and a power calibrator (not shown) as described in the U.S. patent application Ser. No. 14/582,783 entitled “Methods and apparatus to generate and use power, thermal and performance characteristics of nodes to improve energy efficiency and reducing wait time for jobs in the queue.”
In one embodiment, the compute nodes 102 provide the bulk of the processing and computational power. The I/O nodes 103 may provide an interface between the compute nodes 102 and external devices (e.g., separate computers) that may provide input to the HPC system 100 or receive output from the HPC system 100.
The system power allocation (Psys) may be provided to the HPC system 100 by, for example, a utility management facility (e.g., as determined by a system administrator or management software such as a datacenter manager). Typically, the Psys will be a limited amount of power allocated to the HPC system 100 with which the HPC system 100 will use to run one or more of the jobs 120. The jobs 120 comprise one or more jobs requested to be run by the HPC system 100 by one or more users. Each job includes a “power policy,” which will be discussed in-depth below. The power policy will assist the HPC system 100 in allocating power for the job and aid in the management of the one or more jobs 120 being run by the HPC system 100.
In addition, the administrative policies 130 will guide the management of running the jobs 120 by providing an over-arching policy that defines the operation of the HPC system 100. Examples of policies that may be included in the administrative policies 130 include, but are not limited or restricted to, (1) maximize utilization of all hardware and software resources (e.g., instead of running fewer jobs at high power and leaving resources unused, run as many jobs as possible to use as much of the resources as possible); (2) a job with no power limit is given the highest priority among all running jobs; and/or (3) suspended jobs are at higher priority for resumption. Such administrative policies govern the way the HPC system 100 may schedule, launch, suspend and re-launch one or more jobs.
In the following description, certain terminology is used to describe features of the invention. For example, in certain situations, both terms “logic” and “engine” are representative of hardware, firmware and/or software that is configured to perform one or more functions. As hardware, logic (or engine) may include circuitry having data processing or storage functionality. Examples of such circuitry may include, but are not limited or restricted to a microprocessor, one or more processor cores, a programmable gate array, a microcontroller, an application specific integrated circuit, wireless receiver, transmitter and/or transceiver circuitry, semiconductor memory, or combinatorial logic.
The term “power monitoring” should be interpreted as dynamically measuring power consumption by one or more of the components comprising the HPC system. The measurements taken may be used to calculate power consumption by, example, an individual job and/or a group of jobs, as well as to provide statistics on the overall power consumption of the HPC system.
The term “power policy” should be interpreted as an input (e.g., one or more parameters) provided to the HPC system that provide guidance on allocation and management of power for a given job. The input may be provided as part of a job request submission and/or may be provided as a separate input (e.g., via a user interface screen or a configuration file). For example, the input may indicate, among other things, (i) whether a job may be subjected to a power limit, (ii) the maximum and/or minimum power at which a job may run and/or (iii) a minimum and/or maximum frequency at which the one or more nodes processing the job may operate.
The term “node” should be construed as one or more processors optionally grouped with, at least, a system memory and/or one or more input/output (I/O) components. The one or more processors, the system memory and the one or more I/O components may be referred to as the “components” of a node throughout the specification and claims. Throughout the specification and claims, the terms “processor,” “computer processing unit (CPU),” and “core” will be used interchangeably.
The term “job” should be interpreted as predetermined calculations performed on the HPC system. For example, a user (e.g., owner of the job) may request that a job be run by the HPC which means the user is requesting to have one or more compute nodes performing calculations according to input parameters and/or data provided by the user. The job request may specify the one or more calculations (e.g., an application) that are to be used for the processing of the job.
The term “system power (Psys)” should be interpreted as the amount of power provided to the HPC system by, for example, a facility or datacenter manager. The Psys is the total amount of power the HPC system has to allocate to one or more jobs at any given time.
The term “guard band” should be interpreted as a mechanism to assist in the management of a power budget of a HPC system. In one embodiment, the guard band may an extra power allocation, which may be a predetermined percentage of the power allocated to the job. For example, if a HPC system has 3 MW of power to allocate to a job, the HPC system may only allocate 2.8 MW and maintain 0.2 MW as the guard band to prevent a spike in calculations to cause the power consumption of the job to exceed 3 MW. One purpose of the guard band is to maintain consistent power consumption by a job.
The invention may be utilized for power management of a distributed computer system, such as a High Performance Computing (HPC) system. In particular, embodiments of the disclosure relate to managing power allocation to one or more jobs run in a HPC system based on estimates of the power consumption for each job as a result of calibration of the nodes within the HPC system. As this invention is susceptible to embodiments of many different forms, it is intended that the present disclosure is to be considered as an example of the principles of the invention and not intended to limit the invention to the specific embodiments shown and described.
II. Power Aware Distributed Computer System
Referring to FIG. 2, an exemplary block diagram of logic comprising the HPC system 100 is shown. The logic of the HPC system 100 illustrated in FIG. 2 comprises the power management for the HPC system 100 and includes a resource manager 210 including a power aware job scheduler 211 and a power aware job launcher 212, a calibrator 220, an estimator 230, one or more job managers 240 (each job will have its own job manager), a job 250, the user policies 205 and the administrative policies 130. In one embodiment, the resource manager 210 and job manager 240 are configured to collect job power data.
The calibrator 220 calibrates the power, thermal dissipation and performance of each node within the HPC system 100. The calibrator 220 may provide a plurality of methods for calibrating the nodes within the HPC system 100. In one embodiment, the calibrator 100 may provide a first method of calibration in which every node within the HPC system 100 runs a sample workload (e.g., a mini-application, a portion of an application and/or a test script) so the calibrator 220 may sample one or more measurements (e.g., power consumed) at a predetermined time interval in order to determine, inter alia, (1) the average power, (2) the maximum power, and (3) the minimum power for each node. In addition, the sample workload may be run on each node at every operating frequency of the node. In another embodiment, the calibrator 220 may provide a second method of calibration in which calibration of one or more nodes occurs during the run-time of a job. In such a situation, the calibrator 220 may sample the one or more nodes on which a job is running (e.g., processing). In the second method, the calibrator 220 obtains power measurements of each node during actual run-time.
The estimator 230 provides the resource manager 210 with estimates of power consumption for each job enabling the resource manager 210 to efficiently schedule and monitor each job requested by one or more job owners (e.g., users). The estimator 220 may provide a power consumption estimate based on, for example, maximum and average power values stored in a calibrator database, wherein the calibrator database is populated by the processing of the calibrator 220. In addition, the minimum power required for each job may be considered. Other factors that may be used by the estimator 230 to create a power consumption estimate include, but are not limited or restricted to, whether the owner of the job permits the job to be subject to a power limit, the job power policy limiting the power supplied to the job (e.g., a predetermined fixed frequency at which the job will run, a minimum power required for the job, or varying frequencies and/or power supplied determined by the resource manager 210), the startup power for the job, the frequency at which the job will run, the available power to the HPC system 100 and/or the allocated power to the HPC system 100.
The each job requested by a user (e.g., the owner of the job) is accompanied by a user policy 205 (also illustrated in FIG. 1). The user policy includes at least a decision on whether the job 250 may be subjected to a power limit, if a power limit is permitted the policy to limit the power (e.g., fixed frequency, minimum power required, or varying frequency and/or power determined by the resource manager 210), and whether the job 250 may be suspended.
In one embodiment, a power aware job scheduler 211 is configured to receive a selection of a mode for a job (e.g., included within the user policies 205), to determine an available power for the job based on the mode and to allocate a power for the job based on the available power. In one embodiment, the power aware job scheduler 211 is configured to determine a uniform frequency for the job based on the available power. In one embodiment, the power aware job scheduler 211 is configured to determine the available power for the job based on at least one of a monitored power, an estimated power, and a calibrated power. The power aware job scheduler 211 and resource manager 210 are configured to receive information regarding power consumption, to distribute the power budget to each job, and to implement a uniform frequency mechanism to limit power, as described in further detail below.
The resource manager 210 uses power aware job scheduler 211 and power aware job launcher 212 to schedule and launch a job based on the received power inputs, e.g., the user policies 205 and the administrative policies 206. In one embodiment, the resource manager 210 is a software object that is responsible for allocation of compute and I/O resources for interactive and batch jobs that one or more users want to run. Typically, the resource manager 210 is also responsible for scheduling the jobs out of the job queue and launching the jobs to run as scheduled. A job manager 240 is configured to control a job to stay within an allocated power budget for the job, as described in further detail below. In one embodiment, the job manager 240 is responsible for operating a job within the constraints of one or more power policies after the job has been launched. In one embodiment, the job manager 240 is used to control power performance of all components (e.g., nodes, or other components) involved in execution of a job as per policies specified by at least one of the user and/or administrator. The power aware job scheduler 211 and the job manager 240.
Referring to FIG. 3, a flowchart illustrating a sample method of managing jobs within the HPC system 100 when the power consumed by the system exceeds the power allocated to the system is shown. Each block illustrated in FIG. 3 represents an operation performed in the method 300 of managing the jobs within the HPC system 100 by suspending one or more jobs and/or terminating one or more jobs. The method 300 illustrates operations conducted by the job scheduler 211, the job manager 240 and the estimator 220 on the compute nodes 102 in efforts to, automatically, without human interaction, manage the jobs of the HPC system 100 when the power allocated to the HPC system 100 is decreased.
In operation 301, the method 300 is started. In operation 302, a determination is made as to whether the power consumed to the HPC system 100 (Pcsys) is less than or equal to the power allocated to the HPC system 100 (Psys). When the Psys is greater than the Pcsys (yes at operation 302), the HPC system 100 monitors whether any further fluctuations in power allocation to the HPC system 100 result in the Pcsys being greater than Psys. When Psys is less than the Pcsys (no at operation 302), a determination as to whether all current jobs are operating at their lowest power state is made (operation 303). In one embodiment, a job may operate at its lowest power state when a minimum power for processing is requested by the owner of the job and the job is processing at the minimum power requested. In another embodiment, the user may have requested a specific frequency at which the compute nodes 102 are to process the job and the power state of the compute nodes 102 while the compute nodes 102 operate at the requested frequency may be the lowest power state for the given job. In another embodiment, in which the job manager 240 automatically adjusts the frequency of a job (e.g. lowers the frequency to the lowest operating frequency of the compute nodes 102), the job may be operating at its lowest power state (e.g., consuming the least amount of power for the job).
When all current jobs are not operating at their lowest power state (no at operation 303), the HPC system 100 reduces the operating power state or power allocation of one or more jobs (operation 304). In one embodiment, the lowest frequency of each job is determined by the resource manager 210 while analyzing the power policies associated with the HPC system 100 and/or the individual job (e.g., in one embodiment, operating at the lowest frequency may consume the least amount of power placing the job in its lowest power state). In another embodiment, a job may have a power policy that no power limit should be applied to the processing of the job. In such a situation, the compute nodes 102 processing the job are to always operate at their highest frequency and consume the maximum amount of power possible; therefore, the operating frequency and power state cannot be reduced. In a second embodiment, a job may have a power policy that allows that HPC system 100 to set the power allocated to the job and/or the operating frequency of the compute nodes 102 processing the job. In such a situation, the HPC system 100 may determine whether the operating frequency of the compute nodes 102 may be reduced, thereby reducing the power state. Subsequently, the Pcsys is recalculated (operation 305) and the method 300 returns to operation 302 to determine whether Pcsys is less than or equal to Psys.
When all current jobs are operating at their lowest power state (yes at operation 303), the lowest priority running job that may be suspended is identified (operation 306). In some embodiments, whether a job may be suspended depends on whether the owner of the job elected such a policy, whether the HPC system 100 has certain policies regarding suspension of jobs (e.g., all jobs may be suspended) and/or whether the job supports checkpointing. Checkpointing involves saving the state of the job in order to preserve the progress of processing the job In case of a failure in hardware or software, checkpointing allows a job to be resumed from the last known working state. In some embodiments, the checkpointing may occur at predefined intervals, immediately preceding the suspension of the job or both. At operation 307, the identified job undergoes checkpointing and at operation 308, the identified job is suspended.
At operation 309, the Pcsys is recalculated taking into consideration the recently suspended job. The term “recalculating” may mean to measure one or more components of one or more of the compute nodes 102, reference the calibrator database, request a new power estimate from the estimator 230 and/or perform calculations based on one or more measurements. At operation 310, a determination is made as to whether the Pcsys is less than or equal to the Psys. If the Pcsys is less than or equal to the Psys (yes at operation 310), the method 300 is complete (311).
If the Pcsys is not less than or equal to the Psys (no at operation 310), a determination is made as to whether all suspendable jobs have been suspended (operation 312). When all suspendable jobs have not been suspended (no at operation 312), the method 300 returns to operation 306 in which the lowest priority running job is identified. Alternatively, when all suspendable jobs have been suspended (yes at operation 312), the current job having the lowest priority is identified (operation 313). At operation 314, the identified job is terminated.
At operation 315, the Pcsys is recalculated taking into consideration the recently terminated job. At operation 316, a determination is made as to whether the Pcsys is less than or equal to the Psys. If the Pcsys is less than or equal to the Psys (yes at operation 316), the method 300 is complete (operation 317).
If the Pcsys is not less than or equal to the Psys (no at operation 316), a determination is made as to whether all jobs have been terminated (operation 318). When all jobs have not been terminated (no at operation 318), the method 300 returns to operation 313 to identify the lowest priority job that cannot be suspended. When all jobs have been terminated (yes at operation 318), the HPC system is forced to power off (operation 319).
FIGS. 4A-4C illustrate one embodiment of the method 400 for managing one or more jobs when the power consumed by the HPC system 100 (Pcsys) is less than or approximately equal to the power allocated to the HPC system 100 (Psys). Referring to FIG. 4A, a determination is made as to whether the power consumed is between the power allocated to the system plus a first system guardband and the power allocated to the system minus a second system guardband (operation 401). When the determination of operation 401 yields a positive answer (yes at operation 401), the method 400 is taken to operation 428, which is discussed in detail in FIG. 4C. When the determination of operation 401 yields a negative answer (no at operation 401), a determination as to whether the power consumed is less than the power allocated minus the first system guardband (operation 402). It should be noted that Pcsys and Psys are average values, e.g., an average rate of consumption of energy over time. The system guardbands serve the purpose of accounting for minor fluctuations in power consumption.
When the determination at operation 402 yields a positive result (yes at operation 402), the method 400 is taken to operation 417, which is discussed in detail in FIG. 4B. When the determination at operation 402 yields a negative result (no at operation 402), a determination as to whether power can be reduced for one or more of the current jobs (e.g., currently running on the HPC system 100) (operation 403). When power can be reduced to one or more current jobs (yes at operation 403), the one or more current jobs for which power is to be reduced are identified (operation 404). Examples of reducing the power are discussed above. At operation 405, the power of the one or more identified current jobs is reduced and the method 400 subsequently returns to operation 401.
When the power of one or more current jobs cannot be reduced (no at operation 403), a determination is made as to whether any current jobs may be suspended (operation 406). When one or more current jobs may be suspended (yes at operation 406), a prioritized list of suspendable jobs is generated (operation 407). At operation 408, the lowest prioritized suspendable job on the generated list is instructed to implement checkpointing. When checkpointing as completed, the lowest prioritized suspendable job will be suspended (operation 409). At operation 410, a determination is made as to whether the power consumed is less than or equal to the power allocated minus the first system guardband. When the power consumed is less than or equal to the power allocated minus the first system guardband (yes at operation 410), the method 400 returns to operation 401. When the power consumed is not less than the power allocated minus the first system guardband (no at operation 410), the method 400 returns to operation 406 to determine if any current jobs may be suspended.
When no current jobs may be suspended (e.g., user policies and/or system policies prevent such action and/or no current jobs support checkpointing) (no at operation 406), a determination is made as to whether one or more jobs are currently running (operation 411). When no jobs are currently running on the HPC system 100, the HPC system 100 is forced to power down (operation 412) and subsequently, the method 400 ends (operation 413).
When one or more jobs are currently running (yes at operation 411), a prioritized list of current jobs is generated (operation 414). At operation 415, the lowest prioritized job on the generated list is terminated (in one embodiment, the list may prioritize both currently running jobs and suspended jobs in one list). At operation 416, a determination is made as to whether the power consumed is less than or equal to the power allocated minus the first system guardband. When the power consumed is less than or equal to the power allocated minus the first system guardband (yes at operation 416), the method 400 returns to operation 401. When the power consumed is not less than or equal to the power allocated minus the first system guardband (no at operation 416), the method 400 returns to operation 411 to determine if there are any jobs currently running.
Referring to FIG. 4B, when the determination at operation 402, of FIG. 4A, yields a positive result (yes at operation 402), a determination is made as to whether a preference to allocate excess power to current jobs (operation 417). In one embodiment, the preference may be derived from user policies and/or system policies. When a preference exists to allocate excess power to current jobs (yes at operation 417), the excess power is allocated to currently running jobs (operation 420) and the method 400 returns to operation 401.
Excess power may be allocated to currently running jobs in a plurality of ways. Examples of ways the resource manager 210 may allocate the excess power include, but are not limited or restricted to, equally among each currently running job, to the currently running job having the longest running time then the second longest (once the first job is operating at its highest power state), etc., to the currently running job having the highest priority then the second highest priority (once the first job is operating at its highest power state), etc., or proportionally to each currently running job (e.g., proportional to running time for each job, proportional to priority for each job, etc.). Examples of ways of prioritizing jobs for allocation of excess power include, but are not limited or restricted to, a prioritized listing of the currently running jobs, the amount of time each currently running job has been running, the power state of each currently running job (e.g., comparison of jobs running at their lowest power state with those running near their highest power state), the number of compute nodes 102 currently processing each currently running job and/or the amount of additional power each currently running job may require to run at its next power state.
When a preference does not exist to allocate excess power to current jobs (no at operation 417), a determination is made as to whether any unallocated resources are available (operation 418). Examples of resources include, but are not limited or restricted to, one or more compute nodes and/or one or more storage devices. When unallocated resources are not available (no at operation 418), the excess power is allocated to currently running jobs (operation 419) and the method 400 returns to operation 401. When unallocated resources are available to begin processing another job (yes at operation 418), a determination is made as to whether the job queue is empty (operation 420). The HPC system 100 maintains a job queue for jobs waiting to be processed by the system. In one embodiment, the job queue may contain both new jobs that have yet to start processing on the system as well as jobs that were previously processing on the system but have been suspended. An alternative embodiment may include separate queues, one for new jobs and one for suspended jobs. The disclosure will assume the job queue is a single queue that may include both new jobs and/or suspended jobs.
When the job queue is empty (yes at operation 420), the excess power is allocated to currently running jobs (operation 421) and the method 400 returns to operation 401.
When the job queue is not empty (no at operation 420), Jnext, being the next highest priority job in the queue, is retrieved (operation 422). At operation 423, a determination is made as to whether sufficient resources are available for Jnext (e.g., whether enough compute nodes of the compute nodes 102 available). When sufficient resources are not available for Jnext (no at operation 423), a determination is made as to whether additional jobs are in the queue (operation 427). When more jobs are present in the queue (yes at operation 427), the method 400 returns to operation 422 to retrieve the next highest priority job from the queue. When no additional jobs are in the queue (no at operation 427), the excess power is allocated to currently running jobs (operation 419) and the method 400 subsequently returns to operation 401.
However, when sufficient resources are available for Jnext (yes at operation 423), a power estimate for Jnext is retrieved (operation 424). The power estimate may be generated by the estimator 230. When the power estimate has been generated, a determination is made as to whether there is sufficient unallocated power to run Jnext on the unallocated resources (operation 425). When there is sufficient unallocated power to run Jnext (yes at operation 425), Jnext is started when Jnext is a new job or Jnext is resumed when Jnext is a suspended job (operation 426). When there is not sufficient unallocated power to run Jnext (no at operation 425), a determination is made as to whether additional jobs are in the queue (operation 427), as discussed above.
Referring to FIG. 4C, when the determination of operation 401 yields a positive answer (yes at operation 401), a determination is made as to whether jobs are currently in the queue (operation 428). When jobs are not currently in the queue (no at operation 428), the method 400 returns to operation 401. When jobs are currently in the queue (yes at operation 428), a determination as to whether any currently running jobs may be operated at a lower power (operation 429). The determination as to whether any currently running jobs may be operated at a lower power is made to determine whether power can be power can be reallocated to run additional jobs (e.g., one or more jobs waiting in the queue).
When no current jobs can be processed at a lower power (no at operation 429), the method 400 returns to operation 401. However, when one or more current jobs can be operated at a lower power (yes at operation 429), the amount of power that can be made available (Pfreed) by operating current jobs at lower power states. Pfreed freed plus any power unallocated by the system may be reallocated to a new or suspended job.
At operation 431, the additional resources that can be made available by reducing power to one or more jobs are determined Subsequently, at operation 432, Jnext is retrieved from the job queue and at operation 433, a power estimate (Pnext) 1 is generated from Jnext. The job queue and retrieval (e.g., organization and retrieval based on priority) is discussed above. Similarly, the generation of a power estimate is also discussed above.
At operation 434, a determination is made as to whether there are sufficient unallocated resources available to run Jnext. When there are not sufficient unallocated resources available to run Jnext (no at operation 434), a determination is made as to whether additional jobs are in the queue (operation 435), as discussed above. When no additional jobs are present in the queue (no at operation 435), the method 400 returns to operation 400. When additional jobs are present in the queue (yes at operation 435), the method 400 returns to operation 432 to retrieve the next highest priority job from the queue.
However, when there are available resources to run a job (e.g., available nodes and unallocated power) (yes at operation 434), the additional resources that can be made available by reducing the power consumed by one or more jobs are determined (operation 432). At operation 433, the job having the next highest priority is retrieved from the queue. At operation 434, the power estimate for Jnext is obtained (e.g., by requesting an estimate from the estimator 230). At operation 435, a determination is made as to whether sufficient resources are available to run Jnext without reducing power consumed by one or more jobs.
When sufficient resources are available to run Jnext (yes at operation 434), a determination is made as to whether the power required to start Jnext (represented by a portion of the power estimate Pnext) is less than or equal to Pfreed plus the unallocated power. When Pnext is greater than Pfreed plus the unallocated power (no at operation 436), the method 400 returns to operation 435 to determine whether there are additional jobs currently waiting in the queue. When Pnext is less than or equal to Pfreed plus the unallocated power (yes at operation 436), the power of one or more jobs is reduced by, at most, Pnext (operation 437) and subsequently, Jnext is started when Jnext a new job or Jnext is resumed when Jnext is returns suspended job (operation 438). Upon starting is Jnext at operation 438, the method 400 returns to operation 401.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining whether a power consumed by the distributed computer system is greater than a power allocated to the distributed computer system, responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system and responsive to determining all jobs being processed by the distributed computer system are processing at a lowest power state for each job, suspending a job having a lowest priority among all jobs being processed by the distributed computer system.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining whether a power consumed by the distributed computer system is greater than a power allocated to the distributed computer system, responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system and responsive to determining all jobs being processed by the distributed computer system are processing at a lowest power state for each job, checkpointing the job having a lowest priority among all jobs being processed by the distributed computer system, responsive to checkpointing the job having the lowest priority, suspending a job having a lowest priority among all jobs being processed by the distributed computer system.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining whether a power consumed by the distributed computer system is greater than a power allocated to the distributed computer system, responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system and responsive to determining all jobs being processed by the distributed computer system are processing at a lowest power state for each job, suspending a job having a lowest priority among all jobs being processed by the distributed computer system, wherein a lowest power state of a first job of all jobs being processed by the distributed computer system differs from a lowest power state of a second job of all jobs being processed by the distributed computer system.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining whether a power consumed by the distributed computer system is greater than a power allocated to the distributed computer system, responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, responsive to determining all jobs being processed by the distributed computer system are processing at a lowest power state for each job, suspending a job having a lowest priority among all jobs being processed by the distributed computer system and after suspending the job having the lowest priority among all jobs being processed by the distributed computer system, recalculating the power consumed by the distributed computer system, wherein a lowest power state of a first job of all jobs being processed by the distributed computer system differs from a lowest power state of a second job of all jobs being processed by the distributed computer system.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining whether a power consumed by the distributed computer system is greater than a power allocated to the distributed computer system, responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, responsive to determining all jobs being processed by the distributed computer system are processing at a lowest power state for each job, suspending a job having a lowest priority among all jobs being processed by the distributed computer system, after suspending the job having the lowest priority among all jobs being processed by the distributed computer system, recalculating the power consumed by the distributed computer system and after recalculating the power consumed by the distributed computer system, suspending a second job having a second lowest priority, wherein a lowest power state of a first job of all jobs being processed by the distributed computer system differs from a lowest power state of a second job of all jobs being processed by the distributed computer system.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining whether a power consumed by the distributed computer system is greater than a power allocated to the distributed computer system, responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, responsive to determining all jobs being processed by the distributed computer system are processing at a lowest power state for each job, suspending a job having a lowest priority among all jobs being processed by the distributed computer system and responsive to suspending the job having the lowest priority among all jobs being processed by the distributed computer system, determining whether the power consumed by the distributed computer system is greater than the power allocated to the system and responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the system, terminating a second job having a second lowest priority.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including determining whether a power consumed by a distributed computer system is less than a power allocated to the distributed computer system, responsive to determining a power consumed by the distributed computer system is less than the power allocated to the distributed computer system, determining whether unallocated resources are available, responsive to determining unallocated resources are available, determining whether sufficient resources for a highest priority job in a job queue are available, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, when sufficient resources for a highest priority job in a job queue are available, determining whether a power to process the highest priority job is available and when the power to process the highest priority job is available, starting the highest priority job.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including determining whether a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, responsive to determining the power allocated to the distributed computer system is less than the power consumed by the distributed computer system, determining whether a preference exists to allocate excess power to jobs in the job queue, responsive to determining no preference exists to allocate excess power to jobs in the job queue, determining whether unallocated resources are available, responsive to determining unallocated resources are available, determining whether sufficient resources for a highest priority job in a job queue are available, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, when sufficient resources for a highest priority job in a job queue are available, determining whether a power to process the highest priority job is available and when the power to process the highest priority job is available, starting the highest priority job.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including determining whether a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, responsive to determining the power allocated to the distributed computer system is less than the power consumed by the distributed computer system, determining whether a preference exists to allocate excess power to jobs in the job queue, responsive to determining a preference exists to allocate excess power to jobs in the job queue, allocating an excess power to one or more jobs being processed by the distributed computer system.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including determining whether a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, responsive to determining a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, determining whether unallocated resources are available, responsive to determining unallocated resources are available, determining whether sufficient resources for a highest priority job in a job queue are available, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, when sufficient resources for a highest priority job in a job queue are available, determining whether a power to process the highest priority job is available and when the power to process the highest priority job is available, starting the highest priority job, wherein determining whether a power to process the highest priority job is available includes determining whether a power required to process the highest priority job is less than the power allocated to the distributed computer system minus the power consumed by the distributed computer system minus a first system guardband.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including determining whether a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, responsive to determining a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, determining whether unallocated resources are available, responsive to determining unallocated resources are available, determining whether sufficient resources for a highest priority job in a job queue are available, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, when sufficient resources for a highest priority job in a job queue are available, determining whether a power to process the highest priority job is available, and when the power to process the highest priority job is available, starting the highest priority job, wherein determining whether a power to process the highest priority job is available includes determining whether a power required to process the highest priority job is less than the power allocated to the distributed computer system minus the power consumed by the distributed computer system minus a first system guardband and the first system guardband includes a predefined amount of power not to be allocated to one or more jobs in order to prevent fluctuations in power consumption of the distributed computer system from exceeding the power allocated to the distributed computer system.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including determining whether a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, responsive to determining a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, determining whether unallocated resources are available, responsive to determining unallocated resources are available, determining whether sufficient resources for a highest priority job in a job queue are available, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, when sufficient resources for a highest priority job in a job queue are available, determining whether a power to process the highest priority job is available and when the power to process the highest priority job is available, starting the highest priority job, wherein starting the highest priority job includes starting a new job from the job queue.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including determining whether a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, responsive to determining a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, determining whether unallocated resources are available, responsive to determining unallocated resources are available, determining whether sufficient resources for a highest priority job in a job queue are available, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, when sufficient resources for a highest priority job in a job queue are available, determining whether a power to process the highest priority job is available and when the power to process the highest priority job is available, starting the highest priority job, wherein starting the highest priority job includes resuming a suspended job in the job queue.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including determining whether a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, responsive to determining a power allocated to a distributed computer system is less than a power consumed by the distributed computer system, determining whether unallocated resources are available, responsive to determining unallocated resources are available, determining whether sufficient resources for a highest priority job in a job queue are available, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, when sufficient resources for a highest priority job in a job queue are available, determining whether a power to process the highest priority job is available and when the power to process the highest priority job is available, starting the highest priority job, wherein the job queue is a prioritized queue configured to include one or both of (1) one or more new jobs or (2) one or more suspended jobs.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining (1) a power allocated to the distributed computer system minus a first system guardband is less than a power consumed by the distributed computer system and (2) the power consumed by the distributed computer system is less than the power allocated to the system plus a second system guardband, calculating a freed power, wherein the freed power is an amount of power that may be made available to allocate to one or more jobs in a job queue by reducing amount of power consumed by one or more jobs being processed by the distributed computer system, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, responsive to calculating the freed power, determining whether a power required for a highest priority job in the job queue is less than the freed power plus an unallocated power, wherein the unallocated power is power allocated to the distributed computer system but unallocated to a job being processed by the distributed computer system, responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power, reducing the amount of power consumed by one or more jobs being processed by the distributed computer system and responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power and reducing the amount of power consumed by one or more jobs being processed by the distributed computer system, starting the highest priority job.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining (1) a power allocated to the distributed computer system minus a first system guardband is less than a power consumed by the distributed computer system and (2) the power consumed by the distributed computer system is less than the power allocated to the system plus a second system guardband, calculating a freed power, wherein the freed power is an amount of power that may be made available to allocate to one or more jobs in a job queue by reducing amount of power consumed by one or more jobs being processed by the distributed computer system, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, responsive to calculating the freed power, determining whether a power required for a highest priority job in the job queue is less than the freed power plus an unallocated power, wherein the unallocated power is power allocated to the distributed computer system but unallocated to a job being processed by the distributed computer system, responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power, reducing the amount of power consumed by one or more jobs being processed by the distributed computer system and responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power and reducing the amount of power consumed by one or more jobs being processed by the distributed computer system, starting the highest priority job, wherein starting the highest priority job includes starting a new job from the job queue.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining (1) a power allocated to the distributed computer system minus a first system guardband is less than a power consumed by the distributed computer system and (2) the power consumed by the distributed computer system is less than the power allocated to the system plus a second system guardband, calculating a freed power, wherein the freed power is an amount of power that may be made available to allocate to one or more jobs in a job queue by reducing amount of power consumed by one or more jobs being processed by the distributed computer system, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, responsive to calculating the freed power, determining whether a power required for a highest priority job in the job queue is less than the freed power plus an unallocated power, wherein the unallocated power is power allocated to the distributed computer system but unallocated to a job being processed by the distributed computer system, responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power, reducing the amount of power consumed by one or more jobs being processed by the distributed computer system and responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power and reducing the amount of power consumed by one or more jobs being processed by the distributed computer system, starting the highest priority job, wherein starting the highest priority job includes resuming a suspended job in the job queue.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining (1) a power allocated to the distributed computer system minus a first system guardband is less than a power consumed by the distributed computer system and (2) the power consumed by the distributed computer system is less than the power allocated to the system plus a second system guardband, calculating a freed power, wherein the freed power is an amount of power that may be made available to allocate to one or more jobs in a job queue by reducing amount of power consumed by one or more jobs being processed by the distributed computer system, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, after calculating the freed power, generating a power estimate for the highest priority job, responsive to calculating the power estimate, determining whether a power required for a highest priority job in the job queue is less than the freed power plus an unallocated power, wherein the unallocated power is power allocated to the distributed computer system but unallocated to a job being processed by the distributed computer system, responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power, reducing the amount of power consumed by one or more jobs being processed by the distributed computer system and responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power and reducing the amount of power consumed by one or more jobs being processed by the distributed computer system, starting the highest priority job.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining (1) a power allocated to the distributed computer system minus a first system guardband is less than a power consumed by the distributed computer system and (2) the power consumed by the distributed computer system is less than the power allocated to the system plus a second system guardband, calculating a freed power, wherein the freed power is an amount of power that may be made available to allocate to one or more jobs in a job queue by reducing amount of power consumed by one or more jobs being processed by the distributed computer system, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, responsive to calculating the freed power, determining whether a power required for a highest priority job in the job queue is less than the freed power plus an unallocated power, wherein the unallocated power is power allocated to the distributed computer system but unallocated to a job being processed by the distributed computer system, responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power, reducing the amount of power consumed by one or more jobs being processed by the distributed computer system and responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power and reducing the amount of power consumed by one or more jobs being processed by the distributed computer system, starting the highest priority job, when sufficient resources are not available to process the highest priority job and sufficient resources are available to process a second highest priority job, starting the second highest priority job.
A non-transitory computer readable storage medium having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including determining (1) a power allocated to the distributed computer system minus a first system guardband is less than a power consumed by the distributed computer system and (2) the power consumed by the distributed computer system is less than the power allocated to the system plus a second system guardband, calculating a freed power, wherein the freed power is an amount of power that may be made available to allocate to one or more jobs in a job queue by reducing amount of power consumed by one or more jobs being processed by the distributed computer system, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, responsive to calculating the freed power, determining whether a power required for a highest priority job in the job queue is less than the freed power plus an unallocated power, wherein the unallocated power is power allocated to the distributed computer system but unallocated to a job being processed by the distributed computer system, responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power, reducing the amount of power consumed by one or more jobs being processed by the distributed computer system and responsive to determining the power required for the highest priority job in the job queue is less than the freed power plus the unallocated power and reducing the amount of power consumed by one or more jobs being processed by the distributed computer system, starting the highest priority job, when the power required for the highest priority job is greater than the freed power plus the unallocated power and a power required for a second highest priority job is less than the freed power plus the unallocated power, starting the second highest priority job.
1. A non-transitory computer readable storage medium, which is not a signal bearing medium, having stored thereon instructions, the instructions being executable by one or more processors of a distributed computer system to perform operations including:
determining whether a power consumed by the distributed computer system is greater than a power allocated to the distributed computer system;
responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by the one or more processors of the distributed computer system, and wherein each job includes a user power policy set forth by a job owner, wherein the user power policy for each job includes a parameter that indicates a minimum amount of power at which each job can run, wherein each job's lowest power state is constrained by the user power policy for that job and wherein the user power policy indicates whether the job is suspendable; and
responsive to determining all jobs being processed by the distributed computer system are processing at a lowest power state for each job,
identifying, among all jobs being processed by the distributed computer system, one or more jobs that are suspendable based on the policy;
suspending, starting with a lowest priority, the one or more jobs that the user policy indicates as suspendable;
allocating, to currently running jobs, excess power obtained as a result of suspending the one or more jobs, with allocation being done equally, proportionally or according to job priority.
prior to suspending the job having the lowest priority among all jobs being processed by the distributed computer system and responsive to determining all jobs being processed by the distributed computer system are processing at the lowest power state for each job, checkpointing the job having a lowest priority among all jobs being processed by the distributed computer system.
3. The non-transitory computer readable storage medium of claim 1, wherein a lowest power state of a first job of all jobs being processed by the distributed computer system differs from a lowest power state of a second job of all jobs being processed by the distributed computer system.
after suspending the job having the lowest priority among all jobs being processed by the distributed computer system, recalculating the power consumed by the distributed computer system.
5. The non-transitory computer readable storage medium of claim 4, further comprising:
after recalculating the power consumed by the distributed computer system, suspending a second job having a second lowest priority.
6. The non-transitory computer readable storage medium of claim 1, further comprising:
responsive to suspending the job having the lowest priority among all jobs being processed by the distributed computer system, determining whether the power consumed by the distributed computer system is greater than the power allocated to the system; and
responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the system, terminating a second job having a second lowest priority.
7. The non-transitory computer-readable storage medium defined in claim 1, wherein the power policy for each job includes one or more parameters that indicate at least one of:
a maximum amount of power at which each job can run,
a minimum frequency at which the one or more processors processing each job can operate,
a maximum frequency at which the one or more processors processing each job can operate, or
whether for each job can be subjected to a power limit.
determining whether a power consumed by a distributed computer system is greater than a power allocated to the distributed computer system;
responsive to determining the power consumed by the distributed computer system is greater than the power allocated to the distributed computer system, determining whether all jobs being processed by the distributed computer system are processing at a lowest power state for each job, wherein a job includes one or more calculations performed by one or more processors of the distributed computer system, and wherein each job includes a user power policy set forth by a job owner, wherein the user power policy for each job includes a parameter that indicates a minimum amount of power at which each job can run, wherein each job's lowest power state is constrained by the user power policy for that job and wherein the user power policy indicates whether the job is suspendable; and
suspending, starting with a lowest priority, the one or more jobs that the user policy indicates as suspendable; and
10. The method defined in claim 8, wherein a lowest power state of a first job of all jobs being processed by the distributed computer system differs from a lowest power state of a second job of all jobs being processed by the distributed computer system.
12. The method defined claim 11 further comprising:
13. The method defined claim 8, further comprising:
14. The method defined in claim 8, wherein the power policy for each job includes one or more parameters that indicate at least one of:
US14/582,772 2014-08-22 2014-12-24 Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system Active US10289183B2 (en)
US201462040576P true 2014-08-22 2014-08-22
US14/582,772 US10289183B2 (en) 2014-08-22 2014-12-24 Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system
CN201580040030.8A CN107003706A (en) 2014-08-22 2015-06-18 Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system
PCT/US2015/036403 WO2016028374A1 (en) 2014-08-22 2015-06-18 Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system
KR1020177002008A KR20170021864A (en) 2014-08-22 2015-06-18 Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system
EP15833856.6A EP3183628A4 (en) 2014-08-22 2015-06-18 Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system
JP2017510311A JP6386165B2 (en) 2014-08-22 2015-06-18 Method and apparatus for managing job and not be interrupted job may be interrupted when there is a change in the power allocation to the distributed computer system
US20160054781A1 US20160054781A1 (en) 2016-02-25
US10289183B2 true US10289183B2 (en) 2019-05-14
ID=55348281
US14/582,783 Pending US20160054783A1 (en) 2014-08-22 2014-12-24 Method and apparatus to generate and use power, thermal and performance characteristics of nodes to improve energy efficiency and reducing wait time for jobs in the queue
US14/582,743 Pending US20160054779A1 (en) 2014-08-22 2014-12-24 Managing power performance of distributed computing systems
US14/582,756 Active US9927857B2 (en) 2014-08-22 2014-12-24 Profiling a job power and energy consumption for a data processing system
US14/582,772 Active US10289183B2 (en) 2014-08-22 2014-12-24 Methods and apparatus to manage jobs that can and cannot be suspended when there is a change in power allocation to a distributed computer system
US14/582,764 Active 2035-03-08 US9921633B2 (en) 2014-08-22 2014-12-24 Power aware job scheduler and manager for a data processing system
US14/582,795 Active 2035-03-23 US9575536B2 (en) 2014-08-22 2014-12-24 Methods and apparatus to estimate power performance of a job that runs on multiple nodes of a distributed computer system
US (6) US20160054783A1 (en)
EP (3) EP3183654A4 (en)
JP (2) JP2017531245A (en)
KR (2) KR20170021338A (en)
CN (3) CN106537348A (en)
WO (3) WO2016028371A1 (en)
BR112016024645A2 (en) * 2016-08-11 2018-05-15 Ericsson Telecomunicacoes Sa ? Method for managing a data processing task requested by a client, cloud management node, computer program and carrier?
WO2019014321A1 (en) * 2017-07-12 2019-01-17 Wiliot, LTD. Energy-aware computing system
US6125450A (en) 1996-12-19 2000-09-26 Intel Corporation Stop clock throttling in a computer processor through disabling bus masters
WO2002019078A2 (en) 2000-08-31 2002-03-07 Advanced Micro Devices, Inc. System and method for monitoring and controlling a power manageable resource
US20030041272A1 (en) 2001-08-27 2003-02-27 Nguyen Don J. Method and apparatus for regulating the voltage supplied to a computer system
US20030163745A1 (en) 2002-02-27 2003-08-28 Kardach James P. Method to reduce power in a computer system with bus master devices
US20030221026A1 (en) 2002-05-22 2003-11-27 Sean Newman Automatic power saving facility for network devices
US20040025063A1 (en) 2002-07-31 2004-02-05 Compaq Information Technologies Group, L.P. A Delaware Corporation Power management state distribution using an interconnect
US20040022225A1 (en) 2002-08-02 2004-02-05 Jie Liang Low power packet detector for low power WLAN devices
US6745335B1 (en) 1999-10-12 2004-06-01 Nec Corporation Power consumption control of multiprocessor system using separate timers for monitoring processor load rise and drop
WO2004070619A2 (en) 2003-01-30 2004-08-19 Newisys, Inc. Methods and apparatus for distributing system management signals
US20050113103A1 (en) 2003-11-21 2005-05-26 Snowden Gregory O. Portable communication device and network and methods therefore
US20050273633A1 (en) 2004-06-02 2005-12-08 Intel Corporation Hardware coordination of power management activities
US7143300B2 (en) 2001-07-25 2006-11-28 Hewlett-Packard Development Company, L.P. Automated power management system for a network of computers
US20070220293A1 (en) 2006-03-16 2007-09-20 Toshiba America Electronic Components Systems and methods for managing power consumption in data processors using execution mode selection
US20080104428A1 (en) 2006-10-30 2008-05-01 Naffziger Samuel D Inter-die power manager and power management method
US20080178019A1 (en) 2007-01-19 2008-07-24 Microsoft Corporation Using priorities and power usage to allocate power budget
US20090113221A1 (en) 2007-10-29 2009-04-30 Microsoft Corporation Collaborative power sharing between computing devices
US20090254660A1 (en) 2008-04-07 2009-10-08 Hanson James E Systems and methods for coordinated management of power usage and runtime performance in performance-managed computing environments
US20090271046A1 (en) 2008-04-24 2009-10-29 Sun Microsystems, Inc. Estimating a power utilization of a computer system
US20100106985A1 (en) 2008-01-03 2010-04-29 Broadcom Corporation System and method for global power management in a power over ethernet chassis
US20100205469A1 (en) 2009-02-06 2010-08-12 Mccarthy Clifford A Power budgeting for a group of computer systems
US20100235840A1 (en) 2009-03-10 2010-09-16 International Business Machines Corporation Power management using dynamic application scheduling
US20100257531A1 (en) 2009-04-03 2010-10-07 International Business Machines, Corporation Scheduling jobs of a multi-node computer system based on environmental impact
US20110022857A1 (en) 2009-07-24 2011-01-27 Sebastien Nussbaum Throttling computational units according to performance sensitivity
US20110178652A1 (en) 2010-01-15 2011-07-21 International Business Machines Corporation Dynamically Adjusting an Operating State of a Data Processing System Running Under a Power Cap
US20110271283A1 (en) * 2010-04-28 2011-11-03 International Business Machines Corporation Energy-aware job scheduling for cluster environments
US20120054512A1 (en) 2010-08-25 2012-03-01 International Business Machines Corporation Managing Server Power Consumption In A Data Center
US20120060170A1 (en) 2009-05-26 2012-03-08 Telefonaktiebolaget Lm Ericsson (Publ) Method and scheduler in an operating system
US20120072745A1 (en) 2010-09-22 2012-03-22 International Business Machines Corporation Server power management with automatically-expiring server power allocations
US20120084580A1 (en) 2010-04-07 2012-04-05 International Business Machines Power Budget Allocation in Multi-Processor Systems
US20120165963A1 (en) 2010-12-23 2012-06-28 DongA one Corporation Apparatus for controlling power of sensor nodes based on estimation of power acquisition and method thereof
US20120324264A1 (en) 2009-06-30 2012-12-20 International Business Machines Corporation Priority-Based Power Capping in Data Processing Systems
US20130103968A1 (en) 2005-08-25 2013-04-25 Apple Inc. Methods and apparatuses for dynamic power control
US20130124885A1 (en) 2011-11-14 2013-05-16 Microsoft Corporation Estimating and managing power consumption of computing devices using power models
US20130185576A1 (en) 2006-05-04 2013-07-18 Michael A. Brundridge Power profiling application for managing power allocation in an information handling system
US20130212410A1 (en) 2012-02-10 2013-08-15 University Of Florida Research Foundation, Incorporated Renewable energy control systems and methods
US20130227557A1 (en) 2012-02-29 2013-08-29 Jiri Pechanec Systems and methods for providing priority build execution in a continuous integration system
WO2013172816A1 (en) 2012-05-14 2013-11-21 Intel Corporation Managing the operation of a computing system
US20130339776A1 (en) 2012-06-13 2013-12-19 Cisco Technology, Inc. System and method for automated service profile placement in a network environment
US20140006534A1 (en) 2012-06-27 2014-01-02 Nilesh K. Jain Method, system, and device for dynamic energy efficient job scheduling in a cloud computing environment
US20140075222A1 (en) 2008-04-21 2014-03-13 Adaptive Computing Enterprises, Inc. System and Method for Managing Energy Consumption in a Compute Environment
WO2014053610A1 (en) 2012-10-03 2014-04-10 Imperial Innovations Limited Frequency estimation
US20140114107A1 (en) 2012-10-24 2014-04-24 Lummus Technology Inc. Use of hydrocarbon diluents to enhance conversion in a dehydrogenation process at low steam/oil ratios
US20140137121A1 (en) 2012-10-05 2014-05-15 Hitachi, Ltd. Job management system and job control method
US20140214107A1 (en) 2006-10-20 2014-07-31 Cardiac Pacemakers, Inc. Dynamic battery management in an implantable device
US20140245054A1 (en) 2012-11-07 2014-08-28 Dell Products L.P. Power over ethernet dynamic power allocation system
US20140282425A1 (en) 2013-03-18 2014-09-18 Microsoft Corporation Application testing and analysis
US20140317635A1 (en) 2011-11-24 2014-10-23 Hitachi, Ltd. Computer system and divided job processing method and program
US20150169026A1 (en) 2012-05-17 2015-06-18 Devadatta V. Bodas Managing power consumption and performance of computing systems
US20150177814A1 (en) 2013-12-23 2015-06-25 Dell, Inc. Predictive power capping and power allocation to computing nodes in a rack-based information handling system
US20150248312A1 (en) 2014-02-28 2015-09-03 Lenovo Enterprise Solutions (Singapore) Pte. Ltd. Performance-aware job scheduling under power constraints
US20160011914A1 (en) 2013-06-20 2016-01-14 Seagate Technology Llc Distributed power delivery
JP5029823B2 (en) * 2007-09-06 2012-09-19 コニカミノルタビジネステクノロジーズ株式会社 Image forming apparatus, the power consumption management system, the power consumption management method and program
CN101419495B (en) * 2007-10-22 2012-05-30 国际商业机器公司 Method and device for reducing I /O power in computer system and the computer system
JP2011013764A (en) * 2009-06-30 2011-01-20 Hitachi Ltd Method, system and program for managing power consumption
CN101694631B (en) * 2009-09-30 2016-10-05 曙光信息产业(北京)有限公司 Real-time job scheduling system and method
CN102685808A (en) * 2012-05-18 2012-09-19 电子科技大学 Distribution type clustering method based on power control
CN102819460B (en) * 2012-08-07 2015-05-20 清华大学 Budget power guidance-based high-energy-efficiency GPU (Graphics Processing Unit) cluster system scheduling algorithm
JP5787365B2 (en) * 2012-09-18 2015-09-30 Ｎｅｃフィールディング株式会社 Power controller, power control system, power control method, and program
US9250858B2 (en) * 2013-02-20 2016-02-02 International Business Machines Corporation Dual-buffer serialization and consumption of variable-length data records produced by multiple parallel threads
2014-12-24 US US14/582,783 patent/US20160054783A1/en active Pending
2014-12-24 US US14/582,743 patent/US20160054779A1/en active Pending
2014-12-24 US US14/582,756 patent/US9927857B2/en active Active
2014-12-24 US US14/582,772 patent/US10289183B2/en active Active
2014-12-24 US US14/582,764 patent/US9921633B2/en active Active
2014-12-24 US US14/582,795 patent/US9575536B2/en active Active
2015-06-17 JP JP2017510501A patent/JP2017531245A/en active Pending
2015-06-17 EP EP15833962.2A patent/EP3183654A4/en active Pending
2015-06-17 KR KR1020177002070A patent/KR20170021338A/en active Search and Examination
2015-06-17 WO PCT/US2015/036294 patent/WO2016028371A1/en active Application Filing
2015-06-17 CN CN201580040949.7A patent/CN106537348A/en active Search and Examination
2015-06-18 CN CN201580040030.8A patent/CN107003706A/en active Search and Examination
2015-06-18 CN CN201580040005.XA patent/CN106537287A/en active Search and Examination
2015-06-18 JP JP2017510311A patent/JP6386165B2/en active Active
2015-06-18 EP EP15834071.1A patent/EP3183629A4/en active Pending
2015-06-18 WO PCT/US2015/036435 patent/WO2016028375A1/en active Application Filing
2015-06-18 EP EP15833856.6A patent/EP3183628A4/en active Pending
2015-06-18 WO PCT/US2015/036403 patent/WO2016028374A1/en active Application Filing
2015-06-18 KR KR1020177002008A patent/KR20170021864A/en active Search and Examination
"Office Action issued in U.S. Appl. No. 14/582,756, dated Aug. 25, 2016", 26.
"Office Action issued in U.S. Appl. No. 14/582,756, dated Feb. 16, 2016", 22.
"Office Action issued in U.S. Appl. No. 14/582,756, dated Jan. 12, 2017", 26.
"Office Action issued in U.S. Appl. No. 14/582,764, dated Jul. 14, 2017", 12.
"Office Action issued in U.S. Appl. No. 14/582,764, dated Jun. 13, 2016", 16.
"Office Action issued in U.S. Appl. No. 14/582,764, dated Nov. 16, 2016", 12.
"Office Action issued in U.S. Appl. No. 14/582,795, dated Jun. 1, 2016", 14.
2013. Slurm Workload Manager. (Nov. 2013). http://slurm.schedmd.com.
Abstract from W00219078, corresponding to reference CN 1449517A cited in CNOA above, 7 pgs.
Activity Monitor Demystified, Skvorc, 2012.
Alvarruiz, F., de Alfonso, C., Caballer, M. and Hernández, V. 2012. An Energy Manager for High Performance Computer Clusters. ISPA '12 Proceedings of the 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications.
Anonymous: "Benchmark (computing)-Wikipedia", Aug. 15, 2014 (Aug. 15, 2014), XP055458168, Retrieved from the Internet: URL:https://en.wikipedia.org/w/index.php?title=Benchmark (computing)&oldid=621396505.
Anonymous: "Benchmark (computing)—Wikipedia", Aug. 15, 2014 (Aug. 15, 2014), XP055458168, Retrieved from the Internet: URL:https://en.wikipedia.org/w/index.php?title=Benchmark (computing)&oldid=621396505.
Bhattacharya, A. 2013. Constraints and Techniques for Software Power Management in Production Clusters. Technical Report No. UCB/EECS-2013-110, Electrical Engineering and Computer Sciences, University of California at Berkeley. http://www.eecs.berkeley.ed.
Brehm, M. 2013. Energy Aware Scheduling SuperMUC@ LRZ. Application Support Group. Leibniz Supercomputing Centre. http://www.autotune-project.eu/system/files/Matthias_Brehm_Energietag.pdf.
Cai, C., Wang, L., Khan, S. and Tao, J. 2011. Energy-aware High Performance Computing-A Taxonomy Study. Parallel and Distributed Systems (ICPADS), 2011 IEEE 17th International Conference on. (Tainan, Taiwan. Dec. 7, 2009.
Cai, C., Wang, L., Khan, S. and Tao, J. 2011. Energy-aware High Performance Computing—A Taxonomy Study. Parallel and Distributed Systems (ICPADS), 2011 IEEE 17th International Conference on. (Tainan, Taiwan. Dec. 7, 2009.
Department of Energy. 2013. CORAL procurement benchmarks. LLNL-PRE-637694. (May 31, 2013). https://asc.llnl.gov/CORAL-benchmarks/CORALBenchmarksProcedure-v26.pdf.
Deva Bodas, et al, "Simple Power-Aware Scheduler to Limit Power Consumption by HPC System within a Budget," Intel Corporation (U.S. Appl. No. 62/040,576). 10 pages.
Etinski, M., Corbalan, J. and Labarta, J. Power-Aware Parallel Job Scheduling. Barcelona Supercomputing Center. http://nsfcac.rutgers.edu/GreenHPC/EEHiPC/eehipc_etinski.pdf.
European Search Report and Search Opinion Received for EP Application No. 15833856.6, dated May 14, 2018, 14 pages.
European Search Report and Search Opinion Received for EP Application No. 15833962.2, dated Mar. 19, 2018, 9 pages.
European Search Report and Search Opinion Received for EP Application No. 15834071.1, dated Mar. 29, 2018, 14 pages.
First Office Action (and English translation) from CN Application No. 200410101227.1, dated Dec. 1, 2006, 27 pgs.
HP, Intel, Microsoft, Phoenix, Toshiba. 2011. Advanced Configuration and Power Interface Specification Revision 5.0. http://www.acpi.info/DOWNLOADS/ACPIspec50.pdf.
Intel® Corp. 2014. Intel® 64 and IA-32 Architectures Software Developer Manuals. http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html.
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2015/036294, dated Mar. 9, 2017, 9 pages.
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2015/036403, dated Mar. 9, 2017, 10 pages.
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2015/036435, dated Mar. 9, 2017, 13 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2015/036294, dated Sep. 25, 2015, 10 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2015/036435, dated Sep. 18, 2015, 14 pages.
Karthik Ganesan et al: "System-level max power (SYMPO)", Proceedings of the 19th International Conference on Parallel Architectures and Compilation Techniques, Pact '10, ACM Press, New York, New York, USA, Sep. 11, 2010 (Sep. 11, 2010), pp. 19-28.
Laboratory. JSSPP 2013, total 20 pages. http://www.cs.huji.ac.il/.about.feit/parsched/jsspp13/zhou.pdf.
Lefurgy, C., Allen-Ware, M., Carter, J., El-Essawy, W., Felter, W., Ferreira, A., Huang, W., Hylick, A., Keller, T., Rajamani, K., Rawson F. and Rubio, J. 2011. Energy-Efficient Data Centers and Systems. 2011 IEEE International Symposium on Workload Charac.
Mämmelä, O., Majanen, M., Basmadjian, R., De Meer, H., Giesler, A. and Homberg, W. Energy-aware job scheduler for high-performance computing. Computer Science-Research and Development 27, No. 4 (2012): 265-275.
Mämmelä, O., Majanen, M., Basmadjian, R., De Meer, H., Giesler, A. and Homberg, W. Energy-aware job scheduler for high-performance computing. Computer Science—Research and Development 27, No. 4 (2012): 265-275.
Matthieu, H. Power capping in SLURM. Green days @ life, (Nov. 2013).
Office Action from U.S. Appl. No. 10/871,638, dated Sep. 28, 2006, 13 pgs.
Office Action from UK Patent Application No. 0425264.9, dated Sep. 22, 2006, 6 pgs.
Office Action from UK Patent Application No. 0609876.8, dated Sep. 22, 2006, 5 pgs.
Office Action issued in U.S. Appl. No. 10/859,892, dated Dec. 6, 2006, 9 pgs.
Office Action issued in U.S. Appl. No. 10/859,892, dated Jul. 25, 2006, 10 pgs.
PCT International Search Report and Written Opinion of the International Searching Authority for Int'l Application No. PCT/US2015/036403, dated Sep. 4, 2015 (14 pages).
PCT International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2012/038405, dated May 17, 2012, 3 pages.
PCT Notification concerning Transmittal of International Preliminary Report on Patentability (Chapter I of the Patent Cooperation Treaty) for PCT Counterpart Application No. PCT/US2012/037812, 7 pgs., (dated Nov. 27, 2014).
PCT Notification concerning Transmittal of International Preliminary Report on Patentability (Chapter I of the Patent Cooperation Treaty) for PCT Counterpart Application No. PCT/US2012/038405, 7 pages (Nov. 27, 2014).
PCT Notification concerning Transmittal of International Preliminary Report on Patentability (Chapter I of the Patent Cooperation Treaty) for PCT Counterpart Application No. PCT/US2012/038405, 7 pgs., (dated Nov. 27, 2014).
PCT Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority or the Declaration for PCT Counterpart Application No. PCT/US2012/037812, 10 pgs., (dated Jan. 23, 2013).
PCT Written Opinion of the International Searching Authority for PCT Counterpart Application No. PCT/US2012/038405 5 pgs., (dated Feb. 28, 2013).
Rountree, B., Ahn, D., de Supinski, B., Lowenthal, D. and Schulz, M. 2012. Beyond DVFS: A First Look at Performance Under a Hardware-Enforced Power Bound. 8th Workshop on High-Performance, Power-Aware Computing (HPPAC). (May 2012). https://e-reports-ext.II.
Silberschatz et al., Operating System Concepts, 2013, John Wiley & Sons, Inc.
Supplementary European Search Report for European Patent Application No. EP15833962, dated Mar. 9, 2018.
The Office Action for counterpart U.S. Appl. No. 14/582,743, 38 pages, dated Mar. 9, 2017.
The Office Action for U.S. Appl. No. 14/582,764, 13 pages, dated Mar. 14, 2017.
The Office Action for U.S. Appl. No. 14/582,772, 21 pages, dated Nov. 11, 2017.
The Office Action for U.S. Appl. No. 14/582,783, 42 pages, dated Mar. 9, 2017.
Yoo, A., Jette, M. and Grondona, M. 2003. SLURM: Simple Linux utility for resource management. In, Feitelson, D., Rudolph, L. and Schwiegelshohn, U. editors. Job Scheduling Strategies for Parallel Processing. 9th Springer Verlag International Workshop, JS.
Zhai et al., HaPPy: Hyperthread-aware Power Profiling Dynamically, 2014 USENIX Annual Technical Conference, Jun. 2014.
Zhou, Z., Lan, Z., Tang, W. and Desai, N. 2013. Reducing Energy Costs for IBM Blue Gene/P via Power-Aware Job Scheduling. Department of Computer Science, Illinois Institute of Technology; Mathematics and Computer Science Division, Argonne National Laborato.
CN106537348A (en) 2017-03-22
JP2017531245A (en) 2017-10-19
US20160054779A1 (en) 2016-02-25
US20160054774A1 (en) 2016-02-25
WO2016028371A1 (en) 2016-02-25
EP3183654A1 (en) 2017-06-28
EP3183629A4 (en) 2018-05-02
US20160054783A1 (en) 2016-02-25
KR20170021864A (en) 2017-02-28
US20160054781A1 (en) 2016-02-25
KR20170021338A (en) 2017-02-27
WO2016028374A1 (en) 2016-02-25
US9575536B2 (en) 2017-02-21
CN107003706A (en) 2017-08-01
CN106537287A (en) 2017-03-22
WO2016028375A1 (en) 2016-02-25
EP3183628A4 (en) 2018-06-13
EP3183654A4 (en) 2018-04-18
EP3183628A1 (en) 2017-06-28
US20160054780A1 (en) 2016-02-25
US9921633B2 (en) 2018-03-20
US9927857B2 (en) 2018-03-27
US20160054775A1 (en) 2016-02-25
JP6386165B2 (en) 2018-09-05
JP2017530449A (en) 2017-10-12
EP3183629A1 (en) 2017-06-28
JP5157717B2 (en) 2013-03-06 Program for the virtual machine system having a virtual machine system and a virtual battery with a virtual battery
US8566285B2 (en) 2013-10-22 Method and system for scheduling and controlling backups in a computer system
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:RAJAPPA, MURALIDHAR;HOFFMAN, ANDY;BODAS, DEVADATTA;AND OTHERS;SIGNING DATES FROM 20150106 TO 20150123;REEL/FRAME:037131/0394