Patent Description:
Many server workloads exhibit stable homogeneous load behavior with a fixed amount of work utilizing various load balancing techniques. These workloads are usually not latency critical and do not have data dependent threads, i.e., the time to process an individual request does not matter as long as average throughput is kept constant. System-on-chip (SoC) power management algorithms need to identify these throughput-based workloads so that frequency can be lowered to minimize idle times without degrading performance, thereby achieving optimal power efficiency. Existing power management algorithms either look at the behavior of individual instruction processors (e.g., single processor), or contain ad-hoc solutions to model the interaction between SoC and workloads, making it unable to differentiate between throughput and latency critical workloads and resulting in sub-optimal operating points.

For example, a collection of local optimization algorithms are unlikely to achieve a globally optimal operating state targeting a specific Quality-of-Service (QoS). Instead, this solution would lead to poor performance and power efficiency. Therefore, as SoCs are becoming increasingly complex, it is becoming important to understand SoC and workload interactions and manage power in a scalable way while optimizing for a global QoS objective.

) discloses power control logic of a processor which may be adapted to analyze activity levels of one or more cores and/or other processing engines of the processor to determine the concurrency of processing on such cores or other engines. Based at least in part on such information, the power control logic can determine an appropriate performance state at which one or more of the engines may operate to provide for energy efficient operation levels. ) discloses a system for conserving resources in a multi-processor computing environment which monitors usage of the processors in the environment and maintains a sorted list of usage changes that occur in each of a plurality of periodic intervals. The system uses the sorted list to predict, according to configurable parameters, how many processors will need to be available during a subsequent interval.

The disclosure will be more readily understood in view of the following description when accompanied by the below figures and wherein like reference numerals represent like elements, wherein:.

Briefly, methods and apparatus provide power efficiency optimization in throughput-based workloads. In one example, a method for optimizing power efficiency in a computing device is described. The computing device may be a server, for example. The computing device determines processing core activity deviation data for processing cores (e.g., CPU, GPU, etc.) in the computing device based on processing core activity data from the processing cores. The computing device adjusts a frequency of at least one of the processing cores in the computing device based on the processing core activity deviation data.

The computing device determines whether the processing core activity deviation data indicates a homogenous workload for the processing cores. In response to determining that the processing core activity deviation data indicates the homogenous workload for the processing cores, the computing device decreases the frequency of the at least one of the processing cores.

In determining the processing core activity deviation data, the computing device determines a number of the processing cores that are active, and an expected number of the processing cores that will be active. The computing device then determines an activity deviation between the number of the processing cores that are active and the expected number of the processing cores that will be active.

Determining the number of the processing cores that are active is based on a processing core activity level for the processing cores. Determining the expected number of the processing cores that will be active is based on a binomial distribution that takes into account the number of the processing cores and the processing core activity level for the processing cores. A processing core is determined to be active when a percentage of active cycles in the processing core in a preceding interval is greater than a threshold, where the preceding interval is based on a moving average window. When determining the processing core activity deviation data, the computing device determines how much a current QoS for the processing cores deviates from a target QoS.

In one example, a system-on-chip or SoC includes a plurality of processing cores and processing core frequency adjusting logic coupled to the plurality of processing cores. The processing core frequency adjusting logic determines processing core activity deviation data for the plurality of processing cores based on processing core activity data from the plurality of processing cores, and adjusts a frequency of at least one of the plurality of processing cores based on the determined processing core activity deviation data.

In another example, a server includes a plurality of processing cores, a network interface, and processing core frequency adjusting logic coupled to the plurality of processing cores and the network interface. The processing core frequency adjusting logic determines processing core activity deviation data for the plurality of processing cores based on processing core activity data from the plurality of processing cores, and adjusts a frequency of at least one of the plurality of processing cores based on the determined processing core activity deviation data.

The present disclosure describes a power management algorithm framework that proposes: <NUM>) a QoS metric for throughput-based workloads; <NUM>) heuristics to differentiate between throughput and latency sensitive workloads; and <NUM>) an algorithm that combines the heuristic and QoS metric to determine target frequency for minimizing idle time and improving power efficiency without any performance degradation.

The present disclosure provides a management algorithm framework to enable optimizing power efficiency in server-class throughput-based workloads while still providing desired performance for latency sensitive workloads. The disclosure aims to achieve power savings by identifying workloads in which one or more processing cores can be run at a lower frequency (and consequently lower power) without a significant negative performance impact. The disclosure addresses at least two problems to make a technique of this kind workable in practice: <NUM>) how to identify workloads whose performance has low frequency sensitivity; and <NUM>) how much can frequency be safely lowered before excessive performance degradation.

The present disclosure presents a solution for, inter alia, <NUM>) differentiating between homogeneous, stable class of workloads with independent threads, with each thread doing a fixed amount of work from workloads that have data dependent threads where the threads are latency-critical; and <NUM>) optimizing power efficiency by reducing idle time in throughput oriented workloads.

For example, a particular QoS algorithm takes a particular "target central processing unit (CPU) idle percentage" (between <NUM>% and <NUM>%) as parameter, and modulates the CPU frequencies to achieve that QoS target. If the CPU idleness is greater than the threshold, the frequency is lowered, and vice versa. For a throughput-based load, desirably the system would want to reduce idleness to zero, taking up all of the slack provided by idle time. However, in practice this will not be possible because the system will need to account for variation in workload behavior and granularity of power state changes. Therefore, the actual idle time being targeted is left as a tunable configuration parameter, to be optimized for the trade-off between power saving and performance degradation.

The QoS metric for system idleness may be calculated using the following formula: <MAT> where N is the total number of processing cores in the system, idle cycles of Core i is the sum of non-C0 (i.e., CC1 and CC6 cycles) of the i-th core, and total cycles is the total number of CPU cycles (both active and idle) at the current CPU frequency during the synchronization management unit (SMU) sample time (<NUM>).

This algorithm makes several assumptions about application behavior.

Since the QoS algorithm requires specific conditions to work properly, and runs the risk of lowering performance if it is turned on for applications where these conditions are not satisfied, it is necessary to have conservative heuristics for when it turns on.

Further observation about throughput-based homogeneous and stable loads is that the distribution of the number of active processing cores at a given moment closely follows a binomial distribution. The number of active cores is defined as those whose percentage of active cycles in a preceding short interval is above a high threshold, e.g., <NUM>%. This is because for such loads, the following characteristics can be observed.

Therefore, as shown in the following formula, when N processing cores are observed at a given point in time, the expected number of active processing cores is given by the binomial distribution with N trials and probability A, which is equal to activity level.

For example, if activity level is <NUM>% and the workload is running on <NUM> processing cores, the expectation would be that exactly <NUM> processing cores are active for the fraction of time is equal to <MAT>, i.e., about <NUM>% of the time.

By measuring the actual distribution of the number of active processing cores, and comparing with the theoretical binomial distribution (where N is the number of processing cores and A the average activity level measured for the measurement time interval), how much the application deviates from the type targeted by idle QoS can be estimated. Specifically, this measure will capture the following.

It is property <NUM>) that adds value relative to measuring only that the workload is stable and homogeneous. Consider, for example, a workload in which a single thread, fully CPU-bound, is run without CPU affinity and consequently scheduled across N processing cores in a round-robin fashion. This workload would appear stable and homogeneous, with each processing core being active for <NUM>/N of the time. However, turning idle QoS would degrade performance, since it is not throughput-based, but rather each time slice executed on a CPU is dependent on the previous one.

This case is successfully identified by the binomial distribution heuristic, since the distribution will show that exactly one processing core is active for <NUM>% of the time (instead of the binomial distribution with A = <NUM>/N). While this is a simple example, similar reasoning shows that for any sort of workload that presents a series of dependent computations, so that reducing CPU frequency would extend the critical path and degrade performance, the distribution will deviate from binomial because the times when individual processing cores are active are correlated.

In order to avoid performance degradation, the algorithm should be turned on only for workloads that have characteristics described in the previous section. The approach is based on the assumption that certain regularities observed for homogeneous (load-balanced) throughput-based applications will not be observed for any other application type. Specifically, the algorithm turns on QoS when the following conditions are observed.

The long time scale required for the stable load level is because for shorter time scales, too much variation in practice is observed. For example, consider the activity diagram from a benchmark program that measures the power and performance characteristics of servers (such as the SpecPower® benchmark provided by Standard Performance Evaluation Corporation (SPEC)), since the length of active intervals varies between a few milliseconds and a few hundred milliseconds, an interval of only a second or two can still have a lot of random variation in how much active time versus idle time it encompasses.

This makes the technique effective for workloads that exhibit a stable load level on a scale of minutes. This will be the case for typical server benchmarks (such as the SpecPower® or SERT™ benchmarks provide by SPEC), which measure stable throttled load levels for several minutes and moreover feature long warmup intervals at each given level before the measurement kicks in.

In order to detect the stable average load, a moving average of activity could be used, but given the time resolution of sampling (<NUM>), this would require a lot of memory. Therefore, instead a calculation of ordinary averages on the scale of <NUM> can be used and a moving average window for these can be used. The size of the basic averaging interval (<NUM>), the moving average window (tens of s), the time threshold for a stable workload (tens of s), and the allowed variations (within a single processing core, between processing cores, and divergence from binomial distribution), are all tunable parameters. Similarly, the algorithm is not used when these conditions no longer apply, and load variation within and between processing cores (computed the same way) exceeds the given thresholds.

Turning to the drawings, one example of the presently disclosed computing device <NUM> is shown in <FIG>. The computing device <NUM> may be a server, for example, or any other suitable device. The computing device <NUM> may be part of a datacenter, or part of a distributed system, such as a cloud-based (e.g., cloud-computing) system.

As indicated in <FIG>, the computing device <NUM> includes a processing core frequency adjusting logic <NUM>. The computing device <NUM> also includes memory <NUM>, such as RAM or ROM or any other suitable memory, which may be used to store parameters, such as parameters associated with the aforementioned algorithms. The memory <NUM> can also store executable instructions that can be accessed and executed by the processing core frequency adjusting logic <NUM>, as described further below.

The computing device <NUM> also includes a processing core <NUM><NUM>, a processing core <NUM><NUM>, a processing core <NUM><NUM>, and a processing core N <NUM>. As recognized by those of ordinary skill in the art, the number of processing cores can vary. The processing cores <NUM>, <NUM>, <NUM>, <NUM> can be, for example, a processing core associated with a CPU, an accelerated processing unit (APU), or a graphics processing unit (GPU). In addition, the processing cores <NUM>, <NUM>, <NUM>, <NUM> can be part of a SoC <NUM>. In other examples, the SoC <NUM> can be a heterogeneous SoC, an APU, a dGPU, a CPU, or a semi-custom SoC. In addition, although not shown, the computing device <NUM> can include additional SoCs with additional processing cores.

The computing device <NUM> also includes I/O device(s) <NUM>, which can include, for example, a display, a keypad, a keyboard, or any other suitable I/O device. The computing device <NUM> also includes one or more network interface(s) <NUM> to communicate with one or more networks. For example, the network interface(s) <NUM> can support communications with, for example, any suitable network that allows communication amongst multiple devices (e.g., Ethernet, WiFi, WAN, Internet).

As indicated in <FIG>, the processing core frequency adjusting logic <NUM>, the processing cores <NUM>, <NUM>, <NUM>, <NUM>, the memory <NUM>, the I/O device(s) <NUM>, and the network interface(s) <NUM> are each connected to a bus <NUM>. The bus <NUM> can be any suitable bus, such as a wired or wireless bus, that allows devices to communicate with each other.

In some embodiments, some or all of the functions of the computing device <NUM> may be performed by any suitable processor or processors that may, for example, execute a software driver, firmware, or any other suitable executable code stored in memory. For example, some or all of the functions of the processing core frequency adjusting logic <NUM> may be performed by any suitable processing core. In one example, the processing core frequency adjusting logic <NUM> reads and executes executable instructions from the memory <NUM>. In some embodiments, the processing core frequency adjusting logic <NUM> may be a CPU, an APU, a GPU, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a microcontroller, as one or more state machines, or as any suitable logic and/or suitable combination of hardware and software, or any other suitable instruction processing device.

The computing device <NUM> with the processing core frequency adjusting logic <NUM> adjusts the frequency of the one or more processing cores <NUM>, <NUM>, <NUM>, <NUM> as described, for example, with respect to <FIG> below. In one example, the computing device <NUM> with the processing core frequency adjusting logic <NUM> can, either additionally or alternatively, adjust the frequency of processing cores associated with another computing device, such as a remote server. For example, the computing device <NUM> may communicate with the remote server over one or more networks.

<FIG> is a more detailed functional block diagram of the processing core frequency adjusting logic <NUM> of <FIG>. As indicated in <FIG>, the processing core frequency adjusting logic <NUM> includes a processing core activity determination logic <NUM>, a processing core workload sensitivity determination logic <NUM>, and a processing core frequency determination logic <NUM>.

The processing core activity determination logic <NUM> obtains (e.g., receives) processing core activity data <NUM> from, for example, one or more processing cores, such as the processing cores <NUM>, <NUM>, <NUM>, <NUM> of <FIG>. The processing core activity data <NUM> may include data that indicates whether a particular processing core is active (e.g., executing a workload) or idle. The processing core activity determination logic <NUM> determines, based on the processing core activity data <NUM>, a processing core activity level for a processing core. The processing core activity level indicates, for example, a processor activity percentage over a period of time. In one example, the processing core activity determination logic <NUM> executes an algorithm that includes Eq. <NUM> described above. The processing core activity determination logic <NUM> provides the processing core activity level as processing core activity level data <NUM> to the processing core workload sensitivity determination logic <NUM>.

The processing core workload sensitivity determination logic <NUM> determines how much a current QoS for one or more processing cores deviates from a target QoS. For example, based on the processing core activity level data <NUM>, the processing core workload sensitivity determination logic <NUM> determines a number of active processing cores. For example, the processing core workload sensitivity determination logic <NUM> can determine that a processing core is active when a percentage of active cycles in a preceding short interval is above a high threshold (e.g., <NUM>%). The processing core workload sensitivity determination logic <NUM> also determines an expected number of active cores based on, for example, the execution of an algorithm that includes Eq. <NUM> described above. The processing core workload sensitivity determination logic <NUM> then determines an activity deviation between the number of active processing cores and the expected number of active processing cores, for example as described above, and provides the activity deviation as processing core activity deviation data <NUM> to the processing core frequency determination logic <NUM>.

The processing core frequency determination logic <NUM> then adjusts the frequency of the one or more processing cores based on the processing core activity deviation data <NUM>. For example, the processing core frequency determination logic <NUM> can cause the frequency of one or more of the processing cores of processing cores <NUM>, <NUM>, <NUM>, <NUM> of <FIG> to be adjusted (e.g., increased or decreased) via processing core frequency adjustment data <NUM>.

To illustrate the operation of an example algorithm as disclosed herein, <FIG> shows a plot of CPU activity for a single core in a system running a benchmark program (such as the SpecPower® benchmark provided by SPEC) in which the workload is at the <NUM>% load level. Although <FIG> shows only one processing core for clarity, the load is homogeneous and may look similar for other processing cores. Each point in <FIG> shows the percentage of active (i.e., C0) cycles in a <NUM> sample, plotted for the total period of two seconds. From <FIG>, it is clear that this workload keeps a CPU fully busy while a request is being processed, so that periods of <NUM>% C0 activity alternate with idle periods. In this case, about <NUM>% of the time is idle, reflecting the load level for this phase in the benchmark. The benchmark program (such as the SpecPower® benchmark provided by SPEC) controls the load by measuring the maximum throughput of the system, and then throttles the rate of requests between <NUM>% to <NUM>% in order to measure how power consumption scales with load.

For throughput-based loads similar to this, where processing core utilization is significantly below <NUM>%, the processing cores can be slowed down while maintaining the same throughput, thereby lowering the percentage of idle time. This is the basis for the idle-time QoS algorithm.

<FIG> provides a flowchart <NUM> of an example method for measuring QoS for executing workloads in accordance with one example set forth in the disclosure. The method illustrated in <FIG>, and each of the example methods described herein, may be carried out by the computing device <NUM>. As such, the methods may be carried out by hardware or a combination of hardware and hardware executing software. Suitable hardware may include one or more GPUs, CPUs, APUs, ASICs, state machines, FPGAs, digital signal processors (DSPs), or other suitable hardware. Although the methods are described with reference to the illustrated flowcharts (e.g., in <FIG>), it will be appreciated that many other ways of performing the acts associated with the methods may be used. For example, the order of some operations may be changed, and some of the operations described may be optional. Additionally, while the methods may be described with reference to the example computing device <NUM>, it will be appreciated that the methods may be implemented by other apparatus as well, and that the computing device <NUM> may implement other methods.

As indicated in <FIG>, at block <NUM> a basic input/output system (BIOS) setting determines whether QoS is enabled. If QoS is enabled, a workload detect loop is started at block <NUM>. The method includes obtaining start timing parameters from block <NUM>. Otherwise, the feature is disabled at block <NUM>. Once a workload is detected at block <NUM>, the method proceeds to block <NUM> to where a QoS loop is started. At block <NUM>, the QoS loop runs to measure QoS on the executing workload. The method includes obtaining, from block <NUM>, idle time target parameters. At block <NUM>, a determination is made as to whether the workload has ended. The method also includes obtaining end timing parameters from block <NUM>. If the workload has ended, the method proceeds back to block <NUM>. Otherwise, the QoS loop continues running at block <NUM>. At block <NUM>, the method also provides a power state limit signal to provide power state changes to one or more processing cores. In one example, one or more of the start timing parameters, idle time target parameters, and end timing parameters are obtained from memory, such as the memory <NUM> from <FIG>.

Claim 1:
A method for optimizing power efficiency in a computing device with a plurality of processing cores (<NUM>-<NUM>), the method comprising:
determining, by the computing device, processing core activity deviation data (<NUM>) for the plurality of processing cores based on processing core activity data (<NUM>) from the plurality of processing cores, wherein the processing core activity deviation data is based on an activity deviation between an actual number of active processing cores and an expected number of active processing cores; and
in response to the processing core activity deviation data (<NUM>) indicating a homogenous workload for the plurality of processing cores, decreasing, by the computing device, a frequency of at least one of the plurality of processing cores.