Prediction-based power management strategy for GPU compute workloads

A device for processing data includes a processing unit configured to predict an execution time of a compute kernel on a secondary processing unit and, based on the predicted execution time, make a power management decision for the secondary processing unit.

TECHNICAL FIELD

The disclosure relates to processor power management.

BACKGROUND

Parallel programming models may support one or both of task-parallelism and data-parallelism in order to solve computational problems. Task-parallelism may allow computational problems to be divided up into multiple tasks. The tasks may be executed sequentially, concurrently, and/or in parallel on one or more processor cores. Data-parallelism may allow the same set of operations to be performed in parallel on different sets of data by distributing the data to different processing elements and causing each of the processing elements to perform the same set of operations on their assigned set of data.

Multi-core processors may be used to support task-parallelism where each core is configured to execute a particular task. In some cases, one or more of the cores in a multi-core processor may be a single instruction, multiple data (SIMD) processor or a single program, multiple data (SPMD) processor that may include multiple processing elements to support data-parallelism. In such cases, tasks that support data-level parallelism may be able to be executed either sequentially or in parallel on a multi-core processor.

Several different types of processors may support task-parallelism and/or data-parallelism including a multi-core central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a Cell Broadband Engine (Cell/B.E.) processor, etc. Although GPUs were traditionally designed to support the rendering of three-dimensional (3D) graphics to a display, the programmable shader architecture included in many modern GPUs can be used to efficiently support both task-parallelism and data-parallelism found in general-purpose, non-graphics specific programs that are programmed using a parallel programming model. Using the parallel architecture of a GPU to execute non-graphics specific programs may be referred to as general-purpose computing on graphics processing units (GPGPU).

SUMMARY

This disclosure describes power management techniques for processing units processing compute workloads.

In one example, a method of processing data on a secondary processing unit includes predicting an execution time of a compute kernel on the secondary processing unit; and based on the predicted execution time, making a power management decision for the secondary processing unit.

In another example, a device for processing data includes a processing unit configured to predict an execution time of a compute kernel on a secondary processing unit and, based on the predicted execution time, make a power management decision for the secondary processing unit.

In another example, an apparatus for processing data includes means for predicting an execution time of a compute kernel on the secondary processing unit and means for making a power management decision for the secondary processing unit based on the predicted execution time.

In another example, a computer-readable medium storing instructions that when executed by one or more processors cause the one or more processors to predict an execution time of a compute kernel on the secondary processing unit and make a power management decision for the secondary processing unit based on the predicted execution time.

DETAILED DESCRIPTION

There are various techniques such as Dynamic Clock and Voltage Scaling (DCVS) and inter-domain power collapse that can be employed to reduce power (energy) consumption of a given application running on a (mobile) GPU while maintaining certain performance requirements. For graphics applications, various algorithms exist for determining when to put the GPU into a low power mode, such as a power collapse mode, based on one or more target performance metrics, such as a frame rate of the video to be generated. Inter-frame power collapse of certain GPU hardware blocks is a technique used by GPUs during graphics processing, but such a technique is not easily extendable to or compatible with compute workloads processed on a GPU. For compute workloads, i.e. non-graphics workloads, being executed on a GPU, however, easily identifiable target performance metrics such as frame rate cannot be used.

Additionally, for graphics applications, various techniques exist that can save energy by reducing voltage and frequency when such reductions may still result in a desired performance level. As with power collapse mode, voltage and/or frequency adjustment techniques are not easily extendable to or compatible with compute workloads processed on a GPU. This disclosure describes techniques for the adjusting of voltage and/or frequency of a GPU and/or for the power collapse, e.g. shutting down, of GPU blocks that are unused while a compute workloads is being executed. In this regard, the techniques of this disclosure may be unique to the running of compute workloads on a GPU (and hence present a power saving opportunity that is unique to GPGPU and which does not apply to graphics workloads).

This disclosure proposes a framework for predicting the execution time of a compute kernel. A computing system may use the predicted execution time of the kernel as an input for various power saving techniques. Thus, according to the techniques of this disclosure, a computing system may make power management decisions for compute workloads being executed by a secondary processor, such as a GPU, of the computing system.

A kernel may define a function or task that is performed by the GPU. In order to execute a kernel, the program code is divided into work items (e.g., a basic unit of work in a GPU), which are organized into one or more workgroups (e.g., a set of work items). A work item may be analogous to a thread in graphics processing and a workgroup may be analogous to a warp. Some applications may include multiple kernels for carrying out multiple functions on the same input data. Further, applications having multiple kernels may include some kernels that are dependent on other kernels. For example, an application may include two kernels, with a second kernel that is dependent on the results of the first kernel.

This disclosure introduces techniques for using performance counters to measure the processing time for a compute kernel, or portion thereof, as part of a profiling phase. Based on the measured processing times and other information available at compile time or kernel launch time, a system may predict total execution time of the kernel. An example of such other information may be the number of workgroups in a kernel or number of kernels in a virtual frame, where a virtual frame is a virtual construct for converting compute workloads that are theoretically unbounded and non-periodic, into execution units with associated (e.g. implied) deadline or performance requirements.

This disclosure describes two separate techniques for the profiling phase. The first technique includes kernel level profiling, and the second technique includes sub-kernel level. Both techniques calculate average execution clock cycles per workgroup for a specific kernel with different granularities. Total execution cycles for the kernel can subsequently be calculated by multiplying the estimated workgroup cycles by the total number of workgroups in the kernel. The prediction model can be described by the following equation:
Kernelexeccycle=WGexeccycle*Numwg+constantoverhead+ε

Kernel level profiling measures the total execution cycles of a kernel, and divides the total execution cycles by the number of workgroups in the kernel to estimate average workgroup execution cycles. The calculated average workgroup execution cycles can be used to predict execution cycles for the subsequent runs of the kernel. The calculated average workgroup execution cycles may also be updated according to the actual average workgroup execution cycles in those subsequent runs to account for system level variability effects as well as variations related to the dynamic nature of the application. Sub-kernel level profiling measures the execution cycles of the first few workgroups of a kernel and calculates the average execution cycles of a workgroup. The number of workgroups to be used for profiling is a tunable parameter, equal to or greater than the number of workgroups that can be executed in parallel, depending on the available hardware resource.

The constant_overhead term in the above equation can be tuned to account for various kernel startup times including (but not limited to) cache warmup, and GPU state setup. The variable ε accounts for estimation error and can be used for updating predicted execution cycles iteratively.

Either of the two profiling techniques can be used separately as they calculate the same parameter with different granularity. Each of the two technique may have its own advantages and disadvantages, and in some implementations, the two techniques may be combined together to potentially achieve better results.

While not limited to mobile GPUs, the techniques of this disclosure may offer particular benefits to mobile GPUs. The nature of the work performed by desktop GPUs and mobile GPUs is frequently different, with desktop GPUs typically running longer duration tasks than mobile GPUs. With longer duration tasks, the decisions with regards to entering and exiting power savings modes can be made more slowly than with short duration tasks, where such decisions need to be made relatively quickly or else the use of such modes may reduce system performance and even potentially increase power consumption rather than reduce power consumption. As mobile devices typically rely on battery power, reducing power consumption, and thus extending battery life, may significantly improve overall user experience.

FIG. 1is a block diagram illustrating an example device that may implement the techniques of this disclosure for prediction-based power management strategy for GPU compute workloads.FIG. 1illustrates device10that includes GPU12, system memory14, and processor16, which may be a central processing unit (CPU). Examples of device10include, but are not limited to, video devices such as media players, set-top boxes, wireless handsets such as mobile telephones, personal digital assistants (PDAs), desktop computers, laptop computers, gaming consoles, video conferencing units, tablet computing devices, and other such devices. Device10may include components in addition to those illustrated inFIG. 1.

System memory14may be considered as the memory for device10. System memory14may comprise one or more computer-readable storage media. Examples of system memory14include, but are not limited to, a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), flash memory, or any other medium that can be used to carry or store desired program code in the form of instructions and/or data structures and that can be accessed by a computer or a processor.

In some aspects, system memory14may include instructions that cause processor16and/or GPU12to perform the functions ascribed to processor16and GPU12in this disclosure. Accordingly, system memory14may be a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors (e.g., processor16and GPU12) to perform various functions.

System memory14may, in some examples, be considered as a non-transitory storage medium. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted to mean that system memory14is non-movable or that its contents are static. As one example, system memory14may be removed from device10, and moved to another device. As another example, memory, substantially similar to system memory14, may be inserted into device10. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM).

In some examples, such as examples where device10is a wireless handset communication device, processor16and GPU12may be formed in an integrated circuit (IC). For example, the IC may be considered as a processing chip within a chip package or may be considered to be a system on a chip or portion thereof. In some examples, processor16and GPU12may be housed in different integrated circuits (i.e., different chip packages) such as examples where device10is a desktop or laptop computer. However, it may be possible that processor16and GPU12are housed in different integrated circuits in examples where device10is a wireless handset communication device.

Examples of processor16and GPU12include, but are not limited to, a digital signal processor (DSP), general purpose microprocessor, application specific integrated circuit (ASIC), field programmable logic array (FPGA), or other equivalent integrated or discrete logic circuitry. In some examples, GPU12may be specialized hardware that includes integrated and/or discrete logic circuitry that provides GPU12with massive parallel processing capabilities suitable for graphics processing. In some instances, GPU12may also include general purpose processing capabilities, and may be referred to as a general purpose GPU (GPGPU) when implementing general purpose processing tasks (i.e., non-graphics related tasks).

Processor16may execute various types of applications. Examples of the applications include web browsers, e-mail applications, spreadsheets, video games, or other applications that generate viewable objects for display. Instructions for execution of the one or more applications may be stored within system memory14. Processor16may transmit graphics data of the viewable objects to GPU12for further processing.

For instance, processor16may offload processing tasks to GPU12, such as tasks that require massive parallel operations. As one example, graphics processing requires massive parallel operations, and processor16may offload such graphics processing tasks to GPU12. Processor16may communicate with GPU12in accordance with a particular application processing interface (API). Examples of such APIs include the DirectX® API by Microsoft®, the OpenGL® API by the Khronos group, and the OpenCL™ API; however, aspects of this disclosure are not limited to the DirectX, the OpenGL, or the OpenCL APIs, and may be extended to other types of APIs. Moreover, the techniques described in this disclosure are not required to function in accordance with an API, and processor16and GPU12may utilize any technique for communication.

To perform graphics operations, GPU12may implement a graphics processing pipeline. The graphics processing pipeline includes performing functions as defined by software or firmware executing on GPU12and performing functions by fixed-function units that are hardwired to perform very specific functions. The software or firmware executing on the GPU12may be referred to as shader programs (or simply shaders), and the shader programs may execute on one or more shader cores of GPU12. Shader programs provide users with functional flexibility because a user can design the shader program to perform desired tasks in any conceivable manner. The fixed-function units, however, are hardwired for the manner in which the fixed-function units perform tasks. Accordingly, the fixed-function units may not provide much functional flexibility.

For example, processor16may execute an application, such as a video game, and processor16may generate graphics data as part of the execution. Processor16may output the graphics data for processing by GPU12. GPU12may then process the graphics data in the graphics pipeline. In some examples, to process the graphic data, GPU12may need to execute one or more shader programs. For example, the application executing on processor16may cause processor16to instruct GPU12to retrieve a shader program from system memory14and instruct GPU12to execute the shader program.

GPU12may also be configured to execute commands that are issued to GPU12by processor16. The commands executed by GPU12may include general-purpose computing commands, task execution commands (e.g., kernel execution commands), memory transfer commands, etc. GPU12may be configured to perform general-purpose computing for applications executing on processor16. For example, when a host program, which is executing on processor16, decides to off-load a computational task to GPU12, processor16may provide general-purpose computing data to GPU12, and issue one or more general-purpose computing commands to GPU12. The general-purpose computing commands may include, e.g., kernel execution commands, memory transfer commands, etc. In some examples, processor16may provide the commands and general-purpose computing data to GPU12by writing the commands and data to system memory14, which may be accessed by GPU12.

GPU12may also be configured to operate in one or more low power modes or implement one or more power management mechanisms in order to reduce power consumption. An example of one such power management mechanism is DCVS, where device10may conserve power by reducing the clock rate and operating voltage of GPU12. An example of another such low power mode is a power collapse mode, where GPU12may power down (e.g., cut power to) certain blocks when not being used. GPU12may, for example, power down hardware blocks that are not required for a particular compute operation (e.g., hardware blocks dedicated to performing graphics operations).

Device10may also optionally include display18, user interface20, and transceiver module22. Device10may include additional modules or units not shown inFIG. 1for purposes of clarity. For example, device10may include a speaker and a microphone, neither of which are shown inFIG. 1, to effectuate telephonic communications in examples where device10is a mobile wireless telephone. Furthermore, the various modules and units shown in device10may not be necessary in every example of device10. For example, user interface20and display18may be external to device10in examples where device10is a desktop computer. As another example, user interface20may be part of display18in examples where display18is a touch-sensitive or presence-sensitive display of a mobile device.

Examples of user interface20include, but are not limited to, a trackball, a mouse, a keyboard, and other types of input devices. User interface20may also be a touch screen and may be incorporated as a part of display18. Transceiver module22may include circuitry to allow wireless or wired communication between device10and another device or a network. Transceiver module22may include modulators, demodulators, amplifiers and other such circuitry for wired or wireless communication. Display18may comprise a liquid crystal display (LCD), a cathode ray tube (CRT) display, a plasma display, a touch-sensitive display, a presence-sensitive display, or another type of display device.

As will be explained in more detail below, in accordance with the techniques of the disclosure, GPU12may be configured to predict an execution time of a compute kernel and based on the predicted execution time, make a power management decision, such as determining whether to change a DCVS operating performance point (OPP) or to put GPU12into an inter-domain power collapse mode. GPU12may predict the execution time of the compute kernel by estimating an average execution clock cycles per workgroup for the compute kernel and by estimating a total number of execution cycles for the compute kernel based on the average execution clock cycles per workgroup for the compute kernel and a total number of workgroups in the kernel. In other implementations, a kernel driver running on processor16may be configured to predict the execution time of the compute kernel and based on the predicted execution time, make the power management decision for GPU12.

To estimate the average execution clock cycles per workgroup for the compute kernel, GPU12may estimate the average execution clock cycles per workgroup for the compute kernel at a kernel level, a sub-kernel level, or utilizing a combination of both. Kernel-level profiling may provide for a relatively simple implementation in the kernel driver, without the need for special hardware support, produce higher accuracy in common cases, and not require any need for access to kernel source. Kernel-level profiling, however, may also produce lower accuracy for some specific use-cases and may only work after a first run of the same kernel. Additionally, kernel-level profiling may require a more complicated implementation when integrated with DCVS.

Sub-kernel level-profiling may avoid the need for an additional book keeping mechanism for each kernel and may work even on the first run of the kernel. Sub-kernel level profiling may also allow for a more seamless integration with DCVS. Sub-kernel level profiling, however, may require special power management hardware, produce lower accuracy than kernel level profiling for common use-cases, and require more initial implementation effort (but potentially less tuning and overall effort when integrated with DCVS).

FIG. 2Ais a block diagram illustrating components of the device illustrated inFIG. 1in greater detail. As illustrated inFIG. 2A, GPU12includes controller30, oscillator34, shader core36, and fixed-function pipeline38. Shader core36and fixed-function pipeline38may together form an execution pipeline used to perform graphics or non-graphics related functions. Although only one shader core36is illustrated, in some examples, GPU12may include one or more shader cores similar to shader core36.

The commands that GPU12is to execute are executed by shader core36and fixed-function pipeline38, as determined by controller30of GPU12. Controller30may be implemented as hardware on GPU12or software or firmware executing on hardware of GPU12. Controller30may receive commands that are to be executed from command buffer40of system memory14or directly from processor16(e.g., receive the submitted commands that processor16determined should now be executed by GPU12). Controller30may also retrieve the operand data for the commands from data buffer42of system memory14or directly from processor16. Controller30may determine which commands are to be executed by shader core36(e.g., software instructions are executed on shader core36) and which commands are to be executed by fixed-function pipeline38(e.g., commands for units of fixed-function pipeline38).

In some examples, commands and/or data from one or both of command buffer40and data buffer42may be part of a local memory of GPU12. For instance, GPU12may include an instruction cache and a data cache that stores commands from command buffer40and data from data buffer42, respectively. In these examples, controller30may retrieve the commands and/or data from the local cache.

Shader core36and fixed-function pipeline38may transmit and receive data from one another. For instance, some of the commands that shader core36executes may produce intermediate data that are operands for the commands that units of fixed-function pipeline38are to execute. Similarly, some of the commands that units of fixed-function pipeline38execute may produce intermediate data that are operands for the commands that shader core36is to execute. In this way, the received data is progressively processed through units of fixed-function pipeline38and shader core36in a pipelined fashion. Hence, shader core36and fixed-function pipeline38may be referred to as implementing an execution pipeline.

In general, shader core36allows for various types of commands to be executed, meaning that shader core36is programmable and provides users with functional flexibility because a user can program shader core36to perform desired tasks in most conceivable manners. Shader core36may be used for performing graphics operations or compute operations depending on how shader core36is configured or depending on what code shader core36is executing. The fixed-function units of fixed-function pipeline38, however, are hardwired for the manner in which the fixed-function units perform tasks. Accordingly, the fixed-function units may not provide much functional flexibility.

As also illustrated inFIG. 2A, GPU12includes oscillator34. Oscillator34outputs a clock signal that sets the time instances when shader core36and/or units of fixed-function pipeline38execute commands. Although oscillator34is illustrated as being internal to GPU12, in some examples, oscillator34may be external to GPU12. Also, oscillator34need not necessarily just provide the clock signal for GPU12, and may provide the clock signal for other components as well.

Oscillator34may generate a square wave, a sine wave, a triangular wave, or other types of periodic waves. Oscillator34may include an amplifier to amplify the voltage of the generated wave, and output the resulting wave as the clock signal for GPU12.

In some examples, on a rising edge or falling edge of the clock signal outputted by oscillator34, shader core36and each unit of fixed-function pipeline38may execute one command. In some cases, a command may be divided into sub-commands, and shader core36and each unit of fixed-function pipeline38may execute a sub-command in response to a rising or falling edge of the clock signal. For instance, the command of A+B includes the sub-commands to retrieve the value of A and the value of B, and shader core36or fixed-function pipeline38may execute each of these sub-commands at a rising edge or falling edge of the clock signal.

The rate at which shader core36and units of fixed-function pipeline38execute commands may affect the power consumption of GPU12. For example, if the frequency of the clock signal outputted by oscillator34is relatively high, shader core36and the units of fixed-function pipeline38may execute more commands within a time period as compared the number of commands shader core36and the units of fixed-function pipeline38would execute for a relatively low frequency of the clock signal. However, the power consumption of GPU12may be greater in instances where shader core36and the units of fixed-function pipeline38are executing more commands in the period of time (due to the higher frequency of the clock signal from oscillator34) than compared to instances where shader core36and the units of fixed-function pipeline38are executing fewer commands in the period of time (due to the lower frequency of the clock signal from oscillator34). Moreover, higher frequencies typically require higher supply voltages, which further increases the power consumption of GPU12.

As described above, processor16may offload tasks to GPU12due to the massive parallel processing capabilities of GPU12. For instance, GPU12may be designed with a single instruction, multiple data (SIMD) structure. In the SIMD structure, shader core36includes a plurality of SIMD processing elements, where each SIMD processing element executes same commands, but on different data.

A particular command executing on a particular SIMD processing element is referred to as a thread (or a work item for compute workloads). Each SIMD processing element may be considered as executing a different thread/work item because the data for a given thread may be different; however, the thread/work item executing on a processing element is the same command as the command executing on the other processing elements. In this way, the SIMD structure allows GPU12to perform many tasks in parallel (e.g., at the same time). For such SIMD structured GPU12, each SIMD processing element may execute one work item on a rising edge or falling edge of the clock signal.

To avoid confusion, this disclosure uses the term “command” to generically refer to a process that is executed by shader core36or units of fixed-function pipeline38. For instance, a command includes an actual command, constituent sub-commands (e.g., memory call commands), a thread, or other ways in which GPU12performs a particular function. Because GPU12includes shader core36and fixed-function pipeline38, GPU12may be considered as executing the commands.

Also, in the above examples, shader core36or units of fixed-function pipeline38execute a command in response to a rising or falling edge of the clock signal outputted by oscillator34. However, in some examples, shader core36or units of fixed-function pipeline38may execute one command on a rising edge and another, subsequent command on a falling edge of the clock signal. There may be other ways in which to “clock” the commands, and the techniques described in this disclosure are not limited to the above examples.

Because GPU12executes commands every rising edge, falling edge, or both, the frequency of clock signal (also referred to as clock rate) outputted by oscillator34sets the amount of commands GPU12can execute within a certain time. For instance, if GPU12executes one command per rising edge of the clock signal, and the frequency of the clock signal is 1 MHz, then GPU12can execute one million commands in one second.

As described above, in some cases, GPU12may execute submitted commands (e.g., the set of commands that GPU12retrieves from command buffer40) within a set time period. However, the amount of commands in the submitted commands may be variable (i.e., the number of commands that GPU12needs to execute within the time period need not be constant for every set of submitted commands).

For instance, as illustrated inFIG. 2A, processor16executes application26, as illustrated by the dashed boxes. During execution, application26generates commands that are to be executed GPU12, including commands that instruct GPU12to retrieve and execute shader programs (e.g., vertex shaders, fragment shaders, compute shaders for non-graphics applications, and the like). In addition, application26generates the data on which the commands operate (i.e., the operands for the commands). Processor16stores the generated commands in command buffer40, and stores the operand data in data buffer42.

After processor16stores the generated commands in command buffer40, processor16makes available the commands for execution by GPU12. For instance, processor16communicates to GPU12the memory addresses of a set of the stored commands and their operand data, where GPU12is to execute the set of commands, and information indicating when GPU12is to execute the set of commands. In this way, processor16submits commands to GPU12for execution.

As illustrated inFIG. 2A, processor16may also execute graphics driver28. In some examples, graphics driver28may be software or firmware executing on hardware or hardware units of processor16. Graphics driver28may be configured to allow processor16and GPU12to communicate with one another. For instance, when processor16offloads graphics or non-graphics processing tasks to GPU12, processor16offloads such processing tasks to GPU12via graphics driver28. For example, when processor16outputs information indicating the amount of commands GPU12is to execute, graphics driver28may be the unit of processor16that outputs the information to GPU12. Graphics driver28may be divided into two components, a kernel driver and a user-space driver. The techniques of this disclosure, which relate to compute workloads, may be implemented in some examples by the kernel driver of the graphics driver.

As additional examples, application26produces graphics data and graphics commands, and processor16may offload the processing of this graphics data to GPU12. In this example, processor16may store the graphics data in data buffer42and the graphics commands in command buffer40, and graphics driver28may instruct GPU12when to retrieve the graphics data and graphics commands from data buffer42and command buffer40, respectively, from where to retrieve the graphics data and graphics commands from data buffer42and command buffer40, respectively, and when to process the graphics data by executing one or more commands of the set of commands.

Also, application26may require GPU12to execute one or more shader programs. For instance, application26may require shader core36to execute a vertex shader and a fragment shader to generate images that are to be displayed (e.g., on display18ofFIG. 1). Graphics driver28may instruct GPU12when to execute the shader programs and instruct GPU12with where to retrieve the graphics data from data buffer42and where to retrieve the commands from command buffer40or from other locations in system memory14. In this way, graphics driver28may form a link between processor16and GPU12.

Graphics driver28may be configured in accordance to an application processing interface (API); although graphics driver28does not need to be limited to being configured in accordance with a particular API. In an example where device10is a mobile device, graphics driver28may be configured in accordance with the OpenGL ES API. The OpenGL ES API is specifically designed for mobile devices. In an example where device10is a non-mobile device, graphics driver28may be configured in accordance with the OpenGL API.

In the example techniques described in this disclosure, there may be various ways in which processor16may determine the amount of commands that GPU12is to execute within the time period. For example, the amount of commands in the submitted commands may be based on the commands needed to render one frame of the user-interface or gaming application. For the user-interface example, GPU12may need to execute the commands needed to render one frame of the user-interface within the vsync window (e.g., 16 ms) to provide a jank-free user experience. If there is a relatively large amount of content that needs to be displayed, then the amount of commands may be greater than if there is a relatively small amount of content that needs to be displayed.

To ensure that GPU12is able to execute the submitted commands within the set time period, controller30may adjust the frequency (i.e., clock rate) and/or voltage of the clock signal that oscillator34outputs. However, to adjust the clock rate of the clock signal such that the clock rate is high enough to allow GPU12to execute the submitted commands within the set time period, controller30may receive information indicating the performance level needed to execute the set of commands (e.g., submitted commands) within the time period, and determine the clock rate based on the received information.

For instance, if controller30receives information indicating what the performance level is needed based on how many commands GPU12is to execute within the time period, either as an absolute value or a relative value, prior to GPU12executing the commands, controller30may determine whether to increase or decrease the frequency of the clock signal. Controller30may increase the clock rate if controller30determines that a higher performance level (e.g., higher clock rate) is needed to execute the commands within the time period than the current clock rate for timely execution. To reduce power consumption, controller30may decrease the clock rate if controller30determines that as high as a performance level is not needed to execute the commands within the time period (i.e., a lower clock rate is sufficient to timely execute the commands).

In the techniques described in this disclosure, controller30may determine the performance level based on information received from processor16that indicates the performance level. The performance level is based on an amount of commands GPU12is to execute within a time period. Controller30may then increase or decrease the frequency of the clock signal outputted by oscillator34based on the determination of the performance level. In this manner, the frequency of the clock signal may increase before GPU12is to execute the commands that were used to determine the performance level of GPU12.

Because application26generates the commands that GPU12is to execute, application26may determine the amount of commands GPU12is to execute within a set time period, and may, therefore, be able to determine the performance level of GPU12. Processor16may then output information indicating the performance level of GPU12, where the performance level is based on an amount of commands GPU12is to execute within the set time period.

However, in some cases, while application26may generate the commands and the operand data, application26may not have been designed to determine the performance level of GPU12. In some examples, if application26does not determine the performance level of GPU12or does not cause processor16to output information indicating the performance level, GPU12may still be able to receive information indicating the performance level needed to timely execute the commands that were used to determine the performance level. As illustrated inFIG. 2A, processor16also executes operating system24. Operating system24is configured to manage resources of processor16, such as allocate memory resources and handle the transfer of commands and data to and from memory, such as the memory that includes command buffer40or data buffer42(system memory14in the example illustrated inFIG. 2A).

Therefore, operating system24may be configured to determine the amount of commands GPU12is to execute because operating system24determines the memory resources needed to store the commands and operand data in command buffer40and data buffer42. Moreover, when processor16submits the commands, it is operating system24that determines the memory addresses for where GPU12is to retrieve commands from command buffer40and operands data from data buffer42. Accordingly, operating system24may determine the amount of commands GPU12is to execute within a time period. Operating system24may determine the performance level of GPU12based on the determined amount of commands GPU12is to execute within a time period, and cause processor16to output the information indicating the performance level.

There may be other ways in which processor16may utilize application26and/or operating system24to proactively determine the performance level of GPU12before GPU12executes commands that were used to determine the performance level of GPU12. The following provides two additional examples for how processor16may utilize information from application26and/or operating system24to determine the performance level of GPU12.

As a first example for how processor16proactively determines the expected performance level of GPU12, assume that application26is a video game such as, but not required to be, one that produces high definition graphics, and device10is a mobile device. In this example, if a video player is displaying the output of application26when device10is in the portrait orientation, application26may need to submit commands to GPU12to render only a small area of display18. However, when a user rotates device10to the landscape orientation for a transitory period of a few frame, GPU12, at the current clock rate, may not be able to produce a frame within 16 ms.

In this example, operating system24may notify application26about a change in the orientation, and processor16, in turn, may determine the performance level of GPU12. For instance, as illustrated inFIG. 2A, processor16may execute power management module32; however, power management module32may be hardware of processor16or a combination of hardware and software or firmware. In one example of frequency management performed by power management module32, power management module32may determine that application26will be increasing the amounts of commands GPU12is to execute, and in turn may determine a performance level needed by GPU12before GPU12executes the commands, and may be even before GPU12receives the commands.

Power management module32may then indicate to graphics driver28that a performance level of GPU12is to increase. Graphics driver28may then output information indicating the expected performance level of GPU12, and controller30of GPU12may in turn increase the frequency of oscillator34.

In some examples, the techniques implemented by power management module32may be dynamic voltage and clock scaling (DVCS) or dynamic clock and voltage scaling (DCVS) control that provide control based on running average, variant, and/or trend. One example way in which power management module32may implement frequency management is described in U.S. Pat. No. 8,650,423.

In general, power management module32may maintain running statistics of the workload of GPU12. For instance, power management module32may maintain one or more of a running average busy and/or idle duration, an idle/busy ratio, a variance of the running average, and a trend of the running average of the workload. Based on the statistics of the workload of GPU12, power management module32may continuously determine the frequency of the clock signal outputted by oscillator34.

However, there may be some latency in power management module32determining the frequency of the clock signal. For instance, power management module32utilizes the recently executed commands to determine what the clock rate should have been for the recently executed commands. But, by the time power management module32determines the clock rate, GPU12has already started to execute the next commands. Also, for low end examples of GPUs, the latency may be relatively high. Accordingly, the clock rate determination from power management module32may be slightly delayed from what the clock rate should actually be.

In some examples, processor16may utilize the outputs from power management module32to adjust the frequency of the clock signal outputted by oscillator34. For instance, based on information received from power management module32of the performance level of GPU12, controller30may determine and set the clock rate of oscillator34to the determined clock rate.

Power management module32may additionally be configured to power collapse some hardware blocks of GPU12that contribute to the power consumption of GPU12. For example, for graphics data at a low frame rate (e.g. 15 frames per second), power management module32may collapse certain hardware blocks of GPU12because GPU12does not need to utilize those blocks for data processing in order to meet a desired performance level. For graphics data at a higher frame rate (e.g. 30 frames per second), however, power management module32may need all available power blocks active in order for GPU12to meet a desire performance level.

The above description of power management module32has thus far focused on how power management module32may implement aspects of DCVS and power collapse for graphics workloads. According to the techniques of this disclosure, power management module32may also implement DCVS and power collapse for non-graphics workloads, e.g. compute workloads. As will be described in more detail below, power management module32may predict an execution time of a compute kernel that is to be executed by GPU12, and based on the predicted execution time, make a power management decision for GPU12. The power management decision may, for example, include adjusting a frequency and/or voltage of oscillator34in the manner described above. The power management decision may, for example, also include adjusting operating parameters of resources, such as a memory, that are shared by GPU12but separate from GPU12.

Power management module32may also power collapse some, but not all, hardware blocks of GPU12based on the predicted execution time. Compute workloads executed on GPU12may not use some hardware blocks with considerable contribution to the power consumption of GPU12, and therefore, GPU12may be able to reduce power consumption by power collapsing those blocks. In this regard, power collapsing some, but not all, hardware blocks of GPU12may be considered to be a different mode than modes in which all of GPU12is powered down. According to the techniques of this disclosure, power management module32may determine if blocks of GPU12should be power collapsed based on a predicted execution time of a compute kernel. For small execution times, the power collapsed blocks of GPU12may need to be turned on again quickly, and changing rapidly from a power collapsed state to an active state may decrease performance, and in some cases even increase power consumption.

As introduced above, GPU12may predict a kernel execution time by estimating an average workgroup execution time, and based on the estimated average workgroup execution time, predict a kernel execution time. The predicted kernel execution time may, for example, correspond to the average workgroup execution time multiplied by the number of workgroups in the kernel. GPU12may, for example, estimate the average workgroup execution time by implementing either kernel-level profiling or sub-kernel (e.g. workgroup) level profiling.

FIG. 2Bshows an alternative implementation of device10. In the implementation of device10, power management module32is located in GPU12instead of processor16. Power management module32may, for example, be implemented as hardware on GPU12or software or firmware executing on hardware of GPU12. In some implementations, power management module32may be implemented as firmware executed by a dedicated processor inside GPU12. Device10as shown inFIG. 2Bis functionally equivalent to device10shown inFIG. 2Aother than for the location of power management module32.

FIG. 3is a graphical representation of kernel50. Kernel50includes a plurality of workgroups, which are shown inFIG. 3as three-dimensional rectangular boxes. Workgroup52represents one of the plurality of workgroups of kernel50, and is shown inFIG. 3. Each workgroup in kernel50includes a plurality work items, which are shown inFIG. 3as smaller cubes. Work items54A and54B represent two of the work items of workgroup52. Kernel50inFIG. 3is shown as a three-dimensional structure, similar to the three-dimensional structures used in OpenCL and other software frameworks.

FIG. 4is a graphical representation of sub-kernel level profiling for determining an average workgroup execution time. In the example ofFIG. 4, kernel60includes a plurality of workgroups, which are shown inFIG. 4as rectangles. Each of the workgroups includes a plurality of work items, which are shown as squiggly arrows within each rectangle. The number of workgroups in kernel60may be expressed as X*Y*Z, with Z being assumed to be equal to 1 for the example ofFIG. 4, but in other examples Z may be an integer greater than 1. To perform sub-kernel level profiling, processor16or GPU12may be configured to determine a workgroup execution time for a subset of the workgroups in kernel60. In the example ofFIG. 4, processor16or GPU12may be configured to determine a workgroup execution time by profiling workgroups62A-62D (workgroups62). Processor16or GPU12may, for example, determine an average workgroup execution time for workgroups62. Processor16or GPU12can determine an estimated kernel execution time (Exec_kernel inFIG. 4) based on the average workgroup execution time (Exec_WG inFIG. 4) and the number of workgroups in the kernel (X*Y inFIG. 4).

In the example ofFIG. 4, processor16or GPU12predicts the execution of time for kernel60based on profiling workgroups of kernel60. Based on the predicted execution time for kernel60, which is determined based on the profiling of workgroups62, processor16or GPU12makes a power management decision for how GPU12will process the remaining workgroups of kernel60, i.e. the workgroups processed after workgroups62.

FIG. 5is a graphical representation of kernel level profiling for determining an average workgroup execution time. In the example ofFIG. 5, kernel70includes a plurality of workgroups, which are shown inFIG. 5as rectangles within kernel70. Each of the workgroups includes a plurality of work items, which are shown as squiggly arrows within each rectangle. The number of workgroups in kernel70may be expressed as X0*Y0*Z0, with Z0being assumed to be equal to 1 for the example ofFIG. 5, but in other examples Z0may be an integer greater than 1. To perform kernel level profiling, processor16or GPU12may be configured to determine a workgroup execution time for all workgroups in kernel70. Processor16or GPU12may, for example, determine an average workgroup execution time (Exec_WG) for the workgroups of kernel70.

For a second kernel76, processor16or GPU12can determine an estimated kernel execution time (Exec_kernel inFIG. 5) based on the average workgroup execution time (Exec_WG inFIG. 4) determined for kernel70and the number of workgroups in second kernel76(X1*Y1inFIG. 5). In the example ofFIG. 5, second kernel76represents a second execution of the same kernel used for kernel70. In some examples, characteristics of second kernel76, such as input data size or number of workgroups in second kernel76, may be different than kernel70, but the kernel code for kernel70and second kernel76may be the same. In other examples, the profile determined for kernel70may only be used for second kernel76if second kernel76has the same work group configuration as kernel70. In such an example, separate instances of the same kernel that have different workgroup configurations may be associated with different profiles.

In the example ofFIG. 5, processor16or GPU12predicts the execution of time for second kernel76based on profiling workgroups of kernel70. Based on the predicted execution time for second kernel76, processor16or GPU12makes a power management decision for how GPU12will process the workgroups of second kernel76.

FIGS. 6 and 7are flow diagrams illustrating techniques of this disclosure for prediction-based power management of compute workloads. The techniques ofFIGS. 6 and 7will be described with reference to a system that includes a primary processing unit and a secondary processing unit. In this context, a primary processing unit generally refers to a processing unit that controls, either wholly or partially, the execution of workloads on a secondary processing unit. For example, the primary processing unit may issue commands to the secondary processing unit, through a driver of the secondary processing unit, which cause the secondary processing unit to process a workload. In many examples, the primary processing unit may be more of a general purpose processor while the secondary processor may be more of a special purpose processor.

The primary processing unit may, for example, correspond to processor16ofFIGS. 1 and 2, while the secondary processing unit may correspond to GPU12ofFIGS. 1 and 2. While the techniques of this disclosure have generally been described with respect to CPUs and GPUs, it should be understood that the techniques of this disclosure may be implemented with other types of secondary processing units and may be of particular benefit for secondary processing units that support data parallelism and for which there is going to be a high correlation or similarity between the execution time of the instructions on different, similarly sized blocks of data (in this case workgroups).

In the example ofFIG. 6, the system predicts an execution time of a compute kernel on the secondary processing unit (600). To predict the execution time of the compute kernel, the system may estimate an average execution clock cycles per workgroup for the compute kernel. To predict the execution time of the compute kernel, the system may additionally estimate a total number of execution cycles for the compute kernel based on the average execution clock cycles per workgroup for the compute kernel and a total number of workgroups in the kernel. To estimate the average execution clock cycles per workgroup for the compute kernel, the system may estimate the average execution clock cycles per workgroup for the compute kernel at a kernel level or at a sub-kernel level, in the manners described above.

Based on the predicted execution time, the system makes a power management decision for the secondary processing unit (602). The power management decision for the secondary processing unit may, for example, include putting the secondary processing unit into a low power mode, such as an inter-domain power collapse mode or other such mode. The power management decision may alternatively or additionally include dynamically scaling one or both of a clock frequency or a voltage for the secondary processing unit.

In the example ofFIG. 7, the system predicts an execution time of a compute kernel on the secondary processing unit (700). Based on the predicted execution time, the system makes a power management decision for the secondary processing unit. For example, in response to the predicted execution time being greater than a threshold value (702, YES), the system may put the secondary processing unit into a low power mode such as an inter-domain power collapse mode, or other such low power mode (704). In response to the predicted execution time being less than a threshold value (702, NO), the system may leave the secondary processing unit in a full power mode (706). To predict the execution time of the compute kernel, the system may estimate an average execution clock cycles per workgroup for the compute kernel.

To predict the execution time of the compute kernel, the system may estimate a total number of execution cycles for the compute kernel based on the average execution clock cycles per workgroup for the compute kernel and a total number of workgroups in the kernel. To estimate the average execution clock cycles per workgroup for the compute kernel, the system may estimate the average execution clock cycles per workgroup for the compute kernel at a kernel level or a sub-kernel level as described above.

The techniques described in this disclosure may also be stored, embodied or encoded in a computer-readable medium, such as a computer-readable storage medium that stores instructions. Instructions embedded or encoded in a computer-readable medium may cause one or more processors to perform the techniques described herein, e.g., when the instructions are executed by the one or more processors. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer readable storage media that is tangible.

Computer-readable media may include computer-readable storage media, which corresponds to a tangible storage medium, such as those listed above. Computer-readable media may also comprise communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, the phrase “computer-readable media” generally may correspond to (1) tangible computer-readable storage media which is non-transitory, and (2) a non-tangible computer-readable communication medium such as a transitory signal or carrier wave.

Various aspects and examples have been described. However, modifications can be made to the structure or techniques of this disclosure without departing from the scope of the following claims.