Methods and apparatus to manage power of deep learning accelerator systems

Apparatus, systems, methods, and articles of manufacture to manage power of deep learning accelerator systems are disclosed. An example apparatus includes a power manager and a power controller. The power manager is to generate a power table to allocate power frequencies between an accelerator and memory based on a ratio of compute tasks and bandwidth tasks in a first workload; update the power table based on a request to at least one of add a second workload or remove the first workload; and determine an index into the power table. The power controller is to determine a power consumption based on the power table; determine whether to update the index based on a power budget and the power consumption; and allocate power to the accelerator and the memory according to the power table.

FIELD OF THE DISCLOSURE

This disclosure relates generally to power management and, more particularly, for managing power of deep learning accelerator systems.

BACKGROUND

Computer hardware manufacturers develop hardware components for use in various components of a computer platform. For example, computer hardware manufacturers develop motherboards, chipsets for motherboards, central processing units (CPUs), hard disk drives (HDDs), solid state drives (SSDs), and other computer components. Additionally, computer hardware manufacturers develop processing circuitry, known as accelerators, to accelerate the processing of a workload. For example, an accelerator can be implemented by dedicated circuitry, an integrated circuit, a CPU, a graphics processing unit (GPU), a vision processing unit (VPU), an application specific integrated circuit (ASIC), and/or a field programmable gate array (FPGA).

The figures are not to scale. In general, the same reference numbers will be used throughout the drawing(s) and accompanying written description to refer to the same or like parts. Connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and/or physical or logical couplings between the various elements.

DETAILED DESCRIPTION

Disclosed herein are systems, apparatus, methods, and articles of manufacture to dynamically manage power allocation to components in a computing device, such as an accelerator, etc.

Many computer hardware manufacturers develop processing circuitry, known as accelerators, to accelerate the processing of a workload. For example, an accelerator can be implemented by dedicated circuity, an integrated circuit, a central processing unit (CPU), a graphics processing unit (GPU), a vision processing unit (VPU), an application specific integrated circuit (ASIC), and/or a field programmable gate array (FPGA). Moreover, while accelerators are capable of processing any type of workload, accelerators are designed to improve (e.g., optimize, etc.) execution of one or more particular types of workloads. For example, while CPUs and FPGAs are designed to handle more general processing, GPUs are designed to improve the processing of video, games, and/or other physics and mathematically based calculations, and VPUs are designed to improve the processing of machine vision tasks.

Additionally, some accelerators are designed specifically to improve the processing of artificial intelligence (AI) applications. While a VPU is a specific type of AI accelerator, many different AI accelerators are available. For example, many AI accelerators can be implemented by application specific integrated circuits (ASICs). Such ASIC-based AI accelerators can be designed to improve the processing of tasks related to a particular type of AI, such as machine learning (ML), deep learning (DL), and/or other artificial machine-driven logic including support vector machines (SVMs), neural networks (NNs), recurrent neural networks (RNNs), convolutional neural networks (CNNs), long short term memory (LSTM), gate recurrent units (GRUs), etc.

Computer hardware manufactures also develop heterogeneous systems that include more than one type of processing circuit. For example, computer hardware manufactures may combine both general purpose processing circuit, such as CPUs, with either general purpose accelerators, such as FPGAs, and/or more tailored accelerators, such as GPUs, VPUs, ASICs, dedicated circuitry, and/or other AI accelerators. Such heterogeneous systems can be implemented as systems on a chip (SoCs).

As used herein, a workload is defined to be a capacity or function that is assigned or allocated to an accelerator and/or other processing circuitry for execution. For example, implementing a neural network, provisioning a virtual machine, establishing a database, configuring an application, etc., are workloads for execution by an accelerator and/or other processing circuitry. When a developer desires to run a function, algorithm, program, application, and/or other executable instruction on a heterogeneous system, the developer and/or software generates a schedule (e.g., a graph) for the function, algorithm, program, application, and/or other executable instruction at compile time. Once a schedule is generated, the schedule is combined with aspects of the function, algorithm, program, application, and/or other executable instruction to generate an executable file (either for Ahead of Time or Just in Time paradigms). Moreover, the schedule combined with specification for the function, algorithm, program, application, and/or other executable instruction may be represented as a graph including nodes, where the graph represents a workload and each node (e.g., a workload node) represents a particular task of that workload. Furthermore, the connections between the different nodes in the graph represent the data inputs and/or outputs needed in order for a particular workload node to be executed and the vertices of the graph represent data dependencies between workload nodes of the graph.

In certain examples, hardware accelerators are implemented as specialized hardware circuitry that may be used for deep learning and/or artificial intelligence applications. Accelerators may be utilized to accelerate tasks of deep learning and/or artificial intelligence workloads. For example, the accelerators may increase the speed of artificial neural network training for the artificial neural network. When at least one or more workloads are executed by at least one or more accelerators, power to the accelerators is managed to improve performance. Additionally, power allocated to additional processing circuitry and/or memory within the system (referred to herein as an accelerator system or a deep learning accelerator system) is to be managed to execute workloads.

AI applications and associated topologies involve a variety of numerical precision representations, different types and sizes of data, and advanced latency and power requirements. The workloads, topologies, and algorithms related to AI technologies frequently change. Additionally, some AI workload characteristics include inner layer-to-layer variation with respect to compute and/or bandwidth usage of computing devices.

Disclosed herein are advancements to fine tune different power requirements of components of a computing device (e.g., an accelerator) during the execution of a workload such as an AI workload, etc. Traditional power improvement (e.g., optimization) methods do not account for differences in AI workload behavior. Additionally, in some examples, AI workloads feature domain specific improvement (e.g., optimization) opportunities because such AI workloads may have more regular (e.g., predefined, expected, etc.) behavior than usual client applications. In some examples disclosed herein, prior or advanced knowledge can provide power improvement (e.g., optimization) hints and/or directions that can be better utilized in hardware assisted power management circuits.

Examples disclosed herein include a hardware/software co-design that leverages compiler knowledge of an AI workload to drive power management of a power control circuit (also referred to herein as a power control unit or punit) driving configuration and execution of an accelerator and/or other computing device. Components of the computing device including, for example, an AI accelerator (or portion thereof) and/or other power consumption circuits, negotiate with the power control circuit for power and performance improvement (e.g., optimization). The knowledge of workload performance that is analyzed, processed, and shared by the compiler includes, in some examples, data related to the operation of the power consumption units in the form of a configurable table or graph of power levels tuned according to different workload scenarios of the power consumption units. A power manager consolidates requests for power from power consumption circuits and determines a table of frequency values (e.g., clock frequencies, etc.) to drive the power control circuit to deliver power to the power consumption circuits using the knowledge shared by the compiler. An index or level in the power table of frequency values indicates the level of power to be provided to the power consumption circuits for an amount of time, for example.

Examples disclosed herein provide a unique and flexible method of power and performance improvement (e.g., optimization) for AI training and inference workloads that enables fast, efficient improvement (e.g., optimization) of power allocation on-the-fly and/or otherwise dynamically. Utilizing the advanced knowledge analyzed and developed by the compiler is unique for AI workloads and graphs. Furthermore, the compiler activity can occur offline rather than during a runtime or execution phase of the computing device. In addition, power aware compilation or compiler guided power allocation utilizes the advanced knowledge or data in the compilation process and enables the power management hardware such as the power manager, the power control circuit, etc., to make accurate power determinations, to dynamically fine grain control the power policy, to reduce the need for power guard bands, and to provide better quality of services to the different circuitry running in the computing device, each of which has its own power usage characteristics.

Certain examples provide a dynamic power controller or manager to determine frequency changes based on combined requirements of workloads that are running on an accelerator and to regulate power budget based on the determined frequency changes. The combined requirements are known in advance.

Prior power improvement (e.g., optimization) and management solutions relied only on a history of executed workloads. Based on that history of heuristics and actual performance, they tried to improve power decisions over time. However, such prior approaches did not have prior knowledge (e.g., meta-data determined from analysis of instructions, etc.) of the nature of the workloads that are executed. Thus, such prior approaches are not able to detect execution patterns. As such, static heuristics that exist in the power controller may be beneficial in some cases, but, in other cases, may cause performance degradation. With such prior approaches, there is no ability to modify behavior in a dynamic manner.

In contrast, certain examples disclosed herein address these deficiencies by providing a device and associated method for a software/hardware interface that determines the power management policy to reduce or increase power budget. The interface exposes a power table that can be updated through software power management that factors in both precompiled requirements from different workloads that are running on the computing device and management interface requests that can come during execution.

Certain examples provide dynamic power improvement (e.g., optimization) or management heuristics including power heuristics that can be dynamically modified by updating a power table, executing performance experiments, and fine tuning of power levels or indices and level/index selection within the power table. Further, during compilation of a workload, the compiler determines a desired frequency ratio between compute and bandwidth resources for different portions or sections of the workload execution. The compiler can aggregate frequency ratio requests for different sections of workload execution to form levels or indices of frequency distribution in the power table that may change over time. Runtime software on the computing device can manage executed workloads and provide heuristics to serve running workloads, for example.

With deep learning accelerators, for example, a roughly finite set of operations are used to build workloads. These workloads are compiled using a dedicated compiler and executed on the accelerator. These workload operations have compute and bandwidth characteristics that can change significantly based on, for example, which layer of the network model is being implemented by the accelerator (e.g., is it a compute-intensive layer involving many computing operations, a memory access-intensive layer involving many data transfer/loading/storing operations, etc.), and, thus, have different frequency ratio requirements.

To improve performance, a power controller on the computing device is to have a holistic view of frequency requirements of overall executed workloads on the accelerator. The power controller factors in external power requirements (e.g., thermal constraints, etc.) and workload requirements to determine frequencies for power allocation to system components (e.g., memory resources, computing resources, etc.).

Certain examples provide a power manager or power controller to generate, update, and/or otherwise manage a power table that stores power levels and frequency ratios between computing structures (e.g., accelerators, etc.) and memory structures of the computing system (e.g., the deep learning accelerator system, etc.). Frequency requirements (e.g., a frequency ratio, also referred to as a work ratio, etc.) for components of the system form part of the compiling of deep learning applications for execution on the system, for example.

AI accelerators have a finite set of workloads running on the accelerator at any given time. The compiler can improve (e.g., optimize) workload execution and provide different frequency ratios (e.g., cache versus computing circuit, memory versus accelerator, etc.) to different parts of the graphs representing the compiled workload(s). A runtime stack can take these hints and build a dynamic power table and perform power heuristics that are based on compiler power hints along with runtime considerations of multiple workloads that are running on the computing device.

The compiler can generate a set of dynamic voltage and frequency scaling (DVFS) points that can be resolved at a whole-graph level and/or inside the graph, for example. At the graph level, a DVFS ratio for the whole graph (e.g., of memory to cache to accelerator, etc.) can be generated, for example. In some examples, compute circuit frequency can be changed within a graph. Thus, the compiler works with a power manager to generate a plurality of power solutions in conjunction with compilation of workload functions, and the power manager of the computing device (e.g., of the accelerator system, etc.) can select from among the power solutions for operation.

The power manager includes and/or is formed from runtime software to build a dynamic power table used by the power controller when a power change is to be made. As such, rather than only heuristics, a decision on power change is additionally or alternatively based on real workload information, for example. The power table can be updated based on requests to add and/or remove workload(s) from execution by the computing device, for example. The power manager can dynamically determine and update frequency ratios for each workflow and associated relative frequency change of components based on adding and/or removing workloads from execution by the accelerator system and/or other computing device, for example. The power manager is to determine an index or level corresponding to each frequency/work ratio in the power table, for example. The power controller can determine a power consumption of the system based on the index/level in the power table and can determine whether to update the index/level based on a power budget and the power consumption, for example.

FIG. 1is a block diagram of an example apparatus100for configurable workload execution. The example apparatus100includes a processor system110, memory120, and a compiler130. The example apparatus100can be an AI accelerator to execute a plurality of AI workloads, etc. The example apparatus100can have a plurality of implementations. For example, the apparatus100can be implemented as a system-on-a-chip (SoC). The apparatus100can be implemented as a plurality of separate circuits, for example. In some examples, the processor system110and memory120are implemented as a first device with the compiler130located on a second device in communication with the first device. In certain examples, the processor system110and/or the memory120are in communication with the compiler130through a compilation blob and/or other wireless communication without a direct hardware link. In other examples, the processor system110, memory120, and compiler have a hardwired communication link.

Operation of the apparatus100are described in connection with certain examples further below. In brief, the example processor system110determines a configuration of its constituent processor circuitry and/or memory120to execute one or more workloads based on data obtained from the example compiler130and/or executes deep learning and/or other AI workloads using the determined configuration. In certain examples, the processor system110includes one or more accelerators. Other compute circuitry such as processors, other circuitry, etc., can also be included in the processor system110. The processor system110also includes power management, such as a power manager or power driver, a power controller, etc. In some example, the processor system110can also include on-board memory separate from the memory120to cache instructions, a power table, other configuration information, etc.

The example memory120stores data such as executable application information, configuration information, power table information, etc. The example memory120can be implemented as a random access memory (RAM) such as a double data rate (DDR) synchronously dynamic RAM (SDRAM), last level cache (LLC), etc.

The example compiler130compiles at least one workload of at least one deep learning and/or other AI application. In response to the compilation of at least one workload, the example compiler system140determines data utilized by system components, such as work ratios for each workload and/or the relative frequency change of at least one accelerator, etc. The example compiler system140provides at least one deep learning application access to the example processor system110by compiling the application for execution using the processor system110and by providing the processor system110with characteristics and/or other execution information to allow the processor system110to determine frequency ratios for power allocation to circuitry of the processor system110, for example.

As such, the example apparatus100includes one or more AI accelerators to execute AI workloads. For example, the apparatus100can be configured to execute deep learning inference workloads. The compiler130receives a neural network representation, for example, an outputs information that can be executed with specific inferences, optimized or otherwise tailored to a deep learning workload, for example. An example workload includes compute tasks and memory bandwidth tasks. Each workload includes a different ratio of compute tasks to memory tasks. The compiler130generates instructions for execution by one or more accelerators of the processor system110and can also generate information for the compiler130and/or the processor system110to determine a ratio of frequencies (also referred to herein as power frequencies such as clock frequencies, etc.) between compute resources and memory resources in the processor system110involved in executing the workload. Information from the compiler130can be used to generate tailored, dynamic, more accurate determinations of power allocation and generation for workload execution by the processor system110.

In certain examples, one or more accelerators can be used to execute one or more workloads. A plurality of accelerators can be integrated, for example. Additionally or alternatively, a plurality of accelerators can maintain coherency through shared memory120, a converged coherence fabric (CCF), etc. A specific ratio of frequencies can be generated to allocate power among computing and memory resources to accommodate the one or more workloads to be executed using the processor system110and memory120. Thus, rather than relying on pre-set heuristics to allocate power, characteristics of currently executing workloads can be leveraged to generate and update power allocation on the fly. As workload(s) change, the processor system110and memory120can adapt. Information regarding a ratio of compute tasks to memory bandwidth tasks can be determined by the compiler130in a compilation phase and can be utilized by the processor system110(e.g., by a power manager, also referred to as a power driver, in the processor system110) to dynamically determine, at workload runtime, frequency allocation to provide power to circuitry of the processor system110and memory120. Characteristics, such as compute instructions (e.g., arithmetic instructions, logic instructions, etc.), memory instructions (e.g., load instruction, store instructions, move instructions, other data transfer instructions, etc.) of each executing workload are considered in the allocation, and the ratio (e.g., a ratio of compute instructions to memory instructions (e.g., 3:2, 2:3, 1:5, 5:1, etc.) for a workload or portion thereof, etc.) remains until a workload is added or removed from execution, for example.

FIG. 2Ais a block diagram of an example implementation of the example apparatus100ofFIG. 1. In the implementation ofFIG. 2A, the example apparatus100includes the example processor system110and memory120implemented as a system on chip200in communication with the example compiler130. As shown in the example ofFIG. 2A, the compiler130provides workload information210including one or more deep learning workload executables and associated characteristics (e.g., compute and memory requirements/ratio, etc.) to execute the workload. For example, the compiler130can provide workload information210such as deep learning inference workload information including a trained neural network model configuration to infer or predict testing samples and associated characteristics regarding compute and memory bandwidth usage for phases of the inference. For example, the compiler130can analyze the source code representing executable instructions for the workload to extract instruction type, instruction function, etc., to classify the instruction as a compute-related instruction (e.g., an arithmetic action, a logic action, etc.) or a memory access-related instruction (e.g., a memory or other data transfer action, etc.).

The example processor system110shown inFIG. 2Aincludes the processor system110, the memory120, a power manager220, a power table memory240, and a power controller260. The example power manager220includes a power management processor222, a table generator224, and an index determiner226. The example processor system110ofFIG. 2Aincludes a plurality of accelerators232-236as well as a workload management processor238. While the example ofFIG. 2Ashows three accelerators232-236and one workload management processor238, any number of accelerator(s)232-236and workload management processor(s)238may be implemented in the processor system110. The example power table memory240, which can be separate from and/or included with memory120, the power controller260, etc., stores one or more indices241allocating a frequency range242-245to each of a plurality of components. The example memory120ofFIG. 2Aincludes a shared cache252(e.g., an LLC, etc.) and a memory storage254, such as DDR DRAM, DDR SDRAM, etc.

The power management processor222of the power manager220processes the workload information210from the compiler130to determine a ratio of compute resources to memory bandwidth resources to execute the workload. For example, as shown inFIG. 2A, the compiler130includes a receiver211to receive program source code for compilation, an analyzer213to analyze the code, a code generator215to compile or generate an executable from the received source code, and an output217to provide the executable for execution by the SoC200. The code can include instructions for configuration and execution of a workload, such as deep learning neural network inferencing, etc. The code generator215forms the executable from the source code, and the analyzer213processes the code to determine a comparison or ratio between computing tasks and memory access tasks in the executable. For example, the analyzer213can determine dynamic voltage and frequency scaling (DVFS) transition/optimization points and associated frequency or work ratios from the code analysis. For example, by analyzing machine instructions, the analyzer213can determine which operations involve computation and which operations involve memory access or transfer (e.g., by determining which instructions are data transfer instructions (e.g., move, load, input, output) and which instructions are arithmetic instructions (e.g., add, subtract, increment, decrement, convert, compare, etc.) or logic instructions (e.g., AND, OR, exclusive OR, shift, rotate, test, etc.)). The analyzer213can then determine at which point(s) the execution of the workload is to be compute intensive, memory bandwidth intensive, etc.

In certain examples, the analyzer213performs two passes of code and power options to generate one or more DVFS transition (e.g., optimization) points. For example, the analyzer213performs an initial pass to determine a DVFS solution as an approximated inference per second with no power changes. Based on power budget information, a second pass can utilize any power budget remaining after the first pass to add a subset of frequencies to run the workload. A power configuration can be determined that provides a lowest execution time using a valid power budget, for example.

The output217generates workload information210including the executable and associated ratio, DVFS point, other code information, etc., from the analyzer213and the code generator215, for example. In some examples, a plurality of workload transition points (e.g., a top three, five, etc.) can be output for use by the power manager220.

The power management processor222receives a new workload scheduled for execution on the set of accelerators232-236. The workload information210from the compiler130can include (e.g., represented as meta-data) workload power characteristics such as a list of frequencies and/or a set of one or more optimization points at which the workload should run. For example, an initial portion of the workload may be bandwidth intensive and involve much memory access, while a later portion of the workload may be compute intensive and involve one or more accelerators232-236and/or other processor238in the processor system110. The power management processor222consolidates power information for one or more running workloads based on the workload information210and triggers the power table generator224to generate and/or update the power table240when workload(s) executing on the SoC200change, for example. The power management processor222reads a current power level from the power controller260and determines new power levels by consolidating workload requests and associated power characteristics.

The example power table generator224generates and/or updates the power table in the power table memory240based on the compute versus memory power and frequency ratio(s) (also referred to as work ratios) determined by the power management processor222. The example table generator224stores each integrated work ratio to a row or entry of the power table within the example power table memory240. Each row of the power table corresponds to one work ratio option (e.g., operating frequencies for the processor system110versus operating frequencies for memory120, etc.) for the example processor system110and memory120to execute at least one workload. As shown in the example ofFIG. 2A, the power table memory240stores an index241which sets a power frequency allocation (and, in some cases, acceptable variation) for each of the available components, such as the workload management processor allocation242, cache allocation243, accelerator allocation244, memory allocation245, etc. The table generator224triggers the index determiner226to update an index into the power table when the power table has been generated and/or updated in power table memory240.

For example, Table 1 below illustrates an example power table stored in the power table memory240defining levels or indices241of frequency information for the power controller260. For each level/index241, the processor system110and memory120have an associated frequency and/or power level to be allocated/assigned by the power controller260(also called the power control unit or “punit”). Each index241represents a step or increment of change for the associated circuit (e.g., 25 MHz, 100 MHz, etc.). For example, high indicates high memory accesses, medium indicates some memory access requests, and low indicates no use of the memory during inference. In some examples, low, medium, and high are determined at time of manufacture of the circuit and correspond to a particular setting or range (e.g., low=700 MHz, medium=1700 MHz, high=3000 MHz, etc.).

In certain examples, a ratio can be expressed as a transition or improvement (e.g., optimization) point at which a balance between compute resources and memory resources changes in accordance with workload execution. As shown in the example of Table 1, a DVFS transition/improvement point varies according to the index or level in the power table. In certain examples, an acceptable variation or amount of change can be specified to aid the power controller260in adjusting and allocating power frequencies to the SoC200. For example, at index 2 in the example of Power Table 1, the first accelerator ACC1 can be allocated at 600 MHz. However, the power controller260can adjust this value up to 630 MHz before necessitating a different index in the table. At index N, accelerator 1 can still be allocated at 600 MHz, but a downward adjustment to 580 MHz is allowed at index N before triggering selection of a different index.

The example index determiner226determines an index corresponding to one row or level of the example power table (e.g., Table 1, etc.) in memory240. In certain examples, a default index, such as index 0, etc., can be specified for automatic selection by the power controller260to configure power allocation/usage of the SoC200. However, the index determiner226can analyze the power table in memory240and SoC200resource information to determine and recommend another index (e.g., index 2, index N, etc.) rather than a default of index 0 to the power controller260, for example. For example, based on a current index to the power table, the index determiner226can compute an updated index to the power table in power table memory240and can transmit the updated index to the power controller260.

The power controller260can operate according to a power configuration specified at a selected index in the power table stored in memory240. In certain examples, the power controller260calculates power consumption for the SoC200to determine whether the systems of the SoC200are within an allocated power budget. The power controller260can adjust power consumption by increasing or reducing allocation accordingly by moving up or down among indices in the power table. Thus, when more power is available for allocation, an index in the power table can allocate more power, and, when less power is available for allocation, another index in the power table can allocate less power. Further, when the executing workload(s) call for greater computing resources, an index in the power table can allocate more power to the accelerator(s)232-236, workload management processor238, etc., rather than memory120. However, when the executing workload(s) involve greater need for memory bandwidth, another index in the power table can allocate more power to memory120, rather than the processor system110, for example.

As shown in the example ofFIG. 2A, the example power controller260includes a consumption calculator262, a comparator264, and a selector266. As discussed above, the power controller260is notified by the power manager220regarding an update to the power table memory240. The consumption calculator262calculates power consumption from executing workload(s) on the SoC200and information from the power manager220. As such, the consumption calculator262can determine a current power level for the SoC200. The comparator264can compare the current power level to a threshold (e.g., maximum power level, limit, etc.). When the current power level is less than the threshold, the comparator264can trigger a reselection of an index or level in the power table memory240. The selector266selects an index or level in the power table memory240for frequency configuration and power allocation in the SoC200.

In the example ofFIG. 2A, the power manager220and its power management processor222and power table generator224implement means for generating a power table and means for updating the power table. For example, the power management processor222processes workload meta-data to determine frequency allocation, and the power table generator224forms and updates the power table using the frequency allocation. The example index determiner226of the power manager220implements means for determining an index into the power table. The example power controller260implements means for allocating power among accelerator(s)232-236and memory120using the power table. For example, the consumption calculator262of the power controller260determines a power consumption. The example comparator264determines whether to update the index based on a power budget and the power consumption. The example selector266allocates power to the accelerator232-236and the memory120according to the power frequencies at the index of the power table.

FIG. 2Billustrates an alternative arrangement of the example apparatus100. In the example ofFIG. 2B, the power manager220and memory storage254are located external to the SoC200, along with the compiler130. Additionally, the power controller260generates power and frequency controls271-279associated with the accelerators232-236, shared compute resource (e.g., the workload management processor)238, shared cache252, etc. Memory254storing program code, executables, etc., is located external to the SoC200and separate from, or in conjunction with, the compiler130. A shared bus280connects the components of the SoC200to execute workload tasks such as neural network inferencing, convolutional layers, etc. As described above with respect toFIG. 2A, the power manager220generates and/or updates a power table of settings for accelerators232-236, memory252,254, etc.

In the example ofFIG. 2B, the power manager220transmits the power table to the power controller260, which generates power and frequency controls271-279for the SoC200based on a selected index of the power table (e.g., a default index, an index specified by the power manager220, an index determined by the power controller260, etc.). As such, a frequency is assigned to respective components of the SoC200to execute/implement workload(s). For example, during workload execution, an accelerator232-236frequency can be adjusted according to operations that are executed. Optimization points for adjustment to accelerator frequency are determined by the compiler130, and such frequencies are assigned by the power manager220. In certain examples, a relative frequency change can be added or subtracted within a power level of the power table for an accelerator232-236. The relative frequency change is local to the particular accelerator232-236and does not affect memory252,254(e.g., LLC, DDR, etc.) frequency.

In the example ofFIG. 2B, the power manager220implements means for generating a power table, means for updating the power table, and means for determining an index into the power table. The example power controller260implements means for allocating power among accelerator(s)232-236and memory120using the power table by determining a power consumption, determining whether to update the index based on a power budget and the power consumption, and allocating power to the accelerator232-236and the memory120according to the power frequencies at the index of the power table.

FIG. 3depicts an example workload300to implement convolution filters for a deep learning neural network. During the workload, behavior changes between compute-intensive execution to bandwidth-intensive execution. For example, in the workload300, a first half310is compute-limited, and a second half330is bandwidth-limited, with an optimization point320for DVFS frequency change in between the phases or stages310,330of the workload300. In certain examples, the power manager220computes a weighted sum of frequency requests for currently executing workloads and sends an aggregated request to the power controller260. The power manager220can provide options for flexibility of the power controller260via a plurality of levels and associated indices for power frequency allocation in the SoC200, for example.

FIG. 4is an example data flow diagram400illustrating a flow of instructions among the power manager220, power controller260, and an accelerator232-236to execute two deep learning inference workloads on the example apparatus100ofFIG. 2A. At405, the example power manager220sends a request for DVFS frequency (e.g., specifying a ratio of accelerator to shared memory to other memory to shared compute processor, etc.), also referred to as the work ratio, to the example power controller260, which allocates frequencies to provide power to circuitry of the SoC200. At410, the example power manager220sends an instruction to execute the inference request to at least one of the example accelerators232-236, which will execute the inference request workload. At415, during execution of the inference request, at least one of the example accelerators232-236sends a request for a relative frequency change to the example power controller260, and the power controller260determines whether to grant the relative frequency change to the accelerator(s)232-236based on power consumption and availability according to the power table memory240. At420, a result or status update is returned by the accelerator232-236to the power manager220.

At425, the example power manager220sends a DVFS frequency request for a second inference to the example power controller260. At430, the example power manager220sends an instruction to execute the second inference request to at least one of the example accelerators232-236. At435, the example power manager220sends an instruction to execute the first inference request to at least one of the example accelerators232-236. At440, in response to the first and second inference requests, at least one of the example accelerators272-276sends a relative frequency change request to the example power controller260, which determine whether to grant or deny the relative frequency change to the accelerator(s)232-236based on power consumption and availability according to the power table memory240. At445, the example accelerator(s)232-236provide a status update on execution of the first inference request to the example power manager220. At450, the example accelerator(s)232-236provide a status update on execution of the second inference request to the example power manager220.

As such, the power manager220can coordinate execution of inference requests by the accelerator(s)232-236, and the power controller260can allocate power frequency accordingly. The power manager220receives a DVFS request generated by the compiler130that includes a relative ratio of work frequencies between components of the SoC200, and an instruction by the power manager220to load the inference to the accelerator232-236triggers a request for DVFS allocation by the power controller260. The compiler130can determine DVFS ratios based on empirical measured data, calculated according to a model or formula of workload layer behavior, overall system workload requirements, etc. During an inference request, an accelerator232-236can send a request to the power controller260to increase or decrease its allocated frequency, and the power controller260can adjust according to the power table, for example.

While example implementations of the example apparatus100are illustrated inFIGS. 1-2B, one or more of the elements, processes, and/or devices illustrated inFIGS. 1-2Bmay be combined, divided, re-arranged, omitted, eliminated, and/or implemented in any other way. Further, the example processor system110, example memory120, the example compiler130, the example SoC200, the example receiver211, the example analyzer213, the example code generator215, the example output217, the example power manager220, the example power management processor222, the example power table generator224, the example index determiner226, the example accelerator232-236, the example workload management processor238, the example power table memory240, the example shared cache252, the example memory storage254, the example power controller260, the example consumption calculator262, the example comparator264, the example selector266, the example power and frequency control271-279, the example bus280, and/or, more generally, the example apparatus100can be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPU(s)), digital signal processor(s) (DSP(s)), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)), and/or field programmable logic device(s) (FPLD(s)). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example processor system110, example memory120, the example compiler130, the example SoC200, the example receiver211, the example analyzer213, the example code generator215, the example output217, the example power manager220, the example power management processor222, the example power table generator224, the example index determiner226, the example accelerator232-236, the example workload management processor238, the example power table memory240, the example shared cache252, the example memory storage254, the example power controller260, the example consumption calculator262, the example comparator264, the example selector266, the example power and frequency control271-279, the example bus280, and/or, more generally, the example apparatus100are hereby expressly defined to include a non-transitory computer readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc. including the software and/or firmware. Further still, the example processor system110, example memory120, the example compiler130, the example SoC200, the example receiver211, the example analyzer213, the example code generator215, the example output217, the example power manager220, the example power management processor222, the example power table generator224, the example index determiner226, the example accelerator232-236, the example workload management processor238, the example power table memory240, the example shared cache252, the example memory storage254, the example power controller260, the example consumption calculator262, the example comparator264, the example selector266, the example power and frequency control271-279, the example bus280, and/or, more generally, the example apparatus100may include one or more elements, processes, and/or devices in addition to, or instead of, those illustrated inFIGS. 1-2B, and/or may include more than one of any or all of the illustrated elements, processes, and devices. As used herein, the phrase “in communication,” including variations thereof, encompasses direct communication and/or indirect communication through one or more intermediary components, and does not require direct physical (e.g., wired) communication and/or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and/or one-time events.

A flowchart representative of example hardware logic, machine readable instructions, hardware implemented state machines, and/or any combination thereof for implementing the example apparatus100ofFIG. 1is shown inFIG. 5. The machine readable instructions may be one or more executable programs or portion(s) of an executable program for execution by a computer processor such as the processor912shown in the example processor platform900discussed below in connection withFIG. 9. The program may be embodied in software stored on a non-transitory computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a DVD, a Blu-ray disk, or a memory associated with the processor912, but the entire program and/or parts thereof could alternatively be executed by a device other than the processor912and/or embodied in firmware or dedicated hardware.

Further, although the example program is described with reference to the flowcharts illustrated inFIGS. 5-8, many other methods of implementing the example apparatus100may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware.

FIG. 5illustrates a process or method500implemented by executing program instructions to drive the example apparatus100to dynamically configure power allocation for workloads executing on the accelerator apparatus100. At block510, the example program500includes preparing, by the compiler130, a workload for execution by the apparatus100(e.g., by the processor system110and memory120of the example apparatus100). For example, the code generator215compiles one or more workloads for execution by the processor system110and memory120of the apparatus100. The analyzer213analyzes the code for the workload(s) to identify memory accesses, compute instruction executions, etc., to determine a DVFS ratio of compute to memory loads for each workload.

For example, the analyzer213analyzes the composition of a workload to determine a transition or improvement (e.g., optimization) point. For each mapping of a workload to execution instructions for the apparatus100, the analyzer213can analyze each layer (e.g., each layer of a deep learning neural network to be implemented by the workload, etc.). For each layer, the analyzer213can calculate an execution time per layer. Additionally, the analyzer213can determine a desirable (e.g., optimal) accelerator232-236frequency for a given memory120frequency to be involved in the workload. Execution time can then be updated. The example analyzer213can build a DVFS interval to determine a sequence of layers. When a prior execution time minus a new execution time is greater than or equal to a threshold (e.g., old_exec_time-new_exec_time>=threshold), then a DVFS change/transition point has been identified inside the workload. This change point can be noted as an improvement (e.g., optimization) point. Alternatively or in addition, a compute to bandwidth ratio can be computed for each layer, and an absolute ratio can be determined by weighted sum, in which the weights are an approximated time of execution per layer. A DVFS request can be formulated for the power manager220based on the transition point. For example, the DFVS request can be formulated as:

The output217generates workload information210including the executable workload and associated meta-data (e.g., ratio, transition/improvement point, other code information, etc.) from the analyzer213and the code generator215, for example. In some examples, a plurality of workload transition points (e.g., a top three, five, etc.) can be output for use by the power manager220.

At block520, the example program500includes allocating power in the apparatus100to execute the workload. For example, the power manager220receives the workload executable and associated meta-data from the compiler130and determines the power table to store in power table memory240using the meta-data associated with the workload for execution.

For example, the power management processor222receives a new workload scheduled for execution on the set of accelerators232-236as well as associated meta-data such as a list of frequencies and/or a set of one or more optimization points at which the workload should run. For example, an initial portion of the workload may be bandwidth intensive and involve much memory120access, while a later portion of the workload, occurring after a DVFS optimization point, may be compute intensive and involve one or more accelerators232-236and/or other processor238in the processor system110. The power management processor222consolidates power information for one or more running workloads based on the workload information210and triggers the power table generator224to generate and/or update the power table240when workload(s) executing on the SoC200change, for example. The power management processor222reads a current power level from the power controller260and determines new power levels by consolidating workload requests and associated power characteristics.

The example power table generator224generates and/or updates the power table in the power table memory240based on the compute versus memory power and frequency ratio(s) (also referred to as work ratios) determined by the power management processor222. The example table generator224stores each integrated work ratio to a row or entry of the power table within the example power table memory240. Each row of the power table corresponds to one work ratio option (e.g., operating frequencies for the processor system110versus operating frequencies for memory120, etc.) for the circuitry of the example processor system110and memory120to execute at least one workload. The power table memory240stores an index241which sets a power frequency allocation (and, in some cases, acceptable variation) for each available components, such as the workload management processor allocation242, cache allocation243, accelerator allocation244, memory allocation245, etc. The table generator224triggers the index determiner226to update an index into the power table when the power table has been generated and/or updated in power table memory240.

The example index determiner226determines an index corresponding to one row or level of the example power table (e.g., Table 1, etc.) in memory240. In certain examples, a default index, such as index 0, etc., can be specified for automatic selection by the power controller260to configure power allocation/usage of the SoC200. However, the index determiner226can analyze the power table in memory240and SoC200resource information to determine and recommend another index (e.g., index 2, index N, etc.) rather than a default of index 0 to the power controller260, for example. For example, based on a current index to the power table, the index determiner226can compute an updated index to the power table in power table memory240and can transmit the updated index to the power controller260.

The power controller260can operate according to a power configuration specified at a selected index in the power table stored in memory240. As such, the power controller260can configure the SoC200with power allocation based on allotted frequencies for the selected index.

At block530, the example program500includes monitoring and adjusting power allocation during execution of the workload by the apparatus100. For example, resources of the processor system110and memory120execute the workload according to power frequencies allocated by the selector266of the power controller260according to the selected index of the power table. In certain examples, the power controller260calculates power consumption for the SoC200to determine whether the systems of the SoC200are within an allocated power budget. The power controller260can adjust power consumption by increasing or reducing allocation accordingly by moving up or down among indices in the power table. Thus, when more power is available for allocation, an index in the power table can allocate more power, and, when less power is available for allocation, another index in the power table can allocate less power. Further, when the executing workload(s) call for greater computing resources, an index in the power table can allocate more power to the accelerator(s)232-236, workload management processor238, etc., rather than memory120. However, when the executing workload(s) involve greater need for memory bandwidth, another index in the power table can allocate more power to memory120, rather than the processor system110, for example.

For example, the consumption calculator262of the power controller260determines a current power level for the SoC200. The comparator264compares the current power level to a maximum power level, threshold, limit, etc. When the current power level is less than or greater than the maximum power level, the comparator264can trigger a reselection of an index or level in the power table memory240. The selector266selects an index or level in the power table memory240for frequency configuration and power allocation in the SoC200.

FIG. 6illustrates an example program600to prepare a workload (e.g., an AI workload, etc.) for execution by the apparatus100(e.g., an example implementation of block510of the example ofFIG. 5). At block610, the workload is input to be compiled by the compiler130for execution by the processor system100and memory120. For example, the code generator215compiles one or more workloads for execution by the processor system110and memory120of the apparatus100.

At block620, power is evaluated for each layer of the workload. At block630, a bandwidth/compute ratio is estimated for a given layer of the workload. For example, source code representing instructions involved in the workload to implement a layer of an AI network model (e.g., an inferencing layer of a convolutional neural network, etc.) is analyzed to determine a ratio between compute instructions and memory access instructions in the workload. A transition or improvement (e.g., optimization) point at which execution tasks switch from memory-intensive to compute-intensive can be determined for the layer. For example, the analyzer213analyzes the composition of the workload layer to determine a transition point. The analyzer213can calculate an execution time for the layer. When a prior execution time minus a new execution time is greater than or equal to a threshold (e.g., old_exec_time-new_exec_time>=threshold), then a DVFS transition/change point has been identified inside the workload layer. This change point can be noted as an optimization point. Additionally, the analyzer213can determine an improved (e.g., optimal or other desirable) accelerator232-236frequency for a given memory120frequency to be involved in the workload layer.

This process repeats at block620for each layer in the workload. At block640, when all layers have been evaluated, a DVFS request is determined for the workload. For example, execution times, transition points, and other ratio information can be evaluated by the analyzer213across the layers of the workload. The example analyzer213can build a DVFS interval using individual execution times and associated ratios to determine a sequence of layers, for example. Alternatively or in addition, a compute to bandwidth ratio can be computed for each layer, and an absolute ratio can be determined by weighted sum, in which the weights are an approximated time of execution per layer. Thus, a DVFS request ratio for the workload can be computed and saved as meta-data associated with the executable workload for relay to the power manager220, for example.

FIG. 7illustrates an example program700to allocate power in the apparatus100to execute the workload(s) (e.g., an example implementation of block520of the example ofFIG. 5). At block710, a request to add or remove a workload is received. For example, the compiler130sends the power manager220meta-data regarding the addition or removal of a workload. The power management processor222triggers a processing of the meta-data when it is received.

At block720, a power table is created and/or updated in the power table memory240. For example, the power management processor222receives a new workload scheduled for execution on the set of accelerators232-236as well as associated meta-data such as a list of frequencies and/or a set of one or more DVFS transition points at which the workload should run. For example, an initial portion of the workload may be bandwidth intensive and involve much memory120access, while a later portion of the workload, occurring after a DVFS transition point, may be compute intensive and involve one or more accelerators232-236and/or other processor238in the processor system110. The power management processor222consolidates power information for one or more running workloads based on the workload information210and triggers the power table generator224to generate and/or update the power table240when workload(s) executing on the SoC200change (e.g., are added, removed, etc.), for example.

The example power table generator224generates and/or updates the power table in the power table memory240based on the compute versus memory power and frequency ratio(s) (also referred to as work ratios) determined by the power management processor222. The example table generator224stores each integrated work ratio to a row or entry of the power table within the example power table memory240. Each row of the power table corresponds to one work ratio option (e.g., operating frequencies for the processor system110versus operating frequencies for memory120, etc.) for the circuitry of the example processor system110and memory120to execute at least one workload. The power table memory240stores an index241which sets a power frequency allocation (and, in some cases, acceptable variation) for each available components, such as the workload management processor allocation242, cache allocation243, accelerator allocation244, memory allocation245, etc. The table generator224triggers the index determiner226to update an index into the power table when the power table has been generated and/or updated in power table memory240.

At block730, an index into the power table is determined. For example, the example index determiner226determines an index corresponding to one row or level of the example power table (e.g., Table 1, etc.) in memory240. In certain examples, a default index, such as index 0, etc., can be specified for automatic selection by the power controller260to configure power allocation/usage of the SoC200. However, the index determiner226can analyze the power table in memory240and SoC200resource information to determine and recommend another index (e.g., index 2, index N, etc.) rather than a default of index 0 to the power controller260, for example. For example, based on a current index to the power table, the index determiner226can compute an updated index to the power table in power table memory240and can transmit the updated index to the power controller260.

At block740, power allocation is triggered. For example, the index determiner226communicates an index into the power table to the power controller260. The power controller260can then allocate power to components of the processor system110and memory120according to a power configuration specified at a selected index in the power table stored in memory240. As such, the power controller260can configure components of the SoC200with power allocation based on allotted frequencies for the selected index.

FIG. 8illustrates an example program800to monitor and adjust power allocation in the apparatus100during workload execution (e.g., an example implementation of block530of the example ofFIG. 5). At block810, the power controller260receives a request for power consumption. For example, the power manager220and/or the compiler130triggers workload execution, which involves power consumption by the apparatus100. At block815, the power table is read. For example, the power controller260accesses the power table memory240to determine power allocation frequencies for the SoC200and/or other circuitry of the apparatus100. The power controller260can read and/or otherwise access the power table according to an index provided by the power manager220, a default index (e.g., 0, 1, etc.), etc. The power controller260can then allocate power to components based on configuration information at the index in the power table (e.g., frequency allocation information and/or other limit defined at the index241in the power table memory240, etc.).

At block820, power consumption is calculated based on work ratio and/or other frequency information in the power table. For example, the consumption calculator262of the power controller260calculates power consumption for the SoC200using actual consumption and/or information at the index241to the power table240to determine a power level the SoC200. The power level can be a current power level and/or a projected power level accommodating the new workload, for example. The comparator264can compare the power level to a maximum power level, threshold, limit, etc. The power limit/threshold may be set based on physical characteristics of the SoC200, such as heat tolerance, performance degradation, other material constraint, etc., to avoid damage to the SoC200and/or other system impact, for example. The power limit/threshold can include a margin of error or other range in value to provide a buffer or opportunity for error or variance with respect to a true, damaging power limit, for example.

At block825, based on the comparison, the power controller260determines whether the SoC200satisfies, meets, or uses its power budget. For example, the allocated power level may be below a power budget limit or threshold; the allocated power level may be at or exceeding the power budget limit or threshold; the allocated power level may be satisfying its power budget at or within a specified range or tolerance below the power budget; etc. For example, addition or subtraction of a workload may bring the power level above or below the power limit/threshold. If the SoC200satisfies its power budget, then, at block835, power is allocated to the SoC200and/or other part of the apparatus100.

However, when the SoC200does not satisfy its power budget, the power level of the SoC200is further analyzed. For example, at block840, the comparator264of the power controller260determines whether the power level of the SoC200is above or below its power budget limit/threshold. If the SoC200power level is not above its threshold or limit (e.g., is not greater than a specified maximum power level and/or associated range, etc.), then, at block845, the selector266of the power controller260can increase to a higher index241of the power table. For example, a higher index can correspond to a greater overall power allocation and/or a different distribution of power within the SoC200, and, when additional power is available to be allocated, the power controller260can select a higher index for power allocation from the power table. However, if the SoC200exceeds its maximum power allocation, then, at block850, a lower index241is selected from the power table. For example, a lower index241corresponding to a lesser and/or different distribution of power can be selected by the selector266in the power table memory240. At block855, the index241is updated to reflect a new power allocation from the power table, and control reverts to block830to determine an updated compliance of power allocation with power budget for the SoC200. When the power budget is satisfied, then, at block835, power is allocated. As such, the power controller260can react to changes in workload to adjust power allocation according to the power table, which can be updated by the power manager220as workload(s) are added and/or removed for the SoC200.

FIG. 9is a block diagram of an example processor platform900structured to execute the instructions ofFIG. 5to implement the example apparatus100ofFIG. 1. The processor platform900can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a headset or other wearable device, or any other type of computing device.

The processor platform900of the illustrated example includes a processor912. The processor912of the illustrated example is hardware. For example, the processor912can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs (including GPU hardware), DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor912implements the example SoC200including the example processor system110, the example memory120, the example power manager220, the example power table memory240, and/or the example power controller260.

The processor912of the illustrated example includes a local memory913(e.g., memory120, etc.). The processor912of the illustrated example is in communication with a main memory including a volatile memory914and a non-volatile memory916via a bus918. The volatile memory914may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®), and/or any other type of random access memory device. The non-volatile memory916may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory914,916, which can also be used to implement memory120, is controlled by a memory controller.

The processor platform900of the illustrated example also includes an interface circuit920. The interface circuit920may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), a Bluetooth® interface, a near field communication (NFC) interface, and/or a PCI express interface.

In the illustrated example, one or more input devices922are connected to the interface circuit920. The input device(s)922permit(s) a user to enter data and/or commands into the processor912. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint, and/or a voice recognition system.

The processor platform900of the illustrated example also includes one or more mass storage devices928for storing software and/or data. Examples of such mass storage devices928include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

The machine executable instructions932ofFIGS. 5-8may be stored in the mass storage device928, in the volatile memory914, in the non-volatile memory916, and/or on a removable non-transitory computer readable storage medium such as a CD or DVD.

FIG. 10is a block diagram of an example processor platform1000structured to execute the instructions ofFIG. 5to implement the compiler130ofFIG. 1. The processor platform1000can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a gaming console, a personal video recorder, a set top box, a headset or other wearable device, or any other type of computing device.

The processor platform1000of the illustrated example includes a processor1012. The processor1012of the illustrated example is hardware. For example, the processor1012can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor1012implements the compiler130including the example receiver11, the example analyzer231, the example code generator215, and the example output217.

The processor1012of the illustrated example includes a local memory1013(e.g., a cache). The processor1012of the illustrated example is in communication with a main memory including a volatile memory1014and a non-volatile memory1016via a bus1018. The volatile memory1014may be implemented by SDRAM, DRAM, RDRAM®, and/or any other type of random access memory device. The non-volatile memory1016may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory1014,1016is controlled by a memory controller.

The processor platform1000of the illustrated example also includes an interface circuit1020. The interface circuit1020may be implemented by any type of interface standard, such as an Ethernet interface, a USB, a Bluetooth® interface, an NFC interface, and/or a PCI express interface.

In the illustrated example, one or more input devices1022are connected to the interface circuit1020. The input device(s)1022permit(s) a user to enter data and/or commands into the processor1012. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint, and/or a voice recognition system.

One or more output devices1024are also connected to the interface circuit1020of the illustrated example. The output devices1024can be implemented, for example, by display devices (e.g., an LED, an OLED, an LCD, a CRT display, an IPS display, a touchscreen, etc.), a tactile output device, a printer, and/or speaker. The interface circuit1020of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip, and/or a graphics driver processor.

The processor platform1000of the illustrated example also includes one or more mass storage devices1028for storing software and/or data. Examples of such mass storage devices1028include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, RAID systems, and DVD drives.

The machine executable instructions1032ofFIGS. 5-8may be stored in the mass storage device1028, in the volatile memory1014, in the non-volatile memory1016, and/or on a removable non-transitory computer readable storage medium such as a CD or DVD.

From the foregoing, it will be appreciated that example methods, apparatus and articles of manufacture have been disclosed that improve dynamic power allocation to a processor system and memory. The disclosed apparatus, systems, methods, and articles of manufacture improve the efficiency of the processor system, memory, and other associated circuitry, such as system-on-a-chip circuitry by leveraging compiler meta-data to dynamically extract, update, and tailor power allocation for one or more accelerators, memory, and/or other circuitry for a particular workload or set of workloads to be executed. In certain examples, a deep learning hardware accelerator apparatus is improved through modification of the compiler and addition of a power manager to build and update a power table of frequencies for power allocation to be leveraged by a power controller to dynamically allocate and update allocation of power frequencies to circuitry as workload is added and/or subtracted from the apparatus, as a ratio of compute to memory tasks changes within a given workload, etc. Certain examples improve an accelerator apparatus and its associated compiler by providing an ability to analyze a workload and changes or transitions in work ratio within the workload to allocate power to the workload and change the allocation of power during the workload in a manner previously unavailable. The disclosed methods, apparatus and articles of manufacture are accordingly directed to one or more improvement(s) in the functioning of a computer and/or other processor.

Examples apparatus, systems, methods, and articles of manufacture for managing power of deep learning accelerator systems are disclosed herein. Further examples and combinations thereof include the following.

Example 1 includes an apparatus including a power manager and a power controller. The power manager is to: generate a power table to allocate power frequencies between an accelerator and memory based on meta-data, the meta-data generated by compiling a first workload to execute on the accelerator and the memory, the meta-data indicating a ratio of compute tasks involving the accelerator and bandwidth tasks involving the memory in the first workload; update the power table based on a request to at least one of add a second workload or remove the first workload; and determine an index into the power table. The power controller is to: determine a power consumption based on the power table; determine whether to update the index based on a power budget and the power consumption; and allocate power to the accelerator and the memory according to the power frequencies at the index of the power table.

Example 2 includes the apparatus of example 1, further including the memory and the accelerator.

Example 3 includes the apparatus of example 2, wherein the accelerator is one of a plurality of accelerators, the memory and the plurality of accelerators to execute a plurality of workloads.

Example 4 includes the apparatus of example 3, further including a workload management processor.

Example 5 includes the apparatus of example 1, wherein the apparatus is implemented as a system on a chip.

Example 6 includes the apparatus of example 1, further including a compiler.

Example 7 includes the apparatus of example 1, wherein the ratio defines a dynamic voltage and frequency scaling transition point.

Example 8 includes the apparatus of example 1, wherein the workload includes an artificial intelligence workload.

Example 9 includes the apparatus of example 8, wherein the artificial intelligence workload is to implement one or more inference layers of a deep learning neural network.

Example 10 includes the apparatus of example 1, wherein the power controller is to adjust the power allocation to the accelerator in response to a relative frequency change request from the accelerator.

Example 11 includes the apparatus of example 1, wherein the power controller is to: decrement the index when the power consumption is above the power budget and increment the index when the power consumption is below the power budget.

Example 12 includes the apparatus of example 1, further including a power table memory to store the power table.

Example 13 includes the apparatus of example 1, wherein the power manager includes: a power management processor to process the meta-data; a power table generator to at least one of generate or update the power table using the processed meta-data; and an index determiner to determine the index to provide to the power controller.

Example 14 includes the apparatus of example 1, wherein the power controller includes: a consumption calculator to determine a power consumption of the apparatus based on the power table; a comparator to compare the power consumption to a power budget; and a selector to select a level in the power table based on the index and the comparison of the power consumption to the power budget to allocate power to the accelerator and the memory according to the power frequencies at the selected level of the power table.

Example 15 includes at least one non-transitory computer readable storage medium including computer readable instructions that, when executed, cause at least one processor to at least: generate a power table to allocate power frequencies between an accelerator and memory based on meta-data, the meta-data generated by compiling a first workload to execute on the accelerator and the memory, the meta-data indicating a ratio of compute tasks involving the accelerator and bandwidth tasks involving the memory in the first workload; update the power table based on a request to at least one of add a second workload or remove the first workload; determine an index into the power table; determine a power consumption based on the power table; determine whether to update the index based on a power budget and the power consumption; and allocate power to the accelerator and the memory according to the power frequencies at the index of the power table.

Example 16 includes the at least one non-transitory computer readable storage medium of example 15, wherein the workload includes an artificial intelligence workload and wherein the instructions, when executed, cause the at least one processor to adjust the power allocation to the accelerator in response to a relative frequency change request from the accelerator.

Example 17 includes the at least one non-transitory computer readable storage medium of example 15, wherein the instructions, when executed, cause the at least one processor to: decrement the index when the power consumption is above the power budget and increment the index when the power consumption is below the power budget.

Example 18 includes a method including: generating, by executing an instruction with at least one processor, a power table allocating power frequencies between an accelerator and memory based on meta-data, the meta-data generated by compiling a first workload to execute on the accelerator and the memory, the meta-data indicating a ratio of compute tasks involving the accelerator and bandwidth tasks involving the memory in the first workload; updating, by executing an instruction with the at least one processor, the power table based on a request to at least one of add a second workload or remove the first workload; determining an index into the power table; determining a power consumption based on the power table; determining whether to update the index based on a power budget and the power consumption; and allocating power to the accelerator and the memory according to the power frequencies at the index of the power table.

Example 19 includes the method of example 18, wherein the workload includes an artificial intelligence workload, and further including adjusting the power allocation to the accelerator in response to a relative frequency change request from the accelerator.

Example 20 includes the method of example 18, further including: decrementing the index when the power consumption is above the power budget and incrementing the index when the power consumption is below the power budget.

Example 21 includes an apparatus including: memory including machine reachable instructions; and at least one processor to execute the instructions to: generate a power table to allocate power frequencies between an accelerator and memory based on meta-data, the meta-data generated by compiling a first workload to execute on the accelerator and the memory, the meta-data indicating a ratio of compute tasks involving the accelerator and bandwidth tasks involving the memory in the first workload; update the power table based on a request to at least one of add a second workload or remove the first workload; determine an index into the power table; determine a power consumption based on the power table; determine whether to update the index based on a power budget and the power consumption; and allocate power to the accelerator and the memory according to the power frequencies at the index of the power table.

Example 22 includes the apparatus of example 21, wherein the workload includes an artificial intelligence workload and wherein the instructions, when executed, cause the at least one processor to adjust the power allocation to the accelerator in response to a relative frequency change request from the accelerator.

Example 23 includes the apparatus of example 21, wherein the instructions, when executed, cause the at least one processor to: decrement the index when the power consumption is above the power budget and increment the index when the power consumption is below the power budget.

Example 24 includes system-on-a-chip including: a power manager to: generate a power table allocating power frequencies between an accelerator and memory based on meta-data, the meta-data generated by compiling a first workload to execute on the accelerator and the memory, the meta-data indicating a ratio of compute tasks involving the accelerator and bandwidth tasks involving the memory in the first workload; update the power table based on a request to at least one of add a second workload or remove the first workload; and determine an index into the power table; and a power controller to: determine a power consumption based on the power table; determine whether to update the index based on a power budget and the power consumption; and allocate power to the accelerator and the memory according to the power frequencies at the index of the power table.

Example 25 includes the system-on-a-chip of example 24, further including the memory and the accelerator.

Example 26 includes an apparatus including: means for generating a power table allocating power frequencies between an accelerator and memory based on meta-data, the meta-data generated by compiling a first workload to execute on the accelerator and the memory, the meta-data indicating a ratio of compute tasks involving the accelerator and bandwidth tasks involving the memory in the first workload; means for updating the power table based on a request to at least one of add a second workload or remove the first workload; means for determining an index into the power table; and means for allocating power among the accelerator and memory using the power table to execute the first workload by: determining a power consumption based on the power table; determining whether to update the index based on a power budget and the power consumption; and allocating power to the accelerator and the memory according to the power frequencies at the index of the power table.